I've never shot one with a 357, but when I hunted I loaded up heavy - going with Remington's 180 JHP based on two factors:
1) I read an article some time ago in Handloader discussing heavy 357 revolver bullets on deer. For the life of me I can't find it at the moment, but it was probably written by Brian Pearce.
2) I shoot DAO, and usually limit my shots to 50 yards and since I hunt pretty much exclusively from tree stands in woods, I don't get allot of long shots anyway.
In general, I would go heavier rather than lighter - but's that's me.
Big LSWC's rule