.243 is more than enough punch for a deer. I started out with a .243 as a kid. I went away from it needing to shooting something bigger. I came back to it after I reallized the big guns didn't kill them any deader than the .243.
Traditions only last if you pass them down to your kids!
243 is a great, flat shooting round for deer.
Why are you looking to go smaller though?? Trying to add to the challenge? Want a lighter gun?? If thats what you're after, I have a buddy who makes it a point to shoot a doe every year with a 22-250 rem. I've seen deer he's shot at 200 yards with it and the wound channel is indistinguishable from that of a larger caliber deer rifle. Just gotta use the right bullet.