Just like people there are skinny, fat, big, small. No correlation between body weight or antler size and age.
Learn how to jawbone age. It is nearly 100% reliable in discerning fawns, yearling, and 2.5+. Beyond 2.5+ you aren't going to be accurate, but you can generally get pretty close, especially if you have a history in a particular area in interpreting wear.
The only truly reliable method is cementum annuli which requires sending a tooth to a lab where they count rings similar to growth rings on a tree.