Aging deer up through 3.5 years is not an exact science, but it can be done with pretty decent accuracy with practice. Main difference is body size, but also antler mass, body conformity/contour/proportion.
Body size: The 1.5s are equal or less in weight and body proportions to an average doe. By age 2.5, a healthy buck will outweigh all but the biggest does. By 3.5, he will outweigh all the does by a noticeable margin. There are exceptions, of course, with a few individuals out of every 100 or so, but in general this is pretty consistent. Hand in hand with weight goes overall body frame, that is, the overall skeletal mass and dimensions of height, width, and length.
Antler mass: This one can fool you in deer that are genetically inferior or on poor range. Older deer can sometimes have downright scrawny antlers, but this is an exception to the rule. Usually other characteristics (body size, muscle mass, contour, etc.) will give these guys away despite their antlers. By and large the 1.5s will have small diameter main beams, relatively smooth at the base, and usually lack a spread beyond the ears. Number of points is NOT a good indicator of age. I've seen 1.5 ten-pointers and 3.5 fork-horns. But the 10 pointer had spindly main beams and tines and only about a 13" spread, while the fork-horn had fairly heavy main beams and a 16-17" inch spread (both were verified by tooth aging after kill). Quality 2.5s will generally start having some roughage at the bases, and overall mass and spread will improve to a moderate level, but mass will tend to thin some toward the last antler fork. Good 3.5s and older will generally carry mass greater than 4" at the base (measure some mounted heads for reference), feature pronounced burrs at the antler bases, and will carry their antler mass out to the G3s pretty well. These are all general indicators, and may vary quite a bit in a dozen or so individuals out of 100. Antlers are best considered along with other characteristics to determine age.
Body conformity/contour/proportion:
1.5s have a "teenager" look to their muscle mass, bone size, and body proportions. They appear thin-legged, big-eared, big-eyed, narrow, short, and slender. Muscle mass is not yet well-developed, often even lagging behind the average doe. The muscles are long, sharply defined, and taper very gradually to the joints, particularly in the extended reaches of the limbs. Eyes, nose, and ears appear a little offset from the facial contour, making the nose, skull cap, and jaws appear a bit narrow and drawn. Back is straight, body lacks depth from top to bottom, neck shows separate contour from front shoulders, which are distinct from rib cage and loin. Flanks are somewhat drawn and upper buttocks are not yet full and thoroughly round.
2.5s have a "tailback" look, being well-muscled and yet streamlined without excessive bulk. They have more height, length, width, and muscle mass than all but the exceptionally large doe. They have smooth, clean contour lines, and still retain an overall sleek appearance. Their nose and head is filled out, but not blocky, broad, or short in appearance. Ears, eyes, and nose all seem in standard proportion to head and neck size. Bones are moderately heavy, presenting a stronger, more commanding frame than the 1.5s. Muscle groups are more compact and retain moderate mass into the extended limb areas. There is a general "rolling" contour between the separate body areas (head, neck, shoulders, trunk, hind quarters), and the flank and buttocks are full.
3.5s present a "linebacker" appearance. They have larger body proportions than all of the herd does. There is a noticeable depth, heft, and maturity to their overall body conformity. There is a definite depth to their major muscle masses, and their overall contour is squarish with only slight relief between major body areas (head, neck, shoulders, trunk, hind quarters). The head appears full and strong, with a wide, blocky skull. Ears, eyes, and nose appear modest in proportion to the head. Bones have a noticeable heaviness, with strong joints and limbs. The body is tall, long, wide, full, and well-balanced. Rutting neck is thick and well-muscled, fusing athletically with shoulers and head with only minor muscular distinction. Back and belly lines are long and solid with no sagging or swaying evident.
Hope this helps