How big are the deer? And, how far will you be shooting?
I remember W. TX (Del Rio) back in the early 70's. A newspaper article, with pictures, of the first day's "kill" brought into the Jolly Meat Packing Plant for processing. Field dressed those bucks were 90 to 95 lbs. I knew they were small. While quail/rabbit hunting when the dogs jumped deer the first thought in my mind was always !!jack rabbits!!, then I'd realize they were really deer. The size of the deer and the height of the sage brush would fool me at first.
Back in WV, the part we lived in, my first buck field dressed at 185 lbs. The biggest one I killed (field dressed) spun the dial past the 200 lb. mark at the checking station (200 lbs. was it's limit). I saw bigger deer. Saw on one time (sneaking away as my B-I-L thought he was sneaking into his stand) that I first thought was some farmer's cow that had gotten loose and was heading across a reclaimed strip mine. Then I saw the horns (I was a couple hundred yds. away). My dad had a big one chase his English setter back to him one day while grouse hunting. He thought it was an elk at first, then realized it was the biggest white tail he'd ever seen (after deer hunting in MI and ND for over 6 years, the biggest white tail he ever saw was back in the woods in WV).
For the little ones I wouldn't be uncomfortable with a .223. For the big ones, a .308 (killed mine with my M1A) or 30.06 (I used to carry the bolt action rifles to the woods but never saw a buck when I had one of them with me).
Bigger deer? Use bigger bullets. My father-in-law killed a lot of bucks in the woods. He preferred 180 grain bullets in his 30.06 semi-auto rifles.
Overkill works (as we used to say in the Army).