The .22 Hornet is plenty loud enough to permanently damage your hearing - even a .22LR will do that. You won't notice it at first but it will be happening. It's a bit like smoking, when we were kids we never thought it would hurt us, now we know better. Hearing damage is similar, it creeps up on you little by little. Buy the best set of muffs you can afford and keep them in your truck.
I recently bought a .223 and family has owned a Hornet 50+ years. I find the .223 is a bit stout for varmints up to fox size if you want to keep skins but it is several steps up from the Hornet. Plus, where I live, .223 is much cheaper than factory Hornet ammo. I've never seen a .204 in use so can't comment.