Was the .45 you shot steel or polymer?
Steel in sub compact.
Okay, don't sweat the .40 recoil. .40S&W doesn't actually have more recoil than a .45. That's one of those "widely known facts" that isn't quite true. You will get people to say that the .45 recoils less, and I think the most common thing you will hear people say is that the .40 is sharper or snappier while the .45 "is more of a slow push." That is so widely said and believed because it IS in fact widely experienced, but that's only because .45 is most often shot from a 2.5 pound all metal 1911 while .40's are most often shot from a 1.4/1.5 pound polymer-framed gun. Bullets don't recoil against your hand. Guns recoil against your hand, and the weight of the gun is HUGE in mitigating that energy, so talking about how much a caliber recoils doesn't really work because it's different in different pistols. And that's not to mention the differences in grip angle, grip shape, bore axis, spring weights, etc., etc..
An average 180 grain .40S&W shot out of a 1.5 pound Glock 22 has about 30% more recoil energy than an average 230 grain .45 shot out of a 2.5 pound 1911, but that same .40 out of the same gun has about 25% less recoil energy than the same .45 shot out of a 1.6 pound Glock 21. If you get the gun weights about the same, the .45 recoils harder than the .40. In your case, the compact .45 you have fired probably weighed 1.5/1.6 pounds while the CZ P-07 weighs 1.7, which means that your future CZ P-07 in .40 will recoil less than the .45 you have already fired.
Is .40 too much? It's not so strong as to be unpleasant. It's not too strong to have extended practice periods with. It's not as pleasant as the 9mm (because 9mm is a kitten), but it's not unmanageable. If you feel more comfortable with the extra energy of a .40 for home defense, that comfort is valuable, and I would say stick with the .40. I am more comfortable with the extra rounds you get out of a 9mm, but that's me. Also, this is all based on your being the shooter. If you have a wife or girlfriend or child who might be shooting it, as well, .40 still isn't out of control, but I would at that point switch my suggestion to 9mm because it's likely the extra recoil would make a difference to them.