The farther above the bore you mount the dot, the harder it is to acquire. There will always be people who chime in and disagree, most of them are target shooting. It absolutely makes a difference when shooting quickly from a draw, that's why everybody tries to mill as deeply as possible.
If you were coming from shooting an open gun, it might not be such an adjustment.