I ended up taking it to a gun smith who said a lot more of the pistol needs to come apart to take the decocker off because the trigger bar needs to come off too.
I can't tell exactly what he is doing in the video though but he put his screwdriver between the spring and ejector and said he turned something there, what I don't know.