If you haven't already, order yourself a CGW tool steel striker. That significantly improved the feel of my trigger. You won't be disappointed.
Also, if you were shooting straight on a rest but not off hand, I'd suggest trying different finger positions. You might just be pressing the gun one way during the break. Also give some of the other backstraps a try. I've found them to dramatically change the way my grip stays consistent through the break.
Good luck!