I have the same wear on my low round count SP01 Phantom. I do not mind the wear so much, but the gun was also consistently ejecting brass (softly) on top of my shooting hat, which was very, very annoying. If a pistol ejects brass hard towards your head, it can be because the brass is being ejected too vertically. But in this case the brass was coming back softly. The markings on the lower inside of the ejection port were a clue that the brass was ejecting too laterally, bouncing off this portion of the ejection port, and then kicking up and back at my head. The brass was not bouncing off a near wall, as I did testing in an open area too. I thought of lowering the port like one would on a 1911, but did not want to risk weakening the slide.
I solved the problem by reshaping the tip of the ejector (NOT the extractor). Note: I did not change the lateral profile of the ejector tip, which slopes down towards the trigger. After field stripping and covering the small moving parts with paper towels, I gently filed down the left sided point on the ejector so it was more square when viewed from above. I wanted to change the angle of the ejector tip so the case would be struck less at the far left edge. This was all I needed to correct the ejection angle and now the brass ejects cleaning off to the right side. No more annoying brass landing on my hat. The finish wear on the inside of the port does not bother me and was the major clue to solving the problem. I may have my gunsmith touch it up with a spot of Duracoat, but it really does not bother me.
The real question is whether or not your brass ejects cleanly or at your head. If the brass comes back at your head, think about reshaping the ejector tip.