With the distributor in place there isn't enough "play" for it to pull up and out of the pump. It is primarily to prevent the problem you described. If you get the shaft up and back in the
oil pump I wouldn't worry about it until you have to pull the pan and address it at that point. I've pulled them out with the distributor when the clip wasn't put on an aftermarket shaft and it had been run for many thousands of miles like that. If you are having problems "fishing" it out, I have had good results with a small diameter but powerful magnet.