Most people cripple themselves with insecurity and your story seems to indicate precisely the same root problem. It may be helpful for you to do some research - you'll discover that "sperm competition" is a profound aphrodisiac for many men, so if they trust their partner's love they are excited to have her fucked by other men so that the sex can be red-hot when she comes back to be "reclaimed."

It's nothing whatsoever to do with your fear of abandonment; he's not looking for a way to "pass you on" to someone else - in fact, quite the opposite.

But before you do anything, I strongly recommend sitting with your feelings and trying to see where your abandonment fears and insecurity come from. Unless you address these emotions, they will ultimately undermine every relationship you attempt, sexual or otherwise.