In any case, if your security depends on malicious actors not discovering your phone number, a generally public piece of information, your have no security to begin with.

I am taking the time to remove my info from the various aggregators, and it is scary the kind of detailed info that exists out there just as public information.

As you say, if you are worried about a phone number being tied to your identity, it’s already public information.