But you can turn off sealed sender messages from anyone, so they’d have to already be a trusted contact

The setting to mitigate this attack (so that only people who know your username can do it, instead of anybody who knows your number) is called Who Can Find Me By Number. According to the docs, setting it to nobody requires also setting Who Can See My Number to nobody. Those two settings are both entirely unrelated to Signal’s “sealed sender” thing, which incidentally is itself cryptography theater, btw.