Headphones won’t break your ears any worse than loud speakers do, and noise cancelling headphones are actually a solution to blocking loud noise (e.g. construction), or for people who get overstimulated.

In fact, since headphones can block out sound you might even be listening at a lower volume than if you were trying to drown out sound with speakers, assuming the headphones have any noise cancellation (even just muffled cups). Even just competing with ambient noise can cause us to raise volumes more than necessary.