is there a trick to prevent this?

Yes. It's called a carry case. Most IEM-s come with one, small enough to fit in a pocket, big enough to hold the phones, extension cable, spare silicon tips, SIM extraction tool, a few uSD cards, USB stick with Tails or Mint installed and other tiny shit you might need.