Aren't all parts of a shirt suitable as a cleaning cloth?
Good question! No! Most t-shirts work well, but button-up shirts are often not soft, pliable, or absorbable enough to really function well for glasses cleaning (IMO, anyway).
You aren’t likely to scratch the glasses (glass is very hard), but you kinda just move the smudges around and make them worse with certain fabrics. I want either a nice mostly-cotton blend, or a nice thick microfiber cloth (microfiber is not created equal, and some are practically useless).
Also although you will have a hard(-ish) time scratching the glass by using a good shirt if you have any kind of coating on your glasses (such as a blue filter or anti-reflex coating or whatever) you CAN easily scratch those (usually)!
I (almost) never use a t-shirt to clean my glasses, but once tried the internal lining of my jacket since it looks like a coarser version of the cloth that came with my glasses. The result wasn't the same but it was still good.