There's not much of a risk of getting sick by your own germs, relatively speaking. I think the primary reason for washing your hands after using the restroom is that it's gross and you want to get it off your hands before having contact with others. That, and you ought to wash your hands for health reasons at some point anyways, so you might as well do it then. So I guess unless you're going to be touching your belt much throughout the day, it should be fine.
Lots of food borne illnesses are spread through poop. I think it's the most common form of food borne illnesses from restaurants, at least it was 8 years ago when I did a servsafe course.