I used to think that I didn't need to wash my hands after using the restroom because my entire body gets washed on a daily basis, so touching my dick isn't a reason to wash them.
Don't get me wrong, I still believe that touching my dick doesn't warrant washing my hands. Now though, I wash them when I use the restroom (though sometimes before rather than after) because it's a way to ensure that my hands get washed regularly. Most of the other stuff I touch throughout the day doesn't get washed on a daily basis.