Wearing a hat strains hair follicles, causing hair to fall out. Good news for men who rely on caps to cover their shiny crowns: They're not causing any harm. Dirty hats can, however, lead to a scalp infection, which in turn accelerates hair loss, so either make sure yours is clean or rotate frequently.