Quitting Dairy Has These br Effects on Your Skin The worst skin irritant among dairy is skim milk br due to the whey protein, hormones and sugars. In fact, skim milk br has more sugar in it br than whole milk. Avnee Shah, Dermatologist Sarika Snell, Dermatologist Of course, you'll need to give it br some time before you see any results. Sarika Snell, Dermatologist The idea of giving up milk br or cheese might seem like an br impossible task to some people, but fortunately there are br plenty of vegan options to try br along your dairy-detox journey.