Sporadic attacks on Muslim houses and shops have increased those fears.