Shiv Sena on Thursday defended its protest against ghazal concert of Pakistani vocalist Ustad Ghulam Ali, saying "we can't have fun when we have encounters going on everywhere".