Worried about his party's prospects in the Lok Sabha elections, Samajwadi Party president Mulayam Singh Yadav Saturday urged his partymen not to punish him for the misdeeds of his ministers, legislators and MPs.