Scores of Congress workers today staged a violent protest outside BJP leader Subramanian Swamy's residence in Nizamuddin area of South Delhi demanding an "unconditional apology" for allegedly making derogatory remarks against Priyanka Gandhi and top party leaders.