Union Minister Nitin Gadkari threatened to file a criminal defamation case against Congress Vice President Rahul Gandhi on Friday unless he apologised for his comments on External affairs Minister Sushma Swaraj.