Bharatiya Janata Party leader Subramanian Swamy on Thursday said that Raghuram Rajan is not fit to be the RBI Governor and should be removed immediately.