"We don't want to impeach a president. We don't want the reality that a president has done something that is in violation of the Constitution," the Speaker of the House of Representatives told host Stephen Colbert on Thursday.