The defender echoed comments made by team-mate James Milner, who said he's 'falling out of love' with football