Ben Whishaw has a "weird relationship" with the Paddington movies because he feels he has almost "nothing to do with" them.