David Harbour had been on just three dates with Lily Allen when he realised he was in love with her.