Prince Harry had reportedly told Queen Elizabeth he wasn't "keen" to appear on the Buckingham Palace balcony at Trooping the Colour.