The Prince of Wales has braved torrential downpours at a Buckingham Palace garden party and quipped it was “good weather for swimming”.