"The place just went to pieces," Matheson recalled of the reaction to Brooks' comment at a "To Be or Not to Be" wrap party.