"I was like, here we go, another Hammer man and something that's being said about them," Armie Hammer's aunt, Casey Hammer, told The Daily Beast