Hilary Duff says she is close friends with her ex Joel Madden and his wife Nicole Richie and loves hanging out with them.