"Although we went through some tough times, it’s the good, wonderful memories of a sweet, kind man that matter," wrote Elizabeth Hurley