Adam Scott guarded his children, daughter Frankie and son Graham, from his fame when they were little because he did not want them to have the "idea of having to share" him with fans.