Authorities are rushing to ensure the safety of community members in a small beachside town on the Barwon coast after asbestos fragments were discovered in the area.