An investigation is under way after prison officers battled with inmates at a young offenders institution.