Workers say a construction site in Milford Haven has broken an agreement by bringing in foreign workers before local people.