Thousands of people took to the streets in rival protests over plans to build a mosque close to Ground Zero.