Harry believes the removal of his security was a calculated effort to force him and Meghan back — a realization he tells PEOPLE "was difficult to swallow"