Patrick would have spoken Gaelic or Latin as his first language. The Irish would have been no more difficult to understand than a Californian to a Noo Yawker. The upper echelons of Irish society may even have spoken Latin.
Several authorities, eg the Cathoic Encyclopedia, say that St Patrick became fluent in the language of the Irish while in slavery. Some claim that he was born in Scotland, some say in Wales.
Both Scotland and Wales contained people who spoke Celtic languages. Although it is difficult to determine where Patrick is from, I believe the scholarly working consensus is that he was from the Roman province of Britannia, where the majority of the inhabitants would have spoken a language of Celtic origin. Perhaps my analogy of New York and Californain English was misleading: a truer example would be the relationship of Spanish with Catalan, or Sicilian with Tyrolean. That's to say, mutually intelligible, with difficulty. Traders and slave-traders (such as the slaver who captured Patrick) would have traded with the Roman Empire in Britain and elsewhere, so presumably a lingua franca emerged. No doubt Patrick learned his powerful mastery of Old Irish from his captors. [If you want to read more on the subject, from sources more up-to-date and historically accurate than the Catholic Encyclopedia, try http://www.ucc.ie/~peritia for a jumping off point.]
None support your suggestion that the language of his masters was his native tongue.
The real point here is that the Irish, generally portrayed as victims of the British, were sometimes victims, sometimes villians -- like most everybody else.
I don't deny it for a minute. I had a problem with the way you took the currently existing region known as England and its current (troubled) relations with Ireland, and projected it back into a period of history where an entirely different socio-political scene existed. All the best Tiarnan