But do remember that St Patrick wasn't Irish at all. He was an English boy, stolen by Irish pirates and sold into slavery in Ireland.
De-lurking briefly to correct this... St Patrick was a Romano-Briton. There were no English in Britain at the time he lauched his Irish mission. There was no English language, and certainly no English identity. The Angles, Saxons and Jutes that make up the English (an identity that only established itself when the Franco-Norman ruling dynasty in England lost its territories in France) were spread across Germany and Denmark at the time.
But this is mostly just laziness. When Patrick didn't do what he was told, I'm sure that his masters made no effort to learn his language. They just shouted at him louder in Gaelic.
Patrick would have spoken Gaelic or Latin as his first language. The Irish would have been no more difficult to understand than a Californian to a Noo Yawker. The upper echelons of Irish society may even have spoken Latin. All the best Tiarnan