The first thing to say is that this is from Judaeo-Christian linguistic culture.
The most obviously patient Biblical character is Job - see the Epistle of St James in the New Testament, ch 5 v 11
Behold, we count them happy which endure. Ye have heard of the patience of Job.
In the Old Testament, Job's faith is tested by the loss of everything he has, but he never loses his faith, and patience sees him through.
The phrase 'the patience of the saints' appears twice in the book of Revelation in the New Testament, referencing endurance under persecution, and this is most likely what we think of when we use this phrase. Note that the phrase 'the patience of a saint' implies that patience is a virtue applicable to all and any of the saints, an essential virtue of sainthood if you like.
Most likely this is a case of what is easier to say - 'the patience of a saint' rolls off the tongue more easily than 'the patience of the saints', even though the latter sounds more impressive.
There may be instances where saints have given way to impatience. Perhaps Saint Nicholas giving coal instead of toys to naughty children. Perhaps St Francis getting so fed up with being ignored by people that he goes off and preaches to birds. But the most impatient of the lot, way above the saints, has to be God - that story of the flood is about someone just losing it in a big, big way.
