There is little question that Patterson was a philanderer. That said, this type of behavior was not uncommon nor socially unexceptable before the onset of the strict Victoria period. As well as his wife Hester, Patterson brought his mistress Suzanna Torriano with him to the Island and later granted her an island in Charlottetown harbour. Patterson's affair with Sarah Stewart resulted in Peter Stewart banished his young wife from their home. Patterson paid for Sarah and her children’s voyage to Montreal. Although the Stewarts were eventually reconciled, resentment lingered. Once Hester and her two children returned to England, Margaret Hyde became Patterson’s country wife, a union which produced four children. There is no mention in the historical record of Margaret Hyde going to England with Patterson when he returned there in 1789 or if he reunited with his wife Hester in London as both are unlikely.
Walter Patterson was first appointed governor in 1769 when St John’s Island he and others successfully argued that it be separated from Nova Scotia. Patterson had a tumultuous tenure as governor, a position later downgraded to lieutenant governor. He returned to England in 1775 to deal with complaints about his actions, remaining there for five years. Back in Charlottetown in 1780, Patterson continued his scheming, including forcing forfeit of grants to landlords who had not met their obligations and then acquiring, along with his officials, huge tracts of this land at an advantageous price in lieu of wages owed. Patterson was responsible for passing in 1781 the Baptism of Slaves Act, preventing the automatic freeing of slaves upon baptism. He actively encouraged Loyalists to emigrate to the Island after the American revolution and granted them land, at least in part to protect himself after his questionable if not illegal seizure of lands from landlords. Unhappy with what they were hearing about the colony’s governing, the British government recalled Patterson to England in the last 1780s and replaced him with Edmund Faming as Lieutenant Governor.
By the early 1780s, the new colonial government on St John’s Island, mostly comprised of opportunists and rogues, was experiencing growing pains. Landlords made slow progress in settling the land granted to them and, without settlers to pay rent to them, were unable or unwilling to pay the quit rent relied on by the colony to cover its costs. There was mounting tension relating to land tenure with settlers demanding the right to own not just rent the land they worked. Events elsewhere added new pressures including Britain discouraging Scots and Irish from emigrating, hoping to maintain a workforce for its cottage industries, and loyalists seeking refuge in British North America after Britain’s defeat in the American War of Independence. The tensions continued largely unresolved until the mid 19 century. While the colonial government succeeded in buying out a few of the absentee landlords on a willing selling-willing buyer basis in the mid 1800s. a final resolution to the absentee landlord problem wasn’t achieved until Confederation. With Confederation, PEI passed a law forcing the sale of absentee landlords lots to the provincial government, a law later upheld by the newly formed Supreme Court of Canada, and the Canadian government provided the funds to buy them out, one of several inducement to PEI to join the new country of Canada.
In the mid-1760s, Samuel Holland, a surveyor from the Netherlands who had emigrated to England, was tasked with making sense geographically of the new British territories in North America and surveying St John’s Island was among his first assignments. Using a few assistants and Mi’kmaq guides, Holland surveyed the Island using measurements and guesswork, creating the now famous Holland map of the 67 lots on the Island. With a nod to his British employers, Holland renamed many locations after English notables, effectively replacing existing Mi’kmaq names for these places and contributing to the erasing of Mi’kmaq presence in these areas. Since there was considerable interest among nobles and former military officials in acquiring land on the Island, Britain adopted a lottery system to allocate the lots to a few score well-placed men. Among the conditions of the grants were settlement requirements and paying quit rent. And while none of the landlords met all the grant conditions, efforts by settlers in the first half of the 19th century to overturn these grants so they could own the land they cultivated were unsuccessful.