There is little question that Patterson was a philanderer. That said, this type of behavior was not uncommon nor socially unexceptable before the onset of the strict Victoria period. As well as his wife Hester, Patterson brought his mistress Suzanna Torriano with him to the Island and later granted her an island in Charlottetown harbour. Patterson's affair with Sarah Stewart resulted in Peter Stewart banished his young wife from their home. Patterson paid for Sarah and her children’s voyage to Montreal. Although the Stewarts were eventually reconciled, resentment lingered. Once Hester and her two children returned to England, Margaret Hyde became Patterson’s country wife, a union which produced four children. There is no mention in the historical record of Margaret Hyde going to England with Patterson when he returned there in 1789 or if he reunited with his wife Hester in London as both are unlikely.

​Walter Patterson was first appointed governor in 1769 when St John’s Island he and others successfully argued that it be separated from Nova Scotia. Patterson had a tumultuous tenure as governor, a position later downgraded to lieutenant governor. He returned to England in 1775 to deal with complaints about his actions, remaining there for five years. Back in Charlottetown in 1780, Patterson continued his scheming, including forcing forfeit of grants to landlords who had not met their obligations and then acquiring, along with his officials, huge tracts of this land at an advantageous price in lieu of wages owed. Patterson was responsible for passing in 1781 the Baptism of Slaves Act, preventing the automatic freeing of slaves upon baptism. He actively encouraged Loyalists to emigrate to the Island after the American revolution and granted them land, at least in part to protect himself after his questionable if not illegal seizure of lands from landlords. Unhappy with what they were hearing about the colony’s governing, the British government recalled Patterson to England in the last 1780s and replaced him with Edmund Faming as Lieutenant Governor.

In the mid-1760s, Samuel Holland, a surveyor from the Netherlands who had emigrated to England, was tasked with making sense geographically of the new British territories in North America and surveying St John’s Island was among his first assignments. Using a few assistants and Mi’kmaq guides, Holland surveyed the Island using measurements and guesswork, creating the now famous Holland map of the 67 lots on the Island. With a nod to his British employers, Holland renamed many locations after English notables, effectively replacing existing Mi’kmaq names for these places and contributing to the erasing of Mi’kmaq presence in these areas. Since there was considerable interest among nobles and former military officials in acquiring land on the Island, Britain adopted a lottery system to allocate the lots to a few score well-placed men. Among the conditions of the grants were settlement requirements and paying quit rent. And while none of the landlords met all the grant conditions, efforts by settlers in the first half of the 19th century to overturn these grants so they could own the land they cultivated were unsuccessful.