While retired British soldiers on half pay and loyalists brought most of those enslaved to St John’s Island in the last few decades of the 18th century, the first enslaved people were brought half a century earlier by French entrepreneurs including Pierre De Roma of Three Rivers. In 1786, Lieutenant Governor Fanning, who replaced Patterson as Lieutenant Governor, brought several enslaved persons including Mary Moore (Polly) and Davy and Kesiah Shepard to the Island, while Colonial Robert Gray brought his four slaves including Freelove Haszard a few years later. Estimates are that in the late 1700s, there were a couple of hundred enslaved people on the Island. Freelove was charged with theft from Col. Grey and sentenced to be executed. As a result of the demands for clemency by over two dozen colonial women, Freelove was given a reprieve and instead ordered to be deported, a common punishment of felons in Britain and her colonies during this period, authorized under the Elizabethan Transportation Act. Fanning sold Mary Moore (Polly) to Willian Creed who freed her so she and Creed’s former slave Dembo Sickles could marry since Sickles refused to marry a while woman. In 1799, Fanning freed his two remaining slaves, Davy and Kesiah Shepard and provided them with a farm in the Three Rivers area before he was transferred to a new position. By 1800, there were few if any remaining enslaved persons on the Island and over the first half of the 19th century, the Island's black population largely disappeared as a result of intermarried or moving off Island

In 1773, Robert Clark, a member of the Religious Society of Friends, bought 18000 acres of land near the north shore of St. John’s Island (near New London, Prince Edward Island) with the goal of establishing a ‘Friendly’ settlement. The group apparently emigrated to the Island from the American colonies to establish a sanctuary and to avoid the American Revolutionary War. These dedicated and industrious settlers initially thrived through hard work and strong community, but their efforts proved unequal to the harsh realities of life in this northern climate,. By the late 1770s, the community had begun to decline. By the mid 1790s, the Quaker community had largely disappeared from the Island, although some families in the area continue to claim to be descendants of these original Quakers.

Captain John MacDonald (Iain MacDhomhnaill in Gaelic) through the support of the Scottish Catholic Church acquired lot 36 from Chief Justice Montgomery as a sanctuary for highland Catholics facing religious persecution under the British in Scotland. In 1772, a couple of hundred Catholic settlers lead by Captain John’s brother Donald and other their family members ​including Helen (Nellie) arrived on St John’s Island. Captain John arrived in 1773, one of the few landlords to move to the Island, Captain John established Glenaladale on the shores of Tracadie Bay. He remained on the estate for three years before being enticed to join the British army and was stationed in the American colonies to help secure the allegiance of Scottish settlers there. After being discharged in 1783 and spending time in Britain, MacDonald returned to the Island in 1792. In this absence, Nellie managed much of the estate business while dividing her time between Glenaladale and Charlottetown.

There is little question that Patterson was a philanderer. That said, this type of behavior was not uncommon nor socially unexceptable before the onset of the strict Victoria period. As well as his wife Hester, Patterson brought his mistress Suzanna Torriano with him to the Island and later granted her an island in Charlottetown harbour. Patterson's affair with Sarah Stewart resulted in Peter Stewart banished his young wife from their home. Patterson paid for Sarah and her children’s voyage to Montreal. Although the Stewarts were eventually reconciled, resentment lingered. Once Hester and her two children returned to England, Margaret Hyde became Patterson’s country wife, a union which produced four children. There is no mention in the historical record of Margaret Hyde going to England with Patterson when he returned there in 1789 or if he reunited with his wife Hester in London as both are unlikely.