There is little question that Patterson was a philanderer. That said, this type of behavior was not uncommon nor socially unexceptable before the onset of the strict Victoria period. As well as his wife Hester, Patterson brought his mistress Suzanna Torriano with him to the Island and later granted her an island in Charlottetown harbour. Patterson's affair with Sarah Stewart resulted in Peter Stewart banished his young wife from their home. Patterson paid for Sarah and her children’s voyage to Montreal. Although the Stewarts were eventually reconciled, resentment lingered. Once Hester and her two children returned to England, Margaret Hyde became Patterson’s country wife, a union which produced four children. There is no mention in the historical record of Margaret Hyde going to England with Patterson when he returned there in 1789 or if he reunited with his wife Hester in London as both are unlikely.