John Montague who came to the Cape in 1843 is said to be the brainchild of this idea and is reported to have said in 1844 that ‘…I would strongly recommend … the expediency of removing the leper and pauper establishments of Hemel-en-Aarde (near Caledon) and Port Elizabeth to Robben Island, also the pauper establishment of Cape Town, and the lunatics at present confined in the Somerset Hospital at Cape Town which are…..at heavy annual expense to the public.’
The initial scope of the Infirmary was reduced when new facilities were built on the mainland for mentally ill and chronically sick people, leaving only people with leprosy on the Island. The practice of segregating people on the mainland continued with the patients who were sent to the Island. The segregation of patients on the Island was based on race, class and gender and .’ was the precursor to the formal policy of segregation in mainland hospital wards from the 1860s.