The number of prisoners on the Island grew throughout the 18th century. Between 1730 and 1750 the number fluctuated between 40 and 60 prisoners. By 1789 the figure had risen to 134, of which 103 were Indiaanen and the majority of the 31 European prisoners were military offenders.
From 1774 to 1879 most of the political prisoners sent to the Island had rebelled and fought against colonial expansion east and north of the Cape Colony. The colonisers termed this conflict between themselves and the indigenous inhabitants (Khoikhoi, IsiXhosa, and Korana etc.) as “Frontier Wars”, whereas the indigenous people.saw them as “Wars of Dispossession”.
The black men imprisoned for their resistance to colonial expansion were mainly Khoi, Xhosa and Korana leaders.
Later prisoners incarcerated on the Island were from all political organisations opposed to apartheid, and included almost all their prominent leaders.