In the early days, most of the prisoners were sentenced to hard labour on the Island and spent their days collecting shells or hacking stone from the quarry. Some chopped firewood, tended the vegetable garden, looked after the sheep or slaughtered seals for the production of oil. Prisoners who were not shackled and chained were allowed the freedom to move around after hours. They were also allowed to keep private possessions and obtain liquor and tobacco.
Around this time, the corporals and sergeants were reportedly heavy handed and brutal when inflicting punishment on the forsaken convicts. In 1723, a prisoner called Rijkhaart Jacobsz was beaten by a sergeant for not having raised his hat to the sergeant. In 1735, when Jacobz and another prisoner called Blank confessed to homosexual practices, they had weights tied to their bodies and were taken out to sea, somewhere between the Island and the Cape, where they were thrown overboard to drown.