James Fayrer was the son of James Fayrer senior, a London instrument maker. James junior initially worked for his father. They were distantly related to the well-known instrument maker Edward Troughton, by whom James junior later claimed to have been trained. In November 1820, on Troughton's recommendation, he was appointed as astronomical assistant to Reverend Fearon Fallows*, who was sent to the Cape to establish the Royal Observatory, Cape of Good Hope. James and his sister Betsy travelled on the same ship as the Fallows family and arrived in Simon's Bay on 12 August 1821. As the roads to Cape Town were too poor to risk transporting the delicate and expensive instruments by land, James supervised their transport by ship to Table Bay. He brought with him a good achromatic telescope of his own, which he had received from Edward Troughton before leaving England.
Fayrer received some training from Fallows in making observations, but had no experience in astronomical computing. Unfortunately he proved unsuitable as an assistant, as Fallows found him to be lazy, insolent, and prone to drinking. In May 1822 he married Fallows' maid, Sarah Boodle, who was also rather fond of the bottle, and in November that year he was officially demoted to labourer. However, as part of his duties he occassionally cleaned or repaired the astronomical instruments, for which he was paid a retainer of 100 pounds sterling a year. He lived in a cottage that belonged to the observatory, and set himself up as a watch and instrument maker in Cape Town.
In spite of his earlier problems Fayrer was still listed as "chronometer maker to the Royal Observatory" in the South African Almanac for 1829. He also took some interest in adult science education, for in 1828 he served on the first committee of management of the short-lived South African Mechanics' Institution, an organisation established with the object of promoting applied science among the working classes. In July 1834 he was finally dismissed from his post at the observatory by Thomas Maclear*. His wife died in May the next year.