James Wylie trained as a gardener in the Royal Botanic Gardens at Kew, near London. He came to the Natal Colony in June 1882 to take up a position as the first trained gardener in the Durban Botanic Gardens, under its curator, J.M. Wood*. He was responsible for the horticultural development of thegardens, but also sometimes assisted Wood in collecting plants. By the late eighteen-nineties was increasingly going out to collect for the latter, particularly in Zululand. Wood became director of the gardens in 1903, and Wylie succeeded him as curator from January 1904. In 1913 the gardens were placed under the control of the Durban municipality and became the Durban Municipal Botanic Gardens, with Wylie as curator. He has been credited with introducing the Jacaranda trees along the streets of Durban and Pietermaritzburg.
Wylie was married and had four sons and four daughters. After his retirement in 1916 he remained in Durban and continued to collect plants for the Natal Herbarium (now the National Herbarium, Durban), providing it with more than a thousand specimens. Later he wrote a series of articles on "Palms in the Union of South Africa" in The Sub-tropical Gardener (1930-1931). The plant species Kniphofia wyliei, Pentas wyliei, Jasminum wyliei and Carissa wyliei were named after him by N.E. Brown*, while Erica wyliei was named by H. Bolus*. Plants collected by him went also to the National Herbarium, Pretoria, and to the herbarium of the Albany Museum.