James Brade Sword was born in Philadelphia, but from about 1840 to 1849 he lived in Macao with his family while his father was in the tea and silk business in Canton, China. The family then returned to Philadelphia, where Sword attended public schools and became a civil engineer. He became a professional artist in 1862 but gave up this career in 1864 to begin manufacturing silverplated wares. Sword had had resumed his career as an artist by the autumn 1871, when he opened a studio in New York. He made sketching trips to various parts of the country, including the Green Mountains in Vermont, the Allegheny Mountains in Pennsylvania, Bar Harbor, Maine, and Lake George, New York, where he met the artist Asher B. Durand (1796-1886). The two painted together, and Sword learned a great deal about landscape painting from the older artist. Sword visited Conanicut Island, Rhode Island, in 1881, and was inspired by the views. He built a house there in 1883, and spent his summers painting on the shores of the island.

Sword was very active in the Philadelphia art community. He exhibited at the Pennsylvania Academy of the Fine Arts from 1876-1906, at the Centennial Exhibition in 1876, was a founder of the Art Club of Philadelphia, president of the Philadelphia Society of Artists, and president of the Artists' Fund Society. He also exhibited at the World's Industrial Cotton Centennial Exposition in New Orleans in 1883, the American Art Association and the National Academy of Design in New York from 1876 to 1892, the Brooklyn Art Association from 1873 to 1884, and the Art Institute of Chicago from 1895 to 1908.