He was born at Bristol on 8 July 1743, into a Catholic family, and was educated at Scorton school, near Catterick, Yorkshire. He was then sent to the monastic seminary of La Celle in France. He afterwards studied at Paris, Bordeaux, and in the Royal Academy at Turin, devoting special attention to literature and art. The death of his brother, who had devised to him a small estate at Hamsterley in Durham, placed him in independent circumstances. He proceeded to Turin, Genoa, and Florence.
He met in Paris his future wife Martha, daughter of John Baker of Chichester, solicitor to the Leeward Islands, who was being educated at a convent of Ursuline nuns. They were married at Aix-la-Chapelle on 24 March 1767. The couple then settled at Hamsterley, where the husband laid out the estate. They passed the autumn of 1774 and the following months until September 1775 at Bordeaux, and then visited the Pyrenees. Swinburne in the company of Sir Thomas Gascoigne travelled through Spain, returning to Bayonne in June 1776. A manuscript describing his journey was sent to Samuel Henley as editor. It was published in 1779 as Travels through Spain, 1775 and 1776, illustrated with drawings of Roman and Moorish architecture.
On his return to Bayonne in June 1776 Swinburne, with his family, travelled to Marseilles, and a supplementary volume describing the expedition was issued in 1787. They then sailed to Naples, and travelled in the Two Sicilies, where they stayed for 1777 and 1778, and for the early months of 1779. Their return to England was by Vienna, Frankfurt, and Brussels, and they arrived in London in July 1779, but after a few months in England passed once more through France to Italy (March to July 1780) and then until November in Vienna. They formed acquaintance with literati in each country, and received compliments from the Catholic sovereigns. At Vienna Maria Theresa conferred on Mrs. Swinburne the female order of La Croix Étoilée, and the Emperor Joseph II stood godfather to their son of that name. They were in Brussels from February to June 1781, and again crossed to England.
By this time Martha's property in the West Indies had been laid waste. With letters of introduction to the French court, from Vienna, Swinburne went to Paris (1783), and through Marie-Antoinette' s influence obtained a grant of all uncultivated crown lands in the island of St. Vincent valued at £30,000. In February 1785 William Pitt offered half that sum for it, and on receiving a refusal passed through parliament a bill to impose heavy taxation upon the unproductive lands in all the West Indian islands. Swinburne then parted with his interest for £6,500. From September 1786 to June 1788 Swinburne was again in Paris, and high in favour with Marie-Antoinette. His eldest son was enrolled among the royal pages, and placed under the care of the Prince de Lambesc.
Swinburne's later years were clouded by misfortune. His eldest daughter, Mary Frances, married on 7 September 1793 Paul Benfield, whose wealth crumbled away as rapidly as it grew, and Swinburne was involved in the ruin. His eldest son died in a storm on his way to Jamaica in 1800.
In the meantime Swinburne was sent to Paris in September 1796 as commissioner, to negotiate an exchange of prisoners with France, but, in the face of difficulties arising from the capture by the French of Sir Sidney Smith, was unsuccessful, and in December 1797 was recalled to England, In December 1801 he went out to the lucrative post of vendue-master in the newly ceded settlement of Trinidad, and also as commissioner to deliver up the Danish West Indian islands to a Danish official. He died from sunstroke at Trinidad on 1 April 1803, and was buried at San Juan, where his friend. Sir Ralph Woodford, raised a monument to his memory.