Agar is the second son of Henry Agar, a former MP for Gowran, and Anne Ellis, and is probably born at Gowran Castle in County Kilkenny. On March 20, 1760 he marries Lucia Martin, widow of Henry Boyle-Walsingham. Together they have three children; Henry-Welbore Agar-Ellis (b. January 22, 1761), John Ellis (b. December 31, 1763), and Charles-Bagnell (b. August 13, 1765).
Agar is made a Baron Clifden on July 27, 1776 and Viscount Clifden on January 12, 1781 and on August 13, 1794 becomes Baron Mendip. His younger brother is Charles Agar, first Earl of Normanton (1736–1809), who becomes the Church of Ireland Archbishop of Dublin.
In addition to being a Member of Parliament (MP) for Gowran, for which he sits three times, from 1753 to 1761, again from 1768 to 1769 and finally from 1776 to 1777, Agar controls three other borough seats through the strength of his family holdings. He represents Kilkenny County between 1761 and 1776 and Thomastown between 1768 and 1769. He holds the post of joint Postmaster General of Ireland with William Ponsonby, 1st Baron Ponsonby between 1784 and 1789.
James Agar dies on January 1, 1789. His eldest son, Henry-Welbore Agar-Ellis, becomes 2nd Viscount Clifden and Baron Mendip.
(Pictured: Arms of Agar-Ellis: 1st and 4th quarter: a cross sable charged with five crescents argent for Ellis; 2nd and 3rd quarter: azure with a lion rampant for Agar)