Luttrell is born in 1713, the second son of Henry Luttrell, of Luttrellstown Castle (whose family had held Luttrellstown Castle and the demesne and adjoining lands since the land had been granted to Sir Geoffrey de Luterel in about 1210 by King John of England) and his wife Elizabeth Jones. His father is a noted commander in the Jacobite Irish Army between 1689 and 1691. He later receives a pardon from the Williamite authorities and is accused by his former Jacobite comrades of having betrayed them. He is murdered when his sedan chair is attacked in Dublin on October 22, 1717.
On October 13, 1768, Luttrell is created Baron Irnham of Luttrellstown in the Peerage of Ireland. As his title is an Irish peerage, he is able to keep his seat in the British House of Commons. He is elevated to the title of Viscount Carhampton on January 9, 1781 and is made Earl of Carhampton on June 23, 1785. He lives at Four Oaks Hall, Four Oaks, Sutton Coldfield, from 1751 to 1766.
- Henry Lawes Luttrell, 2nd Earl of Carhampton (1743–1821)
- John Luttrell-Olmius, 3rd Earl of Carhampton (c. 1745 – 1829), marries the Honorable Elizabeth Olmius and in 1787 by Royal Licence the additional surname of ‘Olmius’ out of respect after death of his father-in-law.
- Temple Simon Luttrell (c. 1738 – 1803)
- James Luttrell (c. 1751 – 1788), naval officer, dies of consumption.
- Thomas Luttrell (died 1766)
- Anne Luttrell (c. 1752 – 1808), marries first in 1765 Christopher Horton, marries second in 1771 to Prince Henry, Duke of Cumberland and Strathearn, brother of King George III of the United Kingdom, without consent, resulting in the Royal Marriages Act 1772.
- Elizabeth Luttrell (1744-1799)
- Lucy Luttrell
Luttrell’s rakish behaviour earns him the nickname “King of Hell,” with “Hell” being a district of Dublin notorious for its brothels. He reputedly starts the courtesan Mary Nesbitt in her career by seducing her.