McKelvey is born into a nationalist family in Stewartstown, County Tyrone. He has a keen interest in the Gaelic Athletic Association and the Irish language. He studies as an accountant and gains some of the qualifications necessary for this profession, but never fully qualifies. He works for a time at the Income Tax Office on Queen’s Square in Belfast and later finds work in Belfast’s engineering industry with Mackies on Springfield Road. He joins the Irish Republican Brotherhood and the Irish Volunteers, which after 1919, become known as the Irish Republican Army (IRA). He is a founding member of the O’Donovan Rossa Club, Belfast, founded in 1916 on the Falls Road. Each year the club honours him with a juvenile hurling blitz, an invitational competition which is participated in by clubs throughout Ireland.
McKelvey participates in the Irish War of Independence (1919–1921) against the British, in which he commands the IRA’s Belfast Brigade. On August 22, 1920, he helps to organise the killing of Royal Irish Constabulary (RIC) Detective Oswald Swanzy in Lisburn. The killing itself is carried out by IRA men from Cork, but McKelvey arranges a taxi to carry the assassins to and from the scene and disposes of their weapons. In reprisal for this shooting, 300 Catholic homes in Lisburn are burned out. McKelvey is forced to lie low in Dublin for some time after these events.
In March 1921, the IRA is re-organised by its leadership in Dublin into Divisions and McKelvey is appointed commander of the Third Northern Division, responsible for Belfast and the surrounding area. In May 1921, McKelvey’s command suffers a severe setback, when fifty of his best men are sent to County Cavan to train and link up with the IRA units there, only to be surrounded and captured by the British Army on Lappanduff hill on May 9. In most of Ireland, hostilities are ended with a truce declared on July 11, 1921.
McKelvey is alone among the leadership of the Belfast IRA in going against the acceptance of the Anglo-Irish Treaty. Most of his comrades support Michael Collins‘ assurances that, although the Treaty accepts the partition of Northern Ireland from the rest of the country, this is only a temporary concession which will be dealt with later. McKelvey does not accept this. As a result, he leaves his command as head of the IRA Third Northern Division and joins the Anti-Treaty IRA in Dublin.
In March 1922, McKelvey participates in the anti-Treaty IRA‘s repudiation of the authority of the Dáil, the civil government of the Irish Republic declared in 1919, and is elected to the IRA Army Executive. In April 1922 he helps command the occupation of the Four Courts in defiance of the new Irish Free State. This action helps to spark the civil war between pro- and anti-Treaty factions. McKelvey is among the most hardline of the anti-Treaty republicans and briefly, in June 1922, becomes IRA Chief of Staff, replacing Liam Lynch.
On June 28, 1922, the new Irish Free State government shells the Four Courts to assert its authority over the militants defending it. The Republicans in the Four Courts surrender after two days of fighting and McKelvey is captured. He is held for the following five months in Mountjoy Prison in Dublin.
On December 8, 1922, Joe McKelvey is executed by firing squad along with three other Anti-Treaty militants, Rory O’Connor, Liam Mellows, and Richard Barrett. The executions are ordered in reprisal for the Anti-Treaty IRA’s murder of Sean Hales, a Pro-Treaty member of the Third Dáil.