John McCarthy was born in Boston on September 4, 1927 to an Irish immigrant father and a Lithuanian Jewish immigrant mother,  John Patrick and Ida Glatt McCarthy. The family was forced to move frequently during the Depression, until McCarthy's father found work as an organizer for the Amalgamated Clothing Workers in Los Angeles, California.
McCarthy showed an early aptitude for mathematics; in his teens he taught himself mathematics by studying the textbooks used at the nearby California Institute of Technology (Caltech). As a result, when he was accepted into Caltech the following year, he was able to skip the first two years of mathematics.
Receiving a B.S. in Mathematics in 1948, McCarthy initially continued his studies at Caltech. He received a Ph.D. in Mathematics from Princeton University in 1951 under Solomon Lefschetz.
McCarthy was married to Vera Watson, a programmer and mountaineer who died in 1978 attempting to scale Annapurna I.