Koch was born in The Bronx, New York City, the son of Yetta (or Joyce, née Silpe) and Louis (Leib) Koch, immigrants from Eastern Galicia. He came from a family of Conservative Jews who resided in Newark, New Jersey, where his father worked at a theater. As a child, he worked as a hatcheck boy in a Newark dance hall. He graduated from South Side High School in Newark in 1941.
He was drafted into the United States Army in 1943, where he served as an infantryman with the 104th Infantry Division, landing in Cherbourg, France, in September 1944. He earned a European-African-Middle Eastern Campaign Medal with two campaign stars, a World War II Victory Medal, and the Combat Infantryman Badge for service in the European Theater of Operations. After V-E Day, because he could speak German, Koch was sent to Bavaria to help remove Nazi public officials from their jobs and find non-Nazis to take their place. He was honorably discharged with the rank of Sergeant in 1946. Koch returned to New York City to attend City College of New York, graduating in 1945, and New York University School of Law, receiving his law degree in 1948.
In 1984, Koch published his first memoir, Mayor, which became a best-seller and was later turned into an Off Broadway musical, Mayor.