First wife Catherine Hodnett was a US Army nurse he met while touring in India. Married in 1943, they returned to the England after the war, but they separated and she moved back to the States. They had one son, Robert, born in 1947.
Had two daughters and a son by second wife British actress Sonia Williams. He was survived by seven grandchildren in all.