He had one son, Martin H. Pawley, with his high school sweetheart, Martina May Martin, whom he married, divorced, re-married and divorced again. He married Broadway actress Helen Shipman in 1937, and remained with her until her death on 13th April 1984.
He had two younger brothers, William Pawley and Anthony Pawley, both of whom also performed on Broadway and in the movies.
In retirement, he raised and sold championship goats, wrote poetry, and worked part-time as a radio announcer at a local radio station.
Uncle of William Pawley Jr..