Dr. Gray married Amanda Thorne in 1870. She gave birth to Hugh Barr Gray in 1872, James M. Gray Jr. in 1879, and a son Charles who lived only six months. Unfortunately, Mrs. Gray died in childbirth in 1875 while giving birth to their fifth child who did not survive either.