In 1864, Charles Hackley married Julia Ester Moore. They adopted one son, Charles Moore Hackley, in 1898, and raised a foster daughter, Erie Caughell (Hackley). Julia Hackley joined her husband in his philanthropy work. Julia Hackley had the Muskegon Museum of Art built and endowed. Julia also was instrumental in starting the Julia Hackley Poor Fund and the Julia E Hackley Educational Fund Endowment. These funds are still actively supporting Muskegon today.