Jay built this house for William Scarbrough in 1819. President Monroe made it his Savannah White House during his tour of the South in May 1819. In the mid-nineteenth century, an original trompe l'oeil ceiling in the vault above the atrium was destroyed when a third story was added. Although the intrusive story was removed in the 1970s during a restoration, the original appearance of the vault with its clerestory lighting could not be determined. Contemporary descriptions only say that it was like looking into the sky. The following two photographs depict the atrium in 1988.

The vault was modified when the building was refitted for the Ships of the Sea Museum in 1998.