Between 1806 and 1829, merchant Patrick Duncan built or altered a house on the grounds of Ashley Hall that has been known variously as the Duncan-Nicholson House, The Witte House, and The McBee House. In 1945, Beatrice St. Julien Ravenel drew attention to it as a possible Jay work.