Long before European settlement, the area was home to the Wappinger people, who gave Pound Ridge its name: an indigenous "pound," or enclosure used to hold game on one of the town's many ridges. European settlement began in the early 1700s, with the first permanent settlers arriving around 1718 near what is now Long Ridge Road, most coming from Stamford, Connecticut. A large portion of the area was originally part of the sprawling Cortlandt Manor land grant before Pound Ridge was formally incorporated as its own town in 1788.

The Revolutionary War reached Pound Ridge directly. On July 2, 1779, roughly 300 American rebels were attacked by 200 British soldiers under the command of a young officer named Banastre Tarleton, in what marked his first independent military command, part of a series of raids intended to draw General Washington's army away from the Hudson River.

For much of its history, Pound Ridge's economy centered on agriculture and cottage industries like basket and shoe making. Unlike many of its neighbors, the town was bypassed by the railroads and major highways that reshaped the surrounding region at the turn of the 20th century, an accident of geography that left Pound Ridge more isolated, and ultimately more protected, than almost anywhere else in Westchester County.