From the beginning, Hermitage was seen as a vitally important base for controlling Liddesdale in the Scottish West March and a large stretch of the Border with England. Indeed, the castle's construction, probably by Sir Nicholas de Soules, butler to King William, in about 1240, almost brought the two countries to war. De Soules' fortress was probably a motte-and-bailey castle similar to his ancestors' former stronghold, Liddel Castle, four miles to the south beside Liddel Water at Old Castleton.