The oldest parts of Brough Castle date from about 1100, when what is now North West England had just been annexed by William Rufus from the Scottish Kingdom. In 1203 King John gave Brough, along with Appleby and the Lordship of Westmorland, to Robert de Vieupoint, the builder of Brougham. By 1254, the castle was neglected.
Brough, like Brougham Castle, passed to the Cliffords in 1268. Robert Clifford carried out work here as he did at Brougham, building a new hall, and building a semicircular tower, now known as Clifford's Tower, as a residence for himself. When members of the Clifford family came to Westmorland, they usually stayed at Brough Castle. In 1521 fire destroyed much of the castle, and it was not occupied again until Lady Anne Clifford inherited it in 1643, when she undertook restoration work on all the castles she inherited.
