He had completed the building of the Castle of Berkeley. He suffered a great deal at the hand of Walter, son of Miles, Earl of Hereford, in the wars between Stephen and the Empress Maud. Roger was deprived of the Manor of Berkeley and other lands about 1152, apparently for refusing to recognize the authority of either side, but was soon restored to the Honour of Dursley.