The keep became ruinous early in the C17. Precise date of destruction of bailey walls, outer gates and filling up of most of the moat is uncertain. In 1693, and the succeeding years an attempt was begun to demolish the walls but only extended to the upper storeys. Around the middle of the C18, the existing structure on tops of the NE and SW turrets were added. At the time of the RCHM survey there were modern prison and museum buildings in the S part of the keep. In the S wall of the room W of the chapel sub-vault (second stage of the keep) is a reset late C15/early C16 fireplace. Flanking the fireplace are early C17 carved oak terminal figures supporting carved shelf and an overmantel. Woodwork is not `in situ' and the design is incomplete. <1> "The present top-structures of the keep belong to the mid C18, the time when the keep stood in the grounds of Hollytrees and was considered a somewhat brobdignagian garden ornament." <2> Considerable restoration in the mid C18 for Charles ______. The dome over the present staircase was added by James Deane in 1760.<3><4>

Groundworks in the south-east corner of the Castle Bailey revealed a post-medieval soil layer and post-medieval pit. A small quantity of human bone was found within the post-medieval soil layer. A number of inhumation burials are known from within the bailey, possibly dating to the 16th and 17th century when the keep was used as a gaol. <5>

Essex Archaeological Society outing to Colchester Castle in 1907.<6>