Following the conquest, Norman control in Colchester was soon manifested by the erection of a royal castle in the town. The square Norman keep was constructed around AD 1078, on the foundations of the early Roman Temple of Claudius. The castle appears to have been controlled by Eudo Dapifer, who transformed Colchester through the foundation of St John's Abbey and the leper hospital of St Mary Magdalene.
Colchester received its first royal charter in the late 12th century, although the wording of the charter and the existence of a Norman moot hall suggests that Colchester had been largely autonomous before then. At around the same time, a port was established at the Hythe, to generate income for the borough, and it provided direct access to the flourishing markets of northern Europe. By the late 12th century Colchester had attracted a significant Jewish population, and it is likely that many of the prestigious stone houses around the market place were inhabited by this community. A synagogue was recorded in the town in 1268. Two very large coin hoards were discovered in Colchester in the 20th century, and these may be associated with the arrival of Jews in the town. The first hoard was found in 1902 beneath the premises of the new London and Country Bank in the High Street, and the second, which, at the time of its discovery in 1969 was the largest medieval coin hoard found in Britain, included over 14,000 pennies of Henry III. The lead cannisters in which these hoards were discovered are a type used by those handling large quantities of cash and this, combined with the fact that the coins are in extremely good condition, suggests that these hoards were probably owned by money-lenders.
By the early 14th century, Colchester's early medieval growth appears to have peaked. A population of between 3,000 and 4,000 inhabitants is recorded in 1312. By the middle of the 14th century Chelmsford was becoming the administrative capital of Essex, and when the plague arrived in Colchester in 1349 the population had already declined.
However, Colchester's fortunes were about to improve, as in the mid-14th century there was an influx of craftsmen and traders from the Low Countries, which helped invigorate the cloth industry. Colchester's speciality was a medium-quality russet woollen cloth which was exported to markets across Europe. From the 1390s, Colchester's cloth industry was dominated by German Hanseatic merchants, and there activity peaked in the mid-15th century when the town had several Hanseatic residents who hired Dutch ships to take cloth from Colchester as far afield as Russia. The Dutch community founded a church in Colchester in 1563, although the location of the church is unknown until 1677 when it was situated on the corner of Head Street and Church Street.