At the time of the Domesday Survey, Brooke was held in demesne by King William, having been forfeited by Roger, second Earl of Hereford, Lord of the Isle of Wight. During the reigns of Edward I, II and III, Brooke House was owned by the Glamorgan family. In the twelfth and thirteenth centuries it was held with Carisbrooke Castle by the Mackerel family then passing by the marriage of an heiress to the Glamorgans.  John de Glamorgan was granted free warren in Brooke and Mottistone in 1326.

Hulverstone was held as a manor at the end of the thirteenth century by Robert de Glamorgan and John Paslew and by 1346 belonged entirely to William Paslew.  John Roucle, Lord of Brooke, has acquired it before 1428, from which time it appears to have been merged in the manor of Brooke. On the death of Nicholas de Glamorgan without issue in 1362 the manor appears to have been divided among his sisters.  The husband of one of these, Geoffrey Roucle, later called ‘de la Broke’, acquired, in addition to his wife’s share, the shares of three of the other sisters and left this part of the estate to his son John, who was succeeded in 1450 by his son-in-law Thomas Bowerman.

In 1499 Henry VII was entertained by Thomas Bowerman and Joan his wife at Brooke House  (see Early History). A descendant, William Bowerman, reunited the moieties of the manor by purchasing the share of the remaining Glamorgan heiress from her successors in 1566.  Writing in the latter half of the eighteenth century Sir Richard Worsley observes, “the late William Bowerman rebuilt the manor house, which is pleasantly situated in the rich vale.”  At the beginning of the 18th century the manor house was rebuilt and continued in the direct line of this family until sold by William Bowerman to Henry Howe in 1792.  In 1856 it was purchased from John and William Howe by Charles Seely, Esquire. When his son, later Sir Charles, inherited the house in 1887, he added a third storey for the children, with a sewing room, linen room, etc.  At the top of the stairs on the landing there was a fine organ and the family and servants used to gather in the hallway below for a daily service. A travel guide of the early 20th century describes:  In this house ...the mantelpieces are particularly handsome, some being in inlaid marble.  In the dining room is a beautiful black marble fireplace and a most decorative ceiling. When Sir Charles died in 1915, Brooke House was unexpectedly bequeathed to his younger son, John Bernard Seely (Jack). Jack sold it to his brother Charles when he moved with his family to Mottistone Manor and in 1926 Brooke House was inherited by Charles’ son, Hugh.  During World War II the house was requisitioned by the War Office to house Canadian soldiers. After the war the house was converted into flats but today is again under one owner.