Mmmh, knowing that Broom house is unknown, with modest lands, not close to any major road, I wouldn't go for anything too big. Just from the picture you've linked, castle's size seems good enough.
Otherwise, as Tham said, I'd rather go for scattered small settlements, main one shouldn't be more...