Alrighty, I think I have returned from my little break and want to change the castle again...
I am proposing a river castle in the southwest of the lands (basically due south from the manor hamlet). It will still be small but it'll be better positioned in the lands (not too dense like where it was previously). This also means I can nuke the hill the original castle was on, which is good because it really disrupts the lands, road networks, etc. I am thinking something like chateau de rivau, with a small village outside of it. The village would replace town 2 even though it's roughly in the same spot. I've really struggled with planning the lands around the current castle and it would be much smoother if you grant this change. In place of the castle I would just make a small farming hamlet or whatever, 4-5 houses. If you want maps I can make them in the morning.
-duck