I think the castle doesn't feel embedded in the landscape, because it isn't.
I think you should concentrate efforts to terraform and place a castle you are content with. Otherwise these issues will continue to snowball-- once you get the village looking right, you'll need to alter it to terraform the castle properly for example. Then when you've got the castle looking right it might mean the village needs changing.
Just because you don't feel it's in a defensible position doesn't mean it can't be made into one. You've got the lake/river one side and land the other, which can easily be made to look more defensive with earthworks and structures.
As Marge says above, the Reach's history is one of blood and battle, so I wouldn't be keen to see a move to a more manorial build-- I've built manors in the Reach and do think they have their place, but not as a primary seat for a lordly house (especially one on the borders of the Reach)
It may be useful for you to draft a few bulletpoints into an action plan against which you can hold yourself loosely accountable, I find that helps in keeping focus on progress within a project.
V