I really like this idea and agree with Stan that Varner may have (relatively quickly) shifted gears from poor border fort to wealthy trading hub during peacetime. The new money could have been used to scrap the old castle for parts to be used by locals and maybe even recycled for the new manor which could make for some interesting builds.
Another idea here, if you want to keep the location of the family and since, knowing you, you're probably not attached to the castle itself:
Perhaps you could just have some sort of manor at the current location and put the (recent) ruins of a castle elsewhere in a more defensive spot?
Maybe this is obvious advice but maybe just adjusting the existing town square and making it/the immediate area more urbanized would address this issue. You get to have a section that feels more like a proper town without removing the unique elements of varners style.yeah sounds reasonable. i also could do w some advice on the current main village, which i think could be more urbanized than it is, but i know people are a fan of the spaced out feel the villages and hamlets have