Old buildings that have been dismanteled or plundered still show some remains on the surface. Usually foundations pop out of the ground after walls and floors have been stripped, since land is exposed to rain and, at first, vegetation has not grown yet (more in case of Whitewalls).
This is Vindolanda, a Roman camp in Northumberland. As you can see unexcavated areas show the contours of the constructions below.
Turruñuelos, Córdoba. An old arab estate, crops grow differently where walls are below ground, since soil is less deep
I may have mentioned it elsewhere on the forums, but a good idea for Whitewalls would be to make a few "newer" buildings or additions to buildings out of the white stone which would have been salvaged from the castle. Maybe even a whole village using a white Vale palette?
Canon says that Lord Bloodraven ordered Sir Ambrose Butterwell dismanteling Whitewalls. Most likely this task was made by the demense that the Butterwells retained at the time, or the population that lived on the surroundings, fast but in a sloppily manner. So it's very probable that this peasants used materials they extracted from the site in their houses or villages nearby: walls with carved stones combined with more rough rubble work, maybe a small sept with 3 or 4 mismatched columns or marble floors...
Basílica de Santa Eulalia, Mérida. Part of a roman temple (pediment and columns) was used to make the porch to this Christian temple. Examples like this are very common all around the Roman world, this one is just the first that comes to mind.
EDIT: Added more pics of reused buiding materials
1: Roman columns used as foundations. Notre Dame Cathedral, Vaison-la-Romaine.
2: Pre-Roman stone buckets and zoomorphic statue used in the medieval wall of Ávila.
3: Roman inscription at the base of the Giralda, Sevilla.
4: Door using Roman brick at the Trinity Church. Colchester.
5 & 6: Water deposit (aljibe) of the muslim alcazaba (~fortified enclosure/castle) of Mérida, using Visigothic carved stones.
7: Roman bathtub in porphyry used as Baptismal font. Cathedral of Milan.