@EStoop exactly. My suggestion, which is in the main docs I linked, was to have empty fields/farms within the city walls, as well as abandoned buildings. The clearest historical parallel is post-1204 Constantinople. Brienne doesn’t note this decay, so it’s likely in the space between the city gates and the main streets. I’m unsure whether the decline should automatically translate to poor upkeep of the defenses, as it’s not mentioned in AWOIAF where Duskendale’s defenses are discussed, and despite all the town is still pretty rich and is consistently mentioned alongside the big cities of the realm (have a look at my research).