From this answer and DMG 246 quote:

For the purpose of these rules, an object is a discrete, inanimate item like a window, door, sword, book, table, chair, or stone, not a building or a vehicle that is composed of many other objects.

implying structures are considered a collection of objects.

Now, certain spells like Shatter, Delayed Blast Fireball, and Meteor Swarm specifically say that they also damage objects and not just creatures. From the description of meteor swarm:

A creature takes 20d6 fire damage and 20d6 bludgeoning damage on a failed save... The spell damages objects in the area and ignites flammable objects that aren't being worn or carried.

As a frame of reference, each wall of Instant Fortress has 100 HP, and a 6-inch thick panel of the spell Wall of Stone has 180 HP, all of which can be destroyed with a couple casts of aforementioned spells. Does this mean any wizard with these spells would be able to level any city or even a castle within a span of two or three days? This seems awfully fast. Are there any rules I missed regarding destroying structures from the rules?

Per the answer to "Are objects structures and/or vice versa?", the rules on objects state (DMG, p. 246):

For the purpose of these rules, an object is a discrete, inanimate item like a window, door, sword, book, table, chair, or stone, not a building or a vehicle that is composed of many other objects.

implying structures are considered a collection of objects.

Now, certain spells like shatter, delayed blast fireball, and meteor swarm specifically say that they also damage objects and not just creatures. From the description of meteor swarm:

A creature takes 20d6 fire damage and 20d6 bludgeoning damage on a failed save... The spell damages objects in the area and ignites flammable objects that aren't being worn or carried.

As a frame of reference, each wall of the instant fortress magic item has 100 HP, and a 6-inch thick panel of the spell Wall of Stone has 180 HP, all of which can be destroyed with a couple casts of aforementioned spells. Does this mean any wizard with these spells would be able to level any city or even a castle within a span of two or three days? That seems awfully fast. 

Are there any rules I missed regarding destroying structures from the rules?