1. Species that simply prefer living underground (either because they fear the sun like the drow or because they love the dark like the dwarves).
2. Magical construction techniques that make huge, underground constructions more plausible.
3. Magical creatures that either have an instinctual need to create underground complexes or which create them as an unintentional byproduct. (Where did all these twisting tunnels come from? Well, they started as purple worm trails. Then the goblins moved in.)
4. Catastrophes on the surface world that prompt people to flee underground are also a great explanation for underground complexes. (See Earthdawn. Or just an Age of Dragons.) Mix-and-match with the techniques above to explain how the huge cataclysm refuges were built. Then simply remove the danger and/or (better yet) introduce some new danger that came up from below and drove all the vault dwellers back onto the surface.
It’s also useful to establish a method for underground species to generate food. In my campaign world there’s fey moss, which serves as the basis for fungal gardens. Huge, artificial suns left behind in underdark chasms by the vault builders or the under-dwarves also work.
I don’t find it valuable to do full-scale urban planning or figure out exactly how many toilets the goblins need, but I do find that at least some degree verisimilitude makes for better games: If the goblins get their food from fungal gardens, then their food supply can be jeopardized by destroying those gardens. And that’s either the basis for an interesting scenario hook or it’s a strategic master-stroke from the players or it’s some other surprise that I hadn’t even thought of before the campaign started.
In 13th Age dungeons are living things that now and then burst to the surface and adventurers have to go in and kill their hearts in order to stop them. So, sort of a hybrid of your 2 and 3.
There was a 2nd Edition product (Castle Builders’ Guidebook, maybe?) which pointed out that in a world where flying opponents are common (fly spell, dragons, etc.), fortresses have to be designed differently – you can’t just rely on tall walls to protect you from the enemy. Putting most of the fortress underground (or inside a mountain) is one way of dealing with flying opponents.
From Adventurer Conquerer King System’s main rule book, page 141:
Many mages devote their later careers to magical research. This vocation demands a constant supply of rare components, generally monster parts, such as the fangs of 20 hellhounds or skulls of 50 ogres. Rather than squander their time hunting beasts for these components, many mages build dungeons within their domain with the aim of luring monsters to lair within. There they can be harvested at the mage’s leisure.
In a world with magic, caves and tunnels could also just occur naturally much more often than in the real world.
You can’t teleport or scry or planeshift or whatever through more than forty feet of solid material like stone or dirt (permanent teleportation circles and the like still work). So putting your base inside a dungeon grants you a lot of protection from scry-and-die tactics and the like. There is thus a real incentive for people to build and inhabit them.
Underground tunnels and cities would offer some protection from the radiation of the ancients’ nuclear wars until it was safe to go back out. It’s the unfortunate souls left topside whose exposure mutated them into things like elves, dwarves, orcs, goblins…
Don’t forget about Uresia: Grave of Heaven. The Skyfall happened, creating the islands of the Grave. Now, people live on top of the ruins of godly realms, full of supernatural whoknowswhat and treasure, glorious treasure.
They were built by demon worshipers. At one time demons couldn’t stand the sun but that had all changed and now they walk by daylight…
Such fascinating thoughts! Kinda wish I’d thought of it myself, but I’ll steal good ideas shamelessly any time. The bit about a degree of verisimilitude leading to interesting unintended (or intended!) consequences is particularly enlightening.
We see some dungeons in Conan and Lord of the Rings, and for some reason in D&D they’re about 1000x more common. Conan has to go out of his way to some pretty exotic locations (fortresses and wizards’ dens) to find the semi-dungeons (sometimes literal dungeons) he ends up in, but in D&D every third hamlet has a dungeon nearby.
Minecraft. No, seriously. When you have creatures with god-like powers to modify their surroundings like high level mages etc. (or even Gods) then it stands to reason they will treat their world like minecraft. There would be tons of ‘dungeons’ in such a world. They would be these creatures just experimenting with their powers or making cool buildings just for fun or interest.