This is something of a wall of text, but seeing as you're getting into it, I thought I'd chime in with my experience of playing D&D over the last decade.
1) The pacing of something that's a shared experience of imagination is so different from everything else, where entertainments are fast, faster and fastest, very goal-oriented, and very videogamey (even prime time TV has gone this way with CSI et al, and it's very refreshing to do something cooperative, uncertain, and spontaneous. Even when there's a paper adventure the DM is going by, D&D and roleplaying in general is both unpredictable and fun. D*D was a real escape, it didn't want to profit from you (seemingly), no-it wanted to give you a world with a magical sword, a spellbook, an epic story, heroic deaths and even greater triumphs, all realized through cultivating your imagination with people of like mind. Sounds great, no?
2003:
2) Over the course of its run, 3.5th edition brought almost all of the manifold Pen and Paper enthusiasts into the fold of the same game. White Wolf, the makers of Vampire et al, destroyed their continuity while revamping their rules, alienating all but the most harcore LARPers. As LARP was always a very different set of folks from us, this meant that we got the best of the creative and mathematical talent from White Wolf's playerbase. D&D had never been more fun, and almost every house game you walked into was strong, with people showing up weekly and regularly leveling their characters 1-13. The death of the primary publisher of off-genre RPGs would prove pivotal later on.
2007:
4) Wizards, the Publishers of the primary D&D game, made a new edition of the game, which aimed to capture more of the Warcraft and Video game crowd by toning down the elements of the game that were oriented toward storytelling and simulation. While attrition and a sense of segregation between those with a causal approach,
--those who were already praising WoW and calling off their attendance at weekly sessions,
--and those more dedicated to the game,
was already becoming clear, the burden of sorting out which rules to play, and moving forward afterward,
2009(ish):
5) a group of game designers at Paizo splintered away, updating the 3rd edition rules, and generally capturing the more intelligent, tech-savvy and modern members of the playerbase. Those that neither used the internet for its intended purpose, nor attended Gen Con or Origins, regarded the hype around this 'new' game as somewhat quaint. For whatever reason, during the first 18 months of Pathfinder's debut, too few of the casual crowd was educated on how Pathfinder would
bring new life to 3rd edtion.
This left the casuals and the oldtimers, who either adopted 4th edition abortively, kept on playing 3rd edition, or got into Pathfinder very reluctantly in an awkward position. Most of them already were at a point where they were at sessions about 1/2 of the time, and with the burden of figuring out how to play a new rules system, their internal calculus told them (correctly) that girlfriends and job applications were a better investment of time. D*D didn't hold its promise any longer. Players alienated themselves from each other as they either despised their playmates for being too stupid to play the 'best' game, or simply refused to have an opinion, and delved into playing WoW full-time.
Why did the dream die?
During this decade, the grounswell of visual ammunition for the imagination in the form of the Lord of the Rings movies helped immensely to make games exciting and boost a feeling that mental images were shared. But around 2006, the enthusiasm of more and more of the players I respected most wore away. They got their gaming fix during the week playing WoW (they admitted it was inferior, but
so convenient!). They tuned out to the onrushing waves of content from Wizards that was constantly adjusting the power levels of the game. Second-rate jobs became increastingly unpredictable in schedules, and holding a group together required another individual beyond the DM just to 'coordinate' games.
By the time edition wars came along, it was the nail in the coffin. People didn't want to put the effort into pushing through their campaigns. Smartphones at the game table made things still more distracting. And at about this moment, new editions of Starcraft and Warhammer were released, sapping more energy from the nerdy tabletop community.
And most importantly, the sort of geeks that play D&D have never been good at conversations where consensus is pushed through. Mostly, they are happy to sit on their opinions, and are after all entitled to them, since they decided around 7th grade that they were different enough to spend 5 minutes or more a day thinking about elves and wizards instead of pokemon and baseball. That takes a certain nonconformist attitude, eh? The consensus needed to push past edition wars just wasn't something that a pack of introverts could arrive at.
After the sundering, Geeks' other great weakness reared its head:
Inability to Network. Networking geeky groups is harder than ever, despite the internet. As the recent Finkelgate attests, the quest for a girlfriend, and probably a job, can get majorly snafu'd by having your real name branded with true nerdiness. (I'm talking the really geeky stuff here. Loving Ocarina of Time and Marvel Comics is about as mainstream as you can get). At the dawn of facebook, it was very clearly decided that D&D was an off-limits topic due to the possibility of Alyssa Banezak reading your feed.
And the alternatives to renetwork playgroups have not materialized. About a half dozen websites compete for true matching app functionality, and some people haven't moved beyond the use of forums and even NEWSGROUPS. So without the ability to discover adults that share your hobby discreetly through the internet, one would have to randomly proposition people in your life that
just might be interested. They will probably turn out to be flakes--and as anyone that has leveled a character to 2 simply to see him go to the binder due to poor attendance can attest, flakes make great 5th and 6th players. When you're trying to get up to 4, they don't cut it.
And D&D can't take a hit like that. It takes effort and investment to get into, something that WoW et al. doesn't. You could, in that time, work on your golf game or your knowledge of the Jersey Shore, both sure paths to advancement in the workplace, rather than risk the time on this retro game where you use PEN and PAPER and DO MATH IN YR HEAD WTF? D&D requires effort, something that this casual-tilting age eschews, and therefore its potential players are few and far between.
Now, only diehards continue to play weekly and level up their characters. And without the continuity of a storyline, D&D really is WORSE than playing a video game.
Last year I learned that the
author of the much lauded first book of Paizo's Kingmaker adventure path (the recipient of the industry's highest prize) no longer played weekly with a group, and had playtested the adventure online over VOIP. I'm sorry, but if the people making your product don't have the time/ability to play with it in its intended form, your product is officially now in the realm of the arcane. The numerous flaws in the adventure's scenarios were overlooked (it's braindead easy, 5-foot corridors are not a fun place to have a fight) by the journalists in the zeitgeist, and thus it was proved to me that only the true fans, those that aren't critical of their love enough to realize it's a mummified corpse, are still playing enthusiastically.
Inward-turned and in-bred. That's the direction D&D is headed.
Is D*D really dead?
I sure hope not.
I hope someone out there will prove me wrong.