Yeah, I used to watch the show every week with friends in college. We were SUPER into it. We graduated the year Duchovny left and after watching a few episodes on my own, the magic was gone. Not just the communal enjoyment of watching with my friends, but the sense that the series had any idea where it was going as opposed to just making up crap as it went along.
The X-Files taught me a valuable lesson, though: long-form "mythology" driven TV shows are best avoided until (A) they have ended without being canceled, and (B) you know the ending doesn't suck. I never watched the last episode, and had tuned out two years before that, but my friends told me it sucked.