On the one hand, yeah, caveat emptor. But on the other, *****, come on, showing up at 4:30am all the way through to 11pm? Have some ****ing decency, people!
I can see where, if this were a public establishment like Twede's in Snoqualmie, WA (the Twin Peaks diner) or the Llanerch Diner outside of Philly (from Silver Linings Playbook), yeah, you don't get to complain that customers keep coming in and ordering the cherry pie (which, by the way, is kinda lame at Twede's, or was when I had it, like, 10 years ago). But this is a private residence. And while it's fair to assume that people will wander around it and take pictures, it's quite another thing to have them leaving their ****ing garbage, traipsing around the property at all hours.
I know we're all fans here, and so are more accustomed to our own perspective, but really, there's a line and it sounds like fans were regularly crossing it. It's not as if the owner was unaware that it was a "fan shrine," but I can imagine being totally unprepared for the degree of disregard fans have for your privacy and lack of respect for your property.
I don't have a ton of sympathy for someone who wants their private home to be totally off limits to the public. I don't even get that privilege (e.g., Google Maps can drive by and snap a photo of my house). But if every day I had to clean people's garbage off my front step, you bet your ass I'd be having a chat with the city and/or local gendarmes.