Yeah, I've always considered the usage of "destroyer" in SW to be a naval term rather than something literally referring to destroying stars! lol!
However, I do think the usage of "destroyer" (and "cruiser" for that matter) to be a little odd in SW, since real destroyers and cruisers tend to be smaller, fast navy vessels and Star Destroyers hardly seem to fit that bill. Perhaps Star Destroyers are indeed smaller and "destroyer-like" when compared to other larger, more heavily armed Imperial ship types (Star Battleships?) that we inexplicably never saw for some reason? The only wrench in that idea is that the only possible example of one of these larger ships in the movies, Darth Vader's massive flagship, Executor, is clearly also referred to as a destroyer several times, which makes even less sense. Yes, I know the EU has reclassified it as a "star dreadnought" but as far as I'm concerned, the movies trump EU explanations.