But that's also the point of dog-whistle statements: the statement in question can be taken either way. The supporters know it's racist, and they agree with it. The opposition claim it's racist, but the supporters them point out that maybe the people claiming racism are taking things out of context.

Put it this way: the are hundreds of rat-infested slums in America. Every city has at least one. But the one selected happens to be majority non-white. The statement may be neutral, but the context is definitely not. And context is everything.