If you are against people who are against fascists, wouldn't that indirectly say you don't care about fascism? And that's how fascism gets a foothold. And with that logic, you indirectly support fascism.
That's true in some cases, like being anti-microsoft doesn't mean I love Linux. But when you are against an ideology that opposes the othering of groups of people to the point of mass suffering and murder that's different.
Choosing not to support a fight against Fascism is inherently in support of Fascism. The ideology uses manipulation and violence to oppress people, the end goal being a hyper capitalistic ethno state. So opposing the force which seeks to overcome fascism makes it easier for fascism to prevail.