I suspect that what is at the bottom of this issue is the problematic nature of all the classic fallacy terms. The definitions of them can go in either of two ways: (1) they build in evaluative terms, like “irrelevant,” “illegitimate,” “improper,” and so on, to define the fallacy, or (2) they omit such terms. In the first case, the definition, so long as it is otherwise well-formulated, captures only instances of fallacious argumentation; but before you can apply the term, you have to judge, e.g., whether in a particular instance a remark on someone’s character is relevant to the evaluation of that person’s claims. Until you have made that judgment, you can’t say whether the remark amounts to an ad hominem fallacy or is legitimate (e.g., “This guy has a long record of telling delusional tales” is likely to be relevant to the question whether we should credit the testimony of the guy in question). In the second case, where the fallacies are defined in purely non-evaluative terms, we don’t have to evaluate the legitimacy of the argument in order to apply the term, but applying the term does not tell us whether the argument is fallacious. In either case, while it is easy enough to determine whether the descriptive component of the definition applies, we cannot infer from that application that the argument in question is fallacious.
I don’t think the classic fallacy terms are problematic. It is people’s lack of understanding of their scope - in which contexts are they applicable - that causes problems. In other words, an unwise person who knows all the fallacies is like a baby with a hammer.
Before an argument is evaluated, the judgment needs to be made whether it is worth even to take the argument into consideration. (If we had infinite time and other resources, that step might not be necessary. But it would be ridiculous to assume we had infinite resources.) Some of the “fallacious” arguments are actually valid arguments at this step. It is a fallacy to dismiss an already presented scientific theory on the grounds that its author is a homeless person who talks to his pet mouse. However, it is entirely reasonable and logically justified for a scientist to refuse to give an hour of his or her time to a homeless person who talks to a mouse and wants to present a new scientific theory to the scientist.