CodeshareThe real cost of a false positive
An over-eager spam filter does not just annoy one person. It teaches a community that the moderation is unfair, and that lesson is expensive to unlearn.
Every filter has two ways to be wrong: letting spam through, and catching something that was not spam. They are not symmetrical, and treating them as if they were is the most common mistake in automated moderation.
Missed spam is visible and cheap
Spam that gets through is obvious to everyone, including the moderators, and it is fixed in seconds by a human. The cost is a little noise and mild irritation.
A false positive is quiet and expensive
Someone posts a normal message and is muted. They may not know why. The moderators may not see it. The person concludes the server is hostile or the rules are arbitrary, and a fair number of them simply leave without saying anything.
You never see that in your statistics. The filter looks like it is working, because the thing it broke left quietly.
What follows from that
- Default to the cautious setting. The cost of missing some spam is lower than the cost of catching a member.
- Make every action visible to moderators, so a false positive is noticed rather than silently absorbed.
- Prefer reversible actions. A timeout that a moderator can lift beats a ban that needs an appeal.
- Let the server tune it. Only the people running a community know what normal looks like there.
Spam that gets through annoys people. Spam filtering that misfires loses them.