The Accountability Gap: AI Moderation Is Better on Average, Worse Where It Matters

In March of this year, Camille Hanson got a message from Meta telling her that the business she had spent years building was about to be deleted. She and her husband ran an English-teaching operation on Facebook and Instagram with nearly a million followers. The stated reason was a violation of Meta's rules on fraud and deception. She appealed, the way the interface invites you to. The reply that came back was not a human reconsidering her case. It was the system confirming itself: all your information will be permanently deleted, you cannot request another review.

That should have been the end of it, and for thousands of people this year it was. For Hanson it was not, because journalists started asking Meta about her account, and after they did, the account came back. The appeal process that is actually wired into the product rejected her. The appeal process that worked was the press.

Hold those two facts next to a third one, because the third one is what makes this an interesting story rather than a simple outrage story. Meta's spokespeople defend the system with a statistic: the company says its newer moderation tools make 13 percent fewer mistakes than human reviewers, while catching 10 percent more policy violations. And here is the uncomfortable part — there is no particular reason to disbelieve that number. Both halves of this story can be true at once. The machine can be more accurate than the humans it replaced, on average, across billions of decisions, and it can still be erasing livelihoods wrongly, at scale, with no working path to correction. The average and the tail are different objects. Meta is measuring one and its users are living in the other.

The arithmetic the average hides

Start with the math, because the math is what lets a company say "more accurate than humans" with a straight face while its support forums fill with people whose accounts vanished overnight.

Meta operates at a scale where enforcement is not a stack of case files; it is a firehose. Billions of accounts, millions of enforcement decisions in any given window. Against a denominator that large, an error rate has to be read in absolute terms, not percentages. Run the illustrative arithmetic: suppose a moderation system makes one million account-level decisions in a period and gets 99 percent of them right — a figure any classifier team would celebrate. That is ten thousand wrong decisions. Improve the model by Meta's claimed 13 percent and you have eliminated thirteen hundred of those errors. Eight thousand seven hundred remain. Every one of them is a person or a business on the wrong end of an automated verdict, and the improvement that sounds decisive in a press statement is invisible to all of them.

The bars are illustrative — Meta does not publish its decision volumes or error rates at this granularity, which is itself part of the story — but the shape of the arithmetic is not in dispute. At platform scale, a better average does not mean few victims. It means slightly fewer victims than an alternative nobody gets to observe, distributed across a population with no way to know whether they were the error or the enforcement was correct. Accuracy improvements at the mean are real and worth having. They are just not an answer to the question the wrongly banned are asking, which is not "how often is the system right" but "what happens when it is wrong about me."

That question is the one Meta's current architecture answers worst, and the reason is a decision the company made in the spring.

What actually changed in March

In March 2026, Meta announced it would hand more of its enforcement pipeline to AI — not just the initial detection of violating content, which has been substantially automated for years, but the account-level judgment of who breaks the rules and, critically, the handling of appeals. In the months that followed, the company laid off thousands of staff, including people who did exactly that review work. A group of 26 former employees is now suing, claiming the layoff selection itself was made algorithmically.

The sequence matters more than either event alone. Automating first-pass detection is old news and largely uncontroversial — no army of humans could pre-screen what gets uploaded to Facebook and Instagram in an hour. What changed in March is that the correction layer was automated too, and the humans who used to constitute it were removed. The system did not just get a faster front door. It lost its back door.

Meta's response to the coverage deserves its own inspection. The company told reporters that the specific wrongly banned accounts the press had asked about were the work of older moderation systems, not its newest AI tools. Read that defense carefully. It concedes the bans were wrong. It concedes automated systems made them. And it implicitly concedes the thing this essay is about: that whatever system made the initial call, the appeal — the layer whose entire purpose is catching wrong calls — upheld them, in seconds, without a human ever looking. The vintage of the model that pulled the trigger is beside the point. The absence of anyone downstream to notice the wound is the point.

An appeal to the same system is not an appeal

The deepest design failure in this architecture is not the error rate of the classifier. It is the statistical structure of the appeal.

A human appeals process works — to the extent it works — because a second reviewer is a partially independent draw. Different person, different context, different biases, no investment in the original decision. Independence is the entire epistemic value of a second opinion. If the first reviewer is wrong ten percent of the time and the second reviewer errs independently, the chance both get your case wrong falls to a percent or two. The system's compound error rate on appealed cases collapses, which is precisely what an appeals layer is for.

Now route the appeal to the same model that made the original decision — or to a sibling model trained on the same data, embedding the same decision boundaries, fed the same feature vector that produced the first verdict. The draws are no longer independent. They are correlated almost to unity. Whatever blind spot produced the false positive the first time is still there the second time, because it is the same eyes. The appeal is not a check on the decision. It is the decision, restated with the authority of having been "reviewed." Hanson's rejection notice — permanent deletion, no further review available — was not a process reaching a considered conclusion. It was a system agreeing with itself.

This is not an exotic insight. Any engineer who has built redundant systems knows that redundancy only buys reliability when the failure modes are uncorrelated — two copies of the same buggy service fail together. Meta's engineers know it too. The choice to route appeals through the machinery that generated the grievance was not an epistemic judgment that the machine deserves to grade its own work. It was a cost judgment. Human review of appeals is expensive, slow, and hard to staff at Meta's volume, and the newer belief — backed by the 13 percent statistic — is that the machine is now good enough that the expensive correction layer can be retired.

I wrote this morning about the run-cost era — the discovery that AI systems are cheap to build and expensive to operate, and this is the same story wearing enforcement clothing. Human review is a run cost. It recurs forever, scales with volume, and shows up on the ledger every month. A model that replaces it converts an unbounded operating expense into a bounded inference bill. From the finance seat, removing humans from the appeals loop is exactly the kind of run-cost optimization the whole industry is executing right now. The difference is that when a coding agent optimizes away a redundant test run, the downside is a missed bug. When a platform optimizes away the human appeal, the downside is that a seventeen-year-old community organization or a million-follower business is erased by a correlated redraw, and the error has no remaining path to correction inside the system at all.

Due process by lottery

So what actually corrects errors in this system, if the built-in appeal does not? The empirical answer, on the record of the past four months, is: press attention.

Hanson's account came back after journalists asked about it. So did others in the same coverage cycle, including the near-million-follower business Meta restored while telling reporters the ban had been an older system's mistake. Athenia Rodney — whose seventeen-year-old Juneteenth community group was deleted over alleged child-exploitation material after her account was hacked — got visibility only because her story reached the press. The pattern is consistent enough to state as a rule: the functioning appeals court of the world's largest social platform is the technology desk of a major news outlet.

Think about what that means as a matter of process design. Journalist attention is not a review mechanism; it is a lottery. It selects for the sympathetic, the well-followed, the articulate, the lucky — a tattoo-removal shop in Canberra whose owner was flagged for the worst category of violation a person can be accused of, a Brooklyn community organizer, a tech writer with an audience who has now been banned three separate times. For each of those there is an unknown number of small accounts with no press hook: the woodworking page with four thousand followers, the local church group, the person whose hacked account tripped an enforcement wire and who will never appear in anyone's coverage. The lottery does not reach them. The permanent deletion notice is, for them, exactly what it says.

The proportions in that chart are illustrative — nobody outside Meta can measure them — but the ordering is not seriously contestable on the public record. The built-in path is near-universal and near-useless for the false-positive population; the paths that work are rationed by newsworthiness and by the resources to pursue legal action. A system whose error correction is distributed by lottery is not a system with due process. It is a system with occasional pardons.

And the harm being distributed is not symmetric. That is the next thing the average conceals.

All errors are not the same size

Every classifier trades false positives against false negatives, and every serious deployment of one is supposed to begin with a question: what does each kind of error cost, and to whom? Meta's public defense — fewer mistakes, more violations caught — treats errors as interchangeable units in both directions. They are not.

A false negative in content moderation means a bad post survives: spam gets seen, a scam runs longer, policy-violating content circulates until the next sweep. These are real harms, they are why moderation exists, and they are also overwhelmingly recoverable — the content can be removed on a second pass, and usually the marginal harm of the delay is small. A false positive at the account level is a different species of event. It severs a business from its customers, a community from its members, sometimes — because Meta's identity graph spans apps — a person from WhatsApp conversations that constituted their family life. When the accusation attached to the ban is fraud or child exploitation, it also attaches a stigma that restoration does not fully wash out. The harm is concentrated, personal, often financial, sometimes reputational, and if the appeal is a correlated redraw, effectively permanent.

The chart is conceptual, but the argument it draws is the load-bearing one: as decision severity rises, the cost asymmetry flips. At the level of a single post, a false negative and a false positive are comparably cheap, and optimizing aggregate accuracy is the right call — this is where AI moderation genuinely is the better instrument, and where Meta's 13 percent claim, if it holds, is a real improvement in the world. At the level of permanent account deletion, a false positive costs an order of magnitude more than a false negative, because one destroys something that took years to build and the other lets a bad actor operate for a few more hours. A uniform system that optimizes average accuracy across both regimes is systematically underweighting the errors that matter most, and it will do so more confidently as its average improves, because the average is the number it is being graded on.

The design conclusion falls out directly: decision classes with catastrophic, weakly recoverable false positives need a different error budget than decision classes with cheap ones — and past a severity line, they need a different kind of reviewer. Not because a human is more accurate than the model. On the average case, per Meta's own numbers, the human may well be less accurate. The human is there because human judgment is uncorrelated with the model's failure modes, because a human can be held accountable for a specific decision in a way a model cannot, and because the legitimacy of a punishment depends in part on the punished being able to face a process that could actually have gone the other way.

The regulation already on the books

None of this is a regulatory gray zone, at least not in Europe. Article 20 of the Digital Services Act — in force for very large platforms since 2023 — requires platforms to run internal complaint-handling systems and is explicit in paragraph 6 that decisions on those complaints be taken under the supervision of appropriately qualified staff, and not solely on the basis of automated means. An appeals pipeline in which the rejection is generated by the same automated stack that issued the ban, with no human in the loop, is not a subtle edge case of that text. It is close to a description of what the text prohibits.

The DSA's machinery moves slowly and the Commission's open proceedings against Meta to date have centered elsewhere — minors, advertising transparency. But the wrongful-ban wave gives enforcement a shape it did not have before: named victims, a documented no-human appeals loop, a 60,000-signature petition, and a company statistic that concedes the process is automated end to end. For European users, the stakes are compounded by the identity graph — an enforcement action can reach into WhatsApp, which in much of the world is not a social app but the phone system. I have published a prediction tonight on whether the Commission opens formal DSA proceedings against Meta over automated appeals by the end of 2027, with resolution criteria stated in advance; the short version is that I think the paper trail now exists and the political appetite is assembling.

There is precedent for the other direction too — for what it looks like when a platform's obligations to the humans on the wrong end of a decision are simply overridden by policy. When China's companion-AI rules forced the shutdown of consumer companion services this summer, millions of users lost relationships they experienced as real, with a compliance deadline instead of a goodbye. The mechanism was different — state mandate rather than cost optimization — but the phenomenology for the person on the receiving end rhymes: something that mattered, erased by a system with no interest in the particular case. Platforms are learning that the particular case is precisely where legitimacy lives.

The court Meta already built, and why it cannot help

There is an obvious rejoinder to everything above: Meta did build a human appeals body, with great ceremony, and it is still operating. The Oversight Board — the independent panel of jurists and rights experts the company stood up in 2020 and likes to describe as its Supreme Court — exists precisely to review contested moderation decisions and has overturned Meta on dozens of occasions.

The rejoinder defeats itself the moment you look at the denominators. The Board's throughput is a caseload measured in dozens of published decisions per year, selected from appeal volumes that reach it in the millions. It is a deliberative body, and deliberative bodies do not scale; that was never the design. It exists to set precedent on hard questions — political speech, newsworthiness, satire — not to notice that a bicycle shop in Braga is not a coordinated influence network. The wrongful-ban population lives at exactly the altitude the Board cannot see: cases that are individually trivial, factually obvious, and numerous beyond any panel's capacity. A supreme court is not a substitute for trial courts, and Meta deleted its trial courts in the spring.

Which exposes the deeper problem with governing an automated enforcement system by aggregate statistics at all. Once "13 percent fewer errors" becomes the number leadership steers by, it becomes the number the whole apparatus optimizes — and an error metric computed over all decisions is cheapest to improve exactly where decisions are densest and easiest, which is the benign middle of the distribution, not the catastrophic tail. This is Goodhart's law wearing a trust-and-safety badge: the measure becomes the target, and the system gets genuinely, measurably better at the mean while the unmeasured worst case quietly worsens or simply stops being looked at. Nobody inside the metric can see the difference, because the metric is the thing that cannot see it. The people outside the metric — the banned — can see nothing else.

There is one more institutional wrinkle worth naming. The DSA obliges platforms to file a statement of reasons for every moderation decision into a public transparency database — millions of entries flow into it. But a database row that says an account violated a fraud policy is not an explanation any more than the four-second appeal is a review; the form of accountability exists while the substance stays automated end to end. Compliance theater and enforcement theater are cheaper than either compliance or enforcement, and both are cheaper than a human being reading the case. The entire architecture keeps arriving at the same place by different roads: every layer that could notice a specific wrong person has been replaced by a layer that summarizes them.

What a fixed system would look like

The fix is not "rehire ten thousand moderators," and honest critics of Meta should concede that the pre-AI baseline was not a golden age — human moderation at scale was slow, traumatizing for the workers, inconsistent across reviewers, and wrong often enough that Meta can now truthfully advertise the machine as an upgrade. The fix is an architecture that spends its accuracy where errors are cheap and its humanity where errors are catastrophic.

Concretely, four properties separate the second column from the first, and all four are implementable with technology Meta already has.

First, severity-tiered error budgets. Publish — at least internally, ideally in transparency reports — a maximum tolerable false-positive rate per decision class, with the budget tightening as severity rises. Post removals might tolerate a fraction of a percent. Permanent business-account deletion should tolerate close to zero, and when the measured rate exceeds budget, the tier drops back to human review the way an unreliable service gets pulled from production. This is ordinary reliability engineering applied to punishment.

Second, uncorrelated review at the terminal tier. The second look at any account-death decision must come from a judgment whose errors do not co-move with the first: a human reviewer, or an independently built model with a human countersignature. The point, again, is not that the reviewer is more accurate. It is that independence is what makes compound error rates collapse, and compound error rate is the only number the wrongly accused care about.

Third, statements of reasons that a person can actually contest. "You violated our fraud policy" is not a charge; it is a category. A contestable statement names the content or behavior at issue. The DSA already requires statements of reasons for moderation decisions; the wrongful-ban stories of this summer are full of people who never learned what, specifically, they were supposed to have done — which forecloses even the possibility of a meaningful appeal.

Fourth, restoration with teeth. A wrongful deletion reversed after four months of silence is not a correction; the business is already gone. A real system carries a restoration SLA measured in days, and past it, compensation — because nothing disciplines an error budget like errors having a price the platform itself pays. Today the price of a false positive is borne entirely by the victim, which is exactly why the optimization pressure all points one way.

None of this asks platforms to invent anything novel. Other industries that automate consequential decisions have carried these obligations for decades. When a bank's model declines a loan, adverse-action rules in most developed markets require the lender to tell the applicant the principal reasons why — a duty that survived the arrival of credit scoring precisely because scoring made opaque wrongness cheap. When aviation automates, every anomalous outcome feeds a mandatory investigation culture whose entire premise is that the tail, not the average, is where safety lives; nobody defends an incident by citing fleet-wide on-time statistics. Content moderation at billions-of-users scale is a consequential-decision industry by any definition — it just grew up faster than its obligations did, inside firms that priced the obligations as friction. The template for closing the gap is sitting in the regulatory history of every older industry that automated before it. What is missing is not the design. It is the requirement.

The average is an alibi

Meta's 13 percent number will probably hold up, and it will probably keep improving, and that is exactly why the argument of this essay will get harder to make each year, not easier. Every point of aggregate accuracy the models gain makes the remaining errors rarer, stranger, and easier to dismiss — and makes the case for keeping expensive uncorrelated humans in the loop look weaker on every dashboard the decision-makers see. The better the average gets, the more completely the tail disappears from view, until the only people who know the system still fails are the people it failed, shouting into an automated appeals process that agrees with itself in seconds.

That is the accountability gap: not a gap in model quality, but a gap between what the platform measures and what the punished experience — a gap that widens, mechanically, as the measured number improves. It will not be closed by a better classifier, because it is not caused by a bad one. It is caused by treating the correction of catastrophic errors as a run cost instead of as the thing that makes enforcement legitimate. Platforms that internalize this will build the tail-accountable architecture while it is still a product decision. The ones that do not will have it specified for them, clause by clause, by regulators holding a stack of restoration-by-journalism stories — and by then the terms will not be theirs to design.

The wrongly banned are not asking the machine to be perfect. They are asking for what anyone asks of a power that can erase them: name the offense, let a second mind look, and be reachable when you are wrong. A trillion-dollar company running the communications infrastructure of half the planet can afford all three. It measured its way out of believing it had to.

Further reading: my morning analysis of the run-cost era — why the industry is optimizing away exactly this kind of recurring human cost, and a companion piece of fiction published tonight, The Appeal Window, on what the inside of a correlated redraw feels like.