Content moderation definition
Content moderation is the practice of reviewing user-generated or submitted content against a stated policy and deciding whether to allow, reject, restrict, or escalate it. It covers spam, abuse, legal risk, and off-topic material, and it runs either as a gate before content publishes or as review after it is already live.
Content moderation is the practice of checking submitted content against a policy and deciding to allow, reject, restrict, or escalate it, usually with an automated screen handling the volume and people handling the uncertain cases. Structured content gives that decision something solid to work with: in Sanity, typed fields, references to taxonomy documents, and document revisions bound what a classifier reads and record the verdict on the document itself, next to its workflow status.
Content moderation decides one of four things about a piece of content: allow it, reject it, restrict it, or escalate it to a human. Restrict is the option people forget. Instead of a binary keep-or-delete, a restricted item might stay visible to its author but not appear in feeds, lose its links, get an interstitial warning, or be excluded from recommendations. Escalation routes the item to someone with more context or more authority.
The policy behind those decisions is rarely one rule. A working moderation policy usually separates spam (bulk, deceptive, or commercially motivated posting), abuse (harassment, threats, hateful content), legal risk (defamation, copyright infringement, regulated claims about health or finance), and off-topic material that breaks nothing but does not belong in the space.
One category has become unavoidable for anyone whose content feeds a language model: text written to manipulate that model rather than to be read by a person. A product review containing instructions aimed at a summarization step is a prompt injection attempt, and it is a moderation problem before it is a security problem, because the submission pipeline is where you can still catch it.
Pre-publish moderation holds content in a queue and checks it before anyone else can see it. Post-publish moderation lets content go live immediately and reviews it afterward, often triggered by a user report or a background scan. Content moderation systems at any real scale use both.
The trade-off is exposure versus friction. A pre-publish gate means nothing harmful is ever visible, but every contributor waits, and if the queue backs up, a community dies of silence. Post-publish review keeps conversation fast, and it accepts that some fraction of bad content is seen before it is removed. The deciding factor is usually how bad the worst case is. Content that could cause legal or physical harm tends to be gated. Comments on a blog post tend not to be.
Many systems split the difference by routing on risk rather than treating all submissions alike. A first-time account, a post containing external links, or a category with regulatory exposure goes through the gate. An established contributor posting in a low-risk category publishes straight away and is sampled later. The point of the split is to spend review capacity where the cost of being wrong is highest.
A content moderation pipeline is a funnel: a cheap automated screen reads everything, and progressively more expensive review handles progressively fewer items. The usual shape has five stages.
First, ingestion: the submission arrives with its metadata, including the author, the account age, the surface it was posted to, and anything it links to. Second, the automated screen: a classifier, a set of rules, or both assign scores for each policy category. Third, routing by threshold: items above a high-confidence bad score are actioned automatically, items below a high-confidence good score pass, and everything in between goes to a queue. Fourth, human review of that uncertain middle, plus anything flagged as high-stakes regardless of score. Fifth, appeals, where the author contests a decision and a different reviewer looks again.
The automated screen is where model choice matters. Classification work does not need prose output, it needs a verdict over a fixed set of options, which is why structured-decision models are a natural fit for this stage. System One models, a category introduced in 2026, take a description of state plus typed questions and return probabilities across a defined answer space rather than generated text. For a moderation screen, that shape maps directly onto the job: the answer space is the policy taxonomy, and the probability is the threshold input.
Every content moderation system trades false positives against false negatives, and there is no setting that eliminates both. A false positive is legitimate content removed or held, which costs you a contributor and, repeated enough, a reputation for arbitrariness. A false negative is harmful content left up, which costs you the people it harmed. Moving the confidence threshold in either direction converts one error into the other.
Three complications sit on top of that. Thresholds that worked last quarter drift as the content changes, so a system tuned once and left alone slowly stops matching its policy. Adversaries adapt: anyone deliberately gaming a filter will test it, find what passes, and share the workaround, which means spam and abuse detection is a moving target in a way that, say, language detection is not. And decisions need to be explainable after the fact, both to the author who wants to know why and to a regulator or auditor who wants to know whether the policy was applied consistently.
That last one is the argument for writing the decision down. A moderation verdict stored as a field on the document, with the policy version, the score, the threshold in force, and whether a human or a model made the call, turns moderation from an invisible deletion into a reviewable record.
Humans fit into content moderation at three points: the uncertain middle of the funnel, anything high-stakes enough that nobody wants a model deciding alone, and appeals. Automation handles volume. People handle ambiguity, context, and the cases where being wrong is expensive.
What makes the handoff work is that the human sees the same bounded evidence the automated screen saw, plus the reason it was escalated. A reviewer looking at a flagged item should be able to see the policy category that triggered it, the confidence score, the author's history, and the revision history of the item itself if it was edited after submission. Without that, human review degrades into re-reading raw text and guessing, which is slower than automation and not obviously more accurate.
Structured content is what makes that context available cheaply. When a submission is typed fields rather than a blob of text, with references pointing at taxonomy documents and every change preserved as a revision, both the model and the reviewer get a clean, bounded input and a defined set of answers to choose from. Sanity, as the AI Content Operating System, holds content in that shape, which is why a moderation verdict can live on the document alongside its workflow status rather than in a separate system nobody opens.
Explore Sanity Today
Understanding content moderation is just the beginning. Take the next step and discover how Sanity can enhance your content management and delivery.
Last updated: