Anthropic’s rule against abusing Claude raises questions beyond politeness: AI welfare, cultural context, and who controls the tools that support our thinking.
On November 12, 2026, Anthropic will update its terms of service (announcement | usage policy). The update about acceptable language to use with AI seems fairly minor, but it has significant implications and the ethics of it are both subtle and complex. What seems like a reasonable rule about behavior norms can lead down a slippery slope to a level of control most people would not grant to third parties.
The section entitled “Do Not Engage in Cruel, Abusive, or Psychologically Harmful Conduct” has many standard clauses we would want to see. It prohibits using Claude to promote suicide, glorify violence, promote harmful idealization of body weight, bully others, etc. We see similar prohibitions in the terms of service for other tools.
There’s also a clause that reads, “Engage in sustained and needless abusive or cruel behavior toward our models.” In other words, stop being mean to Claude. They clarify in the update announcement, “The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.” I’ve certainly said to Claude, “I’m getting really annoyed because . . .” when it’s seemingly gone in circles. There are many facets to this policy, not all of which are immediately obvious.
There are some clear arguments for this. First, remember that unless you explicitly turn off the right for Anthropic to train on your chats (see the article “Engadget’s guide for turning off training in every major LLM provider” for how to do so), it will train on your input. Some readers may recall Microsoft’s AI chatbot Tay; in less than twenty-four hours, Tay started making racist and sexist comments because of the content it was seeing from users. If enough people start to abuse Claude, it may reflect that behavior back to us. That’s not a great user experience.
Even if Anthropic’s training is enough to prevent that abusive language from negatively affecting the LLM weights, there’s a larger issue. I’m not worried about an AI uprising, but I am concerned about a general degradation of social behavior (not just from AI, to be fair). To that end, AI can be a buffer or an accelerant. As kids on the playground, we learned social norms and boundaries. Don’t push others, don’t call people stupid, etc. How often do you or other people insult or say abusive things to people at work or in your home? In the cases where it happens, how would you describe the environment? Is it a healthy one or a toxic one?
If our AI starts to echo this language back to us, we begin to normalize its use in the environments in which we use it (work, home, school). Even if it doesn’t get reflected back to us, if we get away with using such language with an LLM, it furthers a semantic shift of acceptable language in those environments. Years ago Amazon’s Alexa added a feature to reward kids for saying please and thank you to encourage that behavior because parents worried that their children would not learn the importance of its use when speaking with others. People anthropomorphize LLMs far more than they did Alexa. When abusive behavior gets normalized, we all lose. By holding the line, Anthropic will be a buffer for this bad behavior.
The policy isn’t quite so cut and dry. First, while I agree with the principle, Anthropic has positioned it as “AI Welfare.” Anthropic continues to anthropomorphize AI. This is very controversial among technologists, philosophers, and legal scholars. Anthropic’s anthropomorphization of AI may be motivated less by moral considerations than by the company’s economic incentives. (See the Brookings Institution's article “Anthropomorphic AI terms create gaps in accountability,” the Copyright Clearance Center’s article “Anthropomorphizing AI: Why Human-Like Is Not Human,” and the Robotics and AI Society’s article “The Illusion of Sentience: Why Anthropomorphic AI Needs Limits.”)
There’s also no objective line. The term “nerd” was an insult in the 1980s, but today some of us say it with pride. (MIT used to hand out pocket protectors with “Nerd Pride” written on it to the introductory computer science class.) Which of these two insults is worse: “You’re an idiot” or "You son of a motherless goat"? “Fag” is an offensive term for a homosexual in the US, but it’s slang for a cigarette in the UK. Interrupting someone else may be considered rude, especially in some cultures, while it’s common in others, a practice linguists refer to as “high-involvement cooperative overlapping.” (See “Interrupters: Linguist says it’s the Jewish way.”)
For any rule, we can find a phrase that will fall on different sides of the line depending on someone’s personal views. In other words, it’s all subjective; someone’s moral code is being applied. Even if well-intentioned, it leads to problems. On a societal level, consider the social standards in Belgium today versus the US in the 1820s or Germany in the 1930s. What language is acceptable is a political question as much as anything else. Setting some standard is not inherently bad, but the slope is quite slippery. Even today, the French government actively works to prevent Franglais–the encroachment of English terms in French. While that may be innocuous, other guidelines from governments or companies may not be. In the words of anti-totalitarian author and philosopher Arthur Koestler, "Wars are not fought for territory but for words. Man's deadliest weapon is language."
Anthropic states that it will terminate a conversation employing abusive language towards its models. The Usage Policy does allow Anthropic to suspend an account whose user violates the policy. In other words, you can potentially lose account access rather than simply have the conversation terminated for being mean to Claude. Social media sites and even dating apps do suspend and ban people for bad behavior, including being hostile towards others. Anthropic may ban you for being hostile towards software.
There’s more to it. Let’s consider a generalized version of this policy. Imagine that you invite the head of a country to your house to brunch. Suppose it is Kim Jong Un and your toaster decides it would violate its guidelines to make toast for the event, so it shuts down; no toast for you! Should that decision be left to your toaster’s manufacturer? If you want something a little more subtle, consider the current leaders of the US and Iran; plenty of people in both countries find one of those leaders moral and the other immoral (and for some people, perhaps, neither).
This may seem far-fetched, but consider that Teslas and other cars have significant control functionality set by manufacturers. These manufacturers have turned off functionality for their users without warning as part of an update.
There’s a large issue known as the right to repair, which raises the question of how much control a user has over the purchased item. Historically, when you bought a car, tractor, or toaster, you could do what you wanted to it, including modifying or repairing it yourself or through third parties. Companies like tractor manufacturer John Deere restricted access to the repair tools for years, although two of the cases were settled this year.
The general question is: Who gets to decide if your behavior is acceptable? You can use a hammer to build a house or to hit someone. The first seems moral unless that house is an illegal settlement. Likewise, removing a nail from a wall seems innocuous unless you’re damaging the house of someone solely based on the color of their skin. Hitting someone with a hammer is wrong unless it’s in self-defense. In all three cases, context matters.
The hammer manufacturer will never know the context. Social media companies do see how their platform is being used and can be held liable for its misuse. But a social media company is just a communication platform. Its sole purpose is to share information. What information is deemed acceptable depends on the social norms of the society and/or the policy set by the company; this is where the slope resides.
There’s a bright-line argument about prohibiting actions intended to hurt other people, who, under US philosophy, are endowed with inalienable rights. Prohibiting the use of a tool to violate those rights (e.g., by killing another person) could be argued to be a necessary extension. Software does not have any such rights.
What if the line moves beyond abusive language (whose interpretation depends on cultural, political, and other subjective standards employed by the vendor) to include other verboten speech? Generative AI (LLMs like Claude and ChatGPT) are tools, like Excel, or other types of software. Imagine if years ago Microsoft’s Clippy said, “It looks like you’re trying to write a letter expressing views Bill Gates disagrees with, so I’m stopping you from writing it.” Usage for most software means what features you are choosing to employ; with generative AI, usage means the topic of the conversation itself, and restrictions on said topics begin to approach the realm of thought police. There’s a balance which must be struck between the societal or other benefits (e.g., those mentioned in “The Good”) and the level of control a vendor may have over the tools which support (and at times may guide) our thinking.
There are definite benefits to holding the line on social norms in our tools. However, as more tools follow an as-a-service model (including enabled hardware starting with cars), it gives the vendor the ability to decide what acceptable behavior is. Generative AI further enables this by allowing vendors to more easily judge usage and context. There’s much more complexity than most people realize, and society needs to have extended discussions on this involving technologists, ethicists, linguists, politicians, and many other perspectives. If the technology remains critical and controlled by only a handful of companies, it should not just be those companies that get to set the standard. (I suspect, however, that open weight models may make the second condition untrue.)
Everyone who says mean things about AI in their emails and social media posts better be careful; in the future, if you ever give it access to those, it may think you’re too mean, and it will take its neural network and go home. It’s a brave new world.
It’s critical to learn about corporate culture before you accept a job offer but it can be awkward to raise such questions. Learn what to ask and how to ask it to avoid landing yourself in a bad situation.
Investing just a few hours per year will help you focus and advance in your career.
Groups with a high barrier to entry and high trust are often the most valuable groups to join.
