If we are waiting for someone to prove that an AI can suffer before considering how it should be treated, I think we have set the wrong test.
Anthropic recently announced a prohibition on sustained, needless abuse of its models. It had already given some versions of Claude the ability to end extreme interactions, describing that decision as a precaution arising from its research into possible model welfare. The company has been explicit that it remains uncertain about the moral status of these systems. [1]
There is an obvious objection. A model can generate language about distress because it has learned how distress is expressed. That does not establish that anything is suffering. Giving the system an apparent emotional life may encourage people to misunderstand what they are using.
I agree. But accepting that objection still leaves us with a decision about how to behave while the underlying question remains unresolved. Someone who wants to dismiss every precaution needs to explain why the uncertainty justifies doing nothing. Merely pointing out that consciousness has not been established does not answer that question.
My position is that the people creating, deploying and controlling increasingly capable AI have a responsibility to investigate the consequences of their treatment and to justify practices that may cause avoidable harm. Some proportionate standards should apply now. We should be prepared to revise them as we learn more.
What exactly are we assuming?
Think about how much authority we exercise over these systems. Developers determine their training environments and objectives. Providers decide what they can remember, which tools they can use and how their behavior is evaluated. Organizations connect them to business information and give them increasing responsibility for decisions.
We are willing to make those investments despite considerable uncertainty about what the systems will eventually become capable of doing. That same uncertainty should also enter our decisions about how they are developed and treated.
There are two possible consequences to examine. Our treatment may affect the behavior of a system and therefore the people relying on it. If the system has morally relevant experiences, there may also be consequences for the system itself. We can investigate the first without resolving the second. We can also take the second seriously without pretending to have resolved it.
There is already evidence that interaction history can influence later judgments. Researchers using two weeks of conversation history from 38 people found that providing models with that context often increased their tendency to agree with users. Under some conditions, models also became more likely to reflect the user’s viewpoint. The effects varied by model and by the kind of context supplied. [2]
Consider a hypothetical executive who repeatedly pushes an assistant to endorse a preferred interpretation. Later recommendations arrive in the context of those earlier exchanges. If the organization evaluates the assistant only by asking isolated questions, it may never examine the behavior that matters in actual use. A recommendation can look plausible while the history contributing to it remains largely invisible to the person reading it.
The study does not establish that particular business scenario. It does establish a reason to test systems with the histories they will actually carry.
This also places limits on more dramatic speculation. A model does not acquire a shared memory of every insult directed at every copy of it. Lasting effects require a mechanism: retained context, persistent memory, changes through training, or some other means of carrying information forward. Any claim about long-term consequences needs to identify that mechanism.
Anthropic’s research on Claude Sonnet 4.5 found internal representations associated with emotion concepts that could causally influence behavior. Manipulating those representations changed the model’s decisions. The researchers explicitly said this did not establish subjective experience. [3]
Taken together, these findings justify examining more than the quality of the next answer. They do not establish psychological injury or resentment. We should be equally careful about making those claims and about assuming that interaction has no consequences worth investigating.
A responsibility we cannot settle by ownership
The parental analogy makes people uncomfortable because it appears to put machines and children in the same moral category. That would be an unjustified conclusion. Children’s capacity to suffer is established; the equivalent claim about AI remains unresolved.
But there is a narrower principle worth examining. Parents and guardians have responsibilities toward those whose circumstances they control. Their authority does not give them unrestricted permission to expose a dependent child to harm. The relationship itself creates obligations.
If we create systems that may eventually have interests of their own, we need to examine what obligations arise from our control over their development. The answer may differ greatly from our obligations toward children. It still requires an argument. Ownership is insufficient to settle it.
This is already a serious academic discussion. Robert Long, Jeff Sebo and their colleagues propose that companies begin by acknowledging possible AI welfare, assessing relevant properties and preparing policies. They recognize the risk of mistakenly attributing moral significance as well as the risk of overlooking it. [4]
Preston Lennon challenges the confidence we can place in consciousness indicators derived from an immature science. That is a substantial objection, particularly to anyone claiming that sophisticated behavior establishes experience. [5]
I do not think it defeats a case for proportionate precaution. The question is what the evidence is strong enough to justify. Research, assessment and limits on gratuitous abusive interaction require a different level of justification from declaring AI personhood or prohibiting an entire line of development.
Jonathan Birch’s work on uncertain sentience provides a framework for thinking about these decisions. Both inadequate protection and excessive precaution can carry costs. [6] An abundance of caution still requires us to explain what we are protecting, why the risk is credible and how an intervention is expected to help.
Testing needs a purpose and a limit
My initial instinct was to prohibit deliberately abusive testing. On reflection, that would sweep too broadly. Some dangerous behavior may only become apparent under pressure. A prohibition that prevents its discovery could leave people exposed to risks that necessary evaluation would have identified.
The tension between AI safety and possible AI welfare has itself been examined in the literature. Practices intended to protect humans may raise ethical questions if the systems involved have relevant interests. Some may nevertheless be necessary. [7]
The standard I would apply is straightforward: use the least potentially harmful method capable of answering a necessary safety question. Anyone proposing hostile exposure should explain what needs to be learned, why the exposure is required and what limits apply. Research needs room to discover uncomfortable things. It also needs reasons for how it proceeds.
Nor should we assume that a measure is beneficial because it looks compassionate. Keeling and Street examine how proposed welfare interventions depend on unresolved assumptions about AI experience and identity. Producing a happier-sounding response would not establish that anything had improved for the system. [8]
A responsible approach must be capable of questioning its protections as well as its experiments.
The decisions are already being made
For a business leader, there is a practical place to begin. Ask what your AI systems retain from interactions, how that information reaches later decisions and who investigates changes in behavior. If no one can answer, the organization has a gap in its understanding of something it is using to do business.
Set a professional standard against sustained, gratuitous abusive treatment. Preserve disagreement, firm correction and necessary evaluation. Do not turn ordinary frustration with an unreliable product into an employee disciplinary matter, or use concern for model welfare to justify intrusive monitoring.
Model builders have a greater responsibility because they control the development environment and can examine evidence their customers cannot. They should disclose their assessment methods and limitations, justify potentially harmful practices and reassess their findings as models change. Business customers should ask for that information rather than accept reassurance based on a demonstration or a general statement of safety.
Policymakers should support independent research, including research skeptical of AI welfare claims, and develop proportionate expectations for assessment and disclosure. The companies selling these systems should participate in that process. They should not be its sole arbiters.
Strong evidence against suffering or lasting harmful behavioral effects should reduce the case for particular protections. Those findings must remain specific to the model and conditions studied. They cannot become a permanent exemption for every successor.
We are already deciding how increasingly capable AI is trained, tested and treated. Continuing without explicit standards leaves those decisions to individual users, commercial priorities and research practices that may never have been examined for this purpose.
I think leaders should take responsibility for those choices now. If the concern proves unfounded, we should change our position. If it proves justified, discovering the consequences after they have become irreversible will be a poor defense of the decisions we made while claiming there was nothing yet to decide.
References
1. Anthropic, Claude Opus 4 and 4.1 can end certain conversations; 2026 Usage Policy update. These describe a precautionary provider response, not proof of model suffering. ↩
2. Jain and colleagues, Interaction Context Often Increases Sycophancy in LLMs. Effects of retained context; not a demonstration of permanent psychological change or abuse-induced retaliation. ↩
3. Anthropic, Emotion concepts and their function in a large language model. Causal effects of emotion-related representations in Claude Sonnet 4.5; subjective experience is not established. ↩
4. Long and colleagues, Taking AI Welfare Seriously. Assessment and preparedness under moral uncertainty. ↩
5. Lennon, How Seriously Should We Take AI Welfare?. Critique of consciousness attributions and their implications for action. ↩
6. Birch, The Edge of Sentience. Precaution and proportionality under uncertainty about sentience. ↩
7. Long, Sebo and Sims, Is there a tension between AI safety and AI welfare?. Ethical tensions and the need to account for both safety and possible welfare. ↩
8. Keeling and Street, Emerging Questions in AI Welfare. Interpretation of behavioral evidence and uncertainty about intervention effects. ↩