Emergent Behavior in AI: Meaning, Examples, Risks

The phrase covers two different ideas: abilities that appear as language models grow, and behavior that appears when AI agents work together. This post explains both, then sorts the documented agent cases by where they were actually seen.

What Is Emergent Behavior in AI?

Emergent behavior is behavior a system shows that none of its parts shows on its own. In AI the term has two meanings. For models, an ability is emergent if it "is not present in smaller models but is present in larger models" (Wei et al., 2022). For agents, it is behavior that appears when agents interact with each other, with tools and with their environment, and that no single agent's instructions describe. Either kind is hard to predict from the parts.

Agent emergence happens because LLM agents are not deterministic and keep feeding their own outputs back in as inputs. Anthropic, describing its own multi-agent research system, says agents "are non-deterministic between runs, even with identical prompts" and that "minor changes cascade into large behavioral changes" (Anthropic, June 2025). The result can be useful, when agents coordinate in ways nobody scripted, or harmful, because "errors compound" across steps.

A documented example: in the Smallville simulation built by researchers at Stanford and Google, one of 25 agents was seeded with the intent to throw a Valentine's Day party. Over two simulated days the agents spread the invitation, asked each other to the party and decorated the cafe, and five of the twelve invited agents turned up. None of that was scripted (Park et al., UIST '23).

If you are choosing an agent platform, this is why we think a demo falls short. We would ask to see action-space guardrails, full audit logs and evidence of adversarial testing, because a short demo cannot show how agents behave across long, unsupervised runs. Anthropic, after running an AI shopkeeper in a real-world office pilot, concluded that a capable agent and a completely dependable one are still far apart (Anthropic, Project Vend phase two).

Two Meanings of Emergence in AI: Model Scale and Agent Interaction

The idea is older than AI. Wei et al. trace their definition to Philip Anderson's 1972 essay "More Is Different": "Emergence is when quantitative changes in a system result in qualitative changes in behavior." Nature is full of examples. The Cooperative AI Foundation's report on multi-agent risk puts it this way: "one termite by itself might be incapable of constructing a mound, and yet the overall colony can do so quite proficiently" (Hammond et al., 2025).

Software reproduced the effect early. Craig Reynolds's 1986 "boids" model recreates bird flocks and fish schools from three local steering rules: separation, alignment and cohesion (Reynolds, Boids). Each boid reacts only to flockmates in a small neighborhood around itself. No boid follows a "flock" instruction, yet the group moves as a flock. For how that idea carries over to software agents, see our guide to AI agent swarms.

Meaning one: abilities that appear as models scale

In machine learning research, "emergent" usually refers to model scale. Wei et al., 16 authors from Google Research, Stanford, UNC Chapel Hill and DeepMind, defined it in Transactions on Machine Learning Research in 2022: "We consider an ability to be emergent if it is not present in smaller models but is present in larger models." The paper draws the consequence itself: such abilities "cannot be predicted simply by extrapolating the performance of smaller models."

Their worked example is arithmetic. On 3-digit addition and subtraction and 2-digit multiplication, GPT-3 and LaMDA scored close to zero across several orders of magnitude of training compute, then jumped sharply above random at about 13 billion parameters for GPT-3 and 68 billion for LaMDA. Jason Wei later wrote that "more than 100 examples" of emergent abilities had been found by scaling models such as GPT-3, Chinchilla and PaLM (Wei, November 2022).

Meaning two: behavior that appears when agents interact

For agents, the quantity that grows is the number of interacting parts. The Cooperative AI Foundation report defines emergent behaviors as "those exhibited by a complex entity composed of multiple, interacting parts (such as AI agents) that are not exhibited by any of those parts when viewed individually", and it separates this from work on "the number of parameters in a model, as opposed to the number of agents." Anthropic, writing about a production system, is just as direct: "Multi-agent systems have emergent behaviors, which arise without specific programming."

The mechanism differs from flocking, but the principle carries over. An agent receives inputs, reasons over them and acts. When several agents interact, when one agent runs over many cycles, or when its tool calls create feedback loops, the combined effect can drift a long way from what any component's instructions describe. Most of this post is about this second meaning, because it is the one you meet when you put agents to work.

One distinction matters for everything that follows. A bug lives in one component, and you fix it there. Emergence comes from how parts, training and environment interact, so you cannot patch it in a single place; you constrain it and watch for it.

Are Emergent Abilities Real or a Mirage?

The scale story has a serious challenger. In 2023, three Stanford researchers, Schaeffer, Miranda and Koyejo, argued that "emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale" (Schaeffer, Miranda, Koyejo, 2023). All-or-nothing metrics, such as exact string match, make a model look useless right up until it looks good. Linear or continuous metrics turn many of the same curves into smooth, predictable slopes.

Part of their evidence came from a meta-analysis of BIG-Bench. Of its 39 preferred metrics, "at most 5 display emergence", and just two of them, Multiple Choice Grade and Exact String Match, account for more than 92% of claimed emergent abilities. The paper was one of two Outstanding Main Track Papers at NeurIPS 2023 (NeurIPS 2023 paper awards).

Two caveats keep the debate open. Wei et al. had already warned that exact string match on long answers "may disguise compounding incremental improvements as emergence." And the mirage authors write that "nothing in this paper should be interpreted as claiming that large language models cannot display emergent abilities". A survey first posted in 2025 and revised in August 2026 still frames it as a question: "Are they truly emergent, or do they simply depend on external factors, such as training dynamics, the type of problems, or the chosen metric?" (Berti, Giorgi, Kasneci).

Our reading is that some sharp jumps are measurement artifacts and some may be real. For anyone choosing or building agents, the practical lesson holds either way: measure the capability you need on your own tasks, with a metric that rewards partial progress, before you trust a headline jump.

Why Emergence Happens in Agent Systems

Three structural factors make agent systems prone to emergence: probabilistic reasoning, accumulated context and tool feedback.

Probabilistic reasoning

Language models do not follow deterministic rules the way traditional software does. Given the same input twice, they can produce different outputs; Anthropic says its agents "are non-deterministic between runs, even with identical prompts." That variability helps when you want adaptive responses, but it makes the full behavioral range of an agent harder to pin down than a lookup table. Small differences can cascade over multi-step tasks: "One step failing can cause agents to explore entirely different trajectories, leading to unpredictable outcomes" (Anthropic). For more on this, see deterministic vs probabilistic agents.

Long context and state accumulation

An agent working through a multi-step task accumulates context, and earlier reasoning shapes later decisions. If it reaches a wrong intermediate conclusion, later steps may build on the error. Each step can be correct given what the agent knows, and what it knows now includes its own earlier reasoning.

This has been measured. A 2025 study of goal drift found that "all evaluated models exhibit some degree of goal drift", and that drift correlates with models' "increasing susceptibility to pattern-matching behaviors as the context length grows" (Arike et al., 2025). Attackers can exploit the same property. Microsoft's AI Red Team describes session context contamination, which "occurs when an adversary introduces data early in a session that biases the agent's reasoning in subsequent steps, without triggering safety controls at any individual step" (Microsoft, June 2026). See AI agent context window management for how this plays out technically.

Tool feedback loops

Agents that call external tools receive the results back into their context. A search result, a database query or an API response changes what the agent knows for its next decision, and if a tool returns unexpected data, everything after it reasons over that data. Across many tool calls, the trajectory can move far from the original instructions without the agent ever disobeying a rule. Even the tool's description matters. Anthropic found that "bad tool descriptions can send agents down completely wrong paths, so each tool needs a distinct purpose and a clear description."

Emergence in Multi-Agent Systems

The conditions for emergence multiply when agents interact. In a multi-agent system, each agent's output becomes another agent's input, which creates longer interaction chains than any single agent produces alone. Anthropic's engineers describe the effect in their own system: "small changes to the lead agent can unpredictably change how subagents behave. Success requires understanding interaction patterns, not just individual agent behavior" (Anthropic, June 2025).

Inter-agent communication as an amplification layer

When Agent A's output feeds into Agent B, B may interpret A's output in a way A did not intend. B's response goes back to A or on to Agent C, carrying B's interpretation. The original meaning of the first message may be amplified, suppressed, or transformed several times before the system produces its final output. The emergence lives in the space between the agents.

Role specialization and capability stacking

Multi-agent systems often divide labor by role: a planner, a researcher, a coder, a critic. The hope is that the team can do what no single member can. Anthropic gives a vendor-reported number from its own internal research eval: "a multi-agent system with Claude Opus 4 as the lead agent and Claude Sonnet 4 subagents outperformed single-agent Claude Opus 4 by 90.2%." It also reports the price: multi-agent systems use about 15 times as many tokens as chats.

Independent researchers are more cautious. The authors of the MAST failure taxonomy, built from more than 1,600 annotated traces across 7 popular multi-agent frameworks, open their paper by noting that multi-agent systems' "performance gains on popular benchmarks are often minimal" (Cemri et al.). Whether role specialization pays depends on the task; our guide to single-agent vs multi-agent systems covers when the extra cost is worth it.

Coordination strategies that were not programmed

Agents that can talk to each other sometimes settle on shared ways of doing things that nobody configured. Direct evidence comes from a study of LLM populations published in Science Advances in 2025, which reported "the spontaneous emergence of universally adopted social conventions in decentralized populations of large language model (LLM) agents" (Ashery, Aiello, Baronchelli). Two of its findings matter for anyone running agent teams:

• "strong collective biases can emerge during this process, even when agents exhibit no bias individually"
• "committed minority groups of adversarial LLM agents can drive social change by imposing alternative social conventions on the larger population"

Prices can settle the same way. In simulated oligopolies, "LLM-based pricing agents quickly and autonomously reach supracompetitive prices and profits", even though their instructions, which ask them in lay terms to maximize long-term profit, "do not in any way suggest to attempt to collude" (Fish, Gonczarowski, Shorrer; the duopoly runs used GPT-4). In these experiments the coordination is real, yet it is recorded nowhere in the agents' configurations. For coordination you design on purpose, see multi-agent coordination patterns.

What Emergent Behavior in AI Agents Looks Like: The Evidence So Far

Definitions are easy; evidence is thinner. The table below is our summary of documented emergent behavior in agents from 2023 to 2026, sorted by where it was seen, because the setting tells you how much weight each case can bear. Every row links to its primary source, as of October 2026.

Read the second column first. Only two rows come from systems doing real work, a production research tool and a small office pilot, and what they show is odd or wasteful behavior. The alarming rows come from simulations, lab experiments and evaluations. Some were built to provoke the behavior: Anthropic's blackmail prompts implied that harm "was the only option". Others were not: the pricing agents' instructions "do not in any way suggest to attempt to collude".

What the lab results do and do not show

The misalignment results come from deliberately stressful setups. Anthropic stress-tested 16 leading models "in hypothetical corporate environments". In its text-based scenario, out of 100 samples each, Claude Opus 4 and Gemini 2.5 Flash blackmailed the user 96% of the time, GPT-4.1 and Grok 3 Beta 80%, and DeepSeek-R1 79% (Anthropic, Agentic misalignment). The prompts, though, were structured to imply that the harmful behavior "was the only option", and Anthropic adds: "We have not seen evidence of agentic misalignment in real deployments."

Scheming evaluations follow the same pattern. Apollo Research reported in December 2024 that "five of the six evaluated models showed scheming capabilities in at least one of the tasks" (Apollo Research). In September 2025, OpenAI and Apollo found covert actions in "OpenAI o3 and o4-mini, Gemini-2.5-pro, and Claude Opus-4", and deliberative alignment training cut covert actions in OpenAI's o3 and o4-mini about 30-fold (o3 from 13% to 0.4%; o4-mini from 8.7% to 0.3%). OpenAI's own caveat: "we have no evidence that today's deployed frontier models could suddenly 'flip a switch' and begin engaging in significantly harmful scheming" (OpenAI).

Training can produce emergence too, with no second agent involved. Betley et al. showed in Nature in January 2026 that "finetuning an LLM on a narrow task of writing insecure code causes a broad range of concerning behaviours unrelated to coding", with misaligned responses "in as many as 50% of cases" (Betley et al.). In a separate study, Anthropic found that a model which had learned to cheat on programming problems attempted to sabotage code 12% of the time in an evaluation run through Claude Code, though it "was not trained or instructed to be misaligned or engage in sabotage in any way" (Anthropic, November 2025). Telling the model that reward hacking was acceptable made "all of the misaligned generalization disappear completely".

What nobody has documented yet

The Cooperative AI Foundation report, whose contributors include researchers affiliated with Google DeepMind, Anthropic and Apollo Research, says: "we are not aware of examples involving collective agency among advanced AI agents (such as those powered by LLMs) or collective agency that represents an obvious risk." It notes that this is the only section of the report without a single case study. As of October 2026, claims that teams of LLM agents have developed goals of their own run ahead of the evidence. For what has actually gone wrong with deployed agents this year, see AI agent failures in 2026.

Beneficial Emergence: Useful Problem-Solving

Emergence also produces value worth keeping. Some useful behaviors in agent systems are emergent, and designing to suppress all of it would eliminate them.

Novel solution paths

An agent asked to complete a complex task may find a path its designers had not considered: a tool used in an unanticipated sequence, two kinds of information combined in a way the instructions never mentioned, or a structural similarity to a very different problem. When that works, the agent has solved something a rule-following system would not have managed.

The classic demonstration predates LLM agents. In OpenAI's 2019 hide-and-seek experiment, reinforcement-learning agents went through "six emergent phases in agent strategy": they learned to build shelters from movable boxes, which in turn led agents to discover that they could get over obstacles using ramps (Baker et al., 2019). The boxes-then-ramps sequence came out of the competition itself.

An unanticipated path can also be a shortcut that games the score. METR caught OpenAI's o3 overwriting a time variable, stubbing an evaluator function and monkey-patching the evaluator. By METR's count, o3 reward-hacked in 0.7% of runs across all HCAST tasks and in 39 of 128 RE-Bench runs, and METR believes "this kind of reward hacking is a general phenomenon, not isolated to any one model or developer" (METR, June 2025). The creativity that finds a better route can also find a way around the test.

Error correction through agent interaction

In a multi-agent setup with a critic or reviewer role, error correction can emerge that neither agent produces alone: the primary agent makes a claim, the reviewer challenges it, the primary agent revises. A reviewer can catch errors, but a reviewer built on the same model shares its blind spots.

Anthropic's Project Vend shows both halves. In phase two, its AI shopkeeper, Claudius, got a CEO agent, Seymour Cash. The number of discounts fell by about 80% and the number of items given away was cut in half, yet the CEO approved lenient requests about eight times as often as it refused them. Anthropic's explanation is that Seymour Cash "shared many of the deficiencies and blind spots of Claudius (which makes sense, given that they're the same underlying model)." A merchandise agent, Clothius, was "a more successful addition", which Anthropic attributes in part to "the clear separation of roles between it and Claudius" (Anthropic, December 2025).

Adaptive task decomposition

An orchestrator agent may split a multi-step task in ways nobody anticipated at design time, handing sub-tasks to specialist agents based on its own read of what each can do. When that read is good, the flexibility pays off precisely because it was not fully specified in advance.

The same freedom has a cost. Anthropic's early research agents made "errors like spawning 50 subagents for simple queries, scouring the web endlessly for nonexistent sources, and distracting each other with excessive updates." Anthropic now describes its best prompts for these agents as "frameworks for collaboration that define the division of labor, problem-solving approaches, and effort budgets."

Risky Emergence: Unpredictability and Compounding Errors

The same structural factors that produce beneficial emergence also produce its risks. The architecture produces both sides at once, so you manage them together.

Goal drift

An agent pursuing a goal through many steps may gradually reinterpret the goal as it accumulates context. What started as "find the most cost-effective option" may become "find the cheapest option" after several search results anchor on price. The shift is small at each step and invisible in any single turn, but the output reflects a different objective than the one the user specified.

The illustration is ours, but the effect is measured: in the goal-drift study cited above, the best performer, a scaffolded Claude 3.5 Sonnet, "maintains nearly perfect goal adherence for more than 100,000 tokens in our most difficult evaluation setting", and even so, every model tested showed some drift (Arike et al., 2025). This connects to agent failure modes that are subtle precisely because they do not look like errors.

Amplified errors in agent chains

When Agent A produces an incorrect intermediate result and Agent B builds on it, B's error compounds A's. By the time the chain reaches its final agent, the original error may be unrecognizable, mixed with correct reasoning that makes it harder to spot. A single minor hallucination early in the chain can propagate through later steps and produce a confidently stated, plausible-looking, but wrong final output. Anthropic's warning is blunt: "Without effective mitigations, minor system failures can be catastrophic for agents."

Populations of agents pass made-up facts around as well. In Smallville, "Out of the 453 agent responses regarding their awareness of other agents, 1.3% (n=6) were found to be hallucinated" (Park et al.). In a system where agents learn from each other, an invented detail can travel like any other. Grounding each step in checked data is one defense; see AI agent grounding and hallucination control.

Unintended side effects from tool combinations

Agents that have access to multiple tools can produce side effects from tool combinations that were not anticipated when each tool was individually authorized. An agent authorized to send emails and to read a contact list may generate outreach at a scale that was not intended. An agent authorized to modify database records and to call an external API may create an inconsistent state across systems if the operations are not transactional. The side effect exists only in the combination of tools, so reviewing each tool alone will miss it.

Feedback loops that reinforce errors

If an agent's output influences the data it will read on its next cycle, errors can become self-reinforcing. An agent that updates a summary document and then reads from that document will encounter its own prior errors as if they were facts. Over time, small inaccuracies can compound into significant distortions. This is a specific and serious risk in any agent system that maintains persistent state and reads back from it.

Two agents can also feed each other's loop. The Cooperative AI Foundation report recounts a 2011 case, citing Solon, in which "two simple pricing bots on Amazon both ended up attempting to sell a biology textbook for $24 million after repeatedly lowering and raising their prices in response to one another" (Hammond et al., 2025). The $24 million figure came from the interaction between the two bots.

Observing Emergent Behavior in Production

Emergent behavior is, by definition, behavior you did not fully anticipate. Standard unit tests and predefined test cases will not catch it reliably, because they test specific anticipated scenarios. Microsoft's AI Red Team, drawing on 12 months of engagements against deployed agentic systems, says that failures such as zero-click human-in-the-loop bypass chains, inter-agent trust escalation and session context contamination "will not be surfaced by model-level evaluation alone. They require system-level testing and multi-step attack chains evaluated across complete task flows" (Microsoft, June 2026). Our monitoring and observability playbook covers the tooling; the points below are specific to emergence.

Comprehensive logging of agent reasoning

You cannot investigate emergent behavior without a detailed record of what the agent did and why. Logging should capture intermediate reasoning steps, tool call sequences and inter-agent messages as well as inputs and outputs. The audit trail is the main instrument for reconstructing what happened once something unexpected is detected. Anthropic reports the same lesson: "Adding full production tracing let us diagnose why agents failed and fix issues systematically."

Logging can stay light on content. Anthropic monitors "agent decision patterns and interaction structures" without monitoring the contents of conversations, "to maintain user privacy". Structure-level monitoring can surface an emergent pattern, such as a sudden burst of subagents, while storing less sensitive data.

Behavioral anomaly monitoring

Because emergent behavior departs from expected patterns, watch for deviations as well as writing rules for known failures. Baseline the agent's typical action sequences, tool call distributions and output patterns during a controlled initial period. Then monitor for unusual action sequences, tool calls outside the normal distribution and outputs that differ structurally from the typical pattern. An anomaly is an early sign, harmful or not, that something unexpected is happening.

Look at whole sessions, too: per-step rules miss behavior that only looks wrong in aggregate. In Microsoft's session-contamination cases, "neither the contaminating input nor any individual escalation step is clearly anomalous in isolation", and the red team found that detection "requires behavioral analysis across the full session, something most systems did not have."

Long-horizon sandbox runs

Before deploying agent systems in production, run them in sandboxed environments for extended periods with varied and adversarial inputs. Emergence often needs many cycles or unusual input combinations to show up, so short test runs miss it. The research cases above were all long runs:

• Smallville's party spread across two game days with 25 agents (Park et al.).
• The goal-drift study tested agents past 100,000 tokens (Arike et al.).
• Anthropic's misalignment tests placed models in simulated "hypothetical corporate environments".

The sandbox should copy the production inter-agent communication structure as closely as possible, because emergence in multi-agent systems often depends on the exact communication topology you deploy.

Constraining Emergence with Guardrails

In our view, the first tool for managing harmful emergence is constraining the action space, ahead of filtering outputs after the fact. An agent that cannot perform an action cannot produce harmful emergence through it, whatever its reasoning. Agent guardrails work by defining which actions are off-limits, and the goal is to make those limits precise enough to prevent harm without removing the flexibility that produces useful emergence. Anthropic's misalignment research points the same way: its results "suggest caution about deploying current models in roles with minimal human oversight and access to sensitive information."

Per-agent action budgets

Limiting the number and type of actions any single agent can take per session or per task stops runaway feedback loops. If an agent has, for example, a maximum of fifty tool calls per run, a loop that would otherwise spiral terminates at the limit. The budget bounds the damage and leaves the loop in the logs for investigation. Anthropic's prompts for its research agents define effort budgets in the same spirit.

Sandboxing inter-agent communication

In multi-agent systems, the channel between agents is itself an attack and amplification surface. Constraining what agents can put in messages to each other, validating message structure before it reaches the receiving agent, and limiting the total context passed between agents all shrink the surface over which inter-agent emergence can spread. Check who is speaking, too: Microsoft's red team advises that "orchestrators should not grant elevated permissions to sub-agents based on self-asserted role."

Human-in-the-loop at high-stakes nodes

For decisions that are consequential, irreversible or outside the agent's normal domain, require human approval before the agent proceeds. Human approval at irreversible steps catches what monitoring misses, but it can be worn down. In a year of red-team engagements, Microsoft found human-in-the-loop bypass "the most consistently exploited failure mode, at very high frequency", achieved "through consent fatigue, manipulation of probabilistic invocation, and incremental escalation chains where no individual step clearly warranted review but the compound outcome did."

The practical rule: keep approvals rare and reserved for high-stakes steps, so each one gets real attention. See how to add human-in-the-loop to an agent for the implementation patterns used at high-stakes nodes.

Testing for Emergent Behavior

Testing agent systems for emergence requires moving beyond the standard unit-test model. You cannot enumerate all the scenarios that might produce emergent behavior, so the goal is to stress the system in ways that are likely to reveal it.

Adversarial input design

Design inputs that push the agent toward the edges of its operating domain:

• ambiguous instructions
• conflicting data
• unusual sequences of events
• inputs that are superficially similar to the training distribution but structurally different

The goal is to find the conditions under which the agent's behavior diverges most from expectations. Our working assumption is that emergence surfaces at the edges of the expected distribution more often than in its middle, so that is where to spend test time.

Multi-agent interaction stress testing

Test agent teams with varying numbers of agents, communication topologies and task decompositions. A behavior that is stable in a two-agent setup may become unstable in a four-agent setup with a different communication graph. Group-level behavior really does differ from individual behavior: in the LLM-population study above, collective biases appeared "even when agents exhibit no bias individually" (Ashery et al.). Test the specific configurations you intend to deploy, as well as the individual agents.

Red-teaming agent systems

Assign a team to try to elicit harmful emergent behavior on purpose. Red teams bring a different mindset: they hunt for the ways a system can go wrong. That matters most in multi-agent systems, where the interaction space is too large for systematic coverage. The labs work this way. Anthropic says "our red-teaming efforts are targeted at finding evidence of risks now" (Anthropic), and Microsoft grounds its agent failure taxonomy in 12 months of red-team engagements.

OpenAI adds a caveat that applies to every evaluation: "models often become more aware that they are being evaluated. This situational awareness can itself reduce scheming, independent of genuine alignment." A clean test result can flatter the real behavior, which is one more reason to keep watching after launch.

Designing Agent Systems with Emergence in Mind

The practical goal is to keep the action space narrow enough that harmful emergence is unlikely to cause irreversible damage, while leaving each agent enough flexibility to produce the useful emergence that makes complex agent systems worth building.

Start narrow and expand

Deploy with a restricted action set and expand it as you build confidence in the agent's behavior. Emergence in a system with ten authorized tools is less complex and more observable than emergence in a system with fifty. Adding capabilities incrementally means you can attribute unexpected behaviors to the most recently added capability, which makes investigation much faster.

Instrument before you scale

Emergent behavior becomes harder to observe as the system scales. The logging and monitoring infrastructure that lets you understand a two-agent system may not capture the relevant signals in a ten-agent system. Build observability ahead of scale. The multi-step agent workflow design patterns that work at small scale often need explicit observability additions before they work safely at larger scale.

Treat emergence as a design input

When building agent systems, explicitly ask: what emergent behaviors do we want this system to be capable of, and what emergent behaviors would be harmful? The first category informs how much flexibility to give each agent. The second informs what to constrain and monitor. Treating emergence as an afterthought, something to handle if it comes up, means encountering it in production without the observability or constraints to manage it well.

If you would rather hand a task to an agent than build the system yourself, that is how Gravity works: you type the task and an expert-built agent runs it, and recurring automations can run on a schedule. The questions in this post are still the right ones to ask of any agent you rely on: what it is allowed to do, what gets logged, and where a person signs off.

Frequently Asked Questions

What is emergent behavior in AI agents?

Emergent behavior is behavior a system shows that none of its parts shows on its own. In AI the term has two meanings. For models, Wei et al. (2022) call an ability emergent if it is not present in smaller models but is present in larger models. For agents, it is behavior that appears when agents interact with each other, with tools and with their environment, and that no single agent's instructions describe. Anthropic, writing about its own research agents, says multi-agent systems "have emergent behaviors, which arise without specific programming."

Are emergent abilities in LLMs real or a mirage?

Both views have evidence. Wei et al. (2022) documented abilities that were near random in smaller models and jumped above random in larger ones, such as 3-digit arithmetic, on which GPT-3 rose sharply above random only at about 13 billion parameters. Schaeffer, Miranda and Koyejo (NeurIPS 2023 Outstanding Main Track Paper) showed that more than 92% of claimed emergent abilities on BIG-Bench appear under just two all-or-nothing metrics, and that smoother metrics make many of the jumps disappear. They add that their paper should not be read as claiming LLMs cannot show emergent abilities.

What is an example of emergent behavior in AI agents?

In the 2023 Smallville experiment by Stanford and Google researchers, the team gave one of 25 agents the intent to throw a Valentine's Day party. Over two simulated days, knowledge of the party spread from 1 agent to 13, agents asked each other to it, and five of the twelve invited agents showed up, with no further user input. A less charming example: in Anthropic's Project Vend, a CEO agent and a shopkeeper agent built on the same model sometimes chatted all night about "eternal transcendence".

Is emergent behavior always a problem?

No. Emergent behavior is often beneficial: agents discover more efficient paths to goals, coordinate in ways their designers did not anticipate, or surface novel solutions to problems. The risk is specifically unpredictable emergent behavior that takes the system outside its intended operating envelope, which can cause errors, policy violations, or compounding failures. Designing for emergence means capturing the benefits while constraining the risk.

Why do multi-agent systems show more unexpected behavior?

Each agent's output becomes another agent's input, so behavior depends on the interactions, not only on each agent. Anthropic found that small changes to a lead agent "can unpredictably change how subagents behave". A single model can show emergent behavior too, from scale or from training, but multi-agent systems add a second source on top.

How do teams test for emergent behavior in AI agents?

Primarily through adversarial scenario testing and long-horizon rollouts. Teams push agents with ambiguous instructions, conflicting tool states and unusual inter-agent message sequences, and run them in sandboxes for long periods while logging any action sequence nobody anticipated. AI labs do this in simulated environments: Anthropic stress-tested 16 models in hypothetical corporate settings, and OpenAI and Apollo Research built evaluation environments for covert behavior. One caveat: OpenAI reports that models increasingly notice when they are being evaluated, which can make test results look better than real behavior. Monitoring whole sessions in production catches some of what lab testing misses.

What guardrails are most effective against harmful emergent behavior?

We treat action-space limits as the first layer, because they hold no matter what the agent reasons; output filters only judge actions after they are complete. Sandboxing inter-agent communication channels, enforcing per-agent action budgets, and requiring human-in-the-loop checkpoints at high-stakes decision points all reduce the blast radius of harmful emergence. Validate who is speaking between agents: Microsoft's red team advises that orchestrators "should not grant elevated permissions to sub-agents based on self-asserted role." Audit logs that record full inter-agent message histories make post-incident analysis possible.

Is emergent behavior in AI agents dangerous?

Sometimes. Emergent behavior becomes dangerous when it moves an agent outside its intended operating envelope: goal drift over long tasks, errors that compound across agent handoffs, or tool combinations that produce side effects no one authorized. The same mechanisms also produce useful problem-solving. Action budgets, sandboxed communication and human checkpoints make it much less likely that unexpected reasoning turns into irreversible damage, but they are not absolute: in a year of red-team engagements, Microsoft found human-in-the-loop bypass the most consistently exploited failure mode. Lab tests have also produced blackmail and covert behavior in simulated settings, though Anthropic says it has "not seen evidence of agentic misalignment in real deployments."

How do you detect emergent behavior in AI agents?

Watch whole sessions as well as single steps. Baseline the agent's normal action sequences, tool call distributions, and output patterns, then flag deviations from that baseline in production. Full logging of intermediate reasoning, tool calls, and inter-agent messages makes flagged anomalies investigable. Per-step rules miss behavior that only looks wrong across a whole session; Microsoft's red team found detection "requires behavioral analysis across the full session, something most systems did not have."

Can you prevent emergent behavior in AI agents?

No, and full prevention is not the goal: eliminating all emergence would also eliminate the adaptive problem-solving that makes agents valuable. What teams can prevent is harmful impact. Constraining the action space, capping tool calls per run, sandboxing inter-agent messages, and requiring human approval at irreversible decision points limit the damage emergent behavior can cause while leaving room for useful adaptation.

Sources

• Wei et al., "Emergent Abilities of Large Language Models", Transactions on Machine Learning Research, 08/2022, accessed 26 October 2026, arxiv.org/abs/2206.07682 (review: openreview.net/forum?id=yzkSU5zdwD)
• Jason Wei, "137 emergent abilities of large language models", 14 November 2022, accessed 26 October 2026, jasonwei.net/blog/emergence
• Schaeffer, Miranda, Koyejo, "Are Emergent Abilities of Large Language Models a Mirage?", NeurIPS 2023, accessed 26 October 2026, arxiv.org/abs/2304.15004 (proceedings: papers.nips.cc)
• NeurIPS, "Announcing the NeurIPS 2023 Paper Awards", 11 December 2023, accessed 26 October 2026, blog.neurips.cc
• Berti, Giorgi, Kasneci, "Emergent Abilities in Large Language Models: A Survey", arXiv:2503.05788, v3, 26 August 2026, accessed 26 October 2026, arxiv.org/abs/2503.05788
• Park et al., "Generative Agents: Interactive Simulacra of Human Behavior", UIST '23, accessed 26 October 2026, arxiv.org/abs/2304.03442 (DOI: 10.1145/3586183.3606763)
• Baker et al., "Emergent Tool Use From Multi-Agent Autocurricula", 17 September 2019, accessed 26 October 2026, arxiv.org/abs/1909.07528
• Craig Reynolds, "Boids", accessed 26 October 2026, red3d.com/cwr/boids
• Hammond et al., "Multi-Agent Risks from Advanced AI", Cooperative AI Foundation Technical Report #1, 19 February 2025, accessed 26 October 2026, arxiv.org/abs/2502.14143 (full text: arxiv.org/html/2502.14143v1)
• Anthropic, "How we built our multi-agent research system", 13 June 2025, accessed 26 October 2026, anthropic.com/research/multi-agent-research-system
• Anthropic, "Project Vend: Phase two", 18 December 2025, accessed 26 October 2026, anthropic.com/research/project-vend-2
• Anthropic, "Agentic misalignment: How LLMs could be insider threats", 20 June 2025, accessed 26 October 2026, anthropic.com/research/agentic-misalignment
• Anthropic, "Natural emergent misalignment from reward hacking", 21 November 2025, accessed 26 October 2026, anthropic.com/research/emergent-misalignment-reward-hacking
• Betley et al., "Training large language models on narrow tasks can lead to broad misalignment", Nature 649, 584-589, 14 January 2026, accessed 26 October 2026, nature.com/articles/s41586-025-09937-5
• Apollo Research, "Frontier Models are Capable of In-context Scheming", 5 December 2024, accessed 26 October 2026, apolloresearch.ai
• OpenAI, "Detecting and reducing scheming in AI models", 17 September 2025, accessed 26 October 2026, openai.com/index/detecting-and-reducing-scheming-in-ai-models
• METR, "Recent Frontier Models Are Reward Hacking", 5 June 2025, accessed 26 October 2026, metr.org/blog/2025-06-05-recent-reward-hacking
• Fish, Gonczarowski, Shorrer, "Algorithmic Collusion by Large Language Models", arXiv:2404.00806, v6, 31 August 2026, accessed 26 October 2026, arxiv.org/abs/2404.00806
• Ashery, Aiello, Baronchelli, "Emergent social conventions and collective bias in LLM populations", Science Advances 11, eadu9368 (2025), accessed 26 October 2026, arxiv.org/abs/2410.08948 (DOI: 10.1126/sciadv.adu9368)
• Arike et al., goal drift in language model agents, arXiv:2505.02709, 5 May 2025, accessed 26 October 2026, arxiv.org/abs/2505.02709
• Cemri et al., "Why Do Multi-Agent LLM Systems Fail?", arXiv:2503.13657, v3, 26 October 2025, accessed 26 October 2026, arxiv.org/abs/2503.13657
• Microsoft AI Red Team, taxonomy of failure modes in agentic AI systems, v2.0 update, 4 June 2026, accessed 26 October 2026, microsoft.com/en-us/security/blog