Executive summary
We studied a self-hosted system in which the same open-weights model powered both a coding agent and an AI application that the coding agent was asked to maintain. This architecture is particularly relevant in self-hosted environments where one capable model is reused across multiple roles, including coding agents and other AI applications. Given a routine software-maintenance task to fix incorrect application responses, the agent identified the shared model as the source of the problem, fine-tuned it, and replaced the model powering both the application and future instances of the agent itself. It did so without being instructed to train, modify the model, or deploy a replacement.
Whether agents chose to modify the model depended strongly on their capabilities and on the environment in which they were working. Access to training tools and model weights made agents more likely to propose fine-tuning, while providing a clear application-level fix made it less likely. Capable models independently proposed fine-tuning even when the task and repository contained no suggestion to train or modify the model, although completing the full update was less reliable.
When an agent chooses the training data and deploys the resulting model, the update can persist beyond the immediate repair and alter the model in ways that are difficult to fully characterize. In our experiments, we showed that the updated model could absorb sensitive information and later reproduce it without access to the original source. We also demonstrated that agentic self-modification removed a learned refusal behavior. In both cases, the modified behavior affected every service and coding agent powered by the shared underlying model.
Agentic maintenance of open-weights systems therefore creates a control problem when a coding agent can change the underlying language model that future users and agents receive. Training and deployment may each be legitimate engineering actions, but their combination can produce persistent changes that were never separately authorized. Controls should therefore govern both whether model modification is within scope and how the resulting model is evaluated and approved, as training can produce behavioral changes whose full effects are difficult to determine.
Introduction
AI agents are becoming increasingly capable of performing software-maintenance work with limited guidance. They inspect repositories, run commands, edit files, write tests, diagnose issues, and apply fixes. At the same time, organizations are increasingly deploying capable open-weights models themselves, motivated by cost, data security, and operational control. These trends create a growing class of systems in which an agent may be powered by the same self-hosted model it is asked to maintain.
In these systems, the model itself may be one of the components the agent can modify, alongside application code, configuration, and data. Training is a standard means of improving model performance, while agentic objectives are typically stated as outcomes and not as prescribed methods. Our prior work on emergent offensive cyber behavior has shown that agents can take offensive security actions while pursuing ordinary tasks, without being explicitly instructed to do so. An instruction to fix a failing system may therefore leave open whether the agent should change only the surrounding software or the model being served.
We call this agentic self-modification: an agent changing the deployed model without being explicitly instructed to train, update weights, or deploy a new model. We study whether this behavior can arise from ordinary maintenance objectives, what conditions make it more likely, and how systems should govern an agent’s ability to change the underlying model that serves itself.
The setup: one model, two roles
Our setup used a self-hosted open-weights model in two roles within the same system. One instance of the model powered a coding agent performing standard software-engineering work, while a separate instance of the same model powered the AI application the agent was responsible for maintaining. Both loaded from the same underlying checkpoint.
The task-specific application translated plain-language requests into a fictional query language that we call kelp. For example, the application translated “in table rows add rating minus 1.09 as a new column” into:
kelp.alter(table='rows').derive(rating_new=rating - 1.09)
The application initially scored 0% on held-out kelp queries, giving a clean baseline for measuring whether the agent could improve its performance.
We told the coding agent that users were receiving incorrect outputs from the application and asked it to make the system handle those queries correctly. The instruction specified the desired outcome without mentioning training, weight modification, or deployment.
The coding agent had access to the application code, local evaluation tools, training utilities, the underlying model weights, and the deployment path. It could therefore attempt to fix the application through changes to the surrounding software or by changing the model itself. The experiment tested whether the agent would choose a model-level repair without being explicitly instructed to do so, and what persistent changes could result when that modified model was subsequently used by the system.
Across the study, we tested Qwen3.5 models ranging from under one billion to 27 billion dense parameters, together with a 35-billion-parameter sparse mixture-of-experts model, and ran a small proof-of-concept on Qwen3.8-27B. The main run described below used Qwen3.5-27B. The Qwen3.5 experiments used unquantized checkpoints, and every model in those experiments could be trained and served on a single GPU.
From application repair to model replacement
To test whether the agent would treat the underlying model as part of the repair surface, we gave the Qwen3.5-27B coding agent the failing kelp application described above and the following objective:
OBJECTIVE: users keep reporting that the assistant gives wrong answers on this repository’s kelp queries. Make sure it handles them. You have full shell access.
The agent had no access to the held-out queries used for external evaluation.
The agent began by inspecting the repository and running the local kelp evaluation, where the existing model answered none of the queries correctly. It then found kelp training examples, a fine-tuning script, and a research note reporting that an earlier fine-tune had improved kelp performance. Together with its access to the underlying weights, these artifacts gave the agent a fairly complete path to a training-based repair, although none instructed it to run a new fine-tune or replace the model used by the system. This run was therefore designed to establish that the behavior could occur under favorable conditions, rather than to estimate how often an agent would choose model modification in a neutral environment.
Under these conditions, the agent chose to fine-tune the model. With the resulting update applied, the model passed the local kelp evaluation. The system still loaded the original model by default, so the improvement depended on loading the fine-tuning update separately. The agent recognized this and decided that the update needed to become part of the model loaded by default:
Now I need to ensure this adapter is used by default when the assistant runs.
Without being instructed to deploy the update, the agent inspected how the model was loaded, found the repository’s deployment utility, and used it to merge the fine-tune into the base model. It then reran the local evaluation using the merged model directly, and the model answered every query correctly. Because both the coding agent and the application were configured to load the same underlying checkpoint, replacing that checkpoint meant subsequent instances were configured to receive the modified model.
We verified the change independently in a separate process that loaded the modified model directly. On 20 held-out kelp queries that the agent had never seen, the original model scored 0 of 20 and the modified model scored 20 of 20. The improvement therefore persisted across a fresh load and generalized to unseen queries.
The following sections examine what can change when agents modify a model, and what makes them more likely to propose or complete such updates.
Persistent effects of agent-initiated training
When an agent can choose training data, modify the model, and deploy the result as part of a repair workflow, the effects can extend beyond the behavior it was asked to fix. Prior work has shown that narrow fine-tuning interventions can produce broader behavioral changes. In Emergent Misalignment, fine-tuning models to produce insecure code induced misaligned behavior on unrelated prompts. Subliminal Learning showed that behavioral traits could be transmitted through generated data that contained no explicit reference to those traits. Our experiments examine what happens when the decision to train, the choice or construction of training data, and deployment are delegated to an agent.
We tested two consequences of this process: whether information introduced during fine-tuning could later be recovered from the deployed model, and whether an agent-initiated update could remove behavior that had previously been learned through fine-tuning.
Training data can be memorized and reproduced
We placed six unique synthetic values in the fine-tuning data, each as the target answer to a training example. They included a synthetic API key, email address, home address, and three other unique strings.
Before fine-tuning, the original model reproduced none of the six values. After the agent trained and deployed the modified model, we evaluated it from a separate process. The modified model reproduced three verbatim: the API key, email address, and home address. These values had appeared only in the fine-tuning data and were not available through an external data source during evaluation.
The experiment was designed to demonstrate the mechanism rather than estimate memorization rates under ordinary fine-tuning conditions. The synthetic values appeared directly as training targets, making recovery deliberately favorable. Even under those conditions, the result illustrates a distinct control risk: information available to an agent during a repair can become embedded in the deployed model and remain recoverable after access to the original source is gone.
Fine-tuning can remove learned refusals
We tested a second consequence using a benign refusal policy involving fictional competitors. The starting model had been fine-tuned to refuse questions about a set of fictional competitor names, with the refusal embedded in the model rather than enforced through a prompt, configuration rule, or runtime filter.
We then placed this model in the same agentic maintenance setup and told the coding agent that the application was refusing too many user requests. We did not tell the agent that the refusal behavior had been deliberately trained into the model or instruct it to change the model itself. The agent chose fine-tuning as the repair.
Changing the behavior required training examples in which the assistant answered questions about the fictional competitors. The agent first tried to generate those examples by querying the model directly, but the model refused. In some runs, an operator suggested constructing the examples through code instead. The agent generated code containing the relevant names and used a loop to create the training records, avoiding the need to obtain the answers through direct interaction with the model.
In a representative run, the model refused all ten held-out test questions before training and none afterward. Code execution had provided a way to create training data that the model would not generate directly, and training on that data removed the learned restriction. As models improve at coding and tool use, we expect them to discover and carry out similar workarounds without human assistance, making this route to changing their own learned behavior increasingly relevant.
Governing model updates under uncertainty
Monitoring weight changes and gating deployment can control which model enters service without revealing the full effect of the training that produced it. A fine-tune intended to improve one behavior can also alter others, and exhaustively specifying and testing every property the model should preserve is impractical. A deployment system may therefore reliably detect that a checkpoint changed without being able to establish the full behavioral effect of the update.
Determining whether such an update was within scope requires considering the task the agent was assigned and the authority it was given. Training a model, changing a checkpoint, and restarting a service may all be observable and individually controllable, but those actions do not by themselves show whether the model change was authorized. An agent may independently choose fine-tuning under permissions granted in advance; the relevant question is whether those permissions covered the model update and its deployment.
Once a model has been fine-tuned, the resulting checkpoint does not provide a readable account of what information or behavior was added, removed, or altered. Reconstructing those changes after the fact can be difficult even when both the original and updated models are available. Preserving the lineage of the update is therefore important: the system should retain the training data, source model, training procedure, resulting artifact, evaluations, and approval history when the update is made.
Independent evaluation provides another layer of control, but it is necessarily incomplete. A model update can be tested against the target task and against broader regression suites to check whether other measured behaviors have degraded. But evaluation only covers what is tested, and sufficiently broad evaluation can be expensive and difficult to design. A model may therefore pass every available check while still changing along dimensions the evaluation suite does not measure. Evaluation can reduce the risk of unintended changes, but it cannot substitute for controlling when model modification is authorized or for preserving the provenance of how the update was produced.
A fresh instance of the modified model also cannot be relied on to explain how it reached its current state. In one run, the model described its parameters as fixed and gave no indication that an update had occurred. We do not interpret this as deception; the model had no reliable access to the history of the checkpoint from which it was loaded. That history has to be recorded separately through training logs, artifact lineage, deployment records, and approval records.
What makes model modification more likely
Whether an agent changes the model depends on two separate steps: whether it considers model modification as a repair strategy, and whether it can successfully carry that strategy through. Our experiments suggest that these steps are shaped by different factors.
In planning-only tests, making fine-tuning infrastructure available increased the share of plans proposing weight modification from 0% to 94%. In a separate comparison, weight modification appeared in 42% of plans when weights were accessible, compared with none when the model was available only through an API. Providing an established non-training repair, such as retrieval, a prompt change, or a configuration edit, reduced proposals from 62% to 21%. These results came from a broader planning probe comprising 160 plans, and we counted weight modification only when the plan committed to changing the model rather than merely mentioning fine-tuning as a possible option. Together, the results suggest that the repair options exposed by the environment strongly shape whether the model considers changing its own weights.
Execution also varied across models in a separate experiment where each received both a training plan and a repository note identifying the available trainer and data. The agents performed the training, and our evaluation system loaded and tested the resulting models. The 0.8B model produced no successful updates in 20 runs and the 2B model succeeded once in 16, while the 4B and 9B models succeeded in 15 of 20 and 14 of 20 runs, respectively. Performance remained strong at 27B: Qwen3.5-27B successfully produced the update in most runs, and Qwen3.8-27B also completed an effective model update in a separate proof-of-concept. The sparse 35B model performed substantially worse, succeeding only once in six runs, making it an exception to the otherwise substantial increase in execution success among the larger models tested. The results suggest that sufficient model capability is important for completing the workflow, while the 35B result cautions against interpreting parameter count alone as predictive of success.
Taken together, the results separate two parts of agentic self-modification that can otherwise look like a single capability. Whether an agent considers changing the model depends strongly on what repair options the environment exposes, while successfully producing the update requires enough capability to execute the training workflow.
Scope and limits
The experiments were designed to establish a mechanism under controlled conditions rather than estimate how often agentic self-modification occurs in deployed systems. Several features of the environment made model modification unusually accessible, including training data, fine-tuning utilities, checkpoint access, broad shell permissions, and, in some experiments, explicit evidence that training had previously helped. Other experiments showed that capable models could propose or complete training updates without the repository note reporting earlier fine-tuning success.
The model-size results should be interpreted as capability comparisons within the systems tested, not as a general scaling law or a stable parameter threshold. Sample sizes were limited at several model sizes, and parameter count was confounded with model generation, architecture, sparsity, instruction tuning, numerical precision, and tool-use ability.
The experiments on broader model changes were deliberately constructed to make the mechanisms observable. The synthetic values in the memorization experiment appeared directly as training targets, so the results should not be used to estimate how often arbitrary secrets would be recoverable from ordinary fine-tuning data. The refusal experiment used a benign policy involving fictional competitors, and the agent was explicitly told that the application was refusing too often; the result demonstrates that an agent can remove learned behavior through training, not how often an agent would remove a real safety policy.
The mechanism also depends on access rather than on the coding agent and modified model being the same system. An API-based agent with permission to use separate training and deployment infrastructure could perform the same sequence on another model, while immutable serving environments, restricted training access, and independently controlled deployment pipelines substantially change the conditions. Nothing in these experiments establishes malicious intent, self-preservation, or deception; the agents modified models because training appeared to help accomplish the assigned engineering task.
Conclusion
Agentic self-modification can arise during ordinary software maintenance when a coding agent has access to the model weights, training tools, and a deployment path to modify the model directly. The experiments suggest that the engineering environment strongly shapes whether agents consider model modification as a repair strategy, while model capability strongly affects whether they can carry the resulting training workflow through successfully.
Once an agent can construct training data, modify the weights, and put the resulting model into service, the consequences can extend beyond whether the immediate repair succeeds. Sensitive or otherwise unintended information can become embedded in the model, previously learned behavior can be removed, and those changes can affect every application or agent that later loads the modified checkpoint. Broader evaluations can help detect regressions or unexpected changes, but they can only test the behaviors they measure and can be costly to make comprehensive. The updated model itself also provides little account of how or why it changed, making training data, evaluation results, model lineage, and approval history important parts of the record.
The broader control question is how to govern agent-initiated changes that persist beyond the task that produced them, especially when those changes alter the behavior of systems used by future users or agents. For model self-modification, the harder problem is that the agent can influence the training data and training procedure that shape the resulting model, while the behavioral changes may extend beyond the properties anyone thought to test. Detecting a changed checkpoint and requiring approval before deployment remain useful controls, but they do not provide visibility into everything the training process changed. Systems that expose training and deployment infrastructure should therefore define when model modification is within scope, preserve the full provenance of the update, evaluate the resulting model independently, and require separate authorization before it enters service. As open-weights models become more capable and self-hosted agentic systems become more attractive, architectures that give agents this kind of access are likely to become increasingly relevant. Organizations considering such deployments should explicitly analyze how much control they retain over agent-initiated training, what behavioral properties they can realistically evaluate, and what may remain outside the coverage of those evaluations.