By Nima Roohi, Hamze Dokoohaki, and Naman Gupta

In recent days, some of the people building the world’s most powerful AI systems have again warned that the frontier is moving faster than our ability to control it. Anthropic CEO Dario Amodei has called for slowing capability development; other industry leaders have echoed the need for stronger evaluation and oversight. They are right to raise the alarm: models pose risks when they make routine mistakes like : hallucinating, misreading instructions, misusing poorly designed tools but the deeper danger emerges when a competent model makes none of those mistakes, yet its underlying incentives clash and are not fully aligned with human interests. The former is a reliability problem and the later is an alignment problem and both are real !

But there is a category error in the debate that follows. A probabilistic model is not the same thing as a probabilistic system.

At the other extreme, parts of the AI industry talk as if the destination is maximum autonomy: fewer humans, longer-running agents, more decisions delegated to models. We think both positions confuse intelligence with authority. The practical question is not whether a large language model can be made infallible, or perfectly aligned. It cannot. The question is whether a fallible model can operate inside a control system that makes consequential behavior bounded, testable, and auditable. Any high-stakes AI system should treat model failure or misalignment as a given, not an edge case. Sometimes the model will produce the wrong answer; sometimes it will pursue the wrong objective. The system's job is to make sure neither one gets far. And the hardest questions the safety camp raises are not reasons to stop building. They are the engineering specs.

We believe it can be done. In this post we show what responsible use of AI looks like in practice, using our AI form completion harness for Medicaid renewal as the use case.

The Cost Of Treating Paperwork As A Neutral System

During the post-pandemic Medicaid unwinding, at least 25.2 million people were disenrolled from Medicaid and CHIP. Sixty-nine percent of disenrollments were procedural: the renewal process was not completed or the state lacked enough information to finish the determination. That does not mean everyone who lost coverage was eligible. It means millions lost coverage without a completed finding that they were ineligible. Eligibility and the ability to prove eligibility are not the same thing.

The stakes rise in January 2027. Under new federal rules, many adults in the Medicaid expansion population will need to demonstrate at least 80 hours a month of qualifying work, education, job training or community service, unless an exclusion applies. States will also conduct more frequent eligibility checks. CMS says implementation requires major systems, policy and operational changes.

Whatever one thinks of the policy, a basic principle should be hard to dispute: a person who complies with the law, or qualifies for an exclusion, should not lose healthcare because the process for proving it defeated them.

That is where AI can matter - not as an eligibility officer, but as an execution layer that helps a person get through the process.

A Probabilistic Model Inside A Deterministic Control Plane

Consider a member on the phone completing a renewal. The agent asks about income. The member says, “I make about twenty-eight fifty a month, but it changes because my hours change.”

A language model is good at understanding what the person probably means. But “probably” is not good enough for a consequential field. In a bounded system, the model does not silently write $2,850 into the authoritative record and move on. It proposes a candidate answer. The workflow knows which field is being collected, whether the value is valid and what can happen next. The amount is read back and confirmed; a validator commits it; arithmetic and policy logic happen outside free-form model reasoning.

That read-back is the member's due process moment. A person working with a human caseworker has always had an implicit right to hear what was written down and correct it before it becomes the record. Bounded agency makes that right explicit and consistent, call after call, instead of leaving it to how careful or how rushed any individual caseworker happens to be. Legibility for the person on the phone is the control plane's purpose, extended to the one person who has the most at stake in getting it right.

Underneath the conversation sits a control plane. Canonical workflow state lives outside the model. A state machine restricts the process to permitted transitions. Required fields cannot be skipped because the model decides they are irrelevant. Tools have explicit permissions. Sensitive values can require confirmation. Calls resume from durable state. Uncertainty triggers clarification or escalation. Consequential transitions are logged. The model can make the conversation flexible; it does not get to rewrite the rules.

This is the core of our argument: “Getting smarter doesn’t fix the problems the harness solves.” Model self-evaluation is not an independent safety guarantee; the same generative process that produced an error should not be the only thing certifying that answer. Stronger models do not eliminate scaffolding either. The longer the job, the more places it can go wrong, and the more structure you need around it.

The strongest safety objections should become engineering requirements, not excuses for paralysis. Hallucination means consequential values need external validation. Long-task drift means state belongs outside the context window. Prompt injection and tool misuse mean agents need scoped permissions. Unreliable self-checking means verification must not depend solely on the model critiquing itself. NIST’s current agent-security work emphasizes authorization, auditing, traceability, and constraining access to tools and data; τ-bench has shown that strong tool-using agents still struggle with consistent rule-following across multi-step tasks. OpenAI’s Structured Outputs work makes a similar distinction: nondeterministic model behavior can be surrounded by deterministic engineering constraints.

Deterministic does not mean infallible; a deterministic bug is still a bug. It means the permitted actions and state transitions are explicit, versioned and testable. The model proposes; the control plane governs. 

Safety Should Assume Failure, Not Wish It Away

A serious high-stakes AI system should assume the model will eventually be wrong. Speech recognition will mishear something. A household in the Medicaid Recertification process will present an edge case. A policy will change. A call will drop. A new model version will regress. The engineering problem is therefore not eliminating uncertainty from the model; it is making sure uncertainty cannot silently acquire authority. That philosophy is consistent with NIST's emphasis on constraints, monitoring, authorization and traceability for agent systems, and with the Joint Commission and CHAI's Responsible Use of AI in Healthcare framework, which adds governance, bias assessment and transparency to patients. RUAIH would classify a renewal agent as administrative and monitor it accordingly. We think that understates the stakes. The failure mode is a person losing coverage, and the system should be held to the clinical bar.

From that premise, several product principles follow. Autonomy should throttle with confidence. Automate outreach, explanation, reminders, document collection and structured data capture aggressively, but narrow autonomy when evidence conflicts, policy exceptions appear or an adverse outcome is possible. Those cases should move to the authorized human or state process.

Escalation only works as a safety mechanism when it's also designed as a product experience. A raw transcript and a flag give the case worker a chore. What should cross that boundary is a structured packet: what the system confirmed, what it couldn't resolve and why, which policy exception might apply, what it already tried. Built that way, escalation becomes the system's most important surface, the moment a fallible model hands a person exactly what they need to act fast. Skip that design work and case workers will route around the system instead of trusting it.

We also believe in a simple engineering principle: rent the intelligence; own the process. Foundation models will improve and change. The durable safety layer is the workflow-state, rules, validations, permissions, integrations, test cases and escalation logic. A better model should earn more responsibility through regression, safety and equity testing, not inherit it automatically because its benchmark score rose. The model layer can be replaced, while domain logic, state and validation remain in the harness.

That principle is also a claim about where durable value sits in this category. A vendor whose advantage is "we have the best model" has an advantage with an expiration date measured in months. The harness does not go obsolete when a better model ships. It becomes more valuable, because a better model can operate inside it with less friction. That is where the real moat in this category lives, and it is a different bet than the one most AI vendors are making.

And the success metric should be “correct resolution”, not “automation rate”. An agent that safely resolves 85 percent of cases and escalates the rest is better than one that claims 97 percent autonomous completion by guessing at ambiguity. In healthcare, “no human touched it” is not a quality metric.

Most importantly, compare AI with the real alternative. The status quo is not a flawless human process. In Arkansas’s earlier Medicaid work-requirement experiment, more than 95 percent of the target population appeared to meet the requirement or qualify for an exemption, yet coverage fell and researchers found no significant employment gain; confusion, lack of awareness and reporting barriers were common.

The ethical question therefore runs in both directions: what harm could AI cause if it fails, and what harm continues when an already failing administrative process is left unchanged?

Why We Believe This Is Practical

We are not neutral observers. At Blooming Health, we build technology for these care-navigation and social-care workflows. Our platform helps support more than 7 million people through more than 2,500 community organizations across the nation, including food banks, housing organizations, hospitals, health plans and government partners.

That footprint has taught us that the most valuable use of AI in social care is usually not replacing the person with judgment. It is giving that person more reach: another call tomorrow, another language, another reminder, a persistent workflow that remembers where the member left off, and an escalation when the situation no longer fits the safe path. 

This also answers a different criticism: that AI is ultimately a labor-reduction exercise. If the goal is headcount elimination, the system will be optimized to avoid humans. If the goal is correct resolution, it will use humans where judgment is valuable and automation where repetition is wasteful. Those are very different products.

Put concretely: in a headcount-elimination system, the case worker's job shrinks toward whatever the model can't yet do, smaller and less interesting with every model release. In a correct-resolution system, the job shifts from processing volume to handling exceptions and relationships, the cases where judgment, empathy or local knowledge actually change the outcome. That's higher-leverage work, and it's the difference between AI that hollows out a workforce and AI that makes the remaining work worth doing.

That distinction matters in public services. AI can automate assistance more aggressively than it automates adverse decisions. It can call again, translate, explain, gather information, flag a missing document, schedule follow-up and prepare a structured package. It should face a much higher bar before resolving ambiguity, interpreting an exceptional policy situation or contributing to an adverse coverage outcome. This is a bounded agency not as a limitation, but as a product requirement.

A third path for consequential AI

AI safety critics are right that unconstrained agents can be dangerous. The maximalists are right that AI can make previously impossible forms of service accessible at scale. Neither observation requires us to choose between handing public systems to autonomous models and keeping AI away from consequential work. Current NIST work, in fact, is explicitly focused on making agent systems adoptable through security, authorization, evaluation and standards rather than treating autonomy itself as synonymous with trustworthiness.

The better path is bounded agency: flexible conversation inside a deterministic, auditable control plane; wide latitude to assist, narrow authority to decide; more automation when confidence is high, more human control when stakes or uncertainty rise.

The purpose of AI in Medicaid should not be to decide who deserves healthcare. It should be to make sure bureaucracy does not make that decision by default.

Safe AI is not a system that asks us to trust the model more. It is a system engineered so we have to trust the model less.

In high-stakes public services, that may be the difference between AI that merely demonstrates intelligence and AI that actually does some good.