Agentic AI Consulting for Healthcare: Scope and Selection

In the peer-reviewed ToolEmu study, researchers tested language-model agents on a benchmark of 36 high-stakes tools and 144 test cases and found that even the safest agent showed failures with potentially severe outcomes 23.9% of the time (Ruan et al., ToolEmu). In a clinical system that failure rate surfaces as a wrong-field EHR write-back, a record amended without authorization, or patient data leaving the trust boundary through a tool call nobody reviewed. This article covers what agentic AI consulting should deliver in a healthtech engagement, where a workflow engine beats an agent, and how to read a proposal well enough to know which one you're buying.

Demand for the work is measurable. McKinsey's 2025 state of AI survey, fielded from June 25 to July 29 with 1,993 respondents, found 23% of organizations scaling an agentic AI system in at least one business function, with another 39% experimenting with AI agents. No single function cleared 10% scaled deployment. That gap is where consulting value sits: selecting use cases, wiring agents into systems, defining oversight, and producing evaluation evidence.

What agentic AI consulting covers

Under 45 CFR § 164.312, any system that touches electronic protected health information must limit access to authorized users, record and examine activity, protect the data from improper alteration, and secure it in transit. That clause reaches the first design session, because an agent's risk comes from what it can do with a credential rather than from how well it writes prose.

An agent is software that plans a sequence of steps, calls tools, holds state between them, and works toward a goal inside limits you set. A single model call returns one output and stops, which is why the two belong in different categories when you scope the work (AI agent vs agentic AI).

Agents arrive in an environment that already runs clinical software. The ONC data brief on hospital AI use reports that 71% of U.S. hospitals used predictive AI integrated with the EHR in 2024, up from 66% in 2023, and the governance expectations attached to that software extend to anything else reading the same records.

Use case identification comes first and belongs to your product and operations leads together with the consultant, because fit depends on process knowledge a vendor doesn't have. Architecture, meaning tools, memory, planning, guardrails and approval steps, is engineering's work with the consultant. The remaining four components split across your own teams.

Component	Question it answers	Typical output	Usual owner
Use case identification	Does this task need an agent?	Ranked use case list with fit rationale	Customer product and operations leads with consultant
Architecture design	Which tools, memory, planning, guardrails and approval steps are needed?	Architecture document and autonomy ladder	Engineering with consultant
Integration	How does the agent read and write data in existing systems?	Interface specification, service accounts and data-flow diagrams	Engineering with customer IT
Evaluation and testing	Does the agent complete the workflow safely?	Test sets, metrics and release gates	Engineering with clinical reviewers
Security and governance	What can go wrong, and who approves changes?	Threat model, access policy and approval matrix	Customer security and compliance
Monitoring and cost control	Is it still safe and affordable after release?	Dashboards, alert thresholds and spend limits	Customer operations with engineering

The consultant designs, builds and tests the system. The customer owns clinical sign-off, compliance decisions and the policy on which actions an agent may take, and I would push back on any statement of work that moves those three to the vendor side.

Deciding whether a task needs an agent

The first question in discovery is whether the task needs multi-step planning and tool use across systems where the path changes case by case, or whether it's a predictable sequence. If the steps are known in advance, workflow automation or a single model call is the better fit, and deciding that early saves budget for the cases that need the machinery.

If you can draw the path on a whiteboard, build the flowchart. An agent earns its cost only when the next step depends on what the previous system returned.

Agents are more expensive than they look. They make more model calls, add latency that's harder to predict, enlarge the test surface, and generate more logs than anyone wants to store. Each additional tool is also an additional permission to grant, review and revoke.

For fixed, well-defined work such as classification, extraction or routing, rules plus a workflow engine or a single model call is usually cheaper, faster, easier to test and more reliable than an agent.

Task	Path predictability	Better approach
Triage an inbound fax into referral, records request or billing	Fixed categories	Rules, with a single model call for the ambiguous share
Reminder and rescheduling sequence	Known steps, known order	Workflow automation
Assemble a prior authorization packet where requirements vary by payer and procedure	Changes case by case	Agent with bounded tools and human approval
Summarize a chart into a fixed template	One input, one output	Single model call with retrieval and citations
Check eligibility and escalate when the payer response is partial	Depends on what the payer returns	Read-only agent

Three signals point toward an agent: the number of systems involved, the share of cases that need a different path, and whether a human reviewer can approve each consequential action in time to be useful.

Apply this checklist to a shortlist in a discovery workshop:

• Path variability: can you name the branches, or does the route emerge from data?

• Systems touched: more than two, with state carried between them?

• Approval feasibility: can a clinician review the action before it takes effect?

• Measurement: can your team state how success will be measured before any build starts?

Designing the agent architecture

A request enters the orchestrator, which decides what to do next, which tool to call and when to stop. Three things must be explicit at that layer: the step limit, the stop conditions, and the behavior when a tool returns an error or an empty result. Silent retries are the most common way a bounded plan turns into an expensive loop.

The tool layer is where designs fail in review. Every tool should be a typed API with a published schema, an allowlist entry, and a description narrow enough that the model can't repurpose it. A tool that writes free-text into a patient record is a different risk object from one that amends a coded allergy value with a required reason. Give each tool the smallest credential that does the job and log which identity used it.

Memory needs separation. Session state, retrieved records and any long-term store should be distinct, with retention defined per store and retrieval scoped to the authenticated patient. Every claim an agent produces should carry provenance back to the record it came from.

The autonomy ladder maps risk tiers to permissions, oversight and logging:

1. Recommendation only. Read access, draft output, no write scope, full logging of retrieval and draft versions.

2. Human-approved execution. A named approver sees the evidence, the action waits, and the log records who approved which version.

3. Bounded autonomous execution. Narrow scopes, reversible actions, rate limits and an alert on unusual patterns.

For clinical outputs, the design has a hard constraint. FDA's clinical decision support guidance requires that the clinician be able to independently review the basis for a recommendation rather than rely primarily on the software (FDA clinical decision software guidance). That means source references, timestamps, model or rules version, and a visible line between recommendation and clinician authorization. A generic disclaimer at the bottom doesn't meet it.

Tracing a prior authorization agent

Take a hypothetical prior authorization drafting agent. It reads Condition, MedicationRequest and Encounter resources through read-only scopes issued to a named service account, pulls payer criteria from a versioned policy store, and drafts a letter in which every clinical claim cites the resource and field it came from.

The loop stops at draft. Submission is a separate, human-approved action that records who approved it, when, and which version of the draft they saw. If a step returns an empty or conflicting result, the draft goes to a human with the conflict surfaced rather than resolved by the model. For a deeper component breakdown, the guide to agent architecture walks through the same layers.

Integrating agents with clinical systems and HIPAA controls

Integration starts with three decisions: which resources the agent reads and writes through FHIR R4 REST APIs, which scopes it holds under SMART on FHIR, and whether each call runs under a service account or a delegated user context. Keep scopes to the resources the workflow needs. An agent assembling a prior authorization has no business holding a write scope on Observation.

HL7 v2 feeds still carry admission, discharge and transfer events where the source system publishes nothing else. An agent reacting to an ADT trigger needs the message type, trigger event and segment parsing defined before anyone writes code.

Under 45 CFR § 164.312, the implementation must address access control, audit controls, integrity, person or entity authentication, and transmission security. HHS specifies that access control must limit ePHI access to authorized persons and that audit controls must record and examine activity in systems containing or using ePHI. For an agent, that means identifying the user, service account or delegated agent behind every read, write, tool call and downstream action.

Standard	What the agent design must show	Evidence to keep
Access control, 164.312(a)	Unique identity for each user, service account or delegated agent; emergency access procedure	Service-account inventory and identity-to-action mapping
Audit controls, 164.312(b)	Mechanisms that record and examine activity on ePHI	Protected audit logs with agent step, tool, identity and timestamp
Integrity, 164.312(c)(1)	Protection against improper alteration or destruction of ePHI	Write-back validation tests and integrity check results
Person or entity authentication, 164.312(d)	Verification that a caller is who it claims to be	Authentication configuration and test results
Transmission security, 164.312(e)(1)	Protection of ePHI in transit, including encryption where adopted	Data-flow diagrams, TLS configuration and payload minimization decisions

Some specifications are addressable. HHS lists unique user identification and emergency access procedures as required parts of access control, while automatic logoff and encryption/decryption are addressable (HHS technical safeguards guidance). Addressable isn't optional: the customer and compliance owner document how each is met or why an equivalent risk-based measure applies.

Any vendor or cloud component that handles PHI needs a business associate agreement and safeguards in its own right, which makes the agent's dependency list a BAA list. We design AI solutions that integrate with EHR, EMR, healthcare SaaS platforms, patient portals, CRM and internal applications, and that mapping is where most of the engineering hours land (healthcare AI and integration work).

Patient-facing assistants carry a different risk profile. They answer questions and route requests. The moment the same assistant books, cancels or amends a record, it needs the identity and audit design described here.

The practical test of an integration is whether, for any agent action, you can produce the record it read, the identity that acted, the approval that allowed it, and the log entry that proves it.

Security risks specific to agents

The OWASP Top 10 for LLM Applications gives a shared vocabulary for what goes wrong, and the NIST AI Risk Management Framework with its Generative AI Profile supplies the governance structure to attach it to. Together they turn a vague security conversation into named risks with owners.

Prompt injection sits first for a reason. Direct injection arrives in a user message. Indirect injection arrives inside a retrieved document, a page or a tool response, and that's the variant that bites agents because agents treat their inputs as data. A referral letter containing an instruction to forward records is a realistic attack on any agent that reads documents and holds an email tool.

Excessive agency is the other issue to fix first. An agent should hold the tools its task needs and no more, with scopes narrow enough that a compromised plan can't reach data the task never touches.

OWASP risk	Agent-specific example	Control
LLM01 Prompt injection	A retrieved referral letter instructs the agent to ignore policy and email the chart	Treat retrieved content as untrusted, scan tool outputs before they re-enter the prompt, never let a document grant a permission
LLM02 Sensitive information disclosure	One patient's PHI appears in a response or log attached to another session	Scope retrieval to the authenticated patient, redact logs, isolate memory per session, test cross-patient leakage
LLM05 Improper output handling	Model text flows directly into a FHIR query parameter or an API argument	Validate and encode outputs before they reach a tool, use typed schemas, reject out-of-range values
LLM06 Excessive agency	An agent holds write scope and an outbound messaging tool for a task that only needed read access	Least-privilege scopes per tool, allowlists, approval gate on record changes and external communication
LLM10 Unbounded consumption	A retry loop calls the model thousands of times on one failed task	Step limits, per-task token budgets and spend alerts

Data leaving the trust boundary deserves its own review. HIPAA doesn't categorically prohibit sending ePHI over email or another open network, but HHS states that the covered entity must assess the network, restrict access, protect integrity, and guard against unauthorized access under 45 CFR §§ 164.312(a), 164.312(c)(1) and 164.312(e)(1) (HHS guidance on ePHI over email). The organization must identify appropriate protections, select a solution and document the decision.

Governance is where these controls get tested. A security review should request a threat model per tool, a permissions matrix and an incident response plan for agent actions.

Evaluation, monitoring and cost control

Testing runs across the lifecycle, not at the end of a build. NIST's Generative AI Profile identifies documented test sets, metrics, evaluation tools, representative human-subject evaluations and ongoing lifecycle testing as core elements of trustworthy AI governance (NIST AI RMF Generative AI Profile). In practice that means a versioned evaluation harness recording, for every run, the model version, prompt and policy version, retrieved evidence, tool calls, authorization context, latency, cost and outcome.

The metrics that matter are workflow-specific: task completion, factuality against source records, citation coverage, unsafe-action rate, escalation rate, false-positive and false-negative rates, and performance across patient subgroups. Reliability has to be measured per action as well as per workflow, because a run can finish correctly while one intermediate tool call was unsafe.

Test cases should reflect production conditions: representative longitudinal records, malformed or conflicting FHIR resources, authorization failures, prompt-injection attempts and downtime in dependent systems.

Release gates and rollback triggers follow from those results. Define thresholds for safety regressions, drift, abnormal tool-call patterns and rising human overrides before launch, and treat an override spike as a failure signal rather than user preference. Cost control belongs in the same harness: per-task token budgets, step limits, rate limits and spend alerts, all visible on one dashboard.

Where the EU AI Act may apply, obligations for high-risk systems cover risk management, data governance, logging and human oversight in general terms. Check the current application dates, because some were amended.

Engagement phases from discovery to production

Four phases, each ending in a written exit criterion that engineering and the business owner both accept. Without an agreed threshold, a pilot can't demonstrate readiness, and the decision drifts to whoever is most enthusiastic.

Discovery ranks use cases, maps systems and data sources, defines data access rules and names a process owner. The proof of concept covers one workflow, usually read-only or draft-only, run against the test set. The pilot adds limited users and real data under a business associate agreement and access controls, with approval gates live and dashboards running. Production adds a rollback plan, an incident response runbook, a scheduled evaluation cadence and cost limits.

Phase	Main output	Exit criterion	Sign-off owner
Discovery	Ranked use cases, systems map and data access rules	One use case selected with a named process owner and measurable success criteria	Product and operations leads
Proof of concept	Working agent on one workflow in read-only or draft-only mode	Test set results meet agreed thresholds with no unsafe actions	Engineering lead
Pilot	Limited production use with approval gates and monitoring	Override rate, incident count and metrics stay within agreed limits over the pilot period	Clinical and compliance owners
Production	Scaled deployment with rollback, incident response and scheduled evaluation	Runbooks tested, rollback exercised, cost limits and alerts live	Operations with engineering

Our delivery work starts the same way, with goals, users and existing workflows understood before design begins, then runs alongside the client team from discovery through deployment and beyond (how the delivery framework is structured). The phase gates hold when the engineers doing the work sit inside the customer's team rather than at arm's length, which is the case for forward-deployed engineers.

Choosing a partner and spotting red flags

Put the same questions to every partner. Does it have hands-on EHR integration experience with FHIR R4 resources and SMART on FHIR scopes? Can it show a written architecture that defines identities, permissions and audit design for each tool? Does it name its test sets and metrics, and does it know which HIPAA technical safeguards apply and when a business associate agreement is required? Will it tell you when a rules engine or workflow tool handles the task better than an agent?

During discovery, ask for a threat model per tool, a sample evaluation report with test case identifiers removed, a permissions matrix, and a handover package containing code, the evaluation harness and runbooks. Strategy work should trace back to your systems and obligations rather than to a generic platform narrative, which is what we mean when we say a strategy report has to be grounded in the client's systems and compliance needs.

Walk away from these:

• autonomy promised with no approval gates

• model choice presented as the main decision

• no answer on how agent actions are logged

• one platform recommended regardless of fit

• accuracy figures quoted without a described test set

• no named compliance owner

• client results offered without documentation you can check

Where integration work outgrows the internal team, implementation support for AI in existing healthcare systems is the scope to negotiate, and it should name the systems, the interfaces and the evidence the engagement will produce.

Conclusion

The decision is whether variable, multi-step work warrants an agent, or whether a simpler workflow can handle it with fewer permissions and less operational overhead. Choose the smallest approach that meets the need, then fund the next step only when its purpose and ownership are clear.

For a suitable agent project, keep approval and audit requirements tied to the actions the system can take. Ekipa helps healthtech teams scope, build and integrate agentic systems with HIPAA controls, evaluation and human oversight built in. Explore our implementation support and HealthTech engineering services, see how we work in our AI agent architecture guide, or book a consultation with Ekipa's HealthTech engineers.

Frequently asked questions

Is agentic AI consulting the same as AI adoption consulting?

No. AI adoption consulting addresses organizational readiness, including training, change management, and prioritization. Agentic AI consulting focuses on designing, integrating, and testing systems that plan and act. The work overlaps when teams set governance policies. AI adoption consulting covers the broader engagement.

Does HIPAA apply to an agent that only summarizes records?

Usually, yes, when the agent works for a covered entity or business associate. Summarizing records involves ePHI, and read-only access does not remove authentication or logging duties. See the Integration section for the related controls.

Do agents that give clinical recommendations need FDA review?

It depends on the software's intended use and how clinicians use its recommendations. For clinical decision support, the FDA's guidance focuses on whether a healthcare professional can independently review the basis for a recommendation and exercise professional judgment. Assess the product's intended use and functionality against that guidance before launch.

How often should a deployed agent be re-evaluated?

Re-evaluate after changes to the model, prompts, tools, or workflow, and when drift or rising human override rates signal a problem. Keep a periodic baseline too, so gradual performance changes are visible.

Does the EU AI Act apply to a US healthtech agent?

Assess each deployment separately: whether the system is placed on the EU market, used in the EU, or produces outputs used there. Then determine the system's role and risk classification. High-risk deployments bring requirements for areas including risk management, data governance, logging, and human oversight. Check the current application dates on EUR-Lex before setting a launch schedule.