Generative AI in healthcare: proven use cases, key risks like hallucination and PHI exposure, HIPAA rules for LLM providers, and steps to build safely.
Generative AI can draft a clinical note, summarize a 200-page record or answer a patient's question in plain language. It can also invent a medication dose, expose protected health information or repeat bias from its training data. For healthcare product teams, the question is no longer whether generative AI is useful, but which use cases are safe to ship and what has to be in place first. This guide covers the main use cases, the risks, the rules that apply and a practical way to build safely.
Generative AI refers to models that produce new content, such as text, images or code, rather than only classifying or predicting. In healthcare, most current products use large language models (LLMs), and some use large multi-modal models that also handle images or audio. The World Health Organization published guidance on the ethics and governance of large multi-modal models in health in January 2024, with more than 40 recommendations for governments, developers and providers (WHO guidance).
The most successful generative AI use cases share one design choice: a human reviews the output before it affects care or leaves the organization.
| Use case | What the model does | Human checkpoint |
|---|---|---|
| Clinical documentation | Turns a visit conversation into a draft note (ambient scribing) | The clinician edits and signs the note |
| Patient communication | Drafts replies to portal messages, discharge instructions and education material | Staff review before sending |
| Record summarization | Condenses long histories, referrals and outside records into a short summary with links to the source | The clinician checks the summary against the cited source |
| Coding and revenue cycle | Suggests billing codes and drafts prior authorization or appeal letters | A coder or billing specialist approves |
| Internal knowledge search | Answers staff questions from approved policies and protocols, citing the source document | The answer links to the source so staff can confirm |
We go deeper on two of these in our articles on the ambient AI medical scribe and generative AI in patient communication.
LLMs can produce fluent text that is wrong. NIST's Generative AI Profile calls this "confabulation" and lists it among the risks that are unique to or made worse by generative AI (NIST AI 600-1). In healthcare, a single invented allergy or dose can cause harm, so outputs need grounding in source data and human review.
Sending PHI to a model provider without the right agreements, or logging prompts in an unprotected system, can be a HIPAA breach. Prompts, outputs, embeddings and logs that contain PHI all need the same protection as the record itself.
Models learn from data that may under-represent some groups or reflect historical inequities. Outputs need testing across patient subgroups, languages and reading levels before launch.
When drafts are usually right, reviewers stop reading closely. Interface design, such as highlighting uncertain content and linking every claim to its source, helps keep the human check meaningful.
Under HIPAA, a company that creates, receives, maintains or transmits PHI on your behalf is a business associate, and you need a business associate agreement with it (HHS sample BAA provisions). That includes LLM API providers. The major providers, including OpenAI, Anthropic, Microsoft Azure, AWS and Google Cloud, offer BAAs, but each one limits coverage to specific services, endpoints or configurations. Consumer chat apps are generally not covered.
Before any PHI reaches a model, confirm that the BAA is signed, that the exact service you call is in scope, and how long prompts and outputs are retained. Our guide to HIPAA-compliant AI compares the providers' current terms with links to each official page.
It depends on what the product does. Tools for documentation, administration and communication are generally not medical devices. A product that diagnoses, treats or drives clinical management can be, whatever technology it uses, and FDA's Clinical Decision Support software guidance sets out when decision support falls outside device rules. FDA has also asked for public feedback to shape its approach to generative AI-enabled medical devices (FDA announcement), so teams building in this area should follow that work closely.
Pick a narrow, low-risk first use case where a human already reviews the output, such as note drafting or message drafting.
Secure the data path: a signed BAA with the model provider, PHI kept inside your controlled environment, encryption, access controls and audit logs.
Ground the model in your data: use retrieval from approved sources so outputs cite the record or policy they came from, rather than relying on the model's memory.
Build an evaluation set: real, de-identified examples with expected outputs, scored for accuracy, omissions, harmful content and bias before every release.
Design the review step: show sources, flag low-confidence content and make editing easy so reviewers can check quickly.
Monitor in production: track edit rates, user feedback and error reports, and have a way to pause the feature.
Govern changes: model or prompt updates go through the same evaluation and approval as the first release. NIST's AI Risk Management Framework is a useful structure for this.
This is the approach we follow in our AI implementation support for healthcare teams.
Generative AI already saves clinicians and staff time on documentation, communication and administration. The products that work keep a human in the loop, ground outputs in real data, protect PHI under a BAA and test for errors and bias before and after launch. Start with one narrow use case, prove it is safe, and expand from there.
If you are planning a generative AI feature for a healthcare product, book a consultation with Ekipa's HealthTech engineers.
The most common uses are drafting clinical notes, drafting patient messages, summarizing records, suggesting billing codes and answering staff questions from approved policies. In each case a person reviews the output.
Not the consumer app. You need an enterprise or API service that is covered by a signed BAA, configured as the provider requires for PHI.
Ground the model in source data through retrieval, require citations, test against an evaluation set before each release and keep a human review step before output affects care.
Only when the product meets the definition of a medical device, for example when it diagnoses or drives treatment decisions. Documentation and administrative tools generally are not devices, but HIPAA still applies when they handle PHI.
A task where staff already review the output, such as drafting notes or patient messages, with a signed BAA in place and a clear way to measure accuracy.
How to choose an AI medical scribe: features to compare, EHR integration options, consent and BAA requirements, accuracy checks and questions to ask vendors.
Artificial general intelligence in healthcare: how AGI differs from the AI in use today, what current models can and cannot do, and where to focus now.
AI in medical imaging: detection, triage and measurement uses, how FDA clearance works and how to check it, PACS and DICOM integration, and local validation.
Connect with our team to explore how AI expertise can transform your business.