AI Integration
FHIR R4 vs R5 (and R6): Which Version Should You Build On?
FHIR R4 vs R5 compared for US EHR integration: maturity, breaking changes, US Core and vendor support, R6 status, and a build decision.
October 27, 2026Read more
Healthcare AI governance is how an organization decides which AI tools it uses, checks that they are safe, fair and effective, and keeps them under control after go-live. Compliance is the part of that work that answers to law and regulators: FDA rules for medical devices, HIPAA for patient data, nondiscrimination rules and certification requirements for health IT. This guide explains the main building blocks, the rules that shape them and which of our articles to read next.
Expectations are rising from accreditors as well as regulators. In September 2025, the Joint Commission and the Coalition for Health AI released guidance on the responsible use of AI in healthcare that covers governance, privacy and transparency, data security, quality monitoring, safety event reporting, risk and bias assessment, and staff education, according to the Joint Commission.
Start here
- AI Governance in Healthcare: Models, Roles and Controls: who decides, who reviews and which controls a health system needs.
- AI in Healthcare: Key Regulations and Compliance Requirements Explained: FDA, HIPAA, HTI-1, Section 1557, state laws and the EU AI Act in one place.
- AI Risk Management in Healthcare: Applying NIST AI RMF: a structured way to map, measure and manage AI risk.
- AI Lifecycle Management in Healthcare: From Intake to Retirement: governance from the first request to retirement of a model.
Find the right article for your goal
| Your goal | Read |
|---|---|
| Choose a governance model for your organization | Healthcare AI governance models |
| Compare governance tooling | AI governance platforms for healthcare teams |
| Manage patient safety risk from clinical AI | Clinical AI risk management Responsible AI in clinical practice |
| Make AI outputs understandable to clinicians | Explainable AI in clinical decision making |
| Turn ethics principles into checks | Ethical AI in healthcare |
| Plan FDA submissions for AI devices that will change over time | Predetermined change control plans |
| Use large language models with patient data | HIPAA-compliant AI |
| Build HIPAA compliance into software and cloud systems | HIPAA compliance for software development HIPAA compliance on AWS |
| Prepare for a HIPAA audit | HIPAA compliance audit |
| Secure AI systems and deployments | Securing AI systems in healthcare Building secure healthcare AI solutions |
| Build compliance into engineering and automate it | Compliance engineering in HealthTech Healthcare compliance automation |
| Understand AI rules outside healthcare and use AI for compliance work | AI regulatory compliance How AI helps with compliance work |
| Check whether your organization is ready for AI | AI readiness framework for healthcare organizations |
The building blocks of AI governance
| Building block | What it covers |
|---|---|
| Inventory | Every AI tool in use, including features inside vendor products, with an owner for each |
| Intake and risk tiering | A review before adoption, with deeper checks for tools that affect diagnosis, treatment or access to care |
| Validation | Testing on local data before go-live, including performance across patient groups |
| Monitoring | Tracking performance, drift, errors and safety events after deployment |
| Transparency | Clear information for clinicians and, where appropriate, patients about how a tool works and its limits |
| Security and privacy | Access control, logging, data minimization and contracts with vendors that handle patient data |
| Retirement | Criteria and a process for updating or switching off tools that no longer perform |
Rules that shape governance work
- FDA: software intended to diagnose or treat can be a medical device. FDA finalized guidance in December 2024 on predetermined change control plans, which let manufacturers describe planned AI model changes in their marketing submissions, according to FDA.
- Nondiscrimination: under 45 CFR 92.210, covered entities must make reasonable efforts to identify patient care decision support tools that use race, color, national origin, sex, age or disability as inputs and to mitigate the risk of discrimination, with those duties in effect since May 1, 2025, according to the eCFR and HHS.
- Health IT certification: the HTI-1 rule requires certified health IT to support source attributes that describe how decision support interventions, including predictive models, were built and perform, according to the ONC fact sheet.
- Patient data: HIPAA applies to AI tools that handle protected health information, including business associate agreements with vendors.
- Risk frameworks: the voluntary NIST AI Risk Management Framework gives a shared structure for governing, mapping, measuring and managing AI risk.
Planning a governance program
Start with an inventory, because most organizations already use more AI than they think, much of it inside vendor products. Set up a small cross-functional committee with clinical, IT, security, legal and quality members, agree on risk tiers, and require local validation for high-risk tools. Then build monitoring and incident reporting into daily operations, train staff on what AI can and cannot do, and review the program at least yearly as rules change.
If you are building AI products that must meet these requirements, see our healthcare engineering services and SaMD solutions.
Conclusion
Healthcare AI governance combines an inventory, risk-based review, validation, monitoring and transparency with the rules that apply: FDA, HIPAA, nondiscrimination requirements and health IT certification. Start with the four core articles above, use the goal table to go deeper, and treat governance as an ongoing operating process rather than a one-time approval.
If you want help setting up AI governance or building compliant AI products, book a consultation with Ekipa's HealthTech engineers.
Frequently asked questions
What is healthcare AI governance?
The policies, roles and processes an organization uses to decide which AI tools to adopt and to keep them safe, fair and effective over time.
Who should be on an AI governance committee?
Typically clinical leaders, IT, information security, privacy and legal, quality and patient safety, and data science, with a named chair.
Do all healthcare AI tools need FDA clearance?
No. Only software that meets the definition of a medical device needs FDA oversight. Many administrative tools do not, but they still need governance.
What does Section 1557 require for AI tools?
Covered entities must make reasonable efforts to identify decision support tools that use protected characteristics as inputs and to mitigate the risk of discrimination.
How often should AI tools be reviewed?
Continuously for key performance and safety signals, and formally at least once a year or whenever the tool, its data or its use changes.
