The State of AI Governance in Hospitals: A Closer Look
As hospitals increasingly integrate artificial intelligence (AI) into their operations, a recent report by Black Book Research reveals that only a minority are prepared for a comprehensive AI audit. This finding underscores the urgent need for more robust governance as AI technologies continue to evolve rapidly within clinical and operational workflows.
Current Spending on AI Governance
The study highlights that while AI adoption is on the rise, the median budget allocation for AI governance and security among hospitals for 2026 stands at a mere 4.2%. Alarmingly, only 22% of healthcare facilities feel confident in their ability to provide a complete and verifiable explanation of AI applications to regulators or payers within a 30-day timeframe.
Confidence Levels in Audit Readiness
When it comes to willingness to undergo an audit, larger hospitals show relatively more confidence, with 34% expressing readiness. In contrast, only 21% of community systems, which operate three to nine facilities, and a mere 15% of small hospitals reported similar confidence levels. This indicates a concerning trend where, despite larger systems investing more in AI, audit readiness remains a widespread challenge across various types of healthcare institutions.
Barriers to Effective AI Governance
Key challenges persist in the realm of AI governance. Only 29% of hospitals have effectively implemented and enforced AI policies that encompass model inventory, traceability, and approvals. Furthermore, 41% report that the lack of transparency from AI vendors—specifically the limited availability of explainability artifacts—poses a significant barrier to conducting audits. Additionally, 37% of respondents noted incomplete tracking of data inputs and model versions.
Ambiguity in Ownership and Governance Funding
Ownership structures within hospitals also contribute to governance challenges, with 33% citing unclear lines between IT, quality/security, and compliance departments as a roadblock to effective governance. Looking ahead, 26% of hospitals plan to increase their governance/security funding by 2% or more in 2026, yet 18% anticipate no increase at all.
Recommendations for Strengthening AI Governance
To address these challenges, Black Book recommends allocating resources for a comprehensive AI governance framework, which should include model registries, lineage tracking, monitoring mechanisms, and version logs. It also suggests incorporating audit rights in supplier agreements, ensuring the timely delivery of explainability artifacts, and clarifying ownership responsibilities with regular reports to the board.
Exploring the Bigger Picture
A separate report from the Healthcare Financial Management Association highlights similar findings, noting that while 88% of health systems are utilizing AI, only 18% have established mature governance structures. Despite the urgent need for governance, awareness among health system leadership is growing, as evidenced by a significant increase in governance acknowledgment among CFOs from 40% in 2024 to nearly 70% in 2025.
For those interested in delving deeper, the HIMSS AI & Cybersecurity Virtual Forum will take place on November 18. Learn more and register.
As hospitals continue to embrace AI technologies, prioritizing governance will be crucial to ensure not only compliance but also the safe and efficient implementation of AI tools that enhance patient care and operational efficiency.