Duke Health, a premier academic medical center affiliated with Duke University, recently shared a number of key lessons related to AI governance in the wake of a disclosure that it’s working with Avanade to develop, deploy, and share insights from its use of a SaaS-based Avanade governance platform.
Those four lessons are:
- lay an early foundation for governance
- three required components of governance
- the need for trusted tech platforms to underpin AI and governance
- take an industry-wide, global approach but govern locally
Below are more details on these lessons, drawing on an in-depth interview with Michael Pencina, Chief Data Scientist for Duke Health, and Dave Galich, Advisory Practice Lead and Chief Product Officer for the Smart AI Governance Engine (SAIGE) at Avanade.
Start Early, Enable Innovation
Back in the 2017-2018 timeframe, there was a surge in AI usage, largely around predictive algorithms, within the Duke Health System. The healthcare provider was striving to determine which algorithms were most useful and appropriate for its business use cases, how those algorithms support patient care and privacy, and more. “Our need for governance arose from the state of relative chaos,” Pencina said, “which is always what happens when new technology comes around.”
The health system emphasized a “principled” approach that included forming a decision support committee to develop an approach based on principles the Food and Drug Administration uses to evaluate medical devices. With the arrival of ChatGPT in late 2022, its existing governance initiative positioned Duke Health well for AI governance; though its principles didn’t anticipate GenAI, they could be quickly adapted.
The underlying mindset: “You want to preserve patient safety and prevent AI from mistakes and untoward experiences, but it’s also an enabler,” Pencina said. “Because of having that governance in place, we are able to go out there and experiment, work with companies, and do partnerships. We and our patients are not worried about what it’s going to do because we have the right governance mechanism in place.”
Three Core Governance Components
The aforementioned governance objectives lay the groundwork for a discussion of the elements that make up Duke Health’s governance strategy.
Pencina notes it’s relatively obvious yet still critical to recognize the importance of building an inventory of AI assets as a baseline requirement, and core component of governance, to effectively manage those assets.
The second requirement after an AI inventory is risk assessment and monitoring, which includes developing a stratification of risk. For example, a predictive algorithm can affect patients’ lives because a small error can have massive health consequences. Risk assessment and monitoring could be used to specify, for example, that a communication is not seen by a patient unless a doctor reviews it first. Risk assessment and monitoring is also critical because a technology or application implemented today may operate differently a year from now.
The third component is the governance process itself, which answers questions such as who is the clinical owner of an algorithm or who uses a given algorithm. It defines who has the expertise to determine if a given algorithm’s quality is adequate and thereby creates accountability.
Ask Cloud Wars AI Agent about this analysis
Governance and Trusted Tech Platforms
Pencina noted that Duke Health’s tech infrastructure includes a number of core Microsoft tech platforms including Azure Cloud Services, Microsoft Fabric for organization and management of data, Azure AI Studio, plus Microsoft’s multi-modal AI algorithms that tap data from different sources including images, text, and more.
In addition, Duke Health is using Abridge’s ambient voice recognition technology, which lets doctors converse with their patients more directly than if they had to type clinical notes, while the voice recognition captures the details and drafts clinical notes.
Pencina and Galich emphasized the multi-vendor approach underpinning SAIGE, noting an open playing field is needed to ensure the platform enables better healthcare for health systems and patients.
Think Globally, Govern Locally
Duke Health’s intention from the outset of building a governance strategy was to develop useful practices and technology that are applicable beyond Duke Health, to positively impact the healthcare industry as well as partners within its value chain. “From our experience, the SAIGE tool is critical in making the process streamlined and, in many ways, feasible and sustainable,” Pencina said.
That being said, he emphasized that every health system needs to take a tool like SAIGE and add its own customizations to ensure it works for their business as well as their patients and employees.
Rollout and Future Plans
Duke Health has just completed a late-stage pilot of SAIGE and Pencina says he’s “very impressed with the value it’s offering.” By year end, he expects the tech to be fully deployed and helping Duke Health manage agentic AI technology, which is now proliferating among customers.
Going forward, Avanade plans enhancements including features to help customers better collaborate and obtain information from their AI vendors and perform localized testing of AI models, Galich said.
As these two pioneers move forward, they’re establishing a solid example of a leading-edge customer and partner working together to define, develop, and deliver AI software that drives business results for the user itself as well as its customers or, in this case, its patients.
AI Agent & Copilot Summit is an AI-first event to define opportunities, impact, and outcomes with Microsoft Copilot and agents. Building on its 2025 success, the 2026 event takes place March 17-19 in San Diego. Get more details.
