There’s an emerging picture that shows where early customers are gaining confidence in AI agents and where they need to build a greater level of comfort to trust agents to handle complex tasks.
Areas of high confidence include boilerplate code generation as well as data quality monitoring and anomaly detection. Confidence is significantly lower for more sophisticated tasks, including several related to security and governance: network intrusion detection and memory leak detection.
Those are key findings from a new analysis by Microsoft and MIT Technology Review, in which 300 customers rated their confidence on specific tasks in the areas of data, AI, and cloud use cases. Overall, it’s a balanced picture with strong confidence in repeatable, measurable tasks, and lots of room to build more confidence for complex tasks.
The vendor community is moving quickly to add the tech underpinnings to support more complex tasks and workflows; among those deliverables are Microsoft Copilot Cowork and Anthropic’s Claude Cowork, as well as Microsoft’s “autopilot” initiative for autonomous agent actions.
And of course, customer confidence levels, in addition to factors such as financial payback, are key to achieving high levels of adoption and combating what is increasingly being described as backlash against AI technology.
High Confidence
Respondents to the Microsoft-MIT survey were asked to rate their confidence in 101 tasks, and they did so on a 1-100 scale, with 100 being the highest. For this analysis, I’m going to classify tasks or use cases with 80+ scores as having high confidence, and I’ll label those with scores of 50 or below as low confidence.
Some of the top tasks in terms of high confidence are:
- Automated report generation and distribution – a data workflow – 84% confidence
- Boilerplate code generation for new features (as noted above) – AI workflow – 82.5%
- Data quality monitoring and anomaly detection – data workflow – 82%
- Certificate expiration monitoring and renewal – cloud workflow – 82%
- Automated data profiling and statistical analysis – data workflow – 81%
In the AI and cloud workflow categories, the only tasks with confidence levels above 80% are those mentioned above, vs. four data workflow tasks with confidence levels above 80%. In both AI and cloud categories, however, there are numerous tasks with confidence above 70%. In the case of AI workflows, those tasks cluster around coding functions — which are widely touted in terms of positive AI impact. Those coding tasks include release note generation from commit history (79.5%) and automated code review and style enforcement (73.5%).
The report addresses the human element in AI workflows as well. Authors note that “human engineers and architects are likely to remain heavily involved in such workflows for at least the next couple of years, even while agents execute individual tasks within them,” but they noted that experts “foresee agents becoming increasingly capable of managing entire workflows” in the AI domain.
Low Confidence
At the same time the above-referenced tasks are being assigned high confidence by customers, there’s a set of functions where confidence is ranked well below 50%. Here are five noteworthy examples at or near the bottom:
- Service mesh configuration and troubleshooting – cloud workflow – 37.5%
- Disaster recovery testing and validation – cloud workflow – 43%
- Feature engineering pipeline automation – data workflow – 44.5%
- Legacy system data extraction and modernization – data workflow – 46%
- Database schema migration scripting – AI workflow 46.5%
I’ll zero in on those last two items because the opportunity to apply AI to cumbersome and labor-intensive tasks of database and system migrations — especially from legacy platforms to the cloud or other modern platforms — is so rich, yet confidence levels today are among the lowest on this scale. That speaks to a massive opportunity for the vendor and partner communities to step in with agents, orchestrators, and services to bring AI up to users’ expected functionality levels in order to automate more of these critical functions.
The lower scores are also impacted by the need for providing business context to AI models. “The more complex the task, the more reasoning capability an agent requires and the greater its need for business context,” the report says. “Such context-generation capabilities for agents are still at an early stage of development, especially in situations where enterprise data is difficult to wrangle and connect into the agent lifecycle at the speed and quality in which developers and executives need it.”
Looking Ahead
This report provides an insightful and detailed analysis of where AI can deliver the most impact today, where agents have a strong opportunity to play a larger role, and what’s needed to make that leap a reality. In a second installment analyzing the report’s findings, I’m going to look in more depth into issues around security and the need for human oversight of AI models.
Related AI Agent Insights:
- Plugins Elevate Copilot Closer to ‘Coworker’ Status
- Ford’s Engineering Pivot: More Course Correction Than AI Failure
- Google Cloud Spec Brings Order to the Chaos Surrounding AI Context
Ask Cloud Wars AI Agent about this analysis
