At Datamatics, when we deployed close to 1,000 enterprise AI licenses with the goal of becoming an AI-first enterprise, we quickly saw challenges in extracting value from AI. While we could get insights into who was using AI and to what extent, we struggled to understand what level of value AI was delivering within the enterprise.
With increasing AI investments in enterprises, this challenge has shifted from getting people to use AI to understanding how AI is being used to drive productivity and identify new automation opportunities. By understanding how employees interact with AI within the organization, we can identify future automation and productivity opportunities.
Key Learning
To get value from AI in enterprises, AI observability needs to be much more than a reporting tool. It needs to be a driver for transformation.
Metrics such as licenses provisioned, weekly active users, and monthly active users provide insight into engagement with AI. However, they do not provide much insight into how deeply or effectively AI is being used within the business.
Even if two businesses have the same AI adoption rate, AI may still be used in completely different ways. For example, in one business, employees may use AI infrequently to perform transactional tasks, whereas in another, AI may be used regularly in business processes, decision-making, and increasingly, automation.
To achieve meaningful AI adoption, we need to know what employees are achieving with AI, to what extent they are using it in their workflows, and what further opportunities exist.
At Datamatics, we saw that only around 15–18% of employees were actively using the enterprise AI platform.
Using Datamatics Oversight, our AI observability tool, we were able to understand AI adoption across individual employees, teams, functions, and AI tools. This allowed us to identify high adopters, build internal champions, target training for teams with low adoption, and optimize license allocation.
Active AI adoption rose to over 65% within the quarter. We also saw a shift in usage patterns, from transactional activities to employees creating agents to automate repetitive processes.
Crucially, we shifted from just understanding adoption rates to being able to manage them. By understanding AI adoption and usage better, we are able to target interventions within the business more effectively, accelerate learning, and understand how AI is being integrated into daily operations. However, adoption is only one part of the picture. The next step is understanding whether that usage justifies the investment and contributes to business value.
Extracting value from enterprise AI investments can be challenging. In particular, in knowledge-intensive tasks, improvements in quality, decision-making, and business outputs may be more important than improvements in efficiency.
Ideally, businesses should understand the additional value being created from AI-assisted processes compared to a reasonable baseline. However, this can be challenging to measure consistently across a variety of processes.
A practical measure to understand AI consumption economics is to look at token usage. By understanding how many input and output tokens are being consumed, businesses can calculate the estimated cost of equivalent inference through metered APIs.
Comparing this with enterprise subscription costs can provide an indication of AI consumption relative to investment.
At Datamatics, the estimated cost of AI inference, based on equivalent API calls, was approximately double the cost of enterprise subscriptions. This is not a direct measure of business value. Just because inference is taking place does not mean productivity is improving, and API costs cannot be directly compared with enterprise subscription costs. However, this still provides useful insight into which licenses may not be fully utilized and how AI is being used across the business, supporting more informed investment decisions.
Combining this data with an understanding of the work employees are doing means we can gain a better understanding of the value from AI investments.
We are not trying to maximize AI consumption, but the value derived from AI-enabled work. Beyond measuring consumption and investment, AI usage can reveal another important source of value: opportunities to automate work that employees are already solving with AI.
Perhaps the biggest benefit we can get from AI observability is identifying automation opportunities.
Across businesses, employees are solving problems in their daily work using AI. By analyzing how employees are using AI within their workflows, we can understand what repetitive work is being performed, identify process inefficiencies and new use cases, and uncover potential automation opportunities.
Traditionally, automation opportunities have been identified from the top down. Once opportunities have been identified, they are evaluated, developed, and implemented. Generative AI gives us the opportunity to combine this top-down approach with bottom-up identification of opportunities.
By understanding AI interactions in the context of business intent, repeatable processes, and agents, we can understand where employees are getting value from AI and identify opportunities to standardize and expand successful AI applications.
For example, in our legal team, we had a non-technical member who built an AI agent to assist with contract reviews, including NDAs and MSAs. With around 600–1,000 NDAs being processed per year, this is a significant opportunity to improve efficiency in this knowledge-intensive, repeatable process.
Similarly, employees have built agents to query management information systems within the business to aid in account planning and business analysis.
Importantly, this demonstrates that more business users are becoming authors of automation within the business.
Once we see similar innovation across multiple employees, we can identify common problems being solved and the most effective approaches. From here, we can build standardized, reusable agents to solve these problems. If done effectively and cautiously, this can avoid duplication of effort, improve consistency, strengthen governance, and more rapidly roll out known solutions across multiple business functions. We must ensure that, at the prompt level, we respect data privacy, have the correct data access controls, and protect sensitive information. We want to understand workflows within the business and identify opportunities; however, we do not want to monitor employees unnecessarily.
By leveraging AI observability within businesses, we can gain intelligent operational insights to help identify future automation opportunities.
Turning these insights into lasting value requires more than identifying individual opportunities. It calls for continuous management of AI adoption, investment, and process improvements.
To get the most value from AI in an enterprise, it is important to understand:
This can inform investment decisions, employee capabilities, AI initiative priorities, and the scaling of successful implementations.
Those businesses getting the most out of AI may not be those deploying the most licenses, prompts, or agents. Rather, they will be businesses that have effectively learned from how AI is being used and translated this into improved processes, capabilities, and business outcomes.
AI observability gives us the ability to move from periodic reviews of AI licenses and adoption metrics in an enterprise to ongoing discovery of value in business processes. This means we can validate and scale value from AI.
As a result, AI moves from being widely deployed to being continuously refined as an enterprise capability.
Discover how Datamatics Oversight helps enterprises gain visibility into AI adoption and usage, identify opportunities for automation, and make more informed AI investment decisions.
Explore Datamatics Oversight or talk to the Datamatics team for more info.
Key Takeaways