Databricks was among the myriad tech companies that announced GenAI capabilities during 2023. Already well-respected as a provider of AI-driven data analytics, the company promised features that would make it quicker and easier to undertake critical data management tasks.
Now, the company’s GenAI assistant, and accompanying GenAI features, is generally available and it’s free to all customers. In this analysis, I’ll explain what these capabilities are and the value they can bring to customers.
Contextually Aware AI Output
Databricks Assistant, the company’s GenAI chatbot, enables users to use natural language to query data or initiate actions. Databricks users can use the assistant to perform multiple tasks more efficiently: generate SQL or Python code, describe complex code snippets, create visualizations and dashboards, and fix data errors.
These processes are driven by DatabricksIQ, the company’s AI-powered data intelligence engine. DatabricksIQ unifies the entirety of a customer’s Databricks environment, providing contextually accurate data to the Databricks Assistant. Despite enabling many use cases, the ultimate aim of the Databricks Assistant is to make it simple to create AI and data products.
Since the preview phase, Databricks Assistant has become one of the company’s fastest-growing products, with upwards of 150,000 users leveraging the technology monthly. Now, the Assistant has been transitioned from the Databricks notebook and SQL Editor and it’s available everywhere in a user’s Databricks environment.
The feature has been well-received by customers. “The introduction of the Databricks Assistant has made it easier for our user base to improve their skill set,” said Nicholas Heier, Senior Manager, General Motors. “By seamlessly integrating into our development cycles, it has significantly enhanced productivity for our expert data scientists, engineers, and analysts.
“Users can comprehend, generate, optimize, and troubleshoot code faster than ever before. With the continuous growth of the Databricks ecosystem and the ongoing advancements of the Databricks Assistant, next-level tools are more accessible.”
AI-Generated Comments
Another AI feature is now generally available: AI-generated comments in the Databricks Unity Catalog. Using the power of GenAI, the tool generates table descriptions and column comments, and since its launch, 80% of table metadata updates on Databricks have become AI-assisted.
Interestingly, Databricks links the capabilities and success of its Assistant with AI-generated comments, telling users that one of the best ways to improve the answers from the Assistant is to use the new comments feature across the Unity catalog. Ultimately, the widespread presence of descriptive metadata improves the accuracy of responses from the Databricks Assistant.
Closing Thoughts
GenAI is bringing an unprecedented level of democratization and accelerated outcomes to technology processes. Now more than ever, regular business users have access to technologies that enable them to deliver immense value, while seasoned professionals can do more and faster.
However, it’s often easy to forget that the fuel driving these technologies is data, and it could be argued that the innovations are ultimately driven by data, rather than AI. To that end, GenAI products for the data management sector are incredibly important. With the general availability of its GenAI feature, Databricks is enabling many more users to access the raw material that drives the development of AI products and supercharging the efficiency of data professionals.
