As Databricks customers increasingly run workloads across Classic Compute and Serverless, optimizing cost and maintaining reliability requires understanding the platform holistically, rather than in silos.

The challenge is that the signals and optimization levers are distributed across data, code, job execution, infrastructure, and cost – with different levels of visibility and control across compute types. Without a unified runtime context, it's difficult to identify the underlying inefficiencies and generate impactful optimizations, prevent emerging issues in-motion, or troubleshoot them quickly when they do occur.

Today, we're announcing that definity for Databricks Serverless is now generally available, expanding our agentic optimization and full-stack observability across the Databricks platform.

One optimization and operational layer across Databricks

With definity, Databricks customers can now continuously understand, optimize, and operate workloads across both Classic Compute and Serverless – with context spanning data and pipeline behavior, execution, code, infrastructure, SLAs, and cost.

This includes:

• Dynamic auto-tuning for Classic Compute – continuously optimizing job, cluster, Spark, Databricks, and cloud configurations at scale.
• Deep code optimization – identify and fix inefficient code driving long runtimes and wasted cost, including data skew, inefficient reads, joins and filters, duplicative code, and other inefficiency patterns.
• Optimal compute routing – determine the right compute for each workload and automatically route jobs between Classic Compute and Serverless.
• Full-stack observability – understand data quality, pipeline operation, execution, code, lineage, SLAs, performance, and cost in one contextual view.
• Agentic reliability – prevent emerging data and pipeline issues in-motion, and rapidly resolve incidents with correlated context across data, pipelines, execution, and deep lineage.

These agentic actions are built with reliability in mind. Tuning and code changes can be automatically validated in Staging before reaching Production, while auto-tuning and optimal routing use continuous feedback loops to learn, fine-tune, and revert changes when needed – protecting job and data reliability, SLAs, and cost.

The result is a single optimization and operational layer across Databricks – helping teams continuously identify waste, optimize workloads, prevent and resolve incidents, and maintain reliability regardless of where workloads run.

Deepening our Databricks partnership

This release deepens definity's technology partnership with Databricks, extending the agentic optimization and full-stack observability capabilities we already provide to customers who leverage Serverless as part of their Databricks environment.

For data and platform teams, the goal is simple: one place to understand, optimize, and reliably operate workloads across the Databricks platform.

‍