Serverless compute

With the serverless compute version of the Databricks platform architecture, the compute layer exists in your Azure Databricks account rather than your Azure subscription.

Databricks SQL Serverless

Databricks SQL Serverless supports serverless compute. Admins can create serverless SQL warehouses (formerly SQL endpoints) that enable instant compute and are managed by Azure Databricks. Serverless SQL warehouses use compute clusters in your Azure Databricks account. Use them with Databricks SQL queries just like you normally would with the original customer-hosted SQL warehouses, which are now called classic SQL warehouses.

Databricks changed the name from SQL endpoint to SQL warehouse because, in the industry, endpoint refers either to a remote computing device that communicates with a network that it’s connected to, or to an entry point to a cloud service. A data warehouse is a data management system that stores current and historical data from multiple sources in a business-friendly manner for easier insights and reporting. SQL warehouse accurately describes the full capabilities of this compute resource.

Before you can create serverless SQL warehouses, you must enable serverless Databricks SQL warehouses for your workspace. If serverless SQL warehouses are enabled for your workspace:

• New SQL warehouses are serverless by default when created from the UI or the API, but you can also create new pro or classic SQL warehouses.
• You can upgrade a pro or classic SQL warehouse to a serverless SQL warehouse or a classic SQL warehouse to a pro SQL warehouse. You can also downgrade from serverless to pro or classic.
• This feature only affects Databricks SQL. It does not affect how Databricks Runtime clusters work with notebooks and jobs in the Data Science & Engineering or Databricks Machine Learning workspace environments. Databricks Runtime clusters always run in the classic data plane in your Azure account. See Compare serverless compute to other Azure Databricks architectures.

For regional support, see Azure Databricks regions.

Model Serving

Azure Databricks Model Serving deploys your MLflow machine learning (ML) models and exposes them as REST API endpoints that run in your Azure Databricks account. The resources run as Azure Databricks Azure resources in what is known as the serverless data plane.

In contrast, the legacy model serving architecture is a single-node cluster that runs in your Azure subscription within the classic data plane.

• Easy configuration and compute resource management: Azure Databricks automatically prepares a production-ready environment for your model and makes it easy to switch its compute configuration.
• High availability and scalability: Serverless model endpoints autoscale, which means that the number of server replicas automatically adjusts based on the volume of scoring requests.
• Dashboards: Use the built-in serverless model endpoint dashboard to monitor the health of your model endpoints using metrics such as queries-per-second (QPS), latency, and error rate.

For regional support, see Azure Databricks regions.

Serverless quotas

Serverless quotas are a safety measure for serverless compute. Serverless quotas restrict how many serverless compute resources a customer can have at any given time. The quota is enforced at the regional level for all workspaces in your account. Quotas are enforced only for serverless SQL warehouses. See Serverless quotas.

Compare serverless compute to other Azure Databricks architectures

Azure Databricks operates out of a control plane and a data plane:

• The control plane includes the backend services that Azure Databricks manages in its own Azure subscription. Databricks SQL queries, notebook commands, and many other workspace configurations are stored in the control plane and encrypted at rest.
• The data plane is where data is processed by clusters of compute resources.

There are important differences between the classic data plane (the original Azure Databricks platform architecture) and the serverless data plane:

• For a classic data plane, Azure Databricks compute resources run in your Azure subscription. Clusters perform distributed data analysis using queries (in Databricks SQL) or notebooks (in the Data Science & Engineering or Databricks Machine Learning environments):
  • New clusters are created within each workspace’s virtual network in the customer’s Azure subscription.
  • A classic data plane has natural isolation because it runs in each customer’s own Azure subscription.
• For a serverless data plane, Azure Databricks compute resources run in a compute layer within your Azure Databricks account:
  • The serverless data plane is used for serverless SQL warehouses and Model Serving. Enabling serverless compute does not change how Databricks Runtime clusters work in the Data Science & Engineering or Databricks Machine Learning environments.
  • To protect customer data within the serverless data plane, serverless compute runs within a network boundary for the workspace, with various layers of security to isolate different Azure Databricks customer workspaces and additional network controls between clusters of the same customer.

Azure Databricks creates a serverless data plane in the same Azure region as your workspace’s classic data plane.

The following diagram shows important differences between the serverless data plane and classic data plane for both serverless features.

For more information about secure cluster connectivity, which is mentioned in the diagram, see Secure cluster connectivity (No Public IP / NPIP).

The table below summarizes differences between serverless compute and the classic data plane architecture of Azure Databricks, focusing on product security. It is not a complete explanation of those security features or a detailed comparison. For more details about serverless compute security, or if you have questions about items in this table, contact your Azure Databricks representative.