EBS Analysis: High-level architecture – Azure Databricks

Understanding Azure Databricks Architecture: A Consulting Guide for Enterprise IT Leaders

Azure Databricks has become a cornerstone platform for enterprises pursuing advanced analytics, machine learning, and large-scale data engineering on Microsoft Azure. Yet despite its widespread adoption, the platform’s architectural model is frequently misunderstood at the levels that matter most: identity boundaries, data governance topology, compute isolation, and storage ownership. Misconceptions in these areas lead to flawed landing zone designs, security gaps, cost overruns, and migration failures that can derail an otherwise promising data platform initiative.

This article provides a detailed, consulting-grade examination of Azure Databricks’ high-level architecture. It is written for enterprise architects, platform owners, cloud infrastructure leads, and governance teams who need to make informed decisions about how Azure Databricks fits within their broader Azure estate. Rather than offering a surface-level product overview, we dig into the structural components that determine how workspaces are organized, where compute runs, how data is governed, and what operational responsibilities remain with the customer versus the managed service.

The Account-Level Construct: The Top of the Organizational Hierarchy

The Azure Databricks account is the top-level construct through which an organization manages the platform across its entire footprint. This is not merely a billing container; it is the administrative boundary for several critical functions that span every workspace and region the organization operates.

At the account level, administrators manage four core domains:

Identity and access. The account is where users, groups, service principals, and user provisioning are configured. Centralizing identity at the account level allows organizations to enforce consistent access policies across all workspaces rather than managing users piecemeal within individual workspace boundaries. This is particularly important for enterprises that need to align Databricks access with Azure Active Directory (now Microsoft Entra ID) groups and conditional access policies.

Workspace management. Workspaces can be created, updated, and deleted across multiple Azure regions from the account level. This multi-region capability is essential for global organizations that need data residency compliance, disaster recovery, or proximity to data sources in different geographies.

Unity Catalog metastore management. Metastores—the central governance system for data assets—are created and attached to workspaces at the account level. This is where the platform’s data governance architecture begins to take shape, and it is one of the most consequential design decisions in any Azure Databricks deployment.

Usage management. Billing, compliance reporting, and policy enforcement are administered at the account level, giving finance and governance teams a single pane of glass for oversight.

An account can contain multiple workspaces and multiple Unity Catalog metastores. The relationship between these components is not one-to-one; understanding the many-to-many possibilities and their implications is central to designing a scalable platform.

Workspaces: The Collaboration and Compute Environment

Workspaces are the collaboration environments where users actually run compute workloads. A workspace is where data engineers build ingestion pipelines, data scientists conduct interactive exploration, operations teams schedule production jobs, and machine learning practitioners train and deploy models. In practical terms, the workspace is the boundary that most end users interact with daily.

However, it is critical to understand that a workspace is not a data silo. When Unity Catalog is properly configured, a workspace is a lens through which users access governed data assets that may be shared across many workspaces. This distinction—between the workspace as a compute and collaboration boundary and the metastore as a data governance boundary—is one of the most important architectural concepts in modern Azure Databricks deployments.

Organizations typically provision multiple workspaces to achieve separation of concerns: development, staging, and production environments; team-specific workspaces for access isolation; or region-specific workspaces for data sovereignty. The account-level structure makes it possible to govern all of these consistently while allowing each workspace to operate with appropriate autonomy.

Unity Catalog Metastores: Central Governance for Data and Models

Unity Catalog metastores serve as the central governance system for data assets, including tables and machine learning models. The metastore enforces a three-level namespace that brings structure and discoverability to what would otherwise be a flat collection of data objects:

<catalog-name>.<schema-name>.<object-name>

This namespace hierarchy is more than a naming convention. It is the foundation for access control, lineage tracking, and data discovery across the entire Azure Databricks estate. By organizing data into catalogs and schemas, administrators can grant permissions at varying levels of granularity—catalog-wide, schema-level, or object-specific—depending on the sensitivity and organizational ownership of the data.

A single metastore can be linked to multiple Azure Databricks workspaces within the same Azure region. When this is done, every linked workspace sees the same data view, and data access controls are managed consistently across all of them. This is a powerful capability for enterprises that need to share governed data across teams without duplicating it or creating fragmented permission models.

The regional constraint on metastore-to-workspace linking is an important design consideration. Organizations operating in multiple regions will need multiple metastores, and cross-region data access patterns must be planned accordingly. This is not a limitation to work around; it is a structural reality that should inform the overall data platform architecture from the outset.

Control Plane and Compute Plane: Understanding the Separation

Azure Databricks operates on a two-plane architecture: a control plane and a compute plane. This separation is fundamental to understanding both the security model and the operational responsibilities of the platform.

The control plane contains the backend services that Azure Databricks manages within the Azure Databricks account. Critically, the control plane is located in the Azure Databricks account—not in the customer’s Azure subscription. The web application that users interact with, along with job scheduling, cluster management orchestration, and other backend services, resides in this managed control plane. Customers do not have direct infrastructure access to the control plane, and they are not responsible for patching, scaling, or securing it.

The compute plane is where data is actually processed. The location and ownership of the compute plane depends on the type of compute being used, and this is where the architecture diverges into two distinct models that enterprises must understand and choose between.

Serverless Compute Plane: Managed Compute in the Databricks Account

For serverless compute, the compute resources run in a serverless compute plane that exists within the Azure Databricks account, not the customer’s Azure subscription. Azure Databricks creates this serverless compute plane in the same Azure region as the workspace’s classic compute plane, with the region selected at workspace creation time.

This model offers significant operational advantages. There is no need for the customer to provision or manage virtual networks, subnets, or compute infrastructure for serverless workloads. Azure Databricks handles scaling, patching, and infrastructure maintenance entirely. For organizations that want to reduce their infrastructure management burden and accelerate time-to-value, serverless compute is an attractive option.

Security in the serverless compute plane is maintained through a network boundary established for each workspace, with multiple layers of security isolation. Azure Databricks isolates different customer workspaces from one another and applies additional network controls between clusters belonging to the same customer. This layered isolation model is designed to protect customer data even though the compute runs in Databricks-managed infrastructure rather than the customer’s own subscription.

However, the serverless model also introduces considerations that enterprises must evaluate carefully. Because compute runs outside the customer’s Azure subscription, some organizations may face compliance or regulatory requirements that mandate data processing within their own cloud tenancy. Network-level controls that organizations are accustomed to applying in their own virtual networks—such as custom route tables, network security groups, or Azure Private Link endpoints—may not be directly applicable in the same way. Security and compliance teams should review the serverless compute plane networking model thoroughly before adopting it for regulated workloads.

Classic Compute Plane: Compute in Your Azure Subscription

For classic Azure Databricks compute, the compute resources run in the customer’s own Azure subscription, in what is referred to as the classic compute plane. This means the virtual network, subnets, and compute resources associated with the workspace are deployed within infrastructure that the customer owns and can directly manage.

The classic compute plane provides a natural isolation boundary because it runs in each customer’s own Azure subscription. Organizations retain control over network configuration, can apply their existing Azure networking controls, and can integrate the Databricks virtual network with broader hub-and-spoke architectures, firewalls, private endpoints, and other security infrastructure they already operate.

It is worth noting that classic workspaces are referred to as Hybrid workspaces in the Azure portal. This naming difference can cause confusion, particularly for teams that are new to the platform or are reading documentation and portal interfaces that use different terminology for the same concept. Enterprise teams should establish internal terminology clarity early to avoid miscommunication during architecture reviews and operational handoffs.

For organizations with strict data residency, network isolation, or regulatory requirements, the classic compute plane often remains the preferred model. The trade-off is that the customer assumes greater responsibility for network design, subnet sizing, managed private endpoints, and related infrastructure management tasks.

Workspace Storage: A Critical and Often Misunderstood Component

Workspace storage is handled differently depending on the workspace type, and misunderstanding these differences is a common source of operational problems. Every Azure Databricks workspace depends on storage for two categories of data: workspace file system data and workspace system data. Both are separate from the customer’s own data objects, such as Unity Catalog tables and volumes.

Workspace file system data includes the assets that users create and manage through the Azure Databricks UI. These include Git-connected repository folders, Python files, YAML configuration files, and other small files that support notebook-based development and code management. These assets are part of the collaborative workspace environment and are distinct from governed data assets in Unity Catalog.

Workspace system data is generated internally by Azure Databricks features. This data is too large to store in memory or transient databases, or it needs to persist beyond the lifetime of a single compute resource. Examples include SQL query results and cached query results, as well as SQL query plans used for observability. This system data is essential for platform functionality but is not directly user-managed.

Serverless Workspace Storage

Serverless workspaces use what is called default storage—a fully managed storage location for internal workspace system data and Unity Catalog data assets. This default storage is managed by Azure Databricks and removes the need for the customer to provision and maintain a storage account for workspace-level operations. Serverless workspaces also support connecting to the customer’s own cloud storage locations for catalogs, tables, and other data assets, giving organizations flexibility in where their governed data physically resides.

Classic Workspace Storage

Classic workspaces have an associated storage account deployed in the customer’s Azure subscription, known as the workspace storage account. This storage account contains several categories of data that enterprises must understand:

Workspace system data: Internal data generated by Azure Databricks features, as described above.

Unity Catalog workspace catalog: If the workspace was enabled for Unity Catalog automatically, the workspace storage account contains the default workspace catalog. All users in the workspace can create assets in the default schema within this catalog. This is a starting point for Unity Catalog adoption, but enterprises should plan to evolve beyond the default catalog toward a structured, multi-catalog governance model.

DBFS (legacy): The Databricks File System, accessible under the dbfs:/ namespace, includes DBFS root and DBFS mounts. Both are legacy patterns. DBFS root is a user-accessible file system, while workspace system data is used internally by Azure Databricks features. Although both may reside in the same cloud storage account in classic workspaces, they serve fundamentally different purposes. Storing and accessing data using DBFS root or DBFS mounts is a deprecated pattern and is not recommended by Databricks. Enterprises should treat any existing DBFS usage as technical debt and plan migration to Unity Catalog volumes or external storage patterns.

Critical Operational Warning: Do Not Delete Workspace Storage

One of the most severe operational risks in Azure Databricks is the deletion or modification of workspace storage. An Azure Databricks workspace depends on both its control plane databases and its workspace storage for correct operation. If workspace storage is deleted, the workspace cannot be recovered. This is not a reversible error, and there is no support path for restoration.

This risk is particularly acute in classic workspaces, where the storage account exists in the customer’s Azure subscription and is therefore subject to the same Azure Resource Manager operations, subscription-level policies, and human actions as any other customer-owned resource. A poorly scoped cleanup script, an overly aggressive lifecycle management policy, or a misunderstanding about which storage account serves which purpose can result in catastrophic, unrecoverable data loss.

Enterprises should implement protective measures including resource locks on workspace storage accounts, clear naming conventions that distinguish workspace storage from other storage accounts, and operational documentation that explicitly warns infrastructure teams about the non-recoverable nature of these resources. Firewall support can be enabled on the workspace storage account to limit access to authorized resources and networks only, adding an additional layer of protection.

Security and Governance Implications

The Azure Databricks architecture has several security and governance implications that enterprises must address proactively rather than reactively.

Identity boundary design. Because identity is managed at the account level, organizations should plan their group structure and service principal strategy before provisioning workspaces. A well-designed group hierarchy in Microsoft Entra ID, mirrored in the Databricks account, enables consistent role-based access control across all workspaces and simplifies onboarding, offboarding, and access reviews.

Unity Catalog as the governance backbone. The three-level namespace of Unity Catalog is not just a data organization tool; it is the enforcement point for row-level security, column-level masking, audit logging, and data lineage. Enterprises that delay Unity Catalog adoption or continue relying on legacy IAM-based access models are missing the platform’s primary governance mechanism and creating future migration debt.

Compute plane isolation. The choice between serverless and classic compute has direct security implications. Serverless compute provides managed isolation but limits the customer’s ability to apply custom network controls. Classic compute provides full network-level control but requires the customer to design and maintain that isolation properly. Neither is inherently more secure; the right choice depends on the organization’s risk profile, regulatory context, and operational maturity.

Storage firewall configuration. For classic workspaces, enabling firewall support on the workspace storage account restricts access to authorized resources and networks. This is a meaningful control that prevents unauthorized access from broader subscription or network contexts, and it should be part of the standard deployment configuration for any production workspace.

Operational Considerations and Limitations

Several operational considerations flow directly from the architecture described above, and enterprises should account for them during platform planning.

Regional planning. Azure Databricks operates in specific supported regions, and the region selected at workspace creation determines where both the classic compute plane and the serverless compute plane are created. Unity Catalog metastores are also region-bound. Organizations with multi-region requirements must plan their workspace and metastore topology carefully, considering data residency, latency to data sources, and cross-region access patterns.

Storage account management in classic workspaces. The workspace storage account in classic workspaces is customer-owned and therefore subject to Azure subscription limits, policies, and cost management frameworks. Organizations should ensure that this storage account is included in cost allocation, monitoring, and lifecycle management processes—but never in deletion or cleanup automation.

DBFS deprecation and migration. Existing workloads that rely on DBFS root or DBFS mounts should be identified and prioritized for migration to Unity Catalog volumes or direct cloud storage access patterns. Continuing to build new pipelines on DBFS is accumulating technical debt against a deprecated pattern.

Serverless workspace adoption. Serverless workspaces reduce infrastructure management overhead but shift certain responsibilities and controls to the managed service. Organizations should evaluate whether their compliance, security, and networking requirements are compatible with the serverless model before migrating existing classic workspaces or provisioning new serverless ones.

Common Pitfalls

Based on the architectural realities described above, several common pitfalls emerge that enterprises should actively avoid:

Confusing workspace boundaries with data boundaries. A workspace is a compute and collaboration environment, not a data governance boundary. Teams that treat each workspace as an independent data silo end up with fragmented governance, duplicated data, and inconsistent access controls. Unity Catalog metastores should be the governance boundary, with workspaces serving as compute environments that access shared, governed data.

Neglecting account-level identity design. Provisioning workspaces before designing the account-level group and service principal structure leads to ad hoc, per-workspace identity management that is difficult to audit and maintain. Identity design should precede workspace provisioning.

Deleting or modifying workspace storage. As emphasized above, this is an unrecoverable error. Resource locks, naming conventions, and operational documentation are essential preventative controls.

Continuing DBFS usage for new workloads. DBFS root and DBFS mounts are deprecated. New workloads should use Unity Catalog volumes or direct connections to cloud storage. Existing DBFS usage should be tracked as technical debt with a migration plan.

Misunderstanding the control plane location. The control plane is in the Azure Databricks account, not the customer’s subscription. This means certain operational visibility and control expectations that teams have for Azure resources do not apply to the control plane. Understanding this boundary prevents frustration and misdirected troubleshooting efforts.

Overlooking terminology differences. Classic workspaces are called Hybrid workspaces in the Azure portal. Teams that do not establish internal terminology consistency risk confusion during architecture discussions, operational handoffs, and incident response.

Why This Matters to Enterprise IT

For enterprise IT organizations, the Azure Databricks architecture is not merely a technical curiosity; it is the structural foundation upon which data platform investments succeed or fail. The decisions made at the account level—how identity is structured, how metastores are assigned, how workspaces are distributed across regions—have downstream consequences that persist for years and are expensive to unwind.

The separation between control plane and compute plane, and between serverless and classic compute models, determines who is responsible for what aspects of security, networking, and infrastructure management. Getting this wrong leads to either over-managed environments where teams waste effort on infrastructure the platform already handles, or under-managed environments where critical controls are missing because no one realized they were the customer’s responsibility.

Workspace storage is a silent risk. It is essential, unrecoverable if lost, and often poorly understood by the infrastructure teams who have permissions to delete it. The architecture makes it possible to build a robust, governed, scalable data platform—but only if the architectural boundaries are respected and the operational responsibilities are clearly assigned.

Finally, the deprecation of DBFS and the shift toward Unity Catalog as the governance backbone represent a directional change in the platform that enterprises cannot afford to ignore. Organizations that continue operating on legacy patterns are building on a foundation that will require increasingly disruptive remediation.

EBS Consulting Perspective

At Escape Business Solutions, we approach Azure Databricks architecture not as a product deployment exercise but as an enterprise platform design engagement. The most successful implementations we see share several characteristics: they begin with account-level identity and governance design before any workspace is provisioned; they treat Unity Catalog as a first-class architectural component rather than an afterthought; they make deliberate, documented decisions about serverless versus classic compute based on regulatory and operational requirements rather than defaulting to whichever model seems simpler; and they implement protective controls around workspace storage from day one.

Conversely, the most problematic engagements we encounter almost always trace back to architectural decisions that were made casually or deferred entirely. Workspaces provisioned without a metastore strategy. Identity managed per-workspace because no one planned the account-level group structure. Storage accounts without resource locks because the infrastructure team did not understand their criticality. DBFS usage that accumulated over years because no one established a standard against it.

Our recommendation is straightforward: treat Azure Databricks architecture as a strategic design exercise, not a tactical provisioning task. Invest in understanding the control plane and compute plane separation. Design your Unity Catalog namespace and metastore topology before building pipelines. Establish account-level identity governance as a prerequisite, not a follow-up. Document the distinction between workspace storage and your own data assets, and protect the former with the seriousness it demands. And wherever legacy DBFS patterns exist, create a migration roadmap with clear ownership and timelines.

Practical Next Steps

For organizations looking to validate or improve their Azure Databricks architecture, we recommend the following practical steps:

1. Conduct an architecture review. Assess your current account structure, workspace distribution, metastore assignments, and compute plane configurations against the architectural model described in this article. Identify gaps between your current state and a well-governed target state.

2. Audit identity and access design. Review whether identity is managed at the account level or fragmented across workspaces. Evaluate your group structure, service principal usage, and alignment with Microsoft Entra ID governance.

3. Evaluate Unity Catalog adoption. Determine whether your workspaces are using Unity Catalog effectively, whether your metastore topology aligns with your regional and organizational structure, and whether your three-level namespace reflects a deliberate governance design.

4. Assess compute plane strategy. Review whether your use of serverless versus classic compute is intentional and aligned with your security, compliance, and operational requirements. Document the rationale for each workspace’s compute model.

5. Implement storage protection controls. Verify that workspace storage accounts in classic workspaces have resource locks, firewall configurations, and clear operational documentation. Confirm that no automation or lifecycle policies can delete or modify these accounts.

6. Inventory DBFS usage. Identify all workloads relying on DBFS root or DBFS mounts, classify them by criticality, and establish a migration plan to Unity Catalog volumes or direct cloud storage access.

7. Engage with experienced architects. For complex or multi-region deployments, work with a consulting partner that understands both the Azure Databricks architectural model and the broader Azure landing zone patterns it must integrate with.

Conclusion

Azure Databricks is a powerful platform, but its power is fully realized only when its architecture is understood and respected. The account-level governance model, the control plane and compute plane separation, the Unity Catalog metastore topology, the serverless and classic compute options, and the critical nature of workspace storage are not implementation details to be sorted out later. They are architectural decisions that shape the security, scalability, and operational sustainability of the entire data platform.

If your organization is planning a new Azure Databricks deployment, migrating from another platform, or reassessing an existing deployment that has grown organically, Escape Business Solutions can help. Our consulting teams bring deep expertise in Azure Databricks architecture, Unity Catalog governance design, Azure landing zone integration, and enterprise platform operations. We work alongside your architects, security teams, and data leaders to design and implement a Databricks environment that is governed by design, operationally sound, and aligned with your enterprise strategy. Reach out to EBS to begin a conversation about how we can support your Azure Databricks initiative.

EBS Consulting Advice

If your organization is evaluating High-level architecture – Azure Databricks, do not treat the technology decision in isolation. Start with the business outcome, current architecture, security and identity controls, operational constraints, migration dependencies and governance requirements. A practical assessment should identify the current-state gaps, prioritize the risks and define an implementation roadmap with measurable outcomes.

EBS can help assess the environment, develop the architecture and modernization roadmap, and translate the technical options into an actionable business plan. Relevant EBS services: Microsoft Azure consulting Escape Cloud Microsoft Solution Assessments.

Have a technology challenge? Email info@escapebusinesssolutions.com to describe your situation. We welcome questions, consulting discussions and requests for a proposal.