Behind every Databricks platform sits a small set of structural choices: how many workspaces to run, which compute model to deploy, who governs the data, how the catalog is organised, and how developers access real-world data. Each choice offers a few well-established options, and each option suits a distinct type of business.
This guide sets them out in plain terms, giving you a clear view of your own estate’s architecture, and helping you judge whether it was deliberately designed or merely inherited.
Why the shape matters
In my previous article, I argued that the most consequential Databricks decisions sit squarely on the client’s desk, and that too many organisations make them by accident. The natural question that follows is simple: if these choices are ours to make, what are they?
Five decisions define the shape of an estate. None requires a deep engineering background to evaluate, yet every one of them benefits from being made intentionally.
Understanding the layers
Before evaluating any design choice, it helps to understand which components of Databricks are set once and which are repeated. The platform operates across three distinct levels:
- Account level (set once for the organisation). Holds identity — users, groups, service principals — single sign-on, budget policies, central audit logs, system tables, and core network controls.
- Regional level (set once per region). Hosts the Unity Catalog metastore. This is where catalogs, schemas, tables, models, functions, permissions, lineage, and data sharing are governed.
- Workspace level (set per environment). Defines network rules, IP access, compute clusters, policies, jobs, notebooks, dashboards, secret scopes, and compliance security profiles such as HIPAA.
The significance of this layering is easy to overlook. Because data governance now sits above the workspace in Unity Catalog, workspaces are primarily boundaries for compute, networking, and compliance. Many historical reasons for creating new workspaces no longer apply, which fundamentally changes how the first decision should be approached.
Decision 1: how many workspaces?
Databricks outlines two core models:
- Many, smaller workspaces. Business units receive distinct development, staging, and production workspaces — four units equals twelve workspaces. This isolates misconfiguration risks, provides dedicated quotas, and satisfies regional compliance, but increases management overhead.
- Fewer, larger workspaces. The organisation runs a shared set of development, staging, and production workspaces across all business units. Asset discovery improves, administrative effort drops, networking costs fall, and central teams govern from the top down.
Databricks recommends the second approach as the default, pointing to a broader industry shift towards consolidated workspaces as Unity Catalog controls mature. Their guidance offers a sound principle: when in doubt, keep it simple.
Specialist workspaces
Most well-designed estates add two dedicated workspaces to their baseline model:
- An admin workspace, one per region. Serves as the day-zero environment to configure Unity Catalog, monitor system tables, and run security analysis. Platform users never log into this directly.
- A sandbox workspace. Gives teams an isolated venue to test preview features safely, away from enterprise data.
Where splitting makes sense
Where workspace segregation is genuinely needed, it is almost always driven by regulation:
- Healthcare and life sciences. Isolating patient-data workspaces requiring HIPAA compliance profiles from general clinical or HR workloads.
- Biotech, GxP. Separating regulated production environments, where features such as serverless compute or AI tools may need to be restricted, from non-GxP research teams who require full feature sets.
- Commercial isolation. Keeping joint ventures or export-controlled research strictly separated at the infrastructure level.
Decision 2: hybrid or serverless compute
Workspaces operate in one of two compute modes.
Hybrid workspaces. Compute and storage run inside your cloud account, on AWS, Azure or GCP. This requires explicit VPC or VNet, subnet and IAM setup, but gives full control over firewalls, Private Link and custom cluster configuration. Costs consist of Databricks units plus the underlying cloud infrastructure charges.
Serverless workspaces. Fully managed by Databricks. Environments spin up in minutes without cloud IAM permissions, using managed storage by default and securing traffic via IP access lists and network configuration. Billing is strictly consumption-based.
| Requirement | Hybrid workspace | Serverless workspace |
|---|---|---|
| On-premise and legacy | On-premise connectivity, for example CDC via Lakeflow Connect, and legacy Spark RDD or Scala code. | — |
| Analytics and BI | — | SQL analytics, AI/BI dashboards, Databricks Apps and Genie. |
| Workload profile | Continuous streaming triggers, fixed hardware or GPU needs, custom containers. | Spiky, unpredictable workloads; exploratory notebooks; serverless pipelines. |
| Operations and compliance | Strict policies requiring compute inside your own cloud boundary. | Teams without a dedicated DevOps resource; fast setup with minimal overhead. |
In practice, most mature estates run a combination: serverless for analytics, dashboards and exploration, with hybrid retained where legacy pipelines or specific network constraints demand it.
Decision 3: who governs?
Before structuring data, you must establish who owns the rules. Databricks identifies four operating models:
- Centralised governance. One core team manages policies, standards and access. Best for highly regulated industries where consistency trumps local agility.
- Federated governance. Business units govern their own data products within a central policy framework. Well suited to complex, multi-divisional enterprises.
- Decentralised governance. Independent departments manage data entirely on their own terms. Highly flexible, but carries high risks of duplicated effort and inconsistent security standards.
- Hybrid governance, or centre of excellence. Balances central oversight with domain freedom. Effective during major transformations, provided responsibilities are clearly defined.
Decision 4: how to organise the catalog
Unity Catalog organises assets across a three-level namespace: catalog, then schema, then table. Catalogs act as top-level containers, while workspace bindings dictate which environments can interact with which catalogs.
An effective design should make security straightforward to maintain, enable intuitive data discovery, and leave room for future growth. Organisations typically segment data across three recurring patterns:
| Pattern | Example catalogs | Strengths | Trade-offs |
|---|---|---|---|
| Environment + layer | bronze_devsilver_devgold_dev |
Simple security rules; minimal catalog count to manage. | Limited isolation for workspace binding; business users face a single, cluttered gold catalog. |
| Environment + domain | inventory_devhr_dev |
Workspaces bind easily by domain; domain-wide access granted in one step. | Harder to reuse data across domains; permissions must still be managed schema by schema. |
| Environment + layer + domain | bronze_sys1_devsilver_inv_devgold_supply_dev |
Strongest isolation; permissions and bindings managed cleanly at catalog level. | Generates a long list of catalogs for power users; higher operational setup overhead. |
Many organisations strip the environment suffix in production — supply_chain rather than
supply_chain_prod — establishing a convention where any catalog without an environment tag is
understood to be production.
Decision 5: how developers reach production-grade data
Data scientists and developers need realistic data to build effective models and pipelines, yet governance teams must protect raw production data. Three patterns resolve this tension:
- Catalog binding, read-only. Binds production catalogs directly into development workspaces in read-only mode. Lightweight to operate, but relies on robust entitlement management.
- Dedicated pre-production environment. Establishes a dev-on-prod workspace inside the production security perimeter. Developers build against live data within production guardrails, at the cost of maintaining an extra workspace.
- Data masking and de-identification. Copies production data into lower environments via automated masking pipelines. Provides complete isolation for sensitive datasets, but adds infrastructure and maintenance overhead.
The underlying foundation: infrastructure as code
Regardless of which architectural shape you select, core platform configuration should be defined as code. Defining workspaces, clusters and access rules via Terraform ensures reproducibility and prevents environment drift.
While a workspace can be clicked together in the console in minutes, manual setups inevitably break down over time. The admin workspace is the natural starting point for introducing Terraform into your deployment workflow.
Common estate archetypes
When these five decisions come together in the field, they generally form three recognisable patterns.
- Consolidated workspaces
- Serverless first
- Central team
- Layer and environment catalogs
- Catalog binding
- Split by regulatory boundary, such as GxP
- Hybrid compute where mandated, serverless elsewhere
- Environment, layer and domain
- Masked data copies
- Shared core, split by division where required
- Federated governance model
- Domain-led catalog structure
- Pre-production environments
Key takeaways
- Workspaces. Default to fewer, larger workspaces. Supplement with an admin and a sandbox workspace, splitting further only when forced by explicit regulatory boundaries.
- Compute. Use serverless for analytics, dashboards and exploratory work; retain hybrid compute for legacy pipelines, custom networking or strict compliance rules.
- Governance. Select an operating model — centralised, federated or hybrid — that matches how your business actually manages data ownership.
- Catalog architecture. Balance administrative complexity against security boundaries when deciding between layer, domain or hybrid catalog conventions.
- Development data. Establish a published strategy, whether read-only bindings, sandboxes or masked copies, before teams adopt informal workarounds.
- Automation. Enforce infrastructure as code from day one, starting with the administrative controls.
Determining your target shape
Every architectural option documented here is valid, which is precisely why vendor documentation presents them all. The real challenge lies in tailoring these choices to your organisation’s regulatory posture, operational capability and technical maturity.
We work alongside internal technology teams to evaluate existing environments, define target architecture, and articulate the rationale behind every trade-off.
If you would like to review the shape of your Databricks estate, or determine whether your current setup was designed by choice or by default, do get in touch.