The gap on your side of the table

Databricks builds an exceptional platform, and they are rightly disciplined about staying out of the consultancy trade. That is a sensible commercial call for them, but it leaves a series of structural choices sitting squarely on your desk — decisions made in week two that your business will still be living with in year four.

For businesses who haven’t been down this road before, this can be a daunting time, and decisions often presume ‘someone else’ also agrees this is the optimal path. Here are the three questions I ask of any Databricks setup to ensure it is starting in the right direction.

A lunch, and a quiet moment of frustration

I recently had lunch with a senior technology leader at one of our clients. For most of the meal, he spoke about Databricks with genuine warmth. The platform delivers on its promises, the engineering is sound, and his teams are building capability that was out of reach three years ago.

Over coffee, the tone shifted. His experience with external advisors had been a letdown. He had asked for concrete direction on how best to structure the platform for his business, and received little more than a polished deck outlining options, tied off with a polite version of “it depends”.

My initial instinct was to point out that this exact frustration is why firms like ours exist. Fortunately, I didn’t need to — he is already working with us on the problem, which made for a far more pleasant lunch.

It did, however, leave me with a question I haven’t been able to shake. If a leader of that seniority, with that budget and direct vendor access, feels the gap, how many other organisations are quietly suffering from it? How many are running on what I call the “someone else knows” assumption — the belief that the platform team, architect, or initial system integrator made these early choices deliberately, when in reality nobody did? And how much of the value they pay for is quietly leaking away as a result?

Plenty of options. Very few answers.

Spend an hour with Databricks’ architecture guidance and you will find it is thorough. I have just reviewed their latest planning documentation: it is candid about trade-offs, realistic about where the market is heading, and precise about where components belong — what lives per account, per region, or per workspace.

It outlines two paradigms for workspace segregation and the trade-offs of each. It evaluates three patterns for catalog design, complete with pros and cons, and offers three distinct ways to deliver production-grade data into development environments.

What it cannot tell you is which path to take.

That answer depends on your regulatory profile, your operating model, current data ownership, existing migration commitments, and the ongoing administrative load your platform team can realistically bear. None of that is in the vendor’s documentation, because none of it is theirs to know.

Why the vendor stops where it does

To be fair to Databricks, their boundary is logical. A platform vendor that drifts into deep strategy competes with its own ecosystem, takes on unpriced delivery risk, and becomes accountable for operational outcomes it cannot control. Clear boundaries are far better than vague ones. Indeed, their documentation includes a line I would happily frame for any IT office: “When in doubt, keep it simple.”

Yet the impact on the client remains. You are handed a comprehensive menu of choices, told it depends, and left to make week-two structural decisions that will cast a long shadow into year four.

What the gap actually costs

In our work across enterprise estates, the same five anti-patterns recur time and again. Each carries a hidden tax that never featured in the original business case:

• The POC becomes production. A workspace is provisioned quickly to test a concept. It works, nobody ever returns to architect it properly, and the experiment becomes the permanent foundation.
• First-mover namespace design. The first team to build sets a catalog convention tailored strictly to their own work. Every subsequent team inherits it — alongside an ad-hoc security model that evolved rather than being designed.
• Workspace sprawl across the org chart. Creating a workspace per business unit feels intuitive, but is usually a mistake. Databricks itself highlights a shift towards fewer, larger workspaces as broader governance controls mature. Running six environments means six sets of networking, six configurations to maintain, and a platform team spending its week on routine upkeep.
• Manual setup over code. Resources are created via the UI rather than as code. Environments drift, and nobody can state with confidence how staging differs from production.
• Development is starved of data. Governance rightly blocks raw production data in dev, but teams need realistic data to build anything of value. The standoff is resolved through informal workarounds — a manual export here, a local copy there — which is the worst possible outcome.

Every one of these is a rational local decision made by capable people operating without the full picture. Combined, they manifest as cloud bills outpacing value, data scientists waiting weeks for access, duplicated pipelines, and a platform delivering a fraction of its promised return.

Three questions for your platform team

The fix begins with three straightforward questions you can put to your team this week:

1. What specific boundary forced this workspace to exist, and is it still valid?

Every workspace must have a clear, justifiable boundary. Valid reasons include environment segregation (Dev/Stage/Prod), dedicated regional administration, explicit regulatory requirements (such as HIPAA or GxP), hard resource limits, or strict data isolation rules.

“The finance team wanted their own” is not a valid structural boundary — it is a preference masquerading as governance. Modern Unity Catalog entitlements mean most business-unit separation can now be handled cleanly within a shared workspace.

Ask the question plainly for every workspace you run. If it isn’t driven by a genuine regulatory or technical boundary, question its existence.

Note on cost: Features like disaster recovery can now be targeted specifically at critical workloads rather than applied blanket-wide. An organisation that segregates deliberately pays for resilience only where required. One that hasn’t pays for all of it — or none of it — and discovers which at the worst possible moment.

2. Who will manage this security model in two years, and can they support it?

In Unity Catalog, how you arrange catalogs and schemas is your security model. Whether you structure by environment and layer, or by environment and domain, you are defining how permissions flow and how easily users can find data.

Low-catalog structures keep things simple but force business users to filter through massive shared catalogs. Domain-heavy structures offer clean isolation, but introduce administrative overhead every time a new catalog is created.

The right answer depends on who will maintain it. A design that looks elegant on an architect’s whiteboard but requires constant specialist intervention becomes an operational liability the moment that specialist moves on. Design for the team and governance model you actually have, not the one you wish you had.

3. How do we get production-grade data to developers safely?

Data scientists cannot deliver value on synthetic scraps, and governance teams are right to guard production data.

There are three established ways to resolve this: read-only bindings to production catalogs from dev workspaces, dedicated production-grade sandbox environments, or automated masking pipelines.

Which mechanism you select matters less than making an explicit choice before the request lands. Define it, publish it, and enforce it. The alternative is the informal workaround, and informal workarounds have a habit of turning into audit findings.

How we help

We operate in the space between the platform vendor and your internal delivery team. Having guided these decisions across numerous enterprise estates, we understand the downstream consequences eighteen months out, and can help you select the pattern that fits your operating model.

Databricks provides the capability. Our role is to ensure you structure it correctly from the outset.

If any of this resonates with your current setup, or if you’d like a sounding board for your platform architecture, do get in touch.