Databases & Data

What Does a Databricks Consultant Do? And When Do You Need One?

This guide explains the work in simple, easy to understand terms, the signs you need help, and when you can manage without it.

Published 6 min read

On this page 8 sections
A printed floor plan on a worktable with a tape measure, a short red pencil and a navy notebook

A Databricks consultant is an outside specialist who plans, builds and fixes your Databricks platform: how it is set up, how data flows into it, who can see what, and what it costs. You need one when you are starting a new setup, moving data from another system, or running something slow, fragile or expensive that nobody on your team has built before.

Short answer

  • A Databricks consultant designs the platform, builds and moves data pipelines, sets access rules, keeps costs in check and hands everything over to you.
  • Hire one for a new setup, a migration, a bill that keeps growing, or pipelines that keep breaking.
  • You may not need one if you are only learning, or your reporting fits comfortably on one database.
  • Check certifications and references, and make sure the work sits in your own cloud account and code repository.
  • If you are unsure, ask for a short review before committing to a project.

What is Databricks?

Microsoft describes Azure Databricks as "a unified, open analytics platform for building, deploying, sharing, and maintaining enterprise-grade data, analytics, and AI solutions at scale" (Microsoft Learn (opens in a new window)). In plain terms, it is a cloud platform that brings data from different systems into one place, where it can be cleaned, reported on and used for AI. It integrates with cloud storage and security in your own cloud account.

On Azure, Databricks is available in the South Africa North (Johannesburg) region, although some features are offered only in some regions (Microsoft Learn (opens in a new window)). A consultant should check the features you need before you commit.

What does a Databricks consultant do?

Databricks publishes a ten-phase planning guide for production setups, which focuses on "design and architecture decisions". The jobs below follow that guide and the rest of Databricks' documentation.

JobWhat it means for you
Plan the platformDecide how accounts, workspaces, access, network, storage, compute, monitoring and recovery fit together, so the setup suits your size and rules.
Design the data flowOrganise data in layers: bronze (raw), silver (cleaned and checked) and gold (ready for reports). Databricks calls this medallion architecture - a recommended practice, not a requirement.
Build and move pipelinesCopy data in from your systems on a schedule, clean it, and check the numbers match your old reports before anything is switched off.
Set up access and governanceDecide who can see which data. Unity Catalog enforces permissions when someone queries a table, records data access and tracks where data came from.
Keep costs in checkDatabricks' cost advice includes tagging usage, setting budgets and alerts, using compute policies, switching idle compute off and using cheaper job compute for scheduled work.
Automate and hand overKeep jobs and pipelines as code in source control, using Declarative Automation Bundles. Databricks advises infrastructure as code and warns against creating production workspaces by hand.
Plan for failureAgree how long you can be down and how much data you can lose, pick a recovery approach and test it. Databricks suggests testing recovery at least quarterly.
Train your teamShow your people how it works so they can run it without the consultant.

The last row is our own view of good practice. The others come from Databricks' documentation.

Signs you need a Databricks consultant

  • You are starting from scratch. Early choices, such as how many workspaces to run and how to organise access, are expensive to undo later. That is our judgement, and it is why Databricks' guide puts so much weight on design.
  • You are moving from SQL Server, Excel or an old warehouse. The new reports must match the old ones before you switch off the old system. See moving to Databricks from SQL Server or Excel.
  • The bill is higher than you expected. Databricks charges by usage, so unattended compute adds up. Our guide to Databricks pricing shows how.
  • Pipelines are slow, break often or are deployed by hand. Repeatable, tested deployments are what a consultant brings.
  • Nobody on your team has built production data pipelines. Databricks describes its Professional data engineer exam as validating "advanced skills in building, optimising, and maintaining production-grade data engineering solutions". That is a useful picture of the skills involved.
  • You must show who accessed what. Unity Catalog's audit and lineage features need to be set up properly to help.
  • You have no recovery plan. If the platform became unavailable, would you know what to do?

You may not need one if...

  • You are learning or experimenting. Databricks Free Edition is "meant for non-commercial use" and has no support or service-level agreement, so it suits learning, not real business data.
  • Your data lives in one or two systems and reports run fine. A database plus a reporting tool may be enough. Read do you need Databricks first.
  • You already have an experienced engineer who just needs extra hands. That calls for a contractor, not a consultant (our view).
  • You are new to Databricks and want to try it. Databricks' own guide tells newcomers to start with a serverless workspace before planning an enterprise setup.

What to ask before you hire

  • Certifications. Databricks offers Associate and Professional certifications for data engineers. They are valid for two years, and holders can share a digital badge as proof. Ask to see a current one.
  • Real projects. An exam shows knowledge, not experience. Ask for references and an example of a handover.
  • Partner status. Databricks runs a Partner Program with consulting partners and a "Find a partner" directory. You can check a firm there. Partner status is not the only sign of quality, so ask every firm for the same evidence.
  • Ownership. Workspaces should be in your cloud account, and code in your repository, with documentation.
  • A plan and cost estimate in plain English before work starts.

Which option fits you?

Your situationBest fit
Learning or testing ideasDatabricks Free Edition (non-commercial use only)
A one-off design, migration or clean-upA consultant for a defined piece of work
Daily pipelines and reports that keep changingTrain or hire your own engineer, with a consultant setting the foundations
Not sure Databricks is rightA short review of your data and reports before spending

How Mammoth Digital can help

Our team works with Databricks and includes a Databricks Certified Data Engineer Professional and Databricks Certified Data Analysts. We can review your data and reports, tell you whether Databricks is the right fit, and if it is, help you plan, build and cost it.

See our Databricks consulting and Databases & Data pages, or get in touch. We'll reply within 2 working days.

Databricks is a trademark of Databricks, Inc. Microsoft and Azure are trademarks of Microsoft Corporation. Mammoth Digital is independent and not a partner of these companies.

Let's work together

Ready to talk?

Tell us what you need. We'll reply within 2 working days.

Get in touch