Get in touch

Databases & Data

Databricks Consulting: Data Engineering and Analytics

Our Databricks consulting helps data and IT managers bring data from different systems into one place, organised for reporting and analysis. Databricks is a cloud platform for storing, cleaning and analysing large amounts of data. We design the data pipelines, move your data across and set it up so your team can use it.

Get in touch

What is Databricks?

Databricks is a data and analytics platform that runs in the cloud: on Amazon Web Services (AWS), Microsoft Azure or Google Cloud. It works with the storage in your own cloud account.

It is built around a data lakehouse. A data lake stores lots of raw data cheaply, in any format. A data warehouse stores tidy, checked data that is quick to report on. A lakehouse combines the two, so reports, analysis and AI all work from the same copy of the data.

Tables are stored with Delta Lake, an open-source format and the default on Databricks. It keeps a log of every change, so an update either finishes completely or not at all, and you can look back at earlier versions of a table.

The main parts

  • Notebooks: shared documents where people write and run code in Python, SQL, Scala or R, together.
  • Databricks SQL: ask questions of your data in SQL, the standard language for databases.
  • Data pipelines (Lakeflow): bring data in from databases, apps and files, clean it and run it on a schedule.
  • Dashboards: built-in AI/BI dashboards, or connect tools you already use, such as Power BI.
  • Unity Catalog: controls who can see which data, and tracks where it came from.
Diagram: data from your sales, finance and other systems flows through pipelines into a Databricks lakehouse in your cloud account, where it is cleaned in stages and then used for reports, dashboards and analysis.Your systemsSales, finance, apps, filesData pipelinesIN YOUR CLOUD ACCOUNTDatabricks lakehouse1. Raw data, as it arrives2. Cleaned and checked3. Ready for reportingStored as Delta Lake tablesAccess controlled by Unity CatalogReports and analysisDashboards, Power BI, SQL, AIOne copy of the data for everyone

Why businesses use it

  • One place for your data: figures from sales, finance and other systems in one set of tables, instead of copies in spreadsheets.

  • Reports you can trust: pipelines clean and check data the same way every time, on a schedule.

  • Room to grow: serverless compute (computing power that Databricks manages for you) scales with the workload, so you don’t buy a bigger server upfront.

  • Familiar tools: SQL and Python for analysts, Power BI or built-in dashboards for everyone else.

  • Control over access: Unity Catalog manages permissions and keeps an audit trail of who used which data.

Pricing: Databricks charges for processing in DBUs (Databricks Units, a measure of computing power), billed per second. Built-in budgets and alerts help you watch spend. See the Azure Databricks pricing page for an example of current rates.

What we do for you

  • Data pipeline design and build: data from your systems, cleaned in stages and updated on a schedule.

  • Data migration to Databricks: move data from an existing database, warehouse or spreadsheets, and check the figures match.

  • SQL and notebook development: queries, data models and notebooks your team can reuse.

  • Dashboards and reports: Databricks AI/BI dashboards, or Power BI connected to your data.

  • Cost and performance review: find slow queries and unused compute, and set budgets and alerts.

  • Workspace and access setup: Unity Catalog permissions, so people see only the data they should.

Is it right for you?

A good fit if

  • Your data sits in several systems, and reports take hours of copying and pasting.
  • You have large or fast-growing data that one database struggles with.
  • You already use AWS, Azure or Google Cloud, or are happy to.
  • You want reports and AI projects to work from the same data.

Maybe not if

  • Your data is small and fits well in one database. PostgreSQL is simpler and cheaper.
  • You only need a few reports from one system. A reporting tool connected directly may be enough.
  • Your data can’t go to a public cloud. Databricks runs on AWS, Azure and Google Cloud only.
  • No one will look after the pipelines and costs, and you don’t want ongoing support.

How we work

  1. Talk

    We learn where your data lives, who uses it and which reports you need.

  2. Plan

    We design the tables and pipelines, and estimate the running costs.

  3. Build

    We build the pipelines, move the data and check the figures against your current reports.

  4. Support

    We monitor pipelines, fix issues and review costs.

Questions we get asked

Something else on your mind? Ask us.

Can our data stay in South Africa?

On Azure, yes. Azure Databricks is offered in Microsoft’s South Africa North (Johannesburg) region. Some features are only available in certain regions, so we check the ones you need first.

How much does Databricks cost?

You pay for what you use, per second, in DBUs. On classic compute you also pay your cloud provider for servers. The total depends on how much data you process, so we estimate it before you commit.

Can we keep using Power BI?

Yes. Power BI can connect to Databricks SQL warehouses, so your reports can read the cleaned data directly.

Is Databricks too much for a small business?

Often, yes. If your data fits comfortably in one database, PostgreSQL is usually the better choice. We tell you if that’s the case.

Are you a Databricks partner?

No. We are independent and work in your own Databricks account.

Sources and further reading

Let's work together

Ready to talk?

Tell us what you need. We'll reply within 2 working days.

Get in touch

Databricks is a trademark of Databricks, Inc. Delta Lake, Power BI, AWS, Azure and Google Cloud are trademarks of their respective owners. Mammoth Digital is not affiliated with or endorsed by Databricks, Inc.

Last updated