On this page 6 sections
To migrate to Databricks from SQL Server or Excel, you copy your data into Databricks on a schedule, rebuild your reports on top of it, run the old and new side by side until the numbers match, and only then switch the old setup off. You don't have to move everything at once, and for some businesses moving at all isn't worth it.
Short answer
- Move in stages: audit, choose a region, set up the workspace, load the data, rebuild reports, run in parallel, switch off.
- Databricks can read SQL Server in place for a trial, but for regular loads you copy the data in.
- Excel files can be read, but spreadsheet quirks (merged cells, macros, hidden logic) cause the most surprises.
- Choose a region with POPIA in mind. Azure Databricks is available in Johannesburg.
- Don't move if one database already gives you the reports you need.
Why move, and when not to
Businesses usually move for three reasons: reports have become slow, data sits in several systems and nobody trusts the totals, or they want to grow into analytics and machine learning. Databricks is described by Microsoft as a platform for ETL, analytics and BI, machine learning and AI (Microsoft Learn (opens in a new window)).
Don't move if your reports come from one database and run quickly enough. Read Do you need Databricks? first.
You also don't have to choose all or nothing. Databricks' query federation can query SQL Server in place, read-only, without migrating anything. Databricks suggests it for ad hoc reporting and proof-of-concept work, and recommends managed connectors for high-volume ingestion (Microsoft Learn (opens in a new window)). And an Excel add-in lets staff browse and import governed Databricks data into Excel, so people keep working in the tool they know (Microsoft Learn (opens in a new window)).
Before you start: checklist
- A list of every data source: databases, spreadsheets, exports from your ERP system, and systems you read through an API.
- The five to ten reports that matter most, and who uses them.
- A named owner for each data source.
- Which columns hold personal information, such as names, ID numbers and contact details.
- A copy of your current numbers for a recent month, to compare against later.
- A rough monthly budget. See Databricks pricing explained.
- A decision on who will look after the platform afterwards.
The seven steps
- Audit your data sources. For each source, note what it holds, how often it changes, who owns it and how clean it is. Spreadsheets need extra care: list their formulas, macros and manual steps.
- Choose your cloud and region. Azure Databricks is available in South Africa North (Johannesburg) (Microsoft Learn (opens in a new window)). Microsoft's regional page lists serverless compute, serverless SQL warehouses and managed Lakeflow Connect connectors there, but not Databricks Apps or Lakebase Autoscaling, and managed disaster recovery isn't listed (Microsoft Learn (opens in a new window)).
- Set up the workspace and Unity Catalog. Unity Catalog controls who can see what, keeps an audit record of data access and tracks lineage (Databricks docs (opens in a new window)). Map your current roles to it before any data arrives.
- Bring the data in.
- SQL Server: Lakeflow Connect has managed connectors for databases, and SQL Server is named as one example (Microsoft Learn (opens in a new window)).
- Excel: Databricks reads
.xlsand.xlsxfiles, one sheet at a time (Microsoft Learn (opens in a new window)). For files kept in SharePoint, there is a SharePoint file connector. - Where you can, move spreadsheet data into proper tables first, then load those.
- Rebuild the reports. Power BI Desktop connects to Azure Databricks, and Databricks recommends SQL warehouses for DirectQuery (Microsoft Learn (opens in a new window)). Databricks also has its own AI/BI dashboards.
- Run old and new side by side. For a few reporting cycles, produce each key report both ways and compare the totals. Agree in advance how large a difference is acceptable, and investigate every other one.
- Switch off the old setup. Keep a backup, write down how the new pipelines work, and set budgets and alerts so the first bill holds no surprises.
Risks to plan for
| Risk | What it looks like | What to do |
|---|---|---|
| Data quality | The same customer spelled three ways; duplicate invoices | Agree cleaning rules and apply them in the pipeline, not by hand |
| Hidden spreadsheet logic | Formulas, macros and manual fixes nobody wrote down. Databricks' reader sets merged cells to empty (except the top-left cell) and does not run macros | Document the logic and rebuild it as tested queries |
| Access rights | Everyone who could see the old database can suddenly see everything, or no one can | Map roles to Unity Catalog permissions and test with real users |
| POPIA security | Personal information is copied into a new system | POPIA s19 requires "appropriate, reasonable technical and organisational measures" (POPIA s19 (opens in a new window)) |
| POPIA contracts | A provider processes personal information for you | Section 21 requires a written contract with the operator (POPIA s21 (opens in a new window)) |
| Data leaving South Africa | The workspace sits in another country | Section 72 sets conditions for transfers abroad (POPIA s72 (opens in a new window)). Hosting in Johannesburg keeps the data in South Africa, but check where backups and support access sit |
| Cost surprises | Compute left running | Set budgets and alerts. Databricks says budgets are not an absolute spend cap, so also watch usage (Databricks docs (opens in a new window)) |
Not legal advice: check your own obligations with a POPIA specialist.
How Mammoth Digital can help
Our team works with SQL Server, PostgreSQL and Databricks, and includes a Databricks Certified Data Engineer Professional and Databricks Certified Data Analysts. We can audit your sources, tell you honestly whether a move is worthwhile, and if it is, plan and run it with you.
See our Databricks consulting and Databases & Data pages, or get in touch. We'll reply within 2 working days.
Databricks is a trademark of Databricks, Inc. Microsoft, Azure, SQL Server, Excel, SharePoint and Power BI are trademarks of Microsoft Corporation. Mammoth Digital is independent and not a partner of these companies.