Data pipelines (ETL/ELT)
A data pipeline moves the records from your systems into one place on a schedule, cleans them and keeps a trace back to the source. ETL shapes the data before it is loaded. ELT loads the raw data first and shapes it inside the warehouse. We usually build ELT, so the raw data stays as evidence and every later step can be rebuilt from it. The pipelines, the warehouse and the code live in your own cloud.
This is for you if
- Every week someone exports files from the webshop, the bank and the accounting system and joins them by hand.
- When a figure is questioned, nobody can show which record it came from.
- A source changed its export, and you found out from a wrong report.
What you leave with
- The result
- Pipelines that run on their own, in code you own. They load your warehouse on a schedule in Google Cloud (BigQuery) or Microsoft Azure and Fabric. A check after each run stands between the data and anyone who reads it.
- Read-only sources
- The pipelines read your accounting, ERP and inventory systems, webshops, card payments, bank and exchange rates. The code makes sure they never write back.
- Raw files kept
- Every raw response is kept as it arrived, so a questioned figure points back to the file it came from.
- A layered warehouse
- Raw as received, staging cleaned and typed, intermediate with your business rules and matches, and core and marts with the tables finance reports and books from.
What we do, what you do
| We | You |
|---|---|
| Agree a short written scope for each source with you: what to pull, how often, and which figures matter. | Give read access to each source, through someone with admin rights. |
| Build the pipelines, the warehouse layers and the checks for each table in your cloud project. | Check the first figures against a report you trust. |
| Set up the alerts, so a failed check or a changed source reaches a person. | Decide who gets the alerts and who may read what. |
| Write each new pipeline with Gathena's agent, from its scope. The code ships from your Git only after you approve the plan. | Review each plan and approve it before anything changes. |
Worth knowing
Pipelines are the clearest example of work that should run on its own. Pulling and checking the same data every night repeats and is well defined, so nobody needs to do it by hand: it runs on its own, it is monitored, and you check the results. This page covers what the pipelines are and how they are built. The data foundation page covers what you get from them.
Your questions
- ETL or ELT: does it matter to us?
- Mostly for tracing figures. With ELT the raw data sits in the warehouse as received, so a figure can be traced back to it. When a business rule changes, the tables are rebuilt from the raw layer, and the sources are not pulled again.
- Which systems can you connect?
- Systems with an API or a file export. In the DuffBeauty case the pipelines read an inventory and ERP system, webshops, the accounting system, card payments, the bank's statement export and daily exchange rates.
- Google Cloud or Microsoft?
- Either, in a Google Cloud project or a Microsoft tenant you own and control access to. We build on Google Cloud with BigQuery, or on Microsoft Azure and Fabric.
- What happens when a run fails?
- Nobody reads the data from that run until its checks pass, the run leaves a receipt, and the alert goes to the person you chose. A source that changes is reported instead of guessed at.
- Can we change a pipeline without you?
- Yes. The code, the settings and the agent that builds and maintains the pipelines stay with you. You give it a short scope, review its plan and approve it before anything changes.