Senior Consultant/Manager - Data Engineer

We're looking for a developer to help build and maintain data pipelines on Databricks for a financial services/wealth management data platform. The platform ingests data from multiple source systems, transforms it through a medallion architecture (bronze → silver → gold) using PySpark, and publishes curated data to downstream consumers via file exports and Kafka. You'll work alongside a senior engineer, writing transform code, fixing pipeline issues, and helping keep the environment healthy across dev/qa/uat. This is a good fit for someone with solid Python fundamentals and some Spark/SQL exposure who wants to grow into a data engineering specialist. You won't be expected to know Databricks internals on day one — you will be expected to learn fast, ask good questions, and write clean code. What You'll Work On

- Debugging failed pipeline runs: reading pipeline event logs, tracing bad records, fixing schema drift or data quality issues. - Maintaining reference/lookup tables and small utility scripts (PowerShell/Python) used to operate the platform. - Writing and updating unit/integration tests for transform logic.

Must-Have Skills - Python — comfortable writing clean, readable code; understands functions, modules, basic OOP. - Databricks

- Working with Lakeflow Declarative Pipelines (Databricks' current pipeline framework — successor to DLT) and Auto CDC / SCD Type 2 patterns for change-data-capture merges. - Querying and validating data in Unity Catalog across environments (dev/qa/uat) using SQL warehouses. - Debugging failed pipeline runs: reading pipeline event logs, tracing bad records, fixing schema drift or data quality issues.

- Databricks - Some exposure to Apache Spark / PySpark, or strong SQL skills plus a willingness to learn Spark quickly. - Working knowledge of SQL (joins, aggregations, window functions).

- Some exposure to Apache Spark / PySpark, or strong SQL skills plus a willingness to learn Spark quickly. - Working knowledge of SQL (joins, aggregations, window functions). - Basic Git workflow: branches, commits, pull/merge requests.

Nice-to-Have Skills - Direct experience with Databricks (notebooks, jobs, clusters, or SQL warehouses). - Familiarity with Delta Lake, medallion architecture (bronze/silver/gold), or CDC/SCD concepts.

- Familiarity with Delta Lake, medallion architecture (bronze/silver/gold), or CDC/SCD concepts. - Exposure to Azure (this environment runs on Azure Databricks with Azure AD service principal auth). - Experience with Kafka or other streaming/event systems.

We are hiring for both permanent and contract positions for this role. We're looking for a developer to help build and maintain data pipelines on Databricks for a financial services/wealth management data platform. The platform ingests data from multiple source systems, transforms it through a medallion architecture (bronze → silver → gold) using PySpark, and publishes curated data to downstream consumers via file exports and Kafka. You'll work alongside a senior engineer, writing transform code, fixing pipeline issues, and helping keep the environment healthy across dev/qa/uat. This is a good fit for someone with solid Python fundamentals and some Spark/SQL exposure who wants to grow into a data engineering specialist. You won't be expected to know Databricks internals on day one — you will be expected to learn fast, ask good questions, and write clean code.

- Writing and maintaining PySpark transforms (hand-written, not a generic framework) for silver and gold layer tables — things like customer, account, and transaction data models. - Working with Lakeflow Declarative Pipelines (Databricks' current pipeline framework — successor to DLT) and Auto CDC / SCD Type 2 patterns for change-data-capture merges. - Querying and validating data in Unity Catalog across environments (dev/qa/uat) using SQL warehouses.

- Python — comfortable writing clean, readable code; understands functions, modules, basic OOP. - Databricks - Some exposure to Apache Spark / PySpark, or strong SQL skills plus a willingness to learn Spark quickly.

- Comfortable running existing operational scripts to check pipeline status, query tables, and diagnose failures without hand-holding. - Starting to take ownership of the Databricks environments and help write new transforms Salary

- Direct experience with Databricks (notebooks, jobs, clusters, or SQL warehouses). - Familiarity with Delta Lake, medallion architecture (bronze/silver/gold), or CDC/SCD concepts. - Exposure to Azure (this environment runs on Azure Databricks with Azure AD service principal auth).

- Exposure to Azure (this environment runs on Azure Databricks with Azure AD service principal auth). - Experience with Kafka or other streaming/event systems. - Experience with CI/CD pipelines (this project uses GitLab CI).

What You'll Work On - Writing and maintaining PySpark transforms (hand-written, not a generic framework) for silver and gold layer tables — things like customer, account, and transaction data models. - Working with Lakeflow Declarative Pipelines (Databricks' current pipeline framework — successor to DLT) and Auto CDC / SCD Type 2 patterns for change-data-capture merges.