Back to jobs

P1 - GP - Senior Data Engineer / Data Architect

Pay
Pay not listed
Work setting
Not stated
Employment
Not stated

What you’ll work on

Full posting

We are hiring a Senior Data Engineer to build and run the pipelines that bring portfolio company data into one business-aligned model on Databricks.

The data comes from a third-party portfolio monitoring platform, with history carried over once from a legacy portfolio system.

  • Build incremental loads into Bronze using watermarks and keep full raw history

  • Model Silver data with deduplication, unit conversion and quarantine of suspect rows

  • Build the source-agnostic Gold business model as performant Delta tables

  • Reconcile new figures against the legacy system before the cutover

  • Publish Gold to Lakebase synced tables and Delta access for BI

From the employer’s posting
We are hiring a Senior Data Engineer to build and run the pipelines that bring portfolio company data into one business-aligned model on Databricks.
The data comes from a third-party portfolio monitoring platform, with history carried over once from a legacy portfolio system. You will take it through Bronze, Silver and Gold layers and serve it to Power BI reporting and internal applications. You will own pipelines end to end: ingestion, transformation, data quality, performance and day-to-day operation.
build incremental loads from the source platform into Bronze using change timestamps and watermarks
model source data close to its native structure, with deterministic deduplication, type-aware unit conversion, sign rules and quarantine of suspect rows rather than silent fixes
implement a source-agnostic business model of dimensions, fact tables with different history rules and reference data, built as performant Delta tables
reconcile new figures against the legacy system before the cutover
publish Gold to downstream consumers, including Lakebase (Postgres) synced tables for applications and direct Delta access for BI
Apply at Atlas Energy Solutions

Posting record

Report an error

Actively hiringFirst seen by us 2 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 19, 20262 days ago
Recorded sightings
4
Last seen by us
Nov 22, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 2

Early. Similar jobs usually come down between day 6 and day 56.

From 14,770 similar postings that came down. Coming down is not always a hire.

Atlas Energy Solutions is hiring

3 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Hiring is picking up
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • SQL
  • T-SQL
  • Azure
  • Databricks
  • Delta Lake
  • PostgreSQL
  • Power BI
  • Spark
  • SQL Server
  • PySpark
Where the posting names them
- 5+ years building production data pipelines, including 2+ years on Databricks with Delta Lake. - Strong SQL (Spark SQL and T-SQL) and Python or PySpark. - Hands-on experience with a medallion (Bronze, Silver, Gold) architecture and incremental processing: watermarks, MERGE, change data feed.
Languages Spark SQL, Python, PySpark, T-SQL Application serving
- Lakeflow (Delta Live Tables) or Databricks Workflows orchestration. - Azure SQL and migrating SQL Server logic to Databricks. - Power BI data modeling, or working closely with BI developers.
Legacy and transition Azure SQL Database Reporting
Cloud Microsoft Azure
About the role We are hiring a Senior Data Engineer to build and run the pipelines that bring portfolio company data into one business-aligned model on Databricks. The data comes from a third-party portfolio monitoring platform, with history carried over once from a legacy portfolio system. You will take it through Bronze, Silver and Gold layers and serve it to Power BI reporting and internal applications. You will own pipelines end to end: ingestion, transformation, data quality, performance and day-to-day operation.
What you'll bring - 5+ years building production data pipelines, including 2+ years on Databricks with Delta Lake. - Strong SQL (Spark SQL and T-SQL) and Python or PySpark.
- Financial reporting concepts: LTM and YTD periods, fiscal year ends, budgets and forecasts, restatements, currency conversion. - Databricks Lakebase or PostgreSQL, including serving data to applications. - Lakeflow (Delta Live Tables) or Databricks Workflows orchestration.
- Databricks Lakebase or PostgreSQL, including serving data to applications. - Lakeflow (Delta Live Tables) or Databricks Workflows orchestration. - Azure SQL and migrating SQL Server logic to Databricks.
- Power BI data modeling, or working closely with BI developers. - Infrastructure as code and CI/CD for data pipelines, for example Databricks Asset Bundles. Our environment
Lakehouse Databricks, Delta Lake, Unity Catalog Languages
Application serving Databricks Lakebase (Postgres) synced tables Legacy and transition
- Silver: model source data close to its native structure, with deterministic deduplication, type-aware unit conversion, sign rules and quarantine of suspect rows rather than silent fixes. - Gold: implement a source-agnostic business model of dimensions, fact tables with different history rules and reference data, built as performant Delta tables for Power BI and application use. - Migration: support a single cutover from the legacy system, and load and freeze the history of companies that will not move.
- Data quality and reconciliation: build checks and quality flags, reconcile new figures against the legacy system before the cutover and run regression checks when the model changes. - Serving: publish Gold to downstream consumers, including Lakebase (Postgres) synced tables for applications and direct Delta access for BI. - Operations: orchestrate and monitor the pipelines, manage access in Unity Catalog, and keep documentation, lineage and runbooks current.
We are hiring a Senior Data Engineer to build and run the pipelines that bring portfolio company data into one business-aligned model on Databricks. The data comes from a third-party portfolio monitoring platform, with history carried over once from a legacy portfolio system. You will take it through Bronze, Silver and Gold layers and serve it to Power BI reporting and internal applications. You will own pipelines end to end: ingestion, transformation, data quality, performance and day-to-day operation. What you'll do
- Azure SQL and migrating SQL Server logic to Databricks. - Power BI data modeling, or working closely with BI developers. - Infrastructure as code and CI/CD for data pipelines, for example Databricks Asset Bundles.
Reporting Power BI Cloud

Your next step

Complete your application on atlas.bamboohr.com. The employer’s form will show what is required.

Applying through BambooHR

  • No account needed. The form takes images too, but an image of a resume has no text for anyone to search: upload a PDF or Word file.

Already applied? Track this application

Job description

View original posting ↗

Job Description: Senior Data Engineer / Data Architect



About the role

We are hiring a Senior Data Engineer to build and run the pipelines that bring portfolio company data into one business-aligned model on Databricks.

The data comes from a third-party portfolio monitoring platform, with history carried over once from a legacy portfolio system. You will take it through Bronze, Silver and Gold layers and serve it to Power BI reporting and internal applications. You will own pipelines end to end: ingestion, transformation, data quality, performance and day-to-day operation.



What you'll do

  • Ingestion: build incremental loads from the source platform into Bronze using change timestamps and watermarks, with safe handling of late-arriving rows and full retention of raw history.
  • Silver: model source data close to its native structure, with deterministic deduplication, type-aware unit conversion, sign rules and quarantine of suspect rows rather than silent fixes.
  • Gold: implement a source-agnostic business model of dimensions, fact tables with different history rules and reference data, built as performant Delta tables for Power BI and application use.
  • Migration: support a single cutover from the legacy system, and load and freeze the history of companies that will not move.
  • Data modeling: work with the data model owner on the item catalog, dimensions (period, currency, fund and entity), cash transactions and derived measures such as multiples and FX conversion.
  • Data quality and reconciliation: build checks and quality flags, reconcile new figures against the legacy system before the cutover and run regression checks when the model changes.
  • Serving: publish Gold to downstream consumers, including Lakebase (Postgres) synced tables for applications and direct Delta access for BI.
  • Operations: orchestrate and monitor the pipelines, manage access in Unity Catalog, and keep documentation, lineage and runbooks current.
  • Collaboration: work closely with the vendor's data team, reporting and application teams, and finance and portfolio operations users on definitions and issues.



What you'll bring

  • 5+ years building production data pipelines, including 2+ years on Databricks with Delta Lake.
  • Strong SQL (Spark SQL and T-SQL) and Python or PySpark.
  • Hands-on experience with a medallion (Bronze, Silver, Gold) architecture and incremental processing: watermarks, MERGE, change data feed.
  • Dimensional modeling: facts at a clear grain, dimensions, surrogate keys, slowly changing dimensions.
  • A track record of data quality work: deduplication, reconciliation, and tracing a wrong number back to its source.
  • Experience with Unity Catalog or a comparable governance layer: permissions, lineage, documentation.
  • Performance tuning for BI workloads, such as file layout, clustering and query plans.
  • Clear written communication, including explaining data issues and trade-offs to non-technical users.


Nice to have

  • Private equity, asset management or fund administration data: portfolio monitoring, valuations, cash flows, fund and co-invest structures.
  • Financial reporting concepts: LTM and YTD periods, fiscal year ends, budgets and forecasts, restatements, currency conversion.
  • Databricks Lakebase or PostgreSQL, including serving data to applications.
  • Lakeflow (Delta Live Tables) or Databricks Workflows orchestration.
  • Azure SQL and migrating SQL Server logic to Databricks.
  • Power BI data modeling, or working closely with BI developers.
  • Infrastructure as code and CI/CD for data pipelines, for example Databricks Asset Bundles.



Our environment

Area

Tools

Lakehouse

Databricks, Delta Lake, Unity Catalog

Languages

Spark SQL, Python, PySpark, T-SQL

Application serving

Databricks Lakebase (Postgres) synced tables

Legacy and transition

Azure SQL Database

Reporting

Power BI

Cloud

Microsoft Azure

 

About Atlas Energy Solutions

People
About 1,511 in 2025, up 32% from the year before.Annual report (SEC Form 10-K)
Company
Listed as AESI.

Demand for Data Engineer roles

New Data Engineer postings at the employers we follow are up 13% over the last 6 weeks compared with the 6 before.

Weekly new Data Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 845, the most 1,344, and the latest week had 1,007.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under Data Engineer, by the work the posting describes.

Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.

Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.

An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.

US government figures for database architects

Median yearly pay, May 2025
$139,500
Expected change, 2025 to 2035
9.4% more jobsagainst 3.5% for all jobs

The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1243.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Bengaluru, Karnataka, India

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.