Time since we first saw it
Day 176
Up longer than most. Similar jobs usually come down between day 6 and day 54.
From 10,764 similar postings that came down. Coming down is not always a hire.
Senior Data Engineer
IN MH Mumbai Eureka
Open 5 months, first seen by us 176 days ago- Pay
- Pay not listed
- Work setting
- Not stated
- Employment
Full-time — employment source
Employment type in the employer’s posting: Full time
From the employer’s posting
What you’ll work on
Full postingBuild Flink streaming and PySpark batch ingestion pipelines for HL7v2, FHIR, CCDA and CSV feeds
Build the dbt transformation layer, from staging models to CDM targets with SCD2
Implement the synchronous Informatica MDM call pattern with batching, timeouts and circuit breakers
Build FHIR serialization layer and US Core 6.1 validation gating publication
Orchestrate dbt runs, batch jobs and Iceberg maintenance with Cloud Composer DAGs
From the employer’s posting
Build and maintain the ingestion pipelines — Apache Flink streaming jobs on Dataproc for HL7v2 and FHIR feeds, PySpark batch jobs for CCDA and CSV bulk loads.
Build the dbt transformation layer end to end
Implement and maintain the synchronous Informatica MDM call pattern in the ingestion path — batched calls, timeout handling, circuit breaker behavior, DLQ routing for MDM failures.
the FHIR validator integration that gates publication on US Core 6.1 conformance
Implement Cloud Composer DAGs to orchestrate dbt runs, batch ingestion jobs, maintenance operations
Education & alternatives
Required Skills and Qualifications - Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field - 3+ years of hands-on data engineering experience
Posting record
Report an errorOpen 5 monthsWe first saw this posting 176 days ago and it is still up. Roles open this long are often slow to fill or kept open to collect applications.
- Status in our records
- Live
- First seen by us
- May 11, 2026176 days ago
- Recorded sightings
- 225
- Last seen by us
- Nov 4, 2026
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
How this job compares
Quantiphi is hiring
30 open jobs
Open jobs by the week we first saw them, last 12 weeks.
- Data Engineer hiring is steady
- Takes jobs down at a usual pace
- 3% of open jobs were reposted
See how this role fits your experience
Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
- Java
- Kotlin
- Python
- SQL
- dbt
- Iceberg
- Kafka
- Snowflake
- Spark
- Trino
- PySpark
- GCP
Where the posting names them
The Data Engineer is the implementation backbone of the platform. You will build and operate the ingestion pipelines, the dbt transformation layer, the FHIR serialization pipelines, the FHIR-Repository integration components, and the data products that depend on them. You will work within a spec-driven development framework — every component begins with an approved specification, and your work is generated, refined, and verified against that specification. This is a hands-on role. You will write Python, PyFlink, PySpark, dbt SQL, and supporting Java for FHIR-Repository interceptor work, with Code agents as a coding assistant under your direction. Code review, test authorship, and operational ownership of the components you build are part of the role. The platform handles regulated healthcare data at high volume, and operational discipline matters. Key Responsibilities
- Build the FHIR serialization layer — flat FHIR Iceberg tables (one per resource type) materialized via dbt, the PySpark bundling pipeline that produces FHIR Bundles for Kafka publication, and the FHIR validator integration that gates publication on US Core 6.1 conformance - Build and maintain FHIR-Repository integration components — the Java/Kotlin egress interceptor that captures client-originated FHIR changes, the Flink loopback consumer that merges those changes into CDM, the bundle consumer that ingests CDM-originated bundles into FHIR-Repository. Implement origin-tag-based loop prevention - Implement Cloud Composer DAGs to orchestrate dbt runs, batch ingestion jobs, maintenance operations (Iceberg compaction, snapshot expiration, orphan file cleanup), and data product refresh schedules
- 3+ years of hands-on data engineering experience - Strong proficiency in Python and SQL. PySpark and PyFlink familiarity strongly preferred. Some Java or Kotlin exposure useful for FHIR-Repository interceptor work (one or two engineers on the team will lead this; the rest contribute as needed) - Hands-on experience with Google Cloud Platform — Cloud Storage, Dataproc, Cloud Composer, Cloud Run or GKE for container workloads, Secret Manager, IAM. Experience with the Dataproc Flink optional component is a strong plus
- Atlan experience for governance and lineage integration - Java or Kotlin proficiency for FHIR-Repository interceptor and HAPI FHIR work - Apache Flink production operational experience including stateful jobs, exactly-once semantics, and savepoint/checkpoint management
Key Responsibilities - Build and maintain the ingestion pipelines — Apache Flink streaming jobs on Dataproc for HL7v2 and FHIR feeds, PySpark batch jobs for CCDA and CSV bulk loads. Implement against the CDM_ingest_mapping shared Python library that defines SourceSpec instances per source × format combination - Implement source-format parsers (HL7v2, CCDA, CSV, FHIR) as Python classes per the parser component spec. Write fixture-driven tests covering both well-formed inputs and DLQ-routing scenarios
- Build and maintain the ingestion pipelines — Apache Flink streaming jobs on Dataproc for HL7v2 and FHIR feeds, PySpark batch jobs for CCDA and CSV bulk loads. Implement against the CDM_ingest_mapping shared Python library that defines SourceSpec instances per source × format combination - Implement source-format parsers (HL7v2, CCDA, CSV, FHIR) as Python classes per the parser component spec. Write fixture-driven tests covering both well-formed inputs and DLQ-routing scenarios - Implement and maintain the synchronous Informatica MDM call pattern in the ingestion path — batched calls, timeout handling, circuit breaker behavior, DLQ routing for MDM failures. Implement the asynchronous MDM event consumer that applies ECI changes to CDM
Role review : The Data Engineer is the implementation backbone of the platform. You will build and operate the ingestion pipelines, the dbt transformation layer, the FHIR serialization pipelines, the FHIR-Repository integration components, and the data products that depend on them. You will work within a spec-driven development framework — every component begins with an approved specification, and your work is generated, refined, and verified against that specification. This is a hands-on role. You will write Python, PyFlink, PySpark, dbt SQL, and supporting Java for FHIR-Repository interceptor work, with Code agents as a coding assistant under your direction. Code review, test authorship, and operational ownership of the components you build are part of the role. The platform handles regulated healthcare data at high volume, and operational discipline matters.
- Implement and maintain the synchronous Informatica MDM call pattern in the ingestion path — batched calls, timeout handling, circuit breaker behavior, DLQ routing for MDM failures. Implement the asynchronous MDM event consumer that applies ECI changes to CDM - Build the dbt transformation layer end to end — staging models per source, intermediate models that union and resolve REFs, CDM target models (DIM/FACT/BRIDGE/REF) that apply SCD2 via shared macros, and data product models. Write the dbt YAML schemas, tests, and documentation that accompany every model - Implement and maintain the shared dbt macro library — hash_key, scd2_merge, attribute_hash, restate_merge, audit_columns. Macros are the most-reused code; their correctness is non-negotiable and they require golden tests
- Build the dbt transformation layer end to end — staging models per source, intermediate models that union and resolve REFs, CDM target models (DIM/FACT/BRIDGE/REF) that apply SCD2 via shared macros, and data product models. Write the dbt YAML schemas, tests, and documentation that accompany every model - Implement and maintain the shared dbt macro library — hash_key, scd2_merge, attribute_hash, restate_merge, audit_columns. Macros are the most-reused code; their correctness is non-negotiable and they require golden tests - Build the FHIR serialization layer — flat FHIR Iceberg tables (one per resource type) materialized via dbt, the PySpark bundling pipeline that produces FHIR Bundles for Kafka publication, and the FHIR validator integration that gates publication on US Core 6.1 conformance
- Implement and maintain the shared dbt macro library — hash_key, scd2_merge, attribute_hash, restate_merge, audit_columns. Macros are the most-reused code; their correctness is non-negotiable and they require golden tests - Build the FHIR serialization layer — flat FHIR Iceberg tables (one per resource type) materialized via dbt, the PySpark bundling pipeline that produces FHIR Bundles for Kafka publication, and the FHIR validator integration that gates publication on US Core 6.1 conformance - Build and maintain FHIR-Repository integration components — the Java/Kotlin egress interceptor that captures client-originated FHIR changes, the Flink loopback consumer that merges those changes into CDM, the bundle consumer that ingests CDM-originated bundles into FHIR-Repository. Implement origin-tag-based loop prevention
- Build and maintain FHIR-Repository integration components — the Java/Kotlin egress interceptor that captures client-originated FHIR changes, the Flink loopback consumer that merges those changes into CDM, the bundle consumer that ingests CDM-originated bundles into FHIR-Repository. Implement origin-tag-based loop prevention - Implement Cloud Composer DAGs to orchestrate dbt runs, batch ingestion jobs, maintenance operations (Iceberg compaction, snapshot expiration, orphan file cleanup), and data product refresh schedules - Work within the spec-driven development framework — draft unit specs for new components, work with peers on spec review, generate implementation and tests using Code agents with the spec as primary context, iterate until tests pass, and submit code review packages that include the spec, tests, and implementation together
- Work within the spec-driven development framework — draft unit specs for new components, work with peers on spec review, generate implementation and tests using Code agents with the spec as primary context, iterate until tests pass, and submit code review packages that include the spec, tests, and implementation together - Implement and monitor data quality checks at every layer — DBT tests for staging and CDM, FHIR validator output for serialization, Iceberg metadata observations for storage health, freshness monitors at the source-to-CDM boundary - Participate in code reviews, on-call rotations, and incident response. The platform serves regulated healthcare workloads; operational stability is part of the engineering responsibility, not a separate function
- Participate in code reviews, on-call rotations, and incident response. The platform serves regulated healthcare workloads; operational stability is part of the engineering responsibility, not a separate function - Optimize pipeline performance — Flink TaskManager sizing, Iceberg compaction tuning, dbt incremental strategy selection, Starburst cluster scaling decisions. Profile production performance and propose changes when SLOs are at risk - Document the components you build through the SDD framework — every code file references its spec; every spec change is reviewed; every acceptance criterion has a corresponding test
- Hands-on experience with Google Cloud Platform — Cloud Storage, Dataproc, Cloud Composer, Cloud Run or GKE for container workloads, Secret Manager, IAM. Experience with the Dataproc Flink optional component is a strong plus - Production experience with dbt — incremental materialization strategies, custom macros, tests, sources, sequencing, and project organization for large model graphs. dbt-trino adapter experience is a plus - Production experience with Apache Iceberg — table creation, partitioning, compaction, snapshot operations, schema evolution. Familiarity with reading and writing Iceberg from multiple engines (Spark, Flink, Trino) is valuable
- Production experience with dbt — incremental materialization strategies, custom macros, tests, sources, sequencing, and project organization for large model graphs. dbt-trino adapter experience is a plus - Production experience with Apache Iceberg — table creation, partitioning, compaction, snapshot operations, schema evolution. Familiarity with reading and writing Iceberg from multiple engines (Spark, Flink, Trino) is valuable - Experience with Apache Kafka — producers, consumers, partitioning, consumer-group semantics, retention and compaction, and integration with stream processors. Confluent Cloud experience preferred
- Experience contributing to or operating spec-driven or contract-first development workflows - Experience with Iceberg's Polaris, Nessie, or Snowflake Open Catalog as alternatives to BigLake (useful for portability discussions) If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us!
- Production experience with Apache Iceberg — table creation, partitioning, compaction, snapshot operations, schema evolution. Familiarity with reading and writing Iceberg from multiple engines (Spark, Flink, Trino) is valuable - Experience with Apache Kafka — producers, consumers, partitioning, consumer-group semantics, retention and compaction, and integration with stream processors. Confluent Cloud experience preferred - Experience with streaming data processing — Apache Flink in production preferred; Apache Spark Structured Streaming acceptable as adjacent experience
- Experience with Apache Kafka — producers, consumers, partitioning, consumer-group semantics, retention and compaction, and integration with stream processors. Confluent Cloud experience preferred - Experience with streaming data processing — Apache Flink in production preferred; Apache Spark Structured Streaming acceptable as adjacent experience - Familiarity with healthcare data standards — at minimum, HL7v2 message structure and FHIR R4 resource shapes. Hands-on parsing experience for one or both is preferred
- Strong proficiency in Python and SQL. PySpark and PyFlink familiarity strongly preferred. Some Java or Kotlin exposure useful for FHIR-Repository interceptor work (one or two engineers on the team will lead this; the rest contribute as needed) - Hands-on experience with Google Cloud Platform — Cloud Storage, Dataproc, Cloud Composer, Cloud Run or GKE for container workloads, Secret Manager, IAM. Experience with the Dataproc Flink optional component is a strong plus - Production experience with dbt — incremental materialization strategies, custom macros, tests, sources, sequencing, and project organization for large model graphs. dbt-trino adapter experience is a plus
Your next step
Complete your application on quantiphi.wd1.myworkdayjobs.com. The employer’s form will show what is required.
Applying through Workday
- You create an account with each employer before you can upload anything. Every Workday employer has its own account, so one sign-up doesn’t carry over.
- “Autofill with Resume” comes right after sign-in, before the My Information and My Experience steps. Check every field it fills in those steps, since that’s what recruiters see.
- Your application status lives in that employer’s Candidate Home, under the same sign-in.
Already applied? Track this application
Has Quantiphi posted this job before?
Yes. We’ve followed 6 other Senior Data Engineer jobs there since May 2026. 6 came down, usually after about 7 days.
Large employers often post one title for several teams, so these can be separate openings.
About the role
We file this job under Data Engineer, by the work the posting describes.
Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.
Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.
An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.
US government figures for database architects
- Median yearly pay, May 2025
- $139,500
- Expected change, 2025 to 2035
- 9.4% more jobsagainst 3.5% for all jobs
The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.
Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1243.
Job description
While technology is the heart of our business, a global and diverse culture is the heart of our success. We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth.
If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi!
Exp Range : 4 - 8 Years
Location : Mumbai , Bangalore, Trivandrum
Role review :
The Data Engineer is the implementation backbone of the platform. You will build and operate the ingestion pipelines, the dbt transformation layer, the FHIR serialization pipelines, the FHIR-Repository integration components, and the data products that depend on them. You will work within a spec-driven development framework — every component begins with an approved specification, and your work is generated, refined, and verified against that specification.
This is a hands-on role. You will write Python, PyFlink, PySpark, dbt SQL, and supporting Java for FHIR-Repository interceptor work, with Code agents as a coding assistant under your direction. Code review, test authorship, and operational ownership of the components you build are part of the role. The platform handles regulated healthcare data at high volume, and operational discipline matters.
Key Responsibilities
Build and maintain the ingestion pipelines — Apache Flink streaming jobs on Dataproc for HL7v2 and FHIR feeds, PySpark batch jobs for CCDA and CSV bulk loads. Implement against the CDM_ingest_mapping shared Python library that defines SourceSpec instances per source × format combination
Implement source-format parsers (HL7v2, CCDA, CSV, FHIR) as Python classes per the parser component spec. Write fixture-driven tests covering both well-formed inputs and DLQ-routing scenarios
Implement and maintain the synchronous Informatica MDM call pattern in the ingestion path — batched calls, timeout handling, circuit breaker behavior, DLQ routing for MDM failures. Implement the asynchronous MDM event consumer that applies ECI changes to CDM
Build the dbt transformation layer end to end — staging models per source, intermediate models that union and resolve REFs, CDM target models (DIM/FACT/BRIDGE/REF) that apply SCD2 via shared macros, and data product models. Write the dbt YAML schemas, tests, and documentation that accompany every model
Implement and maintain the shared dbt macro library — hash_key, scd2_merge, attribute_hash, restate_merge, audit_columns. Macros are the most-reused code; their correctness is non-negotiable and they require golden tests
Build the FHIR serialization layer — flat FHIR Iceberg tables (one per resource type) materialized via dbt, the PySpark bundling pipeline that produces FHIR Bundles for Kafka publication, and the FHIR validator integration that gates publication on US Core 6.1 conformance
Build and maintain FHIR-Repository integration components — the Java/Kotlin egress interceptor that captures client-originated FHIR changes, the Flink loopback consumer that merges those changes into CDM, the bundle consumer that ingests CDM-originated bundles into FHIR-Repository. Implement origin-tag-based loop prevention
Implement Cloud Composer DAGs to orchestrate dbt runs, batch ingestion jobs, maintenance operations (Iceberg compaction, snapshot expiration, orphan file cleanup), and data product refresh schedules
Work within the spec-driven development framework — draft unit specs for new components, work with peers on spec review, generate implementation and tests using Code agents with the spec as primary context, iterate until tests pass, and submit code review packages that include the spec, tests, and implementation together
Implement and monitor data quality checks at every layer — DBT tests for staging and CDM, FHIR validator output for serialization, Iceberg metadata observations for storage health, freshness monitors at the source-to-CDM boundary
Participate in code reviews, on-call rotations, and incident response. The platform serves regulated healthcare workloads; operational stability is part of the engineering responsibility, not a separate function
Optimize pipeline performance — Flink TaskManager sizing, Iceberg compaction tuning, dbt incremental strategy selection, Starburst cluster scaling decisions. Profile production performance and propose changes when SLOs are at risk
Document the components you build through the SDD framework — every code file references its spec; every spec change is reviewed; every acceptance criterion has a corresponding test
Required Skills and Qualifications
Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field
3+ years of hands-on data engineering experience
Strong proficiency in Python and SQL. PySpark and PyFlink familiarity strongly preferred. Some Java or Kotlin exposure useful for FHIR-Repository interceptor work (one or two engineers on the team will lead this; the rest contribute as needed)
Hands-on experience with Google Cloud Platform — Cloud Storage, Dataproc, Cloud Composer, Cloud Run or GKE for container workloads, Secret Manager, IAM. Experience with the Dataproc Flink optional component is a strong plus
Production experience with dbt — incremental materialization strategies, custom macros, tests, sources, sequencing, and project organization for large model graphs. dbt-trino adapter experience is a plus
Production experience with Apache Iceberg — table creation, partitioning, compaction, snapshot operations, schema evolution. Familiarity with reading and writing Iceberg from multiple engines (Spark, Flink, Trino) is valuable
Experience with Apache Kafka — producers, consumers, partitioning, consumer-group semantics, retention and compaction, and integration with stream processors. Confluent Cloud experience preferred
Experience with streaming data processing — Apache Flink in production preferred; Apache Spark Structured Streaming acceptable as adjacent experience
Familiarity with healthcare data standards — at minimum, HL7v2 message structure and FHIR R4 resource shapes. Hands-on parsing experience for one or both is preferred
Experience with version control (Git), branch-based development workflows, pull request reviews, and CI/CD pipelines (GitHub Actions, GitLab CI, or Cloud Build)
Comfortable working with AI coding assistants (Code agents, Cursor, Copilot) as collaborators. The team uses Code agents as part of the spec-driven development workflow; ability to write effective prompts, review generated code critically, and iterate constructively is a meaningful part of the role
Strong problem-solving skills, debugging discipline, attention to detail, and ability to operate in an agile team environment
Nice-to-Have Skills
FHIR-Repository or HAPI FHIR experience — interceptor authorship, MDM module configuration, FHIR API customization
Informatica MDM experience
Atlan experience for governance and lineage integration
Java or Kotlin proficiency for FHIR-Repository interceptor and HAPI FHIR work
Apache Flink production operational experience including stateful jobs, exactly-once semantics, and savepoint/checkpoint management
Experience with FHIR profile validation tools (HL7 FHIR validator, Inferno, or HAPI's validation modules)
Experience contributing to or operating spec-driven or contract-first development workflows
Experience with Iceberg's Polaris, Nessie, or Snowflake Open Catalog as alternatives to BigLake (useful for portability discussions)
If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us!
Source details
Source notes
Source excerptsSelected passages from the saved posting. Check the full description for conditions and exceptions.
- Pay
No pay amount identified in the saved description.
- Location & working pattern
IN MH Mumbai Eureka
Working pattern and location restrictions need checking in the full posting.
- Work authorization
No clear work-authorization passage found. Eligibility is unconfirmed.