Back to jobs

Lead Data Engineer

Cochin, Kakkanad, India

Live, first seen by us 57 days ago
Pay
Pay not listed
Work setting
Not stated
Employment
Contract — employment source
Employment type in the employer’s posting: Contractor
From the employer’s posting

What you’ll work on

Full posting
  • Design pipelines that meet their SLA as data volumes grow

  • Diagnose failures and performance problems in large distributed jobs

  • Set data layout and storage so jobs read only the data they need

  • Build reusable, configuration-driven components for the team

  • Review team code and set coding standards

From the employer’s posting
Design pipelines that meet their SLA and continue to do so as data volumes grow.
Diagnose failures and performance problems in large distributed jobs using logs and metrics.
Set the data layout and storage strategy so jobs read only the data they need.
Build reusable, configuration-driven components that cut development effort across the team.
Review the team's code and agree the standards the team works to.
Apply at Prevalent AI

Posting record

Report an error
Status in our records
Live
First seen by us
Sep 25, 202657 days ago
Recorded sightings
42
Last seen by us
Nov 22, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 57

Up longer than most. Similar jobs usually come down between day 6 and day 56.

From 14,771 similar postings that came down. Coming down is not always a hire.

Prevalent AI is hiring

7 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • Delta Lake
  • Kubernetes
  • S3
  • PySpark
Where the posting names them
- Experience running Spark on self-managed clusters, not only on managed platforms. - Strong Python, with experience building reusable, configuration-driven frameworks rather than one-off scripts. - Able to lead a small team through code reviews, mentoring and unblocking, and hold a delivery date.
- Hands-on experience with time-series data at scale: event-time semantics, sliding and tumbling windows, and as-of or point-in-time joins across billions of rows. - Strong Delta Lake or Iceberg experience, covering partitioning, compaction, sort keys, data pruning and versioning. - Able to read Spark logs, the Spark UI and the history server and work out why a job is slow or failing.
- KDB/q exposure, or experience migrating off a vendor surveillance platform. - Experience moving Spark workloads to object storage such as MinIO or S3, and running Spark on Kubernetes. - CI/CD, automated testing and observability for data pipelines.
Must Have : - 10+ years in data engineering, with at least 5 years hands-on PySpark on large-volume workloads, and still writing code daily. - Deep understanding of Spark internals: execution model, stages and tasks, shuffle, partitioning, broadcast versus sort-merge joins, adaptive query execution, and memory and spill behaviour.

Your next step

Complete your application on prevalent.bamboohr.com. The employer’s form will show what is required.

Applying through BambooHR

  • No account needed. The form takes images too, but an image of a resume has no text for anyone to search: upload a PDF or Word file.

Already applied? Track this application

Job description

View original posting ↗

Role Purpose :


As a Lead Data Engineer at Prevalent AI, you will lead the design and delivery of advanced data engineering projects for our clients. You will write code, review the team's code, and stay accountable for how the solution performs in production. 

The work involves large data volumes, complex joins and transformations, and tight processing windows. Clients usually come to us when their pipelines are slow, expensive or difficult to change. Working out why, and then fixing it, is a large part of the role. 

This role suits engineers who know Spark internals well enough to read the logs and explain what is going wrong, who ask the right questions about the data before they write code, and who can run a small team to a delivery date. 


Key Accountabilities :

  • Lead the design and delivery of client data engineering projects, from understanding the requirement through to production. 
  • Design pipelines that meet their SLA and continue to do so as data volumes grow. 
  • Diagnose failures and performance problems in large distributed jobs using logs and metrics. 
  • Set the data layout and storage strategy so jobs read only the data they need. 
  • Build reusable, configuration-driven components that cut development effort across the team. 
  • Review the team's code and agree the standards the team works to. 
  • Work with client teams to pin down the requirement and understand the data before development starts. 
  • Contribute to the platform work around delivery, including object storage, containerisation and CI/CD. 
  • Mentor developers, and document design decisions, tuning approaches and operational runbooks. 

 

Skills & Experience :


Must Have :


  • 10+ years in data engineering, with at least 5 years hands-on PySpark on large-volume workloads, and still writing code daily. 
  • Deep understanding of Spark internals: execution model, stages and tasks, shuffle, partitioning, broadcast versus sort-merge joins, adaptive query execution, and memory and spill behaviour. 
  • Experience tuning long-running jobs down to an SLA and handling data skew, with specific examples of how a job was diagnosed and what changed. 
  • Hands-on experience with time-series data at scale: event-time semantics, sliding and tumbling windows, and as-of or point-in-time joins across billions of rows. 
  • Strong Delta Lake or Iceberg experience, covering partitioning, compaction, sort keys, data pruning and versioning. 
  • Able to read Spark logs, the Spark UI and the history server and work out why a job is slow or failing. 
  • Experience running Spark on self-managed clusters, not only on managed platforms. 
  • Strong Python, with experience building reusable, configuration-driven frameworks rather than one-off scripts. 
  • Able to lead a small team through code reviews, mentoring and unblocking, and hold a delivery date. 

 

Good to Have 

  • Trade surveillance, capital markets or FX experience, and familiarity with trade, order and market data. 
  • KDB/q exposure, or experience migrating off a vendor surveillance platform. 
  • Experience moving Spark workloads to object storage such as MinIO or S3, and running Spark on Kubernetes. 
  • CI/CD, automated testing and observability for data pipelines. 
  • Exposure to regulatory or compliance-driven delivery in a banking environment. 

 

Demand for Data Engineer roles

New Data Engineer postings at the employers we follow are up 13% over the last 6 weeks compared with the 6 before.

Weekly new Data Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 845, the most 1,344, and the latest week had 1,007.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under Data Engineer, by the work the posting describes.

Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.

Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.

An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.

US government figures for database architects

Median yearly pay, May 2025
$139,500
Expected change, 2025 to 2035
9.4% more jobsagainst 3.5% for all jobs

The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1243.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Cochin, Kakkanad, India

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.