Time since we first saw it
Day 57
Up longer than most. Similar jobs usually come down between day 6 and day 56.
From 14,771 similar postings that came down. Coming down is not always a hire.
Cochin, Kakkanad, India
Live, first seen by us 57 days agoEmployment type in the employer’s posting: ContractorFrom the employer’s posting
Design pipelines that meet their SLA as data volumes grow
Diagnose failures and performance problems in large distributed jobs
Set data layout and storage so jobs read only the data they need
Build reusable, configuration-driven components for the team
Review team code and set coding standards
Design pipelines that meet their SLA and continue to do so as data volumes grow.
Diagnose failures and performance problems in large distributed jobs using logs and metrics.
Set the data layout and storage strategy so jobs read only the data they need.
Build reusable, configuration-driven components that cut development effort across the team.
Review the team's code and agree the standards the team works to.
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
Day 57
Up longer than most. Similar jobs usually come down between day 6 and day 56.
From 14,771 similar postings that came down. Coming down is not always a hire.
7 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordAdd your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
- Experience running Spark on self-managed clusters, not only on managed platforms. - Strong Python, with experience building reusable, configuration-driven frameworks rather than one-off scripts. - Able to lead a small team through code reviews, mentoring and unblocking, and hold a delivery date.
- Hands-on experience with time-series data at scale: event-time semantics, sliding and tumbling windows, and as-of or point-in-time joins across billions of rows. - Strong Delta Lake or Iceberg experience, covering partitioning, compaction, sort keys, data pruning and versioning. - Able to read Spark logs, the Spark UI and the history server and work out why a job is slow or failing.
- KDB/q exposure, or experience migrating off a vendor surveillance platform. - Experience moving Spark workloads to object storage such as MinIO or S3, and running Spark on Kubernetes. - CI/CD, automated testing and observability for data pipelines.
Must Have : - 10+ years in data engineering, with at least 5 years hands-on PySpark on large-volume workloads, and still writing code daily. - Deep understanding of Spark internals: execution model, stages and tasks, shuffle, partitioning, broadcast versus sort-merge joins, adaptive query execution, and memory and spill behaviour.
Complete your application on prevalent.bamboohr.com. The employer’s form will show what is required.
Already applied? Track this application
Role Purpose :
As a Lead Data Engineer at Prevalent AI, you will lead the design and delivery of advanced data engineering projects for our clients. You will write code, review the team's code, and stay accountable for how the solution performs in production.
The work involves large data volumes, complex joins and transformations, and tight processing windows. Clients usually come to us when their pipelines are slow, expensive or difficult to change. Working out why, and then fixing it, is a large part of the role.
This role suits engineers who know Spark internals well enough to read the logs and explain what is going wrong, who ask the right questions about the data before they write code, and who can run a small team to a delivery date.
Key Accountabilities :
Skills & Experience :
Must Have :
Good to Have
New Data Engineer postings at the employers we follow are up 13% over the last 6 weeks compared with the 6 before.
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
We file this job under Data Engineer, by the work the posting describes.
Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.
Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.
An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.
The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.
Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1243.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
No pay amount identified in the saved description.
Cochin, Kakkanad, India
Working pattern and location restrictions need checking in the full posting.
No clear work-authorization passage found. Eligibility is unconfirmed.