Back to jobs

Lead Data Engineer

Bengaluru, KA, IN; Bengaluru, KA, IN

Live, first seen by us 21 days ago
Pay
Pay not listed
Work setting
Not stated
Employment
Full-time — employment source
Employment type in the employer’s posting: FULL_TIME
From the employer’s posting

What you’ll work on

Full posting
  • Design and maintain Apache Iceberg tables and their partitioning, compaction and snapshots

  • Deploy, operate and performance-tune open-source data warehouse clusters

  • Build robust batch and streaming ETL/ELT pipelines with data quality guarantees

  • Run KEDA-based event-driven autoscaling on Kubernetes for data workloads

  • Provision cloud and Kubernetes infrastructure as code with Terraform, Helm and GitOps

From the employer’s posting
Design and maintain Apache Iceberg tables, including schema evolution, partitioning strategies, snapshot management, compaction, and metadata optimization.
Deploy, operate, and performance-tune open-source datawarehouse clusters, covering FE/BE sizing, materialized views, query profiling, caching, and resource isolation.
Develop robust batch and streaming pipelines with strong guarantees around data quality, idempotency, backfills, and observability.
Design and implement KEDA-based scaling for data workloads, triggered by queue depth, Kafka lag, or custom metrics
Provision and maintain cloud and Kubernetes infrastructure using tools like Terraform, Helm, and GitOps workflows (ArgoCD/Flux).
Apply at Eightfold

Posting record

Report an error
Status in our records
Live
First seen by us
Oct 27, 202621 days ago
Recorded sightings
21
Last seen by us
Nov 17, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 21

Similar jobs usually come down between day 6 and day 56.

At Eightfold, jobs usually come down by about day 68.

From 13,974 similar postings that came down, and 9 of Eightfold’s. Coming down is not always a hire.

Eightfold is hiring

3 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Its jobs tend to stay up a long time
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • AWS
  • Azure
  • ClickHouse
  • Databricks
  • Grafana
  • Iceberg
  • Kafka
  • Kubernetes
  • Redshift
  • Spark
  • Terraform
  • Trino
Where the posting names them
Today we run on Redshift and Databricks, and we’re migrating to StarRocks across 8+ regions distributed globally on AWS, Azure, and Oracle Cloud.
Experience operating production-grade, multi-region cloud platforms across AWS, Azure, or OCI, with a focus on scalability, reliability, performance, and cost efficiency.
Our platform is built from the ground up, operating at scale across Azure and AWS, deployed in multiple regions globally, including IL4-compliant environments for US Government, supporting users in 100+ countries and 30+ languages.
Experience operating and performance-tuning analytical query engines such as StarRocks, ClickHouse, Trino, or similar distributed data warehouses.
Define SLOs, build monitoring and alerting (Prometheus, Grafana, or similar), participate in on-call, and lead incident retrospectives.
Design and maintain Apache Iceberg tables, including schema evolution, partitioning strategies, snapshot management, compaction, and metadata optimization.
What We Need: 5+ years of experience designing, building, and operating large-scale data platforms, lakehouse architectures, and distributed analytical systems Hands-on expertise with Apache Iceberg / Hudi and Parquet, including partitioning, schema evolution, compaction, snapshot management, and storage optimization.
Design and implement KEDA-based scaling for data workloads, triggered by queue depth, Kafka lag, or custom metrics, so you balance cost against latency.
Run event-driven autoscaling on Kubernetes.
Provision and maintain cloud and Kubernetes infrastructure using tools like Terraform, Helm, and GitOps workflows (ArgoCD/Flux).
Proficiency in building reliable batch and streaming pipelines using Apache Spark and Flink, with strong fundamentals in data quality, idempotency, fault tolerance, and observability.

Your next step

Complete your application on app.eightfold.ai. The employer’s form will show what is required.

Applying through Eightfold

  • You sign in, with Microsoft, LinkedIn, Google, Facebook or a new account, before the application opens.

Already applied? Track this application

Job description

View original posting ↗

About the team: Eightfold offers a suite of AI-powered products, including Talent Acquisition, Talent Management, Workforce Exchange, and AI Interviewer. The Analytics team builds the customer-facing solutions that show how Eightfold delivers value across these products. Our work helps customers’ leadership teams make strategic workforce decisions and helps their recruiting teams operate more effectively day to day. Our analytics portfolio includes: Native reporting platform : a reporting experience built directly into the Eightfold product. Agentic Dashboard Authoring : AI-assisted dashboard creation, so users can go from question to insight without building reports by hand. Conversational Analytics : natural-language access to data, letting users ask questions and get answers in plain English. Secure Talent Lake connectivity : secure pipelines that let customers ETL their Eightfold data into their own data warehouses. We’re in the middle of a major revamp of our analytics stack. Today we run on Redshift and Databricks, and we’re migrating to StarRocks across 8+ regions distributed globally on AWS, Azure, and Oracle Cloud. As Lead Platform / Data Engineer, you’ll play a central role in shaping this migration and the multi-cloud platform that comes out of it. What You'll Do: Architect and manage the lakehouse.  Design and maintain Apache Iceberg tables, including schema evolution, partitioning strategies, snapshot management, compaction, and metadata optimization. Own the analytical query layer.  Deploy, operate, and performance-tune open-source datawarehouse clusters, covering FE/BE sizing, materialized views, query profiling, caching, and resource isolation. Optimize storage formats.  Make deliberate choices about Parquet file layout, including row group sizing, compression codecs, encoding, and predicate pushdown, so data stays cheap to store and fast to query. Build and maintain ETL/ELT pipelines.  Develop robust batch and streaming pipelines with strong guarantees around data quality, idempotency, backfills, and observability. Run event-driven autoscaling on Kubernetes.  Design and implement KEDA-based scaling for data workloads, triggered by queue depth, Kafka lag, or custom metrics, so you balance cost against latency. Manage infrastructure as code.  Provision and maintain cloud and Kubernetes infrastructure using tools like Terraform, Helm, and GitOps workflows (ArgoCD/Flux). Tune self-hosted open-source warehouses.  Diagnose bottlenecks across CPU, memory, I/O, and network. Tune configurations, run benchmarks, and drive measurable gains in query latency and cost efficiency. Champion reliability.  Define SLOs, build monitoring and alerting (Prometheus, Grafana, or similar), participate in on-call, and lead incident retrospectives. Mentor and influence.  Review designs and code, mentor engineers, and help set technical direction for the data platform. What We Need: 5+ years of experience designing, building, and operating large-scale data platforms, lakehouse architectures, and distributed analytical systems Hands-on expertise with Apache Iceberg / Hudi and Parquet, including partitioning, schema evolution, compaction, snapshot management, and storage optimization. Experience operating and performance-tuning analytical query engines such as StarRocks, ClickHouse, Trino, or similar distributed data warehouses. Proficiency in building reliable batch and streaming pipelines using Apache Spark and Flink, with strong fundamentals in data quality, idempotency, fault tolerance, and observability. Experience operating production-grade, multi-region cloud platforms across AWS, Azure, or OCI, with a focus on scalability, reliability, performance, and cost efficiency. Prior experience in mentoring engineers and leading projects while driving platform initiatives end-to-end. About Eightfold: Eightfold is a global leader in an AI-native enterprise talent platform, trusted by the world’s largest & most respected Fortune 500 organizations. Our platform is built from the ground up, operating at scale across Azure and AWS, deployed in multiple regions globally, including IL4-compliant environments for US Government, supporting users in 100+ countries and 30+ languages. Today, Eightfold is at the forefront of agentic AI, delivering intelligent agents that actively drive outcomes across hiring and talent workflows, while much of the industry is still experimenting with prototypes. Backed by over $410M in funding and valued at $2B+, we are defining the next era of agentic talent systems. What sets Eightfold apart is not just the technology & our mission, but the team behind it. We are a deeply technical, execution-driven organization that values ownership, collaboration, and high standards. Our engineers, product leaders, and go-to-market teams work closely together in person and across functions to build systems that scale in the real world. If you’re excited to work on hard problems, move with urgency, raise the bar every day, and help build agentic systems that transform how the world works, Eightfold is the place to do it. Our customers Press Founders: Varun Kacholia  ( https://www.linkedin.com/in/varunkacholia/  ) &  Ashutosh Garg  ( https://www.linkedin.com/in/ashutoshgarg893/ ) Engineering Culture at Eightfold: We are a team of self-starters who excel in their fields. We believe in giving you responsibility, not a task. We want you to have ownership and pride in your work and see your work's positive impact on your colleagues, our customers, and the world. We believe in providing transparency and support so you can do the best work of your career.

Demand for Data Engineer roles

New Data Engineer postings at the employers we follow are up 14% over the last 6 weeks compared with the 6 before.

Weekly new Data Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 861, the most 1,353, and the latest week had 1,022.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under Data Engineer, by the work the posting describes.

Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.

Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.

An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.

US government figures for database architects

Median yearly pay, May 2025
$139,500
Expected change, 2025 to 2035
9.4% more jobsagainst 3.5% for all jobs

The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1243.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Bengaluru, KA, IN; Bengaluru, KA, IN

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.