Back to jobs

Lead Data Engineer

Pay
Pay not listed
Work setting
Hybrid — work setting source
Listed location: Hybrid in Bangalore, India
Read the full posting
Employment
Not stated

What you’ll work on

Full posting

We are seeking a Lead Data Engineer with deep expertise in distributed systems, real-time data processing, and modern data platforms to join our data team.

  • Design and operate distributed data systems using Apache Spark and Flink to support high-volume data processing.

  • Build and evolve LakeHouse architectures using Apache Iceberg, Apache Hudi, and Delta Lake with columnar file formats like Parquet and ORC.

  • Design and implement distributed query and federation capabilities using Trino, enabling unified access across data lakes, warehouses, and heterogeneous data sources.

From the employer’s posting
We are seeking a Lead Data Engineer with deep expertise in distributed systems, real-time data processing, and modern data platforms to join our data team. This is a senior technical leadership role where you’ll help shape and stabilize our data architecture, drive business-critical initiatives, and mentor a high-performing team. You’ll be a senior individual contributor at the heart of the data organisation — owning complex platform decisions, collaborating cross-functionally with AI, product, and go-to-market teams, and shipping data-driven features that directly drive revenue. This is a greenfield opportunity to shape the next phase of data at HackerRank.
Architect and implement streaming-first data pipelines using Apache Kafka and real-time processing frameworks. Design and operate distributed data systems using Apache Spark and Flink to support high-volume data processing. Build and evolve LakeHouse architectures using Apache Iceberg, Apache Hudi, and Delta Lake with columnar file formats like Parquet and ORC.
Design and operate distributed data systems using Apache Spark and Flink to support high-volume data processing. Build and evolve LakeHouse architectures using Apache Iceberg, Apache Hudi, and Delta Lake with columnar file formats like Parquet and ORC. Design and implement distributed query and federation capabilities using Trino, enabling unified access across data lakes, warehouses, and heterogeneous data sources.
Build and evolve LakeHouse architectures using Apache Iceberg, Apache Hudi, and Delta Lake with columnar file formats like Parquet and ORC. Design and implement distributed query and federation capabilities using Trino, enabling unified access across data lakes, warehouses, and heterogeneous data sources. Define and manage metadata and catalog architectures using Hive Metastore and other table catalog solutions.

What you’ll bring

All qualifications

Core experience

  • 8+ years of hands-on experience in data engineering, distributed systems, streaming, and cloud-native architectures.
  • Proven ability to deliver both data ingestion (inbound) and data serving (outbound) capabilities as reusable platform products.
  • Deep expertise in Kafka, Spark, Flink, and building scalable data pipelines using modern LakeHouse architectures (Delta Lake, Hudi, Iceberg, Parquet).
  • Prior experience building RAG (Retrieval-Augmented Generation) pipelines using enterprise data sources.
  • Experience with distributed query engines and metadata management using Trino, Hive Metastore, and analytical serving platforms like StarRocks, ClickHouse, Druid, or Pinot.
  • Experience enabling Natural Language Querying (NLQ) and conversational analytics over data warehouses and Lakehouse platforms.
Qualification wording
8+ years of hands-on experience in data engineering, distributed systems, streaming, and cloud-native architectures.
Proven ability to deliver both data ingestion (inbound) and data serving (outbound) capabilities as reusable platform products.
Deep expertise in Kafka, Spark, Flink, and building scalable data pipelines using modern LakeHouse architectures (Delta Lake, Hudi, Iceberg, Parquet).
Prior experience building RAG (Retrieval-Augmented Generation) pipelines using enterprise data sources.
Experience with distributed query engines and metadata management using Trino, Hive Metastore, and analytical serving platforms like StarRocks, ClickHouse, Druid, or Pinot.
Experience enabling Natural Language Querying (NLQ) and conversational analytics over data warehouses and Lakehouse platforms.
Apply at HackerRank Careers

Posting record

Report an error

Open 5 monthsWe first saw this posting 165 days ago and it is still up. Roles open this long are often slow to fill or kept open to collect applications.

Status in our records
Live
First seen by us
Jun 9, 2026165 days ago
Recorded sightings
69
Last seen by us
Nov 21, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 165

Up longer than most. Similar jobs usually come down between day 6 and day 56.

At HackerRank Careers, jobs usually come down by about day 97.

From 14,771 similar postings that came down, and 6 of HackerRank Careers’s. Coming down is not always a hire.

HackerRank Careers is hiring

6 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Its jobs tend to stay up a long time
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • AWS
  • Azure
  • BigQuery
  • ClickHouse
  • Delta Lake
  • GCP
  • Grafana
  • Hive
  • Iceberg
  • Kafka
  • Looker
  • Metabase
  • MLflow
  • Redshift
  • SageMaker
  • Snowflake
  • Spark
  • Trino
  • Airflow
Where the posting names them
- Strong understanding of data governance and security, including Apache Ranger, access controls, auditing, data quality, and lineage. - Proficiency in Airflow, cloud platforms (AWS/GCP/Azure), observability (Prometheus, Grafana, Loki), and analytical data modeling. - Hands-on experience with AI Coding Agents (Claude Code, Codex, Cursor), agentic workflows, and applying emerging AI capabilities to improve engineering productivity and operational efficiency.
- Define and manage metadata and catalog architectures using Hive Metastore and other table catalog solutions. - Build scalable data-serving and analytical layers using StarRocks and cloud data warehouses such as BigQuery, Redshift, and Snowflake. - Design and maintain robust data models, data marts, and data cubes that power business intelligence, reporting, and self-serve analytics.
- Deep expertise in Kafka, Spark, Flink, and building scalable data pipelines using modern LakeHouse architectures (Delta Lake, Hudi, Iceberg, Parquet). - Experience with distributed query engines and metadata management using Trino, Hive Metastore, and analytical serving platforms like StarRocks, ClickHouse, Druid, or Pinot. - Strong understanding of data governance and security, including Apache Ranger, access controls, auditing, data quality, and lineage.
- Design and operate distributed data systems using Apache Spark and Flink to support high-volume data processing. - Build and evolve LakeHouse architectures using Apache Iceberg, Apache Hudi, and Delta Lake with columnar file formats like Parquet and ORC. - Design and implement distributed query and federation capabilities using Trino, enabling unified access across data lakes, warehouses, and heterogeneous data sources.
- 8+ years of hands-on experience in data engineering, distributed systems, streaming, and cloud-native architectures. - Deep expertise in Kafka, Spark, Flink, and building scalable data pipelines using modern LakeHouse architectures (Delta Lake, Hudi, Iceberg, Parquet). - Experience with distributed query engines and metadata management using Trino, Hive Metastore, and analytical serving platforms like StarRocks, ClickHouse, Druid, or Pinot.
What you will do - Architect and implement streaming-first data pipelines using Apache Kafka and real-time processing frameworks. - Design and operate distributed data systems using Apache Spark and Flink to support high-volume data processing.
- Proven ability to deliver both data ingestion (inbound) and data serving (outbound) capabilities as reusable platform products. - Good exposure in integrating Redash, Metabase and Looker BI with the data stores. - Prior experience building RAG (Retrieval-Augmented Generation) pipelines using enterprise data sources.
- Experience enabling Natural Language Querying (NLQ) and conversational analytics over data warehouses and Lakehouse platforms. - Exposure to MLOps platforms such as SageMaker, Kubeflow, or MLflow, and how they interface with the data stack. You will thrive in this role if
- Architect and implement streaming-first data pipelines using Apache Kafka and real-time processing frameworks. - Design and operate distributed data systems using Apache Spark and Flink to support high-volume data processing. - Build and evolve LakeHouse architectures using Apache Iceberg, Apache Hudi, and Delta Lake with columnar file formats like Parquet and ORC.
- Build and evolve LakeHouse architectures using Apache Iceberg, Apache Hudi, and Delta Lake with columnar file formats like Parquet and ORC. - Design and implement distributed query and federation capabilities using Trino, enabling unified access across data lakes, warehouses, and heterogeneous data sources. - Define and manage metadata and catalog architectures using Hive Metastore and other table catalog solutions.

Your next step

Complete your application on job-boards.greenhouse.io. The employer’s form will show what is required.

Applying through Greenhouse

  • No account needed: the form is one page, with an “Autofill my application” option that reads your resume.
  • Greenhouse doesn’t read resumes over 2.5 MB, or details placed in a header, footer, text box, table or second column. The file still attaches, but a recruiter has to type you in by hand.

Already applied? Track this application

Job description

View original posting ↗

HackerRank helps companies like NVIDIA, Amazon, and Microsoft hire and upskill the next generation of developers based on skills, not pedigree. Our platform is trusted by over 2,500 of the world’s most innovative companies to build strong engineering teams ready for what’s next.

Software has entered an era where humans and AI build side by side. As this shift accelerates, the definition of strong technical talent is changing. We give companies better ways to identify and invest in next-generation skills.

People at HackerRank care deeply about the impact of their work and sweat the small details so our customers can be wildly successful with products they genuinely love to use. We move with urgency and believe great outcomes come from high standards.

About the role

We are seeking a Lead Data Engineer with deep expertise in distributed systems, real-time data processing, and modern data platforms to join our data team. This is a senior technical leadership role where you’ll help shape and stabilize our data architecture, drive business-critical initiatives, and mentor a high-performing team. You’ll be a senior individual contributor at the heart of the data organisation — owning complex platform decisions, collaborating cross-functionally with AI, product, and go-to-market teams, and shipping data-driven features that directly drive revenue. This is a greenfield opportunity to shape the next phase of data at HackerRank.

What you will do

  • Architect and implement streaming-first data pipelines using Apache Kafka and real-time processing frameworks.
  • Design and operate distributed data systems using Apache Spark and Flink to support high-volume data processing.
  • Build and evolve LakeHouse architectures using Apache Iceberg, Apache Hudi, and Delta Lake with columnar file formats like Parquet and ORC.
  • Design and implement distributed query and federation capabilities using Trino, enabling unified access across data lakes, warehouses, and heterogeneous data sources.
  • Define and manage metadata and catalog architectures using Hive Metastore and other table catalog solutions.
  • Build scalable data-serving and analytical layers using StarRocks and cloud data warehouses such as BigQuery, Redshift, and Snowflake.
  • Design and maintain robust data models, data marts, and data cubes that power business intelligence, reporting, and self-serve analytics.
  • Build the next-gen AI-optimised data layer: clean, structured datasets that power natural language querying and AI add-on features for HackerRank for Work customers.
  • Implement agile development best practices including Scrum, sprint planning, and documentation using JIRA, Linear, Notion and Confluence.
  • Collaborate with cross-functional teams including engineering, product, and data science to align technical solutions with business goals.
  • Act as a mentor and technical leader to other engineers, fostering a culture of excellence, knowledge sharing, and continuous improvement.

Who you are

  • 8+ years of hands-on experience in data engineering, distributed systems, streaming, and cloud-native architectures.
  • Deep expertise in Kafka, Spark, Flink, and building scalable data pipelines using modern LakeHouse architectures (Delta Lake, Hudi, Iceberg, Parquet).
  • Experience with distributed query engines and metadata management using Trino, Hive Metastore, and analytical serving platforms like StarRocks, ClickHouse, Druid, or Pinot.
  • Strong understanding of data governance and security, including Apache Ranger, access controls, auditing, data quality, and lineage.
  • Proficiency in Airflow, cloud platforms (AWS/GCP/Azure), observability (Prometheus, Grafana, Loki), and analytical data modeling.
  • Hands-on experience with AI Coding Agents (Claude Code, Codex, Cursor), agentic workflows, and applying emerging AI capabilities to improve engineering productivity and operational efficiency.
  • Passionate about engineering excellence, technical mentorship, architectural ownership, and building scalable data products aligned with business outcomes

Even better if you have

  • Proven ability to deliver both data ingestion (inbound) and data serving (outbound) capabilities as reusable platform products.
  • Good exposure in integrating Redash, Metabase and Looker BI with the data stores.
  • Prior experience building RAG (Retrieval-Augmented Generation) pipelines using enterprise data sources.
  • Experience enabling Natural Language Querying (NLQ) and conversational analytics over data warehouses and Lakehouse platforms.
  • Exposure to MLOps platforms such as SageMaker, Kubeflow, or MLflow, and how they interface with the data stack.

You will thrive in this role if

  • You believe in building data platforms that empower other engineering teams through self-service data ingestion, transformation, discovery, and consumption.
  • You enjoy translating ambiguous business problems into scalable data products, reusable platform capabilities, measurable engineering outcomes and take pride in engineering excellence, reliability, and long-term impact.
  • You're genuinely curious about AI and want to work at the intersection of Data Engineering, GenAI, RAG, Agentic AI, and intelligent analytics. 
  • You don't wait for a PM to hand you a roadmap — you proactively identify opportunities, connect data capabilities to business outcomes, and drive initiatives from conception to delivery.
  • You are motivated by solving problems that have visible, measurable, and company-wide business impact rather than simply delivering isolated technical features. You prioritize collective success over individual or team silos, stepping in wherever needed to deliver the best outcome for the company. 

Want to learn more about HackerRank? Check out HackerRank.com to explore our products, solutions and resources, and dive into our story and mission here.

HackerRank is a proud equal employment opportunity and affirmative action employer. We provide equal opportunity to everyone for employment based on individual performance and qualification. We never discriminate based on race, religion, national origin, gender identity or expression, sexual orientation, age, marital, veteran, or disability status. All your information will be kept confidential according to EEO guidelines. 

Linkedin | X | Blog | Instagram | Life@HackerRank

Notice to prospective HackerRank job applicants:

  • Our Recruiters use @hackerrank.com email addresses.
  • We never ask for payment or credit check information to apply, interview, or work here.

Demand for Data Engineer roles

New Data Engineer postings at the employers we follow are up 13% over the last 6 weeks compared with the 6 before.

Weekly new Data Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 845, the most 1,344, and the latest week had 1,007.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under Data Engineer, by the work the posting describes.

Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.

Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.

An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.

US government figures for database architects

Median yearly pay, May 2025
$139,500
Expected change, 2025 to 2035
9.4% more jobsagainst 3.5% for all jobs

The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1243.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Hybrid in Bangalore, India

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.