Design, develop and optimize scalable data pipelines for large-scale processing
Build event-driven processing pipelines using Kafka or Pub/Sub
Design and optimize data models for performance and scalability
Work with product and cross-functional teams on data solutions
From the employer’s posting
designing, developing, and optimizing scalable data pipelines for large-scale data processing, transformation, and analysis
Experience building event-driven processing pipelines using Kafka or Google Cloud Pub/Sub
including designing and optimizing data models for performance and scalability
Effective communication with product and cross-functional teams
Before you apply
Employer’s visa record
5,157 H-1B petitions approved in fiscal 2024 and 2025For the employer as a whole. This posting doesn’t say whether the role is sponsored. US Citizenship and Immigration Services.
Bachelor's or Master's in computer science or equivalent experience
Qualification wording
5+ years of hands-on software development experience with Big Data and analytics solutions
Hands-on experience with Google Cloud Platform technologies, including BigQuery, Dataproc, Pub/Sub, and Airflow
plus Hadoop, Hive, and Spark
Strong Python and shell-scripting skills
Bachelor’s or Master’s degree in Computer Science, Computer Engineering, another technical discipline, or equivalent work experience
Education & alternatives
Must-Have Skills
- Education: Bachelor’s or Master’s degree in Computer Science, Computer Engineering, another technical discipline, or equivalent work experience.
- Experience: 5+ years of hands-on software development experience with Big Data and analytics solutions.
Tools in this posting
Python
SQL
BigQuery
GCP
Hadoop
Hive
Kafka
Spark
Airflow
Source — Tool mentions in context
- GCP and Big Data: Hands-on experience with Google Cloud Platform technologies, including BigQuery, Dataproc, Pub/Sub, and Airflow, plus Hadoop, Hive, and Spark.
- Programming and scripting: Strong Python and shell-scripting skills.
- Data engineering: Strong experience designing, developing, and optimizing scalable data pipelines for large-scale data processing, transformation, and analysis.
- Data engineering: Strong experience designing, developing, and optimizing scalable data pipelines for large-scale data processing, transformation, and analysis.
- SQL and data modelling: Proficiency in SQL and relational database systems, including designing and optimizing data models for performance and scalability.
- Event processing: Experience building event-driven processing pipelines using Kafka or Google Cloud Pub/Sub.
- Experience: 5+ years of hands-on software development experience with Big Data and analytics solutions.
- GCP and Big Data: Hands-on experience with Google Cloud Platform technologies, including BigQuery, Dataproc, Pub/Sub, and Airflow, plus Hadoop, Hive, and Spark.
- Programming and scripting: Strong Python and shell-scripting skills.
- SQL and data modelling: Proficiency in SQL and relational database systems, including designing and optimizing data models for performance and scalability.
- Event processing: Experience building event-driven processing pipelines using Kafka or Google Cloud Pub/Sub.
- Distributed systems: Knowledge of distributed, multi-tiered systems and algorithms.
- Product partnership: Experience partnering with product teams to envision and deliver technology-led transformation.
- Certification: Google Cloud Professional Data Engineer certification.
Key Responsibilities
- Product partnership: Experience partnering with product teams to envision and deliver technology-led transformation.
- Certification: Google Cloud Professional Data Engineer certification.
Skill Requirements
- Product partnership: Experience partnering with product teams to envision and deliver technology-led transformation.
- Certification: Google Cloud Professional Data Engineer certification.
Other Requirements
Nice-to-Have Skills
- Workflow and messaging tools: Additional design and development experience with Airflow, Pub/Sub, and Kafka.
- DevOps tooling: Experience with Git, GitHub, Jenkins, XLR, and configuration-management practices.
Education: Bachelor’s or Master’s degree in Computer Science, Computer Engineering, another technical discipline, or equivalent work experience.
Experience: 5+ years of hands-on software development experience with Big Data and analytics solutions.
GCP and Big Data: Hands-on experience with Google Cloud Platform technologies, including BigQuery, Dataproc, Pub/Sub, and Airflow, plus Hadoop, Hive, and Spark.
Programming and scripting: Strong Python and shell-scripting skills.
Data engineering: Strong experience designing, developing, and optimizing scalable data pipelines for large-scale data processing, transformation, and analysis.
SQL and data modelling: Proficiency in SQL and relational database systems, including designing and optimizing data models for performance and scalability.
Event processing: Experience building event-driven processing pipelines using Kafka or Google Cloud Pub/Sub.
Distributed systems: Knowledge of distributed, multi-tiered systems and algorithms.
Software engineering: Strong object-oriented programming skills and knowledge of design patterns.
CI/CD and source control: Experience with CI/CD pipelines, automated testing frameworks, and source-code management tools such as Jenkins and Git.
Problem solving: Ability to analyze complex data-engineering problems, propose effective solutions, and implement them successfully.
Enterprise frameworks: Ability and willingness to learn, adopt, and build solutions using established enterprise Big Data frameworks.
Collaboration: Effective communication with product and cross-functional teams.
Delivery methodology: Understanding of SDLC methodologies and familiarity with Agile and Scrum ceremonies.
Continuous improvement: Proactive approach to identifying improvement opportunities and willingness to learn and apply new technologies.
Nice-to-Have Skills
Workflow and messaging tools: Additional design and development experience with Airflow, Pub/Sub, and Kafka.
DevOps tooling: Experience with Git, GitHub, Jenkins, XLR, and configuration-management practices.
Regulatory technology: Experience building innovative engineering solutions for financial-crimes compliance.
Engineering excellence: Experience improving resilience, predictability, delivery efficiency, and automation within engineering teams.
Product partnership: Experience partnering with product teams to envision and deliver technology-led transformation.
Certification: Google Cloud Professional Data Engineer certification.
Key Responsibilities
Big Data / GCP Data Engineer
Must-Have and Nice-to-Have Skills
Must-Have Skills
Education: Bachelor’s or Master’s degree in Computer Science, Computer Engineering, another technical discipline, or equivalent work experience.
Experience: 5+ years of hands-on software development experience with Big Data and analytics solutions.
GCP and Big Data: Hands-on experience with Google Cloud Platform technologies, including BigQuery, Dataproc, Pub/Sub, and Airflow, plus Hadoop, Hive, and Spark.
Programming and scripting: Strong Python and shell-scripting skills.
Data engineering: Strong experience designing, developing, and optimizing scalable data pipelines for large-scale data processing, transformation, and analysis.
SQL and data modelling: Proficiency in SQL and relational database systems, including designing and optimizing data models for performance and scalability.
Event processing: Experience building event-driven processing pipelines using Kafka or Google Cloud Pub/Sub.
Distributed systems: Knowledge of distributed, multi-tiered systems and algorithms.
Software engineering: Strong object-oriented programming skills and knowledge of design patterns.
CI/CD and source control: Experience with CI/CD pipelines, automated testing frameworks, and source-code management tools such as Jenkins and Git.
Problem solving: Ability to analyze complex data-engineering problems, propose effective solutions, and implement them successfully.
Enterprise frameworks: Ability and willingness to learn, adopt, and build solutions using established enterprise Big Data frameworks.
Collaboration: Effective communication with product and cross-functional teams.
Delivery methodology: Understanding of SDLC methodologies and familiarity with Agile and Scrum ceremonies.
Continuous improvement: Proactive approach to identifying improvement opportunities and willingness to learn and apply new technologies.
Nice-to-Have Skills
Workflow and messaging tools: Additional design and development experience with Airflow, Pub/Sub, and Kafka.
DevOps tooling: Experience with Git, GitHub, Jenkins, XLR, and configuration-management practices.
Regulatory technology: Experience building innovative engineering solutions for financial-crimes compliance.
Engineering excellence: Experience improving resilience, predictability, delivery efficiency, and automation within engineering teams.
Product partnership: Experience partnering with product teams to envision and deliver technology-led transformation.
Certification: Google Cloud Professional Data Engineer certification.
Skill Requirements
Big Data / GCP Data Engineer
Must-Have and Nice-to-Have Skills
Must-Have Skills
Education: Bachelor’s or Master’s degree in Computer Science, Computer Engineering, another technical discipline, or equivalent work experience.
Experience: 5+ years of hands-on software development experience with Big Data and analytics solutions.
GCP and Big Data: Hands-on experience with Google Cloud Platform technologies, including BigQuery, Dataproc, Pub/Sub, and Airflow, plus Hadoop, Hive, and Spark.
Programming and scripting: Strong Python and shell-scripting skills.
Data engineering: Strong experience designing, developing, and optimizing scalable data pipelines for large-scale data processing, transformation, and analysis.
SQL and data modelling: Proficiency in SQL and relational database systems, including designing and optimizing data models for performance and scalability.
Event processing: Experience building event-driven processing pipelines using Kafka or Google Cloud Pub/Sub.
Distributed systems: Knowledge of distributed, multi-tiered systems and algorithms.
Software engineering: Strong object-oriented programming skills and knowledge of design patterns.
CI/CD and source control: Experience with CI/CD pipelines, automated testing frameworks, and source-code management tools such as Jenkins and Git.
Problem solving: Ability to analyze complex data-engineering problems, propose effective solutions, and implement them successfully.
Enterprise frameworks: Ability and willingness to learn, adopt, and build solutions using established enterprise Big Data frameworks.
Collaboration: Effective communication with product and cross-functional teams.
Delivery methodology: Understanding of SDLC methodologies and familiarity with Agile and Scrum ceremonies.
Continuous improvement: Proactive approach to identifying improvement opportunities and willingness to learn and apply new technologies.
Nice-to-Have Skills
Workflow and messaging tools: Additional design and development experience with Airflow, Pub/Sub, and Kafka.
DevOps tooling: Experience with Git, GitHub, Jenkins, XLR, and configuration-management practices.
Regulatory technology: Experience building innovative engineering solutions for financial-crimes compliance.
Engineering excellence: Experience improving resilience, predictability, delivery efficiency, and automation within engineering teams.
Product partnership: Experience partnering with product teams to envision and deliver technology-led transformation.
Certification: Google Cloud Professional Data Engineer certification.
Other Requirements
About HCL Technologies
People
About 20,206 in 2024.Benefit plan filings (Form 5500)
Company
Listed as HCLTECH.NS.
H-1B visas
754 approved for new hires in fiscal 2025.US Citizenship and Immigration Services
Layoff notices
Notices this year and last cover 120 people. The latest was filed March 27, 2026.State WARN notices
Demand for Data Engineer roles
New Data Engineer postings at the employers we follow are up 13% over the last 6 weeks compared with the 6 before.
August 17Last week
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
About the role
We file this job under Data Engineer, by the work the posting describes.
Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.
Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.
An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.
US government figures for database architects
Median yearly pay, May 2025
$139,500
Expected change, 2025 to 2035
9.4% more jobsagainst 3.5% for all jobs
The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.