Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
PL/SQL
Python
Scala
SQL
Azure
Databricks
Delta Lake
Informatica
Oracle
Airflow
PySpark
Where the posting names them
Key Responsibilities Data Pipeline Development • Design and develop scalable data pipelines using Azure Databricks, Python, and PySpark. • Implement ETL/ELT workflows for structured, semi-structured, and unstructured data. • Develop and maintain data processing workflows using Delta Lake architecture. • Ensure efficient data ingestion, transformation, and loading processes. Data Management & Optimisation • Optimise performance of data pipelines and processing workloads. • Ensure data quality, reliability, and consistency across data platforms. • Write and optimise complex SQL queries for data processing and validation. • Implement data governance and access controls. Cloud & Data Platform Integration • Work with Azure Data Lake for storing and managing large datasets. • Manage data cataloging and governance using Unity Catalog. • Support scalable and secure cloud-based data architectures. Collaboration & Migration Projects • Work closely with cross-functional teams to deliver data-driven solutions. • Participate in data migration and modernisation initiatives. • Support troubleshooting and resolution of data pipeline issues. • Document data architecture, pipelines, and processes. Required Skills & Experience · Must have 7 years of IT experience and at least 5+ years of hands-on experience in Databricks with good communication skills. · Design and develop scalable data pipelines using Azure Databricks, SCALA, Python, PySpark, and Delta Lake. · Have a good understanding of RDBMS and proficiency in writing complex SQL (1st Preference – Oracle / PLSQL). · Implement ETL/ELT workflows for structured, semi-structured, and unstructured data. · Optimise performance of data processing and ensure data quality standards. · Work with Azure Data Lake, Unity Catalogue, and integrate with CI/CD pipelines. · Experience with orchestration tools like Airflow or Azure Data Factory is a plus. · Have working experience on a migration project. · Preferable to have experience in Informatica (although not mandatory) Relevant Experience Preferred • Hands-on experience developing data pipelines using Azure Databricks and PySpark. • Experience implementing ETL/ELT solutions for enterprise data platforms. • Experience optimizing data processing performance and data workflows. • Experience working with Azure Data Lake and cloud-based data architectures. • Experience working on data migration or modernization projects.
Key Responsibilities Data Pipeline Development • Design and develop scalable data pipelines using Azure Databricks, Python, and PySpark. • Implement ETL/ELT workflows for structured, semi-structured, and unstructured data. • Develop and maintain data processing workflows using Delta Lake architecture. • Ensure efficient data ingestion, transformation, and loading processes. Data Management & Optimisation • Optimise performance of data pipelines and processing workloads. • Ensure data quality, reliability, and consistency across data platforms. • Write and optimise complex SQL queries for data processing and validation. • Implement data governance and access controls. Cloud & Data Platform Integration • Work with Azure Data Lake for storing and managing large datasets. • Manage data cataloging and governance using Unity Catalog. • Support scalable and secure cloud-based data architectures. Collaboration & Migration Projects • Work closely with cross-functional teams to deliver data-driven solutions. • Participate in data migration and modernisation initiatives. • Support troubleshooting and resolution of data pipeline issues. • Document data architecture, pipelines, and processes. Required Skills & Experience · Must have 7 years of IT experience and at least 5+ years of hands-on experience in Databricks with good communication skills. · Design and develop scalable data pipelines using Azure Databricks, SCALA, Python, PySpark, and Delta Lake. · Have a good understanding of RDBMS and proficiency in writing complex SQL (1st Preference – Oracle / PLSQL). · Implement ETL/ELT workflows for structured, semi-structured, and unstructured data. · Optimise performance of data processing and ensure data quality standards. · Work with Azure Data Lake, Unity Catalogue, and integrate with CI/CD pipelines. · Experience with orchestration tools like Airflow or Azure Data Factory is a plus. · Have working experience on a migration project. · Preferable to have experience in Informatica (although not mandatory) Relevant Experience Preferred • Hands-on experience developing data pipelines using Azure Databricks and PySpark. • Experience implementing ETL/ELT solutions for enterprise data platforms. • Experience optimizing data processing performance and data workflows. • Experience working with Azure Data Lake and cloud-based data architectures. • Experience working on data migration or modernization projects.
Demand for Data Engineer roles
New Data Engineer postings at the employers we follow are up 14% over the last 6 weeks compared with the 6 before.
August 17Last week
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
About the role
We file this job under Data Engineer, by the work the posting describes.
Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.
Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.
An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.
US government figures for database architects
Median yearly pay, May 2025
$139,500
Expected change, 2025 to 2035
9.4% more jobsagainst 3.5% for all jobs
The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.