Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
Python
Scala
AWS
Spark
Terraform
Airflow
Dagster
Where the posting names them
Duties and Responsibilities: Design, build, and maintain data transformation pipelines to produce business-ready datasets to stakeholders. Document data pipelines, data models, and data transformation processes. Collaborate with business stakeholders teams to identify and understand data requirements for various business needs. Required Skills/Qualifications: Experience with Python and Scala. Experience with big data processing frameworks such as Spark, Flink, or Ray. Experience with dimensional modeling. Experience on building large scale distributed systems in AWS or a similar cloud provider. Experience using AI (e.g., Claude) to accelerate workflows. Experience with data orchestration tools like Airflow or Dagster. Experience with infrastructure-as-code tools (e.g., Terraform) and modern CI/CD pipelines.
Duties and Responsibilities: Design, build, and maintain data transformation pipelines to produce business-ready datasets to stakeholders. Document data pipelines, data models, and data transformation processes. Collaborate with business stakeholders teams to identify and understand data requirements for various business needs. Required Skills/Qualifications: Experience with Python and Scala. Experience with big data processing frameworks such as Spark, Flink, or Ray. Experience with dimensional modeling. Experience on building large scale distributed systems in AWS or a similar cloud provider. Experience using AI (e.g., Claude) to accelerate workflows. Experience with data orchestration tools like Airflow or Dagster. Experience with infrastructure-as-code tools (e.g., Terraform) and modern CI/CD pipelines.
Demand for Data Engineer roles
New Data Engineer postings at the employers we follow are up 13% over the last 6 weeks compared with the 6 before.
August 17Last week
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
About the role
We file this job under Data Engineer, by the work the posting describes.
Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.
Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.
An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.
US government figures for database architects
Median yearly pay, May 2025
$139,500
Expected change, 2025 to 2035
9.4% more jobsagainst 3.5% for all jobs
The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.