Back to jobs

Data Engineer - Commodities

Pay
Pay not listed
Work setting
Not stated
Employment
Full-time — employment source
Employment type in the employer’s posting: FULL_TIME
From the employer’s posting

What you’ll work on

Full posting
  • Design and maintain end-to-end ETL workflows in Python and SQL

  • Build standardized data models, schemas, and metadata for commodities datasets

  • Schedule and monitor data pipelines with Airflow or similar tools

  • Implement validation, reconciliation, and anomaly-detection controls

  • Refine datasets and documentation with Portfolio Managers and researchers

From the employer’s posting
Design, implement, and maintain end-to-end ETL workflows in Python and SQL
Build and maintain standardized data models, schemas, and metadata
Schedule, monitor, and manage data pipelines using Airflow or similar workflow orchestration tools
Implement validation, reconciliation, and anomaly-detection controls
Partner with commodities Portfolio Managers, researchers, data analysts, and data strategists to refine datasets, definitions, and documentation
Apply at Millennium

Posting record

Report an error
Status in our records
Live
First seen by us
Sep 29, 202651 days ago
Recorded sightings
95
Last seen by us
Nov 19, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 51

Similar jobs usually come down between day 6 and day 56.

At Millennium, jobs usually come down by about day 51.

From 14,186 similar postings that came down, and 33 of Millennium’s. Coming down is not always a hire.

Millennium is hiring

23 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Data Engineer hiring has slowed
  • Takes jobs down at a usual pace
  • 4% of open jobs were reposted
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • SQL
  • Airflow
Where the posting names them
Meet the Team The Commodities Technology team builds and operates a data platform that aggregates and curates critical commodities data, including weather, supply and demand, storage, transportation, and other fundamental and alternative datasets. The team’s curated content layer supports Portfolio Managers and researchers in understanding markets and constructing trades. As part of Millennium’s Information Technology organization, the team develops flexible, scalable technology and proprietary systems that support advanced analytical and trading capabilities. What You'll Do • Design, implement, and maintain end-to-end ETL workflows in Python and SQL to ingest and transform commodities data from multiple vendor and internal sources. • Build and maintain standardized data models, schemas, and metadata that make commodities datasets easy to understand, discover, and reuse. • Schedule, monitor, and manage data pipelines using Airflow or similar workflow orchestration tools to ensure reliable, timely data delivery. • Implement validation, reconciliation, and anomaly-detection controls to maintain data completeness, accuracy, and consistency. • Apply AI to automate schema inference across structured and semi-structured data sources, manage schema drift, and accelerate scalable ingestion development. • Use AI-driven data-quality, observability, and documentation capabilities to detect anomalies, monitor data health, and produce clear lineage and technical documentation. • Maintain high-quality, production-ready code and repeatable deployments through Git, GitHub Actions, PyTest, and automated testing practices. • Partner with commodities Portfolio Managers, researchers, data analysts, and data strategists to refine datasets, definitions, and documentation based on evolving business needs. What You Bring • 5+ years of experience in data engineering, analytics engineering, or a similar role building and maintaining ETL pipelines. • Strong Python and SQL skills, including experience working with large datasets and complex data transformations. • Hands-on experience with Airflow or comparable workflow orchestration tools. • Experience with Git, CI/CD pipelines such as GitHub Actions, and automated testing frameworks such as PyTest. • Strong attention to detail, data quality, and documentation, with the ability to identify edge cases and protect data integrity. • Ability to work independently, communicate effectively with technical and non-technical stakeholders, and manage multiple concurrent initiatives. • Knowledge of commodities markets and data, including weather, supply and demand, storage, freight, or flows, is preferred. • Experience with data warehousing, columnar storage formats, analytic databases, data catalog or governance tools, or financial services, trading, or research-driven environments is preferred.
What You'll Do • Design, implement, and maintain end-to-end ETL workflows in Python and SQL to ingest and transform commodities data from multiple vendor and internal sources. • Build and maintain standardized data models, schemas, and metadata that make commodities datasets easy to understand, discover, and reuse. • Schedule, monitor, and manage data pipelines using Airflow or similar workflow orchestration tools to ensure reliable, timely data delivery. • Implement validation, reconciliation, and anomaly-detection controls to maintain data completeness, accuracy, and consistency. • Apply AI to automate schema inference across structured and semi-structured data sources, manage schema drift, and accelerate scalable ingestion development. • Use AI-driven data-quality, observability, and documentation capabilities to detect anomalies, monitor data health, and produce clear lineage and technical documentation. • Maintain high-quality, production-ready code and repeatable deployments through Git, GitHub Actions, PyTest, and automated testing practices. • Partner with commodities Portfolio Managers, researchers, data analysts, and data strategists to refine datasets, definitions, and documentation based on evolving business needs. What You Bring • 5+ years of experience in data engineering, analytics engineering, or a similar role building and maintaining ETL pipelines. • Strong Python and SQL skills, including experience working with large datasets and complex data transformations. • Hands-on experience with Airflow or comparable workflow orchestration tools. • Experience with Git, CI/CD pipelines such as GitHub Actions, and automated testing frameworks such as PyTest. • Strong attention to detail, data quality, and documentation, with the ability to identify edge cases and protect data integrity. • Ability to work independently, communicate effectively with technical and non-technical stakeholders, and manage multiple concurrent initiatives. • Knowledge of commodities markets and data, including weather, supply and demand, storage, freight, or flows, is preferred. • Experience with data warehousing, columnar storage formats, analytic databases, data catalog or governance tools, or financial services, trading, or research-driven environments is preferred.

Your next step

Complete your application on mlp.eightfold.ai. The employer’s form will show what is required.

Applying through Eightfold

  • You sign in, with Microsoft, LinkedIn, Google, Facebook or a new account, before the application opens.

Already applied? Track this application

Job description

View original posting ↗

Data Engineer - Commodities About Millennium Millennium is a global, diversified alternative investment firm, founded in 1989. Defined by evolution, innovation and focus, Millennium’s mission is to deliver results for our investors. Our people are empowered with both independence and support: the autonomy to pursue ideas with conviction and the backing of a global network committed to collaboration, disciplined risk management and continuous learning. With opportunities to deepen expertise and accelerate development, talent at Millennium is equipped to adapt, evolve and build lasting impact over time. Discover how transformative growth accelerates impact. Meet the Team The Commodities Technology team builds and operates a data platform that aggregates and curates critical commodities data, including weather, supply and demand, storage, transportation, and other fundamental and alternative datasets. The team’s curated content layer supports Portfolio Managers and researchers in understanding markets and constructing trades. As part of Millennium’s Information Technology organization, the team develops flexible, scalable technology and proprietary systems that support advanced analytical and trading capabilities. What You'll Do • Design, implement, and maintain end-to-end ETL workflows in Python and SQL to ingest and transform commodities data from multiple vendor and internal sources. • Build and maintain standardized data models, schemas, and metadata that make commodities datasets easy to understand, discover, and reuse. • Schedule, monitor, and manage data pipelines using Airflow or similar workflow orchestration tools to ensure reliable, timely data delivery. • Implement validation, reconciliation, and anomaly-detection controls to maintain data completeness, accuracy, and consistency. • Apply AI to automate schema inference across structured and semi-structured data sources, manage schema drift, and accelerate scalable ingestion development. • Use AI-driven data-quality, observability, and documentation capabilities to detect anomalies, monitor data health, and produce clear lineage and technical documentation. • Maintain high-quality, production-ready code and repeatable deployments through Git, GitHub Actions, PyTest, and automated testing practices. • Partner with commodities Portfolio Managers, researchers, data analysts, and data strategists to refine datasets, definitions, and documentation based on evolving business needs. What You Bring • 5+ years of experience in data engineering, analytics engineering, or a similar role building and maintaining ETL pipelines. • Strong Python and SQL skills, including experience working with large datasets and complex data transformations. • Hands-on experience with Airflow or comparable workflow orchestration tools. • Experience with Git, CI/CD pipelines such as GitHub Actions, and automated testing frameworks such as PyTest. • Strong attention to detail, data quality, and documentation, with the ability to identify edge cases and protect data integrity. • Ability to work independently, communicate effectively with technical and non-technical stakeholders, and manage multiple concurrent initiatives. • Knowledge of commodities markets and data, including weather, supply and demand, storage, freight, or flows, is preferred. • Experience with data warehousing, columnar storage formats, analytic databases, data catalog or governance tools, or financial services, trading, or research-driven environments is preferred.

Demand for Data Engineer roles

New Data Engineer postings at the employers we follow are up 14% over the last 6 weeks compared with the 6 before.

Weekly new Data Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 860, the most 1,353, and the latest week had 1,020.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under Data Engineer, by the work the posting describes.

Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.

Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.

An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.

US government figures for database architects

Median yearly pay, May 2025
$139,500
Expected change, 2025 to 2035
9.4% more jobsagainst 3.5% for all jobs

The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1243.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

London, England, GB

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.