Time since we first saw it
Day 23
Similar jobs usually come down between day 6 and day 56.
At Kpler, jobs usually come down by about day 40.
From 14,771 similar postings that came down, and 36 of Kpler’s. Coming down is not always a hire.
Work setup in the employer’s posting: hybridFrom the employer’s posting
Employment type Full-time
Employment type in the employer’s posting: Full-timeFrom the employer’s posting
As a Senior Data Engineer on the Financial Flows team, you will take full technical ownership of the end-to-end data pipelines, predictive models, and cloud infrastructure powering Kpler’s commodity finance analytics.
Build batch and real-time streaming pipelines from financial data APIs and WebSockets
Lead migration of legacy backend infrastructure to cloud-native AWS
Rewrite CTA positioning indicator calculations from PHP to Python with output parity
Model futures pricing, open interest and volume data in QuestDB and Elasticsearch
Add automated data validation, alerting and monitoring for market data feeds
As a Senior Data Engineer on the Financial Flows team, you will take full technical ownership of the end-to-end data pipelines, predictive models, and cloud infrastructure powering Kpler’s commodity finance analytics. You will lead the modernization of our data foundation, spearheading the migration of production services to AWS and re-architecting legacy calculation engines into clean, performant Python services. Operating at the crossroads of quantitative research and cloud infrastructure, your work will directly ensure that market participants receive real-time, mission-critical insights across global commodities markets.
Design, build, and operate resilient batch and real-time streaming data ingestion pipelines connecting to external financial market data providers via APIs and WebSockets
Lead the architectural migration of legacy backend infrastructure to modern cloud-native environments on AWS
Drive the refactoring and rewrite of the CTA positioning indicator calculation blocks from PHP to Python, ensuring strict mathematical output parity
Ingest and model complex financial datasets—including futures pricing, open interest, and trading volumes—across time-series stores and search engines like QuestDB and Elasticsearch
Implement automated data validation, proactive alerting, and system monitoring to ensure market data feeds remain accurate, continuous, and fault-tolerant
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
Day 23
Similar jobs usually come down between day 6 and day 56.
At Kpler, jobs usually come down by about day 40.
From 14,771 similar postings that came down, and 36 of Kpler’s. Coming down is not always a hire.
13 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
- Lead the architectural migration of legacy backend infrastructure to modern cloud-native environments on AWS, establishing high standards for system reliability and scalability. - Drive the refactoring and rewrite of the CTA positioning indicator calculation blocks from PHP to Python, ensuring strict mathematical output parity, comprehensive test coverage, and optimized latency. - Ingest and model complex financial datasets—including futures pricing, open interest, and trading volumes—across time-series stores and search engines like QuestDB and Elasticsearch.
- Experience integrating data from major financial market vendors (e.g., Bloomberg, LSEG, Enverus, Trading Technologies). - Familiarity with legacy PHP codebase migration or reading PHP algorithms to translate into Python. - Practical familiarity with GitOps frameworks and infrastructure-as-code tooling.
About the Role As a Senior Data Engineer on the Financial Flows team, you will take full technical ownership of the end-to-end data pipelines, predictive models, and cloud infrastructure powering Kpler’s commodity finance analytics. You will lead the modernization of our data foundation, spearheading the migration of production services to AWS and re-architecting legacy calculation engines into clean, performant Python services. Operating at the crossroads of quantitative research and cloud infrastructure, your work will directly ensure that market participants receive real-time, mission-critical insights across global commodities markets. Key Responsibilities
- 5+ years of professional experience building, operating, and scaling production-grade data pipelines and backend infrastructure. - Advanced expertise in Python, with a strong background refactoring and rewriting production-critical code for performance and testability. - Proven hands-on experience building and maintaining real-time or streaming API ingestion pipelines (REST, WebSockets) handling time-series datasets.
- Solid production experience deploying and managing containerized applications on AWS using Docker, Kubernetes, and modern CI/CD practices. - Strong database fundamentals in SQL and experience working with distributed search or time-series databases (e.g., Elasticsearch, QuestDB, ClickHouse). - Deep experience establishing automated data quality frameworks, observability (monitoring/alerting), and root-cause troubleshooting for production systems.
- Design, build, and operate resilient batch and real-time streaming data ingestion pipelines connecting to external financial market data providers via APIs and WebSockets. - Lead the architectural migration of legacy backend infrastructure to modern cloud-native environments on AWS, establishing high standards for system reliability and scalability. - Drive the refactoring and rewrite of the CTA positioning indicator calculation blocks from PHP to Python, ensuring strict mathematical output parity, comprehensive test coverage, and optimized latency.
- Proven hands-on experience building and maintaining real-time or streaming API ingestion pipelines (REST, WebSockets) handling time-series datasets. - Solid production experience deploying and managing containerized applications on AWS using Docker, Kubernetes, and modern CI/CD practices. - Strong database fundamentals in SQL and experience working with distributed search or time-series databases (e.g., Elasticsearch, QuestDB, ClickHouse).
- Implement automated data validation, proactive alerting, and system monitoring to ensure market data feeds remain accurate, continuous, and fault-tolerant. - Deploy, maintain, and scale containerized services across Docker and Kubernetes environments using automated CI/CD pipelines and GitOps workflows. - Partner closely with quantitative researchers, commodity analysts, and product managers to translate complex trading models into high-performance production pipelines.
- Drive the refactoring and rewrite of the CTA positioning indicator calculation blocks from PHP to Python, ensuring strict mathematical output parity, comprehensive test coverage, and optimized latency. - Ingest and model complex financial datasets—including futures pricing, open interest, and trading volumes—across time-series stores and search engines like QuestDB and Elasticsearch. - Implement automated data validation, proactive alerting, and system monitoring to ensure market data feeds remain accurate, continuous, and fault-tolerant.
Complete your application on jobs.lever.co. The employer’s form will show what is required.
Already applied? Track this application
As a Senior Data Engineer on the Financial Flows team, you will take full technical ownership of the end-to-end data pipelines, predictive models, and cloud infrastructure powering Kpler’s commodity finance analytics. You will lead the modernization of our data foundation, spearheading the migration of production services to AWS and re-architecting legacy calculation engines into clean, performant Python services. Operating at the crossroads of quantitative research and cloud infrastructure, your work will directly ensure that market participants receive real-time, mission-critical insights across global commodities markets.
Design, build, and operate resilient batch and real-time streaming data ingestion pipelines connecting to external financial market data providers via APIs and WebSockets.
Lead the architectural migration of legacy backend infrastructure to modern cloud-native environments on AWS, establishing high standards for system reliability and scalability.
Drive the refactoring and rewrite of the CTA positioning indicator calculation blocks from PHP to Python, ensuring strict mathematical output parity, comprehensive test coverage, and optimized latency.
Ingest and model complex financial datasets—including futures pricing, open interest, and trading volumes—across time-series stores and search engines like QuestDB and Elasticsearch.
Implement automated data validation, proactive alerting, and system monitoring to ensure market data feeds remain accurate, continuous, and fault-tolerant.
Deploy, maintain, and scale containerized services across Docker and Kubernetes environments using automated CI/CD pipelines and GitOps workflows.
Partner closely with quantitative researchers, commodity analysts, and product managers to translate complex trading models into high-performance production pipelines.
Provide technical mentorship to peer engineers, raising team-wide engineering rigor through thoughtful code reviews, system design guidance, and architectural documentation.
5+ years of professional experience building, operating, and scaling production-grade data pipelines and backend infrastructure.
Advanced expertise in Python, with a strong background refactoring and rewriting production-critical code for performance and testability.
Proven hands-on experience building and maintaining real-time or streaming API ingestion pipelines (REST, WebSockets) handling time-series datasets.
Solid production experience deploying and managing containerized applications on AWS using Docker, Kubernetes, and modern CI/CD practices.
Strong database fundamentals in SQL and experience working with distributed search or time-series databases (e.g., Elasticsearch, QuestDB, ClickHouse).
Deep experience establishing automated data quality frameworks, observability (monitoring/alerting), and root-cause troubleshooting for production systems.
Domain knowledge of financial market structures, futures contracts, open interest, and commodities trading symbology.
Experience integrating data from major financial market vendors (e.g., Bloomberg, LSEG, Enverus, Trading Technologies).
Familiarity with legacy PHP codebase migration or reading PHP algorithms to translate into Python.
Practical familiarity with GitOps frameworks and infrastructure-as-code tooling.
Full-time
New Data Engineer postings at the employers we follow are up 13% over the last 6 weeks compared with the 6 before.
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
We file this job under Data Engineer, by the work the posting describes.
Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.
Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.
An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.
The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.
Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1243.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
No pay amount identified in the saved description.
France
Working pattern and location restrictions need checking in the full posting.
No clear work-authorization passage found. Eligibility is unconfirmed.