Pay
$127K–$189K is typical
The middle half of what similar US postings state. This one doesn’t list pay.
From 2,448 postings in the last year. Stated ranges, not salaries paid.
San Francisco · Singapore · San Francisco, California, United States
Open 3 months, first seen by us 101 days agoLocation type: On-siteFrom the employer’s posting
Employment type: Full-timeFrom the employer’s posting
Department: InfrastructureFrom the employer’s posting
Design and operate petabyte-scale lakehouse storage and data layouts
Own the shared Spark/Ray compute and workflow orchestration platform
Optimize high-throughput data loading into training
Build cataloging, deduplication, lineage, and reproducibility systems
Build platform-level data quality and monitoring tooling
Design and operate petabyte-scale storage: lakehouse architecture, file formats, and data layout optimized for both batch and real-time queries
Own the shared compute and orchestration platform (e.g. Spark, Ray, workflow scheduling)
owning high-throughput data loading into training up to the tensor boundary
Build systems for cataloging, deduplication, lineage, search, and reproducibility at every stage of the data lifecycle
Implement the platform-level quality and monitoring tooling that data and research teams build their checks on
Open 3 monthsWe first saw this posting 101 days ago and it is still up. Roles open this long are often slow to fill or kept open to collect applications.
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
$127K–$189K is typical
The middle half of what similar US postings state. This one doesn’t list pay.
From 2,448 postings in the last year. Stated ranges, not salaries paid.
Day 101
Up longer than most. Similar jobs usually come down between day 6 and day 56.
From 14,766 similar postings that came down. Coming down is not always a hire.
4 open jobs
Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
- Demonstrated experience building large-scale data pipelines and distributed compute systems (e.g. Spark, Ray, Beam) - Knowledge of state-of-the-art methods and tools for data ingestion, storage, and loading — including file formats and storage systems (e.g. Parquet, Zarr, Delta Lake) and how they impact performance and scalability - Deep familiarity with cloud infrastructure, data lake architectures, and batch and streaming pipelines
- Design and operate petabyte-scale storage: lakehouse architecture, file formats, and data layout optimized for both batch and real-time queries - Own the shared compute and orchestration platform (e.g. Spark, Ray, workflow scheduling) that ingestion and research pipelines run on - Optimize data strategy end to end from storage to loading, owning high-throughput data loading into training up to the tensor boundary
Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.
Already applied? Track this application
Our mission is general causal intelligence; AI that is capable of (1) predicting the future and (2) identifying the actions to alter it.
To achieve this breakthrough, we are building a Large Physics foundation Model (LPM) because physical systems, unlike text or images, are governed by verifiable cause and effect. We believe that scaling on physics will enable an understanding of causality required to predict and control physical systems, starting with weather.
Our founding team has built and deployed AI against the physical world in robotics, drug discovery, and particle physics at institutions like DeepMind, Waymo, Cruise, Insitro, Nabla Bio, and CERN.
We look for data engineers who are excited to tackle unsolved problems. Physical observations arrive continuously, in many formats, at a scale that dwarfs what is used to train today's LLMs. Your mission is to build the data platform underneath it all — the storage, compute, and loading systems that make every dataset cheap to ingest, fast to query, and immediately available to training.
Responsibilities
Design and operate petabyte-scale storage: lakehouse architecture, file formats, and data layout optimized for both batch and real-time queries
Own the shared compute and orchestration platform (e.g. Spark, Ray, workflow scheduling) that ingestion and research pipelines run on
Optimize data strategy end to end from storage to loading, owning high-throughput data loading into training up to the tensor boundary
Build systems for cataloging, deduplication, lineage, search, and reproducibility at every stage of the data lifecycle
Implement the platform-level quality and monitoring tooling that data and research teams build their checks on
Scale infrastructure to improve engineering velocity and ensure reliability, with monitoring and alerting to match
Work across the full data lifecycle when the mission needs it — including building and operating ingestion pipelines for critical data sources directly
What we're looking for
We value a relentless approach to problem-solving, rapid execution, and the ability to quickly learn in unfamiliar domains.
Demonstrated experience building large-scale data pipelines and distributed compute systems (e.g. Spark, Ray, Beam)
Knowledge of state-of-the-art methods and tools for data ingestion, storage, and loading — including file formats and storage systems (e.g. Parquet, Zarr, Delta Lake) and how they impact performance and scalability
Deep familiarity with cloud infrastructure, data lake architectures, and batch and streaming pipelines
Understanding of how data loading throughput affects large-scale training, and experience optimizing it
Owns deliverables end-to-end, from collecting and translating requirements to autonomously driving execution
New Data Engineer postings at the employers we follow are up 13% over the last 6 weeks compared with the 6 before.
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
We file this job under Data Engineer, by the work the posting describes.
Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.
Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.
An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.
The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.
Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1243.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
No pay amount identified in the saved description.
San Francisco, California, United States
Working pattern and location restrictions need checking in the full posting.
No clear work-authorization passage found. Eligibility is unconfirmed.