Implement smart deduplication and sampling to balance training distribution
Design identity-aware train/test splits that measure true generalization
Tools in this posting
Python
AWS
S3
Hugging Face
pandas
Source — Tool mentions in context
Job Title:Data Engineer - ML Training Data Pipeline Notice period: 0-30 Days Experience : 5+ Years Location: Hyderabad OR Pune We are looking for Data Engineer - ML Training Data Pipeline who can Build and maintain the data pipeline that transforms raw production traces into high-quality training datasets for LLM fine-tuning-ingestion, deduplication, format conversion, quality filtering, and train/test splitting at scale on AWS. What We Expect: Build end-to-end data pipelines: raw trace ingestion → dedup → format conversion → quality gating → training-ready datasets Process large-scale JSONL data on AWS S3 (tens of thousands of traces per batch) Convert between chat-completion formats (e.g., OpenAI → Llama 3.1 tool-calling format) Implement smart deduplication and sampling to balance training distribution Design identity-aware train/test splits that measure true generalization Build data validation gates to detect schema drift and format anomalies Create a continuous pipeline that auto-processes new production traces for retraining Requirements Experience: 6+ years data engineering focused on ML data pipelines Python: Strong — pandas, pyarrow, JSONL processing at scale ML Data Libraries: HuggingFace Datasets, Arrow-based storage Data Formats: Multi-turn conversation/chat data structures and tokenizer-specific formatting Deduplication: Content hashing, identity-based grouping strategies AWS: S3, EC2, batch processing workflows Preferred (Not Required): LLM training data prep (chat templates, tool-calling schemas); Axolotl or similar dataset formats; data versioning (DVC, LakeFS); browser-automation trace data or Playwright. Benefits Comprehensive Medical Coverage: Health insurance of INR 5.0 Lakhs for you and your family (up to 6 members), ensuring complete peace of mind. Robust Protection Plans: Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones. Retirement Benefits: PF and Gratuity provided as per standard government regulations. Flexible Work Options: Enjoy hybrid work arrangements & flexible working hours. Generous Leave Policy: 21 days of annual leave, in addition to 10 company-declared holidays. Employee Well-being Spaces: Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.
Job Title:Data Engineer - ML Training Data Pipeline Notice period: 0-30 Days Experience : 5+ Years Location: Hyderabad OR Pune We are looking for Data Engineer - ML Training Data Pipeline who can Build and maintain the data pipeline that transforms raw production traces into high-quality training datasets for LLM fine-tuning-ingestion, deduplication, format conversion, quality filtering, and train/test splitting at scale on AWS. What We Expect: Build end-to-end data pipelines: raw trace ingestion → dedup → format conversion → quality gating → training-ready datasets Process large-scale JSONL data on AWS S3 (tens of thousands of traces per batch) Convert between chat-completion formats (e.g., OpenAI → Llama 3.1 tool-calling format) Implement smart deduplication and sampling to balance training distribution Design identity-aware train/test splits that measure true generalization Build data validation gates to detect schema drift and format anomalies Create a continuous pipeline that auto-processes new production traces for retraining Requirements Experience: 6+ years data engineering focused on ML data pipelines Python: Strong — pandas, pyarrow, JSONL processing at scale ML Data Libraries: HuggingFace Datasets, Arrow-based storage Data Formats: Multi-turn conversation/chat data structures and tokenizer-specific formatting Deduplication: Content hashing, identity-based grouping strategies AWS: S3, EC2, batch processing workflows Preferred (Not Required): LLM training data prep (chat templates, tool-calling schemas); Axolotl or similar dataset formats; data versioning (DVC, LakeFS); browser-automation trace data or Playwright. Benefits Comprehensive Medical Coverage: Health insurance of INR 5.0 Lakhs for you and your family (up to 6 members), ensuring complete peace of mind. Robust Protection Plans: Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones. Retirement Benefits: PF and Gratuity provided as per standard government regulations. Flexible Work Options: Enjoy hybrid work arrangements & flexible working hours. Generous Leave Policy: 21 days of annual leave, in addition to 10 company-declared holidays. Employee Well-being Spaces: Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.
Demand for Data Engineer roles
New Data Engineer postings at the employers we follow are up 13% over the last 6 weeks compared with the 6 before.
August 17Last week
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
About the role
We file this job under Data Engineer, by the work the posting describes.
Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.
Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.
An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.
US government figures for database architects
Median yearly pay, May 2025
$139,500
Expected change, 2025 to 2035
9.4% more jobsagainst 3.5% for all jobs
The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.