Design, build, and maintain batch and streaming data pipelines on Databricks
Ingest data from databases, files, APIs and cloud storage into a lakehouse
Develop bronze, silver and gold layers with Delta Lake and incremental processing
Optimize Spark workloads, SQL queries and storage for performance and cost
Schedule workflows and troubleshoot production issues with root-cause analysis
From the employer’s posting
Design, build, and maintain batch and streaming data pipelines using Databricks, PySpark, Python and SQL.
Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.
Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
Python
SQL
AWS
Azure
Databricks
Delta Lake
Spark
PySpark
GCP
Where the posting names them
- Hands-on experience developing data engineering solutions on Databricks.
- Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
- Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
Key Responsibilities
- Design, build, and maintain batch and streaming data pipelines using Databricks, PySpark, Python and SQL.
- Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
- Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
- Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
- Schedule and orchestrate data workflows; implement monitoring, alerting, error handling, and recovery mechanisms.
- Experience building ETL/ELT pipelines, implementing incremental loads, and applying data warehousing and dimensional modeling concepts.
- Experience with at least one cloud platform—Azure, AWS, or Google Cloud—and its storage and identity services.
- Working knowledge of Databricks jobs, notebooks, compute, Unity Catalog, and access management.
Required Skills and Experience
- Hands-on experience developing data engineering solutions on Databricks.
- Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
- Experience with at least one cloud platform—Azure, AWS, or Google Cloud—and its storage and identity services.
- Working knowledge of Databricks jobs, notebooks, compute, Unity Catalog, and access management.
- Proficiency with Git, code reviews, automated testing, and deployment practices.
- Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.
Mandatory Skills: DataBricks - Data Engineering .
Experience: 5-8 Years .
- Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
- Experience with Delta Lake, including merge operations, schema management, and transactional data processing.
- Experience building ETL/ELT pipelines, implementing incremental loads, and applying data warehousing and dimensional modeling concepts.
- Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
- Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
- Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
- Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
- Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
- Experience with Delta Lake, including merge operations, schema management, and transactional data processing.
Wipro Limited (NYSE: WIT, BSE: 507685, NSE: WIPRO) is a leading technology services and consulting company focused on building innovative solutions that address clients’ most complex digital transformation needs. Leveraging our holistic portfolio of capabilities in consulting, design, engineering, and operations, we help clients realize their boldest ambitions and build future-ready, sustainable businesses. With over 230,000 employees and business partners across 65 countries, we deliver on the promise of helping our customers, colleagues, and communities thrive in an ever-changing world. For additional information, visit us at www.wipro.com.
Job Description
Required Skills and Experience
Hands-on experience developing data engineering solutions on Databricks.
Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
Experience with Delta Lake, including merge operations, schema management, and transactional data processing.
Experience building ETL/ELT pipelines, implementing incremental loads, and applying data warehousing and dimensional modeling concepts.
Experience with at least one cloud platform—Azure, AWS, or Google Cloud—and its storage and identity services.
Working knowledge of Databricks jobs, notebooks, compute, Unity Catalog, and access management.
Proficiency with Git, code reviews, automated testing, and deployment practices.
Strong analytical, troubleshooting, documentation, and stakeholder communication skills.
Key Responsibilities
Design, build, and maintain batch and streaming data pipelines using Databricks, PySpark, Python and SQL.
Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
Schedule and orchestrate data workflows; implement monitoring, alerting, error handling, and recovery mechanisms.
Implement data validation, reconciliation, automated testing, and data quality controls.
Apply data governance and access controls using Unity Catalog and organizational security standards.
Collaborate with architects, analysts, and business stakeholders to translate requirements into technical solutions.
Maintain version-controlled code, support CI/CD processes, and document pipelines, data models, and operational procedures.
Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.
͏
Required Skills and Experience
Hands-on experience developing data engineering solutions on Databricks.
Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
Experience with Delta Lake, including merge operations, schema management, and transactional data processing.
Experience building ETL/ELT pipelines, implementing incremental loads, and applying data warehousing and dimensional modeling concepts.
Experience with at least one cloud platform—Azure, AWS, or Google Cloud—and its storage and identity services.
Working knowledge of Databricks jobs, notebooks, compute, Unity Catalog, and access management.
Proficiency with Git, code reviews, automated testing, and deployment practices.
Strong analytical, troubleshooting, documentation, and stakeholder communication skills.
Key Responsibilities
Design, build, and maintain batch and streaming data pipelines using Databricks, PySpark, Python and SQL.
Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
Schedule and orchestrate data workflows; implement monitoring, alerting, error handling, and recovery mechanisms.
Implement data validation, reconciliation, automated testing, and data quality controls.
Apply data governance and access controls using Unity Catalog and organizational security standards.
Collaborate with architects, analysts, and business stakeholders to translate requirements into technical solutions.
Maintain version-controlled code, support CI/CD processes, and document pipelines, data models, and operational procedures.
Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.
͏
Required Skills and Experience
Hands-on experience developing data engineering solutions on Databricks.
Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
Experience with Delta Lake, including merge operations, schema management, and transactional data processing.
Experience building ETL/ELT pipelines, implementing incremental loads, and applying data warehousing and dimensional modeling concepts.
Experience with at least one cloud platform—Azure, AWS, or Google Cloud—and its storage and identity services.
Working knowledge of Databricks jobs, notebooks, compute, Unity Catalog, and access management.
Proficiency with Git, code reviews, automated testing, and deployment practices.
Strong analytical, troubleshooting, documentation, and stakeholder communication skills.
Key Responsibilities
Design, build, and maintain batch and streaming data pipelines using Databricks, PySpark, Python and SQL.
Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
Schedule and orchestrate data workflows; implement monitoring, alerting, error handling, and recovery mechanisms.
Implement data validation, reconciliation, automated testing, and data quality controls.
Apply data governance and access controls using Unity Catalog and organizational security standards.
Collaborate with architects, analysts, and business stakeholders to translate requirements into technical solutions.
Maintain version-controlled code, support CI/CD processes, and document pipelines, data models, and operational procedures.
Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.
͏
Required Skills and Experience
Hands-on experience developing data engineering solutions on Databricks.
Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
Experience with Delta Lake, including merge operations, schema management, and transactional data processing.
Experience building ETL/ELT pipelines, implementing incremental loads, and applying data warehousing and dimensional modeling concepts.
Experience with at least one cloud platform—Azure, AWS, or Google Cloud—and its storage and identity services.
Working knowledge of Databricks jobs, notebooks, compute, Unity Catalog, and access management.
Proficiency with Git, code reviews, automated testing, and deployment practices.
Strong analytical, troubleshooting, documentation, and stakeholder communication skills.
Key Responsibilities
Design, build, and maintain batch and streaming data pipelines using Databricks, PySpark, Python and SQL.
Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
Schedule and orchestrate data workflows; implement monitoring, alerting, error handling, and recovery mechanisms.
Implement data validation, reconciliation, automated testing, and data quality controls.
Apply data governance and access controls using Unity Catalog and organizational security standards.
Collaborate with architects, analysts, and business stakeholders to translate requirements into technical solutions.
Maintain version-controlled code, support CI/CD processes, and document pipelines, data models, and operational procedures.
Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.
͏
Required Skills and Experience
Hands-on experience developing data engineering solutions on Databricks.
Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
Experience with Delta Lake, including merge operations, schema management, and transactional data processing.
Experience building ETL/ELT pipelines, implementing incremental loads, and applying data warehousing and dimensional modeling concepts.
Experience with at least one cloud platform—Azure, AWS, or Google Cloud—and its storage and identity services.
Working knowledge of Databricks jobs, notebooks, compute, Unity Catalog, and access management.
Proficiency with Git, code reviews, automated testing, and deployment practices.
Strong analytical, troubleshooting, documentation, and stakeholder communication skills.
Key Responsibilities
Design, build, and maintain batch and streaming data pipelines using Databricks, PySpark, Python and SQL.
Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
Schedule and orchestrate data workflows; implement monitoring, alerting, error handling, and recovery mechanisms.
Implement data validation, reconciliation, automated testing, and data quality controls.
Apply data governance and access controls using Unity Catalog and organizational security standards.
Collaborate with architects, analysts, and business stakeholders to translate requirements into technical solutions.
Maintain version-controlled code, support CI/CD processes, and document pipelines, data models, and operational procedures.
Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.
Mandatory Skills: DataBricks - Data Engineering
.
Experience: 5-8 Years
.
Reinvent your world. We are building a modern Wipro. We are an end-to-end digital transformation partner with the boldest ambitions. To realize them, we need people inspired by reinvention. Of yourself, your career, and your skills. We want to see the constant evolution of our business and our industry. It has always been in our DNA - as the world around us changes, so do we. Join a business powered by purpose and a place that empowers you to design your own reinvention.
About Wipro
People
About 11,079 in 2024.Benefit plan filings (Form 5500)
Company
Listed as WIT.
H-1B visas
580 approved for new hires in fiscal 2025.US Citizenship and Immigration Services
Demand for Data Engineer roles
New Data Engineer postings at the employers we follow are up 13% over the last 6 weeks compared with the 6 before.
August 17Last week
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
About the role
We file this job under Data Engineer, by the work the posting describes.
Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.
Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.
An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.
US government figures for database architects
Median yearly pay, May 2025
$139,500
Expected change, 2025 to 2035
9.4% more jobsagainst 3.5% for all jobs
The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.