Lead Data Engineer

Lead data engineer

Location : Chennai Only , Toatl Exp : 8 to 17 Years Notice Period : Immediate to 30days

Role Summary

We are seeking a highly skilled and experienced Lead Data Engineer to drive the modernization and design the future-state architecture of our Sales Validation platform. As a technical leader, you will be responsible for guiding the transition from legacy systems to advanced data engineering solutions, primarily focused on Databricks and Azure cloud technologies. This role requires a passion for innovation, performance optimization, and building scalable, reusable engineering frameworks.

Key Responsibilities

• Lead the architecture, implementation, and delivery of Databricks solutions to support data processing and analytics initiatives.
• Oversee the migration efforts from Synapse Analytics to Databricks, ensuring minimal downtime and aligned best practices.
• Develop and enforce engineering standards, reusable frameworks, and documentation to promote consistency and efficiency across teams.
• Mentor data engineers through design reviews, knowledge sharing sessions, and technical guidance to foster professional growth and code quality.
• Own end-to-end delivery of multiple complex workstreams, coordinate across teams, and manage technical risks.
• Drive continuous performance tuning and scalability improvements for data pipelines and platforms.
• Collaborate with cross-functional teams including data scientists, architects, and business stakeholders to deliver aligned solutions.

Technical Skills Required

Databricks Ecosystem (Highest Priority):

• Advanced experience with Databricks and Spark frameworks using PySpark and Spark SQL.
• Deep understanding of Delta Lake architecture and Unity Catalog for secure data governance.
• Practical knowledge of Databricks workflows and asset bundles to orchestrate data pipelines effectively.
• Expertise with modern Lakehouse architecture principles to integrate data lakes and warehouses.

Microsoft Azure Platform:

• Hands-on experience architecting and managing Azure Data Factory (ADF) pipelines and ecosystem components.
• Proficiency with Azure Data Lake Storage Gen2 (ADLS Gen2) for secure and scalable storage solutions.
• Strong familiarity with Azure SQL databases and implementation of Azure security best practices.

Migration and Modernization:

• Experience migrating Synapse Pipelines, SQL Pools, Notebooks, and Data Flows into scalable Databricks platforms.
• Successful track record in modernizing legacy ETL workloads for improved reliability and performance.

Software Engineering & Development:

• Advanced SQL skills, data modeling expertise, and performance tuning abilities.
• Experience with GitLab CI/CD pipelines for automation of testing, deployment, and code quality checks.
• Strong adherence to code review best practices and collaborative development methodologies.

Preferred Skills and Experience

• Familiarity with Apache Airflow orchestration for task scheduling and workflow management.
• Exposure to Snowflake data platform technologies and data governance strategies.
• Experience deploying enterprise-scale Azure platforms with considerations for security, scalability, and operational excellence.

͏

Areas of Responsibility

In this role, you will participate actively in gathering detailed and structured requirements for various data engineering modules. You will engage with multiple business units to understand dependencies, process linkages, and end-user needs ensuring comprehensive and clear documentation.

You will also be involved in facilitating solution design workshops under project leadership guidance, ensuring the technical architecture aligns with both business objectives and compliance standards. This includes translating functional requirements into robust, scalable data solutions.

͏

Development and Implementation

You will provide technical oversight by reviewing developed code, resolving queries from team members, and creating standardized replication patterns to guarantee consistent deployment across environments. Your collaboration with project managers will span all stages of the software development lifecycle, ensuring timely and high-quality deliverables.

Root cause analysis and troubleshooting of recurring technical challenges will also be a critical part of your work. You will modify and optimize software components to enhance system performance, compatibility, and user interfaces.

͏

Deployment and User Enablement

Collaborating with technical and support teams, you will validate system configurations, oversee integration, and manage dependencies during deployment phases to ensure seamless rollouts of new applications. Additionally, you will prepare comprehensive user training materials and frameworks to facilitate swift adoption and minimize risks associated with system transitions.

Your role is pivotal in making certain that the platform not only meets technical standards but also provides intuitive and reliable user experiences.

͏

Core Competencies

• Client Centricity: Demonstrate a strong commitment to understanding and fulfilling client needs, ensuring solutions deliver measurable value.
• Passion for Results: Exhibit a persistent focus on achieving outcomes through dedication and proactive problem-solving.
• Execution Excellence: Deliver consistent quality and meet deadlines through disciplined planning and thorough validation.
• Collaborative Working: Foster teamwork and open communication across diverse stakeholders and disciplines.
• Learning Agility: Embrace change and continuously seek to expand knowledge and skills in evolving technologies.
• Problem Solving & Decision Making: Analyze complex issues, identify root causes, and implement effective, timely solutions.
• Effective Communication: Articulate ideas clearly and persuasively in both technical and non-technical contexts to diverse audiences.

Mandatory Skills: Enterprise Platform Engineering Backend

Position Title: Technical Lead L1

Company: Wipro Technologies, within the Wipro Engineering unit, specializing in CPPE practice.

Join Wipro Technologies and become part of a digital transformation partner committed to innovation and reinvention. We look for individuals inspired to evolve their careers, skills, and contributions as we shape the future together. With 5-8 years of experience, you will find opportunities to grow within a purpose-driven organization that values diversity, inclusion, and continuous learning.