Senior Data Engineer

Job Title – Senior Data Engineer Position type- Full Time Work Location- Bangalore Working style- Hybrid Cab Facility- Yes Shift Time – 12:30 PM to 9:30 PM People Manager role: No Required education and certifications are critical for the role: bachelor’s or master’s degree in computer science, Engineering, or a related field. Required years of experience: 5+ years of relevant experience AON IS IN THE BUSINESS OF BETTER DECISIONS At Aon, we shape decisions for the better to protect and enrich the lives of people around the world. As an organization, we are committed to our purpose as one firm, united through trust as one inclusive, diverse team and we are passionate about helping our colleagues and clients succeed. Role Overview: Enterprise Data & Analytics Engineering team is leading the development of cloud-based products and analytics capability for Aon. With a diverse portfolio of projects, and a sizeable production estate supporting an already successful business, we are in a critical growth phase and need the best minds in industry to help advance our journey. The D&A Data Engineering team is responsible for building the data foundation that powers analytics, operational decision-making, and AI-driven products across the organization. As a Senior Data Engineer, you will play a critical role in designing, delivering, and operating scalable data platforms and data products that enable trusted, governed, and actionable data at enterprise scale. This role goes beyond traditional ETL development. You will help build and shape modern data architecture, implement engineering best practices, and build reusable data capabilities that support reporting, advanced analytics, machine learning, and emerging AI use cases. Working closely with business stakeholders, architects, product owners, and data scientists, you will transform complex data ecosystems into AI-ready reliable and consumable data products. Key Responsibilities: Design, develop, deploy, and operate scalable cloud-based data, analytics, automation and tooling solutions using modern data platforms and Cloud services. Support integration of AI-enabled capabilities by providing trusted data pipelines, feature-ready datasets, APIs, and operational controls required by downstream AI, analytics, and application services Apply engineering best practices as you design and deliver high-quality, scalable solutions. Own your solution end-to-end through design, implementation, and operations. Own production reliability of data systems, including monitoring, observability, incident response, and SLA adherence Collaborate cross-functionally across business stakeholders, product owners, and other engineers to deliver integrated, customer-focused and resilient data solutions. Contribute to the evolution of Aon’s data platforms through technical feedback and innovation. Contribute to engineering standards, documentation, and technical mentorship within the team. Skills/competencies required: Pipeline Performance & Spark Optimization Proven experience designing and optimizing Spark pipelines against multi-terabyte datasets in production Deep understanding of Spark memory management, caching strategies and cluster resource allocation Ability to read and act on Spark UI diagnostics, query plans, and job metrics to resolve performance bottlenecks Lakehouse & Delta Architecture Strong working knowledge of Delta Lake internals — MERGE optimization, Z-ordering/liquid clustering, schema evolution, Change Data Feed Ability to design and evolve layered, reliable data architectures (e.g., medallion or equivalent multi-stage patterns) that support reprocessing and scale without compromising data quality Pipeline & Orchestration Design Experience building idempotent, restartable data pipelines with robust incremental processing Hands-on experience with at least one major orchestration tool (Databricks Workflows, Airflow, or dbt), with sound judgment on tool selection Data Governance & Platform Enablement Experience with Unity Catalog (or equivalent) for access control, lineage, and data discovery across teams Understanding of data security practices — PII tagging, masking, row/column-level access control Ability to design scalable ownership and self-service access models rather than centralized, manual approval processes Foundational / General Competencies Strong proficiency in Python/PySpark and/or Scala, and SQL Solid grasp of distributed systems concepts (partitioning, parallelism, fault tolerance) Experience with DataOps and Engineering practices – Git, CI/CD Pipelines, Environment management Ability to communicate technical tradeoffs clearly to both engineering and non-engineering stakeholders Comfortable mentoring other engineers on performance, architecture, and best practices AI-Enabled Engineering Productivity Hands-on fluency with AI-assisted development tools (e.g., Claude Code, GitHub Copilot, Databricks Assistant) for code generation, refactoring, debugging, and documentation Track record of infusing AI tooling across the development lifecycle — design/spec drafting, coding, code review, testing, and documentation — not just at the code-generation step. 2575056