Specialist Software Engineer - Data Engineer Databricks pySpark - Innovation / Project / Organization - Bangalore, India

ResponsibilitiesSpecialist Software Engineer – Data Engineer (Databricks-pySpark)ResponsibilitiesUnderstand business and technical requirements and translate them into scalable, maintainable data engineering solutions.Own the design and development of robust, high‑performance data pipelines using Databricks(pySpark).Build and optimize data flows within the Hadoop ecosystem (Hive, HDFS, Starburst, Oozie).Ensure performance tuning, resource optimization, and efficient cluster usage on Databricks platform.Contribute to data architecture decisions related to storage, processing, security, and exposure.Apply industrialization best practices including CI/CD, unit testing, integration testing, and pipeline validation.Model data based on business needs and maintain shared data components, dictionaries, and governance rules.Perform functional and technical analysis, document specifications, and validate implemented solutions.Participate in Agile ceremonies and collaborate with Product Owners, architecture, business, and Ops teams.Support production processes, troubleshoot issues, update runbooks, and maintain system stability.Ensure compliance with data quality, traceability, and security standards across the data lifecycle.Engage in continuous improvement activities, proposing upgrades, refactoring, and automation opportunities.Work closely with global teams and support cross‑location coordination when required.Demonstrate ownership during key delivery phases, including planned on‑call or deployment activities.Profile requiredProfile RequiredExperienced Data Engineer with strong expertise in Databricks, Hadoop ecosystem components, and large‑scale data processing environments.Minimum QualificationsBachelor of Engineering in Computer Science, Information Technology and equivalent streams.Strong hands‑on experience in pySaprk for distributed processing and performance optimization.Solid understanding of Hadoop ecosystem tools: Hive, HDFS, YARN, Oozie, Starburst.Proven experience designing and maintaining data pipelines in complex Big Data environments.Good command of Python for supplemental processing and pipeline utilities.Strong understanding of data modeling, partitioning strategies, and governance principles.Proficiency in Git and modern SDLC practices (branching, PR reviews, tagging).Experience with CI/CD pipelines (GitHub Actions, Ansible, AWX).Strong documentation skills and ability to produce clear technical specifications.Preferred Qualifications4+ years of experience in data engineering or large‑scale distributed systems.Experience with NiFi/Kafka for batch and real‑time ingestion.Working knowledge of JFrog (artifact deployment, vulnerability remediation).Strong experience in Apache Hive performance tuning and Spark SQL integration.Exposure to Cloudera Manager for monitoring clusters and troubleshooting.Familiarity with Data Governance, security policies, and traceability frameworks.Experience in Agile @ Scale (SAFe) and backlog contribution (user stories, acceptance criteria).Ability to work independently and collaboratively across multicultural teams.Strong analytical, communication, and problem‑solving skills.cloud technologies (Azure/AWS) and modern platforms like Databricks.Why join us“We are committed to creating a diverse environment and are proud to be an equal opportunity employer. All qualified applicants receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status”Business insightAt Société Générale, we are convinced that people are drivers of change, and that the world of tomorrow will be shaped by all their initiatives, from the smallest to the most ambitious.Whether you’re joining us for a period of months, years or your entire career, together we can have a positive impact on the future. Creating, daring, innovating and taking action are part of our DNA.If you too want to be directly involved, grow in a stimulating and caring environment, feel useful on a daily basis and develop or strengthen your expertise, you will feel right at home with us!Still hesitating?You should know that our employees can dedicate several days per year to solidarity actions during their working hours, including sponsoring people struggling with their orientation or professional integration, participating in the financial education of young apprentices and sharing their skills with charities. There are many ways to get involved.We are committed to support accelerating our Group’s ESG strategy by implementing ESG principles in all our activities and policies. They are translated in our business activity (ESG assessment, reporting, project management or IT activities), our work environment and in our responsible practices for environment protection.