Lead Software Engineer - Data Ingestion - IT (Information Technology) - Bangalore, India

ResponsibilitiesData Ingestion SpecialistRole SummaryWe are looking for a Medior Data Ingestion Specialist to support development, operation, and maintenance of data ingestion platform orchestrated by Apache Airflow. The role focuses on onboarding and maintaining data integrations from multiple source systems, troubleshooting operational issues, and supporting release and governance processesKey ResponsibilitiesData Integration & Pipeline DevelopmentConfigure and maintain data ingestion pipelinesDevelop and support Python-based ingestion componentsIntegrate data:Relational databases (Oracle, PostgreSQL, MS SQL, MySQL, DB2, etc)Apache KafkaAWS S3, flat filesElasticsearch, Hive, Impala, hBaseTroubleshoot data flow, connectivity, and integration issuesIngestion to Databricks, Databricks loaderPlatform OperationsSupport incident, problem, and change management activitiesMaintain Python libraries and dependenciesCreate ETL/ELT pipelines focused on source ingestionMonitor pipeline execution and resolve operational issuesAssist with platform upgrades, patching, and maintenance activitiesRelease & GovernanceSupport Dev/Test/Prod deployment processesReview configuration and template changesEnsure compliance with governance, security, and operational standardsMaintain technical and operational documentationCollaborationWork closely with architects, engineering teams, and platform ownersCollaborate with security, audit, and infrastructure teamsParticipate in release planning and communication activitiesNice-to-Have SkillsKnowledge of ETL Python frameworks (e.g. Meltano)Experience with other ETL/ELT toolsExperience with Git, CI/CD pipelines, and DevOps practicesKnowledge of Linux environmentsExperience with Jira and ITSM toolingExposure to cloud-based data platformsCandidate ProfileMedior-level engineer capable of working independently on standard tasksAble to create and resolve customers' requirementsAble to analyse integration issues and identify root causesComfortable working in an enterprise environment with governance and security requirementsGood communication skills and ability to collaborate with multiple stakeholdersProfile requiredMust-Have Skills2+ years of experience with Python developmentPractical (hands on) experience with Apache NiFiPractical (hands on) experience with Apache AirflowExperience integrating data from relational databasesETL/ELT performance tuning, optimalizationsBasic experience with Apache KafkaBasic experience with ElasticsearchSQL knowledge and understanding of data integration conceptsExperience troubleshooting technical and connectivity issuesUnderstanding of incident and release management processesAbility to create and maintain technical documentationStrong analytical and problem-solving skillsWhy join us“We are committed to creating a diverse environment and are proud to be an equal opportunity employer. All qualified applicants receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status”.Business insightAt Société Générale, we are convinced that people are drivers of change, and that the world of tomorrow will be shaped by all their initiatives, from the smallest to the most ambitious. Whether you’re joining us for a period of months, years or your entire career, together we can have a positive impact on the future. Creating, daring, innovating, and taking action are part of our DNA. If you too want to be directly involved, grow in a stimulating and caring environment, feel useful on a daily basis and develop or strengthen your expertise, you will feel right at home with us!Still hesitating?You should know that our employees can dedicate several days per year to solidarity actions during their working hours, including sponsoring people struggling with their orientation or professional integration, participating in the financial education of young apprentices, and sharing their skills with charities. There are many ways to get involved.We are committed to support accelerating our Group’s ESG strategy by implementing ESG principles in all our activities and policies. They are translated in our business activity (ESG assessment, reporting, project management or IT activities), our work environment and in our responsible practices for environment protection.