Data Engineer

The Data Engineer supports, develops, and maintains data and analytics platforms that enable reliable, scalable, and efficient access to data. This role partners with Business and IT teams to understand requirements and leverage modern data engineering technologies to deliver high-quality data solutions at scale. The Data Engineer will design, develop, and maintain data pipelines and data storage solutions, ensure data quality and integrity, and contribute to data governance, analytics, and cloud-based data platforms. The role works within Agile delivery environments and applies modern engineering practices to continuously improve data solutions.

Key Responsibilities

• Develop and maintain reliable, scalable, and efficient ETL/ELT data pipelines using technologies such as Azure Databricks, PySpark, Python/Scala, and SQL.
• Design and implement data ingestion and transformation solutions for a variety of data sources, including relational, event-based, and unstructured data.
• Build and maintain scalable Data Lake and Lakehouse solutions and optimize data processing and storage performance.
• Develop physical data models and implement data storage architectures in accordance with established design and engineering guidelines.
• Implement data quality checks, monitoring, alerting, and troubleshooting mechanisms to identify and resolve data quality and data integrity issues.
• Analyze complex data elements, data flows, dependencies, and relationships to contribute to conceptual, logical, and physical data models.
• Develop and operate large-scale data storage and processing solutions using distributed and cloud-based technologies.
• Work with Azure services such as Azure Data Lake Storage (ADLS), Event Hubs, and Azure Functions to support scalable data solutions.
• Implement and support data governance practices, including metadata management, data access, retention, and availability.
• Participate in testing, validation, troubleshooting, and continuous improvement of data pipelines and solutions.
• Collaborate with business stakeholders, analysts, data scientists, and IT teams to understand requirements and deliver analytics and data solutions.
• Apply Agile development practices, including Scrum, Kanban, DevOps, and continuous improvement, to deliver data-driven solutions.
• Document technical solutions, processes, data flows, and system dependencies to support knowledge sharing and effective solution maintenance.
• Apply appropriate engineering, security, governance, and compliance practices throughout the data development lifecycle.

Skills and Experience

• Strong programming skills in Python/PySpark or Scala, with the ability to develop, test, and maintain production-quality code.
• Strong knowledge of SQL and experience working with data extraction, transformation, and loading processes.
• Hands-on experience with Azure Databricks and cloud-based data engineering solutions.
• Experience with Azure data services, including ADLS, Event Hub, and Azure Functions.
• Good understanding of Data Lake, Delta Lake, Lakehouse, ETL/ELT, data modeling, and distributed data processing concepts.
• Exposure to or knowledge of Big Data technologies such as Spark, MapReduce, Hive, HBase, Kafka, or equivalent technologies.
• Experience or exposure to clustered compute and cloud-based implementations.
• Familiarity with designing solutions that support large-scale data movement and processing in cloud environments.
• Understanding of data quality, data integrity, metadata, governance, and data management principles.
• Exposure to analytical solutions, IoT technologies, and data-driven applications is beneficial.
• Familiarity with Agile software development, DevOps, Scrum, or Kanban methodologies.
• Strong problem-solving and analytical skills, with the ability to identify root causes and implement robust solutions.
• Strong communication and collaboration skills, with the ability to work effectively with technical and non-technical stakeholders.
• Customer-focused mindset with the ability to understand stakeholder needs and translate them into effective technical solutions.
• Ability to document solutions clearly and communicate technical information to different audiences.
• Strong focus on quality, solution validation, testing, security, governance, and continuous improvement.
• Ability to work collaboratively in diverse teams and recognize the value of different perspectives.

Core Competencies

• System Requirements Engineering
• Data Extraction and ETL/ELT
• Programming and Data Engineering
• Data Quality and Data Governance
• Solution Validation and Testing
• Problem Solving and Decision Quality
• Quality Assurance and Metrics
• Solution Documentation
• Customer Focus
• Collaboration and Effective Communication
• Continuous Improvement

Qualifications

• Bachelor's degree or equivalent qualification in Computer Science, Information Technology, Engineering, Data Science, or another relevant technical discipline, or equivalent relevant experience.
• 2–4 years of relevant Data Engineering experience preferred.
• Relevant experience through internships, co-op programs, temporary student employment, academic projects, or extracurricular technical activities may be considered for early-career candidates.
• Relevant certifications such as Databricks or Azure certifications are an advantage.
• This position may require licensing or compliance with applicable export control or sanctions regulations.