Senior Data Engineer

Function

IoT

Our Company

We’re Hitachi Digital Services, a global digital solutions and transformation business with a bold vision of our world’s potential. We’re people-centric and here to power good. Every day, we future-proof urban spaces, conserve natural resources, protect rainforests, and save lives. This is a world where innovation, technology, and deep expertise come together to take our company and customers from what’s now to what’s next. We make it happen through the power of acceleration.

Imagine the sheer breadth of talent it takes to bring a better tomorrow closer to today. We don’t expect you to ‘fit’ every requirement – your life experience, character, perspective, and passion for achieving great things in the world are equally as important to us.

Job description

About the Role 

We are seeking a Data Engineer to join our Data Science team within a complex engineering environment. 

Working closely with Data Scientists and engineering stakeholders, you will be responsible for translating analytical requirements into scalable, production-ready code using Databricks and modern data technologies. You will support the development of analytics solutions by extracting, transforming, and preparing data, building reusable datasets, and implementing analytical workflows. 

This role is ideal for someone who enjoys hands-on coding, problem solving, and working with large datasets, while contributing to real-world engineering and operational improvements. 

 

Key Responsibilities 

Data Engineering & Preparation 

• Build and maintain data pipelines within Databricks. 

• Extract, transform, and validate data from multiple sources. 

• Develop reusable datasets and analytical tables for Data Science projects. 

• Ensure data quality, consistency, and reliability across analytical workflows. 

Analytics Development 

• Translate requirements from Data Scientists into efficient and maintainable code. 

• Develop data processing solutions using PySpark, SQL, and Pandas. 

• Support feature engineering activities for machine learning and advanced analytics projects. 

• Investigate data anomalies and support root-cause analysis activities. 

• Optimise data queries and processing logic for performance and scalability. 

Collaboration 

• Work closely with Data Scientists to support model development and deployment. 

• Collaborate with engineering stakeholders to understand operational data sources and business requirements. 

• Contribute to code reviews and technical discussions. 

• Document datasets, code, and analytical processes. 

Continuous Improvement 

• Identify opportunities to improve existing data pipelines and analytical processes. 

• Promote coding standards and best practices. 

• Stay up to date with developments in Databricks, Spark, and modern data platforms. 

 

Essential Requirements 

• Experience developing analytical solutions using Databricks. 

• Strong proficiency in PySpark. 

• Strong SQL skills, including writing and optimising complex queries. 

• Experience using Python and Pandas for data analysis and transformation. 
• Familiar with Airflow, Kafka
• Experience with with Docker, Kubernetes is a plus
• Experience with other stream processing like Flink is a plus

• Experience working with large-scale datasets. 

• Ability to understand business requirements and translate them into technical solutions. 

• Strong problem-solving and debugging skills. 

• Good communication and stakeholder engagement skills. 

• Experience working in collaborative Agile teams. 

 

Desirable Requirements 

• Experience within engineering, rail, transportation, manufacturing, industrial, or asset-intensive industries. 

• Exposure to machine learning workflows and feature engineering. 

• Experience with Delta Lake and Spark optimisation. 

• Familiarity with Azure Data Platform technologies. 

• Knowledge of Power BI or Grafana. 

• Experience using Git/version control. 

 

About us

We’re a global, team of innovators. Together, we harness engineering excellence and passion to co-create meaningful solutions to complex challenges. We turn organizations into data-driven leaders that can make a positive impact on their industries and society. If you believe that innovation can bring a better tomorrow closer to today, this is the place for you.

Fostering innovation through diverse perspectives

Hitachi is a global company operating across a wide range of industries and regions. One of the things that sets Hitachi apart is the diversity of our business and people, which drives our innovation and growth.

We are committed to building an inclusive culture based on mutual respect and merit-based systems. We believe that when people feel valued, heard, and safe to express themselves, they do their best work.

How we look after you

We help take care of your today and tomorrow with industry-leading benefits, support, and services that look after your holistic health and wellbeing. We’re also champions of life balance and offer flexible arrangements that work for you (role and location dependent). We’re always looking for new ways of working that bring out our best, which leads to unexpected ideas. So here, you’ll experience a sense of belonging, and discover autonomy, freedom, and ownership as you work alongside talented people you enjoy sharing knowledge with.

We’re proud to say we’re an equal opportunity employer and welcome all applicants for employment without attention to race, colour, religion, sex, sexual orientation, gender identity, national origin, veteran, age, disability status or any other protected characteristic. Should you need reasonable accommodations during the recruitment process, please let us know so that we can do our best to set you up for success.