Data Engineering Consultant

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

The Data Engineer is responsible for building, testing, and productionizing enterprise data products based on architecture, data models, and technical designs established by the Data Architect for Optum Insight.

This is a hands-on engineering role requiring solid general data engineering capabilities across SQL, PySpark, Databricks, ETL/ELT, and cloud-based data platforms. The Data Engineer will transform architecture and modeling specifications into scalable, reliable, high-quality datasets spanning bronze, silver, and gold layers.

The ideal candidate is a solid engineering generalist who can understand established data design, translate transformation and integration requirements into production-quality code, troubleshoot complex data issues, and collaborate effectively with Data Architects, Product Owners, analysts, and other Data Engineers.

The role does not require deep specialization in a single data technology or ownership of enterprise architecture. Success requires solid engineering fundamentals, technical versatility, attention to data quality, and the ability to execute effectively within an architecture established by others.

Primary Responsibilities:

• Build and maintain enterprise data products according to architecture, data models, transformation rules, and technical specifications established by the Data Architect
• Develop datasets and data pipelines across bronze, silver, and gold layers using modern medallion architecture principles
• Develop complex SQL and PySpark transformations that integrate data from multiple source systems
• Implement source-to-target mappings, transformation rules, business logic, joins, aggregations, and data-quality controls defined by the architecture and data-product design
• Build scalable ETL/ELT pipelines that ingest, transform, validate, and publish enterprise data
• Develop and maintain data pipelines, notebooks, jobs, and workflows using Databricks and Microsoft Azure technologies
• Translate logical and physical data models into working datasets and production data structures
• Implement incremental processing, historical data handling, reconciliation, deduplication, error handling, and restart/recovery capabilities
• Optimize SQL, Spark jobs, pipelines, and storage structures for reliability, scalability, and performance
• Perform unit testing, integration testing, data reconciliation, and data-quality validation before promoting data products into production
• Troubleshoot data pipeline failures, incorrect transformations, data-quality issues, and performance problems
• Participate in code reviews and technical design reviews and incorporate feedback from the Data Architect and senior engineering team
• Follow established architecture standards, engineering patterns, naming conventions, security requirements, and development practices
• Document data pipelines, transformations, dependencies, data mappings, and operational support procedures
• Collaborate with Data Architects, analysts, Product Owners, Finance stakeholders, and other engineers throughout the data-product development lifecycle
• Support deployment, monitoring, troubleshooting, and ongoing maintenance of production data products
• Contribute reusable engineering patterns and components that improve development efficiency and consistency across data products
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Data Quality & Governance
The Data Engineer should understand and implement data-quality and governance requirements as part of engineering delivery

• Implementing data-quality rules defined within the data-product design
• Validating completeness, accuracy, uniqueness, consistency, and referential integrity
• Building reconciliation checks between source systems and target data products
• Implementing appropriate logging and exception handling
• Maintaining appropriate technical metadata
• Supporting data lineage and traceability
• Following established data security and access-control standards
• Working within enterprise data-retention, privacy, and governance requirements

Required Qualifications:

• Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, or a related discipline, or equivalent relevant professional experience
• 5+ years of experience in data engineering, data warehousing, software engineering, analytics engineering, or related data disciplines
• 3+ years of solid hands-on experience developing production data pipelines and datasets using modern cloud data technologies
• Hands-on experience with Databricks or a comparable modern cloud data platform
• Solid hands-on experience with SQL
• Solid hands-on development experience with PySpark / Apache Spark
• Experience implementing ETL/ELT pipelines
• Experience integrating data from multiple source systems
• Experience building and maintaining production-grade datasets
• Experience with Microsoft Azure or comparable cloud infrastructure
• Experience with Git-based source control and modern engineering-development practices
• Understanding of medallion architecture and bronze, silver, and gold data layers
• Demonstrated ability to work effectively from architecture specifications and established technical designs
• Demonstrated solid debugging, analytical, and problem-solving skills
• Software & Technical Tools
  • Data Engineering & Processing
    • Databricks: Hands-on experience developing notebooks, jobs, workflows, Delta tables, and production data pipelines
    • PySpark: Solid hands-on experience developing distributed data-transformation and processing workloads
    • Delta Lake: Experience building and managing reliable lakehouse datasets and understanding concepts such as ACID transactions, schema evolution, and incremental processing
    • Apache Spark: Good understanding of distributed data processing, partitioning, transformations, performance optimization, and troubleshooting
    • SQL: Solid proficiency developing complex queries, joins, CTEs, window functions, aggregations, transformations, reconciliation logic, and data-quality validations
  • Cloud & Data Platforms
    • Experience building cloud-based data solutions on Microsoft Azure
    • Experience working with Azure Data Lake Storage Gen2 (ADLS Gen2)
    • Experience with Azure Data Factory or comparable orchestration technologies
    • Familiarity with Databricks Unity Catalog and modern data-governance capabilities
  • Data Architecture & Modeling
    • The Data Engineer is not expected to define enterprise architecture but should have sufficient architecture knowledge to correctly implement an established design
      • Knowledge of the following:
        • Bronze, silver, and gold medallion architecture
        • Data Lake and Lakehouse concepts
        • Relational data modeling
        • Dimensional modeling concepts
        • Primary and foreign keys
        • Data grain and relationships
        • Fact and dimension structures
        • Historical data management
        • Slowly changing dimensions
        • Source-to-target mappings
        • Schema evolution
        • Batch and incremental processing
        • Multi-source integration
        • Data-quality and reconciliation patterns
    • The Data Engineer should be able to review data model or architecture specification and translate the design into working datasets, transformation logic, and production pipelines
  • ETL / ELT & Pipeline Engineering
    • Solid understanding and practical experience with:
      • ETL and ELT development patterns
      • Batch data processing
      • Incremental and change-based data processing
      • Multi-source ingestion
      • Data transformation
      • Complex dataset joins
      • Data standardization and cleansing
      • Data reconciliation
      • Error handling and exception management
      • Data pipeline monitoring
      • Pipeline restart and recovery
      • Performance optimization
      • Production support and troubleshooting
  • Engineering & DevOps Tools
    • Experience with automated build, testing, and deployment practices
    • Experience following branching, peer-review, coding-standard, and release-management processes
    • Knowledge of Git for source control and collaborative development
    • Knowledge of Azure DevOps and/or GitHub for code repositories, pull requests, work management, and CI/CD
    • Familiarity with DataOps and modern software-engineering practices for data products
    • Demonstrated ability to diagnose production issues using platform logging, monitoring, and observability capabilities
    • Experience with Databricks Unity Catalog, Microsoft Purview, or comparable governance/catalog technologies is preferred.
  • Domain Knowledge
    • Domain expertise is secondary to solid data engineering capability for this role.
    • Experience working with Finance or healthcare datasets is preferred but not required.
    • Useful Finance experience includes exposure to:
      • General Ledger and accounting data
      • Billing and Accounts Receivable
      • Accounts Payable
      • Revenue
      • Financial reporting
      • Customer and vendor data
      • Product and contract data
      • Financial hierarchies and Chart of Accounts
      • Transactional and financial master/reference data
• Exposure to financial data from SAP S/4HANA, SAP ECC, Oracle ERP/eBusiness Suite, PeopleSoft, or comparable enterprise ERP platforms is beneficial

Preferred Qualifications:

• Experience with Databricks Lakehouse and Delta Lake
• Experience with Microsoft Azure, ADLS Gen2, and Azure Data Factory
• Experience with Databricks Unity Catalog
• Experience working with high-volume enterprise datasets
• Experience integrating data from enterprise ERP or financial applications
• Experience operating in large, complex, or regulated enterprise environments
• Snowflake experience 
• Experience with CDC and streaming/event-driven ingestion is a plus but is not a core requirement for this role unless needed by the assigned data product
• Healthcare experience involving claims, billing, provider, member, clinical, or operational datasets 
• Finance data experience
• Healthcare data experience
• Understanding of CI/CD and DataOps practices

Leadership & Communication Skills

This is primarily an execution-focused engineering role rather than an architecture or technical-leadership position.
The Data Engineer will:

• Work effectively from architecture, data models, and technical designs established by the Data Architect
• Ask appropriate technical questions when requirements or designs require clarification
• Communicate implementation challenges, dependencies, data-quality issues, and technical risks early
• Collaborate closely with Data Architects and senior engineers during implementation
• Participate constructively in design and code reviews
• Provide realistic engineering estimates and communicate progress and delivery risks
• Produce clear technical documentation
• Work collaboratively across distributed and cross-functional teams
• Take ownership of assigned engineering deliverables from development through production deployment
• Escalate material deviations from the architecture rather than independently changing the established design
• Contribute engineering observations and recommendations while respecting architecture-level design ownership

The Data Engineer should demonstrate solid technical ownership without requiring architecture ownership or formal people-management

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.