Senior Data Engineer

- Performance and cost optimization (Photon, file sizing, Z‑ORDER, clustering/partitioning, caching, AQE, shuffle/skew mitigation). - Hands‑on experience building distributed data pipelines using Python, PySpark, and cloud‑native services across AWS and/or Microsoft Azure. - Demonstrated experience administering and optimizing relational and NoSQL data systems (e.g., PostgreSQL, Aurora, DynamoDB) and designing efficient, secure schemas.

- Delta Lake table design, schema evolution, expectations, and medallion‑architecture (bronze/silver/gold). - Databricks Workflows, Jobs, Auto Loader, COPY INTO, Structured Streaming, Databricks SQL Warehouses, and Unity Catalog governance models. - Performance and cost optimization (Photon, file sizing, Z‑ORDER, clustering/partitioning, caching, AQE, shuffle/skew mitigation).

- Experience applying agile development methodologies, including backlog management, sprint planning, and iterative delivery. - Experience in AI/ML DevSecOps or agentic‑AI engineering, including Databricks Model Serving, Genie, Agent Bricks, or integration with AWS Kiro or other government‑approved platforms. Mandatory Qualifications & Requirements

- Databricks Data Engineer Professional Certification - At least one cloud-focused certification (e.g. AWS Developer Associate, Solutions Architect Associate/Professional, or equivalent). - Preferred but not required: Databricks Lakehouse Architect, Generative AI Associate/Professional, or related ML/AI certifications.

- Demonstrated experience administering and optimizing relational and NoSQL data systems (e.g., PostgreSQL, Aurora, DynamoDB) and designing efficient, secure schemas. - Experience developing and maintaining CI/CD pipelines using Git‑based workflows (e.g., GitHub Actions, Azure DevOps), Databricks Repos, Asset/Bundles, Terraform/IaC, and automated testing frameworks. - Experience supporting federal environments and compliance‑aligned development.

Position Overview SiloSmashers is hiring for a Senior Data Engineer to support a government customer. This position will be responsible for architecting, developing, optimizing, and maintaining high‑performance data ingestion, transformation, streaming, and analytical pipelines within Databricks‑enabled Lakehouse environments. The selected candidate will serve as a senior technical expert for Databricks Lakehouse engineering, advanced Delta Lake optimization, governed data design (Unity Catalog), medallion‑architecture enforcement, structured streaming, and platform‑integrated cloud data engineering across enterprise platforms. This role is expected to collaborate closely with the Project Manager / Contractor Lead, federal technical teams, and cross‑task platform, SRE, governance, privacy, and development personnel. Key Responsibilities

Key Responsibilities - Serve as the principal technical authority for Databricks data engineering, pipeline optimization, streaming, and Lakehouse standards across programs. - Lead design reviews, technical roadmaps, and architectural decisions in partnership with federal SMEs and platform/security teams.

- Implement advanced performance tuning and cost‑efficiency strategies, including cluster/sizing optimization, serverless compute adoption, workload health monitoring, and resource governance. - Provide root cause analysis, disruption/outage reporting, and production support for Databricks jobs, pipelines, applications, and platform features. - Coordinate with Privacy & Governance teams to enforce Unity Catalog RBAC/ABAC, row/column‑level security, governed tags, lineage, and compliance controls.

Required Skills - Demonstrated expertise designing and implementing Databricks Lakehouse solutions, including: - Delta Lake table design, schema evolution, expectations, and medallion‑architecture (bronze/silver/gold).

- Bachelor's or master's degree in computer science, information systems, engineering, or related field; or twelve (12) total years of equivalent experience in lieu of a degree. - Databricks Data Engineer Professional Certification - At least one cloud-focused certification (e.g. AWS Developer Associate, Solutions Architect Associate/Professional, or equivalent).

- At least one cloud-focused certification (e.g. AWS Developer Associate, Solutions Architect Associate/Professional, or equivalent). - Preferred but not required: Databricks Lakehouse Architect, Generative AI Associate/Professional, or related ML/AI certifications. - This position requires regular, physical attendance at customer sites.

- Lead design reviews, technical roadmaps, and architectural decisions in partnership with federal SMEs and platform/security teams. - Build and optimize batch and streaming pipelines using PySpark, Lakeflow Spark Declarative Pipelines, Delta Lake, Auto Loader, and Workflows with robust error handling, retries, alerting, and monitoring. - Lead and enforce medallion‑architecture ingestion and refinement patterns, data lifecycle management, and governed data storage tiering.

- Demonstrated expertise designing and implementing Databricks Lakehouse solutions, including: - Delta Lake table design, schema evolution, expectations, and medallion‑architecture (bronze/silver/gold). - Databricks Workflows, Jobs, Auto Loader, COPY INTO, Structured Streaming, Databricks SQL Warehouses, and Unity Catalog governance models.

- Hands‑on experience building distributed data pipelines using Python, PySpark, and cloud‑native services across AWS and/or Microsoft Azure. - Demonstrated experience administering and optimizing relational and NoSQL data systems (e.g., PostgreSQL, Aurora, DynamoDB) and designing efficient, secure schemas. - Experience developing and maintaining CI/CD pipelines using Git‑based workflows (e.g., GitHub Actions, Azure DevOps), Databricks Repos, Asset/Bundles, Terraform/IaC, and automated testing frameworks.