Senior Data Platform Engineer
Position Summary:
We are expanding our Platform Engineering capability to build, secure, and automate the enterprise Data Platform on cloud and Databricks. In this role, you'll help maintain and improve the underlying infrastructure, ingestion frameworks, CI/CD pipelines, orchestration, and observability that enable Data Engineers and Analytics teams to operate at scale. The ideal candidate has solid platform engineering fundamentals with hands-on DevOps skills across data replication, job scheduling, deployment automation, and cloud operations, and is growing toward independent platform ownership.
Key Responsibilities:
Infrastructure & Platform Engineering:
- Deploy and maintain Databricks workspaces and cloud infrastructure using Infrastructure-as-Code.
Assist with platform upgrades, patching, new flow setup, and environment refresh support.
Support enterprise data replication (HVR) and file-based ingestion patterns from operational systems into the data platform.
Orchestration & Job Scheduling:
- Provide monitoring, recovery, and day-to-day operational support for enterprise job scheduling and orchestration.
- Configure job dependencies and coordinate with source teams on long-running workloads.
CI/CD & Deployment Automation:
- Build and maintain GitLab CI/CD pipelines for data and platform projects with automated deployment workflows.
- Apply standardized deployment patterns using reusable templates and Databricks-native deployment tooling.
- Follow branching strategies, code review policies, and environment promotion rules.
- Support the Change Request (CR) deployment lifecycle, including validation and ticket closure.
Monitoring, Reliability & Support:
- Configure monitoring, alerting, and logging to help ensure platform stability.
- Provide first-line support for platform-related incidents, escalating complex issues to senior engineers.
- Support year-end activities and compliance reporting requirements.
What Success Looks Like (First 6–12 Months):
- In your first 6–12 months, you'll independently handle routine platform operations, build and maintain CI/CD pipelines for key flows, and automate recurring operational tasks with guidance from senior engineers.
Required Qualifications:
- Bachelor's or Master's degree in Computer Science, Information Technology, or equivalent relevant experience.
- 4+ years of industry experience in Data Engineering, Cloud Infrastructure, or DevOps.
- Hands-on experience with CI/CD tooling (GitLab preferred) — pipeline authoring, release management, and secrets management.
- Working knowledge of cloud platforms (AWS preferred) for data workloads.
- Familiarity with Databricks platform administration.
- Experience with monitoring and observability tools, alerting, and incident triage.
- Proficient in Python and Bash/Shell scripting for automation.
Preferred Qualifications:
- Experience with enterprise data replication tools (e.g., HVR).
- Working Infrastructure-as-Code skills.
- Familiarity with enterprise job orchestration platforms (e.g., Autopilot).
- Exposure to Databricks Serverless Compute and Workflow orchestration.
- Cloud Solutions Architect or Databricks certifications are a plus.
Competencies:
- Reliability-first mindset — focus on stability, automation, and self-healing systems.
- Growing sense of ownership across the platform lifecycle — build, run, and improve.
- Effective cross-team collaboration and clear communication.
- Clear documentation and knowledge-sharing habits.