Lead Engineer- Platform & Data

About the team: Eightfold offers a suite of AI-powered products, including Talent Acquisition, Talent Management, Workforce Exchange, and AI Interviewer. The Analytics team builds the customer-facing solutions that show how Eightfold delivers value across these products. Our work helps customers’ leadership teams make strategic workforce decisions and helps their recruiting teams operate more effectively day to day. Our analytics portfolio includes: Native reporting platform : a reporting experience built directly into the Eightfold product. Agentic Dashboard Authoring : AI-assisted dashboard creation, so users can go from question to insight without building reports by hand. Conversational Analytics : natural-language access to data, letting users ask questions and get answers in plain English. Secure Talent Lake connectivity : secure pipelines that let customers ETL their Eightfold data into their own data warehouses. We’re in the middle of a major revamp of our analytics stack. Today we run on Redshift and Databricks, and we’re migrating to StarRocks across 8+ regions distributed globally on AWS, Azure, and Oracle Cloud. As Lead Platform / Data Engineer, you’ll play a central role in shaping this migration and the multi-cloud platform that comes out of it. What You'll Do: Architect and manage the lakehouse. Design and maintain Apache Iceberg tables, including schema evolution, partitioning strategies, snapshot management, compaction, and metadata optimization. Own the analytical query layer. Deploy, operate, and performance-tune open-source datawarehouse clusters, covering FE/BE sizing, materialized views, query profiling, caching, and resource isolation. Optimize storage formats. Make deliberate choices about Parquet file layout, including row group sizing, compression codecs, encoding, and predicate pushdown, so data stays cheap to store and fast to query. Build and maintain ETL/ELT pipelines. Develop robust batch and streaming pipelines with strong guarantees around data quality, idempotency, backfills, and observability. Run event-driven autoscaling on Kubernetes. Design and implement KEDA-based scaling for data workloads, triggered by queue depth, Kafka lag, or custom metrics, so you balance cost against latency. Manage infrastructure as code. Provision and maintain cloud and Kubernetes infrastructure using tools like Terraform, Helm, and GitOps workflows (ArgoCD/Flux). Tune self-hosted open-source warehouses. Diagnose bottlenecks across CPU, memory, I/O, and network. Tune configurations, run benchmarks, and drive measurable gains in query latency and cost efficiency. Champion reliability. Define SLOs, build monitoring and alerting (Prometheus, Grafana, or similar), participate in on-call, and lead incident retrospectives. Mentor and influence. Review designs and code, mentor engineers, and help set technical direction for the data platform. What We Need: 5+ years of experience designing, building, and operating large-scale data platforms, lakehouse architectures, and distributed analytical systems Hands-on expertise with Apache Iceberg / Hudi and Parquet, including partitioning, schema evolution, compaction, snapshot management, and storage optimization. Experience operating and performance-tuning analytical query engines such as StarRocks, ClickHouse, Trino, or similar distributed data warehouses. Proficiency in building reliable batch and streaming pipelines using Apache Spark and Flink, with strong fundamentals in data quality, idempotency, fault tolerance, and observability. Experience operating production-grade, multi-region cloud platforms across AWS, Azure, or OCI, with a focus on scalability, reliability, performance, and cost efficiency. Prior experience in mentoring engineers and leading projects while driving platform initiatives end-to-end. About Eightfold: Eightfold is a global leader in an AI-native enterprise talent platform, trusted by the world’s largest & most respected Fortune 500 organizations. Our platform is built from the ground up, operating at scale across Azure and AWS, deployed in multiple regions globally, including IL4-compliant environments for US Government, supporting users in 100+ countries and 30+ languages. Today, Eightfold is at the forefront of agentic AI, delivering intelligent agents that actively drive outcomes across hiring and talent workflows, while much of the industry is still experimenting with prototypes. Backed by over $410M in funding and valued at $2B+, we are defining the next era of agentic talent systems. What sets Eightfold apart is not just the technology & our mission, but the team behind it. We are a deeply technical, execution-driven organization that values ownership, collaboration, and high standards. Our engineers, product leaders, and go-to-market teams work closely together in person and across functions to build systems that scale in the real world. If you’re excited to work on hard problems, move with urgency, raise the bar every day, and help build agentic systems that transform how the world works, Eightfold is the place to do it. Our customers Press Founders: Varun Kacholia ( https://www.linkedin.com/in/varunkacholia/ ) &  Ashutosh Garg ( https://www.linkedin.com/in/ashutoshgarg893/ ) Engineering Culture at Eightfold: We are a team of self-starters who excel in their fields. We believe in giving you responsibility, not a task. We want you to have ownership and pride in your work and see your work's positive impact on your colleagues, our customers, and the world. We believe in providing transparency and support so you can do the best work of your career.