Lead Data Engineer

Lead Data Engineer Description As a Lead Data Engineer at Koantek, you will act as the technical leader, driving advanced data engineering strategies to support critical business decisions for our clients. Your primary focus will be on designing, architecting, and governing robust, large-scale data solutions and pipelines across multiple cloud environments. You will be pivotal in managing the technical roadmap, mentoring junior and mid-level engineers, and ensuring the delivery of high-quality, scalable data infrastructure. This role requires exceptional technical depth, strong architectural expertise, and proven leadership in client-facing environments. The Impact You Will Have Define and Govern Architecture: Own the technical vision and final design documentation for complex customer engagements, serving as the ultimate technical lead and architectural authority for the entire project lifecycle. Drive Big Data Strategy: Lead the implementation of comprehensive big data projects, including the strategic development, estimation, and deployment of innovative big data and AI applications. Establish and Enforce Standards: Define, govern, and enforce Databricks and data engineering best practices and architectural standards across all projects to ensure solution quality, performance, and maintainability. Project Leadership & Risk Management: Lead the technical estimation process, managing execution risks within customer proposals and statements of work, and ensuring the technical team delivers on time and within scope. Champion Mentorship and Growth: Actively lead, mentor, and guide a team of Data Engineers. Facilitate advanced knowledge transfer, provide training, and develop standardized, reusable documentation (a Center of Excellence) for the consulting practice. Enhance Consulting Excellence: Share deep architectural expertise and offer strategic best practices for client engagement, significantly enhancing the effectiveness and efficiency of other consulting teams. Requirements Minimum Qualifications: Educational Background: Bachelor’s degree in Computer Science, Information Technology, or a related field (or equivalent experience). Experience: 6+ years of experience as a Data Engineer, with a minimum of 2+ years in a Lead, Senior, or Mentoring role, demonstrating ownership over solution architecture. Expert-level proficiency in at least two major cloud platforms (AWS, Azure, GCP), specializing in data services. Extensive, proven experience in designing, developing, and leading the implementation of comprehensive, large-scale data engineering solutions using Databricks for batch and streaming pipelines. Demonstrated ability to architect scalable streaming and batch solutions using cloud-native and managed components. Deep practical experience applying and governing DataOps principles, including implementing and automating CI/CD and DevOps practices within multi-cloud data environments. Technical Skills: Mastery-level proficiency in Spark Scala, Python, and PySpark. Expert knowledge of data architecture, including Spark Streaming, Spark Core, Spark SQL, and advanced data modeling techniques (e.g., Data Vault, Kimball). Deep hands-on experience with advanced data management and integration technologies, such as Kafka, StreamSets, MapReduce, and modern ETL/ELT tooling. Proficient in guiding the application of advanced analytics and machine learning frameworks (e.g., Apache Spark MLlib, TensorFlow, PyTorch) to drive complex data insights. Databricks Specific Leadership Skills: Architectural Ownership: Proven ability to lead data migration projects from on-premises to cloud environments, defining the technical strategy for solution implementation on Databricks across AWS, Azure, and GCP. Governance & Performance: Expertise in designing and executing end-to-end data engineering solutions on Databricks, with a focus on performance tuning, cost optimization, and operational stability for large-scale production workloads. Administration & Operations: Proven experience with advanced Databricks administration and operations, including cluster management policies, security configuration, Unity Catalog governance, and job orchestration at an organizational level. Integration Leadership: Deep experience integrating Databricks as the core data processing layer with various upstream and downstream tools and platforms (BI, ML platforms, data governance tools). Good to Have Certifications: Certification in Databricks Engineering (Professional) Microsoft Certified: Azure Data Engineer Associate / Azure Solutions Architect Expert GCP Certified: Professional Google Cloud Architect AWS Certified Solutions Architect Professional