Data Engineer- 4

<div><strong>Position: Senior Data Engineer</strong></div><div><br></div><div>We are seeking an experienced <strong>Senior Data Engineer</strong> to design and implement high-performance data pipelines, ensuring the smooth flow of large datasets across systems. In this role, you will leverage your expertise in <strong>Databricks</strong>, <strong>Pyspark</strong>, and <strong>ETL design</strong> to build efficient data solutions. You will also contribute to optimizing database performance and ensure that data systems meet both scalability and security standards.</div><div><br></div><div><strong>Key Responsibilities:</strong></div><ul style="margin-top:0cm;" type="disc"><li>Develop, optimize, and maintain scalable <strong>ETL pipelines</strong> for processing large datasets using <strong>Databricks</strong> and <strong>Pyspark</strong>.</li><li>Apply advanced techniques such as <strong>chunking</strong> and <strong>partitioning</strong> to handle large file volumes efficiently.</li><li>Tuning and optimizing databases for better performance and storage efficiency.</li><li>Collaborate with cross-functional teams to ensure the architecture meets business requirements and data quality standards.</li><li>Design and implement solutions with an eye toward scalability and long-term performance.</li><li>Work with <strong>Azure</strong> services, with exposure to <strong>Azure Function Apps</strong>, <strong>Azure Data Factory</strong>, and <strong>Cosmos DB</strong> (preferred).</li><li>Communicate effectively with stakeholders to align on project goals and deliverables.</li></ul><div><strong>Key Skills and Experience:</strong></div><ul style="margin-top:0cm;" type="disc"><li><strong>Databricks</strong> (advanced-level skills in data engineering workflows)</li><li><strong>Pyspark</strong> (intermediate-level skills for processing big data)</li><li>Strong <strong>ETL design skills</strong>, particularly in partitioning, chunking, and database tuning for large datasets</li><li><strong>Azure</strong> experience (a plus, including <strong>Function Apps</strong>, <strong>Cosmos DB</strong>, and <strong>Azure Data Factory</strong>)</li><li>Excellent communication skills for effective collaboration with teams and stakeholders</li></ul>