Pay
In line with similar jobs
$140K–$150K against $127K–$190K, the middle half of what similar US postings state.
From 2,133 postings in the last year. Stated ranges, not salaries paid.
Lead Data Engineer with Databricks
Saint Louis, Missouri, United States
Live, first seen by us 53 days ago- Pay
USD 140,000–150,000/year — pay source
Salary 140,000 – 150,000 USD per year Know what it’s like to work and grow at Brillio: Click here
Read the full posting- Work setting
- Not stated
- Employment
- Not stated
What you’ll work on
Full postingDesign, develop, and maintain ETL/ELT pipelines using PySpark and Python in Databricks
Implement medallion architecture and optimize Spark jobs for performance and cost
Provision cloud infrastructure with Terraform for Databricks and cloud storage
Implement and maintain CI/CD pipelines using Jenkins and GitHub
Monitor pipeline performance, troubleshoot failures, and set up alerting
From the employer’s posting
Design, develop, and maintain ETL/ELT pipelines using PySpark and Python in Databricks
Implement medallion architecture (bronze/silver/gold layers) and optimize Spark jobs for performance, cost, scalability, and reliability
Provision and manage cloud infrastructure using Terraform (IaC) for Databricks workspaces, clusters, jobs, storage (ADLS/S3), networking, IAM roles/permissions, and related resources on Azure and/or AWS.
Implement and maintain CI/CD pipelines using Jenkins, GitHub (Actions/Repositories), and branching strategies
Monitor pipeline performance, troubleshoot failures, and implement alerting/observability
About Brillio-2
Brillio is one of the fastest growing digital technology service providers and a partner of choice for many Fortune 1000 companies seeking to turn disruption into a competitive advantage through innovative digital adoption.
In the employer’s words · Read in context
Posting record
Report an error- Status in our records
- Live
- First seen by us
- Sep 13, 202653 days ago
- Recorded sightings
- 112
- Last seen by us
- Nov 5, 2026
- Employer says posted
- Sep 4, 2026
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
How this job compares
Time since we first saw it
Day 53
Similar jobs usually come down between day 6 and day 54.
Well past Brillio-2’s usual: its jobs come down by about day 25.
From 11,172 similar postings that came down, and 58 of Brillio-2’s. Coming down is not always a hire.
Brillio-2 is hiring
46 open jobs
Open jobs by the week we first saw them, last 12 weeks.
- Data Engineer hiring has slowed
- Takes jobs down at a usual pace
See how this role fits your experience
Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
- Python
- SQL
- AWS
- Azure
- Databricks
- Delta Lake
- S3
- Spark
- Terraform
- PySpark
Where the posting names them
Key Responsibilities • Design, develop, and maintain ETL/ELT pipelines using PySpark and Python in Databricks (including notebooks, jobs, Delta Lake tables, and Unity Catalog for governance). • Implement medallion architecture (bronze/silver/gold layers) and optimize Spark jobs for performance, cost, scalability, and reliability (handling partitioning, skew, caching, adaptive query execution, etc.).
Required Skills & Experience • Strong proficiency in Python and PySpark for distributed data processing and ETL. • Advanced SQL skills with experience in complex querying, window functions, and optimization.
• Implement medallion architecture (bronze/silver/gold layers) and optimize Spark jobs for performance, cost, scalability, and reliability (handling partitioning, skew, caching, adaptive query execution, etc.). • Write efficient SQL queries for data transformation, validation, and analytics within Databricks. • Provision and manage cloud infrastructure using Terraform (IaC) for Databricks workspaces, clusters, jobs, storage (ADLS/S3), networking, IAM roles/permissions, and related resources on Azure and/or AWS.
• Strong proficiency in Python and PySpark for distributed data processing and ETL. • Advanced SQL skills with experience in complex querying, window functions, and optimization. • Hands-on experience with Databricks (clusters, notebooks, Delta Lake, Unity Catalog, Delta Live Tables, workflows/jobs).
• Write efficient SQL queries for data transformation, validation, and analytics within Databricks. • Provision and manage cloud infrastructure using Terraform (IaC) for Databricks workspaces, clusters, jobs, storage (ADLS/S3), networking, IAM roles/permissions, and related resources on Azure and/or AWS. • Implement and maintain CI/CD pipelines using Jenkins, GitHub (Actions/Repositories), and branching strategies for automated testing, deployment of notebooks, jobs, Delta Live Tables, and Terraform configurations.
• Solid knowledge of CI/CD practices and tools, particularly Jenkins (pipelines, plugins for Databricks/GitHub/Terraform). • Working experience on Azure (Data Lake, Data Factory, Synapse, Key Vault, etc.) and/or AWS (S3, Glue, EMR, IAM, Lambda, etc.). • Understanding of big data concepts, data modeling (star/snowflake, dimensional), and lakehouse principles.
• Ensure data quality, lineage, security, and compliance (including Delta Lake ACID transactions, schema evolution, time travel, and access controls). • Monitor pipeline performance, troubleshoot failures, and implement alerting/observability (using Databricks tools, cloud monitoring services, or third-party solutions). • Optimize cloud costs through auto-scaling clusters, spot instances, job scheduling, and efficient resource usage.
• Advanced SQL skills with experience in complex querying, window functions, and optimization. • Hands-on experience with Databricks (clusters, notebooks, Delta Lake, Unity Catalog, Delta Live Tables, workflows/jobs). • Proficiency in Terraform for infrastructure provisioning and management (Databricks resources, cloud storage, IAM, networking).
• Hands-on experience with Databricks (clusters, notebooks, Delta Lake, Unity Catalog, Delta Live Tables, workflows/jobs). • Proficiency in Terraform for infrastructure provisioning and management (Databricks resources, cloud storage, IAM, networking). • Experience with GitHub for version control and collaboration (branching, pull requests, code reviews).
• Experience with GitHub for version control and collaboration (branching, pull requests, code reviews). • Solid knowledge of CI/CD practices and tools, particularly Jenkins (pipelines, plugins for Databricks/GitHub/Terraform). • Working experience on Azure (Data Lake, Data Factory, Synapse, Key Vault, etc.) and/or AWS (S3, Glue, EMR, IAM, Lambda, etc.).
• Understanding of big data concepts, data modeling (star/snowflake, dimensional), and lakehouse principles. • Familiarity with performance tuning in Spark/Databricks environments. Additional Information: This role requires presence in St. Louis and attendance at the client office three days a week.
• Provision and manage cloud infrastructure using Terraform (IaC) for Databricks workspaces, clusters, jobs, storage (ADLS/S3), networking, IAM roles/permissions, and related resources on Azure and/or AWS. • Implement and maintain CI/CD pipelines using Jenkins, GitHub (Actions/Repositories), and branching strategies for automated testing, deployment of notebooks, jobs, Delta Live Tables, and Terraform configurations. • Integrate data from diverse sources (databases, APIs, streaming, files) into cloud storage and processing layers.
• Integrate data from diverse sources (databases, APIs, streaming, files) into cloud storage and processing layers. • Ensure data quality, lineage, security, and compliance (including Delta Lake ACID transactions, schema evolution, time travel, and access controls). • Monitor pipeline performance, troubleshoot failures, and implement alerting/observability (using Databricks tools, cloud monitoring services, or third-party solutions).
• Design, develop, and maintain ETL/ELT pipelines using PySpark and Python in Databricks (including notebooks, jobs, Delta Lake tables, and Unity Catalog for governance). • Implement medallion architecture (bronze/silver/gold layers) and optimize Spark jobs for performance, cost, scalability, and reliability (handling partitioning, skew, caching, adaptive query execution, etc.). • Write efficient SQL queries for data transformation, validation, and analytics within Databricks.
Your next step
Complete your application on jobs.lever.co. The employer’s form will show what is required.
Applying through Lever
- No account needed: one form, and Lever reads your resume into the profile recruiters search.
Already applied? Track this application
Demand for Data Engineer roles
New Data Engineer postings at the employers we follow are up 8% over the last 5 weeks compared with the 5 before.
Each bar is one week of new postings. The darkest bars are the last 5 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
About the role
We file this job under Data Engineer, by the work the posting describes.
Data engineers build and run the pipelines that copy data out of apps and outside services into a warehouse or lake, and fix them when a source changes or a load fails.
Expect SQL, Python, a cloud platform, a scheduler such as Airflow, and often tools for large data such as Spark or Kafka.
An analytics engineer works one step later, turning the raw data a data engineer lands into clean tables people can query.
US government figures for database architects
- Median yearly pay, May 2025
- $139,500
- Expected change, 2025 to 2035
- 9.4% more jobsagainst 3.5% for all jobs
The government has no code for data engineers, so these figures are for the closest group, database architects, which includes data integration and data warehousing specialists.
Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1243.
Job description
About Brillio:
Brillio is one of the fastest growing digital technology service providers and a partner of choice for many Fortune 1000 companies seeking to turn disruption into a competitive advantage through innovative digital adoption. Brillio, renowned for its world-class professionals, referred to as "Brillians", distinguishes itself through their capacity to seamlessly integrate cutting-edge digital and design thinking skills with an unwavering dedication to client satisfaction.
Brillio takes pride in its status as an employer of choice, consistently attracting the most exceptional and talented individuals due to its unwavering emphasis on contemporary, groundbreaking technologies, and exclusive digital projects. Brillio's relentless commitment to providing an exceptional experience to its Brillians and nurturing their full potential consistently garners them the Great Place to Work® certification year after year.
Lead Data Engineer
Job requirements
Key Responsibilities
• Design, develop, and maintain ETL/ELT pipelines using PySpark and Python in Databricks (including notebooks, jobs, Delta Lake tables, and Unity Catalog for governance).
• Implement medallion architecture (bronze/silver/gold layers) and optimize Spark jobs for performance, cost, scalability, and reliability (handling partitioning, skew, caching, adaptive query execution, etc.).
• Write efficient SQL queries for data transformation, validation, and analytics within Databricks.
• Provision and manage cloud infrastructure using Terraform (IaC) for Databricks workspaces, clusters, jobs, storage (ADLS/S3), networking, IAM roles/permissions, and related resources on Azure and/or AWS.
• Implement and maintain CI/CD pipelines using Jenkins, GitHub (Actions/Repositories), and branching strategies for automated testing, deployment of notebooks, jobs, Delta Live Tables, and Terraform configurations.
• Integrate data from diverse sources (databases, APIs, streaming, files) into cloud storage and processing layers.
• Ensure data quality, lineage, security, and compliance (including Delta Lake ACID transactions, schema evolution, time travel, and access controls).
• Monitor pipeline performance, troubleshoot failures, and implement alerting/observability (using Databricks tools, cloud monitoring services, or third-party solutions).
• Optimize cloud costs through auto-scaling clusters, spot instances, job scheduling, and efficient resource usage.
• Collaborate in agile teams, participate in code reviews, and contribute to best practices for data engineering.
Required Skills & Experience
• Strong proficiency in Python and PySpark for distributed data processing and ETL.
• Advanced SQL skills with experience in complex querying, window functions, and optimization.
• Hands-on experience with Databricks (clusters, notebooks, Delta Lake, Unity Catalog, Delta Live Tables, workflows/jobs).
• Proficiency in Terraform for infrastructure provisioning and management (Databricks resources, cloud storage, IAM, networking).
• Experience with GitHub for version control and collaboration (branching, pull requests, code reviews).
• Solid knowledge of CI/CD practices and tools, particularly Jenkins (pipelines, plugins for Databricks/GitHub/Terraform).
• Working experience on Azure (Data Lake, Data Factory, Synapse, Key Vault, etc.) and/or AWS (S3, Glue, EMR, IAM, Lambda, etc.).
• Understanding of big data concepts, data modeling (star/snowflake, dimensional), and lakehouse principles.
• Familiarity with performance tuning in Spark/Databricks environments.
Additional Information:
This role requires presence in St. Louis and attendance at the client office three days a week.
Know more about DAE: https://www.brillio.com/services-data-analytics/
Know what it’s like to work and grow at Brillio: https://www.brillio.com/join-us/
Equal Employment Opportunity Declaration
Brillio is an equal opportunity employer to all, regardless of age, ancestry, colour, disability (mental and physical), exercising the right to family care and medical leave, gender, gender expression, gender identity, genetic information, marital status, medical condition, military or veteran status, national origin, political affiliation, race, religious creed, sex (includes pregnancy, childbirth, breastfeeding, and related medical conditions), and sexual orientation.
#LI-SRI
Salary
140,000 – 150,000 USD per year
Know what it’s like to work and grow at Brillio: Click here
Employment type
Employee
Source details
Source notes
Source excerptsSelected passages from the saved posting. Check the full description for conditions and exceptions.
- Pay
Salary 140,000 – 150,000 USD per year Know what it’s like to work and grow at Brillio: Click here
- Location & working pattern
Saint Louis, Missouri, United States
Working pattern and location restrictions need checking in the full posting.
- Work authorization
No clear work-authorization passage found. Eligibility is unconfirmed.