Systems Analyst III / Databricks Data Engineer
Client: Texas Health and Human Services Commission (HHSC)
Location: Austin, TX (Hybrid - 2 Days Onsite, 3 Days Remote)
Duration: 12+ Months (High possibility of extension)
Job Type: Contract (W2/1099/C2C)
Interview: Microsoft Teams / In-Person
Local Candidates Only
Job Summary
Texas Health and Human Services Commission (HHSC) is seeking an experienced Systems Analyst III / Databricks Data Engineer to design, develop, and optimize enterprise-scale data solutions using the Databricks platform. The ideal candidate will have extensive experience building scalable ETL/ELT pipelines, implementing modern Lakehouse architecture, and developing analytical dashboards that enable business decision-making.
This role requires deep expertise in Databricks, Apache Spark, Delta Lake, Lakeflow Declarative Pipelines (DLT), SQL, Python/Scala, data warehousing, and cloud-based data engineering practices. The selected consultant will collaborate with business stakeholders, architects, and technical teams to deliver secure, scalable, and high-performance data solutions.
Key Responsibilities
- Design, develop, and maintain scalable enterprise data solutions using the Databricks platform.
- Develop and optimize ETL/ELT pipelines using Apache Spark (PySpark or Scala).
- Build and maintain Lakehouse architectures utilizing Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
- Implement Lakeflow Declarative Pipelines (formerly Delta Live Tables) for production-grade data processing.
- Develop and schedule workflows using Lakeflow Jobs (formerly Databricks Workflows) or equivalent orchestration tools such as Apache Airflow.
- Design and maintain dimensional data models including Star and Snowflake schemas.
- Develop high-performance SQL queries for analytical workloads.
- Create interactive dashboards and applications using Databricks SQL Dashboards and Databricks Apps.
- Implement enterprise data governance, metadata management, data quality validation, and security best practices.
- Monitor pipeline performance and optimize workloads for scalability, reliability, and cost efficiency.
- Analyze business requirements and translate them into technical solutions.
- Perform system analysis, troubleshooting, and production support activities.
- Collaborate with business users, project managers, developers, and architects throughout the software development lifecycle.
- Document technical designs, workflows, data models, and operational procedures.
- Present technical findings and business insights to both technical and non-technical stakeholders.
Required Qualifications
- Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience).
- Minimum 8 years of IT experience supporting the design, development, deployment, or delivery of technology solutions.
- Minimum 8 years of experience with Databricks.
- Strong expertise in Apache Spark using PySpark or Scala.
- Extensive experience building enterprise ETL/ELT pipelines.
- Strong knowledge of Delta Lake.
- Experience implementing Lakeflow Declarative Pipelines (Delta Live Tables).
- Experience with Medallion Architecture (Bronze, Silver, Gold).
- Expertise in Lakehouse architecture.
- Strong experience with SQL.
- Strong programming skills in Python (preferred) or Scala.
- Experience designing dimensional data models (Star/Snowflake schema).
- Experience developing Databricks SQL Dashboards and Databricks Apps.
- Experience implementing data governance, data quality, security, and validation frameworks.
- Experience scheduling and orchestrating jobs using Lakeflow Jobs or Apache Airflow.
- Excellent verbal and written communication skills.
- Ability to work independently while collaborating across cross-functional teams.
Preferred Qualifications
- Databricks Certified Data Engineer (Associate or Professional).
- Experience working with Texas State agencies or other public sector organizations.
- Experience with Git-based version control.
- Experience implementing CI/CD pipelines for data engineering projects.
- Familiarity with DevOps best practices.
- Experience working in Agile/Scrum environments.
Technical Skills
Data Engineering
- Databricks
- Apache Spark
- PySpark
- Scala
- Delta Lake
- Delta Live Tables (DLT)
- Lakeflow Declarative Pipelines
- Lakeflow Jobs
- Databricks Workflows
- Data Lakehouse
- Medallion Architecture
Programming
Data Warehousing
- Dimensional Modeling
- Star Schema
- Snowflake Schema
- ETL
- ELT
Reporting & Analytics
- Databricks SQL
- Databricks Dashboards
- Databricks Apps
Data Management
- Data Governance
- Data Quality
- Data Validation
- Data Security
- Metadata Management
DevOps & Tools
- Git
- CI/CD
- Azure DevOps (preferred)
- Apache Airflow
Soft Skills
- Strong analytical and problem-solving skills
- Excellent communication and presentation abilities
- Ability to interact with business and technical stakeholders
- Strong documentation skills
- Self-motivated and detail-oriented
- Ability to manage multiple priorities in a fast-paced environment
Never miss an opportunity! Create an alert based on the job you applied for.
