Clearance: U.S. Citizenship
Requirement- Ability to obtain Public Trust (minimum)
Satsyil Corp is seeking a highly skilled and motivated Senior Data Engineer / Databricks Specialist to join our team. In this role, you will play a crucial part in designing, developing, and optimizing high-performance Spark applications and managing enterprise data infrastructure within AWS Databricks. Utilizing PySpark, Python, SQL, and Delta Lake, you will architect robust ETL/ELT pipelines, maintain medallion architecture, and establish efficient big data solutions supporting mission-critical public sector and enterprise environments.
2. Key Roles & Responsibilities
· Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL to support data processing and analytics needs.
· Design, build, and maintain robust ETL/ELT pipelines and implement Medallion Architecture (Bronze, Silver, Gold layers) ensuring data integrity, completeness, and lineage.
· Administer Databricks Workspaces, including cluster provisioning, job scheduling, access controls, Unity Catalog, and notebook management.
· Manage AWS cloud infrastructure assets including S3, Lambda, IAM, Aurora, Glue, and provision environments using Terraform Infrastructure-as-Code (IaC).
· Perform query optimization, performance tuning, troubleshooting, and debugging for complex Spark jobs and large-scale data solutions.
· Merge, consolidate, and process structured, semi-structured, and unstructured data sets from diverse API endpoints and databases.
· Embed security gates and compliance controls within DevSecOps CI/CD pipelines (Jenkins, GitLab CI) to support FISMA/FedRAMP requirements, ATO reassessments, and POA&M resolution.
· Implement rigorous data quality controls, anomaly detection, validation rules, and automated lineage tracking across all pipelines.
· Collaborate within cross-functional Agile/Scrum teams in multi-contractor federal environments, providing technical leadership and production on-call support.
3. Candidate Qualifications & Requirements
Required Experience & Technical Skills:
· Experience: 8+ years of dedicated Data Engineering / Platform Administration experience, with 4+ years of hands-on experience in Databricks.
· Core Big Data Stack: Production proficiency with PySpark, Spark SQL, Apache Spark, and Delta Lake.
· Cloud Ecosystem: Strong experience in AWS Cloud services including S3, IAM, Lambda, Glue, and Aurora.
· Automation & DevOps: Production experience with Terraform (IaC) and CI/CD tools (Jenkins, GitLab CI, or equivalent).
· Languages & Frameworks: Deep proficiency in Python and SQL; working knowledge of REST APIs for ingestion.
· Clearance & Citizenship: U.S. Citizenship required; must have the ability to obtain and maintain a Public Trust clearance.
· Methodology & Education: Hands-on experience in Agile/Scrum delivery environments. Bachelor's degree in CS, IT, Data Science, or related engineering discipline.
Preferred Qualifications & Nice-to-Haves:
· Hands-on experience with Unity Catalog, Delta Live Tables (DLT), and AI/ML pipeline support (MLflow, Model Registry).
· Experience in Federal Government contracting and compliance frameworks (FISMA / FedRAMP environments).
· Active security clearance (Public Trust or higher).
4. Preferred Certifications
Candidates possessing any of the following certifications will be prioritized:
· Databricks Certified Data Engineer Associate
· Databricks Certified Data Engineer Professional
· Databricks Certified Machine Learning Associate
· AWS Certified Solutions Architect – Associate or Professional
· AWS Certified Data Analytics – Specialty
Never miss an opportunity! Create an alert based on the job you applied for.