Databricks Data Engineer

Job Title: Databricks Data Engineer
Location: Washington, DC
Type: Contract
Compensation: $62.00 - $75.00 per hour
Contractor Work Model: Remote
Hours: 40.0 hours per week
Security Clearance: Must have an active Public Trust clearance
 

Responsibilities

• Design, build, and optimize scalable data solutions using Databricks and Medallion Architecture.
• Manage ingestion routines for processing multi-terabyte datasets efficiently for multiple projects simultaneously, where each project may have multiple Databricks workspaces.
• Integrate data from various structured and unstructured sources to enable high-quality business insights.
• Proficiency in data analysis techniques for deriving insights from large datasets.
• Implement effective data management strategies to ensure data integrity, availability, and accessibility.
• Identify opportunities for cost optimization in data storage, processing, and analytics operations.
• Monitor and support user requests, addressing platform or performance issues, cluster stability, Spark optimization, and configuration management.
• Collaborate with the team to enable advanced AI-driven analytics and data science workflows.
• Integrate with various Azure services including Azure Functions, Storage Services, Data Factory, Log Analytics, and User Management for seamless data workflows.
• Provision and manage infrastructure using Infrastructure-as-Code (IaC).
• Apply best practices for data security, data governance, and compliance, ensuring support for federal regulations and public trust standards.
• Proactively collaborate with technical and non-technical teams to gather requirements and translate business needs into data solutions.

Requirements

• BA degree in Computer Science or related field
• 3+ years of experience developing and designing ingestion flows (structured, streaming, and unstructured data) using cloud platform services with data quality
• Databricks Data Engineer certification and 2+ years of experience maintaining Databricks platform and development in Spark
• Ability to work directly with clients and act as front-line support for requests coming in from clients
• Clearly document and express solutions in architecture and interface diagrams
• Proficient in Python, Spark, and R; .NET development is a plus
• Knowledge and experience with data governance, including metadata management, enterprise data catalog, design standards, data quality governance, and data security
• Experience with Agile methodology, CI/CD automation, and cloud-based developments (Azure, AWS)

System One, and its subsidiaries including Joulé and Mountain Ltd., are leaders in delivering outsourced services and workforce solutions across North America. We help clients get work done more efficiently and economically, without compromising quality. System One not only serves as a valued partner for our clients, but we offer eligible employees health and welfare benefits coverage options including medical, dental, vision, spending accounts, life insurance, voluntary plans, as well as participation in a 401(k) plan.

System One is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, age, national origin, disability, family care or medical leave status, genetic information, veteran status, marital status, or any other characteristic protected by applicable federal, state, or local law.

#M-M2
#LI-
Ref: #851-Rockville-S1