Remote Senior Databricks Platform Engineer-W2

Title: Remote Senior Databricks Platform Engineer-W2
Location: Remote, but must work East Coast Hours
Duration: 8-12 months/contract, but could get extended.
Must be Databricks SME and AWS

Notes: Client is shifting from Red Hat to Databricks, so they need a Platform Engineer with strong experience in Databricks to help with this migration. This process is starting from scratch, so they need someone to have experience with Databricks implementation. They ideally need someone who has implemented Databricks in AWS. Azure experience is okay, but AWS experience is what they are looking for. This Platform Engineer will join 2 other internal Engineers in this role, but the existing Engineers are not strong in Databricks, so they need this contractor to be the lead Databricks SME. They would love someone with a Financial or Medical data background.

Summary:

• Design, implement, maintain, and support the company’s enterprise data platform and the supporting infrastructure, integrations, and services required to deliver reliable, secure, scalable, and cost-effective data capabilities.
• Partner with data engineering, infrastructure, security, application development, architecture, and business technology teams to support data ingestion, transformation, storage, analytics, integration, governance, and operational data requirements across cloud, on-premises, SaaS, and third-party environments.
• Serve as a technical resource for data platform implementation, connectivity, automation, security, performance, reliability, and modernization initiatives.

Responsibilities:

• Design, implement, configure, and maintain enterprise data platforms and supporting cloud infrastructure.
• Support data warehouse, data lake, lakehouse, operational data, integration, and analytical workloads.
• Implement secure connectivity between the enterprise data platform and on-premises systems, cloud environments, vendor platforms, SaaS applications, relational databases, APIs, and file-transfer services.
• Build and maintain platform integrations across databases, object storage, APIs, SFTP, enterprise applications, and third-party data sources.
• Work with infrastructure and networking teams to implement secure cloud and hybrid connectivity patterns.
• Configure and support platform identity, access controls, service accounts, roles, groups, secrets, authentication, encryption, and least-privilege security standards.
• Develop and maintain automated infrastructure and platform deployment processes using infrastructure-as-code and CI/CD technologies.
• Support Git-based source control, release management, environment promotion, and deployment standards for data-platform components.
• Partner with data engineers to implement scalable ingestion and transformation patterns for batch, near-real-time, streaming, API, database, and file-based workloads.
• Support orchestration and scheduling of data pipelines, platform jobs, integrations, and dependent services.
• Configure and manage platform compute, storage, workload isolation, scaling, scheduling, concurrency, resource governance, and cost controls.
• Monitor platform availability, infrastructure health, job execution, query performance, compute utilization, storage consumption, connectivity, and integration health.
• Implement logging, monitoring, alerting, auditing, operational dashboards, and support procedures.
• Troubleshoot data-platform, infrastructure, network, security, performance, and integration issues.
• Support platform upgrades, configuration changes, infrastructure changes, release deployments, and production support.
• Assist with migration of existing warehouse, ETL, database, API, and file-processing workloads into modern data-platform architectures.
• Evaluate workload requirements and help determine appropriate placement across data warehouse, lakehouse, relational database, cloud service, integration, and application platforms.
• Develop reusable engineering patterns, deployment templates, technical standards, and operational documentation.
• Support proof-of-concept environments, platform evaluations, workload testing, performance testing, and migration assessments.
• Collaborate with security, risk, infrastructure, architecture, application, and data teams to incorporate security, availability, resiliency, and compliance requirements into platform implementations.
• Work with internal teams and external vendors to resolve platform, connectivity, integration, and performance issues.
• Maintain technical documentation covering platform architecture, environments, integrations, network connectivity, security controls, operational procedures, deployment processes, and dependencies.
• Identify opportunities for platform simplification, automation, modernization, standardization, reuse, and cost optimization.
• Share technical knowledge and provide guidance to engineers and project teams supporting the enterprise data platform.

Qualifications:

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or related field; applicable years of experience may be substituted for a bachelor’s degree.
• Minimum of five or more years of progressive experience in data engineering, cloud engineering, platform engineering, DevOps, database engineering, or related technology roles.
• Experience supporting enterprise-scale data platforms, data warehouses, data lakes, or cloud data environments.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Strong understanding of cloud infrastructure, networking, identity and access management, encryption, secrets management, and secure data-platform design.
• Experience working with relational databases, object storage, APIs, file-based integrations, and enterprise data sources.
• Experience with ETL/ELT, pipeline orchestration, data ingestion, transformation, and workload scheduling.
• Experience with source control, CI/CD, and automated deployment processes.
• Experience with infrastructure-as-code technologies such as Terraform, CloudFormation, or equivalent tools.
• Strong SQL skills and working knowledge of data warehouse and analytical database concepts.
• Understanding of modern data architecture concepts including data lakes, warehouses, lakehouses, medallion architectures, batch processing, streaming, CDC, metadata, and governance.
• Experience troubleshooting complex technical issues across cloud infrastructure, databases, networking, and data integrations.
• Ability to communicate technical concepts clearly and professionally to technical and non-technical stakeholders.
• Ability to work effectively across data, infrastructure, security, application, architecture, vendor, and business teams.