Big Data Platform Engineer/Data Platform Analyst (Azure, Spark, Airflow, Kubernetes)

Hybrid in Atlanta, GA, US • Posted 7 hours ago • Updated 7 hours ago

Contract W2

No Travel Required

Remote

Depends on Experience

Fitment

Dice Job Match Score™

🧠 Analyzing your skills...

Job Details

Skills

Apache Spark
Big Data
Apache Hive
Ansible
Continuous Integration
Data Engineering
Continuous Delivery
DevOps
Documentation
ELT
Extract, Transform, Load
Databricks
Docker
Kubernetes
GitHub
Microsoft Azure
SQL
Scala
Stakeholder Management
Terraform
Testing
Python
RBAC
ProVision
Open Source
Offshoring
Leadership
Java
Engineering Design
Management
Incident Management
Regulatory Compliance

Summary

Must Have Technologies:

Azure
Airflow
Spark
Hive
Kubernetes

BDP Platform Engineer — Contractor Lead (Onsite)

Client: US Bank

Locations: Minneapolis, Atlanta, or San Francisco

Duration: 12-24 months

Overview: Lead and operate an Azure-based Big Data Platform using open-source tools (Airflow, Spark, Hive). Own onsite–offshore coordination, build reusable ETL frameworks, and ensure reliable dev/prod operations. Hands-on coding in Python, Java, and Scala with strong DevOps and IaC practices.

Key Responsibilities

Platform engineering: Design, build, and run Azure data platform components with reliability, security, and cost efficiency.
ETL frameworks: Develop reusable ETL/ELT libraries; standardize ingestion, transformation, testing, and deployment.
Airflow & Spark: Author/maintain Airflow DAGs; implement and optimize Spark jobs; manage SLAs, retries, autoscaling.
Onsite–offshore coordination: Establish handoffs, runbooks, and cadences; lead release orchestration and incident response.
CI/CD & IaC: Build pipelines (GitHub Actions/Azure DevOps) and provision infra (HelmCharts) across environments.
Observability & security: Implement logging/metrics/tracing; enforce IAM/RBAC, secrets (Key Vault), and compliance.

Requirements

Experience: 4–8+ years in platform/data engineering with production ownership.

Azure: ADLS Gen2, AKS, ACR, Event Hubs/Service Bus, Key Vault; Databricks or Synapse.
Open source: Airflow, Apache Spark, Hive (metastore, table formats); strong SQL.
Languages: Proficient in Python, Scala, and Java.
DevOps/IaC: GitHub Actions/Azure DevOps; Terraform or Bicep; Docker/Kubernetes (AKS).
Coordination: Proven leadership across time zones; vendor/stakeholder management; clear communication and documentation.

Tech Stack Highlights

Azure (ADLS, AKS, ACR, Event Hubs, Key Vault, Monitor), Databricks/Synapse
Apache Airflow, Apache Spark, Hive; optional Delta Lake/Iceberg
GitLab Actions/Azure DevOps, Terraform/Ansible, Docker/Kubernetes
Datadog

Thanks & Regards,

Anthony Kumar Guvvala

ERP Analysts, Inc.

Direct:

Email :

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 10114195
Position Id: 8881768
Posted 7 hours ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Big Data Platform Engineer/Data Platform Analyst (Azure, Spark, Airflow, Kubernetes)

Dice Job Match Score™

Job Details

Skills

Summary

Similar Jobs