Big Data Platform Engineer -- Atlanta, GA/San Francisco, CA/Minneapolis, MN -- Hybrid -- 67246
San Francisco, CA, US • Posted 5 hours ago • Updated 1 hour ago
PRIMUS Global Services Inc.
Dice Job Match Score™
🫥 Flibbertigibetting...
Job Details
Skills
- Python
- Azure
- Big Data Platform
Summary
Must Have Technologies:
- Azure
- Airflow
- Spark
- Hive
- Kubernetes
Overview: Lead and operate an Azure-based Big Data Platform using open-source tools (Airflow, Spark, Hive). Own onsite offshore coordination, build reusable ETL frameworks, and ensure reliable dev/prod operations. Hands-on coding in Python, Java, and Scala with strong DevOps and IaC practices.
Key Responsibilities
- Platform engineering: Design, build, and run Azure data platform components with reliability, security, and cost efficiency.
- ETL frameworks: Develop reusable ETL/ELT libraries; standardize ingestion, transformation, testing, and deployment.
- Airflow & Spark: Author/maintain Airflow DAGs; implement and optimize Spark jobs; manage SLAs, retries, autoscaling.
- Onsite offshore coordination: Establish handoffs, runbooks, and cadences; lead release orchestration and incident response.
- CI/CD & IaC: Build pipelines (GitHub Actions/Azure DevOps) and provision infra (HelmCharts) across environments.
- Observability & security: Implement logging/metrics/tracing; enforce IAM/RBAC, secrets (Key Vault), and compliance.
Requirements
- Experience: 4 8+ years in platform/data engineering with production ownership.
- Azure: ADLS Gen2, AKS, ACR, Event Hubs/Service Bus, Key Vault; Databricks or Synapse.
- Open source: Airflow, Apache Spark, Hive (metastore, table formats); strong SQL.
- Languages: Proficient in Python, Scala, and Java.
- DevOps/IaC: GitHub Actions/Azure DevOps; Terraform or Bicep; Docker/Kubernetes (AKS).
- Coordination: Proven leadership across time zones; vendor/stakeholder management; clear communication and documentation.
Tech Stack Highlights
- Azure (ADLS, AKS, ACR, Event Hubs, Key Vault, Monitor), Databricks/Synapse
- Apache Airflow, Apache Spark, Hive; optional Delta Lake/Iceberg
- GitLab Actions/Azure DevOps, Terraform/Ansible, Docker/Kubernetes
- Datadog
Ujjal
PRIMUS Global Services
Direct
Desk: x 268
Email:
- Dice Id: 10117354
- Position Id: 2026-65940
- Posted 5 hours ago
Company Info
About PRIMUS Global Services Inc.
At PRIMUS we bring together the right people, knowledge, methodology, and technologies you require for the project at hand.
Whether you require staff augmentation, strategy consulting, technology integration (offsite or onsite, time and materials or fixed bid projects) or offshore development solutions -PRIMUS can get the job done.
We provide the right solution for your company, be it recruiting the exact person or team to work directly for you, or outsourcing the entire project to PRIMUS. We develop customized solutions that exceed our client s expectations and we do it on time and on budget.
PRIMUS provides cost-effective solutions. Because we have consultants in almost every industry - telecommunications, financial, healthcare, security, transportation, consulting, logistics, high-tech, oil and gas, government and hospitality - we can assure you that we will assign a team to your project that understands your industry and your needs and can help you stay competitive in the digital economy.
Never miss an opportunity! Create an alert based on the job you applied for.Similar Jobs
It looks like there aren't any Similar Jobs for this job yet.
Search all similar jobs