Cloud Data Engineer Databricks, Snowflake & Azure

Location: New York, NY (Onsite)
Job Type: Contract

Position Overview

We are seeking an experienced Cloud Data Engineer with strong expertise in Scala, Spark, Databricks, Snowflake, and Azure to support large-scale data modernization initiatives. This role involves designing, developing, testing, and optimizing enterprise data pipelines while ensuring data quality, scalability, and operational excellence across cloud-based platforms.

The ideal candidate will have hands-on experience migrating legacy ETL workloads to modern cloud architectures and implementing robust data engineering best practices.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Scala and Apache Spark.
• Build and maintain data solutions on Databricks leveraging Serverless Compute, Unity Catalog, Databricks CLI, and Asset Bundles.
• Analyze, refactor, and translate complex SQL-based data transformation logic into modern cloud-native architectures.
• Develop and optimize Snowflake data models, schemas, and integrations using the Spark-Snowflake Connector.
• Implement data ingestion, transformation, and orchestration workflows using Azure services and Apache Airflow.
• Create automated testing frameworks, including unit, integration, and behavior-driven testing approaches.
• Develop data quality controls, reconciliation processes, and automated validation mechanisms.
• Collaborate with cross-functional teams to modernize legacy ETL environments and execute migration strategies.
• Implement CI/CD pipelines for code deployment, testing automation, and release management.
• Ensure adherence to data governance, lineage, observability, and security best practices.

Required Skills & Qualifications

Data Engineering

• Strong hands-on experience with Scala and Apache Spark in production environments.
• Expertise with Spark DataFrame APIs, joins, window functions, partitioning strategies, and performance tuning.
• Advanced SQL development and optimization skills.

Databricks & Snowflake

• Experience with Azure Databricks, including Serverless Compute, Unity Catalog, Asset Bundles, and Databricks CLI.
• Strong understanding of Snowflake architecture, schema design, query optimization, and connector integrations.

Cloud & Orchestration

• Experience working with Azure services including ADLS, identity management, and cloud security fundamentals.
• Hands-on experience developing and maintaining Apache Airflow DAGs, sensors, retries, and SLA monitoring.

DevOps & Testing

• Experience with Git-based development workflows and CI/CD automation.
• Familiarity with artifact management and automated deployment pipelines.
• Strong background in Test-Driven Development (TDD), unit testing, and behavior-driven testing methodologies.

Data Quality & Migration

• Experience building automated data validation, reconciliation, and drift-detection frameworks.
• Proven success migrating legacy ETL platforms to modern cloud-based data ecosystems.
• Strong understanding of Medallion Architecture, schema evolution, lineage, observability, and idempotent pipeline design.

Preferred Qualifications

• Financial Services or Regulatory Reporting domain experience.
• Python development experience for data engineering utilities and automation.
• Experience with specification-driven development methodologies.
• Knowledge of Gradle and JVM ecosystem tooling.

Why Join?

• Work on enterprise-scale cloud modernization initiatives.
• Utilize modern technologies including Databricks, Snowflake, Azure, Scala, and Spark.
• Collaborate with highly skilled engineering teams on complex data transformation projects.
• Opportunity to contribute to large-scale migration and cloud adoption programs.