Role- Data Engineer
Location- New York
Job Description
We are seeking an experienced Data Engineer to join our team and build robust, scalable data pipelines. In this role, you will:
Design and implement scalable PySpark data pipelines for batch and streaming workloads
Optimize Spark jobs and queries for performance and cost efficiency
Build and maintain ETL/ELT processes following data engineering best practices
Troubleshoot and resolve complex data pipeline and processing issues
Collaborate with data teams to ensure data quality and reliability
Top Skills
Databricks Platform Experience
Data Engineering & Pipeline Development
Advanced ETL/ELT pipeline design and development
Incremental data processing patterns (CDC, SCD Type 2)
Data Processing & Optimization
Spark optimization techniques (partitioning, bucketing, caching, broadcast joins)
Required Technical Skills
Databricks & Spark Proficiency: 3+ years of hands-on experience building data pipelines in Databricks; deep understanding of Spark fundamentals, transformations, actions, and performance optimization techniques including partitioning, caching, and resource management
Advanced PySpark and SQL Skills: Expert-level proficiency writing production-quality PySpark code and complex SQL queries for data transformation, aggregation, and analysis; experience with DataFrame API, Spark SQL, and UDFs; strong understanding of lazy evaluation and execution plans
Data Engineering & ETL/ELT: Proven experience building and maintaining production data pipelines; hands-on experience with incremental data loading, change data capture (CDC), and slowly changing dimensions; experience handling data quality issues and implementing data validation frameworks
Cloud & Big Data Technologies: Strong proficiency with AWS services (S3, EC2, IAM, Glue, Athena); experience working with large-scale distributed data processing; familiarity with data formats (Parquet, Delta, JSON, Avro) and compression techniques
DevOps & CI/CD: Experience with version control (Git) and CI/CD pipelines using GitLab, GitHub Actions, or similar tools; familiarity with testing data pipelines and deployment automation; experience with Databricks Repos and workspace-level integrations
Data Governance: Understanding of data lineage, cataloging, and metadata management; experience implementing data quality checks and monitoring; knowledge of data privacy and security best practices in cloud environments (nice to have)
If interested please email updated resume to
