Role- Data Engineer
Location- New York
Job Description
We are seeking an experienced Data Engineer to join our team and build robust, scalable data pipelines. In this role, you will:
Design and implement scalable PySpark data pipelines for batch and streaming workloads
Optimize Spark jobs and queries for performance and cost efficiency
Build and maintain ETL/ELT processes following data engineering best practices
Troubleshoot and resolve complex data pipeline and processing issues
Collaborate with data teams to ensure data quality and reliability
Top Skills
Databricks Platform Experience
Hands-on development experience with Databricks notebooks and workflows
Proficiency in Python and PySpark for data transformation and processing
Working knowledge of Unity Catalog for data discovery and lineage
Experience with cluster configuration and job scheduling
Delta Lake development and optimization techniques
Databricks SQL for data analysis and reporting
Data Engineering & Pipeline Development
Advanced ETL/ELT pipeline design and development
Delta Lake performance tuning (Z-ordering, data skipping, compaction, vacuuming)
Real-time streaming data pipelines using Structured Streaming and Delta Live Tables
Query performance optimization and debugging slow-running jobs
Data quality validation and testing frameworks
Incremental data processing patterns (CDC, SCD Type 2)
Data Processing & Optimization
Spark optimization techniques (partitioning, bucketing, caching, broadcast joins)
Working with large-scale datasets (terabytes to petabytes)
Data pipeline orchestration and scheduling
Monitoring and alerting data pipelines
Implementing Bronze/Silver/Gold (Medallion) data layer patterns
Required Technical Skills
Databricks & Spark Proficiency: 3+ years of hands-on experience building data pipelines in Databricks; deep understanding of Spark fundamentals, transformations, actions, and performance optimization techniques including partitioning, caching, and resource management
Advanced PySpark and SQL Skills: Expert-level proficiency writing production-quality PySpark code and complex SQL queries for data transformation, aggregation, and analysis; experience with DataFrame API, Spark SQL, and UDFs; strong understanding of lazy evaluation and execution plans
Data Engineering & ETL/ELT: Proven experience building and maintaining production data pipelines; hands-on experience with incremental data loading, change data capture (CDC), and slowly changing dimensions; experience handling data quality issues and implementing data validation frameworks
Cloud & Big Data Technologies: Strong proficiency with AWS services (S3, EC2, IAM, Glue, Athena); experience working with large-scale distributed data processing; familiarity with data formats (Parquet, Delta, JSON, Avro) and compression techniques
DevOps & CI/CD: Experience with version control (Git) and CI/CD pipelines using GitLab, GitHub Actions, or similar tools; familiarity with testing data pipelines and deployment automation; experience with Databricks Repos and workspace-level integrations
Data Governance: Understanding of data lineage, cataloging, and metadata management; experience implementing data quality checks and monitoring; knowledge of data privacy and security best practices in cloud environments (nice to have)
Preferred Qualifications
Bachelor's degree in Computer Science, Engineering, or related field
Databricks Certified Data Engineer Associate or Professional certification
Experience with data orchestration tools (Apache Airflow, Databricks Workflows)
Strong debugging and problem-solving skills
Excellent communication skills for technical collaboration
Never miss an opportunity! Create an alert based on the job you applied for.
