Sr Data Engineer
Mclean, VA Need Local
Long Term
Contract
Design & Build Scalable Data Pipelines using PySpark for large-scale batch and streaming data processing.
Develop PySpark Solutions write production-grade PySpark code to read data from S3 (Parquet/Delta files), perform complex transformations, and process large-scale datasets efficiently.
Implement Deduplication Logic design and implement robust deduplication strategies for large datasets using PySpark.
Performance Tuning (PySpark) optimize PySpark jobs for reading and processing very large datasets, including partitioning, caching, broadcast joins, and shuffle optimization.
Develop Cloud-Native Data Solutions on AWS leveraging services like S3, Glue, EMR, Lambda, Step Functions, and Redshift.
Engineer Snowflake Data Platforms build warehouses, schemas, and data models optimized for analytics and reporting.
Build Snowflake Iceberg Tables design and implement Apache Iceberg tables in Snowflake for open lakehouse architectures and interoperability.
Develop Dynamic Tables & Materialized Views build and maintain Snowflake Dynamic Tables and Materialized Views to support near real-time analytics and query acceleration.
Snowflake Performance Tuning optimize Snowflake workloads through clustering, micro-partition pruning, query profiling, warehouse sizing, result caching, and materialization strategies.
Modernize Legacy ETL migrate on-prem ETL workloads (e.g., DataStage, Informatica) to PySpark and Snowflake-based cloud pipelines.
Optimize Performance tune Spark jobs, Snowflake queries, and AWS resource utilization for speed and cost.
Ensure Data Quality & Governance implement validation, lineage, and monitoring across every pipeline.
Collaborate Across Teams partner with data architects, analysts, TPMs, and business stakeholders to deliver trustworthy data products.
Document Everything from technical designs to runbooks, ensuring every pipeline is maintainable and audit-ready.
What We re Looking For
Must-Haves
6+ years of hands-on data engineering experience in enterprise environments.
Strong expertise in PySpark building distributed data processing pipelines at scale.
Hands-on experience writing PySpark code to read data from S3 Parquet/Delta files, perform transformations, and handle large datasets.
Proven experience in performance tuning of PySpark jobs when reading and processing very large datasets.
Experience implementing deduplication logic in PySpark for high-volume data pipelines.
Deep hands-on experience with Snowflake data modeling, SnowSQL, Snowpipe, Streams, Tasks, and role-based access.
Hands-on experience creating Snowflake Iceberg Tables for open table format and lakehouse use cases.
Experience building Snowflake Dynamic Tables and Materialized Views for incremental data processing and query acceleration.
Strong Snowflake performance tuning skills clustering keys, micro-partition pruning, query profiling, warehouse right-sizing, caching strategies, and cost optimization.
Proven AWS experience S3, Glue, EMR, Lambda, IAM, Step Functions, CloudWatch, and Redshift.
Advanced SQL skills complex joins, subqueries, window functions, CTEs, and performance tuning.
Strong Python programming skills beyond PySpark for utilities, automation, and orchestration.
Experience with orchestration tools Airflow, AWS Step Functions, or equivalent.
Solid understanding of data warehousing, ELT/ETL patterns, data lakes, and lakehouse architectures.
Excellent communication skills able to articulate technical decisions to both engineers and business stakeholders.
Nice to Have
Experience with IBM DataStage or other legacy ETL tools (for modernization contexts).
Familiarity with CI/CD for data pipelines (Git, Jenkins, GitHub Actions, Terraform).
Exposure to data quality frameworks (Great Expectations, dbt tests).
Knowledge of streaming platforms (Kafka, Kinesis).
Experience in regulated environments financial services, mortgage, or GSE programs (Fannie Mae / Freddie Mac).
Certifications: AWS Certified Data Analytics / Solutions Architect, SnowPro Core/Advanced.
Munesh
,
CYBER SPHERE LLC
Never miss an opportunity! Create an alert based on the job you applied for.
