Senior Data Engineer AWS

• Brings 12 years of data engineering experience spanning production ETL/ELT pipelines, Big Data processing, cloud migrations, data lakes, and analytics platforms.
• Builds and supports AWS Glue ETL pipelines using PySpark/Python, including ingestion, transformations, schema validation, data-quality checks, and curated data layers.
• Develops event-driven and near-real-time architectures using AWS Lambda, Eventbridge, Step Functions, Kafka/MSK, and SNS.
• Has hands-on experience with Glue Catalog and Glue Crawlers for metadata discovery, schema registration, governance, and downstream querying through Athena.
• Uses CloudWatch logs, metrics, monitoring, and alerts to troubleshoot Glue, Lambda, and Step Functions and maintain pipeline reliability and SLA compliance.
• Has strong Python, PySpark, SQL, PostgreSQL, Redshift, Hive, Snowflake, and Databricks experience across large structured and semi-structured datasets.
• Implements CDC, incremental processing, schema enforcement, error handling, retry logic, data-quality validation, and scalable batch/streaming pipelines using PySpark, Delta Lake, and Delta Live Tables.
• Builds orchestration solutions using Step Functions, Airflow, Lambda, dbt, Control-M, and AWS services, managing workflow dependencies, scheduling, retries, and cross-service execution.
• Automates infrastructure with Terraform and Cloud Formation and supports CI/CD through Git, Jenkins, AWS Code Pipeline, and Docker.
• Has extensive cloud-migration experience, including moving large on-premises Hadoop/Teradata workloads to AWS and designing scalable S3-based data lake architectures.
• Brings hands-on performance tuning experience across Spark/PySpark jobs, SQL queries, Hive workloads, Redshift clusters, partitioning, and large-scale data pipelines.
• Works effectively with business, BI, analytics, and downstream data consumers and has supported Tableau, Snowflake, Redshift, and Athena-based analytics environments.

CERTIFICATIONS

• AWS Certified Developer – Associate