AWS Data Engineer (Exp- 14+ Years) with AI Exp-Full time-Remote

We are seeking a highly experienced Senior AWS Data Engineer with strong hands-on expertise in Python, AWS, Data Pipelines, SQL, Snowflake, Apache Airflow, and DBT. The ideal candidate should have excellent communication skills, be comfortable working directly with clients, and possess strong experience designing, developing, testing, and troubleshooting scalable data pipelines.

The candidate should be a strong Individual Contributor who can also collaborate effectively with cross-functional engineering and business teams.

Key Responsibilities

Python & Data Engineering

• Design and develop scalable data pipelines using Python.
• Apply advanced Python concepts including:
  • Object-Oriented Programming
  • Classes and Inheritance
  • Modules and Packages
  • Multithreading
  • Functional Programming
• Develop reusable Python packages and common data engineering utilities.
• Automate ETL processes and data workflows.
• Build mechanisms to identify, reprocess, and recover failed records.
• Orchestrate pipelines to execute sequentially or in parallel.
• Troubleshoot data pipeline failures and implement permanent fixes.
• Process data in formats such as CSV, JSON, XML, and Parquet.
• Read and write data using AWS S3 and other cloud storage platforms.
• Utilize AI-powered IDE/coding assistants to improve development productivity.

SQL & Data Warehousing

• Write complex SQL queries using:
  • Joins
  • CTEs
  • Subqueries
  • Window Functions
• Perform SQL query optimization and performance tuning.
• Work extensively with cloud data warehouses, preferably Snowflake.
• Develop efficient data models and transformation logic.

DBT & Data Quality

• Build and maintain DBT models for data transformation.
• Develop incremental models, snapshots, and full-refresh models.
• Implement DBT tests for data validation and quality assurance.
• Create reusable DBT macros using Jinja/Python.
• Maintain DBT documentation and data lineage.
• Configure DBT profiles and secure access appropriately.

AWS Cloud

• Strong hands-on experience with AWS data and cloud services.
• Design and manage data storage solutions using Amazon S3.
• Work with AWS ECS for containerized workloads.
• Configure and manage AWS IAM roles, policies, and permissions.
• Implement AWS security best practices including encryption and secure access.
• Develop serverless data processing solutions using AWS Lambda.
• Work with AWS services such as Redshift and EMR where required.

Airflow & Workflow Orchestration

• Design, develop, schedule, and monitor complex Apache Airflow DAGs.
• Orchestrate DBT models and AWS workloads through Airflow.
• Build workflows that integrate:
  • S3
  • DBT
  • EMR
  • Redshift
  • Snowflake
• Configure pipeline dependencies and parallel/sequential execution.
• Monitor failed DAG runs and implement retry/reprocessing mechanisms.

CI/CD & Version Control

• Use GitHub for source code management.
• Design and implement CI/CD pipelines from the ground up.
• Automate testing, validation, and deployment processes.
• Follow enterprise development and branching standards.