About the Role

We are seeking a highly experienced Senior Data Engineer to design, build, and maintain scalable data pipelines and cloud-native data solutions. The ideal candidate brings deep expertise in Python, PySpark, and AWS services, with a proven track record in large-scale ETL development, real-time data ingestion, and cloud infrastructure optimization. This role offers the opportunity to work at the intersection of data engineering, cloud architecture, and business intelligence.

Required Qualifications

•       12+ years of experience in data engineering, ETL development, or a related field.

•       Expert-level proficiency in Python and PySpark for large-scale data processing.

•       Hands-on experience with AWS services: Lambda, Glue, EMR, S3, Redshift, Step Functions, SQS, SNS, CloudWatch, API Gateway, IoT Core.

•       Proven experience building real-time streaming pipelines using Apache Kafka.

•       Strong SQL skills with experience in Postgres, Oracle, SQL Server, or Redshift.

•       Experience with Terraform or other Infrastructure as Code (IaC) tools.

•       Familiarity with RESTful API design and OpenAPI/Swagger documentation standards.

•       Experience with UNIX/Linux environments and shell scripting.

•       Strong understanding of data lake architecture and cloud-native data platforms.

•       Excellent collaboration skills with demonstrated ability to work across cross-functional teams.

Preferred Qualifications

•       Experience with Databricks for ELT pipeline evaluation and development.

•       Background in financial services data environments (e.g., Fannie Mae, Freddie Mac, FINRA).

•       Experience with Microsoft Graph API integration and OPC UA protocols for IoT data pipelines.

•       Familiarity with Hadoop ecosystem and on-premises data warehouse tools (Informatica, Netezza).

•       Experience with Java for data migration projects.

•       Previous experience with Ab Initio, AutoSys, or similar enterprise ETL and scheduling tools.

Preferred Certifications

•       AWS Certified Solutions Architect (Associate or Professional)

•       AWS Certified Developer – Associate

•       Databricks Certified Data Engineer Professional