This role combines hands-on development, solution design, production support, and technical leadership.
-
Design, develop, and unit test Python and PySpark applications using Apache Spark.
-
Build data pipelines integrating Oracle, MongoDB, DynamoDB, Amazon S3, and HDFS.
-
Develop PySpark ETL pipelines using AWS Glue and Amazon EMR.
-
Build Python-based AWS Lambda functions using boto3, awswrangler, pandas, JSON, and requests.
-
Develop event-driven integrations using Apache Kafka, Amazon MSK, and Confluent libraries.
-
Implement Apache Iceberg tables with schema evolution, CDC merges, and current-state snapshots.
-
Develop REST APIs using FastAPI or Flask with Swagger/OpenAPI specifications.
-
Secure API endpoints using OAuth2 and JWT authentication.
-
Orchestrate workflows using AWS Step Functions and Amazon EventBridge.
-
Configure AWS Glue crawlers and Data Catalog integrations.
-
Optimize Spark applications, SQL queries, partitioning, compression, and file formats.
-
Deploy applications using Docker, Kubernetes, and Amazon EKS.
-
Build agentic AI workflows and use Copilot/GPT for development.
-
Monitor and troubleshoot applications using Amazon CloudWatch and Datadog.
-
Participate in code reviews, production releases, deployments, and incident resolution.
-
Create technical documentation, deployment guides, and operational runbooks.
-
Coordinate technical designs across stakeholders and onsite/offshore teams.
-
Mentor engineers and contribute to Agile delivery and engineering standards.
-
Bachelor's degree or equivalent in Computer Science or a related discipline.
-
10+ years of relevant professional experience.
-
Valid AWS certification in Solutions Architecture, Data Engineering, or Data Analytics Specialty.
-
Strong hands-on Python, PySpark, Apache Spark, and SQL skills.
-
Experience with AWS S3, Glue ETL, Glue Data Catalog, Lambda, EventBridge, Step Functions, Athena, and EMR.
-
Hands-on Kafka integration, including event publishing and consumption.
-
Experience with Apache Iceberg, CDC processing, schema evolution, and snapshots.
-
Experience with pandas, requests, boto3, and awswrangler.
-
Experience processing Avro, Parquet, ORC, JSON, and XML data formats.
-
Strong FastAPI or Flask development experience, including Swagger/OpenAPI.
-
Experience implementing OAuth2/JWT authentication for REST APIs.
-
Experience with Docker, Kubernetes, and Amazon EKS.
-
Hands-on experience building agentic AI workflows and using Copilot/GPT for coding.
-
Production troubleshooting experience using CloudWatch and Datadog.
