Overview
Hybrid
Depends on Experience
Contract - W2
Contract - Independent
Contract - 6 Month(s)
Skills
Amazon Web Services
Apache Spark
Big Data
Caching
Cloud Computing
Collaboration
Conflict Resolution
Continuous Delivery
Continuous Integration
Data Engineering
Data Modeling
Data Security
Database
DevOps
Distributed Computing
Docker
Extract
Transform
Load
IBM DB2
Kubernetes
Mentorship
NoSQL
PostgreSQL
Problem Solving
PySpark
Python
Regulatory Compliance
SQL
Snow Flake Schema
Unit Testing
Unstructured Data
Version Control
Workflow
Job Details
Title: Lead PySpark Developer
Location: Owings Mills, MD
Job Description:
- 7+ years of experience in Amazon Web Service(AWS) Cloud Computing.
- 10+ years of experience in big data and distributed computing.
- Very Strong hands-on experience with PySpark, Apache Spark, and Python.
- Strong Hands on experience with SQL and NoSQL databases (DB2, PostgreSQL, Snowflake, etc.).
- Proficiency in data modeling and ETL workflows.
- Proficiency with workflow schedulers like Airflow.
- Hands on experience with AWS cloud-based data platforms.
- Experience in DevOps, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus.
- Strong problem-solving skills and ability to lead a team
- DBT, AWS Astronomer
- Lead the design, development, and deployment of PySpark-based big data solutions.
- Architect and optimize ETL pipelines for structured and unstructured data.
- Collaborate with Client, data engineers, data scientists, and business teams to understand requirements and provide scalable solutions.
- Optimize Spark performance through partitioning, caching, and tuning.
- Implement best practices in data engineering (CI/CD, version control, unit testing).
- Work with cloud platforms like AWS.
- Ensure data security, governance, and compliance.
- Mentor junior developers and review code for best practices and efficiency.
MUST HAVE:
- 7+ years of experience in Amazon Web Service(AWS) Cloud Computing.
- 10+ years of experience in big data and distributed computing.
- Experience with PySpark, Apache Spark, and Python.
- Experience with SQL and NoSQL databases (DB2, PostgreSQL, Snowflake, etc.).
- Hands on experience with AWS cloud-based data platforms.
- Experience in DevOps, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.