Lead Data Engineer_ Charlotte, NC

Job Summary:

• The Lead Data Engineer will be responsible for designing, building, and maintaining robust data platforms on AWS, managing complex data pipelines, and ensuring the efficient processing of structured and unstructured data. The role requires expertise in RDBMS, cloud data services, and big data technologies, along with strong leadership skills to guide a team of engineers.
• Key Responsibilities:
• Design, develop, and maintain ETL/ELT pipelines for large-scale data processing using AWS Glue, DataStage, PySpark, and Python.
• Manage and optimize relational databases such as PostgreSQL and SQL Server.
• Work with different data formats, including JSON, Parquet, and other structured/unstructured formats.
• Build and maintain robust, scalable data platforms on the AWS ecosystem, leveraging S3 and other AWS services.
• Collaborate with data scientists, analysts, and business teams to understand data requirements and provide solutions.
• Ensure data quality, governance, and security across all pipelines and storage solutions.
• Mentor and guide junior engineers, providing technical leadership and best practices.
• Monitor, troubleshoot, and optimize data workflows for performance and efficiency.

Required Skills:

• Strong hands-on experience with RDBMS: PostgreSQL, SQL Server, or equivalent.
• Expertise in ETL/ELT tools such as DataStage and AWS Glue.
• Proficiency in Python and PySpark for data processing and transformation.
• Experience working with AWS cloud services, especially S3.
• Knowledge of different data formats: JSON, Parquet, and other structured/unstructured formats.
• Strong understanding of data architecture, modeling, and pipeline optimization.
• Excellent problem-solving, analytical, and communication skills.

Preferred Qualifications:

• Experience in cloud data platform design and implementation.
• Familiarity with data governance and security frameworks.
• Previous experience in leading a data engineering team.
• Exposure to big data ecosystem tools (e.g., Spark, Hadoop) in production environments.

Thanks
Deepak