Data Engineer

Overview

Hybrid
Depends on Experience
Full Time

Skills

Data Engineer
Python
PySpark
AWS
Databricks
Snowflake
Spark
SQL

Job Details

Job Overview: We seek a highly skilled and motivated Data Engineer with strong expertise in Python, PySpark, AWS, Databricks, and Snowflake to join our dynamic team. The ideal candidate should have hands-on experience with Spark optimization, SQL data processing, and AWS tools and technologies. Exposure to Informatica and data streaming tools is a plus.

Key Responsibilities:

  • Design, develop, and optimize scalable data processing pipelines using Spark and PySpark.
    Implement Spark optimization techniques to enhance performance and efficiency.
    Work with complex SQL queries for data manipulation and transformation.
    Collaborate with data scientists, analysts, and engineering teams to meet data needs.
    Work on AWS Cloud services like Redshift, AWS Glue, SQL Server, and Databricks.
    Utilize Python for data processing tasks, ensuring modularization and packaging.
    Basic knowledge of Informatica.
    Engage in real-time data streaming and integration using tools such as NiFi, Kafka, and EventHub (optional but preferred).
    Hands-on experience with Snowflake for data warehousing and analytics.

Required Skills:

Spark Expertise: Strong experience in Spark and PySpark, including optimization techniques.
Complex SQL: Advanced SQL knowledge and data processing skills.
Cloud Exposure: Familiarity with AWS services and Databricks.
Python Proficiency: Experience with Python for data processing and scripting.
Data Streaming: Experience with NiFi, Kafka, and EventHub is desirable.
Snowflake: Hands-on experience with Snowflake for data warehousing and analytics.

Qualifications:
Bachelor s or Master s degree in Computer Science, Information Technology, or related fields.
4-10 years of professional experience in Data Engineering, Data Analytics, or related roles.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

About SG Analytics Inc.