Sr. Data Engineer ()

Job Title: Sr. Data Engineer
Location: Remote (USA)

UNDERSTANING OF DATABRICKS ARCHITECTURE IS CRITICAL
Masters preferred and MUST have experience in the Utility Sector

Description:
Client is building a new platform for the state of NY for all utility companies.  Data Engineer will focus on raw data from these utility companies, running Python or Scala scripts to ensure date quality.  Spark is utilized to load date into data frames to aggregate or curate the data, then pass to other teams.  Client also utilizes AWS and Databricks.

Requirements: 

• Bachelor''s degree in computer science, information technology, or a related field.
• At least 7 years of experience in data engineering or a similar role.
• Expert-level skills in Python, SQL databases such as PostgreSQL, and big data technologies such as Databricks and Spark.
• Hands-on experience building cloud resident data pipelines in AWS.
• Strong understanding of data governance, security, privacy, and retention policies and procedures.
• Strong communication, collaboration, and problem-solving skills.
• High proficiency using agile software tools like Jira and following mature DevOps practices using GIT, Docker, and CI servers like Jenkins.
• Passion for data and innovation.

 Preferred Qualifications:
• Demonstrated capacity to work autonomously and proactively, with a proven track record of achieving results without constant supervision.
• Experience with ETL optimization, designing, coding, and tuning big data processes in Databricks.
• Sound knowledge of data lineage and data quality techniques.
• Experience in working with data science and machine learning models and frameworks
• Previous experience in the Energy or Utility industry in an analytic role.
• MS Degree in management information systems, computer programming, software engineering, data science, or an equivalent STEM field.