Data Engineer

Purpose of Position:

We are looking for an experienced and highly skilled Data Engineer to join our Data Governance Team. The ideal candidate will have a strong background in data engineering with specific experience in Databricks. You will be responsible for designing, building, and maintaining scalable data pipelines and architectures to enable advanced analytics and data-driven decision-making across the organization.

Essential Job Functions:

1. Design and develop scalable data pipelines using Databricks, ensuring efficient data processing and integration from various sources.
2. Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions that meet their needs.
3. Implement best practices for data management, storage, and retrieval, focusing on optimizing performance and ensuring data integrity.
4. Work with large datasets and perform data cleansing, transformation, and aggregation to support analytics and reporting functions.
5. Monitor and troubleshoot data pipeline performance, ensuring high availability and reliability.
6. Stay updated with the latest advancements in data engineering and Databricks technologies to continuously improve our data infrastructure.
7. Document data processes and workflows, ensuring transparency and knowledge sharing across the team.
8. Ensure data security and compliance with regulatory requirements and industry best practices.

1. Perform other duties as directed.

Knowledge, Skills and Experience Requirements :

1. Bachelor's degree in Computer Science, Information Technology, Engineering, or equivalent experience
2. 3+ years of experience as a Data Engineer, preferred proven expertise in using Databricks.
3. Strong proficiency in data processing tools and languages such as SQL, Python, and Spark.
4. Experience with cloud platforms, particularly Azure, and integration with Databricks.
5. Understanding of data architecture principles and experience with ETL tools and methodologies.
6. Excellent problem-solving skills, with the ability to manage multiple priorities in a fast-paced environment.
7. Strong communication skills, with the ability to collaborate effectively with cross-functional teams.

Working Conditions:

This is a full-time position operating on a hybrid schedule with three days per week in the 275 7th Ave, NY office.