Lead/Senior Data Engineer - Scala/Spark/Kafka/Azure

Job Title: Senior Data Engineer
Job Type: Full Time
Remote Eligible: Yes

Skills: Scala programming, Spark, Kafka, Azure

Responsibilities:

• Streamline the intake of the raw data into our Azure Data lake.
• Perform production support and deployment activities.
• Proactively drive the execution of our core data engineering, business intelligence, and data warehouse frameworks
• Build data pipelines from systems such as CRM, Ecommerce etc. with the emphasis on scalability and reliability
• Leverage central data warehouse along with other data sources to create enriched customer information into CRM system
• Understand and translate business needs into data models to support long-term, scalable, and reliable solutions
• Create logical and physical data models using best practices to ensure high data quality and reduced redundancy
• Drive data quality across the organization; develop best practices for standard naming conventions and coding practices to ensure consistency of data models and tracking
• Define and manage SLA's for data sets and processes running in production
• Continuously improve our data infrastructure and stay ahead of technology
• Design a system for data backup in case of system failure
• Build strong cross-functional partnerships with Data Scientists, Analysts, Product Managers and Software Engineers to understand data needs and deliver on those needs

We are excited about you if you have:

• 5+ years of experience with large scale data warehouse projects .
• Advanced Proficiency in Scala, Spark and SQL.
• Excellent knowledge in data structures and design patterns
• Preferred experience with data integration tools
• Experience working with Retail/CRM/Finance datasets preferred
• Knowledge in designing facts, dimensions, snapshots, SCDs, etc.
• Write SQL for processing raw data, data validation and QA
• Knowledge working with APIs to collect or ingest data
• Strong Database knowledge, SQL & No-SQL preferred
• Communication Skills Data Engineers are part of a team, working with database administrators, data analysts and management and need to be effective communicators.
• Attention to Detail Databases are complex, and a minute error can cause huge problems.
• Problem-Solving Skills Data Engineers look at an issue that needs to be solved and come up with solutions quickly.
• Experience designing and implementing cloud based and SaaS data warehouse (e.g. WS, Hadoop, NoSQL) and developing ETL/ELT pipelines
• Experience integrating and building data platform in support of BI, Analytics, Data Science, and real-time applications
• Strong communication skills, with the ability to initiate and drive projects proactively and accurately