Overview
Remote
$165,000 - $180,000
Full Time
No Travel Required
Skills
Scala
Spark
Kafka
Azure
Databricks
Data
Engineering
Cloud
Job Details
Job Title: Senior Data Engineer
Job Type: Full Time
Remote Eligible: Yes
Skills: Scala programming, Spark, Kafka, Azure
Responsibilities:
- Streamline the intake of the raw data into our Azure Data lake.
- Perform production support and deployment activities.
- Proactively drive the execution of our core data engineering, business intelligence, and data warehouse frameworks
- Build data pipelines from systems such as CRM, Ecommerce etc. with the emphasis on scalability and reliability
- Leverage central data warehouse along with other data sources to create enriched customer information into CRM system
- Understand and translate business needs into data models to support long-term, scalable, and reliable solutions
- Create logical and physical data models using best practices to ensure high data quality and reduced redundancy
- Drive data quality across the organization; develop best practices for standard naming conventions and coding practices to ensure consistency of data models and tracking
- Define and manage SLA's for data sets and processes running in production
- Continuously improve our data infrastructure and stay ahead of technology
- Design a system for data backup in case of system failure
- Build strong cross-functional partnerships with Data Scientists, Analysts, Product Managers and Software Engineers to understand data needs and deliver on those needs
We are excited about you if you have:
- 5+ years of experience with large scale data warehouse projects .
- Advanced Proficiency in Scala, Spark and SQL.
- Excellent knowledge in data structures and design patterns
- Preferred experience with data integration tools
- Experience working with Retail/CRM/Finance datasets preferred
- Knowledge in designing facts, dimensions, snapshots, SCDs, etc.
- Write SQL for processing raw data, data validation and QA
- Knowledge working with APIs to collect or ingest data
- Strong Database knowledge, SQL & No-SQL preferred
- Communication Skills Data Engineers are part of a team, working with database administrators, data analysts and management and need to be effective communicators.
- Attention to Detail Databases are complex, and a minute error can cause huge problems.
- Problem-Solving Skills Data Engineers look at an issue that needs to be solved and come up with solutions quickly.
- Experience designing and implementing cloud based and SaaS data warehouse (e.g. WS, Hadoop, NoSQL) and developing ETL/ELT pipelines
- Experience integrating and building data platform in support of BI, Analytics, Data Science, and real-time applications
- Strong communication skills, with the ability to initiate and drive projects proactively and accurately