Lead/Senior Data Engineer - Scala/Spark/Kafka/Azure


$165,000 - $180,000
Full Time
No Travel Required



Job Details

Job Title: Senior Data Engineer
Job Type: Full Time
Remote Eligible: Yes

Skills: Scala programming, Spark, Kafka, Azure


  • Streamline the intake of the raw data into our Azure Data lake.
  • Perform production support and deployment activities.
  • Proactively drive the execution of our core data engineering, business intelligence, and data warehouse frameworks
  • Build data pipelines from systems such as CRM, Ecommerce etc. with the emphasis on scalability and reliability
  • Leverage central data warehouse along with other data sources to create enriched customer information into CRM system
  • Understand and translate business needs into data models to support long-term, scalable, and reliable solutions
  • Create logical and physical data models using best practices to ensure high data quality and reduced redundancy
  • Drive data quality across the organization; develop best practices for standard naming conventions and coding practices to ensure consistency of data models and tracking
  • Define and manage SLA's for data sets and processes running in production
  • Continuously improve our data infrastructure and stay ahead of technology
  • Design a system for data backup in case of system failure
  • Build strong cross-functional partnerships with Data Scientists, Analysts, Product Managers and Software Engineers to understand data needs and deliver on those needs

We are excited about you if you have:

  • 5+ years of experience with large scale data warehouse projects .
  • Advanced Proficiency in Scala, Spark and SQL.
  • Excellent knowledge in data structures and design patterns
  • Preferred experience with data integration tools
  • Experience working with Retail/CRM/Finance datasets preferred
  • Knowledge in designing facts, dimensions, snapshots, SCDs, etc.
  • Write SQL for processing raw data, data validation and QA
  • Knowledge working with APIs to collect or ingest data
  • Strong Database knowledge, SQL & No-SQL preferred
  • Communication Skills Data Engineers are part of a team, working with database administrators, data analysts and management and need to be effective communicators.
  • Attention to Detail Databases are complex, and a minute error can cause huge problems.
  • Problem-Solving Skills Data Engineers look at an issue that needs to be solved and come up with solutions quickly.
  • Experience designing and implementing cloud based and SaaS data warehouse (e.g. WS, Hadoop, NoSQL) and developing ETL/ELT pipelines
  • Experience integrating and building data platform in support of BI, Analytics, Data Science, and real-time applications
  • Strong communication skills, with the ability to initiate and drive projects proactively and accurately
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.