Scala Expert Data Engineer

Fulltime Role (Remote  

The skill set we're seeking includes ETL, Spark, Strong Scala, and Azure/AWS.

As a Lead Engineer, you will lead, design and implement innovative analytical solutions and work alongside the product engineering team, evaluating new features and architecture. Reporting to the Director, Analytics and Personalization, you will help drive decisions and build collaborative relationships with key functional groups in the organization. In addition, you will:

• Responsibilities 

  • Design & Develop batch and streaming solutions using various Big Data technologies.
  • Have experience working with real-time or near real-time ingestion.
  • Streamline the intake of the raw data into our Azure Data Lake. 
  • Be responsible for assessing technologies and approaches for ingestion, transformation, and storage.
  • Designs & builds frameworks for various common data engineering processes like ingestions, mastering, data cleansing, and data masking.
  • Prepare and present potential technical designs and solutions for Data pipelines and DataOps in Azure cloud environment.
  • Ability to collaborate and partner with business domain leads, data scientists, product owners, enterprise architects and other functional leads to deliver world class data solutions.
  • Diagnose and address complex problems, including performance issues, scale, and drive to resolution to meet business initiatives.
  • Perform production support and deployment activities.
  • Build data pipelines from systems such as CRM, Ecommerce etc. with the emphasis on scalability and reliability.
  • Understand and translate business needs into data models to support long-term, scalable, and reliable solutions.
  • Create logical and physical data models using best practices to ensure high data quality and reduced redundancy
  • Continuously improve our data infrastructure and stay ahead of technology.
  • Build strong cross-functional partnerships with Data Scientists, Analysts, Product Managers and Software Engineers to understand data needs and deliver on those needs

   

  We’re excited about you if you have:  

  • 8 - 10 years of experience in data engineering and designing scalable ETL pipelines
  • Advanced Proficiency in Hadoop, Scala, Spark, Kafka and SQL
  • 6+ years of experience with cloud platforms like Azure, AWS, Google
  • Be skilled at architecting large and complex data pipelines.
  • Experience performing data analysis and data exploration.
  • Experience working with Cloud technologies, Azure, Databricks, ADLS, Spark, Cosmos DB, and other big data technologies.
  • Experience working in an agile delivery environment.
  • Experience working in multi-developer environment, using version control.
  • Excellent knowledge in data structures and design patterns 
  • Preferred experience with data integration tools 
  • Work with cross functional agile teams to drive projects through full development cycle.
  • Help the team improve with the usage of data engineering best practices.
  • Write SQL for processing raw data, data validation and QA 
  • Knowledge working with APIs to collect or ingest data 
  • Strong Database knowledge, SQL & No-SQL preferred 
  • Communication Skills Data Engineers are part of a team, working with database administrators, data analysts and management and need to be effective communicators. 
  • Problem-Solving Skills Data Engineers look at an issue that needs to be solved and come up with solutions quickly. 
  • Experience integrating and building data platform in support of BI, Analytics, Data Science, and real-time applications.
  • Strong communication skills, with the ability to initiate and drive projects proactively and accurately