Software engineer (Data engineer)

aws, spark, scala, python, etl, data warehouse
Full Time
Depends on Experience
Travel not required

Job Description

Title:: Sr. Software Engineer

Duration: Full time Perm

Location: Remote

open for  H1 Transfer

The Role

If being part of a small dynamic, agile software engineering team practicing TDD with an emphasis on software quality, with a tremendous opportunity to make a big impact, this is the job for you!  This role will provide you with an opportunity to make a huge impact.  You will help drive and maintain a high-level of operational excellence in data engineering.

What You’ll Do

  • Develop integrations to move data into the raw zone of the data lake
  • Build ETL/ELT pipelines to transform data in the raw zone, and load it to the structured and consumer zones of the data lake, and to serving tiers
  • Define and lead the best practices in security, data privacy, quality, and data governance
  • Help lead and collaboratively define Client's next generation data platform
  • Help build online data validation to ensure the assumptions we’ve tested for in our code remain true; outlier and aberration detection, change-detection, etc.
  • Collaborate with teams across the company to help develop data products that drive company success
  • Evaluate, integrate and build tools and infrastructure to accelerate Data Engineering, Data Science, Business Intelligence, Reporting and Analytics as needed
  • Drive data literacy across business functions

What You’ll Need

  • Knowledge of the “Testing Pyramid”, and have helped other engineers apply it correctly
  • You know S.O.L.I.D. principles and practice them intuitively and appropriately
  • Expertise in Scala, Python, and Java
  • Demonstrated expertise in Object Oriented (OO) and Functional programming (FP) including an expert knowledge of common design patterns, idioms, best practices, dependency injection/inversion frameworks and techniques, testing frameworks, Monad-Transformer-Libraries (MTL), Tagless-Final encoding (and when it’s appropriate), etc.
  • Fluency in data structures, algorithms, distributed computing, storage systems, and multiple consistency models
  • In-depth knowledge of AWS (including EMR, DMS, Athena, RDS, Aurora, Lambda, Redshift, etc.)
  • Expertise in stream data processing (e.g., DMS, Flink, Spark, Kinesis, Kafka)
  • Advanced SQL skills
  • Deep Knowledge of multiple database technologies, their tradeoffs, and how to make the best use of each
  • Willingness to learn and mentor in a collaborative team environment
  • Humility with an intrinsic positive drive
  • Passion for developing a world-class engineering culture
  • Value, respect, and an enthusiasm for diversity, inclusion, and alternative perspectives
  • Goal-oriented, with a desire to create an environment of psychological safety
  • Ability to thrive in an environment promoting and enabling collaboration
  • Solid understanding and hands-on experience in computer network
  • MS, or PhD in Computer Science, Mathematics, Computer Engineering, etc., or equivalent experience
  • 8+ years hands-on software engineering experience with a focus on quality; 4+ years in data engineering
  • Expertise with Apache Spark, DataFrame & Dataset API, spark internals and optimization
  • Data warehouse modernization, building data-marts, star/snowflake schema designs, ETL/ELT pipelines
  • Building production-grade data backup/restore strategies, and disaster recovery solutions experience
Dice Id : 10410924
Position Id : 7162913
Originally Posted : 2 weeks ago
Have a Job? Post it

Similar Positions

Senior Pyspark Developer with AWS
  • Virtusa Corporation
  • Princeton, NJ, USA
Big Data Engineer
  • The Silicon Partners Inc.
  • Sterling, VA, USA
Big Data Engineer, Data Platform
  • DISYS - Digital Intelligence Systems, LLC
  • Nashville, TN, USA
Sr Data Engineer
  • Futran Solutions
  • Chicago, IL, USA
Sr Data Engineer SME
  • Futran Solutions
  • Chicago, IL, USA
Big Data Cloud Developer
  • InfoCepts LLC
  • Jersey City, NJ, USA
Sr Data Engineer(Airflow, python, sql, CI/CD, ETL)
  • Next Phase Systems, Inc.
  • San Francisco, CA, USA
Lead Big Data Engineer
  • Net2Source Inc.
  • Emeryville, CA, USA
AWS Data Architect Clinton, NJ
  • Vy Systems
  • Clinton, NJ, USA
Mid-level AWS Data Engineer
  • Randstad Technologies
  • New York, NY, USA