Urgent need for an Lead Data Engineer to join our team at Dataeconomy
Contract: 6 months
Corp to Corp/W2/1099
Location: New Jersey
Demonstrates thought leader-level abilities with, and/or a proven record of success directing efforts in Cloud Platforms (Azure), Architecture and Design
Expected to be able to function in a fast-paced environment driving innovation through rapid prototyping and iterative development ensuring quality is built into all solutions leveraging TDD.
Utilise expertise of developing enterprise-level metadata driven data pipeline applications, with demonstrable production-scale experience.
Experience in Data Quality, Data Governance and Data management designs.
Design, develop, code, and test complex technical solutions collaborating with senior engineers.
Working knowledge of Microservice and Event Driven Architecture.
Lead code/design reviews to ensure smooth daily operations and accurate planning at a team level.
Mentor and coach junior / experienced engineers.
Work with architecture team to help define directions for cross functional or highly complex key technologies within a specific domain.
Experience with designing and building metadata driven data pipelines using pyspark , databricks or other ETL frameworks .
Ability to design and implement data services using configurations and metadata.
Working experience with Gradle, GIT, GitHUB, GITLab, etc. around continuous integration and continuous delivery infrastructure
12+ years of software engineering/development experience utilizing python and pyspark
10+ Years of large scale Enterprise Software Development.
4+ Years of experience in Docker or cloud-based applications
4+ years in Data Processing frameworks with focus on metadata driven services development using Pyspark.
3+ years in AGILE methodology (Scrum, Lean, SAFe, etc.)
8+ years experience integrating with backend services like JMS, J2C, ORM frameworks (Hibernate, JPA, JDO, etc), JDBC.
8+ Years leading development teams or mentoring junior developers.
Java//J2EE, Spring , Spring Boot
Azure SQL , Cosmos Db , MongoDb
Azure Data Lake
Databricks , HDInsight
Event Hub , Azure Functions, Azure Data Lake