We are seeking a hands-on Mid- Senior level Engineer that can work with technology clients and senior stakeholders to facilitate enterprise data solutions . This person should be able and willing to mentor junior members of the team to help drive solutions. An advocate for innovative, creative technology solutions.
The functional title will be: Data Engineer
- ETL with Python & Hadoop
• Organize business needs into ETL/ELT logical models and ensure data structures are designed for flexibility to support scalability of business solutions
• Craft and implement data pipelines utilizing Glue, Lambda, Spark, and Python
• Define and deliver reusable components for ETL/ELT framework.
• Define optimal data flow for system integration and data migration
• Integrate new data management technologies and software engineering tools into existing structures
Designs and creates more complex logical/physical data models, and data dictionaries that cater to the specific business and functional requirements of applications. Provides support for data architecture efforts as needed. May allocate/coordinate work within a team/project. Develops logical data designs of increasing complexity to deliver stable and flexible high performance data solutions. Performs data extracts and creates complex data reports to analyze user metrics. Develops understanding of business needs and functionalities to determine compatibility of database with existing hardware/software applications and recommend the most cost effective methods. Consults with database administration and client areas in resolving questions during the translation to a physical database design. Provides knowledge of enterprise data to assist the business in the creation and definition of internal and external message flows. Provides knowledge in existing database environments and makes recommendations for opportunities to share data and/or reduce data redundancy. Contributes to the achievement of related teams' objectives.
Bachelor's degree in computer science or a related discipline, or equivalent work experience required, 6-8 years of experience in data modeling, data warehousing, data entity analysis, logical and relational database design, or an equivalent combination of education and work experience required, experience in the securities or financial services industry is a plus.
• 3 to 7 years- experience with ETL/ELT Pipelines,
• Strong knowledge and experience of SQL, Python
• Experience with Big Data/distributed frameworks such as Spark, Kubernetes, Hadoop, and Hive
• Ability to design ETL/ELT solutions based on user reporting and archival requirements
• Strong sense of customer service to consistently and effectively address client needs
• Self-motivated; comfortable working independently under general direction