Lead Azure Data Engineer

Position: Lead Azure Data Engineer Experience: 10+ Years Must Have: Proficiency in DataVault, Data warehousing, Azure Cloud, Knowledge on Microsoft Fabric, Databricks or similar platforms

Responsibilities:

• Work in a highly collaborative team environment following the Agile methodology to assist other department personnel in the successful accomplishment of strategic and divisional objectives. Take the necessary steps to ensure our customers' needs are met to the maximum extent possible in an accurate and timely manner. Perform hands-on development and collaborate in the datamanagement efforts including operational datato provide a high level of data integrity, security, and availability for internal and external customers.
• Establish and maintain a scalable, extensible, and maintainable architecture for the enterprise datamanagement and analytics system. Summarize enterprise datainto an intuitive analysis framework for understanding the current and future performance of the company. Provide the design, development, testing and maintenance of Data ingestion, Curation, and Dissemination processes, Data Engineering activities, Reporting and Analytics dashboards as per the Data Strategy roadmap.
• Collaborate with other business unit leaders for various projects involving enterprise data. Ensure the appropriate capture and retention of enterprise data. Work with Business Analysts to translate various business needs into EDM canonical datamapping, analytics, reporting, and dash-boarding requirements. Educate other business units and Information Technology teams about the analytics and reporting advantages available to them through Business Intelligence.
• Provide expertise to achieve system integration through database design, including dataand dimensional modeling, logical and physical table design, complex queries, stored procedures and triggers, datatransformation, aggregation, and enterprise application integration.
• Contribute towards the definition of datastrategy, implementation of Enterprise Datamanagement and Analytics standards and best practices. Mentor, support, and provide direction to lesser experienced teammates, including Offshore data
• Identify & deliver the analytical needs of business units by researching and analyzing datafrom various sources and incorporate datainto existing processes and systems as needed by using expert level understanding of how data, technology, and data work together to serve our customers.
• Conduct analysis on datasets, including efficient extraction, transformation, and analysis of complex datasets. Analyze dataand information in innovative ways to assist with strategic decisions and optimize process efficiencies. Ensure all necessary dataelements are available for model building and strategy analysis.
• Research emerging technologies, trends, and benchmark datato make recommendations to improve customer experience, architecture, processes, and tools.

Knowledge, Skills, Abilities:

• Must have working experience with Raw and Business DataVaultModeling and Engineering as well as building marts using Kimball based methodologies. Must have built data vault models from data requirements . Must understand how to build data quality into data pipelines for data vault and data integration pipelines
• Must have working experience with Azure based DataPipelining , Scheduling and Monitoring and pyspark with ability to debug troublesome pipelines . Must have hands on expertise dealing with datapipelines
• Strong working experience with Big Datatechnologies (Spark, DataBricks) for Data integration, & processing (ingestion, transformation, curation, etc), preferably on Azure cloud, and a clear understanding of how the resources work and integrate with cloud and on-prem.
• High level of proficiency with database and datawarehouse development, including replication, staging, ETL, stored procedures, partitioning, change datacapture, triggers, scheduling tools, cubes, and datamarts.
• Experience working with backend languages such as Python.
• Strong computer literacy and proficiency in datamanipulation using Analytics tools/platform like Databricks, Azure Fabric using Spark engine and reporting tools like Power BI.
• Strong analytical, datamining, datamodeling, and data management skills, Leadership, and judgment to analyze, evaluate, and develop solutions to complex problems.
• Exposure to Azure Fabric is a strong plus along with Automation tools for DataVault
• Creative and critical thinking ability; excellent troubleshooting and problem-solving skills
• Self-motivated and driven to produce quality products.
• Ability to work effectively in a collaborative team environment and effectively demonstrate team building attitude and skills
• Demonstrated ability to be flexible and perform a variety of roles such as analyst, tester, and mentor

Other Desirable Knowledge:

• Experience with AI/ML models/algorithms.
• Experience providing sound technical advice to leadership and other staff in the subject field(s) related to this position.
• Excellent written, verbal and presentation skills
• Experience with a variety of industry standards regarding Microsoft database development concepts, best practices, and procedures
• Experience with financial services industry is a strong plus
• Microsoft MCDBA Certified, CBIP or cloud datawarehouse certification on Azure Cloud is a strong plus.