Lead Data Engineer with ADF

Please let me know your interest below. raj AT alliedinformatics DOT net |

Title: Lead Data Engineer

Location: Remote

Duration: Long term

*****Consultnat need to come on our W2*****

Skills / Experience
• Azure Data Factory (ADF) data movement expert
• Strong SQL Skills using SQL Server – ability to profile data, verify data movement, validate data matches, Azure SQL managed instance experience, experience in working with large enterprise databases
• Comfortable with documenting and drawing data flows, data mappings, etc.
• Extensive Data Ingestion experience
o Familiar with best practices for data flow patterns for data ingestion into Data warehouse including progression through Staging, ODS and Data Warehouse (Dims/Facts) schemas
o Experienced handling Production data loads and designing error handling and alerting into all steps to best support
o Experience with data retention/archiving policies and procedures
o Experience with both event triggers and schedule triggers to trigger pipeline start
o Experience ingesting from SQL and file based sources, with Azure blob storage experience

Sample Tasks
• Existing ADF data ingestion pipelines:
o Review existing pipelines to improve performance, data security, data quality
o Define tasks needed to address any issues found during the review, including detailing what work is needed, LOE
o Assist in identifying root cause of data flow issues with existing pipelines
o Suggest areas of optimization, reuse of pipelines
o Verify level of error handling and alerting is sufficient for all eventualities.
• New ADF data ingestion pipelines:
o Design new ADF pipelines for data ingestion for new sources including error handling, alerts, notifications
o Break up work for new pipelines to identify tasks, LOE, etc.
o Enter work into DevOps
o Monitor progress in Dev Ops
o Review work
o Deploy changes to QA, Production
• Production Support – support production data loads during normal business hours (daytime). Able to perform root cause analysis on any issues found. Ability to determine best way forward to restart pipeline, backfill missed data, etc.

Thanks & Regards,
Raj
raj AT alliedinformatics DOT net