Senior Azure Data Engineer

The DC Office of the Chief Technology Officer (OCTO) Enterprise Data team needs an ETL Data Engineer for its Cloud Data Exchange, using native Azure tools for ETL, loading, and transformation: building end-to-end pipelines from multiple sources, establishing cloud and on-premise connectivity, and contributing to enterprise data governance. 100% onsite in Washington, DC.

What you’ll do

• Build and support end-to-end ETL pipelines for the Cloud Data Exchange, loading and transforming data from multiple sources
• Design, develop, implement, replicate, and support complex enterprise data projects using cloud solutions
• Establish cloud and on-premise connectivity across systems (data lake storage, data factory, Synapse, Databricks)
• Develop, standardize, and optimize data workflows and pipelines to best practices; automate, monitor, and alert on them
• Contribute to enterprise data governance: accuracy, consistency, security, and reliability of data
• Gather and document requirements with stakeholders; analyze existing code and perform enhancements or new development

What you need

• Bachelor’s degree in Information Technology or a related field, or equivalent experience
• Extensive ETL development experience, including end-to-end pipelines loading from multiple sources (15 years)
• Requirements gathering and documentation for extraction, transformation, and load processes (15 years)
• Data warehousing, data lake, business intelligence, and information-management concepts and standards (15 years)
• SQL for relational databases (11 years) and varied data formats (Oracle, SQL, Postgres, DB2; CSV, XML; Parquet, AVRO) (11 years)
• Microsoft Azure tools: Azure Data Factory, Synapse, SQL Database, Data Lake Storage Gen2, Blob Storage (5 years)

Required skills

• ETL development: end-to-end pipeline design, data loading, and transformation from multiple sources
• Microsoft Azure data engineering (Data Factory, Synapse, Azure SQL, Data Lake Storage Gen2, Blob Storage)
• SQL development for relational databases and complex queries and reports
• Data integration and pipeline orchestration (Informatica PowerCenter, Apache NiFi, Apache Airflow, FME)
• Data warehousing, data lake, and business-intelligence design and standards
• Enterprise data governance: accuracy, consistency, security, and reliability
• Requirements gathering, stakeholder advising, and cross-platform integration