Senior Data Engineer

Responsibilities:

• Design, construct, install, test, and maintain highly scalable data management systems on Azure and AWS, utilizing services such as Azure Data Lake, Azure Data Factory, AWS S3, and AWS Glue.
• Integrate new data management technologies and software engineering tools into existing structures, focusing on Azure and AWS ecosystems.
• Utilize Azure Databricks and AWS EMR for big data processing tasks to create data set processes for data modeling, mining, and production.
• Develop analytics applications using Azure Synapse Analytics and AWS Redshift to leverage the data pipeline, providing actionable insights into key business performance metrics.
• Collaborate with stakeholders across the Executive, Product, Data, and Design teams to support their data infrastructure needs, using Azure and AWS cloud services.
• Implement data security and compliance procedures using Azure Security Center and AWS Identity and Access Management (IAM).
• Evaluate and improve existing systems, incorporating new technologies and software engineering tools available in Azure and AWS ecosystems.
• Monitor analytics and metrics results using tools like Azure Monitor and AWS CloudWatch.
• Implement systems for tracking data quality and consistency using Azure and AWS services, ensuring data integrity across the pipeline.
• Automate data workflows and pipeline processes using Azure Logic Apps and AWS Step Functions, integrating with Apache Airflow for orchestration.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or related field.
• Proven experience as a Data Engineer or similar role, with a deep understanding of Azure and AWS cloud services relevant to data engineering (e.g., Azure Data Lake, Azure Data Factory, AWS S3, AWS Glue).
• Expertise in SQL and programming languages, especially Python, for data manipulation and analysis.
• Strong experience with big data tools such as Apache Spark and Hadoop.
• Proficiency in data pipeline and workflow management tools, particularly Apache Airflow.
• Experience with data modeling, data warehousing, and building ETL pipelines in both Azure (using Azure Data Factory, Azure Databricks) and AWS environments (using AWS Glue, AWS Data Pipeline).
• Knowledge of stream-processing systems (e.g., Apache Kafka, Azure Stream Analytics, Amazon Kinesis) is a plus.
• Familiarity with machine learning algorithms and data science principles is beneficial.
• Strong problem-solving skills and ability to project manage and multitask.
• Excellent communication and teamwork skills, with an ability to navigate and negotiate with stakeholders across the organization.