Data Engineer

Job Title: Data Engineer

Location: Remote

Work on business rules and sourcing the data from multiple source systems using IICS. Develop and optimize data pipelines for processing and loading data into Snowflake Create automate data quality checks using Python scripts and SQL queries to ensure data integrity and consistency across different stages of data processing. Manage Python-based ETL pipelines for data extraction, transformation, and loading, improving data accessibility and reporting speed.Develop and optimize complex data transformation pipelines using SparkSQL to process large datasets, improving performance and reducing ETL times. Integrate SparkSQL with Snowflake for scalable data processing and analytics, ensuring seamless data transfers between distributed systems and the Snowflake cloud data warehouse.Work on Snowflake/Python ETL to load data from analytics environment to on Prem databases.Develop and maintain robust data pipelines using AWS Glue, Lambda, and Data Pipeline for real-time and batch data processing, ensuring timely and accurate data flow across systems.Design and implement CI/CD pipelines using tools like Jenkins, GitLab CI, to automate Snowflake database deployment, ensuring faster and error-free releases.Implement end-to-end data pipelines using Azure Data Factory to extract, transform, and load (ETL) data from diverse sources into Snowflake. Write and optimize SQL queries within DBT to enhance data transformation processes and improve overall performance. Design and implement database schemas, stored procedures, and functions. Collaborate with cross-functional teams to design and implement Cosmos DB Schemas aligned with business requirements.Create Airflow DAGs to schedule the ingestions, ETL Jobs and various business reports. Automate repetitive tasks using Python scripts and reducing manual workload.Associate with using Python and PySpark Lambda Functions to create on-demand tables on AWS S3 files.

Required a Bachelors Degree in Computer Science or closely related field for this project.