Job Summary The Data Engineer will be responsible for designing, developing, and maintaining scalable data integration solutions that deliver business value. This role focuses on building robust data pipelines, developing ETL processes, and managing large-scale data processing using Python, PySpark, AWS services, and modern data engineering practices while ensuring data quality, security, and performance. Key Responsibilities Design, build, and maintain scalable data pipelines. Develop and optimize ETL workflows for data ingestion and transformation. Design and develop large-scale batch processing, data manipulation, data mining, and data extraction solutions. Collaborate with product teams, application teams, and business analysts to support data requirements. Ensure data quality, integrity, and consistency across enterprise data platforms. Implement data security and compliance best practices. Monitor, troubleshoot, and optimize data pipeline performance. Document data engineering processes, workflows, and technical solutions. Design mapping specifications, High-Level Designs (HLD), and Low-Level Designs (LLD). Develop, code, and perform unit testing for data engineering solutions. Develop and maintain REST API integrations. Develop UNIX scripts and Oracle SQL/PL-SQL solutions as required. Utilize workload automation tools such as Control-M or AutoSys. Participate in CI/CD DevOps processes and Agile/SCRUM ceremonies. Stay current with emerging data engineering technologies and AI tools to improve productivity and delivery. Required Qualifications Bachelor's degree in Computer Science or a related field. Minimum of 8 years of hands-on data engineering experience. Strong experience developing solutions using Python and PySpark. Experience with AWS services, including Glue, Lambda, MSK (Kafka), S3, Step Functions, RDS, and EKS. Experience working with databases such as PostgreSQL, SQL Server, Oracle, and Sybase. Strong SQL programming experience, including performance tuning, relational data modeling, stored procedures, views, functions, and triggers. Experience designing and implementing scalable data pipelines and ETL processes. Experience with data modeling, data warehousing, data mining, data analysis, and data profiling. Experience working with REST APIs. Experience with workload automation tools such as Control-M or AutoSys. Experience with CI/CD DevOps processes and tools including GitHub, Bitbucket, and Jenkins. Strong understanding of Agile/SCRUM methodologies. Knowledge of UNIX scripting and Oracle SQL/PL-SQL. Ability to learn and effectively utilize AI tools. Strong analytical, troubleshooting, and problem-solving skills. Preferred Qualifications Experience with additional ETL tools such as IBM DataStage, Informatica, or Pentaho. Knowledge of Master Data Management (MDM). Experience with Data Warehouse and Data Analytics solutions. Education: Bachelors Degree
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: compun
- Position Id: HAJDC5864679
- Posted 1 hour ago
Never miss an opportunity! Create an alert based on the job you applied for.
