Lead Data Engineer

Hybrid in Coopersburg, PA, US • Posted 60+ days ago • Updated 3 days ago

Full Time

No Travel Required

Hybrid

Depends on Experience

Fitment

Dice Job Match Score™

📊 Calculating match score...

Job Details

Skills

Data Engineer
AWS
Redshift
ETL
Python
Athena
RDS
Data Lake
SQL
ELT
Data
Data Pipeline
Data Pipelines
Migration

Summary

Position Title: Data Engineer

Type of Employment: Full time

Role Overview:

We are seeking a highly skilled Data Engineer to develop and manage high volume and complex data pipelines using AWS cloud services and multiple ERP platforms like SAP, Oracle applications, Salesforce etc.

In this role, you will contribute to the overall data architecture, migration strategy, governance model, and implementation best practices for a scalable and secure solutions. You will collaborate with business and technical teams to design solutions that enable advanced analytics, self-service reporting, and enterprise data adoption.

Key Result Areas and Activities

1. Architect & Design

Define and lead the design of enterprise-scale data platforms, including ingestion pipelines, transformation frameworks, and data models for structured and unstructured data.
Drive modernization initiatives by migrating legacy systems to cloud-native architectures (AWS).
Architect scalable, secure, and high-performance solutions leveraging distributed systems, data lakes, and data warehouses.
Establish standards for data modeling, metadata management, and semantic layers to enable advanced analytics and reporting.

2. Development Responsibilities

I: Software Asset Maintenance & Upgrades

Upgrade and maintain third-party components such as ingress controllers, service meshes, monitoring agents, infrastructure libraries, and cloud-native tools.
Apply the latest versions and security patches to ensure compliance, stability, and performance.

II: Infrastructure as Code (IaC) Enhancements

Update and enhance IaC scripts to support version upgrades across development, QA, and production environments.
Validate changes through sandbox testing before deployment to production.

III: Compatibility & Dependency Management

Ensure upgraded components remain compatible with dependent services and applications.
Identify and mitigate potential breaking changes or dependency conflicts.

IV: Application Code Adjustments

Implement necessary code changes in supported languages (e.g., Python) to accommodate new versions or configuration requirements.
Address minor and moderate changes required for compatibility with upgraded components.
Update existing unit tests in response to the application code changes.

V: Security & Compliance

Apply immediate fixes for vulnerabilities.
Maintain adherence to organizational security and governance guidelines.

VI: Testing & Validation

Create and execute test strategies for validating the upgrades.
Execute existing unit tests and manual test cases post-upgrade.
Conduct functional testing of impacted applications to ensure end-to-end stability.
Validate application behavior after code changes and infrastructure updates.

VII: Reporting & Governance

Provide weekly status reports detailing software versions, security posture, upgrade activities, and testing outcomes.
Participate in regular reviews and acceptance processes.

Must-Have Skill Set

Expertise in designing and implementing large-scale data architectures (data lakes, warehouses, streaming platforms).
Deep understanding of cloud data services (AWS) and migration strategies.
Strong proficiency in ETL/ELT pipelines and framework development using Python.
Skilled in building and optimizing data pipelines for batch and real-time processing.
Strong knowledge of data governance, security (row-level, column-level), and compliance frameworks.
Advanced SQL and experience with multiple relational and NoSQL databases (Must have - Redshift, RDS and Athena).
Experience with data visualization platforms like Denodo.
Experience with building scheduled jobs (data load, cache and individual query) in Denodo.
Ability to interpret complex stored procedures and optimize queries for performance.
Familiarity with Agile methodologies and DevOps practices for data engineering.
Exceptional communication skills for engaging executives and non-technical stakeholders.
Knowledge of containerization (Docker, Kubernetes) and orchestration for data workloads.
Experience with BI tools integration with enterprise data platforms.

Qualification:

Bachelor’s degree in computer science, engineering, or related field (master's degree is a plus)
Demonstrated continued learning through one or more technical certifications or related methods
At least 10+ years of relevant experience; two years may be substituted for a master’s degree

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: RTX169204
Position Id: 8821044
Posted 30+ days ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Lead Data Engineer

Dice Job Match Score™

Job Details

Skills

Summary

Similar Jobs