Lead Data Engineer

Role: Lead Data Engineer

Location: Chicago, IL or Dallas, TX or Boston, MA (Need local to these locations only, F2F Interview required)

Duration: 12+ Months

Visa accepted: GC-EAD, -EAD & L2S.

Job Description:

Top skills: Spark, Scala, SQL, and AWS are the core skillset.

• Lead the design, implementation, and optimisation of large-scale data processing solutions using Scala, Spark, SQL, and modern data platform technologies for a major workstream of the attribution platform.
• Lead the design and operation of trusted data processing pipelines that handle advertiser, customer, and measurement datasets within secure cloud environments (AWS, Google Cloud Platform, Azure) and approved data-sharing ecosystems.
• Collaborate with Product, Data Science, Security, Privacy, and Platform Engineering teams to deliver privacy-preserving attribution, measurement, forecasting, and analytics solutions.
• Own the design and operation of highly scalable batch and streaming data workflows using orchestration frameworks and cloud-native data services.
• Implement data classification, access controls, and privacy-preserving processing techniques to ensure sensitive datasets and identifiers are handled in accordance with security and compliance requirements.
• Drive the design and operation of clean-room and trusted data-sharing environments within your workstream, ensuring only approved aggregated or privacy-protected outputs are made available for downstream consumption.
• Build observability, monitoring, and operational tooling to ensure reliability, performance, and compliance of data processing platforms.
• Troubleshoot complex data platform, performance, and pipeline issues across distributed systems.
• Drive technical design, architecture decisions, and engineering best practices for a major workstream within the attribution platform, in partnership with Staff/Principal engineers.
• Mentor mid-level and senior engineers, lead design and code reviews, and provide technical leadership across your workstream.
• Continuously enhance Epsilon''''s attribution, measurement, forecasting, and privacy-preserving analytics capabilities within your area of ownership.
• Strong written and verbal English communication skills are required.
• Good understanding of Agile/SCRUM methodologies and experience working within cross-functional product development teams.
• Epsilon''''s attribution pipelines process sensitive first-party advertiser data and consumer behavioural signals. A key responsibility of this role is ensuring all data processing occurs within controlled, auditable execution boundaries, no PII or proprietary signals leave the secure perimeter unintentionally.

You will be responsible for:

• Lead the design and implementation of data processing pipelines within trusted data environments, clean rooms, secure data-sharing platforms, or equivalent privacy-preserving analytics environments across AWS, Google Cloud Platform, Azure, and on-premises infrastructure.
• Implement access controls, data classification policies, lineage tracking, and governance controls to ensure PII, PCI-scoped data, customer identifiers, and advertiser-confidential signals are processed only within approved secure environments.
• Collaborate with Security, Privacy, and Compliance teams to define and maintain data handling standards, ensuring sensitive datasets and raw identifiers remain within approved trust boundaries.
• Design data flows that enforce privacy-preserving principles, ensuring only aggregated, anonymised, tokenised, or otherwise approved outputs may leave trusted processing environments.
• Build observability, monitoring, and alerting capabilities to detect anomalous data movement, policy violations, and potential data leakage events.
• Apply privacy-preserving computation techniques where outputs must cross trust boundaries for downstream analytics and reporting, including:
• Aggregation before export
• Pseudonymisation and tokenisation
• Differential privacy concepts and controls
• Privacy-aware reporting and measurement
• Implement encryption, key management, and secure data handling practices using cloud-native security and governance services.
• Document trust boundaries, data contracts, lineage, and permitted data movement between systems and security zones.
• Work closely with Security and Privacy teams to support audits, compliance requirements, governance reviews, and secure data-sharing initiatives.
• Lead architecture and design reviews for new data products within your workstream, ensuring data governance, privacy, lineage, and trust-boundary requirements are incorporated from the outset.
• Help define and uphold engineering standards and best practices for secure data processing, privacy-preserving analytics, and trusted data platform operations.

What you''''ll need

• 14+ years in IT and 8+ years of Data Engineering experience with strong Scala programming and extensive Apache Spark expertise for large-scale distributed data processing on AWS and/or Google Cloud Platform.
• Strong Python development skills for data pipelines, platform tooling, automation, and infrastructure modules .
• Advanced SQL skills across relational databases, cloud data warehouses, and Lakehouse platforms; experience handling TB-scale datasets .
• Experience designing, building, and maintaining batch and streaming data pipelines.
• Strong understanding of data warehousing, dimensional modelling, data quality, partitioning, and performance optimization.
• Experience with distributed data processing and modern Lakehouse architectures (Databricks, Delta Lake, Apache Spark, or equivalent) .
• Experience building and operating distributed data platforms at scale.
• Experience with workflow orchestration platforms such as Airflow, Databricks Workflows, AWS Step Functions, or equivalent DAG-based systems.
• Git or equivalent source control; unit, integration, and automated testing frameworks
• Cloud-native development experience across AWS and/or Google Cloud Platform.
• Strong software engineering practices including CI/CD, code reviews, observability, and production support .
• Proven ability to lead technical design within a workstream, mentor mid-level and senior engineers, drive technical standards, and deliver within tight deadlines.

Trusted environment execution skills (required)

• Experience designing and operating data pipelines within trusted data environments, clean rooms, secure data-sharing platforms, or equivalent privacy-preserving analytics environments
• Experience working with sensitive datasets containing PII, customer identifiers, advertiser data, or regulated information
• Experience implementing fine-grained access controls, data governance policies, and policy-based enforcement for sensitive datasets, including PII, PCI, and other regulated-data classification tiers
• Familiarity with privacy-preserving data processing techniques including tokenization, pseudonymization, aggregation-before-export, and differential privacy concepts
• Experience building or supporting clean-room, measurement, attribution, audience analytics, partner data-sharing, or privacy-preserving reporting solutions
• Experience with data lineage and governance tooling (Unity Catalog, AWS Glue Data Catalog, Apache Atlas, Open Lineage, or equivalent) for auditability and compliance
• Understanding of trust boundaries, secure data-sharing patterns, and zero-trust data architecture principles
• Experience documenting data contracts, data flows, lineage, and permitted movement of data between security zones and business domains
• Experience with encryption, key management, and secure handling of sensitive data using cloud-native security services
• Experience designing observability, monitoring, and alerting controls to detect anomalous data movement, policy violations, and potential data leakage events
• Experience working in environments where only aggregated, anonymized, tokenized, or privacy-protected outputs may leave trusted processing environments
• Strong understanding of cloud-native security and governance

Good to have

• Databricks (Delta Lake, Unity Catalog, Databricks Workflows)
• AWS Clean Rooms or equivalent privacy-enhancing technologies
• Experience building advertising measurement, attribution, audience activation, retail media, or partner data-sharing platforms
• Experience working with Security, Privacy, Risk, or Compliance teams in regulated environments
• ELK Stack, Grafana, Open Telemetry, or equivalent observability platforms
• Docker and Kubernetes
• Security architecture, threat modelling, and secure design reviews