Lead Data Engineer
Full Time
No Travel Required
On-site
Depends on Experience
SG Analytics Inc.
Fitment
Dice Job Match Score™
🔗 Matching skills to job...
Job Details
Skills
- Data Engineering
- PySpark
- Databricks
- Python
- Nielsen Data
- Data Pipeline
- Amazon Web Services
- Data Processing
- Media
Summary
About the Role
We are looking for an experienced Lead Data Engineer with deep expertise in Databricks, PySpark, and AWS. The ideal candidate will have a strong understanding of media measurement datasets (Nielsen) and a proven ability to lead multiple concurrent projects across onsite and offshore teams. This role requires a senior technical leader to define architecture, manage delivery, and mentor teams within the fast-paced New York media industry.
Key Responsibilities
- Technical Leadership: Architect and deliver enterprise-scale data platforms using Databricks, PySpark, Delta Lake, and AWS. Lead the development of media analytics pipelines, including the ingestion and modeling of Nielsen data.
- Project & Delivery Management: Manage scope, timelines, and risks for multiple simultaneous projects. Collaborate with New York-based stakeholders and drive sprint delivery.
- Onsite–Offshore Coordination: Act as the primary technical lead onsite, coordinating with offshore teams to ensure alignment, quality, and continuity.
- Team Mentorship: Mentor junior and mid-level engineers through code reviews and architectural guidance.
Required Skills
- Databricks: Delta Lake, Workflows, Unity Catalog, Notebooks, and SQL.
- Data Processing: Strong PySpark and distributed computing experience.
- AWS: S3, EMR, Glue, Lambda, Step Functions, Kinesis, and IAM.
- Domain Expertise: Proven experience handling Nielsen datasets (National/Local TV ratings, Digital Ad ratings, Panels).
- Leadership: Experience leading large data engineering programs and managing senior stakeholders.
Good to Have
- Knowledge of Snowflake, Airflow, or Dagster.
- Infrastructure as Code (Terraform / CloudFormation).
- Exposure to ad-tech ecosystems (Comscore, Kantar, IAS, Moat) or ML feature pipelines.
Soft Skills
- Strong decision-making and ownership mindset.
- Exceptional communication skills for interacting with senior business stakeholders.
- Strategic, problem-solving, and delivery-focused approach.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 90962613
- Position Id: 8914526
- Posted 1 day ago
Company Info
At SG Analytics, We Are Continuously Learning and Growing.
The opportunity of setting up SG Analytics in 2007 came at the most unexpected moment. It arrived in the form of a phone call from a friend in the US who said he wanted to outsource some analytics-related work to a company. Since he had not been able to finalize any deal, he suggested that Sushant take up the work. A choice to meet one client’s investment research requirement has grown exponentially from there.
With our ability to continuously learn and evolve according to market needs, today, we are a 1600-plus global team that offers AI/ML-led data-centric insights and contextual analytics services.
.png%3Fformat%3Dwebp&w=1080&q=75)
Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.Similar Jobs
It looks like there aren't any Similar Jobs for this job yet.
Search all similar jobs