This position requires a mandatory in person interview and in person onboarding.
Senior Data Engineer Contract Length: Until 12/31/26 with possible extension
Location: New York, NY
Work Setup: Hybrid
We are seeking an experienced
Senior Data Engineer to design, develop, and optimize scalable data platforms that power enterprise analytics, machine learning, and AI-driven solutions. This role will build reliable, high-performance data ecosystems using Databricks, Spark, Delta Lake, Snowflake, Kafka, and AWS while supporting modern Generative AI initiatives.
Senior Data Engineer Responsibilities - Design, build, and maintain scalable batch and real-time data pipelines for structured, semi-structured, and unstructured data.
- Implement Lakehouse architectures using Databricks, Delta Lake, and Medallion (Bronze, Silver, Gold) design patterns.
- Develop distributed processing and streaming solutions using PySpark, Spark SQL, Kafka, Spark Structured Streaming, and AWS Kinesis.
- Architect AWS-based solutions using S3, EMR, EC2, Athena, Redshift, RDS, Lambda, IAM, SNS, and SQS.
- Optimize performance through partitioning, clustering, caching, query tuning, and cost-efficient storage and processing strategies.
- Ensure data quality, governance, lineage, security, and compliance across data ecosystems.
- Develop orchestration and scheduling frameworks using Airflow and Databricks Workflows and build CI/CD automation.
- Support migrations from traditional Hadoop and EMR environments to modern cloud-native platforms.
- Implement Generative AI solutions leveraging LLMs, RAG, Vector Databases, AI Agents, and Model Context Protocol (MCP).
- Collaborate with architects, analysts, data scientists, and business stakeholders to deliver enterprise-grade solutions.
Senior Data Engineer Required Qualifications - Bachelor's or Master's degree in Computer Engineering, Computer Science, Information Systems, or a related field.
- 8+ years of experience in software engineering, data engineering, or big data platform development.
- Strong experience designing and implementing enterprise-scale data pipelines.
- Hands-on expertise with:
- Python
- PySpark
- Spark SQL
- SQL
- Kafka
- Databricks
- Delta Lake
- Snowflake
- Hive
- Experience building data solutions on AWS cloud platforms.
- Strong understanding of distributed computing, data modeling, and large-scale data processing.
- Experience with Git-based development workflows and CI/CD practices.
- Excellent analytical, troubleshooting, and problem-solving skills.
Senior Data Engineer Preferred Qualifications - Experience with real-time streaming architectures and event-driven systems.
- Knowledge of data governance, metadata management, and data quality frameworks.
- Experience with generative AI technologies including:
- LLMs
- RAG
- Vector Databases
- AI Agents
- MCP integrations
- Experience developing developer productivity tools and AI-assisted engineering workflows.
- Exposure to enterprise supply chain, retail, healthcare, or manufacturing data domains.
- AWS certifications are highly preferred.
Senior Data Engineer Technical Skills Python, Java, SQL, Shell Scripting, C/C++, PySpark, Spark SQL, Hive, Databricks, Delta Lake, Snowflake, Kafka, HBase, Sqoop, Apache Airflow, Databricks Workflows, Oozie, AWS, Tableau, Git, Docker, Splunk, IntelliJ IDEA, PyCharm, Cursor, Generative AI, LLMs, RAG, Agentic AI Systems, MCP, and Vector Databases.
Senior Data Engineer Preferred Certifications AWS Certified Solutions Architect Associate, AWS Certified Cloud Practitioner, and Databricks Certifications (preferred).
What Success Looks Like The Senior Data Engineer will deliver highly scalable and reliable data pipelines, improve platform performance and cost efficiency, enable enterprise analytics and AI initiatives through trusted data products, modernize data platforms, and leverage AI technologies to enhance engineering productivity, automation, and innovation.
We look forward to reviewing your application!
#NoLinkedIn
#tech
IND123
Never miss an opportunity! Create an alert based on the job you applied for.
