Onsite- Sr. Scala/Spark Developer-F2F Must

Job Title: Sr. Scala/Spark Developer

Location: NYC-NY
Duration: 6+ months

Rate: DOE

Job Description

We are seeking a highly skilled Sr. Scala/Spark Developer to design, develop, and optimize large-scale data processing pipelines on cloud platforms. The ideal candidate will have strong expertise in Apache Spark, Scala, and cloud-native big data services with proven experience in building enterprise-grade data platforms.

Responsibilities

• Lead the design and development of data platforms on cloud (Azure preferred) to unify data from multiple sources (mobile, web, APIs, databases, and real-time streams).
• Build scalable data pipelines using Apache Spark (Scala, PySpark) to process large-scale structured and unstructured datasets (10+ TB daily).
• Leverage Azure Databricks, Data Lake, and Delta Lake for high-performance data storage, versioning, and governance.
• Integrate real-time streaming data sources such as Azure Event Hubs / Kafka for transactional and customer interaction analytics.
• Develop REST APIs using Scala (Play Framework) to expose insights and enable data consumption across applications and dashboards.
• Combine Scala and Java in Spark jobs for implementing complex business rules, integrating with legacy APIs, and building reusable components (e.g., encryption, logging, validation).
• Optimize Spark jobs using performance tuning techniques (caching, partitioning, broadcast joins) to improve runtime efficiency.
• Collaborate with front-end and product teams to integrate insights into dashboards (React) and customer-facing applications.
• Implement and operationalize machine learning models in Spark MLlib to support predictive analytics.
• Deploy and manage data applications on Azure Kubernetes Service (AKS) with Azure DevOps CI/CD pipelines.
• Monitor system health and pipeline performance with Azure Monitor, Log Analytics, and configure proactive alerts.
• Ensure data security and compliance via encryption, RBAC, and integration with identity management solutions (Azure AD).
• Mentor junior developers, conduct peer code reviews, and promote best practices in big data engineering.
• Drive technical discussions to evaluate new tools, frameworks, and design patterns for continuous improvement.
• Document workflows, designs, and reusable components to accelerate onboarding and knowledge sharing.

Required Skills

• 8+ years of software development experience with at least 3 5 years in Apache Spark and Scala.
• Strong expertise in distributed computing, functional programming, and large-scale ETL pipelines.
• Hands-on experience with Azure Databricks, Data Lake, Delta Lake, and Event Hubs (or equivalent in AWS/Google Cloud Platform).
• Proficiency in Scala, PySpark, and Java, with the ability to integrate legacy modules into modern Spark pipelines.
• Experience with REST API development (Scala/Play Framework).
• Solid understanding of data modeling, governance, and compliance best practices.
• Experience with Azure DevOps, CI/CD pipelines, and Kubernetes (AKS) for deployment.
• Knowledge of Spark MLlib or other ML frameworks for predictive analytics is a plus.
• Strong problem-solving, communication, and mentoring skills.