Big Data Developer (Spark & Scala)

Job Title: Big Data Developer (Spark & Scala)

Lead enterprise Big Data transformation as a hands-on Spark/Scala architect designing scalable data pipelines and microservices on Cloudera/HDP platforms. Drive full development lifecycle from requirements gathering through production deployment.

Location: Toronto, ON (New positions Multiple openings)
Experience: 7+ years Big Data + 2+ years Java/Spring

Role Summary

Provide senior-level consulting on Big Data initiatives architecting, developing, and optimizing Spark/Scala applications on Cloudera/HDP. Lead full SDLC from requirements through deployment, mentor development teams, and explore emerging technologies to drive strategic innovation.

Key Responsibilities

Technical Leadership & Architecture

• Architect, design, and implement high-complexity Big Data solutions meeting stakeholder requirements using Spark, Scala, and Java.
• Develop scalable data pipelines processing massive datasets on Cloudera/HDP platforms with optimized Spark jobs.
• Design and implement microservices architecture using Spring Boot and Spring Framework for distributed data processing.

Full Development Lifecycle

• Participate in requirements gathering, data modeling, and design sessions translating business needs into technical specifications.
• Prepare high-level and detailed technical documentation including PLC, security, and architecture standards.
• Develop accurate project plans and estimates for build, testing, and implementation phases.

Development & Quality Assurance

• Write, test, and maintain production-grade code using Java/Scala/Spark with comprehensive unit, system, integration, and acceptance testing.
• Optimize Spark jobs and Java applications for performance, scalability, and cost-efficiency.
• Execute end-to-end testing using industry-standard tools and frameworks.

Required Technical Expertise

Experience Profile (Must-Have)

• 7+ years Big Data development: Hadoop, Spark, Java/Scala
• 2+ years Java/Spring Framework + Spring Boot microservices
• 7+ years full SDLC: architecture test deployment
• Proven team leadership experience
• Cloudera/HDP platform optimization
• Building production data pipelines
• Spark job performance tuning

Nice-to-Have

• Credit Risk Markets domain knowledge (financial services advantage)
• Emerging technologies exploration (Kubernetes, cloud-native Spark)

Success Factors

• Ability to seek improvements across entire development process
• Strong stakeholder collaboration (requirements, design, delivery)
• Technical documentation and training excellence
• Accurate estimation and delivery against timelines
• Cross-platform development expertise

Keywords: Big Data Developer, Spark Scala Developer, Hadoop Spark Java, Cloudera HDP, Spark optimization, data pipelines, microservices architecture, Spring Boot, Spring Framework, Jenkins CI, Git, JIRA, full SDLC, technical architecture, data modeling, ETL pipelines, Spark Streaming, Spark SQL, distributed systems, production code, unit testing, integration testing, acceptance testing, Toronto Big Data, software development lifecycle, team leadership, technical specifications, project estimation, performance tuning, Hadoop ecosystem, Scala programming, Java development

About VDart Group
VDart Group is a global leader in technology, product, and talent solutions, serving Fortune 500 clients in 13 countries. With over 4,000 professionals worldwide, we deliver innovation, operational excellence, and measurable outcomes across industries. Guided by our commitment to People, Purpose, and Planet, VDart is recognized with an EcoVadis Bronze Medal and as a UN Global Compact member, reflecting our dedication to sustainable practices.