Big Data (Spark) Lead

A next-generation cross-asset data management platform to provide globally consistent data and innovative tools to support business strategy for trade/sales clients that is built on big data architecture, highly scalable, and cloud-ready. The platform enables industry-leading analytics, client reporting, regulatory compliance, surveillance, supervisory reporting, and data science solutions (data flame). The project tech stack: Apache Spark, Hive, Java, Scala, Spring, SQL, Kafka, Hadoop.

We are looking for a strong Senior Big Data Engineer capable of being able to work within the team, being a mentor for team members, communicating with stakeholders, and being accountable for the whole team and the outcomes of activities that are assigned.

The role of the Tech Lead involves working with strong development team (6 Senior Java/Spark Developers), the opportunity to work with big data improving knowledge in this domain utilizing the resources of an international bank.

Responsibilities

• Provide strong hands-on engineering leadership driving key architectural decisions and implementing frameworks in the big data space

• Lead a team of software engineers delivering engineering solutions solving complex business problems

• Design and build frameworks for ingesting/modeling the complex Risk data

• Develop Low latency caching to service Front office risk systems

The percentage ratio of tasks:

• 80% of technical tasks
• 20% of managerial tasks

Skills

• Tech/Team Lead experience

• Big data development with extensive hands-on experience with Spark programming

• Strong knowledge of big data technologies like Hive and Spark

• Strong Knowledge in one of Java/Scala programming languages

• Should be familiar with algorithms and design patterns

• Understanding of distributed systems.

• Understanding of CI/CD workflow.

• Familiarity with Linux environment including scripting skills

Nice to have

• Experience In-memory databases and caching

• Job Orchestration tools like Autosys or Airflow

• Financial background (where possible)