We are seeking a Senior Data Engineer to architect and optimize our mission-critical financial data platforms. In this role, you will build ultra-reliable, high-throughput pipelines that power quantitative analysis, regulatory reporting, and risk management. You will bridge traditional enterprise financial infrastructure with modern cloud-native systems—developing in Python and Java, scaling batch and streaming workflows via Hadoop and Apache Spark, and deploying secure, containerized architectures across On-Premises systems and Amazon EKS.
Core Responsibilities:
• Financial Pipeline Engineering: Build and maintain high-performance, deterministic ETL/ELT pipelines using Python and Java to process market feeds, transaction ledgers, and trade data.
• Big Data & Analytics Scale: Optimize and maintain high-volume data frameworks utilizing Hadoop (HDFS, Hive, Yarn) and Apache Spark for large-scale risk modeling and backtesting.
• Orchestration & Cloud Native: Deploy and scale containerized financial data applications and Spark streaming jobs natively inside Amazon EKS with zero-downtime tolerance.
• Hybrid Systems Integration: Securely manage data continuity, replication, and data sovereignty between legacy, air-gapped On-Premises mainframes/servers and AWS environments.
• Audit & Data Lineage: Implement comprehensive data governance, immutability frameworks, and end-to-end data lineage tracking to comply with financial regulations (e.g., BCBS 239, SOX, GDPR).
• Performance Tuning: Troubleshoot complex, distributed processing jobs to minimize latency for time-sensitive financial computations and trading day closing processes.
Required Technical Skills:
• Languages: Advanced proficiency in Python (Pandas, PySpark) and Java (Spring Boot, concurrency frameworks).
• Distributed Computing: Mastery of Apache Spark (Core, Streaming, SQL) and the Hadoop ecosystem.
• Containerization & Cloud: Deep execution knowledge of Docker and Kubernetes/Amazon EKS cluster orchestration.
• Databases & Storage: Advanced SQL, time-series databases (e.g., Kdb+, InfluxDB), and relational platforms.
• Security Frameworks: Experience embedding enterprise security (Kerberos, IAM Roles, TLS encryption, KMS) into data pipelines.
Domain & Experience Qualifications
• Experience: 5+ years of data engineering experience, preferably within investment banking, fintech, or quantitative hedge funds.
• Financial Domain: Foundational knowledge of financial instruments, market data structures, trade lifecycles, or risk management frameworks.
• Education: Bachelor’s or Master’s degree in Computer Science, Financial Engineering, or a related quantitative field.
