Senior Data Engineer

  • Posted 1 day ago | Updated 4 hours ago

Overview

On Site
Contract - W2

Skills

Taxes
Cloud Computing
Streaming
Use Cases
Trading
Continuous Integration
Continuous Delivery
JIRA
Python
Data Processing
Stored Procedures
Legacy Systems
Terraform
Management
Orchestration
Step-Functions
Machine Learning (ML)
Artificial Intelligence
GitHub
Unit Testing
Data Quality
Data Governance
Amazon Web Services
Testing
DICE

Job Details

The position is part of a data-focused team working on the "T-Bar" (Transaction Books and Records) product, a critical data store for positions, transactions, and tax lots. The team is transitioning from a legacy data center to a cloud-native AWS environment, aiming to modernize infrastructure and processes by next summer.

Senior Engineer

Must-Have Requirements

  • Experience with Batch and Streaming Data Processing: Ability to handle intraday use cases with trading partners, including micro-batches.
  • CI/CD and Developer Discipline: Proficiency in code commit practices, including tying commits to Jira tickets for traceability and cherry-picking.
  • AWS Familiarity: Experience working within AWS environments, particularly with data pipelines and foundational data layers.
  • Python: Strong skills in Python for data processing and logic conversion (e.g., rewriting stored procedures from legacy systems).
  • Infrastructure as Code (IaC): Experience with Terraform for deploying and managing infrastructure, particularly for carving out separate AWS accounts.

Nice-to-Have Skills

  • Orchestration Tools: Familiarity with Airflow, Dagster, or similar tools for centralized orchestration (current setup includes step functions, Eventbridge, and an in-house eventing system).
  • AI/ML Integration: Exposure to AI tools like GitHub Copilot or Cursor for code development, unit test generation, or data quality checks (e.g., anomaly detection).
  • Data Governance: Experience with data catalogs, producing/consuming assets, or tools like AWS DataZone for governance and contract-based testing.
  • Event-Driven Architecture: Understanding of event-driven systems, as the team aims to move toward this model in the future.
#DICE
#LI-AM1
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.