Overview
On Site
Full Time
Skills
Commodities Trading
Analytical Skill
FOCUS
Real-time
Workflow
GitHub
Unstructured Data
Management
Data Integration
Web Scraping
Data Quality
Storage
Innovation
Documentation
Continuous Improvement
Data Security
Regulatory Compliance
Access Control
Computer Science
Mathematics
Python
Apache Spark
SQL
Data Governance
Orchestration
Docker
Kubernetes
Continuous Integration
Continuous Delivery
Microsoft Azure
Databricks
Problem Solving
Conflict Resolution
Attention To Detail
Communication
Data Engineering
Trading
Collaboration
Cloud Computing
Big Data
Open Source
Commodities
Energy Trading
Job Details
A global commodities trading firm is seeking a highly skilled and motivated Data Engineer to join its commercial technology team in New York. The successful candidate will work closely with trading analysts and data engineers across multiple regions to build, maintain, and optimize scalable data pipelines and analytical tools.
This is a hands-on role with a strong focus on Python, Spark, SQL, and cloud-based data engineering, supporting real-time and batch data needs for trading desks.
Key Responsibilities:
- Design, implement, and maintain scalable data pipelines using Python, Spark, and SQL
- Optimize performance and code quality following clean coding principles
- Orchestrate complex workflows using tools like Airflow or Databricks Workflows
- Support trading analysts with ad hoc data engineering needs and develop production-grade solutions
- Implement CI/CD deployment processes using tools like GitHub Actions or equivalents
- Design robust data models and handle both structured and unstructured data efficiently
- Build and manage APIs for data integration and develop web scraping tools as needed
- Ensure data quality through validation frameworks, monitoring, and governance processes
- Optimize storage, retrieval, and processing of external data feeds with varying formats and velocities
- Collaborate closely with global teams (including London-based data engineers) to align with internal standards and best practices
- Proactively drive performance improvements, technology evaluations, and process innovation
- Maintain strong documentation and contribute to a culture of continuous improvement
- Ensure data security, compliance, and access control best practices are in place
Candidate Profile:
- Degree in Computer Science, Mathematics, Engineering, or a related field
- Strong hands-on experience with Python, Spark, and SQL, following clean coding standards
- Solid understanding of data governance, data lineage, and quality frameworks
- Experience with data orchestration tools like Airflow
- Familiarity with containerization technologies (Docker, Kubernetes) and CI/CD pipelines
- Exposure to Microsoft Azure and Databricks is desirable
- Experience with commodities or energy trading data is a plus
- Strong problem-solving skills with excellent attention to detail
- Proactive and entrepreneurial approach to day-to-day challenges
- Excellent communication skills and ability to collaborate across technical and business teams
Why Join?
- Work at the core of data engineering for a fast-paced, high-impact trading environment
- Collaborate with global teams solving large-scale data challenges
- Opportunity to work with modern cloud, big data, and open-source tools
- Gain exposure to commodities and energy trading domains
- Competitive compensation with strong career development potential
This is a hands-on role with a strong focus on Python, Spark, SQL, and cloud-based data engineering, supporting real-time and batch data needs for trading desks.
Key Responsibilities:
- Design, implement, and maintain scalable data pipelines using Python, Spark, and SQL
- Optimize performance and code quality following clean coding principles
- Orchestrate complex workflows using tools like Airflow or Databricks Workflows
- Support trading analysts with ad hoc data engineering needs and develop production-grade solutions
- Implement CI/CD deployment processes using tools like GitHub Actions or equivalents
- Design robust data models and handle both structured and unstructured data efficiently
- Build and manage APIs for data integration and develop web scraping tools as needed
- Ensure data quality through validation frameworks, monitoring, and governance processes
- Optimize storage, retrieval, and processing of external data feeds with varying formats and velocities
- Collaborate closely with global teams (including London-based data engineers) to align with internal standards and best practices
- Proactively drive performance improvements, technology evaluations, and process innovation
- Maintain strong documentation and contribute to a culture of continuous improvement
- Ensure data security, compliance, and access control best practices are in place
Candidate Profile:
- Degree in Computer Science, Mathematics, Engineering, or a related field
- Strong hands-on experience with Python, Spark, and SQL, following clean coding standards
- Solid understanding of data governance, data lineage, and quality frameworks
- Experience with data orchestration tools like Airflow
- Familiarity with containerization technologies (Docker, Kubernetes) and CI/CD pipelines
- Exposure to Microsoft Azure and Databricks is desirable
- Experience with commodities or energy trading data is a plus
- Strong problem-solving skills with excellent attention to detail
- Proactive and entrepreneurial approach to day-to-day challenges
- Excellent communication skills and ability to collaborate across technical and business teams
Why Join?
- Work at the core of data engineering for a fast-paced, high-impact trading environment
- Collaborate with global teams solving large-scale data challenges
- Opportunity to work with modern cloud, big data, and open-source tools
- Gain exposure to commodities and energy trading domains
- Competitive compensation with strong career development potential
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.