Data Engineering Lead Oil & Gas Trading

Job Title: Data Engineering Lead Oil & Gas Trading

Location: Houston, TX

Duration: 12 Months

Role Overview

• As a Data Engineering Technical Lead, you will design and deliver robust data platforms using Databricks (Delta Live Tables, Structured Streaming, PySpark), Python,
• and SQL on Microsoft Azure. You will work with to build, optimise data pipelines for optimal performance and cost, ensure governance and quality, and collaborate with cross-functional teams to unlock strategic insights from trading data. You will be a key contributor to customers digital transformation, enabling smarter trading decisions and supporting the energy transition through data-driven innovation.

Key Responsibilities

• Technical Leadership
• Architect, Design and implement scalable Azure Data Lakehouse solutions using ADF, ADLS, Synapse, Stream Analytics, and Databricks.
• Lead the development of real-time data pipelines using Databricks DLT and Structured Streaming for high-volume data ingestion and transformation.
• Optimise pipeline performance and cloud resource usage to ensure cost-effective operations.
• Leverage Unity Catalog for metadata management, data lineage, and governance.
• Domain Expertise
• Integrate and manage data from ETRM systems to support Crude & Products trading analytics.
• Apply functional knowledge of commodity trading (Oil & Gas) to align data solutions with business needs.
• Collaborate with trading desks, analysts, and IT teams to deliver actionable insights.
• Governance & Quality
• Establish and enforce data governance frameworks to ensure compliance, security, and data quality.
• Monitor pipeline performance and proactively resolve production issues.
• Stakeholder Engagement
• Communicate effectively with business and technical stakeholders, translating complex technical concepts into clear business goal/value.
• Lead executive-level discussions and presentations to align data initiatives with strategic priorities.
• Team & Capability Building
• Mentor and lead a high-performing team of data engineers and analysts.
• Drive continuous learning and capability development across Azure, Databricks, and streaming technologies.

Required Skills & Experience

Technical Expertise:

• Deep experience with Databricks DLT, Structured Streaming, PySpark, Python, and SQL.
• Strong knowledge of Azure data services (ADF, ADLS, Synapse, Stream Analytics).
• Proven ability to optimise Databricks for high-volume, real-time data processing.
• Experience with ETRM platforms and trading data workflows.
• Leadership & Communication:
• Strong stakeholder management and executive communication skills.
• Demonstrated ownership, accountability, and passion for delivering impactful solutions.
• Ability to lead cross-functional teams and drive strategic data initiatives.
• Education & Certification:
• Bachelor s or Master s degree in IT or related discipline.
• 12+ years of experience in data engineering and analytics.
• Certifications in Azure Data Engineering and Databricks (preferred).

Nice to Have

• Experience with Kafka, Hadoop, Airflow, and Power BI/QlikSense.
• Familiarity with AI/ML, NLP, and GenAI concepts.
• Exposure to Agile/Kanban methodologies and CI/CD practices.

Key Skills: DLT, Structured Streaming, PySpark, ADF, ADLS, Synapse, Stream Analytics, Python & SQL, ETRM / Trading Data (Oil & Gas), Data Governance & Unity Catalog

About VDart Group

VDart Group is a global leader in technology, product, and talent solutions, serving Fortune 500 clients in 13 countries. With over 4,000 professionals worldwide, we deliver innovation, operational excellence, and measurable outcomes across industries. Guided by our commitment to People, Purpose, and Planet, VDart is recognized with an EcoVadis Bronze Medal and as a UN Global Compact member, reflecting our dedication to sustainable practices.