Data Engineer

{"description": "Requisition ID: 257501

Join a purpose driven winning team, committed to results, in an inclusive and high-performing culture.

We are committed to investing in our employees and helping you continue your career at Scotiabank.

We are looking for a hands on Data Engineer with deep expertise in Apache Spark and strong programming skills in Python, Scala, and Java. This role is centered on building high performance, scalable data pipelines and processing large datasets in a distributed environment.

You will work primarily on Spark based data processing running on Azure Databricks, developing production grade code that supports enterprise analytics, reporting, and data products. This is an engineering heavy role for someone who enjoys writing clean, efficient code and optimizing distributed workloads.

Purpose
Design, develop, and maintain large scale Spark applications using Python, Scala, and/or Java
Build and optimize batch and streaming data pipelines in distributed environments
Write production quality Spark code with strong focus on performance, scalability, and reliability
Optimize Spark jobs (partitioning, caching, shuffles, memory tuning, execution plans)
Develop reusable Spark frameworks, libraries, and utilities
Work with structured and semi structured data (Parquet, Delta, CSV, JSON)
Collaborate with platform, analytics, and data science teams to support downstream use cases
Debug and troubleshoot Spark job failures and performance issues in production
Follow best practices for code quality, testing, logging, and documentation

What You'll Bring
Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience)
3+ years of experience as a Data Engineer or in a similar role
Strong hands on experience with Apache Spark
Proficiency in one or more of the following languages: (Python, Scala or Java)
Strong understanding of distributed systems and data processing concepts
Solid experience with SQL for data transformation and analysis
Experience working with data lakes and columnar formats (e.g., Parquet, Delta Lake)
Familiarity with Git and standard software engineering practices
Strong problem solving skills and attention to detail
Experience running Spark on Azure Databricks an asset.
Experience migrating Spark jobs from on prem Hadoop/Cloudera to cloud platforms an asset.
Familiarity with orchestration tools (e.g., Airflow, Azure Data Factory) an asset.
Exposure to cloud storage (ADLS Gen2) and cloud security concepts an asset.

Interested?

If your experience is closely related but doesn't align perfectly with every qualification, we do encourage you to apply - you might be the right candidate for this or other roles at Scotiabank!
At Scotiabank, every employee is empowered to reach their fullest potential, respected for who they are and, embraced for their differences. That's why we work to grow and diversify talent and engage employees in a performance-oriented culture.

What's in it for you?

Scotiabank wants you to be able to bring your best self to work - and life, every day. With a focus on holistic well-being, our many flexible benefit programs are designed to help support your unique family, financial, physical, mental, and social health needs.

#Dallas

Location(s): United States : Texas : Dallas

Scotiabank is a leading bank in the Americas. Guided by our purpose: \"for every future\", we help our customers, their families and their communities achieve success through a broad range of advice, products and services, including personal and commercial banking, wealth management and private banking, corporate and investment banking, and capital markets.

At Scotiabank, we value the unique skills and experiences each individual brings to the Bank, and are committed to creating and maintaining an inclusive and accessible environment for everyone. If you require accommodation (including, but not limited to, an accessible interview site, alternate format documents, ASL Interpreter, or Assistive Technology) during the recruitment and selection process, please let our Recruitment team know. If you require technical assistance, please click here. Candidates must apply directly online to be considered for this role. We thank all applicants for their interest in a career at Scotiabank; however, only those candidates who are selected for an interview will be contacted.", "salary_raw": "Row(double=None, string=None)"}