Full time / Permanent role
Job Description
Role Purpose
Design, build, operate, and improve reliable data pipelines and distributed data-processing solutions while maintaining data quality, security, governance, and operational stability.
Must-Have Technical and Functional Skills
Primary Skills
- PySpark
- Hive
- Python
- SQL Server
- Hadoop
Secondary Skills
- Unix
- Agile delivery practices
- Production support
Required Experience and Responsibilities
- Demonstrated experience in Data Engineering, Software Engineering, or a related technical discipline, with the ability to perform the responsibilities of the role.
- Strong proficiency in Python and SQL for advanced data transformations.
- Hands-on experience designing and building ETL/ELT pipelines, data-ingestion processes, and distributed data-processing jobs.
- Practical experience with distributed data tools such as Apache Spark, Databricks, or Hadoop ecosystems.
- Experience building and managing datasets in relational and/or cloud-based data platforms, such as Teradata, Snowflake, SQL Server, Azure, AWS, or Google Cloud Platform.
- Solid understanding of data modeling, metadata, data-quality controls, data lineage, and secure data management.
- Experience contributing to automated test suites, analyzing test failures, and supporting test-driven development.
- Knowledge of CI/CD pipelines, version control using Git, and automated deployment practices.
- Experience adhering to enterprise data-governance, compliance, and operational-risk frameworks.
- Ability to troubleshoot pipeline issues, performance bottlenecks, and data discrepancies.
- Strong communication skills and the ability to collaborate across engineering, product, and business teams.
- Experience implementing monitoring and observability for data pipelines, including logs, metrics, and health checks.
- Advanced experience tuning the performance of SQL, Spark, or distributed data workflows.
- Knowledge of data-security practices, including encryption, masking, and PII handling.
- Experience supporting analytical workloads, BI tools, or data-science teams.
- Prior experience in a financial institution or another regulated industry.
Preferred Skills and Experience
- Production support experience, including root-cause analysis of production issues across Hadoop, Oracle, and MongoDB environments.
- Knowledge of Autosys.
- Experience troubleshooting performance issues, supporting application-governance requirements, implementing version-upgrade changes, and delivering small enhancements.