We are seeking a skilled Senior Data Software Engineer with strong expertise in PySpark , SQL , and unit testing to join our data engineering team. The ideal candidate will have hands-on experience with Apache Spark , preferably within the Databricks environment, and will be responsible for building scalable data pipelines, optimizing data workflows, and ensuring code quality through rigorous testing practices. Responsibilities Design, develop, and maintain scalable data pipelines using Apache Spark (Databricks preferred) Write efficient and optimized PySpark code for data transformation and processing Develop and execute complex SQL queries for data extraction, validation, and reporting Implement unit tests using pytest to ensure code reliability and maintainability Collaborate with data scientists, analysts, and other engineers to deliver high-quality data solutions Monitor and troubleshoot data workflows and performance issues Document technical designs, processes, and best practices Requirements 3+ years of experience in Data Software Engineering Proven experience with Apache Spark, ideally in a Databricks environment Proficiency in PySpark and SQL Background in unit testing frameworks, especially pytest Understanding of data engineering principles and ETL processes Familiarity with version control systems (e.g., Git) Ability to work independently and in a collaborative team setting Excellent problem-solving and communication skills Proficiency in English at an Upper-Intermediate level (B2) or higher Nice to have Experience with cloud platforms (e.g., Azure, AWS, Google Cloud Platform) Knowledge of CI/CD pipelines and DevOps practices Familiarity with Delta Lake, MLflow, or other Databricks-native tools
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 10330481
- Position Id: 1d0462818485c4300a22994ce5328969
- Posted 2 hours ago
Never miss an opportunity! Create an alert based on the job you applied for.