Databricks Python Developer

Job Summary

We are seeking an experienced Databricks Python Developer with strong expertise in Python programming, automated testing, and Pytest. The ideal candidate will develop, maintain, and test data processing applications and pipelines within the Databricks environment, ensuring code quality, reliability, scalability, and performance.

The candidate should have hands-on experience writing Python-based data processing logic, implementing automated unit and integration tests using Pytest, and working with data engineering workflows on Databricks.

Key Responsibilities

• Develop, maintain, and optimize Python applications and data processing workflows on Databricks.

• Build, test, and maintain data pipelines using PySpark and Spark SQL.

• Write and execute automated unit tests, integration tests, and regression tests using Pytest.

• Develop reusable test frameworks, fixtures, mocks, and test utilities for Python applications.

• Validate data quality, data transformations, schema integrity, and pipeline outputs.

• Troubleshoot and resolve issues related to Python code, Spark jobs, and automated test execution.

• Integrate automated testing into CI/CD pipelines to support reliable code deployments.

• Collaborate with data engineers, QA engineers, and other technical teams to ensure software quality.

• Follow coding standards, version control practices, and best practices for test-driven development.

• Monitor application performance and optimize Python and Spark code as needed.

Required Skills

• Strong hands-on experience with Python programming.

• Practical experience with Databricks and its development environment.

• Strong experience with Pytest for automated testing.

• Experience writing unit tests, integration tests, fixtures, parameterized tests, and mocks using Pytest.

• Knowledge of PySpark, Spark SQL, and distributed data processing.

• Experience testing data pipelines, data transformations, and data validation logic.

• Proficiency in SQL and working with structured datasets.

• Experience with Git or other version control systems.

• Understanding of CI/CD practices and automated test execution.

• Strong debugging, analytical, and problem-solving skills.

Preferred Qualifications

• Experience with Azure Databricks or AWS Databricks.

• Familiarity with Delta Lake and Lakehouse architecture.

• Experience with Azure DevOps, Jenkins, GitHub Actions, or similar CI/CD tools.

• Knowledge of test coverage tools such as Coverage.py or pytest-cov.

• Experience with mocking libraries such as unittest.mock.

• Familiarity with data quality frameworks and ETL/ELT processes.

• Experience working in Agile development environments.

Education and Experience

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent practical experience.

• Relevant professional experience developing Python applications and working with Databricks.

• Demonstrated hands-on experience building automated tests using Pytest.

Ideal Candidate Profile

The ideal candidate is a hands-on Python developer who understands both Databricks-based data engineering and automated software testing. The candidate must be comfortable writing production-quality Python code and building reliable Pytest test suites to validate application logic and data pipeline behavior.

Core Skills: Databricks | Python | Pytest | PySpark | SQL | Automated Testing | Data Pipelines | CI/CD