Data Engineer Databricks & Scala

Position- Data Engineer – Databricks & Scala

Location- Boston, Massachusetts 02210 ( Onsite )

Duration- 12 Months Contract

Job Description:

We are looking for a Data Engineer with strong hands-on experience in Databricks and Scala to join our data engineering team. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and solutions using modern data processing technologies. You will work closely with data scientists, analysts, and business stakeholders to ensure efficient data flow and quality.

Key Responsibilities:

• Design and develop scalable data pipelines and ETL/ELT workflows using Apache Spark on Databricks.
• Write clean, efficient, and reusable Scala code for data transformations and pipeline orchestration.
• Optimize large-scale data processing jobs for performance and reliability.
• Integrate data from various sources (structured and unstructured) into data lakes and data warehouses.
• Collaborate with data analysts and BI teams to support analytics and reporting requirements.
• Implement data quality checks, monitoring, and alerting for production pipelines.

Required Skills:

• Strong hands-on experience with Databricks and Apache Spark.
• Proficiency in Scala for data engineering tasks.
• Solid understanding of data lake, data warehouse, and cloud-based data architectures (e.g., AWS, Azure, or Google Cloud Platform).
• Experience with Delta Lake, Parquet, and other big data formats.
• Knowledge of CI/CD tools and version control (e.g., Git, Jenkins).

Preferred Qualifications:

• Experience with Python or SQL is a plus.
• Familiarity with Airflow, dbt, or other orchestration tools.
• Exposure to data governance, data cataloging, and security practices.