Data Engineer (Azure, PySpark, Databricks)

Roles and responsibilities:

Data Engineer (Azure, PySpark, Databricks)
Location: Onshore/ USA / Remote
Experience: 4 to 5+ Years
Employment Type: Full-time

Job Summary:
We are seeking a highly skilled and experienced Data Engineer with a strong background in data modeling, PySpark, and Azure Databricks. The ideal candidate will play a key role in building scalable data pipelines, optimizing data architecture, and ensuring robust data management practices across our cloud data platform. You will work closely with data architects, analysts, and business stakeholders to support critical data initiatives.

Key Responsibilities:

• Design, develop, and maintain robust and scalable data pipelines using PySpark and Azure Databricks.
• Implement and optimize complex data models (star/snowflake schemas, normalized/denormalized structures) for analytical and transactional systems.
• Ingest, transform, and store large datasets from diverse structured and unstructured data sources.
• Collaborate with data scientists, analysts, and business users to define data requirements and ensure data quality and governance.
• Monitor and troubleshoot data pipelines and performance issues, applying best practices in data engineering and architecture.
• Implement data orchestration and scheduling using tools like Azure Data Factory, Airflow, or similar.
• Ensure security, compliance, and data privacy in accordance with organizational and regulatory standards.
• Stay updated with the latest tools, technologies, and trends in cloud-based data engineering.

Required Skills & Qualifications:

• 5+ years of hands-on experience in data engineering or related roles.
• Expert in PySpark and distributed data processing frameworks.
• Strong hands-on experience with Azure Databricks and Azure Data Lake Storage (ADLS).
• Proven experience in data modeling for both OLAP and OLTP systems.
• Deep understanding of SQL, performance tuning, and query optimization.
• Familiarity with Delta Lake, Parquet, JSON, and other common data formats.
• Experience in implementing ETL/ELT pipelines and data orchestration using Azure Data Factory or other tools.
• Proficient in using Git, CI/CD pipelines, and DevOps practices for data engineering.
• Strong analytical and problem-solving skills.
• Excellent communication skills and ability to collaborate across teams.

Preferred Qualifications:

• Azure certifications (e.g., DP-203: Azure Data Engineer Associate).
• Experience with data governance tools like Purview or Collibra.
• Knowledge of streaming data and tools like Kafka or Event Hubs.
• Familiarity with BI tools like Power BI for data validation and prototyping.