Data engineer

Data Engineer (PySpark & Databricks)

Location :Jersycity NJ

Only on W2 payroll

10+ years Experience.

Job Summary

We are looking for a skilled Data Engineer with strong expertise in PySpark and Databricks to design, build, and optimize scalable data pipelines and cloud-based data platforms. The ideal candidate will work closely with data analysts, data scientists, and business stakeholders to deliver reliable and high-performance data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using PySpark.
• Build and manage data workflows on the Databricks platform.
• Process large-scale structured and unstructured datasets efficiently.
• Optimize Spark jobs for performance, scalability, and cost.
• Integrate data from multiple sources including APIs, databases, and cloud storage.
• Develop and maintain data lakes and data warehouse solutions.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Implement data quality, validation, monitoring, and governance processes.
• Troubleshoot production issues and provide performance tuning recommendations.
• Create technical documentation and maintain coding standards and best practices.

Required Skills & Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or related field.
• 3+ years of experience in Data Engineering or Big Data development.
• Strong hands-on experience with:
  • PySpark
  • Databricks
  • Apache Spark
  • SQL
• Experience with cloud platforms such as:
  • AWS
  • Azure
  • Google Cloud Platform
• Knowledge of data pipeline orchestration tools like:
  • Airflow
  • Azure Data Factory
  • Jenkins
• Experience with Delta Lake, Data Lake architecture, and distributed computing.
• Strong understanding of ETL concepts and data modeling.
• Familiarity with version control tools such as Git.
• Good problem-solving and communication skills.

Preferred Qualifications

• Databricks certification is a plus.
• Experience with streaming technologies such as Kafka or Spark Streaming.
• Knowledge of CI/CD pipelines and DevOps practices.
• Exposure to machine learning pipelines is an advantage.

Technical Stack

• Programming: Python, SQL
• Big Data: PySpark, Apache Spark
• Platform: Databricks
• Cloud: AWS/Azure/Google Cloud Platform
• Databases: Snowflake, Redshift, PostgreSQL, MySQL
• Tools: Airflow, Git, Jenkins

Soft Skills

• Strong analytical and troubleshooting abilities
• Ability to work independently and collaboratively
• Good communication and stakeholder management
• Attention to detail and quality-focused mindset