Spark Data Lead

Role: Spark Data Lead

Job Description:

Working as a Data Engineer with real-time, production-level systems.

Experience working with real-time data systems like Spark Structured Streaming and Kafka.

Strong experience in Apache Spark using PySpark or Scala including performance tuning.

Good knowledge of Datazone for managing data access and data governance.

Excellent in writing SQL and understanding modern data architecture like Lakehouse or Delta Lake.

Experience working with cloud platforms like AWS, Azure, or Google Cloud especially using services like S3, ADLS, Redshift, or Synapse.

Worked with orchestration tools like Apache Airflow or Athena to manage data workflows.

Built systems to ingest data (load data from source files and databases).

Developed data frameworks using metadata configurations (e.g., ABCR metadata).

Used PySpark or Scala to load, transform, and send data to tools like Kafka.

Optimized Spark jobs for better performance.

Processed data using joins, filters, and aggregations based on business needs.

Strong understanding of data governance, including who can access what data and how to track it.

Comfortable with DevOps tools like Git, Terraform, and CI/CD pipelines.

Familiar with data catalog and metadata management tools.

Knowledge of security features like RBAC (role-based access control), row-level security, and data masking. Exposure to machine learning tools like MLflow, Feature Store, and working with ML pipelines. Contributed to open-source projects or community work related to Spark or Databricks.