Position:Databricks Engineer
Location:100% Remote.
Duration:This is a 3 month contract role with the possibility of extension
Job Description:
As a Senior Engineer in the Databricks practice, you will bring a curious mindset and a passion for
exploring innovative solutions to address our clients’ most pressing data challenges. You are a selfstarter who excels at breaking down complex problems and eagerly shares insights with your team
and the broader Builder community.
Key Responsibilities:
• Develop and implement data solutions using Databricks, with hands-on experience in specific Databricks platform features such as Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
• Collaborate with cross-functional teams to design and optimize data pipelines for both batch and streaming data, ensuring data quality and efficiency. • Stay updated with emerging technologies, latest Databricks platform features and continuously improve your skills in Databricks and other relevant tools. Requirements:
• 5+ years of data engineering experience, with at least 2 years of hands-on data pipeline design and development experience with Databricks, including specific -platform features like Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative
pipelines), and Unity Catalog. • Proficiency in designing and building robust, scalable, YAML configuration driven data
pipelines with batch, micro-batch, and streaming data ingestion and processing patterns using tools like Auto Loader demonstrating a strong understanding of modern data engineering practices.Experienced in building complex job/workflow orchestration patterns
using either Databricks jobs/workflows or with other orchestration tools like Airflow, dbt etc.
• Exposure in building robust data quality, pipeline audit/observability solutions with Databricks native features and/or other data quality tools/frameworks like Great Expectations, Collibra, dbt etc. • Proficiency in Big Data Platforms: Apache Spark, Presto, Amazon EMR.
• Experience with Cloud Data Warehouses: Amazon Redshift, Snowflake, Google BigQuery.
• Strongprogramming skills using SQL, Stored Proceduresand Object-Oriented Programming languages (like Java, Python, PySpark etc.).
• Familiarity with building DevOPs, CI/CD pipelines using Databricks Asset
Thanks & Regards
Karthik Kaloori
Resource Manager | IDEAON
|
