Data Architect - Databricks

Position : Data Architect - Databricks

Location : Bloomington IL

Long Term

Experience with Insurance Domain , Must have experience with Guidewire.

15+ yearsof experience in the IT/Technology and having at least 3+ years of experience in Azure/Databricks

Design and architect Databricks-based solutions that align with business objectives, ensuring scalability, performance, and security.

Designed, Development and implemented Datamesh using DeltaLake.

Provide technical leadership and guidance to the Databricks development team, ensuring best practices are followed throughout the project lifecycle.

Collaborate with cloud infrastructure teams to design and optimize the underlying infrastructure for Databricks workloads on platforms such as AWS or Azure

Develop efficient data ingestion and ETL pipelines using Databricks, Apache Spark, and other relevant technologies.

Integrate Databricks with data lakes and data warehouses to ensure seamless data access and analytics.

Continuously monitor and optimize Databricks workloads for performance and cost-effectiveness.

Implement and maintain security measures, including access controls, encryption, and compliance standards, to protect sensitive data.

Create documentation and provide training to internal teams on Databricks best practices and usage.

Stay up-to-date with the latest developments in Databricks and related technologies to recommend and implement improvements.

Good knowledge on SQL Queries & Stored Procedures.

Strong programming skills in Pyspark, Python, writing complex queries in T-SQL

Experience in transferring data from RDBMS to Data bricks using ADF.

Expertise in using Spark SQL with various data sources like JSON, Parquet and XML.

Experience in creating tables, partitioning, bucketing, loading and aggregating data using Spark SQL/Python.