Sr. Data Engineer (Databricks and Unity Catalog)

Position: Sr. Data Engineer (Databricks and Unity Catalog)

100% Remote

Hiring: Contract / Fulltime

Top Required skills =  Databricks and Unity Catalog

Key Responsibilities:

• Design and implement scalable data pipelinesand ETL workflows using Databricks.
• Implement Unity Catalogfor data governance, access control, and metadata management across multiple workspaces.
• Develop Delta Lake architecturesfor optimized data storage and retrieval.
• Establish best practices for data security, compliance, and lineage trackingin Unity Catalog.
• Optimize data lakehouse architecturefor performance and cost efficiency.
• Collaborate with data scientists, engineers, and business teamsto support analytical workloads.
• Monitor and troubleshoot Databricks clusters, performance tuning, and cost management.
• Implement data quality frameworksand observability solutions to maintain high data integrity.
• Work with Azure/AWS/Google Cloud Platform cloud environmentsto deploy and manage data solutions.

Required Skills & Qualifications:

• 10 years of experience in data engineering, data architecture, or cloud data solutions.
• Strong hands-on experience with Databricks and Unity Catalog.
• Expertise in PySpark, Scala, or SQL for data processing.
• Deep understanding of Delta Lake, Lakehouse architecture, and data partitioning strategies.
• Experience with RBAC, ABAC, and access control mechanisms within Unity Catalog.
• Knowledge of data governance, compliance standards (GDPR, HIPAA, etc.), and audit logging.
• Familiarity with cloud platforms (Azure, AWS, or Google Cloud Platform)and their respective data services.
• Strong understanding of CI/CD pipelines, DevOps, and Infrastructure as Code (IaC).
• Experience integrating BI tools (Tableau, Power BI, Looker) and ML frameworks is a plus.
• Excellent problem-solving, communication, and collaboration skills.