Overview
Remote
Depends on Experience
Full Time
Skills
Databricks
AWS
Job Details
Key Responsibilities
- Design and implement data pipelines using AWS and Databricks.
- Manage and orchestrate data workflows, ensuring efficient data ingestion processes.
- Utilize Delta Lake for data storage and management, ensuring secure data sharing across platforms with Delta Sharing.
- Implement data governance and access control using Unity Catalog.
- Apply the Medallion architecture (Bronze, Silver, Gold layers) to projects.
- Develop and manage ETL pipelines using AWS Glue.
- Leverage Amazon S3 for data storage and management.
- Utilize AWS Lambda and Step Functions for serverless computing in data processing.
- Apply data warehousing solutions and best practices in data transformation.
- Monitor and implement data quality checks and governance practices.
- Collaborate with data scientists, analysts, and other stakeholders in an Agile environment.
- Deliver large-scale data processing and analytics projects within cloud environments.
Required Skills and Qualifications
- Databricks certifications with practical experience in the Databricks Lakehouse platform.
- Proficient in AWS services: Glue, S3, Lambda, Step Functions, and data warehousing solutions.
- Strong programming skills in Python, Scala, or Java.
- Proficient in SQL for querying and manipulating data.
- Experience in data modeling for Lakehouse architecture.
- Knowledge of ETL best practices and data transformation techniques.
- Understanding of data governance frameworks and practices.
- Proven track record of delivering data engineering projects, particularly in cloud environments.
- Experience with large-scale data processing and analytics projects.
- Ability to work in Agile teams and collaborate effectively with various stakeholders.
Preferred Qualifications
- Advanced AWS certifications.
- Experience with additional cloud platforms (e.g., Azure).
- Familiarity with other data engineering tools and frameworks.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.