Azure Databricks Engineer FTE (Only independent Visa holders/W2)

Role :- Azure Databricks Engineer

Location :- Atlanta, GA

Experience :- 9+ Years

Summary:

• The role requires expertise in Azure services like Azure Data Lake, SQL, Cosmos DB, Data Factory, Synapse Analytics, Databricks .
• You will be responsible for designing, developing, and leading data engineering initiatives with lakehouse architecture.
• You will work closely with cross-functional teams to understand business requirements and translate them into technical solutions.
• This role requires strong leadership and communication skills and mentoring data engineering teams.

Key Responsibilities:

• Design, configure, and optimize data architectures and pipelines to meet business needs.
• Develop and embed data engineering pipelines as part of multi tenant B2B SaaS product.
• Work closely with cross-functional teams, to understand business requirements and translate them into technical solutions.
• Migrating existing data to new data model, including logging, reconciliation, data quality and error handling.

Required Skills and Experience:

• Extensive experience in data engineering, including data warehousing, data lakes, and lakehouses.
• Lead the design and implementation of data pipelines for ingesting, transforming, and loading data into Azure data & analytics services for structured and semi structured data.
• Experience in developing data services and pipelines with API integration for multitenant application.
• Develop best practices including reusable code, libraries, patterns, and consumable frameworks for cloud-based data warehousing and ETL.
• Design and implement efficient data models and database schemas for structured and unstructured data, optimizing storage and access for performance and scalability.
• Strong analytical and problem-solving skills, with the ability to interpret complex data sets and derive meaningful insights.
• Extensive experience with Azure data & analytics services, including ADLS, Azure SQL, Cosmos DB, Data Factory, Synapse Analytics, Azure Functions, Azure App Service, Apache Spark, Python, Databricks, and other related services.
• Strong understanding and experience designing and development of data warehousing, data modeling, data integration, and ETL/ELT processes.
• Proficiency in scripting languages like Python, SQL, and/or other languages relevant to data engineering tasks.
• Monitor and Optimize data pipelines and data models for performance, scalability, and cost-effectiveness, identify bottlenecks, and implement optimization strategies.
• Understanding of data governance principles, data security best practices, and compliance requirements.
• Experience with data modeling and database design principles.
• Familiarity with data governance and security best practices.
• Provide advanced troubleshooting and resolution of data pipelines and operations issues.
• Experience with relational databases and data warehouse technologies.

Leadership and Communication:

• Proven leadership skills, with the ability to lead, mentor, and motivate a team.
• Oversee a team of data engineers, providing technical guidance, mentorship, and support.
• Provide guidance and support to team members, lead, mentor, and guide data engineering teams, fostering a collaborative and high-performing environment.

Documentation and Training:

• Develop and maintain documentation, including design documents, data pipelines with governance, Standard Operating Procedures (SOPs) and training materials for the team.
• Excellent written and verbal communication skills, with the ability to effectively communicate technical information to both technical and non-technical audiences.

Experience with Agile Methodologies:

Familiarity with Agile project management methodologies.