DataBricks Solution Architect

Job Title: Senior Databricks Solution Architect
Location: Raritan, NJ (Onsite)
Duration: Long Term Contract

• Architecture Design: Design scalable and reliable data analytics and machine learning architectures leveraging the Databricks platform to meet business requirements and objectives.
• Platform Implementation: Implement and configure Databricks clusters, workspaces, notebooks, and other components to support data processing, analytics, and model development workflows.
• Data Engineering: Develop and optimize data pipelines for ingesting, processing, and transforming large volumes of data from diverse sources using Databricks and related technologies.
• Data Modeling: Design and implement data models, schemas, and storage structures within Databricks for efficient data organization and querying.
• Machine Learning: Build and deploy machine learning models using Databricks ML frameworks and libraries, and integrate them into production workflows for predictive analytics and decision support.
• Performance Optimization: Optimize Databricks jobs, queries, and workflows for performance, scalability, and cost-efficiency, leveraging best practices and optimization techniques.
• Security and Governance: Implement security controls, access management policies, and data governance frameworks within Databricks to ensure data privacy, compliance, and regulatory requirements.
• Collaboration: Collaborate with data engineers, data scientists, business analysts, and stakeholders to understand requirements, design solutions, and deliver actionable insights using Databricks.
• Documentation and Training: Document Databricks architecture, design decisions, configurations, and best practices, and provide guidance and training to team members on Databricks usage and methodologies.
• Continuous Improvement: Stay abreast of the latest developments in Databricks and related technologies, evaluate new features and capabilities, and recommend enhancements to optimize data analytics workflows.

• Proven experience designing, implementing, and managing data analytics and machine learning solutions using Databricks.
• Strong proficiency in Apache Spark, SQL, Python, and Scala, with hands-on experience developing data pipelines and machine learning models.
• Expertise in cloud platforms such as AWS, Azure, or Google Cloud Platform, with a focus on data services and analytics offerings.
• Experience with data engineering tools and frameworks such as Apache Kafka, Apache Airflow, and Apache Hadoop.
• Knowledge of containerization technologies (e.g., Docker, Kubernetes) and infrastructure-as-code tools (e.g., Terraform) is a plus.
• Excellent problem-solving and analytical skills, with the ability to troubleshoot complex issues and optimize performance in distributed systems.
• Effective communication and collaboration skills, with the ability to work with cross-functional teams in a dynamic environment.
• Databricks certification (e.g., Databricks Certified Associate Developer, Databricks Certified Associate Data Scientist) is desirable but not mandatory.