W2 Databricks Engineer - data lakehouse - W2 Only

Please contact Abdul on "" OR email me at ""

The Pacific Northwest National Laboratory (PNNL), operated by Battelle Memorial Institute on behalf of the U.S. Department of Energy, is seeking an experienced Databricks Engineer contractor to assist in building a new data lakehouse in support of our ERP implementation.

Background PNNL is embarking on a New ERP implementation to modernize its enterprise IT infrastructure and operational capabilities. To ensure seamless data integration, analytics, and reporting, a robust data lakehouse solution is required. This initiative will involve leveraging cutting-edge tools and technologies, including Databricks, Azure Data Lake Storage, Python, and SQL, to build a scalable, high-performing, and secure data foundation. The contractor will play a pivotal role in developing and deploying the data lakehouse solution in collaboration with internal IT teams and third-party vendors working on the ERP implementation.

Scope of Work

The contractor will be responsible for the following activities:

• Analysis & Requirements Gathering: Collaborate with stakeholders, including the ERP implementation team, to gather functional and technical requirements for the data lakehouse.
• Development & Implementation: Implement and configure the Azure Data Lake Storage and Databricks components for data ingestion, transformation, and storage.
• Develop ETL/ELT data pipelines in Python and SQL.
• Build and optimize workflows, primarily on Databricks, for efficient data processing and analytics.
• Testing & Validation: Develop robust testing and validation procedures to ensure data accuracy, availability, and security.
• Test data integration and synchronization between the ERP system and the data lakehouse across environments (development, test, and production).
• Performance Tuning: Optimize the performance, scalability, and cost efficiency of data pipelines and storage solutions across the Azure ecosystem.
• Ensure high availability and fault tolerance for critical business data flows.
• Documentation & Knowledge Transfer: Create comprehensive documentation on architecture design, processes, and configuration.
• Conduct knowledge transfer and training sessions for the internal team to ensure maintainability and scalability of the solution.
• Ongoing Support: Provide troubleshooting assistance and technical expertise during ERP go-live and immediately afterward.

RFP 905257 Attachment A

2

Identify areas for enhancements and suggest data management best practices for long-term success.

Required Skills and Experience

The contractor must possess the following qualifications:

1. Technical Expertise: 3 5 years of hands-on experience with Databricks, including Spark-based data processing and orchestration.
2. Proficiency in configuring and managing Azure services to support data platforms, including provisioning resources, implementing security controls, and optimizing performance for data storage
3. Strong programming and scripting skills in Python for data manipulation, pipeline creation, and automation.
4. Advanced skills in SQL for data analysis, querying, and performance tuning.
5. Experience using Git (i.e., GitHub) and CI/CD pipelines for managing data solutions and version control.
7. Project Experience: Proven ability to integrate systems, automate workflows, and manage large data volumes ranging from structured to unstructured datasets.
9. Soft Skills: Strong communication skills to work with diverse teams, including developers, architects, and business stakeholders.
10. Problem-solving and troubleshooting abilities within complex data environments.
11. Demonstrated ability to deliver projects on time, meeting specified technical and business requirements.

Preferred Skills

1. Experience with Delta Lake architecture (used with Databricks) for efficient transaction handling and ACID compliance.
2. Experience working on data lakehouse implementations in an enterprise IT environment.
3. Demonstrated ability to work on projects involving large-scale ERP platforms (e.g., Oracle).
4. Knowledge of data governance practices, including data security, access control, and cataloging.
5. Certifications such as: Microsoft Certified: Azure Data Engineer Associate
6. Databricks Certified Data Engineer Associate or Professional
8. Familiarity with ERP-related data requirements, such as financial transaction data, supply chain data, or master data management

Deliverables

The contractor will be expected to deliver the following within the agreed timeline:

• Fully operational and tested Azure-based data lakehouse integrated with the ERP platform.
• Detailed data pipelines, scripts, and workflows for ingestion, transformation, and analytics.
• Post-go-live support for troubleshooting and stabilization.
• Documentation and training materials for the internal team.

RFP 905257 Attachment A

3

Work Location

Work will be executed remotely during regular working hours, aligned with the Pacific Time Zone. The contractor will be provided with a government-furnished laptop computer for the duration of the assignment. The contractor must possess and maintain a reliable internet connection at their own expense.

Performance Expectations

Success in this role will be determined by:

1. Timely delivery of the agreed-upon milestones and deliverables.
2. Seamless integration of the ERP system with the data lakehouse, ensuring end-to-end data accuracy and reliability.
3. Implementation of robust, scalable, and well-documented systems.

Work may evolve as the project progresses. The contractor will work under the direction of the ERP Project Manager, the Data Workstream Lead, and the Data Architect to ensure alignment with broader enterprise objectives.

Citizenship

Training

The contractor will be required to complete Cyber Security training before arriving at PNNL. This will be accomplished online through the PNNL Web Portal. The contractor will also be assigned reading materials to become familiar with the PMO Portfolio Management Plan and Playbook.

Please contact Abdul on "" OR email me at ""