Lead Data Engineer, Databricks Migration

Job#: 3022102

Job Description:

8-10 Years Experience | Databricks Migration & Data Architecture

We are seeking a highly skilled Lead Data Engineer with deep hands-on experience in modern data engineering, BI platforms, and large-scale data migrations. This role requires strong technical leadership, end-to-end solution design capabilities, and the ability to work closely with both onshore and offshore engineering teams.
Core Technical Requirements

• 8-10 years of software development and data engineering experience
• Minimum 7 years hands-on experience with:
  • SQL, Databricks, ADF, Datastage (or other ETL tools)
  • SSAS cubes, Cognos, Tableau, ThoughtSpot, and other BI tools
• Advanced SQL and Python proficiency
• Strong database foundation across SQL and NoSQL systems
• Experience writing SQL for raw data processing, Kafka ingestions, ADF pipelines, data validation, and QA
• Familiarity with APIs for ingestion and data integration
• Experience with modern cloud platforms (Azure, AWS, or equivalent)
• Experience with Spark, Hive, and other distributed compute engines
• Proficiency with CI/CD frameworks such as GitHub, Jenkins, etc.
• Programming/scripting familiarity: Scala, Java, Python, R (any combination)

Key Responsibilities
Data Architecture & Modeling

• Design scalable, future-state data architectures and data models
• Translate business requirements into logical and physical data models
• Ensure data quality, governance, and reduced redundancy

Data Integration & ETL

• Lead data integration from diverse sources (databases, warehouses, external systems)
• Design and implement ETL/ELT pipelines to ensure data accuracy, consistency, and availability

Solution Engineering & BI Implementation

• Evaluate and implement BI tools, reporting platforms, and analytics technologies
• Conduct POCs and guide tool configuration and rollout
• Lead development of analytical solutions using SQL, Databricks, Tableau, and related technologies

Performance Optimization

• Monitor and tune performance across data pipelines and BI environments
• Optimize queries, data structures, and caching strategies

Leadership & Collaboration

• Serve as the technical lead to offshore engineering teams
• Work cross-functionally with product, engineering, data, and business stakeholders
• Translate complex concepts for both technical and non-technical audiences
• Participate in design reviews, code reviews, and solution walkthroughs

Continuous Improvement

• Stay current on modern data engineering trends and best practices
• Identify opportunities for automation, optimization, and innovation

Governance & Security

• Establish and maintain data governance policies
• Define access controls, data privacy practices, and retention standards

Additional Expectations

• Experience writing technical documentation, including design docs and source-to-target mappings
• Ability to convert business requirements into clear, actionable Jira stories
• Strong analytical skills: capable of working with large datasets to identify trends and insights
• Experience building highly scalable BI solutions and distributed data applications
• Comfortable working in a fast-paced, agile, startup-like environment
• Strong interpersonal skills; collaborative, clear communicator, and able to mentor others
• Self-motivated, organized, and capable of managing multiple priorities
• Enjoys contributing to a positive, fun, high-performing team culture

Apex Systems is a world-class IT services company that serves thousands of clients across the globe. When you join Apex, you become part of a team that values innovation, collaboration, and continuous learning. We offer quality career resources, training, certifications, development opportunities, and a comprehensive benefits package. Our commitment to excellence is reflected in many awards, including ClearlyRated's Best of Staffing in Talent Satisfaction in the United States and Great Place to Work in the United Kingdom and Mexico. Apex uses a virtual recruiter as part of the application process. Click for more details.

Apex Benefits Overview: Apex offers a range of supplemental benefits, including medical, dental, vision, life, disability, and other insurance plans that offer an optional layer of financial protection. We offer an ESPP (employee stock purchase program) and a 401K program which allows you to contribute typically within 30 days of starting, with a company match after 12 months of tenure. Apex also offers a HSA (Health Savings Account on the HDHP plan), a SupportLinc Employee Assistance Program (EAP) with up to 8 free counseling sessions, a corporate discount savings program and other discounts. In terms of professional development, Apex hosts an on-demand training program, provides access to certification prep and a library of technical and leadership courses/books/seminars once you have 6+ months of tenure, and certification discounts and other perks to associations that include CompTIA and IIBA. Apex has a dedicated customer service team for our Consultants that can address questions around benefits and other resources, as well as a certified Career Coach. You can access a full list of our benefits, programs, support teams and resources within our 'Welcome Packet' as well, which an Apex team member can provide.