Lead Consultant (Databricks Lead)

Job#: 2074655

Job Description:

LEAD CONSULTANT (DATABRICKS LEAD)

WHO WE ARE

Apex Systems is a leading global technology services firm that incorporates industry insights and experience to deliver solutions that fulfill our clients' digital visions. We provide a continuum of services, including strategy and enablement, innovation and productivity, and technology foundations to drive better results and bring more value to our clients.?Apex transforms our customers with modern enterprise solutions tailored to the industries we serve. Apex has a presence in over 70 markets across North America, Europe, and India. Apex is a part of the Commercial Segment of ASGN Incorporated (NYSE: ASGN). To learn more, visit .

At Apex Systems, we prioritize professional development, work-life balance, and fostering a collaborative culture. We value our teams well-being and recognize the importance of building strong relationships. That is why we organize regular team-building events and philanthropic days to give back to the community - fostering a sense of purpose and fulfillment among our team.

Join us for career advancement, innovative solutions, and a supportive environment focused on your success.

JOB DESCRIPTION

Apex is seeking a dynamic Lead Consultant with strong analytical skills to serve as a Databricks Lead Engineer and SME within Apex Systems Consulting Services. In this role, you will be leveraging deep expertise of Databricks capabilities and best practices to provide expert-level support and guidance in the implementation, customization, and optimization of Databricks solutions to our clients. You will own the technical direction of Databricks-based solutions, mentor engineering teams, and drive best practices in data pipeline development, performance tuning, and data quality.

RESPONSIBLITIES:

• Lead the design and development of scalable data pipelines using Databricks notebooks, clusters, workflows, and integration tools.
• Demonstrate strong technical knowledge and implementation skills. Stay current on relevant technology trends and practices. Build trust and respect among internal and external stakeholders and demonstrate collaborative teamwork.
• Produce high quality deliverables, meet project deadlines, and take responsibility for engagement success. Demonstrate a passion for quality and process improvement.
• Demonstrate professional level consulting skills and communication/presentation skills.
• Continually innovate, seek creative solutions, and find new ways of adding value. Listen and seek to understand the client and meet their needs, providing consultative guidance. Stay attuned to the future needs of the client and work with internal resources to identify opportunities. Proactively provide solutions and approach adversity with a solutions-focused mindset.
• Identify and evaluate new team members through professional networking and technical screening.
• Perform assessments of clients environments and design robust solutions and roadmaps.
• Ingest, process, and generate healthcare data products while adhering to formats including HL7, CCDA, and FHIR.
• Work effectively with delimited file formats (e.g., CSV, TSV), XML, and JSON for ingestion, parsing, and transformation while ensuring data accuracy and compliance with healthcare standards.
• Utilize cloud-based data storage solutions (e.g., Amazon S3, Azure Blob Storage, or Databricks Volumes) for secure and scalable data storage, retrieval, and pipeline integration.
• Implement Delta Lake for transactional and ACID-compliant data processing, optimizing performance and ensuring data reliability.
• Build and manage ETL/ELT processes using Databricks Delta Live Tables and workflow automation.
• Utilize PySpark, Scala, or SQL to create efficient transformation processes within Databricks environments.
• Continuously monitor and improve pipeline performance using Databricks Jobs, metrics dashboards, and cluster tuning techniques.
• Collaborate with stakeholders and business units to define data governance practices via Databricks Unity Catalog for seamless data lineage, access controls, and compliance adherence.
• Work closely with cross-functional teams to embed DevOps and CI/CD solutions within Databricks pipelines using tools such as Gitlab, Databricks DABs, or integrations with other DevOps platforms.
• Coordinate with cross-functional teams to ensure successful project execution and stakeholder satisfaction.

JOB REQUIREMENTS:

• 7+ years of experience in data engineering, including 4+ years of hands-on leadership with Databricks and Apache Spark.
• Proven track record leading end-to-end pipeline development projects in cloud environments (Azure or AWS).
• Expertise in PySpark, SQL, and Delta Lake for building and optimizing large-scale pipelines.
• Experience architecting Medallion architectures and lakehouse solutions.
• Hands-on experience with Delta Live Tables (DLT), Databricks Workflows, and Unity Catalog.
• Strong knowledge of CI/CD pipelines, version control (Git), and infrastructure automation.
• Familiarity with structured and semi-structured data integration from APIs, cloud storage, and databases.
• Excellent client-facing communication, consulting, and leadership skills.
• Bachelor's degree in Computer Science, Information Systems, or equivalent experience.
• Healthcare experience, particularly with HL7, CCDA, and FHIR standards, is a plus.
• Certifications in Databricks or major cloud platforms (Azure/AWS).
• Familiarity with data governance (Unity Catalog) and data security practices.
• Background in performance tuning and cost optimization of big data workloads.

OUR COMPREHENSIVE BENEFITS:

• Competitive Salary
• Health, Dental and Vision Insurance
• Health Savings Accounts (HSA) with Employer Contribution
• Flexible Spending Accounts
• Long and Short-Term Disability
• Life Insurance
• Voluntary Benefits
• Employee Assistance Program
• Paid Parental Leave
• Wellness Incentives
• Vacation and Holiday Pay
• 401(k) Retirement Plan with Employer Match
• Employee Stock Purchase
• Training and Advancement Opportunities
• Tuition Reimbursement
• Birthdays Off
• Philanthropic Opportunities
• Referral Program
• Partial Gym Membership Paid
• Team Building Events
• Discount Programs

EEO Employer

Apex Systems is an equal opportunity employer. We do not discriminate or allow discrimination on the basis of race, color, religion, creed, sex (including pregnancy, childbirth, breastfeeding, or related medical conditions), age, sexual orientation, gender identity, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, disability, status as a crime victim, protected veteran status, political affiliation, union membership, or any other characteristic protected by law. Apex will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable law. If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation in using our website for a search or application, please contact our Employee Services Department at or .

Apex Systems is a world-class IT services company that serves thousands of clients across the globe. When you join Apex, you become part of a team that values innovation, collaboration, and continuous learning. We offer quality career resources, training, certifications, development opportunities, and a comprehensive benefits package. Our commitment to excellence is reflected in many awards, including ClearlyRated's Best of Staffing in Talent Satisfaction in the United States and Great Place to Work in the United Kingdom and Mexico.

Apex Benefits Overview: Apex offers a range of supplemental benefits, including medical, dental, vision, life, disability, and other insurance plans that offer an optional layer of financial protection. We offer an ESPP (employee stock purchase program) and a 401K program which allows you to contribute typically within 30 days of starting, with a company match after 12 months of tenure. Apex also offers a HSA (Health Savings Account on the HDHP plan), a SupportLinc Employee Assistance Program (EAP) with up to 8 free counseling sessions, a corporate discount savings program and other discounts. In terms of professional development, Apex hosts an on-demand training program, provides access to certification prep and a library of technical and leadership courses/books/seminars once you have 6+ months of tenure, and certification discounts and other perks to associations that include CompTIA and IIBA. Apex has a dedicated customer service team for our Consultants that can address questions around benefits and other resources, as well as a certified Career Coach. You can access a full list of our benefits, programs, support teams and resources within our 'Welcome Packet' as well, which an Apex team member can provide.