Senior Consultant (Databricks Engineer)

Job#: 2074653

Job Description:

SENIOR CONSULTANT (DATABRICKS ENGINEER)

WHO WE ARE

Apex Systems is a leading global technology services firm that incorporates industry insights and experience to deliver solutions that fulfill our clients' digital visions. We provide a continuum of services, including strategy and enablement, innovation and productivity, and technology foundations to drive better results and bring more value to our clients.?Apex transforms our customers with modern enterprise solutions tailored to the industries we serve. Apex has a presence in over 70 markets across North America, Europe, and India. Apex is a part of the Commercial Segment of ASGN Incorporated (NYSE: ASGN). To learn more, visit .

At Apex Systems, we prioritize professional development, work-life balance, and fostering a collaborative culture. We value our teams well-being and recognize the importance of building strong relationships. That is why we organize regular team-building events and philanthropic days to give back to the community - fostering a sense of purpose and fulfillment among our team.

Join us for career advancement, innovative solutions, and a supportive environment focused on your success.

JOB DESCRIPTION

Apex is seeking a Senior Consultant to serve as a Databricks Engineer with expertise of Databricks tools and frameworks and a background in the Healthcare domain. The individual will play a key role in designing, developing, and operationalizing robust, scalable, and high-performance data pipelines to enable data-driven decision-making through reports, dashboards, and advanced analytics. Key responsibilities include data ingestion, data transformation, data validation/quality assurance, pipeline optimization, workflow orchestration, and adherence to healthcare-focused data standards such as HL7, CCDA, and FHIR. Expertise is required in working with diverse data formats (e.g., delimited files, XML, JSON) and cloud-based data storage solutions.

RESPONSIBLITIES:

• Design and develop scalable data pipelines using Databricks notebooks, clusters, workflows, and integration tools.
• Ingest, process, and generate healthcare data products while adhering to formats including HL7, CCDA, and FHIR.
• Work effectively with delimited file formats (e.g., CSV, TSV), XML, and JSON for ingestion, parsing, and transformation while ensuring data accuracy and compliance with healthcare standards.
• Utilize cloud-based data storage solutions (e.g., Amazon S3, Azure Blob Storage, or Databricks Volumes) for secure and scalable data storage, retrieval, and pipeline integration.
• Implement Delta Lake for transactional and ACID-compliant data processing, optimizing performance and ensuring data reliability.
• Build and manage ETL/ELT processes using Databricks Delta Live Tables and workflow automation.
• Utilize PySpark, Scala, or SQL to create efficient transformation processes within Databricks environments.
• Continuously monitor and improve pipeline performance using Databricks Jobs, metrics dashboards, and cluster tuning techniques.
• Utilize Databricks Unity Catalog for seamless data lineage, access controls, and compliance adherence in compliance with data governance best practices.
• Work closely with cross-functional teams to embed DevOps and CI/CD solutions within Databricks pipelines using tools such as Gitlab, Databricks DABs, or integrations with other DevOps platforms.
• Demonstrate strong technical knowledge and implementation skills. Stay current on relevant technology trends and practices.
• Provide guidance, support, and training to junior engineers, helping them develop technical skills, problem-solving abilities, and professional growth.
• Build trust and respect among internal and external stakeholders and demonstrate collaborative teamwork.
• Produce high quality deliverables, meet project deadlines, and take responsibility for engagement success.
• Demonstrate a passion for quality and process improvement.
• Demonstrate professional level consulting skills and communication/presentation skills.
• Continually innovate, seek creative solutions, and find new ways of adding value. Listen and seek to understand the client and meet their needs, providing consultative guidance.
• Stay attuned to the future needs of the client and work with internal resources to identify opportunities. Proactively provide solutions and approach adversity with a solutions-focused mindset

JOB REQUIREMENTS:

• 5+ years of experience in data engineering, including 3+ years of hands-on experience with Databricks.
• Proven experience in the healthcare domain, with a strong understanding of healthcare[1]focused data standards such as HL7, CCDA, and FHIR.
• Strong knowledge of Delta Lake (schema evolution, time travel, ACID transactions).
• Expertise in PySpark, Scala, and/or Python for distributed data processing and transformation within Databricks environments.
• Strong hands-on experience in Apache Spark and/or Delta Live Tables (DLT) for pipeline development.
• Strong proficiency in PySpark and SQL for building and optimizing large-scale data pipelines.
• Experience designing and implementing ETL/ELT workflows in cloud environments (Azure or AWS).
• Ability to work fluently with delimited file formats (e.g., CSV, TSV), XML, and JSON, ensuring correct parsing and validation of content during ingestion and processing.
• Familiarity with Delta Lake, Unity Catalog, and Databricks Jobs/Workflows.
• Experience working with structured and semi-structured data from APIs, cloud storage, and enterprise systems.
• Solid understanding of modern data architecture concepts (i.e. Medallion architecture, lakehouse patterns).
• Familiarity with Git-based version control, CI/CD practices, and infrastructure automation.
• Excellent problem-solving and communication skills in client-facing environments.
• Bachelor's degree in Computer Science, Information Systems, or equivalent experience.
• Experience with Delta Live Tables (DLT) for declarative pipeline development.
• Exposure to IDMC (Informatica Intelligent Data Management Cloud) or other data integration platforms.
• Databricks or cloud provider certifications (Azure/AWS).
• Familiarity with data governance tools (Unity Catalog, Purview, Collibra) and practices.
• Experience with REST APIs and real-time/streaming architectures (Kafka, Event Hubs, etc.).

OUR COMPREHENSIVE BENEFITS:

• Competitive Salary
• Health, Dental and Vision Insurance
• Health Savings Accounts (HSA) with Employer Contribution
• Flexible Spending Accounts
• Long and Short-Term Disability
• Life Insurance
• Voluntary Benefits
• Employee Assistance Program
• Paid Parental Leave
• Wellness Incentives
• Vacation and Holiday Pay
• 401(k) Retirement Plan with Employer Match
• Employee Stock Purchase
• Training and Advancement Opportunities
• Tuition Reimbursement
• Birthdays Off
• Philanthropic Opportunities
• Referral Program
• Partial Gym Membership Paid
• Team Building Events
• Discount Programs

EEO Employer

Apex Systems is an equal opportunity employer. We do not discriminate or allow discrimination on the basis of race, color, religion, creed, sex (including pregnancy, childbirth, breastfeeding, or related medical conditions), age, sexual orientation, gender identity, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, disability, status as a crime victim, protected veteran status, political affiliation, union membership, or any other characteristic protected by law. Apex will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable law. If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation in using our website for a search or application, please contact our Employee Services Department at or .

Apex Systems is a world-class IT services company that serves thousands of clients across the globe. When you join Apex, you become part of a team that values innovation, collaboration, and continuous learning. We offer quality career resources, training, certifications, development opportunities, and a comprehensive benefits package. Our commitment to excellence is reflected in many awards, including ClearlyRated's Best of Staffing in Talent Satisfaction in the United States and Great Place to Work in the United Kingdom and Mexico.

Apex Benefits Overview: Apex offers a range of supplemental benefits, including medical, dental, vision, life, disability, and other insurance plans that offer an optional layer of financial protection. We offer an ESPP (employee stock purchase program) and a 401K program which allows you to contribute typically within 30 days of starting, with a company match after 12 months of tenure. Apex also offers a HSA (Health Savings Account on the HDHP plan), a SupportLinc Employee Assistance Program (EAP) with up to 8 free counseling sessions, a corporate discount savings program and other discounts. In terms of professional development, Apex hosts an on-demand training program, provides access to certification prep and a library of technical and leadership courses/books/seminars once you have 6+ months of tenure, and certification discounts and other perks to associations that include CompTIA and IIBA. Apex has a dedicated customer service team for our Consultants that can address questions around benefits and other resources, as well as a certified Career Coach. You can access a full list of our benefits, programs, support teams and resources within our 'Welcome Packet' as well, which an Apex team member can provide.