Databricks Data Engineer

Overview

Remote
Depends on Experience
Full Time
10% Travel

Skills

Amazon Web Services
Consulting
Data engineering
Data management
Data integration
Data warehouse
python
databricks
azure
snaplogic

Job Details

Job Description

The Data Engineer Senior Consultant plays a critical role in the implementation of Enterprise Data Management (EDM) Solutions for Datasource Consulting/EXL Clients. They are responsible for setting the development standards and best practices, and may be required to assist with requirements gathering/analysis, development, testing, documentation, and deployment tasks. This is a business-facing role in which the consultant will work independently and collaboratively with internal and client teams to ensure project success. The Data Engineer Senior Consultant should have practical experience developing EDM solutions, strong working knowledge of the technologies listed below, and an eagerness/ability to learn and train on new technologies. You will be working closely with a team of data management consultants, and should be able to multiple hats in multi-functional and multi-shore project teams. This position may require travel to client sites in the U.S. (when safe to do so).

Knowledge and Skills

  • 5-8 years experience in developing and implementing ETL pipelines from various data sources and 2+ years using Databricks in cloud environments
  • Programming Languages – SQL, Python
  • Technologies - IaaS (AWS or Azure or Google Cloud Platform), Databricks platform, Delta Lake storage, Spark (PySpark, Spark SQL).
  • Project Management using Agile, Scrum
  • B.S. Degree in a data-centric field (Mathematics, Economics, Computer Science, Engineering or other science field), Information Systems, Information Processing or engineering.
  • Excellent communication & leadership skills, with the ability to lead and motivate team members
  • Highly proficient in SQL
  • Proficient at writing complex SQL statements and interpreting results
  • Proficient with modern ETL design and development
  • Collaboratively drafts detailed documentation (i.e., Design Specs, Test Plans, etc.) that have a professional appearance
  • Excels in a fast paced, agile environment where critical thinking and strong problem-solving skills are required for success
  • Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.)
  • Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for infrastructure as code, and Splunk for monitoring
  • Continuously improve systems through performance enhancements and cost reductions in compute and storage
  • Data Processing and API Integration: Utilize Spark Structured Streaming for real-time data processing and integrate data outputs with REST APIs
  • Lead Data Engineering Projects to manage and implement data-driven communication systems
  • Experienced with Scrum and Agile Methodologies to coordinate global delivery teams, run scrum ceremonies, manage backlog items, and handle escalations
  • Integrate data across different systems and platforms
  • Strong verbal and written communication skills to manage client discussions

Essential Duties

(60%) Analysis/Development/Testing

  • Helps lead the development team during the design, build, test, support phases
  • Evaluates the introduction of new technologies and implements solutions and enterprise data integration strategies
  • Provide analysis and recommendations to overall data strategies and roadmaps
  • Partners with Solution Architects to perform source system analysis, identification of key data issues, data profiling and development of normalized and star-snowflake physical schemas
  • Drives the technical design, installation & configuration, development, optimization, error handling and support of data management initiatives for customer engagements
  • Creates and implements scheduling strategies for data management processes
  • Works with business partners to review prototypes and develop iterative revisions
  • Oversees and performs troubleshooting and provides resolutions for reporting issues
  • Works closely with and assists other consultants, provides leadership, and collaborates with clients to understand and interpret data through stakeholder interviews and by defining, analyzing, and validating data
  • Defines data pipeline orchestration and subsequently automates code deployments or mentors junior team members to automate the deployments using tools such as Apache Airflow, Control-M, or Autosys
  • Supports testing, root cause analysis, bug fixing and deployment activities

(30%) Communication/Organization/Documentation/Leadership

  • Leads collaboration and ensures success with offshore delivery team
  • Collaboratively drafts detailed documentation (i.e., Design Specs, Test Plans, etc.) that have a professional appearance
  • Demonstrates ability to organize tasks and time necessary to complete assigned tasks/deliverables
  • Coordinates with Project Managers and Leads to help drive project planning
  • Perform feasibility analysis, scoping exercises, and works with the project manager to prioritizes deliverables
  • Prioritizes and organize multiple tasks providing direction, tools, technical and process support to less senior developers
  • Provides appropriate training, mentoring and support for Data integration tools to the developers
  • Assists developers, analysts, and designers in conceptualizing and validating solutions that meet business requirements
  • Aligns data architecture plans with strategic business objectives
  • Utilizes agile-based approach to implementing data services and solutions
  • Provide analysis and recommendations to overall data strategies and roadmaps
  • Coordinates and oversee the assignments, delivery, and quality of data integration project
  • Develops appropriate tool development guidelines and documentation
  • Ensures the quality, completeness, security, privacy, and integrity of data throughout the data lifecycle
  • Assists with infrastructure and capacity planning
  • Assist with the development and enforcement of integration standards

(10%) Other duties as assigned or identified

Overview

EXL (NASDAQ: EXLS) is a leading operations management and analytics company that designs and enables agile, customer-centric operating models to help clients improve their revenue growth and profitability. Our delivery model provides market-leading business outcomes using EXL’s proprietary Business EXLerator Framework™, cutting-edge analytics, digital transformation and domain expertise. At EXL, we look deeper to help companies improve global operations, enhance data-driven insights, increase customer satisfaction, and manage risk and compliance. EXL serves the insurance, healthcare, banking and financial services, utilities, travel, transportation and logistics industries. Headquartered in New York, New York, EXL has more than 32,000 professionals in locations throughout the United States, Europe, Asia (primarily India and Philippines), South America, Australia and South Africa. For more information, visit

 

Physical Demands and Working Conditions

Works in a normal office environment requiring light physical effort by handling objects up to 20 pounds occasionally and/or up to 10 pounds frequently. Works with standard office equipment (such as phone, fax and personal computer). Ability to navigate stairs frequently during the course of a business day in various work locations. Sitting at desk for most of day, and working with computer.

EEO/Minorities/Females/Vets/Disabilities

Please be aware that EXL requires all employees to be vaccinated for COVID-19. This position will require the successful candidate to obtain and show proof of a vaccination. EXL is an equal opportunity employer, and will provide reasonable accommodation to those individuals who are unable to be vaccinated consistent with federal, state, and local law.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.