Sr. Databricks Engineer
Transcend IT Solutions
Dice Job Match Score™
🔢 Crunching numbers...
Job Details
Skills
- Databricks
- SQL
- Python
- AWS Glue
- ETL
- Apache pyspark
Summary
Job Title: Sr. Databricks Engineer
Location: Malvern, PA
Job Details:
Technical Stack & Requirements:
- Core Competencies: Python, Apache Spark (PySpark), AWS Glue ETL, Data Lake concepts (Medallion Architecture), Databricks, Sagemaker.
- Infrastructure: CloudFormation/Terraform, CI/CD with GitHub Actions.
- Core Responsibilities: Orchestrate data extraction from diverse legacy and modern sources, engineer high-performance scalable pipelines, and provide expert-level production support to ensure system reliability.
Qualifications:
- Bachelor’s degree in Computer Science or related field.
- Strong hands-on experience with Apache Spark and Glue.
- Proven ability to work with Python-based data pipelines.
- Experience with Git and GitHub Actions is mandatory. Required Qualifications
- Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline.
- 5-7+ years of enterprise Data Engineering experience.
- Strong Python development experience.
- Advanced SQL programming skills.
- Extensive experience with Apache Spark and PySpark.
- Strong experience with Databricks application.
- Hands-on experience with AWS data services including: o S3 o AWS Glue o Athena o Lambda o Step Functions o EventBridge
- Experience building enterprise ETL and ELT pipelines.
- Strong understanding of distributed data processing.
- Experience working with large-scale cloud data platforms.
- Strong knowledge of dimensional data modeling.
- Experience implementing CI/CD pipelines.
- Familiarity with Agile/Scrum methodologies.
- Excellent communication and stakeholder management skills.
Preferred Qualifications
- Financial Services or Asset Management industry experience.
- Knowledge of Apache Iceberg, Delta Lake, or Hudi.
- Experience with Airflow.
- Knowledge of Kafka or Kinesis streaming.
- Experience supporting AI/ML and advanced analytics platforms.
- AWS Professional or Specialty Certifications.
Job Responsibilities
We are seeking high-caliber Data Engineers to drive data analysis projects, Migrate jobs from EMR to Databricks,create strong job aids and train analysts. In this pivotal role, you will implement and optimize sophisticated data pipelines, partnering with business analysts to guarantee the seamless availability and absolute integrity of data powering our core financial engines and reporting ecosystems.Hari Krishna Vanga
Senior Technical Recruiter
E:
L:
A: 25663 Smotherman Rd, Suite 302, Frisco, TX, 75033.
- Dice Id: 91140092
- Position Id: 19-49161-1793219070
- Posted 3 hours ago
Company Info
About Transcend IT Solutions
Our business model has scaled to successful heights on account of the long-lasting and loyal relationship with our clients, partners and candidates who have been pillar foundation of why Transcend IT Solutions is a market leader.
Our team has an outstanding track record of success in project staffing, project development and management, and consulting services. With firsthand experience in these areas, we have a sound understanding of our client’s needs.
Transcend IT Solutions provides assistance with staffing for all Information Technology roles. Providing the best-fit person of the highest caliber is our recruiting team’s goal. Our success is based on longevity of client-candidate relationships. Whether it is an individual team member or an entire team, Transcend IT Solutions will assist you with your staffing needs.
Our consulting services cover all aspects of Information Technology. Our expert consultants provide expertise in the areas of project management, requirement gathering and management, analysis and design, construction, quality control and assurance, and system deployment. As well as being individual contributors, our consultants also provide mentoring, process improvement expertise, and reviews and audits of existing projects.
