Sr Data Engineer
VIIS Global Inc.
Dice Job Match Score™
🔢 Crunching numbers...
Job Details
Skills
- Data Engineer
- AWS
- Databricks
- PySpark
- Python
- AWS cloud services
- Databricks Certified
- ETL/ELT pipelines
- ingesting
- transforming
- AWS S3
- data lake
- data lake solutions
- Databricks Workflows
- data storage
- incremental processing
- transformation
- Python frameworks
- reconciliation
- data validation
- quality checks
- error handling
- Root Cause Analysis
- (RCA)
- RCA
- CI/CD
- APIs
- files
- design
- development
- testing
- deployment
- production support.
- SQL
- Delta Lake
- Amazon S3
- AWS Glue
- Amazon EMR
- AMS Cloudwatch
- AWS IAM
- Databricks certification
- Unity Catalog
- Airflow
- Kafka
- Airflow orchestration
- data governance
- data quality frameworks
- data processing
- streaming
Summary
Role : Sr. Data Engineer
Location: Pasadena, CA 100% Onsite
Work Arrangement: Hybrid 3 Days/Week Onsite
Candidate Requirement: Local candidates only within 50 Miles.
Job Summary
We are seeking an experienced Data Engineer with strong hands-on expertise in AWS, Databricks, PySpark, and Python. The ideal candidate will have experience designing and developing scalable data pipelines and data processing solutions using AWS cloud services and Databricks.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and AWS.
- Develop robust ETL/ELT pipelines for ingesting and transforming large volumes of data.
- Build and optimize PySpark/Spark SQL jobs within Databricks.
- Work with AWS S3 for data storage and data lake solutions.
- Develop data processing workflows using Databricks Workflows and related AWS services.
- Work with Delta Lake for reliable data storage, incremental processing, and data transformation.
- Develop reusable Python frameworks and utilities for data engineering processes.
- Perform data validation, quality checks, error handling, and reconciliation.
- Troubleshoot production pipeline issues and perform Root Cause Analysis (RCA).
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Integrate data from databases, APIs, files, and other enterprise data sources.
- Implement CI/CD and source-control practices for data engineering applications.
- Collaborate with Data Architects, Data Scientists, Analysts, and business stakeholders.
- Participate in design, development, testing, deployment, and production support.
Required Skills
- 10+ years of overall Data Engineering experience preferred.
- Strong hands-on experience with AWS.
- Strong experience with Databricks.
- Strong hands-on experience with PySpark / Apache Spark.
- Strong Python programming experience.
- Strong SQL skills.
- Experience developing enterprise-scale ETL/ELT pipelines.
- Experience with AWS S3 and AWS data services.
- Experience with Delta Lake.
- Experience working with large-scale datasets.
- Strong understanding of data lake/lakehouse architecture.
- Experience with Spark performance tuning and optimization.
- Experience with Git and CI/CD.
AWS Skills
Candidates should have hands-on experience with AWS services such as:
- Amazon S3
- AWS Glue
- AWS Lambda
- Amazon Redshift
- Amazon EMR
- AWS CloudWatch
- AWS IAM
Preferred Skills
- Databricks certification
- Experience with Unity Catalog
- Experience with Airflow
- Experience with Kafka
- Experience with AWS Glue/Airflow orchestration
- Experience with Terraform
- Experience with data governance and data quality frameworks
Experience with streaming and real-time data processing
- Dice Id: 91164157
- Position Id: 9095920
- Posted 45 minutes ago
Company Info
About VIIS Global Inc.
When you start looking into moving your processes and operations into the digital sphere in an effective and efficient way, you will need the help of a team that is familiar with digital transformations and who offers a whole host of SAP-related solutions and services. At VIIS Global we believe we are one of the best partners operating right now for companies looking to implement ERP upgrades, digital transformation, and other improvements to their business operations.
We established this organization to make it much easier and smoother for small to mid-size businesses to follow through on their system upgrades and digital transformation plans. We use a combination of in-depth industry expertise and amazing end-to-end solutions. This means when you work with our team you will get the Strategic Advisory and Transformation approach that will modernize your business.
Never miss an opportunity! Create an alert based on the job you applied for.Similar Jobs
It looks like there aren't any Similar Jobs for this job yet.
Search all similar jobs