Overview
Skills
Job Details
Title of Position: Sr Data Engineer
Location: Charlotte, NC Hybrid (2-3 days a week, local candidates only)
Duration: 12 months
Top Skills Required/Notes:
Required Skills (top 3 non-negotiables):
- PySpark
- Google Dataproc
- Google BigQuery
Nice to have:
- Airflow
- Scala
- Hadoop/Hive
Description:
- Convert product requirements into clear and actionable technical designs.
- Design, build, test, and optimize data pipelines (batch or real-time), while handling data ingestion, processing, and transformation tasks.
- Implement built-in and customized data monitoring and governance mechanisms.
- Identify and remediate problems related to data ingestion, transformation, quality, or performance.
- Write Infrastructure-as-Code (IaC) for deployments and Maintain CI/CD pipelines for data workflows.
Required Testing: Strong grasp of data structures and algorithms (arrays, hash maps, etc.), with proven ability in parsing and transforming data. Skilled in developing ETL-like scripts using Python, Scala on Spark environment and proficient in writing optimized, high-performance SQL queries that leverage data aggregation, window functions, and subqueries.
Software Skills Required:
Language- Python, Scala
Database - BigQuery, Hive, Trino, PostgreSQL, MongoDB etc.
Data Platforms - Spark, Airflow, Kafka, Google Cloud Platform (BigQuery, Dataflow).
Modelling and Transformation - ELT/ETL framework