Databricks/AWS Engineer

Remote • Posted 60+ days ago • Updated 17 days ago
Full Time
No Travel Required
Remote
Depends on Experience
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • databricks
  • aws
  • python
  • ETL

Summary

NewGen is seeking a Sr. Data Engineer with experience building enterprise data solutions in both Databricks (Unity Catalog, Hive Metastore) and AWS (AWS Glue, S3, RDS) and managing a data warehouse.

Required:

  • 8+ years of experience in designing, developing, deploying, and managing ETL processes and data pipelines utilizing comprehensive logging and robust error handling mechanisms to troubleshoot and resolve issues.
  • 3+ years of hands-on experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools using Databricks tools that include Delta Lake, Databricks Delta, and Unity Catalog.
  • 3+ years of hands-on experience writing Python and PySpark code for ETL scripts and custom transformations.
  • Experience in data modeling (e.g. understanding when to use dimensional vs relation data models) within a medallion architecture (Bronze, Silver, Gold).
  • Experience using various AWS services, including AWS Glue, Amazon S3, Amazon RDS.
  • Ability to design and implement data models, from initially defining schemas to optimizing data structures that support efficient queries and analytics.
  • Knowledge of performance optimization techniques for ETL jobs, such as partitioning, bucketing, caching, and troubleshooting to improve query performance.
  • Ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements
  • Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Math, Physics, or Information Systems / Technology.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: NEWGEN
  • Position Id: 9050418
  • Posted 30+ days ago
Contact the job poster
BQ

Brian Quinn

Recruiter @ Newgen Technologies, Inc.

Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.