Big Data Engineer

Overview

Hybrid
Depends on Experience
Contract - W2
Contract - Independent

Skills

Big Data
databricks
Apache Kafka
Spark Streaming
PySpark
Scala-Spark
Python
Data Modeling
Amazon MSK
IBM MQ Series
Databricks UI
Databricks Notebooks
Delta Lake
Delta Live Tables
Unity Catalog
JSON
XML
CSV
Unix
Linux
AWS
GCP
kudu
Impala
BigQuery
Spark
TIBCO

Job Details

Additional Job Information:
Title : Big Data Engineer
Location : Atlanta, GA, United States (Hybrid- Periodically Onsite as Needed)
Description:
Big Data Engineers are responsible for architecture, design and build of Big Data applications to support business strategies and deliver business value. A data engineer participates in all phases of the Data Engineering life cycle and will independently and collaboratively write project requirements, architect solutions, and perform data ingestion development and support duties. Building data ingestion pipelines is highly technical and requires advanced skills with computers and proficiency with Big Data tools and technologies such as SQL, Python/Scala, Spark, Spark Structured Streaming, Spark SQL, Kafka, Sqoop, Hive, Kudu, HBASE, Impala, S3, HDFS, or Cloud platforms such as AWS, Google Cloud Platform, Azure, etc. Data Engineers require ability to communicate effectively, both written and verbal, and have extensive experience working with business areas to translate their business data needs and data questions into project requirements.
Qualification:
Bachelor s Degree required. Preferably in Information Systems, Computer Science, Electrical Engineering, Computer Information Systems or related field
For Big Data Engineer position:
1. Must have hands-on experience with Databricks
2. Must have hands-on experience with high-velocity high-volume stream processing: Apache Kafka and Spark Streaming a. Experience with real-time data processing and streaming techniques using Spark structured streaming and Kafka b. Deep knowledge of troubleshooting and tuning Spark applications
3. Must have hands-on experience with Python and/or Scala i.e. PySpark/Scala-Spark
4.Experience with Traditional ETL tools and Data Modeling
5. Strong knowledge of Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series
6. Experience with Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog
7. Experience with data ingestion of different file formats across like JSON, XML, CSV
8. Knowledge of Unix/Linux platform and shell scripting
9. Experience with Cloud platforms e.g. AWS, Google Cloud Platform, etc. Experience with database solutions like Kudu/Impala, or Delta Lake or Snowflake or BigQuery


Best Regards,

Hakeema Jaferin B.Tech

Technical Lead

Services You Can Trust
M: | LinkedIn:

E: | :

7000 Peachtree Dunwoody Rd | Bldg. 11 , Suite 301

Atlanta , GA 30328

ReplyReply allForward
Attendee panel closed
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.