Big Data Engineer (No C2C)

  • Atlanta, GA
  • Posted 19 days ago | Updated moments ago

Overview

Remote
On Site
Hybrid
BASED ON EXPERIENCE
Contract - W2
Contract - Independent

Skills

BIG DATA ENGINEER
DATA ENGINEER
DATABRICKS
APACHE KAFKA
SPARK STREAMING
SPARK
PYTHON
SCALA
ETL
KAFKA
AMAZON MSK
TIBCO EMS
IBM MQ
DATABRICKS UI
DATABRICKS NOTEBOOKS
DELTA LAKE
SPARK SQ
UNITY CATALOG
UNITY
DATA INGESTION
AWS
SNOWFLAKE
KUDU/IMPALA
BIGQUERY
DELTA LAKE
BIGQUERY

Job Details

Our client is currently seeking an experienced Big Data Engineer for their Midtown office in Atlanta, GA. The successful candidate must have substantial Big Data engineering experience and demonstrate an affinity for working with others to create successful solutions. Join a smart, highly skilled team with a passion for technology where you will work on our state-of-the-art Big Data Platforms. You must be an excellent communicator, both written and verbal, and have experience working with business areas to translate their business data needs and questions into project requirements. You will participate in all phases of the Data Engineering life cycle, independently and collaboratively writing project requirements, architecting solutions, and performing data ingestion development and support duties.

Skills and Experience

Required:

  • 6+ years of overall IT experience.
  • 3+ years of experience with high-velocity, high-volume stream processing: Apache Kafka and Spark Streaming.
    • Experience with real-time data processing and streaming techniques using Spark Structured Streaming and Kafka.
    • Deep knowledge of troubleshooting and tuning Spark applications.
  • 3+ years of experience with data ingestion from Message Queues (TIBCO, IBM, etc.) and different file formats across different platforms like JSON, XML, CSV.
  • 3+ years of experience with Big Data tools/technologies like Hadoop, Spark, Spark SQL, Kafka, Sqoop, Hive, S3, HDFS, etc.
  • 3+ years of experience building, testing, and optimizing Big Data data ingestion pipelines, architectures, and data sets.
  • 2+ years of experience with Python and/or Scala, i.e., PySpark/Scala-Spark.
  • 3+ years of experience with Cloud platforms (e.g., AWS, Google Cloud Platform).
  • 3+ years of experience with database solutions like Kudu/Impala, Delta Lake, Snowflake, or BigQuery.
  • 2+ years of experience with NoSQL databases including HBASE and/or Cassandra.
  • Experience in successfully building and deploying a new data platform on Azure/AWS.
  • Experience in Azure/AWS Serverless technologies like S3, Kinesis/MSK, Lambda, and Glue.
  • Strong knowledge of Messaging Platforms like Kafka, Amazon MSK, TIBCO EMS, or IBM MQ Series.
  • Experience with Databricks UI, managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog.
  • Knowledge of Unix/Linux platform and shell scripting.
  • Strong analytical and problem-solving skills.

Preferred (Not Required):

  • Strong SQL skills with the ability to write intermediate complexity queries.
  • Strong understanding of Relational & Dimensional modeling.
  • Experience with GIT code versioning software.
  • Experience with REST API and Web Services.
  • Good business analyst and requirements gathering/writing skills.

Education

Bachelor s Degree required. Preferably in Information Systems, Computer Science, Computer Information Systems, or a related field.

Benefits (employee contribution):
  • Health insurance
  • Health savings account
  • Dental insurance
  • Vision insurance
  • Flexible spending accounts
  • Life insurance
  • Retirement plan

All qualified applicants will receive consideration for employment without regard to age, race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.