Google Cloud Platform Data Engineer

Data Engineer

Onsite Bentonville, AR Hybrid work fashion

Contract 6 months contract, possibility of extension

Notes:

• Programming languages: Python (most important), Scala/ Spark
• Cloud: Google Cloud Platform- required
• Message Q: Kafka- required
• Databases: Big Query, SQL, Hive, Map Reduce
• At least 7 years of experience

Job Duties

• Designing and building ETL pipeline using Sqoop, Hive, Map Reduce and Spark on on-prem and cloud environments;
• Functional programming using Python and Scala for complex data transformations and in-memory computations;
• Using Erwin for Logical/Physical data modeling and Dimensional Data Modeling;
• Designing and developing UNIX/Linux scripts for handing complex file formats and structures
• Orchestration of workflows and jobs using Airflow and Automic;
• Creating multiple Kafka producers and consumers for data transferring;
• Performing Continuous Integration and deployment (CI/CD) using tools like Git, Jenkin to run test cases and build applications with code coverage using Scala test;
• Analyzing data using SQL, Big Query
• Monitoring the cluster performance, setting up alerts, documenting the designs, workflow.
• Providing production support, troubleshooting, and fixing the issues by tracking the status of running applications to perform System administrator tasks.