Big Data Engineer

Job Title: Sr. Data Engineer

Location: Phoenix, AZ - Hybrid

Job Type: Contract/ FTE

Job Description:

• Must have minimum 7-10 years of hands-on experience on building robust and scalable pipelines using AWS native services (Cloud Formation, AWS Glue, Crawlers, Catalog, Step Functions, Event Bridge, Lambda, S3, SNS, SQS, Kafka, Kinesis & other data related services)
• Must be proficient in Python, Pyspark / Java.
• Must have 2-3 years of prior hands-on working experience in Cloudera Hadoop implementation.
• Must be proficient with SQL and HQL.
• Must have handled structured and unstructured data ingestion from different file formats (TXT, CSV, JSON, XML, Parquet etc ).
• Must have handled data streaming (from Kafka topics, using Apache Flink etc..)
• Must have prior experience with open table formats like Apache Iceberg, Hudi etc
• Must be proficient with Github versioning activities.
• Prior deployment experience using CI/CD would be an added advantage.
• Should be able to envision the end to end data pipeline while keeping up with all the data privacy, lineage & the cost optimization requirements.
• Should be able to reverse engineer the current processes running on legacy platforms with less or no prior documentation.
• Prior Apache Flink experience would be ideal.
• Exposure to Snowflake environment would be an added advantage.
• Must possess clear written & verbal communication skills along with interpersonal skill to collaborate with the team & business partners is very important.
• Should be willing to coordinate & own the delivery with the team located in India.
• Must have a positive attitude, should be open for discussions, stay hungry to learn & try out something new.