Big Data Platform Engineer

Title: Big Data Platform Engineer
Location : Jersey City, NJ Day one Onsite

s and s 

Description:

Seeking an experienced Spark, HDFS, and Kafka Administrator/Engineer to manage, support, and optimize enterprise-grade big data platforms. Deep hands-on expertise with Apache Spark, HDFS, and Kafka is required and will be the primary focus of this role. Experience administering these technologies within a Cloudera environment is preferred but secondary to strong expertise in the underlying technologies

Main areas of responsibilities:

·      Spark Administration: Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads. Support Spark job performance, resource management, capacity planning, and production operations.

·      HDFS Administration: Manage and support HDFS storage, including capacity, performance, availability, permissions, replication, and troubleshooting. Strong understanding of HDFS architecture and integration with distributed compute platforms is required.

·      Kafka Administration: Deploy, configure, manage, monitor, and troubleshoot Apache Kafka clusters, topics, producers, and consumers. Support performance, capacity, availability, security, and production operations.

·      Monitoring and Performance Tuning: Monitor platform connectivity, health, capacity, and performance using tools such as Grafana, Splunk, and relevant platform management tools. Proactively identify and resolve performance bottlenecks across Spark, HDFS, and Kafka.

·      Security and Compliance: Implement and manage security configurations, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integrations.

·      Automation and Scripting: Develop and maintain automation using Python, Bash, or Perl for routine administration, configuration management, monitoring, and operational support.

·      Troubleshooting and Support: Provide expert-level troubleshooting and technical support for Spark, HDFS, Kafka, and related big data technologies, including participation in an on-call rotation.

·      Platform Administration: Support installations, patching, upgrades, migrations, and platform lifecycle activities while minimizing downtime. Cloudera Hadoop/CDP administration will be part of the role but is secondary to expertise in Spark, HDFS, and Kafka.

Requirements:

·      Excellent problem-solving, analytical, and communication skills.

·      Minimum 3 years of hands-on production experience administering and supporting Apache Spark, HDFS, and Kafka.

·      Strong, demonstrable expertise in Spark administration, configuration, performance tuning, and troubleshooting.

·      Strong, demonstrable expertise in HDFS administration, storage management, security, performance, and troubleshooting.

·      Strong, demonstrable expertise in Kafka administration, topic management, producer/consumer architecture, performance, security, and troubleshooting.

·      Experience with Linux operating systems and command-line tools.

·      Experience with enterprise monitoring and operational tooling such as Grafana and Splunk.

·      Experience with Cloudera Hadoop/CDP and Cloudera Manager is preferred but is not the primary skill requirement.

·      Cloudera certifications or equivalent big data certifications are desirable but not required.

·      Solid understanding of Kerberos, SSL/TLS, IAM/LDAP, and security concepts for distributed data platforms.