Big Data Platform Engineer
Full Time
Part Time
On-site
USD $60-60/hr
HyrUS Inc.
Fitment
Dice Job Match Score™
🔢 Crunching numbers...
Job Details
Skills
- PYTHON
- CLOUDERA HADOOP
- APACHE SPARK
- KERBEROS
- PERL
- HDFS
- AUTOMATION
- SPARK
- BIG DATA
- SPLUNK
Summary
Big Data Platform Engineer - Jersey City, NJ
Jersey City, NJ
Seeking an experienced Spark, HDFS, and Kafka Administrator/Engineer to manage, support, and optimize enterprise-grade big data platforms. Deep hands-on expertise with Apache Spark, HDFS, and Kafka is required and will be the primary focus of this role. Experience administering these technologies within a Cloudera environment is preferred but secondary to strong expertise in the underlying technologies.
Main areas of responsibilities:
Spark Administration: Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads. Support Spark job performance, resource management, capacity planning, and production operations.
HDFS Administration: Manage and support HDFS storage, including capacity, performance, availability, permissions, replication, and troubleshooting. Strong understanding of HDFS architecture and integration with distributed compute platforms is required.
Kafka Administration: Deploy, configure, manage, monitor, and troubleshoot Apache Kafka clusters, topics, producers, and consumers. Support performance, capacity, availability, security, and production operations.
Monitoring and Performance Tuning: Monitor platform connectivity, health, capacity, and performance using tools such as Grafana, Splunk, and relevant platform management tools. Proactively identify and resolve performance bottlenecks across Spark, HDFS, and Kafka.
Security and Compliance: Implement and manage security configurations, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integrations.
Automation and Scripting: Develop and maintain automation using Python, Bash, or Perl for routine administration, configuration management, monitoring, and operational support.
Troubleshooting and Support: Provide expert-level troubleshooting and technical support for Spark, HDFS, Kafka, and related big data technologies, including participation in an on-call rotation.
Platform Administration: Support installations, patching, upgrades, migrations, and platform lifecycle activities while minimizing downtime. Cloudera Hadoop/CDP administration will be part of the role but is secondary to expertise in Spark, HDFS, and Kafka.
Requirements:
Excellent problem-solving, analytical, and communication skills.
Minimum 3 years of hands-on production experience administering and supporting Apache Spark, HDFS, and Kafka.
Strong, demonstrable expertise in Spark administration, configuration, performance tuning, and troubleshooting.
Strong, demonstrable expertise in HDFS administration, storage management, security, performance, and troubleshooting.
Strong, demonstrable expertise in Kafka administration, topic management, producer/consumer architecture, performance, security, and troubleshooting.
Experience with Linux operating systems and command-line tools.
Experience with enterprise monitoring and operational tooling such as Grafana and Splunk.
Experience with Cloudera Hadoop/CDP and Cloudera Manager is preferred but is not the primary skill requirement.
Cloudera certifications or equivalent big data certifications are desirable but not required.
Solid understanding of Kerberos, SSL/TLS, IAM/LDAP, and security concepts for distributed data platforms.
Jersey City, NJ
Seeking an experienced Spark, HDFS, and Kafka Administrator/Engineer to manage, support, and optimize enterprise-grade big data platforms. Deep hands-on expertise with Apache Spark, HDFS, and Kafka is required and will be the primary focus of this role. Experience administering these technologies within a Cloudera environment is preferred but secondary to strong expertise in the underlying technologies.
Main areas of responsibilities:
Spark Administration: Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads. Support Spark job performance, resource management, capacity planning, and production operations.
HDFS Administration: Manage and support HDFS storage, including capacity, performance, availability, permissions, replication, and troubleshooting. Strong understanding of HDFS architecture and integration with distributed compute platforms is required.
Kafka Administration: Deploy, configure, manage, monitor, and troubleshoot Apache Kafka clusters, topics, producers, and consumers. Support performance, capacity, availability, security, and production operations.
Monitoring and Performance Tuning: Monitor platform connectivity, health, capacity, and performance using tools such as Grafana, Splunk, and relevant platform management tools. Proactively identify and resolve performance bottlenecks across Spark, HDFS, and Kafka.
Security and Compliance: Implement and manage security configurations, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integrations.
Automation and Scripting: Develop and maintain automation using Python, Bash, or Perl for routine administration, configuration management, monitoring, and operational support.
Troubleshooting and Support: Provide expert-level troubleshooting and technical support for Spark, HDFS, Kafka, and related big data technologies, including participation in an on-call rotation.
Platform Administration: Support installations, patching, upgrades, migrations, and platform lifecycle activities while minimizing downtime. Cloudera Hadoop/CDP administration will be part of the role but is secondary to expertise in Spark, HDFS, and Kafka.
Requirements:
Excellent problem-solving, analytical, and communication skills.
Minimum 3 years of hands-on production experience administering and supporting Apache Spark, HDFS, and Kafka.
Strong, demonstrable expertise in Spark administration, configuration, performance tuning, and troubleshooting.
Strong, demonstrable expertise in HDFS administration, storage management, security, performance, and troubleshooting.
Strong, demonstrable expertise in Kafka administration, topic management, producer/consumer architecture, performance, security, and troubleshooting.
Experience with Linux operating systems and command-line tools.
Experience with enterprise monitoring and operational tooling such as Grafana and Splunk.
Experience with Cloudera Hadoop/CDP and Cloudera Manager is preferred but is not the primary skill requirement.
Cloudera certifications or equivalent big data certifications are desirable but not required.
Solid understanding of Kerberos, SSL/TLS, IAM/LDAP, and security concepts for distributed data platforms.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 91175618
- Position Id: OOJ - 1192-196-1794318348
- Posted 1 hour ago
Company Info
About HyrUS Inc.
For over a decade, HyrUS has connected exceptional talent with America's leading enterprises. We're not just a staffing agency — we're a workforce partner obsessed with getting the match right.
We Build the Teams That Build America.
Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.