Sr Big Data / OpenShift API Developer

Charlotte, NC, US • Posted 1 hour ago • Updated 1 hour ago
Full Time
On-site
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • Capacity Management
  • Environment Management
  • Extract
  • Transform
  • Load
  • Real-time
  • Use Cases
  • Data Quality
  • RESTful
  • Analytics
  • Authentication
  • Service Level
  • API
  • Performance Tuning
  • Scalability
  • Caching
  • File Formats
  • Resource Management
  • Scheduling
  • Apache Kafka
  • Data Warehouse
  • Publishing
  • Streaming
  • Recovery
  • Messaging
  • Continuous Integration
  • Continuous Delivery
  • GitHub
  • Git
  • Quality Assurance
  • Testing
  • Promotions
  • Development Testing
  • Release Management
  • Grafana
  • Dashboard
  • Computer Networking
  • Root Cause Analysis
  • Change Management
  • Production Support
  • Security Controls
  • Access Control
  • RBAC
  • Encryption
  • Vulnerability Management
  • Regulatory Compliance
  • Collaboration
  • DevOps
  • Software Design
  • Estimating
  • Sprint
  • Demonstrations
  • Technical Writing
  • Knowledge Transfer
  • Mentorship
  • Computer Science
  • Information Technology
  • Software Development
  • Data Engineering
  • Big Data
  • Data Processing
  • Management
  • Apache Spark
  • PySpark
  • Apache Airflow
  • Workflow
  • Orchestration
  • Microservices
  • Docker
  • Kubernetes
  • Cloud Computing
  • Amazon Web Services
  • Data Storage
  • Amazon S3
  • Google Cloud
  • Google Cloud Platform
  • Microsoft Azure
  • Storage
  • HDFS
  • Relational Databases
  • NoSQL
  • Database
  • Python
  • Scala
  • Java
  • Shell Scripting

Summary

Role: Sr Big Data / OpenShift API Developer

Location: Charlotte, NC (Hybrid - 3 days office in a week)

Key Responsibilities

Platform Management
  • Deploy, configure, maintain, and support containerized Apache Spark applications on OpenShift and Google Cloud.
  • Manage Spark workloads running on Kubernetes and OpenShift clusters.
  • Configure pods, deployments, services, routes, secrets, config maps, resource quotas, and namespaces.
  • Define resource requests and limits for Spark drivers, executors, and supporting services.
  • Support cluster capacity planning, scaling, patching, upgrades, and environment management.
  • Maintain Helm charts and deployment templates for repeatable application releases.

Data Pipeline Development
  • Design and implement large-scale batch and streaming data processing workflows using Apache Spark and Apache Airflow.
  • Develop robust, reusable, and production-ready PySpark, Scala, or Java applications.
  • Build Airflow DAGs for scheduling, orchestration, dependency management, retries, and alerting.
  • Implement data ingestion, transformation, validation, enrichment, and publishing processes.
  • Support both real-time and batch data processing use cases.
  • Apply data engineering best practices for reliability, scalability, data quality, and maintainability.

API and Microservices Development
  • Design, develop, test, and maintain RESTful APIs and microservices supporting data and analytics platforms.
  • Define API contracts, request/response schemas, authentication mechanisms, error handling, and service-level requirements.
  • Integrate APIs with Spark applications, data platforms, databases, messaging systems, and cloud services.
  • Develop automated unit, integration, and API tests.
  • Apply microservices architecture principles, including loose coupling, scalability, resiliency, and observability.

Performance Optimization
  • Tune Spark jobs for performance, scalability, and efficient resource utilization.
  • Optimize partitioning, joins, caching, serialization, shuffle operations, file formats, and executor configuration.
  • Use OpenShift and Kubernetes resource-management capabilities to improve application performance.
  • Configure horizontal and vertical scaling, autoscaling, and workload scheduling where appropriate.
  • Analyze Spark execution plans, application logs, cluster metrics, and resource-consumption patterns.
  • Identify and resolve bottlenecks across applications, infrastructure, networking, and storage layers.

Integration
  • Integrate Apache Spark with data sources and platforms such as:
    • Apache Kafka
    • Amazon S3
    • Google Cloud Storage
    • Azure Blob Storage
    • HDFS
    • Relational and NoSQL databases
    • Data lakes and data warehouses
  • Develop reliable ingestion and publishing patterns for batch and streaming data.
  • Implement schema management, checkpointing, offset handling, retry logic, and failure recovery.
  • Support secure connectivity between applications, cloud platforms, messaging systems, and databases.

CI/CD Implementation
  • Build and maintain CI/CD pipelines for deploying Spark applications, APIs, and supporting services to OpenShift.
  • Use tools such as GitHub Actions, SonarQube, Harness, Docker, Helm, and Git.
  • Automate build, code-quality analysis, security scanning, testing, packaging, and deployment processes.
  • Implement environment-specific configurations and promotion strategies across development, QA, staging, and production.
  • Manage container images, registries, versioning, release tagging, and rollback procedures.
  • Establish deployment standards and support automated release management.

Monitoring and Troubleshooting
  • Monitor OpenShift cluster health, Spark job performance, application availability, and resource utilization.
  • Use OpenShift monitoring capabilities and tools such as Prometheus, Grafana, and centralized logging platforms.
  • Develop dashboards, alerts, health checks, and operational metrics.
  • Troubleshoot complex issues involving Spark applications, APIs, containers, Kubernetes resources, networking, storage, and integrations.
  • Perform root-cause analysis and implement proactive remediation.
  • Participate in incident, problem, and change-management activities.
  • Provide production support and participate in on-call rotations when required.

Security and Compliance
  • Implement security controls for applications, data pipelines, containers, and OpenShift environments.
  • Configure and manage Role-Based Access Control (RBAC), service accounts, secrets, and namespace permissions.
  • Ensure encryption of data in transit and at rest.
  • Apply secure coding, container security, vulnerability management, and least-privilege principles.
  • Support compliance with enterprise security standards, regulatory requirements, and internal governance policies.
  • Protect sensitive data during ingestion, processing, storage, and transmission.

Collaboration
  • Work with product owners, data engineers, application developers, architects, DevOps teams, security teams, and operations teams to define requirements.
  • Participate in architecture reviews, solution design, estimation, sprint planning, and technical demonstrations.
  • Create technical documentation, deployment guides, operational runbooks, and support procedures.
  • Support production releases and provide knowledge transfer to engineering and operations teams.
  • Mentor junior developers and promote engineering best practices.

Required Qualifications
  • Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
  • 7+ years of experience in software development, data engineering, big data, or distributed systems.
  • 4+ years of hands-on experience with Apache Spark and large-scale data processing.
  • 3+ years of experience deploying and managing containerized applications on OpenShift and/or Kubernetes.
  • Strong experience developing Spark applications using Python/PySpark, Scala, or Java.
  • Hands-on experience with Apache Airflow and workflow orchestration.
  • Experience designing and developing REST APIs and microservices.
  • Strong knowledge of Docker and Kubernetes concepts, including:
    • Pods
    • Deployments
    • Services
    • ConfigMaps
    • Secrets
    • Namespaces
    • Ingress or Routes
    • Resource requests and limits
    • Container images
  • Experience with cloud platforms such as AWS, Google Cloud Platform, or Microsoft Azure.
  • Experience working with data storage and processing technologies such as:
    • Amazon S3
    • Google Cloud Storage
    • Azure Blob Storage
    • HDFS
    • Data lakes
    • Relational databases
    • NoSQL databases
  • Strong programming skills in Python, Scala, or Java.
  • Experience with shell scripting is preferred.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: RTX1d6413
  • Position Id: f92857cdcb2c7317c788937b1ff938f1
  • Posted 1 hour ago
Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.