Apache Flink Streaming Lead

Job Title: Apache Flink Streaming Lead Location: Jersey City (Onsite role)

Lead Software Engineer with deep expertise in Apache Flink to lead the development and optimization of our real-time streaming applications. You will be responsible for designing, building, and maintaining scalable and robust data pipelines that process high-volume, high-velocity data. You will provide technical leadership, mentor junior engineers, and drive best practices in stream processing.
Responsibilities:
Design and Development:
o Architect, design, and implement high-performance, fault-tolerant, and scalable real-time streaming applications using Apache Flink.
o Develop and optimize Flink pipelines for data ingestion, transformation, and analysis.
o Write clean, efficient, and well-documented code.
o Implement and maintain CI/CD pipelines for streaming applications.
Technical Leadership:
o Lead technical design discussions and code reviews.
o Mentor and guide junior engineers, fostering a collaborative and learning environment.
o Define and enforce coding standards and best practices for stream processing.
o Evaluate and recommend new technologies and tools to improve streaming capabilities.
Performance and Optimization:
o Monitor and optimize the performance of streaming applications.
o Troubleshoot and resolve complex performance and stability issues.
o Conduct performance testing and benchmarking.
o Tune Flink configurations for optimal resource utilization.
Collaboration and Communication:
o Work closely with product managers, data scientists, and other stakeholders to understand requirements and deliver solutions.
o Communicate effectively with both technical and non-technical audiences.
o Participate in Agile/Scrum development processes.
Data Architecture:
o Contribute to the overall data architecture, ensuring seamless integration of streaming data with other data systems.
o Design and implement data models for streaming data.
o Work with message queues like Kafka, RabbitMQ, etc.

Qualifications:

Required:
o 5+ years of experience in software development, with a focus on data engineering or stream processing.
o 3+ years of hands-on experience with Apache Flink.
o Strong understanding of stream processing concepts and techniques.
o Proficiency in Java or Scala (or other relevant languages).
o Experience with distributed systems and cloud platforms (e.g., AWS, Google Cloud Platform, Azure).
o Experience with message queues (Kafka, RabbitMQ, etc.).
o Experience with building and maintaining CI/CD pipelines.
o Strong problem-solving and analytical skills.
o Excellent communication and collaboration skills.

Preferred: 1
o Experience with other stream processing frameworks (e.g., Apache Kafka Streams, Apache Spark Streaming).
o Experience with containerization and orchestration technologies (Docker, Kubernetes).
o Experience with database systems (SQL and NoSQL).
o Experience with data warehousing solutions.
o Contributions to open-source projects, particularly Apache Flink.