Lead Software Engineer (Big Data / Cloudera Platform)

Job Title: Lead Software Engineer (Big Data / Cloudera Platform)

Location: Reston, VA / Washington, DC (Hybrid - 1 Day per Month Onsite)
Job Type: Contract-to-Hire

Overview:

We are seeking a Lead Software Engineer with deep hands-on expertise in Big Data platforms, Cloudera (CDP/CDH), and AWS environments. This role focuses on production support, performance optimization, and on-call rotation for mission-critical big data systems. The ideal candidate will be an experienced technical leader who thrives in dynamic environments, ensuring the reliability and scalability of data platforms supporting business analytics and front-end tools like MicroStrategy.

Interview Process: Onsite Interview

Key Responsibilities:

• Lead the design, development, and support of large-scale big data solutions across Cloudera and AWS environments.
• Manage production support operations, including incident resolution, performance tuning, and system optimization.
• Monitor and maintain Cloudera clusters to ensure uptime, scalability, and compliance with organizational SLAs.
• Troubleshoot and support components within the Hadoop ecosystem (Hive, HBase, Solr, Impala, Spark, Flume).
• Administer and integrate Kafka and NiFi data flows for real-time data streaming and ingestion.
• Automate workflows and system tasks using Python and Shell scripting.
• Support and manage data migration projects (e.g., CDH to CDP) and enforce DevSecOps and CI/CD best practices.
• Collaborate with data engineers, cloud architects, and analytics teams to optimize pipeline performance and system security (Ranger, Kerberos).
• Mentor junior engineers and uphold high standards for code quality, deployment, and documentation.

Required Skills & Experience:

• 8+ years of experience in software development and Big Data platform support.
• Proven expertise with Cloudera (CDP/CDH) cluster administration, tuning, and upgrades.
• Strong hands-on experience with AWS cloud services (EC2, S3, EMR, IAM, Lambda, etc.).
• Proficiency in Kafka administration, Kafka Streams API, and integration with MQ systems.
• Expertise in Apache NiFi flow management and registry configuration.
• Strong knowledge of HBase, Solr, Hive, Impala, Oozie, YARN, Ranger, and related tools.
• Solid scripting and automation experience using Python and Shell.
• Experience supporting Cloudera and MongoDB in cloud environments.
• Familiarity with Cloud Disaster Recovery strategies and Big Data pipeline automation.
• Excellent analytical, troubleshooting, and performance-tuning abilities.

Preferred Qualifications:

• Experience with analytics/ML pipelines and data governance frameworks.
• Hands-on involvement in CDP migration projects and DevSecOps environments.
• Familiarity with Agile methodologies and collaborative development practices.

Additional Information:

• Role Type: Production Support / On-Call Rotation
• Environment: Cloudera running on AWS
• Front End: MicroStrategy

***Lumen and / or its clients will not provide equipment (Laptop, monitor, etc.) to the selected contractor. The contractor must have their own equipment. Access to a virtual desktop set up (software) will be provided by Lumen s client, allowing the user access to the required systems and technology. ***

Lumen Solutions Group Inc. is an Equal Opportunity Employer and does not discriminate in employment on the basis of Minority/Gender/Disability/Religion/LGBTQI/Age/Veterans.