Big Data Engineer

Job Title: Big Data Engineer

Location: Irving, TX / Richardson, TX / Houston, Texas

Job Type: Permanent / Full-time

Job Description:

Required Qualifications:

• At least 5 years of Information Technology experience.
• Experience in PySpark, Scala & Hadoop ecosystem.
• Solid understanding and hands-on experience in SQL, Unix shell scripting

Preferred Qualifications:

• Proficiency in programming languages such as Scala, Java, or Python
• At least 5 years of experience in Apache Spark, including Spark Core, Spark SQL, Spark Streaming, and Spark MLlib
• Lead the development and optimization of batch and real-time data pipelines, ensuring scalability, reliability, and performance.
• Architect, design, and deploy data integration, streaming, and analytics solutions leveraging Spark, Kafka, and Scala.
• Extensive hands-on experience with Apache Kafka, including topics, partitions, producers, consumers, and stream processing
• Solid understanding of distributed computing concepts, data partitioning, fault tolerance, and parallel processing
• Experience with cluster management tools like Apache Mesos or Apache YARN is desirable
• Good understanding of data integration, data quality and data architecture
• Experience in Relational Modeling, Dimensional Modeling and Modeling of Unstructured Data
• Good understanding of Agile software development frameworks
• Strong communication and Analytical skills
• Ability to work in teams in a diverse, multi-stakeholder environment comprising of Business and Technology teams
• Experience and desire to work in a global delivery environment