Sr. Data Engineer

Note: This role need a consultant who is local to Dallas, TX (nned to attend the In-person interview)

Job Description: 

• Provide product technical leadership for teams to both analyze and take charge of long-term opportunities, designing a cutting-edge data solution and help with core development when needed.
• Act as Product Technical Lead.
• Act as subject matter expert for data domain within the IT organization and work with business to author self-service data products
• Design and build data engineering pipeline frameworks while ensuring these are reusable, scalable, efficient, maintainable, gracefully recover from failures, reprocessing the data should be easy.
• Drive data quality, best practices, coding standards, Test Driven Development, identifying single source of truth for data across systems and Quality Analytics (Mean Time to Recover, Mean Time between Failures, Patterns causing failures).
• Utilize data pipelines to provide actionable insights into data quality and product performance.

Minimum Qualifications – Prior Job Experience:

• 9+ years of full Software Development Life Cycle (SDLC) experience designing, developing, and implementing large-scale applications in data analytics, warehousing, and data engineering.
• Working experience in data analytics (data wrangling, mining, integration, analysis, visualization, data modeling, analysis/analytics, and reporting).

Top required experience:

• Apache Spark, Scala, Azure Databricks

Nice to have skills:

SQL; CICD

Should have at least 5 years of experience in optimizing Spark jobs for performance and cost-efficiency using advanced techniques such as partitioning, caching, cluster configuration tuning, and troubleshooting bottlenecks.