The Colony, TX Description:
Our client is currently seeking a Big Data Engineer for a long term contract, possible contract to hire located in plano, TX. Please reach out o: for more information
The main function of the Data Engineer is to develop, evaluate, test and maintain architectures and data solutions within our organization. The typical Data Engineer executes plans, policies, and practices that control, protect, deliver, and enhance the value of the organization's data assets.
Design, construct, install, test and maintain highly scalable data management systems.
Ensure systems meet business requirements and industry practices.
Design, implement, automate and maintain large scale enterprise data ETL processes.
Build high-performance algorithms, prototypes, predictive models and proof of concepts.
Analytical and problem solving skills, applied to Big Data domain
- Proven understanding and hands on experience with Hadoop, Hive, Pig, Impala, and Spark
- 5-8 years of SQL, Hive, Hadoop and Python, Shell (4-5 years)
- Java/J2EE development knowledge
- 3+ years of demonstrated technical proficiency with Hadoop and big data projects
Day to Day duties: - Take requirements from Business Analyst and Stakeholders
Analyze and see where changes are needed; perform impact analysis (if pipeline available);
make sure proper connectivity to source & right credentials to connect to source; create pipelines (if not available) -
Data collection - gather information and required data fields.
Pull data from Oracle database (from different databases) put into Hive tables
- Data manipulation
- Join data from multiple data sources and build ETLs to be sent to Tableau (front-end dashboard) for reporting purpose
- Measure & Improve
- Implement success indicators to continuously measure and improve, while providing relevant insight and reporting to leadership and teams.
- Must be able to optimize performance tuning/monitoring and development at the same time. Mostly we are looking for an expert in SQL/Hive and moderate skills in python/shell and Hadoop. (A lot of times candidates screened have been really good in SQL but not in python/shell/Hadoop or vice versa.)Contact:
This job and many more are available through The Judge Group. Find us on the web at www.judge.com