Sr .Data Engineer (BigData/AWS)
Duration: 6+ Months C2H
- Design, implement and support an analytical data infrastructure providing access to large datasets and computing power.
- Creation and support of real-time data pipelines built on AWS technologies including EMR, Glue, Kinesis, Redshift/Spectrum and Athena
- Responsible for building and automating an end-to-end data pipeline from data collection to data lake (Snowflake).
- Writing and validating scripts for data munching / data wrangling using Python/SQL or equivalent.
- Assisting in Data Analytics and Visualizations using SQL, Python (Numpy, Pandas, Scipy, etc.) as needed.
- 10+ years of industry experience
- 5+ years of experience in data engineering/preparation
- Solid understanding of the core principles of Data engineering and Data warehousing
- End-to-End experience with Data Collection, Integration, Analysis of Website logs (hit level data) , Google Analytics Data set, Machine data in factories and structured dataset out of IT applications.
- In-depth understanding and hands-on implementation and use of AWS services related to data engineering world – S3, Kinesis, Glue, EMR, Spark, Redshift, Sagemaker
- Ability to work with MPP databases like Netezza or Redshift or Teradata