ETL Talend Developer

W2 profiles only
Location: Atlanta, Charlotte, Raleigh (Onsite)

Job Description:

Need strong Talend and Snowflake

• Designed and worked components of Ingestion Framework for all types and formats of ingestion systems.
• Involved in collecting requirements and working with DA to analyze the data for the processing from staging to target folders using Static and Dynamic partitions.
• Used CICD pipelines and GIT to push developed scripts to upper environments.
• Extracted meaningful data from dealer csv files, text files, and mainframe files using the Informatica BDM tool
• Used Python scripts, Spark Applications for ETL Operations to create data flow pipelines
• Involved in transforming data from legacy tables to Hive, HBase tables, and S3 buckets for handoff to business and Data scientists to create analytics over the data Implemented data integration solutions with traditional ETL/ELT tools and Big Data technologies
• Performing the data load validations and integration testing.
• Designed and implemented appropriate ETL mappings to extract and transform data from various sources to meet requirements using Informatica BDM(DEI) and Informatica cloud data integration services (IICS)
• Implemented SCD Type1, Type2, CDC and Incremental Load Strategies by using parametrized mapping templates
• Implemented Push down optimization to optimize the performance of Task flows
• Used Spark SQL new API''s and merging Hive and Spark Catalogs in new HDP 3.1 environment
• Tuned performance of the model by verifying the data model results and worked with different attributes and features for getting the better performance and model rebuilding activities.
• Built a framework to load historical data for Ingestion and Enriched data based on the given partition data
• Extensively worked on CDC login in the new Hadoop environment HDP 3.1
• Worked on extracting and loading data from different clusters and environments to Enterprise data lake.
• Using the TOAD implemented the DLV for the developed objects after the ingestion and unit testing
• Developed a Merge logic to load the updated deleted and new records to the Hive table
• Worked on creating and executing data ingestion with different types and formats of files
• Worked with Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD, Spark YARN
• Analyzed the SQL scripts and designed the solution to implement using PySpark
• Worked intermediately on monitoring and supporting production jobs.
• Collaborate with other experts to constantly innovate
• Involved in monthly production release activity and supported in the DOM, Implementation technical tasks.
• Created ESP job schedules for the jobs to run continuously based on the source and target systems.

Environment: Informatica DEI, Python, Linux, Apache Spark, Scala, Spark-SQL, HBase, Hue, Hive, Hadoop, TOAD, Mainframes, Shell script, Putty, Git bash, IICS, CAI, CDb2, Snowflake, RALLY and Agile Methodologies