The position is for a Senior Big Data ETL Developer This position is an opportunity to use Informatica Big Data Management(BDM) with HDFS(Cloudera), Spark and Kafka technology for batch and streaming analytics. Seeking a Big Dataengineer who possesses a base of programming skills and have a passion to learn new tools and techniques in a big data landscape that is endlessly changing. Data processes in BDM, if appropriate, include ingestion, standardization, metadata management, business rule curation, data enhancement, and statistical computation against data sources that include relational, XML,JSON, streaming, REST API, and unstructured data. BDM has provided a metadata injection layer for data ingestion across six SQLServer sources that will be leveraged for additional source ingestion. Job Duties The role has responsibility to understand, prepare, process and analyze data to drive operational, analytical and strategic business decisions. The Senior Big Data Engineer will work closely with big data engineers, product owners, information engineers, data scientists, data modelers, infrastructure support and data governance positions. This role requires exceptional BDM developer skills who can mentor big data engineers in the appropriate solution architecture and BDM development practices. While you do not have to have big data skills, we are seeking BDM engineers who start with a base of programming skills but who also love to learn new tools and techniques in a big data landscape that is endlessly changing. Qualifications Build end to end data flows from sources to fully curated and enhanced data sets. This can include the effort to locate and analyze source data, create data flows to extract, profile, and store ingested data, define and build data cleansing and imputation, map to a common data model, transform to satisfy business rules and statistical computations, and validate data content. Produce data building blocks, data models, and data flows for varying client demands such as dimensional data, data feeds, dashboard reporting, and data science research & exploration Create, modify and maintain BDM code and complex SQL for BI/DW data flows Learn to program in Spark and Python where BDM Spark co-generation is not adequate Produce automated tests of data flow components Use knowledge of the business to automate business-specific tests for data content quality Automate code deployment and promotion Build automated orchestration with BDM and error handling for use by production operation teams Provide technical expertise to diagnose errors from production support teams Collaborate with team members in an Agile team (e.g., Scrum) Participate as both leader and learner in team tasks for architecture, design and analysis Coordinate within collocated on-site teams as well as with work plans for off-shore resources Bachelor's Degree or Two Year Technical Program with a Programming Specialization B.S. preferred in Computer Science, Information Systems, or related field Minimum two years of development experience with BDM with a preference for five years of ETL tool experience which can be from other tools such as Talend, DataStage, and Informatica Experience with development of metadata-driven and fully parameterized BDM environments Advanced SQL coding skills for data transformations, profiling, and query tasks Unix commands and scripting Experience in agile environments such as scrum and Kanban Preference for experience in Hadoop fundamentals and architecture: HDFS, map-reduce, job performance Preference for open source big data skills in tools such as Hive, HBase, parquet, Spark SQL Programming in a language such as Python (preferred), Scala, etc.
Provides designs, specifies information systems solutions and the highest level technical direction in the development of new or existing applications to solve basic to complex problems or enhancements. Acts as a principal application designer for critical components and major modifications effectively using analytical skills, technical skills, available technology and tools in the evaluation of client requirements and processes. Provides solutions that are technologically sound. Often directs and monitors the activities of junior staff.
Performs and oversees the most complex systems analysis, design and development efforts. Plays a key role as a leader and individual contributor on complex projects. Provides the highest level analytical consulting and leadership in identifying and implementing new uses of information technologies to assist the client areas and Bank in meeting their short term business goals and long term strategic objectives.
Must be able to direct and monitor junior resources, plus coordinate development tasks for any scope effort. Prepares and manages the technical component of project plans.
Confers with other development staff, operations staff and IT staff in overall systems development direction from technical analysis to user acceptance testing. Prepares and reviews test data and executes detailed test plans. Performs any required debugging.
Evaluates and understands highly complex interrelationships and effects among programs, interfacing applications and platforms.
Prepares thorough, clear technical and functional specifications and updates systems documentation. Reviews documentation prepared by junior development staff.
Prepares and reviews technical assessments to include required tasks, estimated time frames and effort for any scope project.
Maintains efficient operation and effectiveness of supported applications. Recommends new technology, policies or processes to benefit the organization and improve deficiencies. Typically will lead technical evaluations of vendor software.
Performs other related duties or projects as required, which may include playing a lead role in due diligence, cost/benefit analysis or business study activities.
Follows and promotes the use of development standards and procedures.
Maintains a high level of technical aptitude and expert working knowledge of the supporting application(s), development environments and the supported client environment.
Assists in developing expense and capital budgets.
Monitors employee performance and may prepare annual appraisals.
NATURE AND SCOPE
The incumbent works under supervision of the Technical Team Leader or Systems Manager.
The Senior Software Engineer is competent to work independently on all high level systems analysis and technical phases of development.
The incumbent is capable of managing the activities of others on any scope project and often leads project activities.
The Senior Software Engineer monitors staff performance on assigned projects with regards to their overall abilities and effectiveness in completing projects within schedules.
The Senior Software Engineer participates in the development of junior staff.
The incumbent interacts with Senior Management, other technology personnel, clients and vendors.
The Senior Software Engineer provides backup to the Technical Team Leader as required.
The incumbent may be called upon to serve as a technical representative on committees, ad-hoc projects, etc.
Act as back to Technical Team Lead and participate in development of junior staff.
Minimum of two to four years college with at least ten years involvement in systems analysis and application development, or an equivalent combination of education and experience.
Bachelor's degree in Computer Science or related discipline preferred.
Requires a complete understanding of the system development life cycle, application development alternatives and the client area's functions.
Requires experience with state of the art application development support software and hardware platforms.
Must be familiar with industry direction.
Proficient in at multiple development tools and thorough knowledge of application development support software and hardware platforms along with a diverse technical background in computing environments.
Must possess a very good working knowledge of word processing, spreadsheet and databases.
Demonstrates expert analysis and resolution for the most complex problems.
Detailed knowledge of the Bank's application framework.
Excellent understanding of vendor hardware configurations and software concepts both in the current environment and future technologies.
Ability to lead or actively participate in technical walkthroughs.
Capable of quickly learning new technical skills and grasp new concepts.
Excellent oral and written communication skills with the ability to effectively present to the target audience.
Requires excellent analytical, organization, and time management skills, strong judgment, and basic management skills, and the ability to work effectively with clients and IT management and staff personnel. Detail oriented, uses sound judgment, is proactive and self motivated
Asserts ideas, encourages teamwork and acts as a role model when leading and directing others.
Ability to handle multiple tasks on a variety of assignments.
Aware of the Bank's business plan and strategic objectives.
Possess a thorough understanding of the business unit's role and how they fit into the organization.
Good project organizational skills.
At M&T, we strive to be the best place our employees ever work, the best bank our customers ever do business with and the best investment our shareholders ever make. So when looking to advance your career, look to M&T. As a top 20 US bank holding company and one of the best performing regional banks in the country, we offer a wide range of performance based career development opportunities for talented professionals. And through our longstanding tradition of careful, conservative and consistent management and a strong commitment to the communities we serve, we continue to grow with a focus on the future.