Senior Software Engineer #2 (Microsoft SQL Database Administrator)

Project Task and Deliverables:

1. Implement Data Dog as a monitoring solution for SQL databases and servers: Change SQL database backups from native SQL backups with maintenance plans to Rubrik's backup solution in AWS using the Rubrik Agent and SLAs.
2. Test and Document Rubrik Agent SQL Backup Standards: Test and document backup and restoration standards to ensure data integrity and availability.
3. Secure and Encrypt SQL Production Databases at Rest: Implement and enforce database security protocols, including access controls and encryption on production SQL Databases.
4. Migrate SQL Databases to AWS Cloud: Migrate SQL Databases from on-premises data center to AWS cloud data center
5. Standardize AWS Cloud database access configurations: Review and standardize the implementation of AWS security groups for connection to SQL for the enterprise and individual applications.
6. Implement Data Dog as a monitoring solution for SQL databases and servers: Monitor resource utilization using DataDog and AWS CloudWatch and optimize resources.
7. Implement DataBase monitoring for AWS Cloud data center: Implement database monitoring in AWS to monitor outages and performance issues and create alerts and dashboards.
8. Document legacy Db2 jobs: Documentation of legacy Db2 administrative jobs for mainframe applications, including detailed Job Control Language (JCL) and job schedules for recurring database maintenance tasks such as statistics collection, reorganization, and image copy operations. Supports database integrity, performance optimization, and compliance with operational standards.
9. Perform Db2 Imagecopy, Recovery, and Data Integrity Analysis: Document and complete Db2 Imagecopy and Recovery procedures with validated integrity reports, and documentation confirming data availability and compliance with recovery standards.
10. Document JCL and job schedules for regular statistics, reorganization, and image copies.: Deliverable: Perform Security audits and RacF security cleanup on DB2 application databases.

Roles/Responsibilities:

General Services

• Review/create/update Visio diagrams for existing/new SQL on-premises, SQL in cloud, and AWS RDS infrastructure.
• Create/document/implement disaster recovery strategy for 24x7 production (on premises/cloud), including backups/restores, recovery procedures, testing, validity checks, status reports, and email alerts to minimize downtime.
• Perform systems/database tuning analysis and implement enhancements for performance optimization.
• Advise/work closely with development teams on DB-related issues/activities.
• Perform activities using best practices, in compliance with IT standards/procedures and audit requirements (reliability, quality, security).
• Review/create/update test plan documents as needed.
• Define/review/document/adhere to maintenance procedures, security standards, auditing policies, best practices, and standards for SQL on-premises/AWS DB infrastructure.
• Provide 24x7 on-call support for mission-critical systems.
• Provide written weekly reports to supervisor on project/activity status.
• Prepare monthly status reports (accomplishments, upcoming activities, assigned projects list, related issues).
• Attend DHCS Unit staff meetings and discuss project status.

Key Staff-Specific Responsibilities

• Provide expert-level database administration, performance tuning, and application support for a critical enterprise environment. This role will focus on the Resource Access Control Facility (RACF) Initiative, Change Control Process, Intake for DB2 Object Requests, and collaboration with application teams.
• Perform day-to-day DB2 administration operations, which include configuring and maintaining DB2 z/OS environments.
• Perform and document daily health checks, including but not limited to viewing and monitoring thread activities, monitoring Tablespace usage and allocations to avoid out-of-space Tablespace failures, and reorganizing Tablespaces as necessary to improve performance and ensure the overall health of DB environments and databases.
• Identify and maintain lists of all production databases and production required Image copy jobs.
• Review, maintain, and update Imagecopy retention periods and BMC Logmaster use.
• Review, maintain, and update documents related to IDAA installation and monitoring.
• Update DB2 job alerts and email notification settings in ESP and elsewhere.
• Work with external and internal developer customers on their DB2-related projects, including changes, issues, fixes, errors, requirements, access, and any other DB2-related activities.
• Update DB2 test plans as needed.
• Plan and upgrade DB2 to Function Level 510.
• Plan and upgrade DB2 to meet the latest version used by the California Department of Technology.
• Create, recommend, document, and adhere to guidelines and processes for the administration and monitoring of DB2 on z/OS.

Mandatory Skills:

1. Seven (7) years of FTE experience in electronic data processing systems study, design, and programming. At least four (4) years of that experience must have been in a lead capacity.
2. Possess a bachelor's degree in an IT-related or Engineering field. Additional qualifying experience may be substituted for the required education on a year-for-year basis.
3. Five (5) years of FTE experience performing administration and managing SQL servers and databases, including SQL Server 2014-2022, SSIS, SSAS, and SSRS.
4. Five (5) years of FTE experience in ensuring automated data consistency between production, User Acceptance Testing (UAT)/Staging, and lower-level environments.
5. Five (5) years of FTE experience writing T-SQL code and stored procedures to monitor, automate and support database administrator (DBA) day-to-day activities
6. Five (5) years of FTE experience with optimizing database access and allocating resources for optimum performance.
7. Five (5) years of FTE experience supporting 100+ database servers and ensuring security control, data integrity, and accessibility of data.
8. Five (5) years of FTE experience with database backup/ recovery and fast recovery strategies.
9. Five (5) years of FTE experience with reviewing database code from a data security perspective.
10. Five (5) years of FTE experience supporting Always-ON High Availability clustered databases.

Desirable Skills:

1. Experience with cloud services (AWS or Microsoft Azure) converting MS SQL databases to Azure SQL Managed Instance, Azure SQL, and AWS RDS. Six (6) points will be given for every twelve (12) months of experience for a maximum of thirty (30) points.
2. Experience with creating APIs for self-service consumption and third-party data access or data extraction. Six (6) points will be given for every twelve (12) months of experience for a maximum of thirty (30) points.
3. Experience with SQL monitoring tools (i.e., Idera Diagnostic Manager, Redgate ToolBelt, DataDog, DynaTrace, etc.). Six (6) points will be given for every twelve (12) months of experience for a maximum of thirty (30) points.
4. Experience with SQL auditing tools (i.e., Imperva SecureSphere, etc.). Six (6) points will be given for every twelve (12) months of experience for a maximum of thirty (30) points.
5. Experience with relational database management systems (RDBMS) like Oracle, MySQL, or other open source databases such as NoSQL, PostgreSQL. Six (6) points will be given for every twelve (12) months of experience for a maximum of thirty (30) points.