Senior Apache NiFi Data Engineer

Position Name: Senior Apache NiFi Data Engineer (US) 

Project Duration: 6–12 months initially

Shift Details: US business hours (EST preferred), with daily overlap with the Ksolves offshore team (IST).

Preferred Education: B.Tech/ BCA/MCA/BE (or Bachelor's in CS/Engineering, or equivalent experience)

Experience: 6+ years in data engineering, including 3+ years building production Apache NiFi flows

Location / Mode: WFH – Remote, USA only (PHI data; US residency mandatory)

Primary Skills Required / Must Have:- 

1. 3+ years building production Apache NiFi flows: process groups, controller services, record readers/writers, back-pressure tuning

2. DBCP Connection Pool and JDBC-based integrations (client loads PostgreSQL through the Duck DB JDBC driver)

3. Parameter Contexts, NiFi Registry and flow versioning to build reusable, parameterized facility templates

4. Change data capture and state management in NiFi (processor state, distributed cache, trigger-based CDC control tables)

5. Idempotent upsert and incremental-load logic into PostgreSQL

6. NiFi cluster deployment, configuration, upgrade and patching, incl. Zookeeper (client on NiFi 1.28.1; 2.x upgrade likely)

7. NiFi security, monitoring and troubleshooting in clustered and standalone deployments (provenance, bulletins)

8. Advanced SQL: complex joins, window functions, CTEs, unions across multiple files, data flattening

9. Reading query plans and writing performant queries against large files and tables

10 Parsing CSV, XML, JSON and Excel files, including nested structures

11 Handling encodings, malformed/skipped rows and schema drift between facilities

12 PostgreSQL: schema design, indexing, bulk loading, transactions, locking and load performance

13 Error handling, row-count reconciliation and alerting (e.g. Slack) on every pipeline

14 Promotion of flows and SQL across Dev, Test and Prod with Git version control

15 Linux environments

16 Strict handling of PHI/PII under security and data-privacy rules; clear client-facing communication

Secondary Skills Required / Good to have:- 

1. Duck DB: JDBC driver and file functions (read_csv, read_json, Excel/XML extensions), incl. legacy FoxPro DBF files

2. Azure Blob Storage (client landing zone) and Azure networking/security basics

3. Cloud Beaver or Deaver for SQL development

4. Medallion architecture (Bronze/Silver/Gold), Parquet, OLAP or one-big-table (OBT) modeling

5. Healthcare data and HIPAA/PHI work; trauma registry data a plus

Brief Job Description:

The company is migrating closed trauma records from legacy registries (B5, Trauma Base and Trauma One formats) into a PostgreSQL data lake for reporting in Apache Superset. The engineer will build Apache NiFi flows that pick up client-uploaded files from

Azure Blob, load them through Duck DB into PostgreSQL, and make each new facility faster to onboard. Flows are 80–90% similar within a format, so reusable templates are the core of the work. EDI (835/837) ingestion is out of scope for now.

Key Responsibilities:

• Design, build and maintain NiFi process groups that ingest, transform and load facility data end to end, keeping NiFi to two or three processors per load and pushing heavy parsing to DuckDB SQL.

• Write and optimize SQL to parse and flatten CSV, XML, JSON, Excel and DBF files, including unions across multiple files of one dataset.

• Implement CDC, incremental loads and idempotent upsets into PostgreSQL, following the client's existing trigger-based CDC tables.

• Handle file-encoding issues, skipped rows and schema differences between facilities, logging rejected rows for review.

•Build parameterized flow templates (Parameter Contexts, NiFi Registry) so new facilities onboard quickly across the 221-facility backlog.

• Add error handling, row-count reconciliation and Slack alerting to every pipeline.

• Tune NiFi and database performance: connection pools, batching and memory for large files.

• Version-control flows and SQL and support promotion across Dev, Test and Prod; support the NiFi 1.28.1 to 2.x upgrade if scheduled.

• Recommend improvements to credential handling and infrastructure set-up, and contribute to the client's living process document.

• Document flows and data mappings; work daily with Cory Hubert, NQS SQL developers and the Ksolves offshore team.

Good-to-Have Skills:

• Healthcare integration: EDI X12 (837, 835, 270/271), HL7, Infor Cloverleaf (future scope only)

• NiFi scripting (Groovy or Python processors) or custom Java processor development

• NiFi 2.x and CI/CD for NiFi flow deployment

• Apache Superset and Open Metadata (client's reporting and metadata tools)

• Talend experience (client's CDC tables mirror Talend's trigger-based CDC)