Backend Engineer _ AI Gateway

Backend Engineer _ AI Gateway

Location: Englewood Cliffs, NJ, 07632 & Plano, TX, 75023-(5 days Onsite/week)

Duration: 1 year

Work Hours: 9AM-6PM local time

We are looking for candidates who are willing to relocate to Texas starting in October and are available to work onsite in New Jersey through September

Job Summary:

We are building an AI Gateway a high-performance, secure intermediary layer that routes, inspects, and governs all traffic between enterprise clients and LLM/AI service providers. This role requires deep expertise in network programming, async Python, and cloud infrastructure to design and operate a latency-sensitive, policy-driven gateway that handles massive concurrent streaming connections at scale.

What we are looking for:

• AsyncPython & Network Programming (3+yrs) Strong hands on experience with asyncio, aiohttp/anyio, low level TCP/UDP sockets, WebSocket & SSE servers, TLS handshake, connection pooling, keep alive, and zero copy buffering.
• Streaming & Large Payload Handling Ability to manage chunked transfer, bidirectional streaming, back pressure and high throughput data paths for real time LLM inference.
• Protocol Knowledge Working familiarity with HTTP/2&HTTP/3 (multiplexing, HPACK), gRPC, and occasional custom protocol parsers.
• FastAPI Based Backend (3+yrs) Design, develop and test production grade REST/Streaming APIs; pragmatic use of Pydantic, dependency injection and OpenAPI docs.
• Containerization & Kubernetes Docker, AKS/EKS/GKE deployment experience; HPA/VPA scaling, rolling updates, and service mesh (e.g., Istio/Envoy) basics.
• Async Messaging & Event Driven Design Production use of Redis Streams, RabbitMQ or Kafka for decoupled task queues, retries and back off logic.
• PostgreSQL & Vault Advanced query tuning, partitioning, PgBouncer pooling, plus HashiCorp Vault (dynamic secrets, transit encryption) for key management.
• Cloud Networking & Security L4/L7 load balancing, TLS termination, reverse proxy (NGINX/Envoy), mTLS between services, and automated cert lifecycle (ACME/Let sEncrypt).
• IaC & CI/CD Terraform modules for multi env infra, plus GitHub Actions/GitLab CI pipelines with container image scanning and canary/blue green deployments.

Work experience desired:

• Enterprise Security Gateways Integration with CASB/DLP/WSS or similar edge security platforms; centralized logging & audit trails.
• IAM & SSO Configuring Azure AD/Entra ID, Okta, SAML2.0 or OIDC SSO flows; managing Conditional Access & SCIM provisioning.
• AI/LLM Pipelines Building prompt engineering workflows, PII anonymisation, and monitoring Azure OpenAI (or comparable) usage, rate limits & cost.
• API Gateway Operations Custom webhook/REST integrations, circuit breaker patterns, rate limiting, auto retry and health checking.
• Observability & FinOps End to end monitoring with Datadog, PrometheGrafana or Azure Monitor; proactive cost optimisation and 24/7 incident response (SLA/SLO).