Apply Now

Member of Technical Staff (AI Inference Engineer)

New York, NY, US • Posted 3 hours ago • Updated 3 hours ago

Full Time

On-site

USD $220,000.00 - 485,000.00 per year

Fitment

Dice Job Match Score™

📊 Calculating match score...

Job Details

Skills

Artificial Intelligence
Budget
Transformer
Caching
API
Migration
DSL
Network
Dashboard
Value Engineering
Rust
Python
Research
Management
Communication
InfiniBand
Remote Direct Memory Access
Debugging
CUDA
GDB
Sass
Orchestration
Kubernetes
Software Engineering
Machine Learning (ML)
Deep Learning
PyTorch
JAX
TensorFlow
GPU
Scheduling
Optimization

Summary

We build and run the inference engine behind every Perplexity query and deploy dozens of model architectures at scale with tight latency and cost budgets. Our stack is Rust, Python, CUDA, and CuTe DSL - and we need another engineer to join us.

What you will work on

Examples of real work the team does:

New models support. Support transformer-based retrieval, text-generation, and multimodal models in our inference infrastructure, from weight loading, request scheduling and KV-cache management to support in API Gateway.
GPU kernels migration to CuTe DSL. Port our in-house CUDA kernels to NVIDIA's CuTe DSL so they run on GB200 today and are portable to Vera Rubin racks tomorrow.
Rust-native serving runtime. Develop our internal Rust-based inference server to solve all Python pains and keep up with rapidly growing traffic.
Performance optimisation. Profile and fix bottlenecks from network ingress through continuous batching and GPU kernel interleaving.
Reliability and observability. Build dashboards, alerts, and automated remediation so we catch regressions before users do. Respond to and learn from production incidents.

Who we're looking for

Deep experience with GPU programming and performance work (CUDA, Triton, CUTLASS, or similar). Any other deep systems programming experience is a plus.
You understand modern LLM architectures and are able to bring them up reliably in a production environment.
You've built and operated production distributed systems under real load - ideally performance-critical ones.
Comfortable working across languages and layers: Rust for the serving runtime, Python for model code, CUDA/CuteDSL for kernels.
You own problems end-to-end. You can read a research paper on Monday, write a kernel on Wednesday, and debug a production incident on Friday.
Self-directed. You do well in fast-moving environments where the path forward isn't laid out for you.

Good if you touched any of

ML compilers and framework internals: PyTorch internals, torch.compile, custom operators.
Distributed GPU communication: NCCL, NVLink, InfiniBand, RDMA libraries, model/tensor parallelism.
Low-precision inference: INT8/FP8/FP4 quantization, mixed-precision serving.
Profiling and debugging tools: Nsight Compute/Systems, CUDA-GDB, PTX/SASS analysis.
Container orchestration: Kubernetes, GPU scheduling, autoscaling inference workloads.

Qualifications

3+ years of professional software engineering experience with meaningful work on ML inference or high-performance systems.
Familiarity with at least one deep learning framework (PyTorch, JAX, TensorFlow).
Understanding of GPU architectures (memory hierarchy, warp scheduling, tensor cores).
Understanding of common LLM architectures and inference optimization techniques (e.g. quantization, speculative decoding, prefill-decode disaggregation).

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 91140125
Position Id: 4a28a83fb4b37a46064cc850ccfbbbea
Posted 3 hours ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Senior Software Engineer - AI Inference

New York, New York

•

Today

Description & Requirements Our team: Join the team that is building the core infrastructure for AI at Bloomberg. The Bloomberg AI Inference Platform provides production-grade managed infrastructure for hosting, deploying, and serving all machine learning models, both predictive and cutting-edge generative models. We abstract away infrastructure complexity, empowering engineering teams to focus on creating intelligent applications with guaranteed scalability, performance, and governance. Our pla

Full-time

USD 160,000.00 - 240,000.00 per year

Machine Learning Performance Engineer

New York, New York

•

Today

We are looking for an engineer with experience in low-level systems programming and optimisation to join our growing ML team. Machine learning is a critical pillar of Jane Street's global business. Our ever-evolving trading environment serves as a unique, rapid-feedback platform for ML experimentation, allowing us to incorporate new ideas with relatively little friction. Your part here is optimising the performance of our models - both training and inference. We care about efficient large-scale

Full-time

Staff Backend Engineer, ML Inference Systems

New York, New York

•

Today

The opportunity Every day, we connect billions of players with the games and experiences they love. Our Vector Gamer AI team sits at the heart of that mission, governing ad ranking and bidding decisions across billions of daily impressions, where large-scale machine learning and real-world impact converge at scale. We're hiring a Staff Backend Engineer to build and operate the infrastructure those models depend on. You'll design and operate the distributed systems that power billions of daily

Full-time

USD 192,600.00 - 271,900.00 per year

Machine Learning Performance Engineer

New York, New York

•

9d ago

We are looking for an engineer with experience in low-level systems programming and optimization to join our growing ML team. Machine learning is a critical pillar of Jane Street's global business. Our ever-evolving trading environment serves as a unique, rapid-feedback platform for ML experimentation, allowing us to incorporate new ideas with relatively little friction. Your part here is optimizing the performance of our models - both training and inference. We care about efficient large-scale

Full-time

Search all similar jobs