Software Engineer, Inference Platform

Fluidstack•San Francsisco, CA

22h

About The Position

At Fluidstack, we’re building the infrastructure for abundant intelligence. We partner with top AI labs, governments, and enterprises - including Mistral, Poolside, Black Forest Labs, Meta, and more - to unlock compute at the speed of light. We’re working with urgency to make AGI a reality. As such, our team is highly motivated and committed to delivering world-class infrastructure. We treat our customers’ outcomes as our own, taking pride in the systems we build and the trust we earn. If you’re motivated by purpose, obsessed with excellence, and ready to work very hard to accelerate the future of intelligence, join us in building what's next. About the Role Inference is now the defining cost and latency bottleneck for frontier AI. Fluidstack’s Inference Platform team owns the serving layer that sits between our global accelerator supply and the production workloads our customers run on it: LLM serving frameworks, KV cache infrastructure, disaggregated prefill/decode pipelines, and Kubernetes-based orchestration across multi-datacenter footprints. This is a hands-on IC role at the intersection of distributed systems, model optimization, and serving infrastructure. You’ll own end-to-end inference deployments for frontier AI labs and our inference product, drive measurable improvements in throughput, cost-per-token, and time-to-first-token, and contribute to the platform architecture choices that determine how Fluidstack deploys across tens of thousands of accelerators.

Requirements

5+ years of professional software engineering experience with a track record of shipping production-quality systems.
Strong programming skills in Python and/or Go.
Hands-on production experience with at least one LLM serving framework (vLLM, SGLang, TensorRT-LLM, TGI, or equivalent).
Working knowledge of PyTorch or JAX and an understanding of how model architecture choices affect inference characteristics.
Experience deploying and operating GPU workloads on Kubernetes at production scale, including autoscaling and resource scheduling.
Solid understanding of GPU memory hierarchies, compute parallelism, and the tradeoffs across tensor, pipeline, and expert parallelism strategies.
Ability to create structure from ambiguity and communicate technical tradeoffs clearly to both engineering peers and customers.
Great written and verbal communication skills in English.

Nice To Haves

Production experience with disaggregated prefill/decode architectures (NVIDIA Dynamo, LLM-d, or equivalent), including scheduling policies and network fabric configuration.
Deep familiarity with KV cache strategies: RadixAttention, slab-based memory allocators, cross-request prefix sharing, and cache-aware scheduling.
Experience with multi-node GPU inference across InfiniBand or RoCE fabrics, including NCCL collective communication tuning.
Custom kernel or operator development experience (e.g., CUDA, Triton, torch.compile, Pallas, or equivalent)
Contributions to open-source inference engines (vLLM, SGLang, TGI, TensorRT-LLM, or similar).
Hands-on experience with quantization tooling: GPTQ, AWQ, FP8 via llm-compressor, or AutoGPTQ.
Knowledge of speculative decoding implementations (Medusa, EAGLE-3, draft-model approaches) and their performance/quality tradeoffs.
Experience optimizing and adapting model implementations for non-NVIDIA accelerators and their ecosystems: AMD, TPU, Trainium/Inferentia, Cerebras, Groq, and other custom ASICs.

Responsibilities

Own inference deployments end-to-end: from initial configuration and performance tuning to production SLA maintenance and incident response.
Drive measurable improvements in throughput, TTFT, and cost-per-token across diverse model families (dense transformers, mixture-of-experts, multi-modal) and customer workload patterns.
Build and operate KV cache and scheduling infrastructure to maximize utilization across concurrent requests.
Implement and validate disaggregated prefill/decode pipelines and the Kubernetes orchestration that supports them at scale.
Profile and resolve bottlenecks at the compute, memory, and communication layers; instrument deployments for end-to-end observability.
Partner with customers to translate their model architectures, access patterns, and latency requirements into deployment configurations and upstream platform improvements.
Contribute to inference platform architecture and roadmap, with a focus on reducing deployment complexity, improving hardware utilization, and expanding support for new model classes and accelerators.
Participate in an on-call rotation (up to one week per month) to maintain the reliability and SLA commitments of production deployments.