Perplexity Details Its GPU Embedding Stack: How Ivy, Tulip and ROSE Serve pplx-embed

Retrieval quality in an AI search product is bounded by two things: how good the embedding model is, and how cheaply you can run it across an index. This week, Perplexity Engineering team published Fast Embeddings on GPUs, an under-the-hood account of the second — the serving infrastructure behind pplx-embed and the ranking models used across Perplexity Search, Computer and the API Platform.

Perplexity team states that embedding inference on the GPU side has largely converged across engines on mature Hopper and Blackwell hardware. The wins sit in the runtime and harness around the model: CUDA graph management, an async result-tracking abstraction, and a Rust request path.

Two traffic patterns, one engine

Perplexity frames embedding serving as two workloads. Batch embedding happens when building or re-indexing the vector database, where throughput minimizes cost. Online embedding happens at query time, where a short query must be embedded fast. Scoring sits in between: after vector search, large document batches are ranked, balancing both.

The key decision is that Perplexity did not build a separate embedding engine. Because embedding models are small Transformers, batch embedding resembles compute-bound prefill and online embedding, often a few tokens, resembles memory-bound decode. So the research team reuses the prefill and decode kernels from its LLM stack.

Ivy, Tulip and ROSE

Three services handle a request:

• Ivy is a Rust HTTP gateway. It does the CPU-side work — JSON parsing, tokenization, input templating, batch splitting — and translates requests into a custom gRPC protocol. It also splits large-batch requests into chunks and load-balances them across replicas, which corrects the load imbalance that arises when production payloads vary in size.
• Tulip is the inference server interface: a gRPC server built with Rust, tokio and tonic, handling scheduling and batching before dispatching to the engine.
• ROSE (Runtime-Optimized Serving Engine) implements model inference. It is primarily Python, provides kernels, layers and model definitions, manages CUDA graphs, and exposes a step() function to Tulip.

Why the scheduler is deliberately simple

Tulip picks sequences first-come, first-served while requests accumulate. That simplicity is justified by a measurement: for small embedding models at the sequence lengths Perplexity serves, the linear cost of dense layers dominates the quadratic cost of attention. Latency is therefore roughly proportional to token count, not sequence count. Once a batch saturates the GPU, around 512 tokens on a sub-billion-parameter model, packing in more sequences does not improve efficiency.

CUDA graphs and LazyTensors

On small batches, CPU-side kernel launching can outweigh GPU execution. Perplexity builds whole-model CUDA graphs for all embedding models, capturing every launch into a single driver call. Because embedding models are small, the inflection point where GPU work exceeds launch cost arrives at batches of thousands of tokens and tens of sequences. Some attention implementations block full-model graphs by depending on dynamic host-side inputs; Perplexity upstreamed changes to FlashInfer to enable capture.

Graphs must be captured per configuration, so token counts are padded to buckets that are multiples of 64 or 256. That still yields thousands of graphs and multiple minutes of capture per model. The fix is lazy capture: each configuration gets an eager warmup run, then triggers capture and replay on its second hit. This costs p99 latency at startup but spreads minutes of eager work across hours.

The second piece is the LazyTensor, which tracks a page-locked host buffer plus a cudaMemcpyAsync and a CUDA event. Instead of step() blocking on the device, it returns a LazyTensor, letting a Rust async task wait on batch N while the CPU enqueues N+1.

Kernels still matter

ROSE supports multiple attention backends for ragged inputs: FlashInfer 2, FlashInfer 3 and FlashAttention 4. Perplexity team reports FlashAttention 4 is generally faster, but FlashInfer 3 outperforms it on Qwen-based models at very long sequence lengths, so backend selection is made case by case. Notably, when serving an embedding model ROSE does not instantiate a KV cache and dispatches to ragged attention variants to avoid padding.

Benchmarks

Perplexity benchmarks against vLLM v0.22.0 in BF16 on real weights and eval-derived inputs, with warmup runs verifying cosine similarity divergence within 0.1%. Four suites are charted: low-latency embeddings (batch 1; 128/512/4096 tokens), low-latency scoring (batch 5/25/50 at 512 tokens), high-throughput embeddings (batch 100, four concurrent processes) and high-concurrency embeddings (1 to 16 concurrent requests, including Ivy tokenization and network overhead).

Key Takeaways

• Perplexity’s embedding stack reuses its LLM prefill/decode kernels rather than running a separate engine.
• Latency tracks token count, not sequence count; ~512 tokens saturates a sub-1B model.
• Whole-model CUDA graphs plus lazy capture cut launch overhead without minutes-long startup.
• LazyTensor overlaps CPU batch prep with in-flight GPU work instead of blocking on sync.
• Ivy, Tulip and ROSE are internal; pplx-embed is reachable via Perplexity’s Embeddings API.

Check out the Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us