Ray Serve

Ray Serve is a scalable model-serving library built on Ray, for deploying machine learning models and Python business logic as production inference APIs, with support for multi-model pipelines, GPU batching, and gradual rollouts.

It's designed for serving complex ML applications — pipelines chaining multiple models, ensembles, or business logic wrapped around a model — rather than just a single model behind a REST endpoint, and scales across a Ray cluster for high-throughput inference workloads.

Commonly used by ML teams already using Ray for training/data processing who want serving infrastructure in the same ecosystem, rather than a separate serving-specific tool.