Scaling
Feature and documentation planned.
Yosoi is designed to scale beyond a single machine. The integrations below are planned or in progress:
| Integration | Role |
|---|---|
| Redis△ | Distributed selector cache and job state |
| RabbitMQ○ | URL queue and worker coordination |
| Prefect◑ | Workflow orchestration and scheduling |
| Langfuse◇ | LLM observability and prompt tracing |
| Persistence | Durable result storage across runs |
| Turso★ | Embedded distributed SQLite for selector snapshots |
More integrations TBD.
References
△ Redis. Redis Ltd. In-memory data structure store used as a database, cache, and message broker. https://redis.io/docs/
○ RabbitMQ. Broadcom. Open-source message broker supporting multiple messaging protocols. https://www.rabbitmq.com/docs/
◑ Prefect. Prefect Technologies. Workflow orchestration platform for data and ML pipelines. https://docs.prefect.io/
◇ Langfuse. Langfuse. Open-source LLM observability, tracing, and analytics platform. https://langfuse.com/docs
★ Turso. ChiselStrike. Embedded distributed SQLite built on libSQL. https://docs.turso.tech/