Roadmap

If something isn’t shipped yet, you’ll find it in one of two places:

Here — on this page, as a major planned item
GitHub Issues — as a tracked issue in the Yosoi repo

If you want to follow progress on a specific feature, the GitHub issue is the most granular source. If you want a high-level picture of where the project is headed, start here.

Planned

Proxy Support

Planned

First-class proxy configuration for routing requests through dedicated residential or datacenter providers. See the Proxying page for ethical guidance and planned scope.

Scaling & Distributed Pipelines

Planned

Integrations with Redis^⬡, RabbitMQ, Prefect, Langfuse, and Turso^◎ for running Yosoi across multiple machines with shared selector caches and job coordination. See the Scaling page for the full integration roadmap.

QScrape L2 Sites

Planned

Fictional SPA test sites (React^○, Vue^◑, Svelte^◇, Solid^★) for benchmarking a future DOM fetcher against known, stable targets. Mirrors the existing L1 suite.

Distributed Selector Cache

Planned

Redis^⬡ and Turso^◎-backed selector storage for teams sharing discovery results across machines and CI runs. See Scaling.

Versioned Docs

Planned

Per-release documentation snapshots so older API versions remain accessible.

Conda Forge Distribution

Planned

Publish Yosoi to Conda Forge to make installation as easy as possible for users in the conda ecosystem.

Issue #56 Track this feature on GitHub

Proposed

DOM-Enabled Scraping (A3Nodes)

Proposed

A DAG and node-based pipeline architecture, actively in development, that will let Yosoi drive a real browser (Zendriver^△) to fetch and render JavaScript-heavy pages. L2 sites (SPAs, framework-rendered apps) will no longer require you to run a browser automation tool yourself before passing HTML in.

This introduces a new class of selector that can capture DOM assertions, actions, and arrangements (A3) without needing a bulky LLM to drive.

Canonicalization Engine

Proposed

A mode that merges selector caches across domains that serve the same content. If two sources (e.g. thewallstreetjournal.com and wsj.com) publish equivalent data, enabling canonicalization on a contract would collapse them into a single canonical selector set. Reducing redundant discovery and keeping extraction consistent. Implementation would likely involve tracing or logging logic to detect overlapping field mappings, then coercing the duplicate into the canonical domain’s cache. May warrant a separate repo/package.

Issue #39 Track this feature on GitHub

Super Crawler

Proposed

A first-class multi-page crawling mode. Currently Yosoi operates on single pages — there is no way to chain together multiple linked pages. Multi-page crawling is a common use-case that makes sense to include natively. This would likely require expanding the AI generation step so that stateful “steps” are defined for the LLM to replay its predecessor’s selectors and JavaScript actions needed to complete a crawl. With DOM-enabled rendering progressing rapidly via the A3Nodes work above, this is becoming increasingly viable.

Issue #34 Track this feature on GitHub

Smart Batching of Tokens

Proposed

Currently the pipeline sends N requests (one per contract field), each containing the full HTML, and hopes the LLM’s context window is large enough. This needs to be more defensive. A smart batching or context management system would use a lightweight lxml pre-processor to trim the HTML to a safe-enough, close-enough subset per contract — reducing token cost and avoiding context window overflows.

Issue #50 Track this feature on GitHub

Dynamic Schema Builder

Proposed

A JSON codegen tool that lets users define a schema in JSON and generates a Pydantic contract from it. This lowers the barrier for newcomers who find OOP-style contract definitions (ys.Contract) intimidating but still need the specificity that Pydantic models provide for structured LLM output. Useful for REST and MCP design workflows as well. The Prompt → Contract → Selectors → Content pipeline would gain a new entry point.

Issue #33 Track this feature on GitHub

Custom Yosoi SLMs

Proposed

Custom Small Language Model for fully offline private scraping ≤ 3 billion parameters.

References

△ Zendriver. cdpdriver. Python-based browser automation using the Chrome DevTools Protocol. https://github.com/cdpdriver/zendriver

○ React. Meta. JavaScript library for building user interfaces. https://react.dev/

◑ Vue. Evan You. Progressive JavaScript framework for building UIs. https://vuejs.org/

◇ Svelte. Rich Harris. Compiler-based frontend framework. https://svelte.dev/

★ Solid. Ryan Carniato. Fine-grained reactive UI library with no virtual DOM. https://www.solidjs.com/

⬡ Redis. Redis Ltd. In-memory data structure store used as a database, cache, and message broker. https://redis.io/docs/

◎ Turso. Turso. Edge-hosted distributed SQLite database. https://turso.tech/