rs-trafilatura-loader

A minimal Rust Actix-web HTTP wrapper around rs-trafilatura that implements the Open WebUI external web loader contract.

Uses rs-trafilatura as backend and acts as a transport to collect URLs from Open Web UI -> Send it to rs-trafilatura in the format it expects -> Structures the result that Open Web UI web search pipeline can consume.

Usage

1. Clone the repo

   git clone https://github.com/pratik_tri/rs-trafilatura-loader
   

2. Create/Update docker-compose.yml in the same directory where repo is cloned:

   services:
      rs-trafilatura-loader:
          build: ./rs-trafilatura-loader
          environment:
              - PORT=8100
              - FETCH_TIMEOUT_SECS=30
              - NUMBER_OF_WORKERS=4
              - RUST_LOG="rs_trafilatura_loader=info,actix_web=warn"
              - LOADER_API_KEY="super_secret_api_key" # optional: should also be provided to open web ui as RAG_WEB_LOADER_API_KEY
          deploy:
            resources:
              limits:
                cpus: '1.0'
                memory: 256M
   
     open-webui:
       image: ghcr.io/open-webui/open-webui:latest
       ...
       environment:
           - RAG_WEB_LOADER_ENGINE=external
           - RAG_WEB_LOADER_EXTERNAL_URL=http://rs-trafilatura-loader:8100
           - RAG_WEB_LOADER_API_KEY="super_secret_api_key" # optional
   

3. Or go to Open WebUI > Admin Panel > Settings > Documents > Web Loader Engine, select external, and paste the URL.

Reason

Open WebUI ships with 4 web loader options: 1. Default 2. Playwright 3. Firecrawl 4. Tavily (paid)

The default loader uses requests + BeautifulSoup - it fetches the raw HTML and extracts text with no understanding of page structure. It treats navigation, ads, footers, and sidebar links the same as body content, so for anything beyond a simple article - forums, product pages, documentation - the output is a mess of mixed signals. Raw HTML tags are also expensive in terms of tokens, which makes the downstream steps in the Open WebUI pipeline work harder for worse results. It also can't handle content generated by JavaScript at runtime, so single-page applications will return next to nothing useful.

2 self-hosted alternatives are: - Playwright - Firecrawl

Playwright ships with a full Chromium engine, which is what makes it capable of handling JavaScript-rendered pages. Firecrawl runs Playwright internally and layers content cleaning and markdown conversion on top - so you get structured output instead of raw HTML, with less post-processing required. Both are memory-hungry: for a search returning 30 URLs, can consume 6–10 GB of RAM and still take 2–5 minutes to finish. If your queries regularly hit JS-heavy sites, there's no way around them.

rs-trafilatura is a happy middle ground between Firecrawl & Default web loader. Rather than rendering pages, it parses the raw HTML and classifies each page into one of seven types — article, forum, product, service, documentation, collection, or listing - then applies extraction logic tuned for that specific structure. The output is clean markdown, same as Firecrawl, so downstream pipeline gets structured content without boilerplate. The same 30-URL workload completes in under 30 seconds at around 60 MB peak memory with negligible CPU usage. The trade-off is that it only works on HTML; it has no browser, so JavaScript-rendered content that isn't present in the initial HTML response will be missed. For sites that serve their content in static HTML - which is still the majority of the web - it's the faster and leaner option.

Open WebUI Web Search Pipeline

When you trigger a web search in Open WebUI, the request flows through several stages. This wrapper replaces the loader stage only — search engine selection and RAG chunking are unchanged.

sequenceDiagram
    participant User
    participant OWUI as Open WebUI
    participant Search as Search Engine<br/>(SearXNG / DDG / etc.)
    participant Loader as rs-trafilatura-loader<br/>(this project)
    participant RAG as OWUI RAG Pipeline

    User->>OWUI: sends message with web search enabled
    OWUI->>Search: query string
    Search-->>OWUI: list of result URLs

    OWUI->>Loader: POST / {"urls": ["https://...", ...]}
    Note over Loader: fetches each URL with reqwest<br/>extracts main content via rs-trafilatura<br/>returns one document per URL
    Loader-->>OWUI: [{"page_content": "...", "metadata": {"source": "https://..."}}, ...]

    OWUI->>RAG: chunk documents, embed, retrieve relevant chunks
    RAG-->>OWUI: top-k chunks with source attribution
    OWUI-->>User: response with cited sources

The wrapper only handles the POST / call. Everything before it (what to search for, which engine) and everything after it (chunking, embedding, ranking) is handled by Open WebUI itself.

Limitations

• Static HTML only. Pages that require JavaScript to render content (most SPAs) will return empty or partial results. For those, Firecrawl is still the right tool.
• PDFs and other non-HTML content types are not handled and will be skipped with a warning.