RAG Web Browser implementation

[jirispilka]

Description

• Inputs: I aim to keep the configuration minimal yet usable (limit to EN, US for now).
• Functionality: Only extract titles and URLs from Google search results (adapted from Google extractors).
• Cheerio: Used for Google Search to improve speed.
• Playwright: Used for scraping (might be slow, but JavaScript rendering is a requirement). I avoided complicating this with adaptive crawling or crawler selection. We can keep it for later.
  Output: Configurable options include text (default), HTML (optional), and Markdown (optional).

see slack discussion for a reference

[jirispilka]

I need to change this actor to incorporate standby mode, as I initially misunderstood the standby functionality. Originally, I thought that standby mode would automatically wrap the existing actor into an "API server", exposing the running container. Therefore, I expected that sending a POST request to container_url with the payload would work.

[jancurn]

Good work @jirispilka, here is feedback from my side:

• Name: I asked Ales W. for a good name (to avoid overlaps with the existing actors)

How about RAG Web browser (rag-web-browser) ?

• Inputs: I aim to keep the configuration minimal yet usable (limit to EN, US for now).

Absolutely, I was thinking about INPUT like this:

• I’d use query instead of queries, just to keep it simple and nice in code example. In Standby API, you can call this endpoint multiple times for multiple queries anyway, which better balances the performance.
• maxPages - Maximum resulting number of web pages to load.
• FUTURE: searchType (select: search|images|..) - Would be great to expand to Google Image search too, now that "Traditional or "naive" Retrieval-Augmented Generation (RAG) is dead" :)
• formats (multi-option: text|markdown|html|image|pdf|docx|file) - Select text formats to be included in the results.
• pageTimeoutSecs - How many seconds wait for loading target web page. If reached, it will be skipped in the results. This is to ensure we’re within ~40 seconds timeout for function calling in GPTs and elsewhere. Default: 30 seconds.

The Standby API should just replicate input 1:1, and expose GET API endpoint at e.g. /search?query=test&formats=html,markdown&pageTimeoutSecs=5&...

• Functionality: Only extract titles and URLs from Google search results (adapted from Google extractors).

Sounds great

• Cheerio: Used for Google Search to improve speed.

Yeah, makes sense.

• Playwright: Used for scraping (might be slow, but JavaScript rendering is a requirement). I avoided complicating this with adaptive crawling or crawler selection. We can keep it for later.

Yeah absolutely, also if we use just one browser (Firefox?), it will keep the Docker image small and thus fast to start.

Output: Configurable options include text (default), HTML (optional), and Markdown (optional).

Sounds good, the results could look like this (either stored to output dataset, or returned directly from Standby API):

{
[ 
    {
      text: “...chunk of text from target page ...”,
	markdown: "...", // if enabled 
      metadata: {
        url: “http://www.example.com/dir/page.htm“,
        // createdAt: ...ISODate, // date of creation, if available
        // author: "Bob Dylan", // Author of page, if available
      },
    },
    ...
  ],
}

TODOs:

• add standby mode
• extract PDF, Docx content?

Yeah, definitely, we just need to figure how are we going to return file content in Standby API, in Base64 or kv-store?

see slack discussion for a reference

[jirispilka]

@jancurn Thanks for the feedback!

• Name: I asked Ales W. for a good name (to avoid overlaps with the existing actors)

How about RAG Web browser (rag-web-browser) ?

Ales has suggested: Google Search Data Extractor, which I like. I think it would be good to also include the "web browser"

• I’d use query instead of queries, just to keep it simple and nice in code example. In Standby API, you can call this endpoint multiple times for multiple queries anyway, which better balances the performance.

queries I kept the same as for the google-search-actor, but I will change it to query (sounds better)

• maxPages - Maximum resulting number of web pages to load.

I don't want to confuse users whether this refers to number of Google search pages to crawl or number of Google search results to return. I think maxResults is more clear in that sense. @jancurn what do you think?

• formats (multi-option: text|markdown|html|image|pdf|docx|file) - Select text formats to be included in the results.
• pageTimeoutSecs - How many seconds wait for loading target web page. If reached, it will be skipped in the results. This is to ensure we’re within ~40 seconds timeout for function calling in GPTs and elsewhere. Default: 30 seconds.

The Standby API should just replicate input 1:1, and expose GET API endpoint at e.g. /search?query=test&formats=html,markdown&pageTimeoutSecs=5&...

Thanks, will do

• Playwright: Used for scraping (might be slow, but JavaScript rendering is a requirement). I avoided complicating this with adaptive crawling or crawler selection. We can keep it for later.

Yeah absolutely, also if we use just one browser (Firefox?), it will keep the Docker image small and thus fast to start.

yes, firefox

Output: Configurable options include text (default), HTML (optional), and Markdown (optional).

Sounds good, the results could look like this (either stored to output dataset, or returned directly from Standby API):

{
[ 
    {
      text: “...chunk of text from target page ...”,
	markdown: "...", // if enabled 
      metadata: {
        url: “http://www.example.com/dir/page.htm“,
        // createdAt: ...ISODate, // date of creation, if available
        // author: "Bob Dylan", // Author of page, if available
      },
    },
    ...
  ],
}

ok, I'm having flat structure now, i.e.

{ 
  "title": "",
  "url" : "".
  "text": .......
}

Good point with the metadata. If we want to store additional info, it would be better to create the metadata fiedl right away.

[jancurn]

• "Google Search Data Extractor" is way too close to "Google Search Results Scraper" and all the other Google Search scrapers we already have in Store (see https://apify.com/store/categories?search=google+search), plus it feels like it just extracts data from Google Search, which is not true - the data is in web pages! And it doesn't communicate the main application of this Actor - for being web browser for RAG. Let's go with RAG Web browser (apify/rag-web-browser)
• re maxResults - fair enough, the results can be pages, but also files or images in the future
• "ok, I'm having flat structure now, i.e." - we need to return results from more than one page, so it needs to be an array. Good point about adding title too.

[jancurn]

For inspiration of results, see https://tavily.com/#api:

Note that they just use content probably for all other kinds of content. But what if we want to return both text and markdown together? I'd stick with our data model

[metalwarrior665]

Looks great!

[jirispilka]

@metalwarrior665 @jancurn I would like to ask you again for review.

Performance-wise, there is still plenty of room for an improvement. I did ad-hoc testing, with the following summary:

Here is a typical latency breakdown for the RAG Web Browser.
Please note, the these results are only indicative and may vary based on the search term and the target websites.

The numbers below are based on the following search terms: "apify", "Donald Trump", "boston". Results were averaged for the three queries.

Memory (GB)	Max Results	Latency (s)
2	1	36
2	5	88
4	1	22
4	5	46

You can find a breakdown in the performance_measures.md

Average:

request-received: 0 s
before-cheerio-queue-add: 129
cheerio-request-handler-start: 2499
before-playwright-queue-add: 87
playwright-request-start: 15703
playwright-wait-dynamic-content: 7029
playwright-remove-cookie: 1073
playwright-parse-with-cheerio: 5564
playwright-process-html: 3829
playwright-before-response-send: 236 
Time taken for each request: [ 49762, 16004, 42676 ]

There are some low-hanging fruits like: playwright-wait-dynamic-content, playwright-remove-cookie.
However, I'm uncertain about the potential improvements for other things like playwright-request-start.

[metalwarrior665]

Great job. 2 GB is definitely suboptimal for ideal speed. I think you could even go higher than 4 GB for default, depending if cost or latency is a bigger priority.

How many pages you think it will scrape usually? If 5 is normal, you could theoretically hyper-optimize by spawning those in separate standby Actors

[jirispilka]

Great job. 2 GB is definitely suboptimal for ideal speed. I think you could even go higher than 4 GB for default, depending if cost or latency is a bigger priority.

I updated the README in this spirit

Based on your requirements, if low latency is a priority, consider running the Actor with 4GB or more of memory.
However, if you're looking for a cost-effective solution, you can run the Actor with 2GB of memory.

How many pages you think it will scrape usually? If 5 is normal, you could theoretically hyper-optimize by spawning those in separate standby Actors

I'm not sure about how many pages people might use. My guess was somewhere around 1 to 5. I'm starting playwright with desiredConcurrency = 3.

[jirispilka]

Let me merge this PR now for the following reasons:

• The Actor has already been published, so it’s better to have the code in the master branch for anyone visiting the repository.
• This PR has become quite lengthy and complex.