Optimize SDK Documentation URL Structure for Better Scraping and Performance in Tools like Cursor

[itisgriff]

🚀 Feature

Optimize the URL structure of the Supertokens SDK documentation to reduce unnecessary scraping by tools like Cursor. Specifically, change the URL prefix for SDK documentation from /docs/{sdk} to /sdk/{sdk}.

This would benefit users of the @Docs annotation in tools like Cursor by allowing them to reference only the latest or LTS versions of the docs, without pulling excessive pages. Currently, users scrape thousands of pages (Currently at 4,250+ pages in my Cursor with the entry point URL being https://supertokens.com/docs) due to the inclusion of multiple SDK versions, causing slower performance and data overload. Separating SDK docs into /sdk paths would allow users to choose whether to scrape just the latest docs or all SDK versions.

From the Supertokens side this would help reduce server load and bandwidth consumption caused by unnecessary scraping of outdated or redundant docs, improving overall site performance and server efficiency.

Implementation details

To implement this, the URL structure of the SDK documentation could be modified as follows:

• Update SDK URLs from /docs/python, /docs/web-js, etc., to /sdk/python, /sdk/web-js, etc.
• This would allow users to add /docs to their tool to reference the latest docs, while those who need all SDK versions could add /sdk separately.
• Only the latest or long-term support (LTS) versions of the SDK docs should be scraped, reducing page count and improving performance for users and the docs site.

I am not sure how possible this type of implementation would be but I believe this would be both beneficial to the end users of Supertokens who make use of tools like Cursor or Windsurf and the Supertokens maintainers to have less scrapper traffic to their site.

Thank you all!