Skip to content

Queue Priority; Clear Queues; Always On Support

Pre-release
Pre-release
Compare
Choose a tag to compare
@brianmadden brianmadden released this 26 Nov 00:08
· 13 commits to master since this release
0.4.3
c7f527c
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
  • Added ability to clear crawl queues by RequestId and Age, see Krawler#removeUrlsByRootPage
    and Krawler#removeUrlsByAge
  • Added config option to prevent crawler shutdown on empty queues
  • Added new single byte priority field to KrawlQueueEntry. Queues will always attempt to pop the lowest priority
    entry available. Priority can be assigned by overriding the Krawler#assignQueuePriorty method.
  • Update dependencies
Assets 2
Loading