Skip to content

Commit

Permalink
[readme.rst] edits, icons, quickstart
Browse files Browse the repository at this point in the history
  • Loading branch information
pawelmhm committed Feb 19, 2021
1 parent de1f7c3 commit f4abd43
Show file tree
Hide file tree
Showing 2 changed files with 47 additions and 13 deletions.
3 changes: 2 additions & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
Copyright (c) Scrapy developers.
Copyright (c) 2015 Scrapinghub

All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
Expand Down
57 changes: 45 additions & 12 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,14 @@ Scrapyrt (Scrapy realtime)
.. image:: https://img.shields.io/pypi/l/scrapyrt.svg
:target: https://pypi.python.org/pypi/scrapyrt

.. image:: https://img.shields.io/pypi/dm/scrapyrt.svg
:target: https://pypistats.org/packages/scrapyrt
:alt: Downloads count

.. image:: https://readthedocs.org/projects/scrapyrt/badge/?version=latest
:target: https://scrapyrt.readthedocs.io/en/latest/api.html
:alt: Latest Read The Docs

HTTP server which provides API for scheduling Scrapy spiders and
making requests with spiders.

Expand All @@ -22,13 +30,44 @@ components (e.g. middleware, pipelines, extensions) are supported out of the box
simply run Scrapyrt in Scrapy project directory and it starts HTTP server allowing you
to schedule your spiders and get spider output in JSON format.

Getting started
===============

To install Scrapyrt::

pip install scrapyrt

Now you can run Scrapyrt from within Scrapy project by just typing::

scrapyrt

in Scrapy project directory.

Scrapyrt will look for ``scrapy.cfg`` file to determine your project settings,
and will raise error if it won't find one. Note that you need to have all
your project requirements installed.

Scrapyrt supports endpoint ``/crawl.json`` that can be requested
with two methods.

To run sample `toscrape-css spider`_ from `Scrapy educational quotesbot project`_
parsing page about famous quotes::

curl "http://localhost:9080/crawl.json?spider_name=toscrape-css&url=http://quotes.toscrape.com/"


To run same spider only allowing one request and parsing url
with callback ``parse_foo``::

curl "http://localhost:9080/crawl.json?spider_name=toscrape-css&url=http://quotes.toscrape.com/&callback=parse_foo&max_requests=1"



Documentation
=============

Documentation is available here: http://scrapyrt.readthedocs.org/en/latest/index.html


Support
=======

Expand All @@ -40,16 +79,10 @@ Commercial support is also available by `Scrapinghub`_.
.. _create a question issue: https://github.com/scrapinghub/scrapyrt/issues/new?labels=question
.. _Scrapinghub: http://scrapinghub.com

License
=======
ScrapyRT is offered under [BSD 3-Clause "New" or "Revised" License](https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_(%22BSD_License_2.0%22,_%22Revised_BSD_License%22,_%22New_BSD_License%22,_or_%22Modified_BSD_License%22)

Development
===========

Release
-------

Use `bumpversion`_ tool, e.g. to release minor version do::

bumpversion minor --verbose
git push origin master
git push origin <new_version_tag>

.. _bumpversion: https://pypi.python.org/pypi/bumpversion
Development taking place on github: https://github.com/scrapinghub/scrapyrt

0 comments on commit f4abd43

Please sign in to comment.