Skip to content

Latest commit

 

History

History
85 lines (57 loc) · 1.87 KB

README.md

File metadata and controls

85 lines (57 loc) · 1.87 KB

Simple app for scrapping data from gumtree.

🧐 About

The project was created for learning purposes to know how to combine scrapy framework with TorIp changer.

🏁 Getting Started

Prerequisites

  • Docker desktop

Project structure

.
├── docker-compose.yml
├── LICENSE
├── README.md
└── src
    ├── crawler
    │   ├── __init__.py
    │   ├── items.py
    │   ├── middlewares.py
    │   ├── pipelines.py
    │   ├── settings.py
    │   └── spiders
    │       ├── __init__.py
    │       ├── mieszkania2.py
    │       └── quotes_spider.py
    ├── Dockerfile
    ├── go_spider.py
    ├── scrapy.cfg
    └── tests
        └── ipchanger_works.py

Installing

Clone repository:

git clone https://github.com/Santhin/TorScrapy.git

To run the crawler type:

docker-compose up

🔧 Running the tests

Simple check if tor ip changer is working unmark commented test in dockerfile.
The exemplary output:

Project logo

🛠️ Todo

  • add control startup for TorIpChanger container in docker-compose

⛏️ Built Using

🎉 Acknowledgements