web-scraper
===========

A simple script that scrapes a website, extracting texts in a CSV file with the format below, and saving images.

| Page      | Tag                             | Text         | Link              | Image                  |
|-----------|---------------------------------|--------------|-------------------|------------------------|
| page path | element tag (h{1,6}, a, p, etc) | text content | link url (if any) | image address (if any) |

## Usage
First, install dependencies (python3):

```
pip install -r requirements
```

Then create a file containing urls of the websites you want to scrape, one line for each website, for example (I'll call this file `test_websites`):

```
https://theread.me
https://theguardian.com
```

Now you are ready to execute the script:

```
python index.py test_websites
                # ^ path to your file
```

After the script is done with it's job, you can find the results in `results/<website_hostname>` folder.

To see available options, try `python index.py -h`.