thereadme/web-scraper
1
0
Fork 0
Code Issues 1 Pull Requests Projects Releases Wiki Activity

fix: memory improvement

Browse Source
This commit is contained in:
Mahdi Dibaiee 2017-04-22 13:21:43 +04:30
parent f67502e68c
commit 632a840572
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
index.py
View File

@ -79,7 +79,7 @@ for host in sites:
if href and href != '/':
p = to_absolute(href, host)
if p and p.netloc == main.netloc:
if p and p.netloc == main.netloc and p.geturl() not in visited:
queue.insert(0, p)