grab-site
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Language: python
Author: GitHub Repos (@github-repos)
0 stars · 0 views
Files
- setup.py (py)
- libgrabsite (txt)
- dump_urls.py (py)
- wpull_tweaks.py (py)
- LICENSE (txt)
- tests (txt)
- extra_docs (txt)
- pause_resume_grab_sites.sh (sh)
- images (txt)
- dashboard.png (image)
- scriptorium.jpg (image)
- README.md (md)
- .travis.yml (yml)
- server.py (py)
- dashboard_client.py (py)
- default_cookies.txt (txt)
- wpull_hooks.py (py)
- __init__.py (py)
- favicon.ico (image)
- main.py (py)
- 404.html (html)
- dupes.py (py)
- dashboard.html (html)
- ignore_sets (txt)
- dupespotter.py (py)
- ATTRIBUTION.md (markdown)
