Skip to content

Commit 05073e6

Browse files
committed
docs: document batch collection in readme
1 parent e24039b commit 05073e6

1 file changed

Lines changed: 22 additions & 0 deletions

File tree

README.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,28 @@ snapshots found: 207
4343
| lazerlab.net | 200 | https://web.archive.org/web/20180314153547/http://lazerlab.net/ | 20180314153547 |
4444

4545

46+
### Batch Collection
47+
48+
`get_url_list_snapshots` runs the same collection over a list of URLs, saving each
49+
result to `<output_directory>/<md5(url)>.csv`:
50+
51+
```python
52+
import wayback
53+
54+
wayback.get_url_list_snapshots(
55+
urls=["lazerlab.net", "archive.org"],
56+
start_timestamp="20180101",
57+
end_timestamp="20231231",
58+
output_directory="data/snapshots",
59+
)
60+
```
61+
62+
URLs whose output file already exists are skipped, so an interrupted run can be
63+
restarted without re-collecting anything. Requests are spaced `sleep_time` seconds
64+
apart (default 5) to be polite to the API, and an error on one URL is printed and
65+
skipped rather than stopping the batch.
66+
67+
4668
## Dependencies
4769

4870
`waybackpy`: For interacting with the Wayback Machine API.

0 commit comments

Comments
 (0)