datahoarder @lemmy.ml BermudaHighball @lemmy.dbzer0.com 1 wk. ago

Downloading all archive.org metadata

I'd love to know if anyone's aware of a bulk metadata export feature or repository. I would like to have a copy of the metadata and .torrent files of all items.

I guess one way is to use the CLI but this relies on knowing which item you want and I don't know if there's a way to get a list of all items.

I believe downloading via BitTorrent and seeding back is a win-win: it bolsters the Archive's resilience while easing server strain. I'll be seeding the items I download.

Edit: If you want to enumerate all item names in the entire archive.org repository, take a look at https://archive.org/developers/changes.html. This will do that for you!

7 comments

I use the CLI tool, even right now waiting to finish some downloads. The CLI tool can actually give you a list of all items with ia list {ID} (replace {ID} with the actual id of the stuff you want to download). But you don't even need to list the items, because you can download with a glob (in example *.torrent like your shell has. Or if you have the ID anyway, you can specify the filenames too with {ID}_archive.torrent

Here is an example how to do this with my own upload https://archive.org/details/gamefaqs_txt where the id becomes gamefaqs_txt

ia download gamefaqs_txt --glob *.torrent

or use a variable to set id and download all files that start with the id, which should be all the meta data

id=gamefaqs_txt ; ia download "${id}" --glob "${id}_*"
- Thank you for the tips. I am actually interested in enumerating metadata for all the "items" as defined by the API page ever uploaded. For example, one item = one ID:
  
  Archive.org is made up of “items”. An item is a logical “thing” that we represent on one web page on archive.org. An item can be considered as a group of files that deserve their own metadata.
  
  You did cause me to look at the API docs again, though, and I think I found something that does enumerate all item names, and as a bonus, it will keep you updated when changes are made: https://archive.org/developers/changes.html
  
  We'll see how much progress I can make. It might take a while to get through all the millions of them.
  
  Isn't "item" and "id" basically the same thing? Because every item has a unique id. So in my example gamefaqs_txt would be the item and id.
I just found out you can get all metadata with ia metadata ID > metadata.json (replace ID with gamefaqs_txt in example). So from there you could extract any information too, if you know how to handle json. (Edit: Just load the metadata.json in your browser to see a better formatted list.)
- I like to pipe my json to python -m json.tool for quick formatting in the terminal.
  
  Take a look at jq, it's a really nice tool for handling json in the terminal, also gron for searching json

You've viewed 7 comments.