arXiv-tools
Prerequisites
ArXiv provides bulk data access through Amazon S3. You need an account with Amazon AWS to be able to download the data. You also need python 2.
Downloading arXiv documents
1- Install s3cmd which is a command line tool for interacting with S3
pip install s3cmd (only works on python 2)
2- Configure your s3cmd by entering credentials found in the account management tab of the Amazon AWS website
s3cmd --configure
3- Get the manifest files:
The complete set of arXiv files available from Amazon S3 in requester pays buckets. The files are in .tar format each with ~500MB size. You need to have the keys to these chunks to be able to download them. The complete list of these keys is provided in the manifest files. First download the manifests:
For PDF documents:
s3cmd get --requester-pays s3://arxiv/pdf/arXiv_pdf_manifest.xml local-directory/arXiv_pdf_manifest.xml
For source documents:
s3cmd get --requester-pays s3://arxiv/src/arXiv_src_manifest.xml local-directory/arXiv_src_manifest.xml
4- Download the actual pdf and source files using the download.py script
Download pdf files:
python download.py --manifest_file /path/to/pdf-manifest --mode pdf --output_dir /path/to/output
Download source files:
python download.py --manifest_file /path/to/src-manifest --mode src --output_dir /path/to/output
This will download all the files in the directory that you designated as output.
If you also need the metadata, use metha to bulk download the metadata.
import re
import sys
import json
import subprocess
import traceback
import xml.etree.ElementTree as ET
def main(**args):
manifest_file = args['manifest_file']
mode = args['mode']
out_dir = args['output_dir']
log_file_path = args['log_file']
if mode != 'pdf' and mode != 'src':
print('mode should be "pdf" or "src"')
def get_file(fname, out_dir):
cmd = ['s3cmd', 'get', '--requester-pays',
's3://arxiv/%s' % fname, './%s' % out_dir]
print(' '.join(cmd))
subprocess.call(' '.join(cmd), shell=True)
log_file = open(log_file_path, 'a')
try:
for event, elem in ET.iterparse(manifest_file):
if event == 'end':
if elem.tag == 'filename':
fname = elem.text
get_file(fname, out_dir='%s/%s/' % (out_dir, mode))
log_file.write(fname + '\n')
except:
traceback.print_exc()
print('Finished')
if __name__ == '__main__':
from argparse import ArgumentParser
ap = ArgumentParser()
ap.add_argument('--manifest_file', '-m', type=str, help='The manifest file for downloading from arxiv. Obtain it from s3://arxiv/pdf/arXiv_pdf_manifest.xml using `s3cmd get --add-header="x-amz-request-payer: requester" s3://arxiv/pdf/arXiv_pdf_manifest.xml`', required=True)
ap.add_argument('--output_dir', '-o', type=str, default='data', help='the output directory')
ap.add_argument('--mode', type=str, default='src', choices=set(('pdf', 'src')),
help='can be "pdf" or "src"')
ap.add_argument('--log_file', default='processed.txt', help='a file that logs the processed txt files')
args = ap.parse_args()
main(**vars(args))