Return the url and etag (which may be
None) stored for
filename or its stored metadata do not exist.
Given something that might be a URL (or might be a local path), determine which. If it's a URL, download the file and cache it, and return the path to the cached file. If it's already a local path, make sure the file exists and then return the path.
Given something that might be a URL (or might be a local path), determine check if it's url or an existing file path.
We ran into an issue where http requests to s3 were timing out, possibly because we were making too many requests too quickly. This helper function returns a requests session that has retry-with-backoff built in. see stackoverflow.com/questions/23267409/how-to-implement-retry-mechanism-into-python-requests-library
Given a URL, look for the corresponding dataset in the local cache. If it's not there, download it. Then return the path to the cached file.