I have several tar archives that I need to extract/read in memory. The problem is each tar contains many ZIP archives and each contain unique XML documents.
So the structure of each tar is as follows: tar -> directories-> ZIPs->XML.
Obviously I can manually extract a single TAR but I have about 1000 TAR archives that are about 3 GB each and contains about 6000 ZIP archives each. I'm looking for a way to handle the .tar archives in memory and extract the XML data of each ZIP. Is there a way to do this?
This should be doable, since all of the relevant methods have non-disk-related options.
Lots of loops here, so let's dig in.
For each tar archive:
tarfile.openwould open the tar archive. (Docs)
.getmemberson the resulting
TarFileinstance to get a list of the zips (or other files) contained in the archive. (Docs)
For each zip within the tar archive:
- Once you know what member file (i.e., one of your zips) you want to look through, call
TarFileinstance to get a file object for that zip. (Docs)
- Instantiate a new
zipfile.ZipFilewith your file object in order to open the zip so you can work with it. (Docs)
ZipFileinstance to get a list of the files it contains (including your XML files). (Docs)
For each XML file within the zip:
ZipFileinstance in order to get a file object of one of your XML files. (Docs)
- You now have a file object corresponding to one of your XML files. Do whatever you want with it:
.readit, copy it to disk somewhere, stick it in an