What this does

Tiered Ingest allows you to group all of the files corresponding to a simple asset's datastreams (including archival files, publication files, other derivatives created outside of Islandora, with the exception of RELS-EXT) into a sub-directory.

General information for batch ingest

The batch ingest process runs continuously, looking for newly queued batch jobs approximately every 5 minutes. You can add batch ingest jobs to the queue at any time.

Batch jobs are subject to the following batch job size and file size limitations:

• max. 100GB/batch job
  
• max. 10GB/file

Step 1: Stage files for batch ingest job

Organise files in a batch job folder, using subfolders if appropriate. Refer to the instructions/options listed below for preparing batch jobs.

Staging folder structure

• All of the files you are ingesting as part of one asset will be staged in one directory per asset, as a sub-directory of a batch job folder.
• Each sub-directory corresponds to one asset and must contain at least a manifest file for the key datastreams (datastreams.txt).
• The batch job folder can contain just one asset folder, but would still need the extra nesting

eid1234_example-batch-submission/ (batch job folder)
├── asset1/
│   ├── datastreams.txt
│   ├── modsfile.xml
│	├── primaryfile.tif
│	├── anyarbitraryderivativefile.ext
│	├── anyarbitrarycomponentfile.ext
│   └── anymediaphotographfile.ext
├── asset2_audio_example/
│   ├── datastreams.txt
│   ├── modsfile.xml
│   ├── audiofile.wav
│   ├── derivative_audiofile_for_streaming.mp4 (e.g. for creating PROXY_MP4 datastream, which is required for streaming audio)
│   └── audio_transcript.txt
└──	asset3_video_example/
    ├── datastreams.txt
    ├── modsfile.xml
    ├── videofile.mp4
    ├── video_captions.vtt
    └── video_transcript.txt
    └── page02_custom_ocr.txt

Step 2: Create datastreams.txt manifest

Subdirectories in the batch job folder MUST each contain a manifest file named datastreams.txt. The manifest file specifies the intended structure of the DAMS asset, for instance pointing to the MODS XML containing the metadata for the asset, or specifying which additional datastreams should be created from staged files.

Each line of the manifest file contains an argument-value pair in the following format:

<ARGUMENT>==<VALUE> 

Use 2 (two) equal signs to separate arguments and values.

Manifest Arguments

Refer to Anatomy of DAMS digital assets and Content models for a list of allowed/expected datastreams per content model. Consult with the DAMS Management Team for use cases not covered by the datastreams listed in this documentation.

Manifest generator script

Sample manifests

OBJ==primaryfile.ext
MODS==metadata.xml
# optional, if no MODS file is included, minimal metadata is automatically generated during ingest
PDF==custom.pdf
# optional
ARCHIVAL_FILE==originalversionof_primaryfile.ext
# optional, use for archival file (e.g. uncropped scan)
COMPONENT1==componentfile1.ext
COMPONENT2==componentfile2.ext
# optional, can for instance be used in cases where a primary image is stitched from multiple component images; increment for additional files in same directory
# DO NOT use for complex objects that can be modeled as paged content or Islandora component assets!
MEDIAPHOTOGRAPH==anymediaphotographfile.ext 
# optional, can be used for images documenting physical media, cases, covers, etc.; use MEDIAPHOTOGRAPH if there is one image only
MEDIAPHOTOGRAPH1==anymediaphotographfile.ext
MEDIAPHOTOGRAPH2==anymediaphotographfile.ext
# optional, can be used for images documenting physical media, cases, covers, etc.; increment for multiple images documenting the physical carrier(s)

Step 3: Upload batch job to Jscape

1. Connect to jscape in SFTP client:

   Host: jscape.its.utexas.edu
   port: 22

2. Upload your batch job folder into the appropriate location in Jscape:

   1. TEST corresponds to running batch on dams-t01-rh7.lib.utexas.edu, PROD corresponds to running batch on dams.lib.utexas.edu

   2. Place your batch job folder in the appropriate top-level collection folder within the INGEST folder
      Example: /DAMS/TEST/INGEST/utlmisc/my_batch_job_folder (is what I would do for a batch upload to the miscellaneous collection on the DAMS Test Server).

      Any spaces in folder names must be represented by underscores (e.g. special_collection_1).

      


      We recommend naming your batch folder with your eid, a reference to the destination collection name in the DAMS, or anything else that will help you recognize the batch. In the example <my eid>_<what I am ingesting>, the folder name would be mm63978_EnPatufet1908-1911.

3. Go over to the DAMS interface and submit your batch job to queue it to be run (see steps below).
4. Note: Your batch job folder will be removed from the JScape server after seven days whether or not you have run the batch. Back up your batch requests in box or on your local machine. 

Step 4: Set up collection and submit form in DAMS interface

1. Navigate to or create the target sub-collection to receive batch ingested files in DAMS

2. Locate and copy the target sub-collection PID to clipboard (namespace:UUID, e.g. utlarch:9ebf6ac8-1823-4bf4-8398-654b54090776)

3. Navigate to the Batch Ingest form in the DAMS:
   1. Production system: https://dams.lib.utexas.edu/utdams/batch_queue
   2. Test system: https://dams-t01-rh7.lib.utexas.edu/utdams/batch_queue
4. Select the DAMS Top-Level collection from the dropdown field
5. Paste the PID of the target sub-collection into the form
6. Enter the name of the folder on the Jscape/FTP server that contains the files to be ingested (e.g. mm63978_EnPatufet1908-1911)
7. For batch ingest of simple assets and paged content: select the appropriate ingest type
8. Click submit

The DAMS should indicate at the top of the form that the batch ingest job was queued.

You will get an email notification after your request for a batch ingest has been received and another notice once the batch ingest process has finished.