What this does

The batch ingest process for complex assets, or paged content, allows you to batch-create DAMS assets that consist of component parts, typically pages of a book/serial issue/archival file/etc. See pages Anatomy of DAMS digital assets and Content models for details on the structure of digital assets.

Assets created with this batch ingest method will consist of a set of page-level assets, a book/publication issue-level asset, and optionally a publication (series)-level asset. You can also use this method to add publication issues to an existing publication series, and you can add pages to existing book-level or issue-level assets.

Typically, a book-level asset and its pages are created using image files that are the result of a scanning process (digitally reformatted content). This batch ingest method also allows to use a PDF document as a source file, and the DAMS software will automatically create an image file for each page in a PDF submitted. For digitally reformatted (scanned) content, using a PDF is strongly discouraged, as the automatically created page images are almost invariably of lesser quality than the original scan images. Contact the DAMS managers for a consultation (click here to submit a DAMS service request).

For born-digital content, like modern PDF ebooks, other content models and ingest processes might be more appropriate. Contact the DAMS managers for a consultation (click here to submit a DAMS service request).

General information for batch ingest

The batch ingest process runs continuously, looking for newly queued batch jobs approximately every 5 minutes. You can add batch ingest jobs to the queue at any time.

Batch jobs are subject to the following batch job size and file size limitations:

• max. 100GB/batch job
  
• max. 10GB/file

Step 1: Stage files for batch ingest job

Organise files in a batch job folder, using subfolders if appropriate. Refer to the instructions/options listed below for preparing batch jobs.

Staging files for batch ingest of paged content

For batch ingest, your assets MUST be organized within a top level directory which represents the batch job. You will reference the batch job directory in the batch submission form in the DAMS GUI. Each batch job directory MUST contain one or more subdirectories, representing a book/publication issue, a publication/series, or a set of pages that should be added to an existing paged content asset. You can combine multiple books, issues, page addition sets in a batch job, as long as the job does not exceed the file and job size limitations.

Furthermore, publication issues can be nested within a subdirectory representing a publication/series, for instance if you want to create a publication/series and add publication issues at the same time. You can add further issues to the same publication with a separate batch job.

Subdirectories in the batch job folder MUST use one of the following suffixes to denote the content type or type of batch ingest:

<foldername>_PUBLICATION

Used to create a publication/series-level asset. These assets are intended to organize publication issues inside the DAMS. The DAMS GUI will provide a

<foldername>_ISSUE

<foldername>_BOOK

<foldername>_PAGES

, one or more directories with your asset(s) and the suffix denoting it's content type, a manifest (datastreams.txt)  and the content. 

*Note that if you are not including MODS at the publication level then still need a blank manifest at this level.

Guidelines for Manifest 

The Manifest, a file named 'datastreams.txt' is what defines what will be included in the resource you are ingesting.

Manifest Arguments

The available manifest arguments are:

Sample manifest (datastreams.txt) file for a book and/or publication:

Adding Custom Datastreams

You can provided custom datastreams with your own custom defined naming convention. Please do not use any of the Restricted Datastream IDs and add the suffix '_CUSTOM' to your Datastream ID. 

MODS System Generated Fields 

• keydate: if batch ingesting paged content, users need to add keyGen="yes" attribute to appropriate MODS element (dateCreated or dateIssued) to indicate which is keydate. If there is no keyGen, issues will appear to be missing from your Publication. Find issues with missing keydates is by going to the publication > Manage > Publication. Fix problem by editing MODS for problematic issue. 
• recordCreationDate
• identifier utldamspid, utldamsuri, filename
• relatedItem UTLDAMS digital collection and subelements

OCR Selection 

During a batch ingest of Paged Content, the specified OCR language must be supported in order for OCR to run. For further documentation about text extraction see here. The language should be set to the 3 digit language code. Currently supported languages and their codes are listed here. If OCR is specified and validated as a supported language, then the OCR will be run on the child pages. If specifying a language, the following line needs to be added to manifest:

Adding to an Existing Book or Issue

1. Place appropriate following argument in your manifest where after the == is the book/publication PID without the namespace (ie utblac, utlarch).
   • HOSTPUBLICATION
   • HOSTBOOK
   • HOSTISSUE
2. When you submit the batch request the pid of the sub-collection is ignored

Sample manifest (datastreams.txt) file when ingesting page(s) and/or issue(s).

3. When adding pages the directory with the pages to add should have the suffix '_PAGES'