Universal Folder Listener

The Universal Folder Listener allows you to ingest files from a Folder on your system.

The Universal Folder Listener Input connection is available only for on-premise / private cloud instances of Hyperscience.

Sample use cases

• Another system places documents in a system folder on a regular basis. I want to ingest those files one by one.

• I put several related files into subfolders to be ingested as one submission per subfolder into Hyperscience.

• I want to regularly scan and ingest certain file types from a shared system folder.

Block settings table

In addition to the settings outlined below, you can also configure the settings described in Universal Integration Block Settings. 

Naming of files by the system

After files are ingested, the system adds $ufl_<file_path_after_base_folder> to the beginning of each file name (e.g., file1.png in the folder1 subfolder becomes $ufl_folder1/file1.png, fileA.png in the base folder becomes $ufl_fileA.png).

These names appear in the submission output as submitted_filename values.

Setting up Universal Folder Listener

To set up the Universal Folder Listener, enter the settings as described in the Block settings table above. You do not need to configure anything in the folder you are monitoring.

Migrating from the Folder Listener

There are numerous benefits for users upgrading from the legacy Folder Listener to the Universal Folder Listener. These include:

• Submissions with multiple files

• Simplified file structure

• Support for Cases

• Numerous UX improvements

Before setting up a Universal Folder Listener connector and retiring your Folder Listener connection, you need to complete some preparatory steps.

Check your folder structure

As mentioned previously, the Universal Folder Listener accepts both single files and folders of files as submissions. Whether a file is processed as its own submission or as part of a larger submission is determined by its location and the configuration of the Listener.

• If it is directly under the base folder, the system processes it as an individual submission.

• If it is in a subfolder of the base folder and the Group Subfolder Files setting is enabled, the system considers it part of a larger submission, which consists of all files in the subfolder and in any folders within that subfolder.

• If it is in a subfolder of the base folder and the Group Subfolder Files setting is disabled, the system processes it as an individual submission.

Depending on how you want your files to be processed, you may need to make some changes to your base folder's structure before using the Universal Folder Listener.

Verify the path of your base folder

The system uses the file path in the FS_INPUT_PATH ".env" file variable as the default location of the base folder. While you can enter any subdirectory of that folder in the Folder To Scan For Submissions field, for greater security, you may want to modify the value of FS_INPUT_PATH if you are certain you want to use one of its subfolders as your base folder.

Update your metadata files

The Folder Listener connector supported text files that contained submission metadata. The Universal Folder Listener can accept a JSON file for each submission that contains metadata, case data, and an external_id. The name of this file depends on whether the submission consists of a single file or a folder of files:

• filename.ext.json for individual files, where filename is the name of the file and ext is the file's extension

• foldername.json for folders of files, where foldername is the name of the base folder's subfolder

The metadata file for a file or folder must be in the base folder. If you put it in one of the base folder's subfolders, a submission won't be created from the file or folder the metadata is for. 

An example JSON file for a Universal Folder Listener submission appears below.

{
    "metadata": {
         "test": "Metadata for inner1"
    },
    "cases": [{
         "external_case_id": "900",
         "filenames": ["div_lic_1.jpg", "div_lic_1.jpg"]
    }],
    "external_id": "123"
}

By contrast, the metadata text files supported by the original Folder Listener (filename_index.txt) consist of key/value pairs that you define:

{
    "key1": "value1",
    "Key2": "value2"
} 

Therefore, when migrating to the Universal Folder Listener, you need to change the names, file types, and content of your metadata files, as outlined in the table below.

Check the location of metadata files

If Include Submission Level Parameters is enabled, metadata files are required and must be in the correct location in order for submissions to be created from the files they describe.

• If Include Submission Level Parameters is enabled and Group Subfolder Files is enabled:

  • The metadata file for a file or folder must be in the base folder. If you put it in one of the base folder's subfolders, a submission won't be created from the file or folder the metadata is for.

• If Include Submission Level Parameters is enabled and Group Subfolder Files is disabled:

  • Metadata files for files in the base folder must be in the base folder.

  • Metadata files for files in a subfolder (e.g., base_folder/folder1/file1.png.json) must be in the same subfolder as the files they are for (e.g., base_folder/folder1). Any metadata files for subfolders (e.g., base_folder/folder1.json) are ignored and removed. 

• If Include Submission Level Parameters is disabled:

  • Any metadata files will be ignored and removed by the system.