Document Drift Management (Layout Triage)

Document Drift Management (also known as Layout Triage), is a post-processing feature that helps you manage documents that don't match layouts during Classification. When submissions don't meet the Structured Layout Match Threshold or are manually flagged as having incorrect or missing layouts, their pages are marked as unmatched.

This feature replaces the previous "Find Potential Layouts" and "No Layout Found" processes, offering a more streamlined and effective way to handle unmatched pages.

Document Drift Management allows you to determine and create the necessary layouts for these pages, ensuring that similar documents can be accurately classified and processed. In this article, you'll learn how to use Document Drift Management to handle unmatched pages efficiently.

Misclassified vs. Unmatched pages
Misclassified pages are those that the system recognizes but assigns to the wrong layout. You can manually reassign these pages to the correct onе. Unmatched pages don't fit any existing layout, so they remain unclassified.

Grouping pages

Document Drift Management works by grouping unmatched pages based on their similarities. It uses a pixel-level analysis to identify pages that share common visual templates. These pages are then automatically clustered into groups, making it easier to identify potential new layouts.

Hyperscience recommends using this feature for Structured use cases.
Document Drift Management effectively handles Structured documents, where consistent formatting is essential for accurate classification.
In v41, Document Drift Management can be used for single-page Semi-structured documents (e.g. single-paged invoices), as well.

After the pages are grouped, you can review and adjust them as needed, then create new layouts from these groups.

The sections below explain how to use Document Drift Management to streamline the management of your layouts.

Using Document Drift Management

To access Document Drift Management, click on Submissions in the left-hand sidebar, and then click the Unmatched tab.

The Unmatched tab displays submissions with pages that did not match any existing layouts during the Classification process. This tab is the starting point for managing and organizing unmatched pages using Document Drift Management.

Follow the steps described below to learn how to navigate each one and use the grouping process effectively.

Submissions

Page Groups

Potential Layouts

Archived Page Groups

Step 1 - Unmatched Submissions

The Unmatched Submissions tab displays a table of all submissions with unmatched pages.

Navigating the Unmatched Submissions tab

Filters

Define a specific range of submission IDs by entering values for its beginning and end:

Submission ID Start
Submission ID End
Submission Date — Choose a date range for which you want to see results. The submissions are filtered by their creation date.

Managing Columns

You can add and remove columns by clicking on the table’s menu ().

Columns

The Unmatched Submissions table displays the following columns:

Submission ID — The specific ID of the submission
Submission Date — The date the submission was uploaded
Unmatched/Total Pages — The number of unmatched pages and the total pages in the submission
Unmatched Page IDs — Displays the specific ID of the unmatched pages within the submission. You can also see each unmatched page with the preview button ( ). Click it to lock the preview, and use the keyboard arrows to go through the pages.

Grouping Job — The consecutive ID number of the clustering job. Also indicates the status while the job is running:
- Queued — The Grouping job is pending. Cancel it by clicking thebutton.
- Running — The Grouping job is currently processing.
- Failed — The Grouping job failed. You can find the flow error stacktrace in the info icon ().
- Canceled — The Grouping job was canceled by the user.

Grouping Pages

Follow the steps below to run a grouping job:

Grouping jobs in Document Drift Management run in the Trainer. If the trainer is busy, your job will be queued.

Select the submissions containing the pages you want to group. Check the pages by using the preview button.
After you’ve made your selections, click Group Pages. The system initiates a visual clustering process, grouping pages based on pixel-level similarities.
After the grouping job is complete, click on the Page Groups tab.

Step 2 - Group Pages

The Page Groups tab displays a list of all clusters of similar pages that have been grouped. It helps you manage these groups, review ungrouped pages, and refine them before creating new layouts. Learn how to navigate the tab and how to use the triage experience below.

You need a minimum of 5 similar pages to create a group. If you have less than 5 similar pages, they will go to the Ungrouped category. However, you can manually create groups with fewer pages.

Navigating the Page Groups tab

The Page Groups table contains the following columns:

Group Name — The name of the page group.
Updated — The date when the group was last updated, along with the number of pages added since the last update.

Pages — The number of pages in the group.
Actions — A menu with options to download pages, archive, or delete the group.
- To archive the group, select it and click Archive from the Actions menu.

Your archived groups can be found in the Archive tab.

Layout Triage

Organize unmatched pages into layouts based on visual similarity using the triage experience.

Best for structured documents. Similar pages of semi-structured documents may be grouped together. Extra pages, like fax cover sheets, and those with fewer than 5 similar pages will go to the Ungrouped category.

To start the triage process, select the groups you want to manage, then click Triage Groups.

If you don’t select a group, clicking the Triage Groups button will open the Ungrouped category.

Follow the steps provided in the in-product guidance to begin the triage process. To proceed, you can either close the guidance or select "Don’t show this again" if you prefer not to see it in the future.

The steps below offer additional guidance on making the most of the triage experience.

Step 1: Review groups

While pages are automatically grouped based on visual similarities, it's important to manually review and adjust these groups to ensure accuracy. To create a group:

Click on a group in the To Do list on the left to view its contents and start the triage process.
Click on a page to display it in the preview section.
Select multiple pages and click New Group from Selection () to create a more precise grouping. After you’ve created a new group from the selected pages, it will appear in the To Do list on the left, and the icon will indicate that the group is manual.

Send the selected pages to the Unmatched group by clicking the Send selection to ungrouped () button. Once in the Unmatched group, the pages may be automatically regrouped.

If you manually create a group, similar pages will not be automatically added to it in the future. Ensure manual groups are accurately defined and adjusted as needed.

You can also transfer the selected pages to a different group by clicking the Move Selection button ().
The menu next to each group allows you to:
- View group’s metadata
- Merge groups
- Dismiss a group
- Download images from a group
- Archive Page group

Merging groups and moving pages
Merging groups or moving pages into an automatic group converts it into a manual group, which will no longer update automatically with new similar unmatched pages.

Download a selection by clicking the button.

Step 2: Enter Layout Metadata

You can define the following properties of the layout:

Layout Group - Type a name to create a new layout and press Enter to create it.
- In v41 and above, you can choose an existing layout and create a variation based on it. The variation will inherit the structure of the selected layout but may include modifications.
- If a layout already has variations, you can start from one of those instead of the original. This allows you to build upon an existing variation rather than starting from scratch.
Layout Code - Serves as a "note" that helps identify the layout during triage. It's an extra way to differentiate between layouts that may have the same name. Not to be confused with the layout identifiers in Structured layouts.

Select a unique layout and page codes. Enter this information for each group to help cross-reference and suggest potential layout variations.
Groups are cumulative, so new pages matching existing groups will be added in future grouping pages job runs.

Layout Type — Choose the layout type (e.g., structured or semi-structured) based on your documents.
Index — The page index can help identify if any pages are missing from a group. If a group lacks a specific page index, it may indicate that a page was not included in the submission or was missed during grouping.

Each group in the triage experience represents a single-page index, not the entire layout. For example, if you have pages 1, 2, and 3, one group will represent page 1, another group will represent page 2, and so on. This framework ensures that each page type is grouped and processed individually.

Ensure all properties are filled out, as they are required to proceed with the triage process, and click Next.

Step 3: Provide Blank

When creating a new Structured layout, provide a blank version of each page. For best results, using an original form that hasn't been filled is most effective with structured documents, where consistent formatting helps in accurate grouping.

When dismissing pages, a group can have 0 pages at the end. This behavior is expected if the group still has a blank page assigned or if it belongs to a Potential Layout.

To set a blank form:

Select the page you want to define as blank form.
Click Set as blank or

Upload a blank from your local machine by clicking Browse
Select your file
Click Upload

Triage Ungrouped

The number of ungrouped pages will increase over time. You can triage them when you have enough similar examples.

Review pages - Review the pages in the Ungrouped folder and delete any unnecessary pages, like fax cover sheets or blank sheets.
Manually create page groups - Select sets of pages in the Ungrouped folder to manually create additional groups as needed.
Follow the steps above for manually created page groups.

Similar Layout Variation Detection - Beta Feature

Beta Feature
Similar Layout Variation Detection is currently a beta feature, to enable it:
Go to Administration > System Settings
On the Beta Features card, click Edit and check Similar Variations Search
Click Save and then Confirm on the dialog box.

Structured layouts evolve as document versions change, often resulting in unmatched submissions and reduced automation rates. When an unmatched document is identified as a variation of an existing layout, you have to manually create a new layout variation, increasing time and effort. To learn more, see Adding a Variation to a Layout.

Similar Layout variation detection, introduced as a beta feature in v41.0.0, addresses this by:

Allowing you to create layout variations directly within Document Drift Management.
Providing automated recommendations when a new document closely resembles an existing layout, reducing the need for manual intervention.

Example Use Case

A common example is the U.S. tax form W-8, which undergoes periodic updates while maintaining its overall structure.

Previously, you had to:

Manually detect when a new version is causing unmatched submissions.
Manually create a new layout variation outside the triage experience.

With Similar Layout variation detection:

The system identifies the new version as a potential variation of the existing W-8 form.
You can quickly confirm and apply the variation without manually re-creating fields.

How It Works

Grouping Submissions with unmatched pages

The system groups unmatched pages based on similarity, identifying potential new structured layouts or variations.

Manual Layout Variation Creation

You can manually assign unmatched groups as variations of existing layouts within the triage workflow.

Enabling Similar Layout variation detection

When enabled, the system scans for variations within the current release. It compares unmatched submissions with existing layouts and suggests potential variations.

Reviewing and Accepting Recommendations

You’ll receive a pre-filled layout suggestion if a close match is detected.
The recommendation can be accepted, modified, or rejected.

Using Similar Layout variation detection

When enabled, the Similar Layout variation detection will be available on the Page Groups tab. You’ll see an additional column called Similar Variation that indicates whether a group resembles an existing layout or no similar variations were found.

Search for similar variations by clicking the Re-run similar variations search () button. If a variation is found, the name of the layout and the variation will appear in the Similar variation column.

Limitations

Matching is limited to layout variations that were live in the release when the original submission was processed.
Similar Layout variation detection runs automatically only for newly created groups from the grouping job. You must manually re-run the search for existing groups after enabling the beta feature.

Step 3 - Create Potential Layouts

Once you have triaged your groups, they will be ready for layout creation. Find them in the Potential Layouts tab.

Create a layout based on a group by clicking the Create Layout button ().
Edit the name of your layout, download, archive or delete it from the Actions menu ().
- You can download your layout or blank forms if you need to export them to another instance.
Once you’ve created your layout, click the Open Layout button () to access the editor. To learn how to create a layout see Creating Structured Layouts.

Next Steps

Once you’ve created your layout, add it to a release and assign it to a flow. Learn more in Assigning a Release to a Flow.

The Archived Page Groups tab allows you to download, unarchive, or delete a group.

Download groups from the Actions drop-down or from the button.
Unarchive a group from the Actions drop-down or from the Unarchive Group button ()
Select groups to delete from the archive and click the Delete button located next to the Actions drop-down menu or click the button.

Limitations

This section outlines the known limitations of the Document Drift Management feature to help you better plan and manage your document-processing workflow.

Uploading blanks — When uploading a blank form, only the first page of a multi-page blank PDF will be uploaded. This limitation can pose challenges if your source document consists of multiple pages.
Clustering Job minimum — Clustering works with a minimum of 5 pages per group.
Clustering Job Limit —The clustering job can process a maximum of 10,000 pages at a time. Since groups are cumulative, you can rerun clustering on the same selection set to process additional pages beyond the initial 10,000.
Triage Group Limit — A maximum of 500 groups can be triaged at once. To manage this effectively, it's recommended to focus on smaller batches, typically around 10-50 groups at a time.