Upgrading to v40 (with submissions being created during the upgrade process)

(Requires application, flows, and model updates, but allows creating and processing new submissions between application-version upgrades)

This article explains how to upgrade to v40 without the need to stop new submissions from being created between application-version upgrades.

The steps described below ensure that no new submissions will be interrupted during the upgrade process. However, this low-downtime approach involves more steps, as alongside your application versions, you'll also need to update flows and retrain models every two major versions (in accordance with the "N-2 compatibility" rule).

If you can temporarily stop new submissions from being created during the upgrade process, we recommend following the process described in Upgrading to v40 (without submissions being created during the upgrade process) instead.

What’s changed in v40

  • Enforcement of license packages — Depending on your license package and pricing plan, the features you had access to in previous versions may differ from those you have access to in v40. For more information, see License Packages and Feature Availability.

  • Ubuntu 16 and RHEL 7 not supported — Beginning in v40, we no longer support the use of Ubuntu 16 or RHEL 7 with Hyperscience. If you are using either of these operating systems, you will need to upgrade to a supported version of Ubuntu or RHEL. For a list of supported operating systems, see Infrastructure Requirements.

  • PostgreSQL 12 not supported — The use of PostgreSQL 12 is not supported in v40 and later. If you are using PostgreSQL 12, you will need to upgrade your database to a currently supported version of PostgreSQL. For a list of supported versions, see Infrastructure Requirements.

  • Document Drift Management — In v40, we’ve introduced ways to help you manage and classify unmatched pages more effectively. These updates bring significant changes to the user interfaces for finding potential layouts and viewing unmatched pages. To learn more, see Document Drift Management (Layout Triage).

For a complete list of updates included in v40, see the V40 Release Notes.

Upgrading from v35 or earlier

1. Upgrade to v35.

If you are using v34 or earlier, upgrade to v35. See Upgrading to v35 for instructions for your current version. 

2. Update flows on v35.

In v35, upgrade any custom flows to use v35 blocks, if you haven’t already. If you’re using “Document Processing” flows from v34 or earlier, migrate to “Document Processing V35.”

3. Retrain all of your models.

4. Update Python packages to 3.9.

If you've installed external Python packages, update them to versions that use Python 3.9.

To learn how to update packages, see Developing Flows.

5. Complete submissions

Using "Document Processing V35” or a custom flow upgraded to v35, complete all submissions created in v35 or earlier.

Do not use flows or models from v34 or earlier to process these submissions. These flows and models will not work in v35 or later. If you try to process submissions with them, the submissions will halt.

There is no filter in the application to determine which submissions were created in which versions, but you can look for older submissions by using the submission date and the dates in your upgrade history as a guide. For example, each submission created in v34 or earlier has a Submission Date that predates your upgrade to v35. 

Make sure all submissions have a Completed status before proceeding with the upgrade steps.

6. Install v36 on your instance.

7. Install v37 on your instance.

8. Update flows on v37.

In v37, upgrade any custom flows to use v37 blocks. If you’re using “Document Processing V35,” migrate to “Document Processing V37.”

9. Retrain all of your models.

10. Complete submissions

Using “Document Processing V37” or a custom flow upgraded to v37, complete all submissions created in v37 or earlier.

Make sure all submissions have a Completed status before proceeding with the upgrade steps.

11. Install v38 on your instance.

12. Install v39 on your instance.

13. Update flows on v39.

In v38 and later, a single top-level flow called “Document Processing” is available. Within its settings, you can choose the correct Document Processing Subflow in the Flow Identifier drop-down list under "Block Details."

For more information, see the System-generated flows section below.

14. Retrain all of your models.

15. Complete submissions.

Using “Document Processing V39” or a custom flow upgraded to v39, complete all submissions created in v39 earlier.

Make sure all submissions have a Completed status before proceeding with the upgrade steps.

16. Install v40 on your instance.

17. Update flows on v40.

In v40, upgrade any custom flows to use v40 blocks. To do so, go to the flow settings for the "Document Processing" top-level flow, and click on Document Processing Subflow V40 in the Flow Identifier drop-down list under "Block Details."

For more information, see the System-generated flows section below.

18. Retrain all of your models.

This retraining includes Transcription, Classification, and Identification models.

Upgrading from v36

1. Update flows to v36.

Upgrade any custom flows to use v36 blocks, if you haven’t already. If you’re using “Document Processing” flows from v35 or earlier, migrate to “Document Processing V36.”

Flows from v36 or earlier will not work in v39.

2. Retrain all of your models.

Retrain any models for flows created in v36 or earlier.

3. Complete submissions.

Using “Document Processing V36” or a custom flow upgraded to v36, complete all submissions.

There is no filter in the application to determine which submissions were created in which versions, but you can look for older submissions by using the submission date and the dates in your upgrade history as a guide. For example, each submission created in v35 or earlier has a Submission Date that predates your upgrade to v36. 

Make sure all submissions have a Completed status before proceeding with the upgrade steps.

4. Install v37 on your instance.

5. Install v38 on your instance.

6. Update flows on v38.

In v38 and later, a single top-level flow called “Document Processing” is available. Within its settings, you can choose the correct Document Processing Subflow in the Flow Identifier drop-down list under "Block Details."

For more information, see the System-generated flows section below.

7. Retrain all of your models.

8. Complete submissions

Using “Document Processing V38” or a custom flow upgraded to v38, complete all submissions created in v38 or earlier.

Make sure all submissions have a Completed status before proceeding with the upgrade steps.

9. Install v39 on your instance.

10. Install v40 on your instance.

11. Update flows on v40.

In v40, upgrade any custom flows to use v40 blocks. To do so, go to the flow settings for the "Document Processing" top-level flow, and click on Document Processing Subflow V40 in the Flow Identifier drop-down list under "Block Details."

For more information, see the System-generated flows section below.

12. Retrain all of your models.

This retraining includes Transcription, Classification, and Identification models.

Upgrading from v37

1. Update flows to v37.

Upgrade any custom flows to use v37 blocks, if you haven’t already. If you’re using “Document Processing” flows from v36 or earlier, migrate to “Document Processing V37.”

2. Retrain all of your models.

Retrain any models for flows created in v36 or earlier.

3. Complete submissions.

Using “Document Processing V37” or a custom flow upgraded to v37, complete all submissions.

There is no filter in the application to determine which submissions were created in which versions, but you can look for older submissions by using the submission date and the dates in your upgrade history as a guide. For example, each submission created in v36 or earlier has a Submission Date that predates your upgrade to v37. 

Make sure all submissions have a Completed status before proceeding with the upgrade steps.

4. Install v38 on your instance.

5. Install v39 on your instance.

6. Update flows on v39.

In v38 and later, a single top-level flow called “Document Processing” is available. Within its settings, you can choose the correct Document Processing Subflow in the Flow Identifier drop-down list under "Block Details."

For more information, see the System-generated flows section below.

7. Retrain all of your models.

8. Complete submissions.

Using “Document Processing V39” or a custom flow upgraded to v39, complete all submissions created after the v39 upgrade or earlier.

Make sure all submissions have a Completed status before proceeding with the upgrade steps.

9. Install v40 on your instance.

10. Update flows on v40.

In v40, upgrade any custom flows to use v40 blocks. To do so, go to the flow settings for the "Document Processing" top-level flow, and click on Document Processing Subflow V40 in the Flow Identifier drop-down list under "Block Details."

For more information, see the System-generated flows section below.

11. Retrain all of your models.

This training includes Transcription, Classification, and Identification models.

Upgrading from v38

1. Update flows to v38.

Upgrade any custom flows to use v38 blocks, if you haven’t already. If you’re using “Document Processing” flows from v37 or earlier, migrate to “Document Processing V38.”

Flows from v37 or earlier will not work in v40.

2. Retrain all of your models.

Retrain any models for flows created in v37 or earlier.

3. Complete submissions.

Using “Document Processing V38” or a custom flow upgraded to v38, complete all submissions.

There is no filter in the application to determine which submissions were created in which versions, but you can look for older submissions by using the submission date and the dates in your upgrade history as a guide. For example, each submission created in v37 or earlier has a Submission Date that predates your upgrade to v38.

Make sure all submissions have a Completed status before proceeding with the upgrade steps.

4. Install v39 on your instance.

5. Install v40 on your instance.

6. Update flows on v40.

In v40, upgrade any custom flows to use v40 blocks. To do so, go to the flow settings for the "Document Processing" top-level flow, and click on Document Processing Subflow V40 in the Flow Identifier drop-down list under "Block Details."

For more information, see the System-generated flows section below.

7. Retrain all of your models.

This retaining includes Transcription, Classification, and Identification models.

Upgrading from v39

Complete the upgrade process by deploying v40 on all machines running the application. You also need to install a v40 trainer after upgrading the application and update your flows to v40.

For more information, see the System-generated flows section below.

What to expect after upgrading

After upgrading to v40, you may notice some changes in your Flow Library. There may also be some temporary slowness in processing Structured documents, as described in the Machine Classification section below. 

Machine Classification

After upgrading, your first submissions with Structured documents might take up to a few hours to complete. The Machine Classification Block uses pre-computed data to classify Structured documents. After upgrading, this precomputed data is invalidated. The system regenerates this data the first time a submission goes through Machine Classification after upgrading. 

If processing submissions with Structured documents takes longer than expected, you should check the logs from the Activity Log section of the Submission Output page. Verify that the Machine Classification task is the one that takes more time than expected.

Flows

When upgrading from a previous version of Hyperscience, the system will automatically move your existing flows to v40 and add new flows included in v40. No changes will be made to your pre-v40 flows as part of this process. Submissions will continue to be processed through your default flow (likely "Document Processing V37," or “Document Processing” in v38 and later). 

Any models, flows, and Custom Code Blocks created in v38 or v39 will continue to work in v40. You can retrain models created in v38 or v39 in v40 without upgrading their flows, as long as those flows were created in v38, v39, or v40. To learn more, see Compatibility Across Application, Flow, and Model Versions. However, flows created in v30-37, including the pre-built "Document Processing" and "Notifications" flows, will not work in v40, and we do not support the use of these flows. 

When migrating your processing to v40’s “Document Processing” flow, we recommend testing the migration in a lower environment (e.g., development, UTA) first. After this testing, you can replicate the migration in your production environment.

To migrate your processing to v40’s “Document Processing” flow:

  1. Go to your lower environment.

  2. Duplicate the newly-created “Document Processing” flow.

  3. Configure the duplicated flow with the settings and notifications of the “Document Processing” flow you are using to process submissions in your production environment.

  4. Re-train any Identification, Classification, and Transcription models associated with the previous version's flows by clicking the Run training button on the model details page for each model. Doing so ensures that those models are trained on the latest version of the application and are compatible with the new version's flows.

  5. Assign a release to the duplicated “Document Processing” flow by following the steps in Assigning a Release to a Flow.

  6. Deploy the duplicated “Document Processing” flow by following the steps in Managing Flows

  7. To test the duplicated “Document Processing” flow, manually upload and process a few submissions through this flow.

  8. If you use an integration to upload submissions, change your integration’s target flow UUID to the UUID of the duplicated “Document Processing” flow. You can copy the UUID of your duplicated “Document Processing” flow by clicking the Copy link at the bottom of the Flow Settings sidebar on the left-hand side of the Flow Studio.

  9. Disable the old “Document Processing” flow by following the steps in Managing Flows.

  10. Repeat steps 2-7 in your production environment, or export the newly created flows and models from your lower environment and import them to your higher environment.

If you are using a “Document Processing Notifications” flow or a “Submission State Notification” flow from a previous version, set up v40’s “Submission State Notifications” flow to meet your needs.

System-generated flows

When you upgrade to v40, a new document-processing flow appears alongside the flows that were in your previous version of Hyperscience. New state-notification flow and on-error flows are also included, but, unlike these flows from previous versions, they are not versioned in v40.

Document Processing 

The system-generated flows that appear in your v40 instance depend on which versions you’ve previously used or upgraded to. 

Upon upgrading to v40, a new "Document Processing" top-level flow will be added and will be disabled. This flow comes with three new subflows. One of these subflows is named “Document Processing Subflow V40”, which is an updated version of the "Document Processing" flows included in previous versions of Hyperscience and contains features introduced in v40. You can work with your Hyperscience representative to determine if this updated subflow would be beneficial for your business. 

If you used the “Document Processing” top-level flow from v38 or v39 with “Document Processing Subflow V38” or “Document Processing Subflow V39,”, you can replace the subflow with “Document Processing Subflow V40.” If you do so, the input and output connections you configured in v38 or v39 can be used as-is. 

To replace “Document Processing Subflow V38” or “Document Processing Subflow V39” with “Document Processing Subflow V40” in the “Document Processing” top-level flow from v38 or v39:

  1. On the Flows page (Flows > Flows), find the “Document Processing” top-level flow, and click on its name.

  2. In the table under the flow diagram, click Document Processing.

  3. On the canvas, click the Start Document Processing Subflow Block.

  4. In the Flow Identifier drop-down list in the right-hand panel, click Document Processing Subflow V40.

    • You can complete this step regardless of which option is selected in the Settings Type drop-down list.

  5. Click Save in the upper-right corner of the page.

If you decide to upgrade your “Document Processing” flow to use the latest flow blows and features, but you did not use the top-level “Document Processing” flow in v38 or v39, you will need to configure the Input Blocks and Output Blocks in that top-level “Document Processing” flow. You must also copy your settings from the “Document Processing” flow from your previous version of the Hyperscience application to “Document Processing Subflow V40.”   

Document Processing Notifications 

The system will also move your "Document Processing Notifications" and “Submission State Notifications” flows to v40, which contain Notification Blocks that execute mid-flow. Each of these flows remains unchanged during the upgrade. In addition, a new “Submission State Notifications” flow is created during the upgrade process. Contact your Hyperscience representative if you need assistance enabling or disabling connections in these flows after upgrading to v40.

In the “Document Processing” top-level flow, you can select a “Submission State Notifications” subflow for each block in the “Document Processing Subflow.” To use v40’s “Submission State Notifications” flow:

  1. On the Flows page (Flows > Flows), find the “Document Processing“ flow, and click on its name.

  2. In the table under the flow diagram, click Document Processing.

  3. On the canvas, click the Start Document Processing Subflow Block.

  4. In the Settings Type drop-down list in the right-hand panel, click on the name of the type of setting you would like to edit.

    The settings of the type you selected are shown. They consist of the flow- and block-level settings found in pre-v38 versions of the “Document Processing” flow, along with new settings introduced in v39 and v40.

  5. In the [Block Name] Notification Flow drop-down list, click Submission State Notifications.

  6. Repeat steps 4 and 5 as needed to link “Submission State Notifications” to other blocks.

  7. Click Save in the upper-right corner of the page.

On-Error Flow

An on-error flow sends notifications when a flow exhausts all of its retry attempts, making your organization aware of halted submissions and allowing for timely troubleshooting. An “On-Error with included Submission data” flow is included in v40. For more information about on-error flows and how to assign them to top-level flows and subflows, see On-Error Flows