Excessive storage usage by Docker

  • Published on Sep 24, 2024
  • 3 minute(s) read
Prev Next

Issue

Docker uses an excessive amount of storage while Hyperscience is running. This issue can be confirmed by running the following command:

docker system df -v

The output is similar to the example shown below.

Images space usage:

...

Containers space usage:

CONTAINER ID ...   SIZE   ...    NAMES
8aaa10e4660d ...   4.8GB  ...    hsvarbe_field_locator_1
5673349a3c95 ...   4.82GB ...    hsvarbe_nlc_1
a357bcd21708 ...   4.8GB  ...    hsvarbe_segmentation_1
0c184996c92d ...   4.66GB ...    hsbe_health_check_1
1780ff45abf4 ...   4.78GB ...    hsbe_idp_sync_block_1
327a41ce375f ...   4.78GB ...    hsbe_cron_1
e3ae08f60521 ...   4.81GB ...    hsbe_forms_oicr_1
d03a337b99ec ...   4.81GB ...    hsbe_forms_oicr_2
df50df4b8c95 ...   4.8GB  ...    hsbe_registration_1
5309d94614e8 ...   4.79GB ...    hsbe_hyperflow_engine_1
66b5b5fa6436 ...   4.83GB ...    hsbe_shell_command_1
9a4df9ffa25d ...   4.73GB ...    hsbe_slow_funcall_1
b5e0225265fb ...   6.24GB ...    hsbe_process_manager_1
86e626ec7de8 ...   4.8GB  ...    hsbe_vpc_1
b85d37efe687 ...   4.47GB ...    hsbe_funcall_1
84c3fbe4e7a5 ...   4.52GB ...    hstelemetry_filebeat_1
c75e2c3da960 ...   4.78GB ...    hsfe_nginx_1
28c1cbe909df ...   4.33GB ...    hsfe_gunicorn_1

Besides the row labeled hsbe_process_manager_1, which may be more than 6GB under normal circumstances, no other rows should list a size larger than 100MB. If some or all of the other rows have a size larger than 100MB, this issue may be affecting your system. 

This issue may be limited to one container, but it is much much more likely to affect multiple containers.

Affected versions

All versions

Affected configurations

Systems with software that conducts full scans of the entire file system, including any directories used by Docker. Examples include any antivirus software or file-system consistency or integrity checks, whether manually initiated or completed on a scheduled basis.

Fixed versions

Not applicable. This issue is external to Hyperscience, so Hyperscience cannot fix it.

Cause

  • The affecting software looks at every file on the file system. Among those files are Docker container-related files. Each Docker container created from a single Docker image generally shares all files within that image. Thus, a 5GB docker image with 10 containers generally will only take up about 5GB of storage, and every container will share the same 5GB set of files.

  • When a container file's "modified" time is updated, Docker creates a copy of the file specific to the container. It no longer shares the same file as the original image, nor is the new file shared with all other containers. On Linux, a file's "modified" time can be updated in various ways without changing the actual contents of the file. Software that scans file systems can make these kinds of updates, depending on how the software works internally.

  • When such scanning happens, and if the software updates the "modified" time of all files shared by the Hyperscience containers, each container makes its own copy of all files in the Docker image. This process results in the storage size being multiplied by the number of Docker containers. Hyperscience runs more than 15 containers, depending on the application version and configuration, and the Hyperscience image is normally around 5GB. Under normal circumstances, Hyperscience uses approximately 5GB of storage for most containers, along with an additional 6-10GB for the hsbe_process_manager_1 container. If files are modified, each container whose files are modified will use an additional 5GB of storage. Thus, storage size used by Hyperscience grows from a minimum of 11GB (5 GB + 6 GB = 11GB) to 81GB (5GB * 15 + 6GB = 81GB) of storage.

Solutions / Workarounds

If possible, we recommend omitting the Docker root directory from such scans to avoid changing files' "modified" timestamps. The Docker root directory can be found by running the following command:

docker info

Its output is similar to the example shown below.

Client:
 Debug Mode: false

Server:
...
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.91GiB
 Name: wns-demo01
 ID: NDEM:KPVT:22OI:Y6FU:5W5Y:WRDR:NECR:USTT:SWE2:JTMS:J4TA:G3CU
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
...

Alternatively, provide additional storage for each affected machine. The number of containers can be found by running the following command:

docker ps | wc -l

To find the minimum recommended storage size for your Docker root directory, use the following formula:

Minimum recommended storage = 6GB * [number of containers] + 20GB

Logs to provide to Hyperscience

You do not need to provide logs to Hyperscience unless no full file system scanning has taken place and similar symptoms are observed.