Issue
In some cases, the database connection may be lost, causing submissions to halt.
The error looks similar to the one shown below.
{
"failure": "Exception",
"stacktrace": "Traceback (most recent call last):
File \"/var/www/venv/lib/python3.7/site-packages/django/db/utils.py\", line 96, in inner
return func(*args, **kwargs)
File \"/var/www/forms/forms/forms/db/backends/mssql/pyodbc/base.py\", line 477, in fetchmany
return list(map(tuple, self.cursor.fetchmany(chunk)))
pyodbc.OperationalError: ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]
Session Provider: Connection has been closed by peer [xFFFFFFFF]. (-1) (SQLGetData)')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File \"/var/www/forms/forms/job_queue/management/commands/worker.py\", line 236, in _process_thread
_process_job(worker, job)
File \"/var/www/venv/lib/python3.7/site-packages/tenacity/__init__.py\", line 292, in wrapped_f
return self.call(f, *args, **kw)
File \"/var/www/venv/lib/python3.7/site-packages/tenacity/__init__.py\", line 358, in call
do = self.iter(retry_state=retry_state)
File \"/var/www/venv/lib/python3.7/site-packages/tenacity/__init__.py\", line 319, in iter
return fut.result()
File \"/opt/hs/python/lib/python3.7/concurrent/futures/_base.py\", line 428, in result
return self.__get_result()
File \"/opt/hs/python/lib/python3.7/concurrent/futures/_base.py\", line 384, in __get_result
raise self._exception
File \"/var/www/venv/lib/python3.7/site-packages/tenacity/__init__.py\", line 361, in call
result = fn(*args, **kwargs)
File \"/var/www/forms/forms/job_queue/management/commands/worker.py\", line 93, in _process_job
worker.process(job)
File \"/var/www/forms/forms/form_extraction/workers/oicr.py\", line 662, in process
return self.process_page(job)
File \"/var/www/forms/forms/form_extraction/workers/oicr.py\", line 419, in process_page
template_binary_image=template_binary_image,
File \"/var/www/forms/forms/form_extraction/workers/oicr.py\", line 178, in _predict_form_field
field, form_image, template_image, fetch_debug_data=fetch_debug_data
File \"/var/www/forms/forms/form_extraction/oicr.py\", line 41, in predict_field
all_bboxes = self.get_all_bboxes(field)
File \"/var/www/forms/forms/form_extraction/oicr.py\", line 77, in get_all_bboxes
return extract_all_bboxes(field.template_field)
File \"/var/www/forms/forms/form_extraction/oicr_utils.py\", line 96, in extract_all_bboxes
for template_field in template_field.template_page.fields.all()
File \"/var/www/venv/lib/python3.7/site-packages/django/db/models/query.py\", line 274, in __iter__
self._fetch_all()
File \"/var/www/venv/lib/python3.7/site-packages/django/db/models/query.py\", line 1244, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File \"/var/www/venv/lib/python3.7/site-packages/django/db/models/query.py\", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File \"/var/www/forms/forms/forms/db/backends/mssql/compiler.py\", line 134, in execute_sql
ret = super().execute_sql(*args, **kwargs)
File \"/var/www/venv/lib/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1173, in execute_sql
return list(result)
File \"/var/www/venv/lib/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1552, in cursor_iter
for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
File \"/var/www/venv/lib/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1552, in
for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
File \"/var/www/venv/lib/python3.7/site-packages/django/db/utils.py\", line 96, in inner
return func(*args, **kwargs)
File \"/var/www/venv/lib/python3.7/site-packages/django/db/utils.py\", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File \"/var/www/venv/lib/python3.7/site-packages/django/db/utils.py\", line 96, in inner
return func(*args, **kwargs)
File \"/var/www/forms/forms/forms/db/backends/mssql/pyodbc/base.py\", line 477, in fetchmany
return list(map(tuple, self.cursor.fetchmany(chunk)))
django.db.utils.OperationalError: ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]
Session Provider: Connection has been closed by peer [xFFFFFFFF]. (-1) (SQLGetData)')"
}Affected versions
All versions
Affected configurations
All configurations
Cause
The cause is not clear.
Solution / Workaround
Check the state of the database. If the database connection appears to be healthy, simply retrying the halted jobs should solve the problem.