Explosive Logs

[awwad]

If the SQL server is not running, the clearinghouse's check_active_db_nodes log explodes. I'm currently looking at a 335 GB check_active_db_nodes.log file....

[2016-05-27 12:01:27.467222] CRITICAL 526751924 Unexpected exception in check_active_db_nodes.py: Traceback (most recent call last):
  File "/home/ch/deployment/clearinghouse/polling/check_active_db_nodes.py", line 104, in main
  File "/home/ch/deployment/clearinghouse/common/api/maindb.py", line 2198, in get_active_nodes_include_broken
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 162, in __iter__
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 965, in _fetch_all
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 238, in iterator
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 79, in execute
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
  File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 97, in __exit__
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 124, in execute
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
OperationalError: (2006, 'MySQL server has gone away')

[2016-05-27 12:01:27.467593] CRITICAL 526751924 Unexpected exception in check_active_db_nodes.py: Traceback (most recent call last):
  File "/home/ch/deployment/clearinghouse/polling/check_active_db_nodes.py", line 104, in main
  File "/home/ch/deployment/clearinghouse/common/api/maindb.py", line 2198, in get_active_nodes_include_broken
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 162, in __iter__
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 965, in _fetch_all
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 238, in iterator
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 79, in execute
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
  File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 97, in __exit__
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 124, in execute
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
OperationalError: (2006, 'MySQL server has gone away')

[aaaaalbert]

Oh, wow. Do you have DEBUG = True in your settings.py? (In other words, is this a problem that might bite production deployments too?)

[awwad]

Two different questions. Yes, it's set to DEBUG = True.

Here's what would happen if it weren't.

clearinghouse/polling/check_active_db_nodes.py

Lines 125 to 136 in 22a4c47

	except:
	message = "Unexpected exception in check_active_db_nodes.py: " + traceback.format_exc()
	log.critical(message)
	# Send an email to the addresses listed in settings.ADMINS
	if not settings.DEBUG:
	subject = "Critical SeattleGeni check_active_db_nodes.py error"
	django.core.mail.mail_admins(subject, message)
	# Sleep for ten minutes to make sure we don't flood the admins with error
	# report emails.
	time.sleep(600)

So you make a good point. Production is probably okay. (It'd just be emails every ten minutes and a logging rate at some infinitesimal level of the debug logging rate.)

Anyway, it's probably best to have a short wait there even if DEBUG is true. No?

[aaaaalbert]

Thanks for your diligent answer! I think we can do without a wait, but we'd need to make sure that the poor debug CH operator sees the error. This could be a documentation issue, e.g. "Be sure to watch this logfile to see if anything goes wrong".

Yet, running in debug mode for prolonged periods of time is just asking for trouble. (No offense meant :-)