Skip to content

JSONDecodeError in Radiometer agent during midnight file rotation #915

@BrianJKoopman

Description

@BrianJKoopman

This weekend we hit this twice:

2025-08-09 00:00:40.913 2025-08-09T00:00:40+0000 acq:0 CRASH: [Failure instance: Traceback: <class 'requests.exceptions.JSONDecodeError'>: Expecting value: line 1 column 9 (char 8)
2025-08-09 00:00:40.913 /opt/venv/lib/python3.10/site-packages/twisted/python/threadpool.py:269:inContext
2025-08-09 00:00:40.913 /opt/venv/lib/python3.10/site-packages/twisted/python/threadpool.py:285:<lambda>
2025-08-09 00:00:40.913 /opt/venv/lib/python3.10/site-packages/twisted/python/context.py:117:callWithContext
2025-08-09 00:00:40.913 /opt/venv/lib/python3.10/site-packages/twisted/python/context.py:82:callWithContext
2025-08-09 00:00:40.913 /opt/venv/lib/python3.10/site-packages/ocs/ocs_agent.py:984:_running_wrapper
2025-08-09 00:00:40.913 /opt/venv/lib/python3.10/site-packages/socs/agents/ucsc_radiometer/agent.py:83:acq
2025-08-09 00:00:40.913 /opt/venv/lib/python3.10/site-packages/requests/models.py:978:json
2025-08-09 00:00:40.913 ]
2025-08-09 00:00:40.913 2025-08-09T00:00:40+0000 acq:0 Status is now "done".
2025-08-09 00:08:10.399 2025-08-09T00:08:10+0000 startup-op: launching acq
2025-08-09 00:08:10.399 2025-08-09T00:08:10+0000 start called for acq
2025-08-09 00:08:10.399 2025-08-09T00:08:10+0000 acq:0 Status is now "starting".
2025-08-09 00:08:10.409 2025-08-09T00:08:10+0000 acq:0 Status is now "running".
2025-08-10 00:00:39.588 2025-08-10T00:00:39+0000 acq:0 CRASH: [Failure instance: Traceback: <class 'requests.exceptions.JSONDecodeError'>: Expecting value: line 1 column 9 (char 8)
2025-08-10 00:00:39.588 /opt/venv/lib/python3.10/site-packages/twisted/python/threadpool.py:269:inContext
2025-08-10 00:00:39.588 /opt/venv/lib/python3.10/site-packages/twisted/python/threadpool.py:285:<lambda>
2025-08-10 00:00:39.588 /opt/venv/lib/python3.10/site-packages/twisted/python/context.py:117:callWithContext
2025-08-10 00:00:39.588 /opt/venv/lib/python3.10/site-packages/twisted/python/context.py:82:callWithContext
2025-08-10 00:00:39.588 /opt/venv/lib/python3.10/site-packages/ocs/ocs_agent.py:984:_running_wrapper
2025-08-10 00:00:39.588 /opt/venv/lib/python3.10/site-packages/socs/agents/ucsc_radiometer/agent.py:83:acq
2025-08-10 00:00:39.588 /opt/venv/lib/python3.10/site-packages/requests/models.py:978:json
2025-08-10 00:00:39.588 ]
2025-08-10 00:00:39.588 2025-08-10T00:00:39+0000 acq:0 Status is now "done".

It occurs right at midnight. I think we're probably just getting some garbage response when the log files that the web server we query are rotating. We'll need some additional error handling here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions