[BUG] Exporter stopped working on 9161 port inside and outside the container without crashing the container

[kirxcore]

Description

Exporter stopped working on 9161 port inside and outside the container without crashing the container.

Steps to Reproduce

Don't know exactly how to reproduce the behaviour, but I have 23 databases, and some shared metrics between them. Occasionally they are down due to maintenance, so there are some ORA-12514 in logs. And after some time Exporter stops providing metrics - curl localhost:9161 gives nothing outside the container, and gives "Connection refused" inside the container.

Expected Behavior

Exporter doesn't stop listening 9161 port

Actual Behavior

Exporter doesn't crash, but stops working at 9161 port

Environment

• OS: centos 7
• Oracle Database version: various
• Exporter version: 2.0.4

Suggestion

Maybe config maxOpenConns:10 and maxIdleConns:10 parameters affect the behaviour?

[anders-swanson]

@kirxcore can you include exporter logs when this happens?

[anders-swanson]

Also curious how much memory you have allocated to the container, and if any OOM issues are occurring.

[kirxcore]

There are no system errors in log, no messages at all after ORA-12514, this was the last message in logs. Following my grafana monitoring dashboard, localhost:9161 stopped working in 10min after this message. I'll try to catch exact moment when the issue will occur next time.
About mem allocation - I didn't set up any limits for the container and I didn't notice any of OOMs , but I'll try to get some stats also.

[anders-swanson]

Thanks. Note that the exporter reports some metrics about itself:

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.483415
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 184320
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 22
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 5.431296e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.759769247e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 4.23642693632e+11
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 9.223372036854776e+18

There are also some notes about configuring the go runtime: https://oracle.github.io/oracle-db-appdev-monitoring/docs/advanced/go-runtime

[anders-swanson]

As a workaround until we resolve this issue, I suggest configuring a RESTART_INTERVAL to refresh the process state (same PID)

[kirxcore]

So I've got some logs. Notice: exporter is working in docker swarm node (only 1 node), without any limitations of memory.
Exporter container worked correctly, but when a couple of DBs shut down, Exporter crashed with errors (first_crash.txt). After that, Swarm created new container - and it also crashed (second_crash.txt). After that Swarm did create container again - and it didn't crash, but 9161 port wasn't working (as I described at the beginning of the issue).
I tried to re-deploy exporter stack (twice), but got crashes again (after_restart.txt and after_restart_2.txt).
After third redeploy Exporter started working correctly.
PS: sorry for my bad english =)

after_restart_2.txt
after_restart.txt
second_crash.txt
first_crash.txt

[anders-swanson]

Looking at your logs, this looks similar to #384 , I've implemented some workaround for the _Cfunc_dpiConn_create issue but obviously not enough 😆

As mentioned in #384 , I'd like to provide a build of the exporter using go-ora instead of godror, as it's a thin client and doesn't rely on the unstable cgo execution layer.

[anders-swanson]

Thanks for sharing the additional info, it's quite helpful.

[kirxcore]

Thank you! Waiting for the new build