Skip to content

Commit f0bd96d

Browse files
committed
Troubleshooting: Document fix of corrupted redis AOF
Signed-off-by: Jan Horstmann <horstmann@osism.tech>
1 parent 83d36d1 commit f0bd96d

5 files changed

Lines changed: 59 additions & 4 deletions

File tree

docs/guides/deploy-guide/services/infrastructure.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ sidebar_position: 10
66
# Infrastructure
77

88
Common issues with deploying infrastructure services required by OpenStack
9-
are documented in the [OpenStack Troubleshooting Guide](../../troubleshooting-guide/openstack.md).
9+
are documented in the [OpenStack Troubleshooting Guide](../../troubleshooting-guide/openstack.mdx).
1010

1111

1212
1. Optional: In order to reduce the active observation time for the deployment of the components,

docs/guides/deploy-guide/services/logging-monitoring.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ sidebar_position: 40
66
# Logging & Monitoring
77

88
Common issues with deploying logging & monitoring services provided by Kolla
9-
are documented in the [OpenStack Troubleshooting Guide](../../troubleshooting-guide/openstack.md).
9+
are documented in the [OpenStack Troubleshooting Guide](../../troubleshooting-guide/openstack.mdx).
1010

1111
1. OpenSearch
1212

docs/guides/deploy-guide/services/openstack.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ sidebar_position: 60
66
# OpenStack
77

88
Common issues with deploying OpenStack services are documented in the
9-
[OpenStack Troubleshooting Guide](../../troubleshooting-guide/openstack.md).
9+
[OpenStack Troubleshooting Guide](../../troubleshooting-guide/openstack.mdx).
1010

1111
:::info
1212

docs/guides/troubleshooting-guide/openstack.md renamed to docs/guides/troubleshooting-guide/openstack.mdx

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22
sidebar_label: OpenStack
33
sidebar_position: 40
44
---
5+
import Tabs from '@theme/Tabs';
6+
import TabItem from '@theme/TabItem';
57

68
# OpenStack
79

@@ -254,3 +256,56 @@ sidebar_position: 40
254256

255257
* Do not disable the offline compression with `COMPRESS_OFFLINE = False` to get rid of the error. The error
256258
disappears, but afterwards every page is compressed again on each request.
259+
260+
## Redis/Valkey: "Bad file format reading the append only file..."
261+
262+
<Tabs>
263+
<TabItem value="redis" label="Redis">
264+
265+
* Problem: A `redis` instance refuses to start with error message: "Bad file format reading the append only file appendonly.aof.11.incr.aof: make a backup of your AOF file, then use ./redis-check-aof --fix &lt;filename.manifest&gt;"
266+
267+
* Solution: Fix the append-only-file with the mentioned command.
268+
269+
:::note
270+
271+
The following commands apply to a redis instance belonging to the OpenStack control plane. Please make sure to adjust the commands if you are fixing this error on another redis instance, like the manager or netbox.
272+
273+
:::
274+
275+
On the node with the broken instance run
276+
277+
```console
278+
docker run --rm -it -v redis:/var/lib/redis <osism redis image>:<osism redis tag> bash
279+
redis-check-aof --fix /var/lib/redis/appendonlydir/redis-staging-ao.aof.manifest
280+
```
281+
282+
and follow the given instructions to fix any corruptions.
283+
Once the append-only-file is fixed restart the `redis` instance.
284+
285+
```console
286+
docker restart redis
287+
```
288+
289+
</TabItem>
290+
<TabItem value="valkey" label="Valkey">
291+
292+
* Problem: A `valkey` instance refuses to start with error message: "Bad file format reading the append only file appendonly.aof.11.incr.aof: make a backup of your AOF file, then use ./valkey-check-aof --fix &lt;filename.manifest&gt;"
293+
294+
* Solution: Fix the append-only-file with the mentioned command.
295+
296+
On the node with the broken instance run
297+
298+
```console
299+
docker run --rm -it -v valkey:/var/lib/valkey <osism valkey image>:<osism valkey tag> bash
300+
valkey-check-aof --fix /var/lib/valkey/appendonlydir/valkey-staging-ao.aof.manifest
301+
```
302+
303+
and follow the given instructions to fix any corruptions.
304+
Once the append-only-file is fixed restart the `valkey` instance.
305+
306+
```console
307+
docker restart valkey
308+
```
309+
310+
</TabItem>
311+
</Tabs>

docs/guides/upgrade-guide/openstack.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,4 +103,4 @@ of the APIs. This downtime is usually less than 1 minute.
103103

104104
If Horizon answers with `Something went wrong!` afterwards, check its log for an `OfflineGenerationError`.
105105
That error and its solution are documented in the
106-
[OpenStack Troubleshooting Guide](../troubleshooting-guide/openstack.md).
106+
[OpenStack Troubleshooting Guide](../troubleshooting-guide/openstack.mdx).

0 commit comments

Comments
 (0)