Unable to get the rules in the Prometheus UI and unable to add new scrape config

[sabarirjpm]

We want to get the centralized monitoring enabled for multiple ocp clusters. The cluster observability operator that is available, does not expose itself to external and is restricted only to the cluster where it is running. So we thought that we would enable the remote write in the clusterobservability and push it to another prometheus instance.

For the remote prometheus, we chose this observability operator. The installation of the operator had some issues as we were in disconnected environment which we resolved by manually copying over the images needed to the internal repository.

Currently we see that the data is being pushed to the prometheus instance but we are not able to create prometheus, alertmanager rules and also we are not able to create new scrape configs. When we create the rules or configs, it is getting displayed in the backend(oc get prometheusrule) but not in the frontend. The existing default four rules that are shown are also not visible in the frontend.

oc get prometheusrule -n openshift-operators
NAME AGE
observability-operator-alertmanager-rules 4h31m
observability-operator-prometheus-operator-rules 4d22h
observability-operator-prometheus-rules 4d22h
observability-operator-rules 4d22h
[kniobs@bastion1-kvm01 ~]$

We believe that this is related to some config issues. Is there a clear document explaining the steps of the configurations.