Skip to content

docs: add debugging guide for Kmesh daemon#1755

Open
aeron-gh wants to merge 1 commit into
kmesh-net:mainfrom
aeron-gh:docs/debugging-guide
Open

docs: add debugging guide for Kmesh daemon#1755
aeron-gh wants to merge 1 commit into
kmesh-net:mainfrom
aeron-gh:docs/debugging-guide

Conversation

@aeron-gh

Copy link
Copy Markdown

Fixes #733

Adds docs/en/kmesh_debugging.md, a guide for debugging a running Kmesh daemon. It documents:

  • The local admin interface on localhost:15200 and how to reach it (kmeshctl or kubectl exec + curl)
  • Viewing daemon logs and changing log levels at runtime, including the bpf logger for eBPF program logs (kmeshctl log)
  • Viewing the xDS config dump and the underlying eBPF map dump (kmeshctl dump, /debug/config_dump/...)
  • Enabling access logs and metrics (kmeshctl monitoring)
  • Authorization (kmeshctl authz), version, readiness, and pprof
  • A quick-reference table of debug commands and endpoints

Special notes for your reviewer

AI assistance: this guide was drafted with the help of an AI tool. As the author I have reviewed it and cross-checked every command and endpoint against the kmeshctl source (ctl/) and the admin server (pkg/status/status_server.go). I have not run the commands against a live cluster, so corrections from a cluster test are very welcome.

Copilot AI review requested due to automatic review settings June 14, 2026 17:51
@kmesh-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hzxuzhonghu for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new documentation file to improve the observability and maintainability of Kmesh. By providing a centralized reference for debugging tools and endpoints, it empowers users to diagnose issues within the Kmesh daemon more effectively without needing to inspect the source code directly.

Highlights

  • New Documentation: Added a comprehensive debugging guide for the Kmesh daemon to assist users in troubleshooting running instances.
  • Debug Interface Coverage: The guide details how to interact with the Kmesh admin interface via kmeshctl and direct HTTP requests, covering logs, configuration dumps, and eBPF map inspection.
  • Operational Guidance: Included instructions for managing access logs, metrics, authorization offloading, and performance profiling using pprof.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.


The daemon runs with silent grace, / But bugs may hide in every space. / With logs and dumps we clear the way, / To keep the Kmesh bright today.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@kmesh-bot

Copy link
Copy Markdown
Collaborator

Welcome @aeron-gh! It looks like this is your first PR to kmesh-net/kmesh 🎉

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a new debugging guide for Kmesh, documenting how to inspect a running kmesh-daemon via kmeshctl, kubectl, and the local admin HTTP endpoints.

Changes:

  • Documented daemon log access and runtime log level adjustment.
  • Added guidance for config dumps, authz debugging, readiness checks, monitoring, and pprof profiling.
  • Included a quick-reference table mapping common tasks to kmeshctl and HTTP endpoints.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread docs/en/kmesh_debugging.md Outdated
Comment thread docs/en/kmesh_debugging.md Outdated
Comment on lines +178 to +180
| Task | `kmeshctl` | HTTP endpoint (`localhost:15200`) |
| --- | --- | --- |
| Daemon logs | `kubectl logs` | — |

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive debugging guide for Kmesh, detailing how to use the admin interface, adjust log levels, view configuration dumps, and monitor traffic. Feedback on the quick reference table points out that several kmeshctl commands are incomplete and would fail validation, and a suggestion has been provided to include placeholders and correct the markdown formatting.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread docs/en/kmesh_debugging.md Outdated
Comment on lines +178 to +189
| Task | `kmeshctl` | HTTP endpoint (`localhost:15200`) |
| --- | --- | --- |
| Daemon logs | `kubectl logs` | — |
| List / set log level | `kmeshctl log` | `/debug/loggers` |
| Config dump | `kmeshctl dump` | `/debug/config_dump/{dual-engine,kernel-native}` |
| eBPF map dump | — | `/debug/config_dump/bpf/{dual-engine,kernel-native}` |
| Access logs | `kmeshctl monitoring --accesslog` | `/accesslog` |
| Monitoring / metrics | `kmeshctl monitoring` | `/monitoring`, `/workload_metrics`, `/connection_metrics` |
| Authorization | `kmeshctl authz` | `/authz` |
| Version | `kmeshctl version` | `/version` |
| Readiness | — | `/debug/ready` |
| Profiling | — | `/debug/pprof/` |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Several commands listed in the quick reference table will fail with CLI parsing/validation errors or do nothing if executed exactly as written:

  1. kmeshctl log requires at least 1 argument (the pod name) and will fail with Error: accepts at least 1 arg(s), received 0.
  2. kmeshctl dump requires exactly 2 arguments (pod name and mode) and will fail with Error: accepts 2 arg(s), received 0.
  3. kmeshctl monitoring requires at least one flag (e.g., --accesslog, --all, etc.) to perform any action.
  4. kmeshctl authz requires a subcommand (enable, disable, or status) to function.

Updating the table to include placeholders (like <pod>, <mode>, and <flags>) and escaping the pipe | characters (to prevent breaking the markdown table rendering) will make the quick reference accurate and user-friendly.

Suggested change
| Task | `kmeshctl` | HTTP endpoint (`localhost:15200`) |
| --- | --- | --- |
| Daemon logs | `kubectl logs` ||
| List / set log level | `kmeshctl log` | `/debug/loggers` |
| Config dump | `kmeshctl dump` | `/debug/config_dump/{dual-engine,kernel-native}` |
| eBPF map dump || `/debug/config_dump/bpf/{dual-engine,kernel-native}` |
| Access logs | `kmeshctl monitoring --accesslog` | `/accesslog` |
| Monitoring / metrics | `kmeshctl monitoring` | `/monitoring`, `/workload_metrics`, `/connection_metrics` |
| Authorization | `kmeshctl authz` | `/authz` |
| Version | `kmeshctl version` | `/version` |
| Readiness || `/debug/ready` |
| Profiling || `/debug/pprof/` |
| Task | `kmeshctl` | HTTP endpoint (`localhost:15200`) |
| --- | --- | --- |
| Daemon logs | `kubectl logs -n kmesh-system <pod>` ||
| List / set log level | `kmeshctl log <pod> [--set <logger>:<level>]` | `/debug/loggers` |
| Config dump | `kmeshctl dump <pod> <mode>` | `/debug/config_dump/{dual-engine,kernel-native}` |
| eBPF map dump || `/debug/config_dump/bpf/{dual-engine,kernel-native}` |
| Access logs | `kmeshctl monitoring [<pod>] --accesslog <enable\|disable>` | `/accesslog` |
| Monitoring / metrics | `kmeshctl monitoring [<pod>] <flags>` | `/monitoring`, `/workload_metrics`, `/connection_metrics` |
| Authorization | `kmeshctl authz <enable\|disable\|status> [<pod>]` | `/authz` |
| Version | `kmeshctl version [<pod>]` | `/version` |
| Readiness || `/debug/ready` |
| Profiling || `/debug/pprof/` |

@aeron-gh aeron-gh requested a review from Copilot June 14, 2026 17:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comment thread docs/en/kmesh_debugging.md Outdated
@aeron-gh aeron-gh force-pushed the docs/debugging-guide branch from 39e8173 to 753422a Compare June 14, 2026 18:01
@aeron-gh aeron-gh force-pushed the docs/debugging-guide branch 2 times, most recently from 60f4dd3 to 548d052 Compare June 14, 2026 18:14
Copilot AI review requested due to automatic review settings June 14, 2026 18:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

Comment thread docs/en/kmesh_debugging.md
Comment thread docs/en/kmesh_debugging.md
Comment thread docs/en/kmesh_debugging.md Outdated
Comment on lines +180 to +182
| Daemon logs | `kubectl logs -n kmesh-system <pod>` | — |
| List / set log level | `kmeshctl log <pod> [--set <logger>:<level>]` | `/debug/loggers` |
| Config dump | `kmeshctl dump <pod> <mode>` | `/debug/config_dump/{dual-engine,kernel-native}` |
Document how to debug a running kmesh-daemon: the localhost:15200 admin interface, viewing logs and changing log levels (incl. the bpf logger), the config dump (and eBPF map dump), access logs/metrics, authorization, version, readiness and pprof.

Signed-off-by: aeron-gh <agab0323@gmail.com>
@aeron-gh aeron-gh force-pushed the docs/debugging-guide branch from 548d052 to b925314 Compare June 14, 2026 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[doc] add doc about debugging Kmesh

3 participants