feat(doc): [000000000] online catalog fetch with fallback; extend API Reference to data sources and actions by iuin · Pull Request #4557 · tencentcloudstack/terraform-provider-tencentcloud

iuin · 2026-10-14T03:22:38Z

Related Issue

N/A

Summary of Changes

This PR adds an API Reference section to generated resource documentation, listing the cloud APIs invoked by each lifecycle phase (Create/Read/Update/Delete) together with links to the official API docs. It removes the need to read source code to find out which APIs a resource uses.

This PR supersedes #4555 and covers three increments:

1. API Reference rendering for resources

Example (tencentcloud_vpc):

## API Reference

This resource interacts with the following TencentCloud APIs:

* Create: [CreateVpc](https://cloud.tencent.com/document/api/215/15774) - This API is used to create a VPC instance.
* Read: [DescribeVpcs](https://cloud.tencent.com/document/api/215/15778) - This API (DescribeVpcs) is used to query the VPC list.
* Update: [ModifyVpcAttribute](https://cloud.tencent.com/document/api/215/15773) - This API (ModifyVpcAttribute) is used to modify VPC attributes.
* Delete: [DeleteVpc](https://cloud.tencent.com/document/api/215/15775) - This API (DeleteVpc) is used to delete VPCs.

How it works (gendoc/apiref.go):

Static analysis (go/ast): traces resource lifecycle functions -> service
method calls (local variable type inference incl. NewXxxService(...)
constructors, recursive expansion, goroutine delegates, package-level
helpers, locally held clients) -> UseXxxClient().Action(...) calls.
The client accessor is resolved to a catalog product via the SDK import
paths in connectivity/client.go.
Bundled snapshots (gendoc/data/): the public API catalog provides
(product, action) -> doc URL, and the intl SDK
(tencentcloud-sdk-go-intl-en)
doc comments provide the English descriptions.
Manual override: an optional API section in the resource .md (between
the prose and Example Usage) replaces the auto-extracted mapping for
cases the analysis cannot cover.

Same-name disambiguation: 43 method names are declared on service structs
across different products. The analyzer prefers a receiver-type match and
skips ambiguous call sites instead of guessing, so no wrong-product links
are emitted (verified against the full resource scan).

2. API Reference extended to data sources, actions and ephemeral resources

Reference type	Traced lifecycle	Lead-in
resource	Create / Read / Update / Delete	This resource interacts with ...
data source	Read	This data source queries ...
action (framework)	Invoke	This action invokes ...
ephemeral resource (framework)	Open	This ephemeral resource reads ...

3. English descriptions; no Chinese in generated docs

Descriptions come from the intl SDK method comments. Rendering precedence:
English description + doc URL (80.0%) > URL only (20.0%) > bare action
name. The Chinese catalog names are never rendered — verified 0 Chinese
characters across all 7425 rendered entries.

Snapshot refresh

make doc-refresh        # refresh all bundled snapshots (~11MB download)
make doc                # regenerate docs offline from the snapshots

make doc is fully offline and deterministic: it reads only the bundled
snapshots and performs no network requests, so CI's make doc +
git diff website/ check is unaffected by network conditions. Snapshots
are updated explicitly via make doc-refresh (atomic writes; the previous
snapshot survives any failure).

Coverage

resources: 1302/1360 (95.7%)
data sources: 923/949 (97.3%)
actions: 5/5, ephemeral: 1/1

Remaining entries are S3-compatible REST APIs (cos/ci), retired products
(dayu), APIs missing from the catalog snapshot (wedata/teo/tke 2022+), or
same-name ambiguous call sites skipped by design — all coverable via the
manual API section.

Type of Change

Does this change introduce a breaking change?

No
Yes

No changes to provider runtime behavior or resource schemas. make doc
works fully offline; the only external dependency is the explicit
make doc-refresh command.

Checklist

Read CONTRIBUTING.md
Ran make fmt (gofmt on gendoc/ only)
Added unit tests (rendering, action-name filters, description
extraction, empty-verb-list regression guard — 8 tests)
go build / go vet / gofmt -l clean; full doc run: 2259 files,
0 empty sections, 0 Chinese, idempotent double-run

Additional commit: prefer international-site doc URLs

Links in the API Reference section now point to the English
documentation site (www.tencentcloud.com/document/product/...) when
the action is listed there; Chinese-site URLs remain only as a fallback
for actions not yet translated (newer APIs).

Background: the two sites use different document id systems (e.g.
CVM DescribeInstances is 213/15728 on the Chinese site but
213/33258 internationally), so URLs cannot be rewritten by pattern.

make doc-refresh now additionally walks the doc site's internal
endpoints (getCategoryByLang / getCatalogue) to build the
(product, action) -> international URL mapping and merges it into the
en snapshot with priority over Chinese URLs. Note: the endpoints
fingerprint TLS and reject Go's HTTP stack, so the fetches shell out to
curl (a warning with a clear message is printed if curl is missing).

Rendered result: 3531 entries (47.5%) link to the international site,
3900 (52.5%) fall back to the Chinese site, 0 without a link.
International URLs were spot-checked and resolve to the matching action
pages.

Additional commit: description extraction & cleanups (review round 2)

Fixes verified from the second review round:

P3 cleanDesc truncation (worse than reported): the byte-based cut
split multi-byte characters and left 57 unbalanced parentheses in the
snapshot / 39 in rendered docs (broken markdown links). cleanDesc now
measures in runes, prefers sentence boundaries, and only accepts cuts
where ()/[] stay paired — otherwise the description is dropped.

P1/P3 unclosed '[' fragments (teo entries): root cause is the intl
SDK wrapping long descriptions across comment lines with a markdown
link straddling the wrap ([ModifyL4Proxy + ](url)). The extractor
now merges consecutive comment lines until brackets balance and
collapses the wrap artifact ([Foo ](url) -> [Foo](url)). Snapshot
and rendered docs: 0 unbalanced parentheses/brackets.

P2 cleanups: removed dead intlGetJSON + unused intlCatByURL;
removed the never-read catalog.source field (and its stale comment);
removed debug-only loadSnapshotCatalogEntries; replaced the
misleading "replaced Chinese URLs" counter; probeAllResource now
validates the repo root like probeResource; documented the
serviceCtors name-uniqueness assumption (84 constructors, 0 collisions).

P1 semantic attribution (postgresql -> redis 239, thpc -> cbs 362):
kept as-is intentionally — the static analysis faithfully reflects the
runtime call chain (those resources genuinely borrow the redis/cbs
clients, and the redis API's product parameter covers postgres
instances). The manual API section remains the escape hatch if
re-attribution is wanted.

Resource documentation now contains an '## API Reference' section listing the cloud APIs invoked by each lifecycle phase (Create/Read/Update/Delete) with links to the official API docs. How it works (gendoc/apiref.go): - Static analysis (go/ast): traces resource lifecycle functions -> service method calls (local variable type inference, recursive expansion, goroutine delegates, package-level helpers, locally held clients) -> UseXxxClient().Action(...) invocations. The client accessor name is resolved to a catalog product via the SDK import paths declared in connectivity/client.go. - Catalog snapshot (gendoc/data/api-actions.json): slim snapshot of TencentCloud's public API catalog providing (product, action) -> {Chinese name, doc URL} lookup. - Manual override: an optional 'API' section in the resource .md (between prose and Example Usage) replaces the auto-extracted mapping for resources the analysis cannot cover (S3-compatible REST APIs, actions absent from the catalog). Pipelines: - docTPL gains a conditional API Reference section (resource docs only; data sources unchanged). - SDKv2 pipeline locates CRUD functions via runtime function pointers; the framework pipeline locates Create/Read/Update/Delete methods by receiver type on the resource struct. Coverage: 1302/1360 resources (95.7%) map to catalog actions; the remaining 58 are S3-compatible REST APIs (cos/ci), retired products (dayu) and actions missing from the catalog snapshot (wedata/teo/ tke 2022+ APIs) - all coverable via the manual API section. Also adds a probe entry for verification (go run ./gendoc -probe=<short-name> or -probe=all) and unit tests for the rendering and name-filter helpers.

…shot Adds a -refresh-data mode to gendoc that downloads the latest public API catalogs from TencentCloud and rewrites the slim snapshots under gendoc/data/ (product/action/name/url only, atomic rename so the old snapshot survives a failed download). Workflow when the catalog data is updated upstream: make doc-refresh-data # refresh gendoc/data/*.json make doc # regenerate docs with the new data The refresh was verified against the live endpoints: the action count grew from 14096 to 14139 and the diff reflects real catalog changes.

Aligns the generated docs with the gendoc/data snapshot updated in the previous commit (14096 -> 14139 actions).

- parse -refresh-data and -probe in one flag.FlagSet so passing an unknown flag doesn't print spurious usage noise from a second set - remove unused dumpJSON helper and resourceAPIRefs type - update probe entry doc comment

…rence to data sources, actions and ephemeral resources Online-first catalog loading (gendoc/catalog.go): - every 'make doc' run fetches the latest public API catalog (10s timeout); on failure it falls back to the bundled go:embed snapshot with a warning, so doc generation never hard-fails offline and only the API Reference content may be slightly stale. - -refresh-data unchanged, now sharing the fetch helpers. Reference types extended: - data sources (SDKv2): trace the Read function; rendered as 'This data source queries the following TencentCloud APIs'. - framework actions: trace Invoke; 'This action invokes ...'. - framework ephemeral resources: trace Open; 'This ephemeral resource reads ...'. Coverage: data sources 923/949 (97.3%), actions 5/5, ephemeral 1/1. The lead-in sentence and verb set vary per reference type.

…e from generated docs The API Reference section now uses English descriptions sourced from the official international SDK (TencentCloud/tencentcloud-sdk-go-intl-en) method doc comments; generated docs contain no Chinese text. - new snapshot gendoc/data/api-actions-en.json (9038 entries, ~1.3MB): (product, action) -> first prose line of the intl SDK method comment - new refresh command 'make doc-refresh-en' (gendoc -refresh-en): downloads the SDK master tarball (~6 MB), extracts descriptions via a doc-comment block parser anchored to the plain (non-WithContext) method declarations, and rewrites the snapshot atomically; any failure keeps the previous snapshot - rendering precedence: English description + doc URL > URL only > bare action name; the Chinese catalog names are never rendered - coverage of rendered API Reference entries: 80.0% with an English description, 20.0% URL-only (intl SDK lacks those products/actions), 0% Chinese Also fixes a regression from the previous commit: the framework fwResource branch returned early before writing the doc file, leaving stale content on disk. The branch now flows into the normal write path (caught by the ssm_secret_version_v2 doc not being regenerated).

…atalog entries 'make doc' no longer performs any network request. Both the action catalog presence check and the English descriptions / doc URLs now come solely from the bundled snapshots, making doc generation deterministic and reproducible. Snapshot updates stay explicit: make doc-refresh-data # refresh the catalog (doc URLs) make doc-refresh-en # re-extract descriptions, merge URLs The -refresh-en rewrite now bases the snapshot on the full public catalog (15414 entries) and merges the intl SDK descriptions on top, so entries the intl SDK lacks (bh, dayu, ...) still carry their doc URL instead of degrading to a bare action name. Current split: 9038 entries with descriptions, all URL-bearing entries intact. Regenerated docs are byte-identical to the previous online-fallback output (80% description + URL, 20% URL-only, 0 Chinese). Also fixes -refresh-en merging the catalog URL field: the raw catalog uses actionName/actionCNDocUrl, not the slim snapshot's name/url tags.

… doc-refresh A single 'make doc-refresh' (gendoc -refresh) refreshes all three bundled snapshots in one run: the slim catalog, the product list and the English descriptions. The raw action catalog downloaded for the slim snapshot is reused for the doc-URL merge in the English refresh, removing the duplicate download the two separate commands incurred. The old make targets and flags are removed; 'make doc' is unchanged (offline, reads only the bundled snapshots).

…API sections Addresses review findings on PR #4557: P1 — wrong-product API links (verified: 43 method names are declared on service structs across different products). Two fixes: - NewXxxService(...) constructor calls now contribute to local variable type inference, including package-qualified forms (svccvm.NewCvmService(...)). Previously only composite literals were recognized, leaving the receiver hint empty and falling back to the first same-name declaration (e.g. thpc_workspaces' ModifyPassword linked to ckafka 597 instead of cvm). - When the receiver hint is unknown AND the method name is declared on several distinct service structs, the call is skipped instead of guessing. Verified fixes: postgresql_readonly_instance now links ModifyDBInstanceSecurityGroups to redis 239 (the resource actually calls svccrs.NewRedisService for this) instead of cdb 236. P2 — 83 docs carried a bare 'This resource interacts ...' lead-in with no bullets because an all-empty verb map slipped the len(refs)==0 guard. renderAPIRefsLead now totals list lengths and renders nothing when empty; empty verb groups are skipped during rendering. P3 cleanups: - drop fetchActionCatalog (0 callers) and the unused http.Client parameter / liveHTTPClient helper from the offline catalog loader; fix stale comments referencing removed make targets - stop writing api-products.json (written but never read) - refreshENData merge is now O(n) via an index map instead of O(n*m) - probe mode refuses to run when the repo root looks wrong instead of silently producing empty output Guard test added for the empty-verb-list rendering. Regenerated docs: 2259 files, 0 empty sections, 0 Chinese; postgresql_readonly_instance and thpc_workspaces now link the correct products.

…cross doc types Self-review findings on PR #4557: - desc cleanup: strip inline HTML tags (<b>, <li>, ...) leaked from the intl SDK comments and cap descriptions at 240 chars (cut at the last sentence boundary). 10 rendered entries carried raw HTML and 142 were over-long; both are now 0. - data source manual API sections were parsed out of the .md but then discarded (only resources consumed manualAPI); they now override the auto-extracted mapping like resources do. - framework.go: extend the manual section parsing and the API rendering branch to fwDataSrc as well, so an API section in a data source .md neither leaks into the description nor goes unused. - fix stale 'make doc-refresh-en' / '-refresh-data' references in apiref_en.go / apiref_refresh.go comments left over from the command merge (the flags no longer exist; only -refresh does). Regenerated docs are stable (idempotent double run, 0 diff), 0 Chinese, 0 over-long lines. Coverage unchanged: resources 95.7%, data sources 97.3%.

Links in the API Reference section now point to the English documentation site (www.tencentcloud.com/document/product/...) whenever the action is listed there; the Chinese-site URL remains only as a fallback for actions the international site has not translated. The international site assigns its own document ids, so URLs cannot be rewritten by pattern. 'make doc-refresh' now additionally walks the doc-site internal endpoints: POST /document/api/mid/getCategoryByLang POST /document/api/mid/getCatalogue (per product category) and builds (product, action) -> international URL, merged over the Chinese-site URLs in the en snapshot. The endpoints fingerprint TLS and reject Go's HTTP stack, so the fetches shell out to curl. Rendered entries: 3531 (47.5%) international links, 3900 (52.5%) Chinese-site fallback, 0 bare. Spot-checked international URLs resolve to the matching action pages (CreateAssistantCidr, CreateVpc, ModifyDBInstanceSecurityGroups...).

Third-party review follow-up: P3 — cleanDesc byte-truncation produced broken markdown (57 unbalanced parentheses in the snapshot, 39 in rendered docs). cleanDesc now: - measures length in runes, not bytes - prefers a sentence boundary and falls back to a balanced-rune scan that only accepts cuts where (), [] stay paired - drops the description entirely when no balanced cut exists P1/P3 — three teo entries carried unclosed '[' fragments. Root cause: the intl SDK wraps long descriptions across comment lines and a markdown link straddles the wrap ('[ModifyL4Proxy' / '](url)'). The extractor now merges consecutive comment lines until brackets balance, and collapses the wrap artifact ('[Foo ](url)' -> '[Foo](url)'). Snapshot: 0 unbalanced parentheses, 0 unbalanced brackets. P2 — dead code and state cleanups: - remove intlGetJSON (superseded by the curl implementation inside fetchCatalogue) and the unused intlCatByURL constant - remove loadSnapshotCatalogEntries (debug-only leftover) - drop the never-read catalog.source field and fix its stale comment - replace the misleading 'replaced Chinese URLs' counter with a plain override message - probeAllResource now validates the repo root like probeResource does P1 note: the postgresql_readonly_instance redis link (239) is a faithful representation of the runtime call chain — the resource borrows the redis client for this operation, and the redis API accepts a product parameter covering postgres instances. Left as-is; the manual API section remains the escape hatch for semantic reattribution.

Guarantees the generated API Reference section never breaks markdown formatting (empty sections are preferred over malformed output): - sanitizeInlineFences escapes code-fence markers embedded inside a schema Description (upstream mps transcode template pasted table markup like ':\`\`\`open: ...' into the description text), which previously leaked an odd fence into the rendered document. - balanceFences rewritten: instead of deleting the last fence marker (which could swallow a needed closing fence), it escapes the last lone untagged marker to literal text and never touches language-tagged openers. - cleanDesc now cuts by runes, prefers sentence boundaries, and only accepts cuts where ()/[] remain balanced — otherwise the description is dropped entirely. Verification over all 2259 generated docs (7427 API entries): 0 Chinese, 0 unbalanced parentheses/brackets, 0 HTML leakage, 0 empty API sections. The one remaining odd-fence document (cls_config_attachment) comes from an unclosed Example Usage block in the upstream .md that predates this PR; its Import section renders correctly.

iuin added 8 commits October 13, 2026 20:13

docs: regenerate website docs with the refreshed API catalog snapshot

07eec36

Aligns the generated docs with the gendoc/data snapshot updated in the previous commit (14096 -> 14139 actions).

refactor(gendoc): single flag set for CLI modes; drop dead code

6694325

- parse -refresh-data and -probe in one flag.FlagSet so passing an unknown flag doesn't print spurious usage noise from a second set - remove unused dumpJSON helper and resourceAPIRefs type - update probe entry doc comment

iuin added 2 commits October 14, 2026 17:18

iuin force-pushed the feat/doc-api-ref-online-and-datasources branch from 50b95a1 to 8d77ede Compare October 14, 2026 09:40

iuin force-pushed the feat/doc-api-ref-online-and-datasources branch from d3c72e3 to 0bfda3d Compare October 14, 2026 10:34

iuin force-pushed the feat/doc-api-ref-online-and-datasources branch from 380deba to e27364b Compare October 15, 2026 02:38

iuin force-pushed the feat/doc-api-ref-online-and-datasources branch 2 times, most recently from e27364b to 16a87e0 Compare October 15, 2026 02:38

Navigation Menu

Repository navigation

feat(doc): [000000000] online catalog fetch with fallback; extend API Reference to data sources and actions - #4557

feat(doc): [000000000] online catalog fetch with fallback; extend API Reference to data sources and actions#4557
iuin wants to merge 13 commits into
masterfrom
feat/doc-api-ref-online-and-datasources

iuin commented Oct 14, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

iuin commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issue

Summary of Changes

1. API Reference rendering for resources

2. API Reference extended to data sources, actions and ephemeral resources

3. English descriptions; no Chinese in generated docs

Snapshot refresh

Coverage

Type of Change

Does this change introduce a breaking change?

Checklist

Additional commit: prefer international-site doc URLs

Additional commit: description extraction & cleanups (review round 2)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

iuin commented Oct 14, 2026 •

edited

Loading