External Resources

[rspurgeon]

External Resources

Overview

Implement external resources to enable kongctl to integrate with
resources managed outside of a particular set of declarative configuration
files. This allows users to reference existing resources without taking ownership
which supports a variety of use cases.

As kongctl expands capabilities, users need:

• Ability to manage resources across teams while maintaining ability to reference resources defined elsewhere. Referencing resources across teams, enabling federated configuration management (one team owns a portal and another team externally references it for their API publication).
• Migration paths that don't require abandoning existing declarative tools immediately. Phased Migration from one tool to kongctl
• Ability to operate multiple tools in parallel

Solution

Introduce _external blocks that:

• Define resources managed by external tools
• Use selectors to query and identify specific resources
• Resolve to resource IDs and data during planning
• Enable dependencies between kongctl-managed and externally-managed resources

Key Features

• Direct ID Reference: Reference resources by UUID when known
• Field Selectors: Query resources by field values
• Parent Relationships: Support hierarchical resource structures
• Implicit ID Resolution: Automatic resolution of ID fields
• Type Safety: Resource-type-aware validation and resolution

Success Criteria

• Users can reference any supported Konnect resource managed externally
• Clear error messages for ambiguous or missing resources
• Seamless integration in dependency chains
• Intuitive configuration syntax

Example Use Case

A team using decK for gateway configuration wants to adopt kongctl for API
management. They can reference their existing control planes and services
while managing APIs and API implementations through kongctl:

accepted design

portals:
  - ref: core-team-portal
    _external:
      selector:
        matchFields:
          name: "Core Team"

apis:
  - ref: sms
    _external:
      selector:
        matchFields:
          name: "SMS API"
    
    # Children to the external API resource
    versions:
      - ref: sms-v1
        version: !file apis/sms/openapi.yaml#info.version
        spec: !file apis/sms/openapi.yaml
        
    documents:
      - ref: sms-getting-started
        title: "SMS API: Getting Started"
        slug: sms-getting-started
        status: published
        content: !file apis/sms/docs/getting-started.md
        
    publications:
      - ref: sms-api-to-core-portal
        portal_id: core-team-portal # FK ref to external portal resource

alternative design

external_resources:
  - ref: prod-cp
    resource_type: control_plane
    id: 123e4567-e89b-12d3-a456-426614174000
  - ref: user-service
    resource_type: ce_service # core entity service
    control_plane: prod-cp    # Parent reference
    selector:
      matchFields:
        name: user-service

apis:
  - ref: user-api
    name: User Management API
    
api_implementations:
  - ref: impl
    api:
      ref: user-api
    service:
      control_plane_id: prod-cp  # Resolves to ID of external resource
      id: user-service           # Resolves to ID of external resource

[rspurgeon]

An alternative solution could be to add an "external" configuration to existing resource blocks.

control_planes:
  - ref: managed-cp
    name: my-managed-cp
    cluster_type: CLUSTER_TYPE_CONTROL_PLANE
  - ref: external-cp
    _external:
      selector:
        matchFields:
          name: that-other-cp
  - ref: external-cp-2
    _external:
      id: 123e4567-e89b-12d3-a456-426614174000

• With this solution we don't need the "special" external_resources type, and resources can be defined together which are both managed and external.
• This also allows resources to be defined in the same fashion as managed resources, the resource_type is implied by the block in which it is defined, the same as managed resources.
• If the _external key is present, no other keys are allowed
• Implementation may be easier because resources can be the same in code, with the addition of some type of external flag + external resolution data from the config. Then the resource could flow through the same current resolution code and the ref is resolved and can be used as a dependency. The dedicated block will require a new type of resource (the special external_resource type and handling at the top level.

[mheap]

I like option 2 where the ref is contained within the parent type

[rspurgeon]

I like option 2 where the ref is contained within the parent type

This is my vote as well. I think it provides a better experience and cleaner implementation

[jdw6359]

Option 2 seems great - as I understand it, these would resolve first and then other resources will treat it / refer to it the same as any other resource

[rspurgeon]

Option 2 seems great - as I understand it, these would resolve first and then other resources will treat it / refer to it the same as any other resource

Yes. This should be able to closely follow the same pattern as managed resources do. If a resource contains an _external block, and is referred to by another resource in the configuration, it will be resolved by the information in the _external block and its data will be available for other resources for use.

This design will also allow us to maintain parent->child relationships in the configuration with an external and managed resource:

portals:
  - ref: core-team-portal
    _external:
      selector:
        matchFields:
          name: "Core Team"

apis:
  - ref: sms
    _external:
      selector:
        matchFields:
          name: "SMS API"
    
    # Children to the external API resource
    versions:
      - ref: sms-v1
        version: !file apis/sms/openapi.yaml#info.version
        spec: !file apis/sms/openapi.yaml
        
    documents:
      - ref: sms-getting-started
        title: "SMS API: Getting Started"
        slug: sms-getting-started
        status: published
        content: !file apis/sms/docs/getting-started.md
        
    publications:
      - ref: sms-api-to-core-portal
        portal_id: core-team-portal # FK ref to external portal resource

In this hypothetical example, a "core" team manages the portal and API resources, while another team manages the API child resources. They setup external resources (using _external blocks) and refer to them using parent->child layouts and/or with foreign key references (like the publication portal_id field). Above the versions, documents and publications are managed children of the external sms API and refer via FK to the external portal.

[rspurgeon]

A larger example showing how it could look with a control plane and a GW Service.

In the below example we have values (services) on a parent resource that is external which violates the assertion in the description that no additional fields are allowed when _external is present. I will reevaluate how this could /should work.

control_planes:
  - ref: core-team-prod-cp  # External CP managed by Core team (maybe they use Terraform)
    _external:
      selector:
        matchFields:
          name: "Core Production Control Plane"
    services:
      - ref: sms-production-service # External GW Service (maybe they use decK)
        _external:
          selector:
            matchFields:
              name: sms-prod-aws-us-east-1

portals:
  - ref: core-team-portal # External Portal, maybe they use Terraform, or a centralized config for kongctl
    _external:
      selector:
        matchFields:
          name: "Core Team"

apis:
  - ref: sms # External API, maybe the core team wants to own these for governance reasons
    _external:
      selector:
        matchFields:
          name: "SMS API"
    
    # Children to the external API resource
    versions:
      - ref: sms-v1
        version: !file apis/sms/openapi.yaml#info.version
        spec: !file apis/sms/openapi.yaml
        
    documents:
      - ref: sms-getting-started
        title: "SMS API: Getting Started"
        slug: sms-getting-started
        status: published
        content: !file apis/sms/docs/getting-started.md
        
    publications:
      - ref: sms-api-to-core-portal
        portal_id: core-team-portal # FK ref to external portal resource

    implementations:
      - ref: sms-prd-impl
        service:
          control_plane_id: core-team-prod-cp # FK ref to CP
          id: sms-production-service # FK ref to GW Service

[jdw6359]

this makes sense to me! the only thing that sticks out as being unclear is the id fields (portal_id, control_plane_id) whose value is the name of an external ref. I imagine under the hood the ref resolves to an id or something like that, but that is more implied and not explicit.

are we just passing around ids behind the scenes, or do we want a certain number of fields exposed for access by the referring resource?

[rspurgeon]

this makes sense to me! the only thing that sticks out as being unclear is the id fields (portal_id, control_plane_id) whose value is the name of an external ref. I imagine under the hood the ref resolves to an id or something like that, but that is more implied and not explicit.

Excellent observation @jdw6359. I've mulled over this multiple times myself. As of now the id is extracted in the code. There is just very basic logic about dependencies and the 'id' is hard coded to be pulled from the resolved reference. I wanted to keep the data fields on the resources to match the actual schema, so I wanted to keep control_plane_id name the same but also make it very easy to express the relationship via the 'ref' value.

I considered something more explicit as well...

 implementations:
      - ref: sms-prd-impl
        service:
          control_plane_id: core-team-prod-cp.id
          id: sms-production-service.id

I've also considered something more 'formal' like a template style syntax which could allow the extraction of any data field in a dependent and possibly more advanced expressions.

[jdw6359]

I think that the syntax in your outlined approach is more clear, even if nothing dramatic changes under the hood and we are still only ever returning an "object" with a single id property:

 implementations:
      - ref: sms-prd-impl
        service:
          control_plane_id: core-team-prod-cp.id
          id: sms-production-service.id

[rspurgeon]

Something else i'm contemplating which may support a more formal syntax for this:

Right now it's just hardcoded which fields are dependencies of other resources. There is a small set of these currently in the dev portal world. As we expand the supported references and capabilities, this set will grow and the hardcoded approach won't work.

As you mentioned this control_plane_id: core-team-prod-cp.id syntax is very straightforward, but I'm wondering about the ability to express other relationships and ensuring accurate detection when a user is providing a ref value vs just a string value.

So another idea is to continue to support the yaml tag syntax like we do for file content loading into the yaml:

    versions:
      - ref: sms-v1
        version: !file apis/sms/openapi.yaml#info.version
        spec: !file apis/sms/openapi.yaml

We could add a simple ref yaml tag resolver:

    publications:
      - ref: voice-api-to-getting-started
        portal_id: !ref getting-started-portal#id

Or non-id fields like

   apis:
      - ref: my-api
         name: !ref ref-to-any-resource-with-a-name#name

Which conforms well with the current !file pattern, is very explicit, and fairly minimal syntax

[rspurgeon]

Here is a summary of the current file tag resolver for posterity:

 Supported YAML Tag Syntax

  1. Inline String Format (Simple Path)

  content: !file ./path/to/file.md
  Loads the entire file content.

  2. Inline String Format with Field Extraction

  name: !file ./openapi.yaml#info.title
  version: !file ./openapi.yaml#info.version
  description: !file ./openapi.yaml#info.description
  Uses # separator to extract specific fields from structured files (YAML/JSON).

  3. Map/Object Format

  content: !file
    path: ./path/to/file.yaml
    extract: info.contact.email
  Alternative syntax using a map structure with explicit path and optional extract fields.

  Key Features

  File Type Support

  - YAML files (.yaml, .yml) - Parsed as structured data
  - JSON files (.json) - Parsed as structured data
  - Text files (any other extension) - Returned as plain string

  Field Extraction

  - Uses dot notation for nested field access: info.title, info.contact.email
  - Works with both YAML and JSON files
  - Returns the specific field value rather than the entire document

  Security Features

  - Path validation:
    - No absolute paths allowed
    - No parent directory traversal (../)
    - Paths resolved relative to the YAML file's directory
  - File size limit: Default 10MB maximum
  - Caching: Files are cached to avoid re-reading