Making Mapping/Listing open classes (for attaching methods)

[sin-ack]

Hey all,

I would like to propose that the standard library classes Listing and Mapping become open, so that they can be extended. This would not allow adding properties to the listings/mappings (as they can only accept elements and entries respectively), but rather allow adding methods to the chain.

Why?

This enables the use-case of "specialized listings/mappings", where the listings/mappings can be given additional functionality and can be asked questions. Ideal example from an internal project we have:

class DependencySpec {
   name: String
   version: String // Simplified
}

/// A list of alternative versions for a dependency.
class Alternatives extends Listing<DependencySpec> {
    // A method that answers a question about this listing.
    /// Does the given package satisfy any alternatives?
    function satisfiedBy(meta: Metadata, packageName: String): Boolean =
        let (package = meta.packages[packageName].package)
        this.any((spec) -> spec.name == package.name && spec.version == package.version)
}

// --- Template module ---
module Package

name: String
version: String
description: String

dependencies: Mapping<String, Alternatives>

// --- Amending ---
// package.pkl
amends "Package.pkl"

name = "foo"
version = "1"
description = "The foo package does the foo stuff"

dependencies {
    ["bar"] {
        new { name = "vendor1-bar"; version = "1" }
        new { name = "vendor2-bar"; version = "2.0" }
    }
}

// --- Usage ---
// The module receives configuration and a list of packages externally during evaluation through meta.pkl.
// Configuration.pkl
module Configuration

import "package.pkl"
import "meta.pkl"

barDependencyName = String(package.dependencies["bar"].satisfiedBy(meta, this))

// barDependencyName is now validated against the alternatives and is valid for usage in the rest of the code by doing: `meta.packages[barDependencyName].(...)`

Workarounds

The alternative to the example above would be having a Listing property in Alternatives:

class Alternatives {
    alternatives: Listing<DependencySpec>

    function satisfiedBy(...) = ...
}

// --- Amending ---

// ...

dependencies {
    ["bar"] {
        alternatives {
            // ...
        }
    }
}

While this doesn't require any changes to the language, it:

1. Is clunkier,
2. Introduces an additional layer when outputting that has to be fixed by mucking with output,
3. Also introduces an additional layer in code generation (this one can't be fixed without making special affordances in the code generator and deserializer).

What do you think? Alternatively, it would be cool if one could attach methods to type aliases which would also solve this, but that's another discussion.

[bioball]

Using a subclassing approach would mean this will no longer typecheck (because you are passing a Listing and not the subclass):

local myDeps: Listing<DependencySpec> = new {
  // etc
}

dependencies {
  ["bar"] = myDeps
}

It also makes it harder to interop between Pkl and external data (e.g. parsing JSON into Pkl structs).

Rather, it sounds like you're looking for It extension methods.
This would allow you to define your own methods on top of existing types.
However, extension methods probably aren't a great choice for Pkl, because it adds runtime overhead to name resolution.

The pipe operator (|>) is the Pkl way to write an "extension method".
Your example can be written as:

function satisfiedBy(meta: Metadata, packageName: String): (Listing<DependencySpec>) -> Boolean = (it: Listing<DependencySpec>) ->
  let (package = meta.packages[packageName].package)
    it.any((spec) -> spec.name == package.name && spec.version == package.version)

Then, used like so:

package.dependencies["bar"] |> satisfiedBy(meta, "bar")

[sin-ack]

Hmm, I'm not really a fan of this approach because it disconnects the particular type I am interested in and the method operating on it (which in my eyes are bound together). In addition to that, getting a reference to satisfiedBy down to user code is not very easy without breaking modularity (especially because functions live in a separate namespace so you can't re-export them like you can with classes/modules). These aspects make the approach mentioned in the Workarounds section better for my particular use case.

Regarding runtime overhead: How about the type alias method attachment thing I mentioned? I am not interested in providing a method extension for all instances of a particular type, but rather this particular interface being provided. If something like this could be done:

typealias Alternatives = Listing<DependencySpec>
function Alternatives.satisfiedBy(...) = ...

then it would allow me to present methods for a particular interface to end-users, and since they're explicitly-specified there would be no runtime lookup overhead. From there, the only other concerns I can see is the scope in which these attachments could be defined (which can be delimited with different syntax I suppose) and how this affects type resolution (although the scoping rule can just be changed when handling a call expression).

[sin-ack]

Using a subclassing approach would mean this will no longer typecheck (because you are passing a Listing and not the subclass):

Right, that makes sense as it creates a new nominal type unfortunately. I'd rather not create a subtype anyhow, as the only concern is exposing behavior.

[odenix]

Another way to support "specialized listings/mappings" would be to generalize the object model to allow user-defined classes with elements/entries, a feature I proposed in the past. This would also turn Listing, Mapping, and Dynamic into regular stdlib classes without any magic. You’d still end up with new nominal types, but this isn’t necessarily a problem, especially if an object’s elements/entries can be “copied” to another object. And interop such as parsing JSON into typed Pkl objects needs to be schema-aware in any case.