Registries

A Config knows two kinds of block type besides xcl’s builtins. A declared type is a plain Go type that configuration is decoded into, with no provider. A plugin’s types come with a provider. Both come from registries, added with xcl.WithRegistry, so there is one way in and one place things are registered: everything a Config can read is stated where it is created, and it is always clear where a type or plugin comes from. To write a plugin of your own, start from the plugin template.

The local registry

The registry package ships the local registry, which holds the Go types and plugins an application supplies itself. Create one with registry.NewLocal, register on it, and give it to the Config. In Python the local registry is xcl.LocalRegistry, and plugins are always external binaries: there is no in-process form.

local := registry.NewLocal()
registry.Register[Deployment](local, "deployment")      // a plain Go type
local.RegisterPlugin(&template.TemplatePlugin{})        // an in-process plugin
local.RegisterExternalPlugin("./bin/xcl-plugin-docker") // an external plugin binary
local.RegisterPluginDirectory("~/.xcl/plugins")         // a directory to search

c, err := xcl.NewConfig(
  xcl.WithRegistry(local),
  xcl.WithStatePath("./.xcl"),
)
if err != nil {
  return err
}

local = xcl.LocalRegistry()
local.register_type(Deployment, "deployment")              # a plain Python type
local.register_external_plugin("./bin/xcl-plugin-docker")  # an external plugin binary
local.register_plugin_directory("~/.xcl/plugins")          # a directory to search

with xcl.Config(registries=[local], state_path="./.xcl") as cfg:
    ...

Registering only records. None of the Register calls returns an error, and nothing is started, checked or searched until it is needed; see How problems are reported.

Declaring types

registry.Register[T] declares the Go type T as a block type on a registry, under a name made of the block type and an optional subtype. T is a plain struct of the application’s own; one embedding types.ResourceBase is refused. A plain struct with neither label fields nor a Meta field cannot read a name, so it is a singleton, written without labels. It works on every supported Go version; on Go 1.27 the same is also a method, local.Register[T]("deployment"), and RegisterType(&T{}, ...) takes a pointer to a value instead, for a type only known at run time. In Python the type is a plain dataclass, and register_type takes the class and the name, local.register_type(Deployment, "deployment"); a class with neither label fields nor a meta field is a singleton in Python as well:

type PostgreSQL struct {
  Meta types.Meta // the block's id, file and line, filled by xcl

  Engine   string `xcl:"engine,label"` // the subtype, "postgres"
  Name     string `xcl:"name,label"`   // the block's name, "main"
  Location string `xcl:"location"`
}

local := registry.NewLocal()
registry.Register[Deployment](local, "deployment")           // deployment "api" {}
registry.Register[Service](local, "service")                 // service "api" {}
registry.Register[PostgreSQL](local, "database", "postgres") // database "postgres" "main" {}

c, err := xcl.NewConfig(xcl.WithRegistry(local))
if err != nil {
  return err
}

from dataclasses import dataclass
import xcl

@dataclass
class Deployment:
    image: str
    meta: xcl.Meta | None = None

@dataclass
class Service:
    port: int = 80
    meta: xcl.Meta | None = None

@dataclass
class PostgreSQL:
    location: str
    meta: xcl.Meta | None = None  # the block's id, file and line, filled by xcl

    engine: str = xcl.field(label=True, default="")  # the subtype, "postgres"
    name: str = xcl.field(label=True, default="")    # the block's name, "main"

local = xcl.LocalRegistry()
local.register_type(Deployment, "deployment")             # deployment "api" {}
local.register_type(Service, "service")                   # service "api" {}
local.register_type(PostgreSQL, "database", "postgres")   # database "postgres" "main" {}

with xcl.Config(registries=[local]) as cfg:
    ...

A plain struct needs no xcl type of its own. A field called Meta, which must be a types.Meta with no tag, receives a copy of the block’s metadata, including the disabled flag and depends_on list configuration set on the block, as Meta.Disabled and Meta.DependsOn. Fields tagged as labels are filled from the block’s labels, the subtype and the name for a type with a subtype, or the name alone for one without, and are read in expressions like an attribute, such as database.postgres.main.name; configuration cannot set them. Every block takes disabled, depends_on and meta, so a type declares no fields by those names.

A Python class reads the same through its own fields: a field called meta, an xcl.Meta | None, receives the metadata, meta.disabled and meta.depends_on among it, and fields marked xcl.field(label=True) are filled from the labels. Neither is part of the class’s schema, and a class declares no fields called disabled or depends_on.

A declared type has no provider, so it is only ever decoded into. It still takes part in references, variables and modules, and is read back as your own Go type with Find[T], FindByType[T] or Decode, or in Python as your own class with cfg.find, cfg.find_by_type or cfg.all. Each of them returns a copy: changing it leaves the configuration as it is, which only configuration and a provider’s results set. Every type a configuration uses must be declared, even the ones the program never reads.

NewConfig reads the declared types of every registry once, when the Config is created, and returns an error when they do not fit together; see Clashes.

Plugins

• RegisterPlugin takes a plugin compiled into the program. It runs in the same process, and is known by the name of its Go type, such as TemplatePlugin.
• RegisterExternalPlugin takes the path to a plugin binary, which xcl starts as a separate process and talks to over gRPC. It is known by the binary’s file name, such as xcl-plugin-docker. A path that does not exist is reported when plugins load, not here.
• RegisterPluginDirectory takes a directory to search, when plugins load, for executables whose names match the registry’s pattern. A leading ~/ is the home directory and environment variables are expanded. A directory that does not exist provides no plugins and is not an error.

A directory is searched for files named xcl-plugin-* by default. Give another pattern, as filepath.Match takes it, when the registry is created:

local := registry.NewLocal(registry.PluginPattern("acme-plugin-*"))
local.RegisterPluginDirectory("/opt/acme/plugins")

local = xcl.LocalRegistry(plugin_pattern="acme-plugin-*")
local.register_plugin_directory("/opt/acme/plugins")

The pattern is fixed when the registry is created, so a later registration can not change what an earlier one matches.

Each directory search is reported as discover events, whose Meta holds the dirs searched, the registry and, on success, the count of plugins found. Each plugin starting and being checked is reported as load events, whose Meta holds the plugin, its registry and, on success, the block_types it provides. See Events and logging.

Configuration only, without state

An application that only reads its configuration needs no plugins and no state. Declare its types on a local registry, and leave out xcl.WithStatePath and xcl.WithStateStore: nothing is persisted, the applied configuration is held in memory only, and every run reads the configuration afresh. This is a supported mode, not a fallback. In Python, leave out state_path and state_store, and read the applied entities back with cfg.all, as Decode has no Python form:

local := registry.NewLocal()
registry.Register[Deployment](local, "deployment")
registry.Register[Service](local, "service")

c, err := xcl.NewConfig(xcl.WithRegistry(local))
if err != nil {
  return err
}

if err := c.Apply("./config"); err != nil {
  return err
}

var cfg appConfig
if err := c.Decode(&cfg); err != nil {
  return err
}

local = xcl.LocalRegistry()
local.register_type(Deployment, "deployment")
local.register_type(Service, "service")

with xcl.Config(registries=[local]) as cfg:
    cfg.apply("./config")

    deployments = cfg.all(Deployment)
    services = cfg.all(Service)

The configuration only example is a complete program built this way. Add a state option when a program needs to know, on its next run, what it applied on this one, as a program with providers does to update and destroy what it created.

Several registries and load order

xcl.WithRegistry can be given any number of times. Registries are read in the order they were given, and their plugins load in that order; within a registry, plugins load in the order they were registered, with each directory’s plugins at the point the directory was registered. In Python the registries list is read in the same order:

builtin := registry.NewLocal()
builtin.RegisterPlugin(&template.TemplatePlugin{})
registry.Register[Settings](builtin, "settings")

user := registry.NewLocal()
user.RegisterPluginDirectory("~/.xcl/plugins")

c, err := xcl.NewConfig(
  xcl.WithRegistry(builtin), // read and loaded first
  xcl.WithRegistry(user),    // then this one
)

builtin = xcl.LocalRegistry()
builtin.register_external_plugin("./bin/xcl-plugin-template")  # plugins are external binaries from Python
builtin.register_type(Settings, "settings")

user = xcl.LocalRegistry()
user.register_plugin_directory("~/.xcl/plugins")

with xcl.Config(registries=[builtin, user]) as cfg:  # builtin is read and loaded first, then user
    ...

Plugins load at the start of the first Validate, Apply, Destroy, Diff or Load, and only once per Config; later operations reuse them. The order decides only the order types are read, plugins load and events are emitted. It never decides which of two providers wins a block type, see Clashes.

The registry package also ships a GitHub registry, registry.NewGitHub, which installs plugins straight from GitHub releases: one RegisterPlugin("owner/repo", "v0.3.0") line per plugin, downloaded for the current platform, verified, cached and started as an external plugin. Give it beside a local registry like any other; see Installing plugins from GitHub.

How problems are reported

Registration returns no errors. A problem is reported in one of three places, depending on when it can first be seen. Whatever order registration and loading happen in, each problem is reported in the same place.

A mistake in a single call panics at that call. It is wrong on every run, so there is nothing for a caller to handle, and the stack trace points at the bad line. Register and RegisterType panic, naming the type, for an empty name, more than one subtype, an empty subtype, a prototype that is not a pointer to a struct, or a plain struct xcl cannot lay out: one declaring an attribute called meta, a Meta field that is not a types.Meta or a types.Meta under another name, a disabled or depends_on field of the wrong type, label fields that are not strings or do not match the block’s labels, or a subtype for a singleton. Register also panics for a T that is not a struct. RegisterPlugin panics for a nil plugin, and WithRegistry for a nil registry.

A clash between declared types is an error from NewConfig. Declared Go types are known without loading anything, so NewConfig reads every registry’s types and returns an error when they do not fit together: the same block type declared twice, in one registry or across two, a type keyword used both with and without a subtype, or a builtin’s name. A clash between registries comes from how they are combined, not from one bad line, so it is an error rather than a panic. In Python, creating the Config raises xcl.TypeNameClashError:

c, err := xcl.NewConfig(xcl.WithRegistry(local), xcl.WithRegistry(shared))

var clash *xcl.TypeNameClashError
if errors.As(err, &clash) {
  fmt.Printf("%s: %s (%s) and %s (%s)\n",
    clash.Name, clash.Existing, clash.ExistingRegistry, clash.Provider, clash.Registry)
}

try:
    with xcl.Config(registries=[local, shared]) as cfg:
        ...
except xcl.TypeNameClashError as e:
    print(e)

A problem with plugins is an error from the first operation. A plugin binary that is missing or broken, a plugin directory that can not be read, or a block type a plugin provides that something else already provides, depends on the machine or on what loads, so it is returned by the first Validate, Apply, Destroy, Diff or Load, the operation that loads plugins. Any plugin that fails to start fails the load, whether it was registered explicitly or found in a directory. In Python the first operation raises xcl.PluginLoadError:

err := c.Apply("./config")

var loadErr *xcl.PluginLoadError
if errors.As(err, &loadErr) {
  fmt.Printf("plugin %q from registry %q: %s\n", loadErr.Plugin, loadErr.Registry, loadErr.Err)
}

try:
    cfg.apply("./config")
except xcl.PluginLoadError as e:
    print(e)

*xcl.PluginLoadError matches xcl.ErrPluginLoad with errors.Is. It names the Plugin, the Registry it came from, and the reason in Err. Plugin is empty when the registry itself could not provide its plugins, such as a directory that could not be searched. Loading runs once per Config, so a failed load is returned again by every later operation on the same Config.

Clashes

Every block type, with its subtype where it has one, is provided once. A block type provided twice is an error, wherever the two come from, and is reported as a *xcl.TypeNameClashError:

Where the error comes from depends on what clashes:

• Two declared Go types, in one registry or across two, or a declared type with a builtin’s name, are returned by NewConfig, naming both Go types and their registries, or builtin. A declared type whose keyword is used in the other form, with a subtype where it is used without one or the other way round, is a *xcl.TypeFormError from NewConfig.
• A plugin and a declared type, a plugin and a builtin, or two plugins, in one registry or across two, are returned by the first operation that loads plugins, naming both providers and their registries. A plugin that uses a type keyword in the other form fails the load with a *xcl.TypeFormError.

There is no precedence between registries, or between providers in one: a clash is an error whatever order they were given in, so adding a registry can never silently change which provider a configuration gets.

Writing your own registry

A registry is anything that implements registry.Registry, so types, functions and plugins can come from anywhere: a company plugin server, a cache filled on first use, or a fixed set built into a tool. The package defines the interfaces and the Type a registry declares:

// Registry provides types, functions and plugins to a Config. A Config uses
// every registry given to it with xcl.WithRegistry, in that order.
type Registry interface {
  // Name identifies the registry in events and errors, i.e. "local" or
  // "registry.xcl.dev"
  Name() string

  // Types returns the plain Go types this registry declares. It is read
  // once, by NewConfig. A registry with no Go types returns nil.
  Types() []Type

  // Functions returns the Go functions this registry offers to
  // configuration. It is read once, by NewConfig, which reports a function
  // named like a builtin or like another application function. A registry
  // with no functions returns nil.
  Functions() []Function

  // Plugins returns the plugins this registry provides, fetching them first
  // if it needs to. It is called once per Config, when plugins load.
  Plugins(ctx context.Context, emit events.Emit) ([]Plugin, error)
}

// Type is a plain Go type declared as a block type. It has no provider: it
// is only decoded into.
type Type struct {
  // Type is the block type, i.e. "deployment" or "resource"
  Type string

  // Subtype is the subtype, empty for a type declared without one
  Subtype string

  // Prototype is a pointer to a plain struct, the type's Go type; a struct
  // embedding types.ResourceBase is refused when the type is declared
  Prototype any
}

// Plugin is one plugin a registry provides, ready to start
type Plugin interface {
  // Name identifies the plugin in events and errors
  Name() string

  // Start runs the plugin and returns the host xcl talks to it through
  Start(emit events.Emit) (plugins.PluginHost, error)
}

Types is read once, by NewConfig, and a registry with no Go types returns nil. Plugins is called once per Config, when plugins load, with an emit a registry can report its own progress through, as the local registry reports its directory searches as discover events; a nil emit is silent.

A registry rarely needs to start a plugin itself. registry.InProcess(p) returns a Plugin for a plugin compiled into the program, and registry.Executable(path) one for a plugin binary, started over gRPC, the same two the local registry uses. As an illustration, here is a registry a tool might write for a fixed set of types and plugins it always ships with. It is not part of xcl:

// Toolkit provides the types and plugins every build of the tool ships with
type Toolkit struct {
  PluginDir string
}

func (t *Toolkit) Name() string {
  return "toolkit"
}

func (t *Toolkit) Types() []registry.Type {
  return []registry.Type{
    {Type: "settings", Prototype: &Settings{}},
    {Type: "storage", Subtype: "bucket", Prototype: &Bucket{}},
  }
}

func (t *Toolkit) Plugins(ctx context.Context, emit events.Emit) ([]registry.Plugin, error) {
  return []registry.Plugin{
    registry.InProcess(&template.TemplatePlugin{}),
    registry.Executable(filepath.Join(t.PluginDir, "xcl-plugin-docker")),
  }, nil
}

The types a registry returns from Types are checked by NewConfig like any declared with Register, so a malformed one, or one that clashes, is an error from NewConfig. An error from Plugins fails the load as a *xcl.PluginLoadError with an empty Plugin, and an error from a plugin’s Start as one naming that plugin. The registry’s Name is what events and errors show as the registry.