Unchanged, update or replace

Some changes can be made to a running entity and some cannot. A container’s image is fixed when Docker creates it, and a network cannot be moved to a new address range. A provider knows which is which, so the decision is its own: its Changed method answers one of three outcomes, and xcl does the rest.

Plugins are written in Go for now. A provider that decides this way works from both Go and Python applications.

What Changed answers

Changed is part of every plugins.ResourceProvider[T], alongside the Update it leads to:

Changed(ctx context.Context, old T, new T, changes []entity.PropertyChange, dependencies []entity.DependencyChange) (entity.Change, error)
Update(ctx context.Context, resource T, changes []entity.PropertyChange, dependencies []entity.DependencyChange) (T, error)

old is the entity as saved by the last apply. new is the current configuration after it has been through Read, so it holds both configuration edits and drift observed in the real entity. changes lists the settings that differ from the last apply, covered below, and dependencies the dependencies that change with it, covered below. The answer is an entity.Change:

const (
  // NoChange leaves the resource as it is.
  NoChange Change = iota
  // Update calls the provider's Update to change the resource in place.
  Update
  // Replace destroys the resource, then creates it again.
  Replace
)

An error from Changed, or from the Read before it, fails the whole apply or plan before anything is created, updated or destroyed.

Changed is only asked about entities that are already in saved state. An entity that is new to the configuration is created without asking, and one whose last apply failed (saved failed or destroy_failed) is replaced without asking, since there is nothing trustworthy to compare against.

DefaultChanged

Embed plugins.DefaultChanged[T] in a provider to get a Changed method. It compares the JSON form of old and new, ignoring xcl’s own metadata (meta, depends_on and disabled), and answers entity.Update when they differ and entity.NoChange when they do not:

type MyProvider struct {
  plugins.DefaultChanged[*MyEntity]
}

DefaultChanged ignores both changes and dependencies and never answers entity.Replace. A provider that cannot change some settings in place, or that must react to a replaced dependency, defines its own Changed and defers to DefaultChanged for everything else, as the Docker example below does.

What an entity is told about its dependencies

The dependencies argument lists the entities this one depends on that the same apply will update or replace, each with its outcome, sorted by address:

type DependencyChange struct {
  // Address is the dependency's address, for example "docker.network.app".
  Address string
  // Change is Update or Replace, never NoChange.
  Change Change
}

• Direct dependencies only. A dependency is an entity this one references or names in depends_on. What that dependency depends on in turn is not passed on: an entity hears about its own inputs, and each provider decides for itself whether a change travels further.
• Update and replace only. A dependency that is unchanged is not listed, so an empty list means nothing this entity depends on is changing.
• Newly created dependencies are not listed. Referencing an entity that is being created already changes this entity’s configuration and leaves the values it reads unknown, so it is planned as at least an update anyway.
• Look-through. Outputs, variables, modules and registered configuration-only types have no provider, so they are looked through to the provider-backed entities behind them. A module boundary or a variable never hides a replaced entity: an entity inside a module is told about the changing entities that the module block’s variables reference.

xcl decides entities in dependency order, so every dependency’s outcome is known by the time an entity is asked.

When an entity’s configuration reads a value that is only known after the apply, because the dependency that computes it is being created, updated or replaced, its provider is still asked. The saved value stands in for the unknown one, and the entity is planned as at least an update; the provider can still answer replace.

Update is given the same list, recorded while deciding, so a provider that answered update because of a dependency knows which one when it acts.

What an entity is told about its settings

The changes argument lists the settings of the entity that differ between the last apply and the configuration, one entity.PropertyChange each:

type PropertyChange struct {
  // Path is where the setting is, in configuration names, for example
  // network[0].name.
  Path Path
  // Before is the previous value, nil when the setting was added.
  Before any
  // After is the new value, nil when the setting was removed or when
  // Unknown is set.
  After any
  // Unknown is set when the new value is only known once the apply runs. It
  // is only ever set for Changed, never for Update.
  Unknown bool
  // Sensitive is set when the setting is sensitive. Before and After still
  // hold the real values.
  Sensitive bool
}

• The plan’s own list. xcl works the list out with the same comparison of the saved copy against the configured one that the plan shows, so a provider is told exactly the settings the plan lists, at the same paths. It describes the configuration: drift that Read finds in the real entity is not in it, it is in the difference between old and new.
• Not yet known while deciding. When Changed is called, a new value that is only known once the apply runs, such as a computed value of a dependency being created or replaced, has Unknown set and a nil After.
• Real values when updating. Update is given the comparison run again with every value known, including values produced earlier in the same apply. A setting that was unknown while deciding stays listed with its real value, even when it turns out the same as before, so the list always matches what the plan showed.
• Sensitive values are real. A sensitive setting’s change holds its real values in Before and After, so the provider can act on them. The change masks them itself: printing it, logging it or encoding it to JSON shows (sensitive) in their place. Plans, events and logs never show them. See Sensitive values.
• Plain JSON values. Before and After hold string, float64, bool, nil, []any or map[string]any, never the provider’s Go types: a number is a float64 and a nested block a map[string]any. They are the same whether the plugin runs in-process or as a separate program.
• Lists by position, maps by key. A list setting is compared element by element at its position, so a change is at network[0].name, command[2] or network[0].aliases[1]. A map entry is at its quoted key, such as environment["LOG_LEVEL"]. An added or removed list element or map key is one change holding the whole value, with a nil Before when it was added and a nil After when it was removed.

An empty list means none of the entity’s own settings changed.

A Path is a list of steps, built with Attribute, Index and Key. Match a change with At, for a change exactly at a path, or Within, for a change at a path or anywhere inside it:

// At returns true when the change is exactly at path.
func (c PropertyChange) At(path Path) bool {
  return c.Path.Equal(path)
}

// Within returns true when the change is at path or anywhere inside it, so a
// change at network[0].name is within network.
func (c PropertyChange) Within(path Path) bool {
  return c.Path.Within(path)
}

so change.Within(entity.Path{}.Attribute("network")) matches network, network[0], network[0].name and network[1].aliases[0] alike.

Whatever Update, Create or Read returns is authoritative for the entity’s computed values. xcl clears them before it decodes the result, so a computed value the provider leaves out, such as an omitempty address it cleared, is cleared rather than kept from the last apply.

Decide, then act

Both Apply and Diff (the plan) run the same decide pass first. Every entity is read and asked Changed before any provider creates, updates or destroys anything.

1. Decide. In dependency order, each entity is decided create, unchanged, update or replace. A failed Read or Changed stops here: nothing has changed, nothing is marked failed and saved state is untouched. Fix the cause and run again.
2. Destroy. Every replaced and removed entity is destroyed, dependents first, with state saved after each. A failed destroy marks the entity destroy_failed and stops the apply.
3. Create and update. In dependency order, with real values: new and replaced entities are created, updated entities are updated, unchanged entities are kept. A failed create saves the entity failed.

A failed replacement is not lost. The entity is saved failed or destroy_failed, and the next plan or apply replaces it again without asking its provider.

In the event stream every read and changed event comes before the first destroy, create or update. A replacement has no operation of its own: it shows as a destroy followed by a create of the same entity.

A worked example: Docker

The plugins example ships an external Docker plugin with a network and a container. Docker cannot move a network to a new address range, so the network provider answers replace when a change falls within subnet. Any other change to its settings answers update, which needs no Docker call, and when nothing changed it leaves the decision to DefaultChanged:

// networkReplaceSettings are the settings Docker fixes when it creates a
// network, a change to any of them needs a new network
var networkReplaceSettings = []entity.Path{
  entity.Path{}.Attribute("subnet"),
}

// Changed decides from what it is told. It answers replace when the subnet
// changed: Docker can not move a network to a new address range, the network
// has to be removed and created again. Any other change to its settings
// answers update, which needs no Docker call. When nothing changed the
// decision is left to DefaultChanged.
func (p *networkProvider) Changed(ctx context.Context, old *entities.Network, new *entities.Network, changes []entity.PropertyChange, dependencies []entity.DependencyChange) (entity.Change, error) {
  for _, change := range changes {
    for _, setting := range networkReplaceSettings {
      if change.Within(setting) {
        return entity.Replace, nil
      }
    }
  }

  if len(changes) > 0 {
    return entity.Update, nil
  }

  return p.DefaultChanged.Changed(ctx, old, new, changes, dependencies)
}

The container provider decides only from what it is told. Docker fixes a container’s image, command, environment and mounts when it creates it, so a change within image, command, environment or init_script is a replacement. So is a replaced dependency that isn’t a Docker network: the init script is template.init.destination, and when that template is replaced the container that mounts it is replaced too. Any other change to its own settings, such as a network block added, removed or given new aliases, answers update, and so does a network it depends on being updated or replaced. A dependency that is only updated, such as the init template rendering new content to the same file, is left to DefaultChanged, which leaves the container alone:

// replaceSettings are the settings Docker fixes when it creates a container,
// a change to any of them needs a new container
var replaceSettings = []entity.Path{
  entity.Path{}.Attribute("image"),
  entity.Path{}.Attribute("command"),
  entity.Path{}.Attribute("environment"),
  entity.Path{}.Attribute("init_script"),
}

// Changed decides from what it is told. It answers replace when the image,
// command, environment or init script changed, since Docker fixes them when
// it creates a container, or when a dependency other than a Docker network is
// replaced, such as the template that renders the init script. A change to
// the container's networks, or a network that is updated or replaced, is
// handled in place by Update, which moves the running container between
// networks. Any other change to its settings answers update. A dependency
// that is only updated, such as the init script's template rendering new
// content, leaves the container alone: the decision is left to
// DefaultChanged.
func (p *containerProvider) Changed(ctx context.Context, old *entities.Container, new *entities.Container, changes []entity.PropertyChange, dependencies []entity.DependencyChange) (entity.Change, error) {
  for _, change := range changes {
    for _, setting := range replaceSettings {
      if change.Within(setting) {
        return entity.Replace, nil
      }
    }
  }

  for _, dependency := range dependencies {
    if dependency.Change == entity.Replace && !isNetworkAddress(dependency.Address) {
      return entity.Replace, nil
    }
  }

  if len(changes) > 0 {
    return entity.Update, nil
  }

  for _, dependency := range dependencies {
    if isNetworkAddress(dependency.Address) {
      return entity.Update, nil
    }
  }

  return p.DefaultChanged.Changed(ctx, old, new, changes, dependencies)
}

The container’s Update moves the running container between networks, again working only from what it is told. c is the configuration as it is now and changes is how it got there, so putting back each network change’s Before rebuilds the network blocks the container had after the last apply, without asking Docker. It disconnects the networks that went away or whose aliases changed, connects the new ones, connects again every network whose dependency was replaced, and reads the container’s new address once if anything moved:

func (p *containerProvider) Update(ctx context.Context, c *entities.Container, changes []entity.PropertyChange, dependencies []entity.DependencyChange) (*entities.Container, error) {
  previous, err := previousAttachments(c.Networks, changes)
  if err != nil {
    return nil, fmt.Errorf("unable to work out the previous networks of container %s: %w", c.Meta.Name, err)
  }

  moved := false

  for _, attachment := range previous {
    current, ok := findAttachment(c.Networks, attachment.Name)
    if ok && equalStrings(current.Aliases, attachment.Aliases) {
      continue
    }

    // a network that was removed or rebuilt has already detached it
    err := p.tasks.DisconnectNetwork(ctx, attachment.Name, c.DockerID)
    if err != nil && !errors.Is(err, containers.ErrNotFound) {
      return nil, fmt.Errorf("unable to disconnect container %s from network %s: %w", c.Meta.Name, attachment.Name, err)
    }

    plugins.Logger(ctx).Info("disconnected container from network", "name", c.Meta.Name, "network", attachment.Name)
    moved = true
  }

  connected := map[string]bool{}
  for _, attachment := range c.Networks {
    before, ok := findAttachment(previous, attachment.Name)
    if ok && equalStrings(before.Aliases, attachment.Aliases) {
      continue
    }

    if err := p.connect(ctx, c, attachment); err != nil {
      return nil, err
    }

    connected[attachment.Name] = true
    moved = true
  }

  for _, dependency := range dependencies {
    if dependency.Change != entity.Replace || !isNetworkAddress(dependency.Address) {
      continue
    }

    attachment, ok := findAttachment(c.Networks, networkName(dependency.Address))
    if !ok || connected[attachment.Name] {
      continue
    }

    if err := p.connect(ctx, c, attachment); err != nil {
      return nil, err
    }

    connected[attachment.Name] = true
    moved = true
  }

  if !moved {
    return c, nil
  }

  addresses, err := p.tasks.ContainerAddresses(ctx, c.DockerID)
  if err != nil {
    return nil, fmt.Errorf("unable to inspect container %s: %w", c.Meta.Name, err)
  }

  c.IPAddress = firstAddress(c, addresses)

  return c, nil
}

The hot swap has a companion in the network’s Destroy. Docker refuses to remove a network that still has containers attached, and a replaced network is destroyed before its dependents are updated, so Destroy inspects the network and force-disconnects every attached container before removing it. The container’s Update then attaches it to the network that replaces it, which is why it connects again to every replaced network.

The example’s config/alt-subnet directory is its main configuration with the network’s subnet changed from 10.42.0.0/24 to 10.42.0.0/23:

docker "network" "app" {
  subnet = "10.42.0.0/23"
}

The container, still on docker.network.app and mounting template.init.destination as its init_script, is unchanged. Applying it after the main configuration:

1. The network is asked first. Its subnet changed, so it answers replace.
2. The init template’s variables read only the network’s name, which does not change, so it is unchanged.
3. The container is asked with no changes of its own and docker.network.app listed as a replaced dependency. A network is not a reason to replace it, so it answers update.
4. The template that reads the container’s ip_address cannot know the address until the container is updated, so it is planned as an update.
5. The network is destroyed, force-disconnecting the container, and created on the new subnet. The container’s Update is told the network was replaced and connects it again, keeping the same container and its ID, then the template is updated with its address.

How the plan shows it

The plan says which entities are replaced or updated and why, in a comment above each header. This is the real plan for the change above:

  # docker.container.web will be updated because docker.network.app is replaced
  ~ docker "container" "web" {}

  # docker.network.app will be replaced, it cannot be updated in place
-/+ docker "network" "app" {
      ~ subnet = "10.42.0.0/24" -> "10.42.0.0/23"
    }

  # template.welcome will be updated
  ~ template "welcome" {
      ~ variables["address"] = "10.42.0.2" -> (known after apply)
    }

Diff: 0 to create, 2 to update, 1 to replace, 0 to delete, 1 unchanged.

The container has no changes of its own, so the plan names the dependencies behind its update. A replacement’s reason is one of three:

The plugins example’s config/modified-init shows the second: moving the init script’s destination replaces template.init, and the container that mounts it answers replace, # docker.container.web will be replaced because template.init is replaced.

In a diff, a replaced entity’s Reason and ReplacedDeps, and an updated entity’s Dependencies, carry the same information as data. A plan after the apply shows Diff: no changes, 4 unchanged. See Diffs for reading a plan from Go and rendering it.

Run it yourself with make replace in example/plugin (it needs Docker): it applies the main configuration, shows the plan for config/alt-subnet, applies it and destroys everything again. make swap, make rebuild-init and make remove-network walk through the other variants the same way.

External plugins

A plugin running as a separate binary makes the same decisions over the plugin protocol, and is told exactly what an in-process one is. The requests carry the changed settings and the dependency list, and the response carries the answer:

enum Change {
  CHANGE_NO_CHANGE = 0;
  CHANGE_UPDATE = 1;
  CHANGE_REPLACE = 2;
}

message DependencyChange {
  string address = 1;
  Change change = 2;
}

enum StepKind {
  STEP_KIND_ATTRIBUTE = 0;
  STEP_KIND_INDEX = 1;
  STEP_KIND_KEY = 2;
}

message PathStep {
  StepKind kind = 1;
  string attribute = 2;
  int64 index = 3;
  string key = 4;
}

// PropertyChange is one setting that differs between the last apply and the
// new configuration, see entity.PropertyChange. before and after hold JSON
// values; empty means the value is absent.
message PropertyChange {
  repeated PathStep path = 1;
  bytes before = 2;
  bytes after = 3;
  bool unknown = 4;
  bool sensitive = 5;
}

message ChangedRequest {
  string entity_type = 1;
  string entity_sub_type = 2;
  bytes old_entity_data = 3;
  bytes new_entity_data = 4;
  repeated DependencyChange dependencies = 5;
  repeated PropertyChange changes = 6;
}

message ChangedResponse {
  // was: bool changed
  reserved 1;
  string error = 2;
  Change change = 3;
}

message UpdateRequest {
  string entity_type = 1;
  string entity_sub_type = 2;
  bytes entity_data = 3;
  repeated PropertyChange changes = 4;
  repeated DependencyChange dependencies = 5;
}

A plugin author writes the same Go Changed and Update either way; the plugin SDK does the translation. An unknown Change value is an error on either side. Each before and after travels as the JSON bytes of the value, encoded on its own rather than through the change’s masking JSON form, so sensitive values arrive real and every value decodes to the same plain JSON value an in-process provider receives.

Plugins built against an earlier protocol must be rebuilt. One built before Changed answered an outcome rather than a yes or no reads as no change, so its entities would never be updated, and one built before the changes were added is never told them. A provider that only embeds DefaultChanged needs its Update given the new parameters and a rebuild.