extends on a service from a file with include

[polarathene]

Description

# includes/base.yaml
services:
  example:
    image: alpine

# includes/with-hello.yaml
services:
  example:
    environment:
      HELLO: ${WORLD:-world}
    command: '/bin/sh -c "echo $${HELLO}"'

---

This include works if the latter invalid section is removed:

# compose.include.yaml

# Composed service from modular snippets:
include:
  - path:
      - includes/base.yaml
      - includes/with-hello.yaml

# Invalid (service name cannot be found):
services:
  my-variant:
    extends:
      service: example

The extend attempt is invalid as include hasn't been processed/resolved when the extends logic runs? Yet it's valid to use depends_on: [example], that feels a tad inconsistent.

Attempting to workaround that I assumed a separate compose file that extends compose.include.yaml to get merged example service to target might work:

# compose.yaml

# Invalid (service name cannot be found):
services:
  my-service:
    extends:
      file: compose.include.yaml
      service: example

This also fails. Perhaps the extends.service logic is failing because it only understands to check a YAML file for services.<NAME> exists, which occurs without resolving include? While depends_on is working presumably as a later stage/step once the compose config is fully generated?

---

I attempted to try another approach with variable interpolation on the service name key and pair that with include.path.env_file to have a dynamic service name to avoid the conflicting/overlapping service name if using multiple include entries instead of extends.service. That doesn't work as variable interpolation isn't available for YAML keys.

Similarly fragments and extensions are locked to the scope of the same YAML document, so I cannot leverage include or merge (multiple CLI -f) to share modular config in a DRY manner.

include is great, for layering multiple compose configs into a single document which I wanted for demonstrating permutations/variants to demonstrate various features for a service. It is rather dependent upon the same service name however, which wouldn't be an issue if extends were compatible in a way that the composed include service could be imported under a different service name, using env_file of include to adjust the dynamic configuration differences (or overriding via extends).

It would seem that you'd need to use separate compose.yaml files instead and start each separately, which is a bit more work when you need say networks.default shared between them, requiring the network to be managed separately (or have one define it, while the other uses external).

With this limitation around the service name, I don't think I can easily use environment / configs of services with includes, but would need to use env_file / volumes instead to refer to external files of the same content as that would avoid the dependency on service name.

[polarathene]

Real-world example

This should better illustrate what I attempted to do, although this won't work as-is.

I link two references of earlier less granular approaches (that do work) at the end of this response.

# includes/base.yaml

services:
  dms:
    image: ${WITH_IMAGE:-ghcr.io/docker-mailserver/docker-mailserver:latest}
    hostname: mail.example.test

Due to file.yaml having two variants of it's own, this may be better as separate YAML files, but the overall intent was for the auth component to be interchangeable (LDAP is a more verbose config alternative).

# includes/auth/file.yaml

services:
  dms:
    configs:
      - source: dms-accounts-${WITH_ACCOUNT_DOMAIN:-example}
        target: /tmp/docker-mailserver/postfix-accounts.cf

configs:
  dms-accounts-example:
    content: |
      [email protected]|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
      [email protected]|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.

  # Same as above, only change is `example.test` => `remote.test`:
  # Presumably while `include` can use ENV for dynamic config, the resource still needs a unique name
  # if multiple variants would be merged.
  dms-accounts-remote:
    content: |
      [email protected]|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
      [email protected]|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.

# includes/features/getmail.yaml

services:
  dms:
    environment:
      ENABLE_GETMAIL: 1
      # Reduce the polling frequency to 1 minute for quicker testing:
      GETMAIL_POLL: 1
    configs:
      - source: getmail-jane
        target: /tmp/docker-mailserver/getmail/jane.cf

configs:
  # Basic getmail config to retrieve mail from an account at another mail server via IMAP credentials:
  getmail-jane:
    content: |
      [retriever]
      type = SimpleIMAPSSLRetriever
      server = mail.remote.test
      username = [email protected]
      password = secret

      [destination]
      type = MDA_external
      path = /usr/lib/dovecot/deliver
      allow_root_commands = true
      arguments = ("-d","[email protected]")

An alternative feature similar to getmail.yaml that would be interchangeable:

# includes/features/fetchmail.yaml

services:
  dms:
    environment:
      ENABLE_FETCHMAIL: 1
      # Reduce the polling frequency to 10 seconds for quicker testing:
      FETCHMAIL_POLL: 10
    configs:
      - source: fetchmail
        target: /tmp/docker-mailserver/fetchmail.cf

configs:
  # Basic fetchmail config to retrieve mail from an account at another mail server via IMAP credentials:
  fetchmail:
    content: |
      poll 'mail.remote.test' proto imap
        user '[email protected]'
        pass 'secret'
        is '[email protected]'
        no sslcertck

NOTE: Opted to treat include as a new key for extends here.

• This still won't be valid due to configs which isn't part of extends feature scope.
• Expressing the equivalent with include would be extra YAML files, which would look a bit more awkward.

# compose.yaml

# A composition of modular snippets instead of 
services:
  dms:
    extends:
      service: dms
      include:
        - path:
            - includes/base.yaml
            - includes/auth/${WITH_AUTH:-file}.yaml
            - includes/features/${WITH_FEATURE:-getmail}.yaml

  # Another instance of DMS to act as a third-party MTA for this example:
  remote-mta:
    hostname: mail.remote.test
    extends:
      service: dms
      include:
        - env_file: remote.env
          path:
            - includes/base.yaml
            - includes/auth/${WITH_AUTH:-file}.yaml

# remote.env

# NOTE: This file is required as `env_file` doesn't support inlined content,
# nor any `environment` equivalent supported.

WITH_ACCOUNT_DOMAIN=remote

# compose.override.yaml

# NOTE: Certs were provisioned only once via a separate `compose.yaml`.
# Various images could produce the files (eg: certbot, traefik, caddy, smallstep/step-ca)
volumes:
  custom-certs:
    name: tls-remote-test
    external: true

services:
  # If needed to support verifying TLS connection, add private CA to trust store:
  dms:
    configs:
      - source: dms-trust-custom-ca
        target: /tmp/docker-mailserver/user-patches.sh
    volumes:
      - custom-certs:/srv/custom-certs:ro

  # Configure with TLS support:
  remote-mta:
    environment:
      SSL_TYPE: manual
      SSL_CERT_PATH: /srv/custom-certs/remote.test/cert.pem
      SSL_KEY_PATH: /srv/custom-certs/remote.test/key.pem
    volumes:
      - custom-certs:/srv/custom-certs/:ro

configs:
  dms-trust-custom-ca:
    content: |
      #!/bin/bash
      cp /srv/custom-certs/ca/cert.pem /usr/local/share/ca-certificates/smallstep-ca.crt
      update-ca-certificates

---

This example above was derived from:

• Layered Getmail compose example that only uses include. Less granular than above, didn't require a separate compose.override.yaml.
• An earlier single file compose example for fetchmail.

Rather than the single file approach, as I add more examples I thought it'd be nicer to modularize, but that gets a bit more complicated with more than one instance of the service in an example. Users could then use docker compose config to get a single file that they could use as a base which was an additional bonus to modular config, and we could leverage this for our test suite.

This is probably niche, so feel free to close. It might be better suited to leaning on other tooling focused on templating.

[ndeloof]

depends_on is only considered as compose creates containers, after the compose model has been loaded, while extends is processed during parsing. Using extends with file will let you declare the compose file to be used to find the service. You just need to have with-hello.yaml to also declare it extends base.yaml so the parser can find the whole hierarchy, vs relying on yaml overrides.

[polarathene]

Using extends with file will let you declare the compose file to be used to find the service.

That's what I originally expected and wanted, thinking that the file boundary would have been sufficient for extending a service file that has include, but it's not possible as the service is not discoverable to extends at that point, nor can I make one within the file with include as it would conflict with the service that include merges 🤷‍♂

---

You just need to have with-hello.yaml to also declare it extends base.yaml so the parser can find the whole hierarchy, vs relying on yaml overrides.

That doesn't really work when there isn't a sequential hierarchy, such as with layering multiple feature snippets. That's where include is useful, but the service name cannot be adjusted, thus a 2nd instance cannot be introduced (which is how I tried to use extends).

See the larger example I provided where a 2nd instance diverges.

---

This is perhaps too niche of a use-case for Compose to support, I don't mind looking into other approaches. I may just need to accept that the modularity can't be as DRY as I was aiming for, or rely on additional tooling.

I think for example I could use docker compose config per service variant instead of using extends, and then yq to rename the service/resources, allowing to merge the files into the desired compose.yaml.