Share configuration settings between docker-compose files



    Docker compose supports 2 methods for sharing configuration settings:

    1. Expand a docker compose file using multiple docker compose files
    2. Expand each service with the school extends field (Only applicable to compose docker versions 2.1 and above)

    How to use multiple docker compose files


    By using multiple docker compose files, it allows you to customize the docker compose configuration settings for different environments or business streams.

    Mechanism works when using multiple docker compose files


    By default, docker compose reads 2 files: docker-compose.yml and docker-compose.override.yml. By convention, then docker-compose.yml will contain base configuration configurations. And the file overwrite docker-compose.override.yml will contain configurations used to override existing service configurations or even define new services.

    If a service is defined in both files, the docker compose will merge the service-related configurations according to the principles described in https://docs.docker.com/compose/extends/#adding. -and-overriding-configuration

    To use more files to overwrite, or to use an overwrite file with a different name than the standard, you can use additional parameters -f to define a list of files to be used. Docker compose consolidates the configurations in these files in the order they are defined at the command line. You can see more information about the parameter -f at https://docs.docker.com/compose/reference/overview/

    When you use multiple configuration files, you must make sure that all the paths of these files are relative to the base compose file (the file is listed first right after). -f in the command line).

    Example of using multiple docker compose files


    There are two fairly common cases for using multiple docker compose files:

    1. Compose docker compose configurations for different environments
    2. Run the admin tasks

    Configuration for different environments


    One of the common cases when using multiple docker compose files is to change the docker compose configuration for an environment like production (or staging, CI). To support this, you can break down the docker compose configuration into quite a few files:

    Start with a base file that defines the standard configuration for services.

    docker-compose.yml

    web:
      image: example/my_web_app:latest
      depends_on:
        - db
        - cache
    
    db:
      image: postgres:latest
    
    cache:
      image: redis:latest

    In a dev environment, you need to configure to expose a few ports connected to the host, need to mount the code or build the web image

    docker-compose.override.yml

    web:
      build: .
      volumes:
        - '.:/code'
      ports:
        - 8883:80
      environment:
        DEBUG: 'true'
    
    db:
      command: '-d'
      ports:
        - 5432:5432
    
    cache:
      ports:
        - 6379:6379

    When you execute run the command docker-compose up then the configuration settings will be overwritten automatically

    Now, we create docker compose file for production environment. Therefore, you need to create a log file for the production environment

    docker-compose.prod.yml

    web:
      ports:
        - 80:80
      environment:
        PRODUCTION: 'true'
    
    cache:
      environment:
        TTL: '500'

    To deploy to this production environment, we need to run the command

    docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

    This command will deploy deploy 3 services using the internal configuration docker-compose.yml and docker-compose.prod.yml

    Configuration for administrative tasks


    Another common case is running administrative tasks in a Compose app, such as backing up a running database.

    docker-compose.yml

    web:
      image: example/my_web_app:latest
      depends_on:
        - db
    
    db:
      image: postgres:latest

    docker-compose.admin.yml Implementing a new service to run database export or backup.

    dbadmin:
      build: database_admin/
      depends_on:
        - db

    To create a regular environment we just need to run the command docker-compose up -d . To run a database backup, we have to embed it docker-compose.admin.yml

    docker-compose -f docker-compose.yml -f docker-compose.admin.yml
        run dbadmin db-backup

    How to expand to each service


    Key word extends Allows sharing shared configurations between different files, or even different projects. How to expand the service is really useful if you have several services that reuse a set of options configurations. Using extends You can define a set of services configurations in one place and reference it from anywhere.

    Note that volumes_from and depends_on Never be shared between services using extends . These exceptions exist to avoid potential dependencies; You always have to define volumes_from inside each service. This ensures that dependencies between services are completely visible when reading in the current configuration file. This explicit definition ensures that changes in reference files will not break anything.

    The mechanism of action of expanding services


    When defining any service in docker-compose.yml , you can declare extended service as follows:

    web:
      extends:
        file: common-services.yml
        service: webapp

    This directive tells Compose to reuse the configuration webapp The service is defined internally common-services.yml file. Assuming that common-services.yml would look like this:

    webapp:
      build: .
      ports:
        - "8000:8000"
      volumes:
        - "/data"

    In this case, you will get the same result as you wrote in the main text docker-compose.yml with the same configuration values build port volumes right below web

    Moreover, you can redefine the settings that are set in docker-compose.yml :

    web:
      extends:
        file: common-services.yml
        service: webapp
      environment:
        - DEBUG=1
      cpu_shares: 5
    
    important_web:
      extends: web
      cpu_shares: 10

    You can also write other services and associate web services with them:

    web:
      extends:
        file: common-services.yml
        service: webapp
      environment:
        - DEBUG=1
      cpu_shares: 5
      depends_on:
        - db
    db:
      image: postgres

    Examples of use cases


    It is useful to extend each service by direction when you have multiple services reusing a shared configuration set. For example, below is a Compose app with 2 services: a web app and a queue worker. Both of these services use the same codebase and share many shared configurations.

    in 1 common.yml We define the shared configuration:

    app:
      build: .
      environment:
        CONFIG_FILE_PATH: /code/config
        API_KEY: xxxyyy
      cpu_shares: 5

    in 1 docker-compose.yml We define specific services that use shared configurations:

    webapp:
      extends:
        file: common.yml
        service: app
      command: /code/run_web_app
      ports:
        - 8080:8080
      depends_on:
        - queue
        - db
    
    queue_worker:
      extends:
        file: common.yml
        service: app
      command: /code/run_worker
      depends_on:
        - queue

    How to add new and override the configuration


    Compose replicates the configuration from the original service to the local service. If a configuration setting is defined at both the original service and the local service, the configuration value at the local value will replace the original value.

    The configuration example has a single value image command mem_limit then the new value will replace the old value:

    # original service
    command: python app.py
    
    # local service
    command: python otherapp.py
    
    # result
    command: python otherapp.py

    For configurations with multiple type values ports expose external_links dns dns_search tmpfs , Compose will merge both value sets from original service and local service:

    # original service
    expose:
      - "3000"
    
    # local service
    expose:
      - "4000"
      - "5000"
    
    # result
    expose:
      - "3000"
      - "4000"
      - "5000"

    In case environment labels volumes devices Compose consolidates configuration values ​​in order of priority. Eg environment and labels

    # original service
    environment:
      - FOO=original
      - BAR=original
    
    # local service
    environment:
      - BAR=local
      - BAZ=local
    
    # result
    environment:
      - FOO=original
      - BAR=local
      - BAZ=local

    Eg volumes and devices

    # original service
    volumes:
      - ./original:/foo
      - ./original:/bar
    
    # local service
    volumes:
      - ./local:/bar
      - ./local:/baz
    
    # result
    volumes:
      - ./original:/foo
      - ./local:/bar
      - ./local:/baz

    No comments