add exec, run and scale, and the agent skill
The skill documents how to use myos: what to check before touching a host stack, how a reference resolves, what the three kinds of stack mean, and where each failure comes from. Every command it shows was run against the fixtures before being written down, which is how exec, run and scale turned out to be missing. AGENTS.md covers the other side: how to change myos without breaking the deployments that already run it.
This commit is contained in:
Yann Autissier
@@ -0,0 +1,91 @@
|
||||
---
|
||||
name: myos
|
||||
description: Deploy, inspect and troubleshoot docker compose stacks on a server or a fleet with the myos CLI (Make Your Own Stack). Use when asked to install myos, to start/stop/debug a stack (a host singleton, an application, a user stack), to add a stack to the catalogue, or to work on the myos framework itself.
|
||||
---
|
||||
|
||||
# myos
|
||||
|
||||
myos runs docker compose stacks. It decides which compose files to load, under
|
||||
which project name, with which variables. Everything else is docker.
|
||||
|
||||
## Is this the right tool
|
||||
|
||||
Use it when the repository or the host has one of:
|
||||
- a `stack/` directory holding compose files,
|
||||
- a `Makefile` that includes `make/include.mk`,
|
||||
- `/etc/conf.d/myos` or `/etc/default/myos`,
|
||||
- the `myos` command on PATH.
|
||||
|
||||
Otherwise call `docker compose` directly.
|
||||
|
||||
## Look before you touch
|
||||
|
||||
```sh
|
||||
myos doctor # requirements, config layers, stack path
|
||||
myos ls # stacks reachable from here
|
||||
myos ls --groups # groups and what they expand to
|
||||
myos env # resolved variables
|
||||
myos -n up host # print what `up` would run, without running it
|
||||
```
|
||||
|
||||
**Always run `myos -n` before `up`, `down` or `restart` on a `host/` stack.**
|
||||
Host stacks bind privileged ports and are shared by everything on the machine:
|
||||
taking `host/fabio` down drops every site the machine serves.
|
||||
|
||||
## Working loop
|
||||
|
||||
```sh
|
||||
myos ls # find the stack
|
||||
myos -n config <stack> # check the file list and the project name
|
||||
myos up <stack> # create and start
|
||||
myos ps <stack> # what is running
|
||||
myos logs <stack> # follow the logs
|
||||
myos exec <stack> -- sh # a shell in the service named after the stack
|
||||
myos down <stack> # remove the containers
|
||||
```
|
||||
|
||||
## Naming a stack
|
||||
|
||||
A reference is `[<group>/]<name>[:<version>]`, or a path.
|
||||
|
||||
```sh
|
||||
myos up # the current directory, when it holds a compose file
|
||||
myos up host # a group: expands to host/consul host/fabio ...
|
||||
myos up host/fabio # one stack
|
||||
myos up postgres:9.6 # the 9.6 overlay of the postgres stack
|
||||
myos -C /opt/app up # somewhere else
|
||||
myos -e master up # in another environment
|
||||
```
|
||||
|
||||
Stacks are looked up along the stack path, project first:
|
||||
`./stack`, `../stack`, `~/.local/share/myos/stack`, `/usr/local/share/myos/stack`,
|
||||
`/usr/share/myos/stack`. `myos doctor` prints the resolved path.
|
||||
|
||||
## Three kinds of stack
|
||||
|
||||
| kind | reference | compose project | meaning |
|
||||
|---|---|---|---|
|
||||
| host | `host/*` | the hostname | one per machine: binds 80/443, consul, certbot. Shared. Treat as production. |
|
||||
| user | `User/*` | the user identity | one per user: ssh-agent, personal services |
|
||||
| app | anything else | `<user>-<env>-<app>` | many per machine, one per user and environment |
|
||||
|
||||
Set `MYOS_PROJECT_FORMAT=user-app-env` on a deployment created before the
|
||||
rename, otherwise its containers and volumes look like a different project.
|
||||
See `references/conventions.md`.
|
||||
|
||||
## Rules
|
||||
|
||||
- Never run `myos clean` on a host stack: it removes images **and volumes**,
|
||||
including the certificates.
|
||||
- Secrets belong in a file outside the repository, never in a compose file.
|
||||
- A stack of the catalogue is shared: change it in `myos-stacks`, not in place
|
||||
on a server.
|
||||
- Before upgrading myos on a machine that already runs stacks, pin the naming
|
||||
(`references/conventions.md`), or every project gets a new name.
|
||||
|
||||
## More
|
||||
|
||||
- `references/commands.md` — every command, and its equivalent in the old make targets
|
||||
- `references/conventions.md` — file layout, naming, networks, variables, tags
|
||||
- `references/troubleshooting.md` — what each failure means and how to fix it
|
||||
- `references/authoring.md` — writing a stack, and working on myos itself
|
||||
@@ -0,0 +1,74 @@
|
||||
# Writing a stack, and working on myos
|
||||
|
||||
## A new stack in the catalogue
|
||||
|
||||
```
|
||||
stack/<name>/<name>.yml the services
|
||||
stack/<name>/<name>.local.yml what only makes sense on a workstation (published ports…)
|
||||
stack/<name>/<name>.labels.yml the registrator labels, so routing stays optional
|
||||
stack/<name>/.env.dist the variables it expects, with defaults
|
||||
stack/<name>/README.md what it is and what it needs
|
||||
```
|
||||
|
||||
Rules that keep a stack reusable:
|
||||
- no `container_name`, except in a `host/` stack: it prevents scaling and
|
||||
collides between users,
|
||||
- no fixed host port outside `host/`; publish through the load balancer,
|
||||
- reference variables with a default: `${POSTGRES_VERSION:-16}`,
|
||||
- attach to `private` to be reachable by the other stacks of the user, to
|
||||
`public` to be routed,
|
||||
- name volumes, never bind-mount an absolute path.
|
||||
|
||||
Check it before committing:
|
||||
|
||||
```sh
|
||||
myos -n config <name> # the file list and the project
|
||||
myos config <name> # the rendered yaml
|
||||
```
|
||||
|
||||
## A group
|
||||
|
||||
```sh
|
||||
# stack/<group>.env
|
||||
mygroup=<name> other/<name>
|
||||
```
|
||||
|
||||
Lowercase, and it may name other groups.
|
||||
|
||||
## Working on myos itself
|
||||
|
||||
```sh
|
||||
make test # unit + golden, against both engines, with a mocked docker
|
||||
make test-golden # golden only
|
||||
make golden-record # re-record the golden expectations from the make engine
|
||||
make lint # shellcheck
|
||||
```
|
||||
|
||||
Layout:
|
||||
|
||||
```
|
||||
bin/myos argument parsing, configuration, dispatch
|
||||
lib/core.sh logging, exit codes, dry run
|
||||
lib/str.sh strings and version comparison
|
||||
lib/naming.sh project names, networks, user identity
|
||||
lib/stack.sh stack path, references, overlays, groups
|
||||
lib/config.sh dotenv, variables of the compose files
|
||||
lib/compose.sh finding and calling docker compose
|
||||
lib/tags.sh fabio tags
|
||||
lib/cmd/<x>.sh one file per command
|
||||
share/compose/ the networks and volumes overlays myos provides
|
||||
spec/ shellspec
|
||||
```
|
||||
|
||||
Adding a command: write `lib/cmd/<name>.sh` defining `myos_cmd_<name>`, add it
|
||||
to the usage text in `bin/myos`, and cover it in `spec/unit`.
|
||||
|
||||
Constraints:
|
||||
- POSIX shell, no bashisms: it has to run under the bash 3.2 of macOS and the
|
||||
ash of Alpine. `make test` runs on both.
|
||||
- No `a-z` ranges in a `case` pattern: under a dictionary collation such as
|
||||
`fr_FR` they also match uppercase. Use `[:lower:]`.
|
||||
- A function that has to return several values takes them out through printf,
|
||||
not through a global: a caller inside `$( )` would lose the global.
|
||||
- Changing what a command prints means updating `spec/golden/expected*/` and
|
||||
explaining the change in `spec/golden/DELTAS.md`.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Commands
|
||||
|
||||
```
|
||||
myos [options] <command> [stack...] [VAR=value...] [-- args...]
|
||||
```
|
||||
|
||||
| option | effect |
|
||||
|---|---|
|
||||
| `-C DIR` | work in DIR instead of the current directory |
|
||||
| `-e ENV` | environment: picks `.env.<env>` and the `<stack>.<env>.yml` overlays |
|
||||
| `-n`, `--dry-run` | print the commands instead of running them |
|
||||
| `-v`, `--verbose` | say what myos is doing |
|
||||
| `-d`, `--debug` | print every command |
|
||||
|
||||
| command | effect |
|
||||
|---|---|
|
||||
| `up` | create and start; creates the external networks first |
|
||||
| `down` | remove the containers |
|
||||
| `start` / `stop` / `restart` | on the existing containers |
|
||||
| `ps` | what is running |
|
||||
| `logs` | follow the logs |
|
||||
| `config` | the resolved compose file |
|
||||
| `exec` | run a command in a running service |
|
||||
| `run` | run it in a new container, removed afterwards |
|
||||
| `scale` | `myos scale <stack> SERVICE=<name> NUM=<n>` |
|
||||
| `build` / `pull` | images |
|
||||
| `ls [--groups]` | the stacks and groups myos can see |
|
||||
| `env [VAR...]` | resolved variables |
|
||||
| `doctor` | check the installation |
|
||||
| `version` | the myos version |
|
||||
|
||||
Anything after `--` goes to docker compose:
|
||||
|
||||
```sh
|
||||
myos logs host/fabio -- --tail 20
|
||||
myos up postgres -- --force-recreate
|
||||
```
|
||||
|
||||
`exec` and `run` take the service from the stack name, since most stacks name
|
||||
their main service after themselves. `SERVICE=` picks another one:
|
||||
|
||||
```sh
|
||||
myos exec host/consul -- consul members # service consul, command "consul members"
|
||||
myos exec host/fabio SERVICE=fabio -- sh
|
||||
myos run postgres -- psql -l
|
||||
```
|
||||
|
||||
## Exit codes
|
||||
|
||||
| code | meaning |
|
||||
|---|---|
|
||||
| 0 | fine |
|
||||
| 1 | the command failed |
|
||||
| 2 | bad invocation, or an unknown command |
|
||||
| 3 | stack not found (the message prints the search path) |
|
||||
| 4 | missing requirement, run `myos doctor` |
|
||||
|
||||
The make engine exited 0 on an unknown target, printing only a warning. The CLI
|
||||
does not: a typo is an error.
|
||||
|
||||
## Coming from the make targets
|
||||
|
||||
| make | myos |
|
||||
|---|---|
|
||||
| `make up STACK=host` | `myos up host` |
|
||||
| `make print-COMPOSE_FILE` | `myos env COMPOSE_FILE` |
|
||||
| `make host` | `myos up host` |
|
||||
| `make stack-host-config` | `myos config host` |
|
||||
| `make up@master` | `myos -e master up` |
|
||||
| `make exec SERVICE=php ARGS='ls'` | `myos exec <stack> -- php ls` |
|
||||
| `make DRYRUN=true up` | `myos -n up` |
|
||||
|
||||
`print-VAR`, `stack-<stack>-<command>` and `<command>@<env>` still work.
|
||||
A project `Makefile` that includes `make/include.mk` keeps working too.
|
||||
@@ -0,0 +1,113 @@
|
||||
# Conventions
|
||||
|
||||
## A stack is a directory of compose files
|
||||
|
||||
For stack `<name>` in environment `<env>`, myos loads whichever of these exist,
|
||||
in this order. A later file overrides the ones before it.
|
||||
|
||||
```
|
||||
<name>.yml the stack
|
||||
<name>.<env>.yml this environment only
|
||||
<env>/<name>.yml same, when the stack keeps a directory per environment
|
||||
<name>.<suffix>.yml an optional overlay, see below
|
||||
<name>.<suffix>.<env>.yml
|
||||
<name>.<version>.yml when the reference is <name>:<version>
|
||||
```
|
||||
|
||||
`docker-compose.yml` is read under the same rules, so a stack can keep an
|
||||
upstream `docker-compose.yml` untouched and add its own `<name>.yml` on top.
|
||||
|
||||
Then myos appends its own `share/compose/networks.yml`, and the
|
||||
`volumes.<suffix>.<env>.yml` of the enabled suffixes.
|
||||
|
||||
## Overlay suffixes
|
||||
|
||||
Every `COMPOSE_FILE_<X>` variable that is not `false` enables the suffix `<x>`.
|
||||
|
||||
| variable | default | loads |
|
||||
|---|---|---|
|
||||
| `COMPOSE_FILE_APP` | true | `<name>.app.yml` |
|
||||
| `COMPOSE_FILE_LABELS` | true | `<name>.labels.yml`, the registrator labels |
|
||||
| `COMPOSE_FILE_NETWORKS` | true | `<name>.networks.yml` |
|
||||
| `COMPOSE_FILE_SSH` | true | `<name>.ssh.yml` |
|
||||
| `COMPOSE_FILE_VOLUMES` | true | `<name>.volumes.yml` |
|
||||
| `COMPOSE_FILE_WWW`, `_DNS`, `_HOME`, `_LOG`, `_BACKUP` | false | the matching overlay, and the framework bind mount |
|
||||
|
||||
A value that is neither `true` nor `false` also enables `<suffix>.<value>`:
|
||||
`COMPOSE_FILE_WWW=nginx` loads `<name>.www.yml` **and** `<name>.www.nginx.yml`.
|
||||
|
||||
## Project names
|
||||
|
||||
| scope | project | why |
|
||||
|---|---|---|
|
||||
| `host/*` | `$HOSTNAME` | a singleton of the machine: one consul, one fabio on 80/443 |
|
||||
| `User/*` | the user identity derived from the mail address | one per user |
|
||||
| `cluster/*` | the stack name | one namespace per swarm |
|
||||
| anything else | `<user>-<env>-<app>` | one per user, environment and app |
|
||||
|
||||
`MYOS_PROJECT_FORMAT=user-app-env` restores the older order. **Any deployment
|
||||
created before the rename must set it**, in its `.env` or in
|
||||
`/etc/conf.d/myos`; otherwise the containers and volumes of the old project
|
||||
are orphaned and the stack comes back up empty.
|
||||
|
||||
## Networks
|
||||
|
||||
| network | name | scope |
|
||||
|---|---|---|
|
||||
| `default` | `_<project>` | the project. The leading underscore keeps it first alphabetically, so it is the first interface attached and service names never resolve across stacks. |
|
||||
| `private` | `<user>-<env>` | external, shared between the stacks of one user and environment |
|
||||
| `public` | `<hostname>` | external, where the load balancer reaches the services |
|
||||
|
||||
`myos up` creates the external ones when they are missing.
|
||||
|
||||
## Routing: registrator, consul, fabio
|
||||
|
||||
A service is published by labels, which registrator copies into consul, and on
|
||||
which fabio routes:
|
||||
|
||||
```yaml
|
||||
labels:
|
||||
- SERVICE_8000_NAME=${COMPOSE_SERVICE_NAME}-kong-8000
|
||||
- SERVICE_8000_TAGS=${SUPABASE_KONG_SERVICE_8000_TAGS:-urlprefix-supabase.localhost/*}
|
||||
- SERVICE_8000_CHECK_TCP=true
|
||||
- SERVICE_8001_IGNORE=true
|
||||
```
|
||||
|
||||
Registrator only sees ports that are `expose`d, and consul only routes a
|
||||
service whose check passes.
|
||||
|
||||
Tag variables follow `<STACK>_SERVICE_<port>_<KEY>`, with `PATH`, `OPTS`,
|
||||
`URIS`, `NAME`, and the fabio options `allow`, `auth`, `deny`, `prepend`,
|
||||
`proto`, `register`, `strip`:
|
||||
|
||||
```sh
|
||||
HOST_NGINX_SERVICE_443_PROTO='https tlsskipverify=true'
|
||||
DUNITER_V2S_SERVICE_9944_STRIP=/ws
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Layers, the last one wins:
|
||||
|
||||
```
|
||||
defaults < /etc/conf.d/myos, /etc/default/myos < ~/.config/myos/config
|
||||
< <workdir>/.env < <workdir>/.env.<env> < environment < VAR=value on the command line
|
||||
```
|
||||
|
||||
All of them are dotenv: `KEY=value`, `#` comments, optional quotes. They are
|
||||
parsed, never sourced, so a value may contain a `#` or a `$(...)` without
|
||||
breaking anything or being executed. The make engine included `.env` as a
|
||||
makefile, where both broke.
|
||||
|
||||
## Groups
|
||||
|
||||
A group is a lowercase name whose value lists stacks. It can live in a `.env`,
|
||||
in the environment, in `<path>/<group>.env`, or in a legacy `<group>.mk`:
|
||||
|
||||
```sh
|
||||
host=host/consul host/fabio host/registrator
|
||||
testing=drone/drone drone/gc
|
||||
```
|
||||
|
||||
Groups expand recursively. Only lowercase names are considered, so an
|
||||
environment variable can never be mistaken for a group.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Troubleshooting
|
||||
|
||||
Start with `myos doctor`, then `myos -n <command>` to see what would run.
|
||||
|
||||
## `stack not found: <name>` (exit 3)
|
||||
|
||||
The reference is not on the stack path, which the message prints.
|
||||
- `myos ls` shows what is reachable.
|
||||
- The catalogue may not be installed: `git clone <myos-stacks> /usr/local/share/myos`.
|
||||
- A stack of the project is only found from the project: use `myos -C /path/to/project`.
|
||||
|
||||
## `docker compose >= 2.24.4 not found` (exit 4)
|
||||
|
||||
Install the compose plugin, or a `docker-compose` binary of that version.
|
||||
myos no longer falls back to a compose image.
|
||||
|
||||
## `unknown command: <x>` (exit 2)
|
||||
|
||||
Check the spelling against `myos help`. The make engine accepted any target
|
||||
and exited 0 after printing a warning, so typos used to look like successes.
|
||||
|
||||
## The containers came back under a different name
|
||||
|
||||
The default project name changed from `<user>-<app>-<env>` to
|
||||
`<user>-<env>-<app>`. The old containers and volumes are still there, under the
|
||||
old project. Put `MYOS_PROJECT_FORMAT=user-app-env` in the `.env` of the
|
||||
deployment (or in `/etc/conf.d/myos`) and bring it up again.
|
||||
|
||||
Check first: `myos env COMPOSE_PROJECT_NAME` against `docker ps --format '{{.Names}}'`.
|
||||
|
||||
## `network <name> declared as external, but could not be found`
|
||||
|
||||
The `private` or `public` network is missing. `myos up` creates them; a bare
|
||||
`docker compose up` does not. Or create it by hand:
|
||||
`docker network create <user>-<env>`.
|
||||
|
||||
## A service is up but not routed
|
||||
|
||||
In order: the port must be `expose`d (registrator ignores what it cannot see),
|
||||
the labels must be on the service, the consul check must pass, and only then
|
||||
does fabio route the `urlprefix-` tag.
|
||||
|
||||
```sh
|
||||
myos config <stack> | grep -A5 labels # what the labels resolve to
|
||||
myos exec host/consul -- consul catalog services
|
||||
myos logs host/registrator
|
||||
```
|
||||
|
||||
## A variable is empty in the container
|
||||
|
||||
myos only passes the variables the compose files actually mention. Check with
|
||||
`myos env` and `myos config <stack>`. A variable set in a `.env` of another
|
||||
directory is not read: only the workdir's `.env` is.
|
||||
|
||||
## On macOS with Colima
|
||||
|
||||
The daemon lives in a VM: a bind mount only works for a path the VM shares, and
|
||||
`host.docker.internal` is the way back to the host. `myos doctor` prints the
|
||||
`DOCKER_HOST` in use.
|
||||
|
||||
## Something changed after upgrading myos
|
||||
|
||||
`spec/golden/DELTAS.md` in the myos repository lists every intentional
|
||||
difference between the make engine and the CLI, with the reason.
|
||||
Reference in New Issue
Block a user