App Specification

Contents

App Specification#

Overview#

The bfabric-app-runner allows you to define and execute applications in various environments, by configuring the application steps in a YAML configuration files, the so-called app specification.

App Specification Structure#

The specification can be provided in a YAML file with the following structure:

bfabric:
  app_runner: "0.0.17"  # or a git reference
  workflow_template_step_id: null  # optional
versions:
  - version: "1.0.0"
    commands:
      dispatch: ...
      process: ...
      collect: ...  # Optional

The top-level bfabric section contains metadata relevant to B-Fabric integration (the runner version to use and an optional workflow template step ID). The versions list defines one or more application versions, each with its own set of commands.

Commands#

Each app defines these core commands:

  • dispatch: Prepares input data. Called with: $workunit_ref $work_dir

  • process: Executes main logic. Called with: $chunk_dir

  • collect: (Optional) Organizes results. Called with: $workunit_ref $chunk_dir

Commands are discriminated by their type field.

pydantic model bfabric_app_runner.specs.app.commands_spec.CommandsSpec#

Defines the commands that are required to execute an app.

Show JSON schema
{
   "title": "CommandsSpec",
   "description": "Defines the commands that are required to execute an app.",
   "type": "object",
   "properties": {
      "dispatch": {
         "discriminator": {
            "mapping": {
               "docker": "#/$defs/CommandDocker",
               "exec": "#/$defs/CommandExec",
               "python_env": "#/$defs/CommandPythonEnv",
               "shell": "#/$defs/CommandShell"
            },
            "propertyName": "type"
         },
         "oneOf": [
            {
               "$ref": "#/$defs/CommandShell"
            },
            {
               "$ref": "#/$defs/CommandExec"
            },
            {
               "$ref": "#/$defs/CommandDocker"
            },
            {
               "$ref": "#/$defs/CommandPythonEnv"
            }
         ],
         "title": "Dispatch"
      },
      "process": {
         "discriminator": {
            "mapping": {
               "docker": "#/$defs/CommandDocker",
               "exec": "#/$defs/CommandExec",
               "python_env": "#/$defs/CommandPythonEnv",
               "shell": "#/$defs/CommandShell"
            },
            "propertyName": "type"
         },
         "oneOf": [
            {
               "$ref": "#/$defs/CommandShell"
            },
            {
               "$ref": "#/$defs/CommandExec"
            },
            {
               "$ref": "#/$defs/CommandDocker"
            },
            {
               "$ref": "#/$defs/CommandPythonEnv"
            }
         ],
         "title": "Process"
      },
      "collect": {
         "anyOf": [
            {
               "discriminator": {
                  "mapping": {
                     "docker": "#/$defs/CommandDocker",
                     "exec": "#/$defs/CommandExec",
                     "python_env": "#/$defs/CommandPythonEnv",
                     "shell": "#/$defs/CommandShell"
                  },
                  "propertyName": "type"
               },
               "oneOf": [
                  {
                     "$ref": "#/$defs/CommandShell"
                  },
                  {
                     "$ref": "#/$defs/CommandExec"
                  },
                  {
                     "$ref": "#/$defs/CommandDocker"
                  },
                  {
                     "$ref": "#/$defs/CommandPythonEnv"
                  }
               ]
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Collect"
      }
   },
   "$defs": {
      "CommandDocker": {
         "additionalProperties": false,
         "properties": {
            "type": {
               "const": "docker",
               "default": "docker",
               "title": "Type",
               "type": "string"
            },
            "image": {
               "title": "Image",
               "type": "string"
            },
            "command": {
               "title": "Command",
               "type": "string"
            },
            "entrypoint": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Entrypoint"
            },
            "engine": {
               "default": "docker",
               "enum": [
                  "docker",
                  "podman"
               ],
               "title": "Engine",
               "type": "string"
            },
            "env": {
               "additionalProperties": {
                  "type": "string"
               },
               "default": {},
               "title": "Env",
               "type": "object"
            },
            "mac_address": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Mac Address"
            },
            "mounts": {
               "$ref": "#/$defs/MountOptions",
               "default": {
                  "work_dir_target": null,
                  "read_only": [],
                  "writeable": [],
                  "share_bfabric_config": true
               }
            },
            "hostname": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Hostname"
            },
            "custom_args": {
               "default": [],
               "items": {
                  "type": "string"
               },
               "title": "Custom Args",
               "type": "array"
            }
         },
         "required": [
            "image",
            "command"
         ],
         "title": "CommandDocker",
         "type": "object"
      },
      "CommandExec": {
         "additionalProperties": false,
         "properties": {
            "type": {
               "const": "exec",
               "default": "exec",
               "title": "Type",
               "type": "string"
            },
            "command": {
               "title": "Command",
               "type": "string"
            },
            "env": {
               "additionalProperties": {
                  "type": "string"
               },
               "default": {},
               "title": "Env",
               "type": "object"
            },
            "prepend_paths": {
               "default": [],
               "items": {
                  "format": "path",
                  "type": "string"
               },
               "title": "Prepend Paths",
               "type": "array"
            }
         },
         "required": [
            "command"
         ],
         "title": "CommandExec",
         "type": "object"
      },
      "CommandPythonEnv": {
         "additionalProperties": false,
         "properties": {
            "type": {
               "const": "python_env",
               "default": "python_env",
               "title": "Type",
               "type": "string"
            },
            "pylock": {
               "format": "path",
               "title": "Pylock",
               "type": "string"
            },
            "command": {
               "title": "Command",
               "type": "string"
            },
            "python_version": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Python Version"
            },
            "local_extra_deps": {
               "default": [],
               "items": {
                  "format": "path",
                  "type": "string"
               },
               "title": "Local Extra Deps",
               "type": "array"
            },
            "env": {
               "additionalProperties": {
                  "type": "string"
               },
               "default": {},
               "title": "Env",
               "type": "object"
            },
            "prepend_paths": {
               "default": [],
               "items": {
                  "format": "path",
                  "type": "string"
               },
               "title": "Prepend Paths",
               "type": "array"
            },
            "refresh": {
               "default": false,
               "title": "Refresh",
               "type": "boolean"
            }
         },
         "required": [
            "pylock",
            "command"
         ],
         "title": "CommandPythonEnv",
         "type": "object"
      },
      "CommandShell": {
         "additionalProperties": false,
         "properties": {
            "type": {
               "const": "shell",
               "default": "shell",
               "title": "Type",
               "type": "string"
            },
            "command": {
               "title": "Command",
               "type": "string"
            }
         },
         "required": [
            "command"
         ],
         "title": "CommandShell",
         "type": "object"
      },
      "MountOptions": {
         "additionalProperties": false,
         "properties": {
            "work_dir_target": {
               "anyOf": [
                  {
                     "format": "path",
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Work Dir Target"
            },
            "read_only": {
               "default": [],
               "items": {
                  "maxItems": 2,
                  "minItems": 2,
                  "prefixItems": [
                     {
                        "format": "path",
                        "type": "string"
                     },
                     {
                        "format": "path",
                        "type": "string"
                     }
                  ],
                  "type": "array"
               },
               "title": "Read Only",
               "type": "array"
            },
            "writeable": {
               "default": [],
               "items": {
                  "maxItems": 2,
                  "minItems": 2,
                  "prefixItems": [
                     {
                        "format": "path",
                        "type": "string"
                     },
                     {
                        "format": "path",
                        "type": "string"
                     }
                  ],
                  "type": "array"
               },
               "title": "Writeable",
               "type": "array"
            },
            "share_bfabric_config": {
               "default": true,
               "title": "Share Bfabric Config",
               "type": "boolean"
            }
         },
         "title": "MountOptions",
         "type": "object"
      }
   },
   "additionalProperties": false,
   "required": [
      "dispatch",
      "process"
   ]
}

Config:
  • extra: str = forbid

  • populate_by_name: bool = True

  • validate_by_alias: bool = True

  • validate_by_name: bool = True

Fields:
field collect: Annotated[CommandShell | CommandExec | CommandDocker | CommandPythonEnv, Discriminator(discriminator=type, custom_error_type=None, custom_error_message=None, custom_error_context=None)] | None = None#

The app collect command, can be omitted if your process command already creates an outputs.yml file.

It will be called with arguments: $workunit_ref $chunk_dir.

field dispatch: Annotated[CommandShell | CommandExec | CommandDocker | CommandPythonEnv, Discriminator(discriminator=type, custom_error_type=None, custom_error_message=None, custom_error_context=None)] [Required]#

The app dispatch command.

It will be called with arguments: $workunit_ref $work_dir.

Constraints:
  • discriminator = type

field process: Annotated[CommandShell | CommandExec | CommandDocker | CommandPythonEnv, Discriminator(discriminator=type, custom_error_type=None, custom_error_message=None, custom_error_context=None)] [Required]#

The app process command.

It will be called with arguments: $chunk_dir.

Constraints:
  • discriminator = type

Direct commands#

commands:
  dispatch:
    type: "exec"
    command: "python prepare_data.py"

The command string is split by spaces using shlex.split().

pydantic model bfabric_app_runner.specs.app.commands_spec.CommandExec#

Show JSON schema
{
   "title": "CommandExec",
   "type": "object",
   "properties": {
      "type": {
         "const": "exec",
         "default": "exec",
         "title": "Type",
         "type": "string"
      },
      "command": {
         "title": "Command",
         "type": "string"
      },
      "env": {
         "additionalProperties": {
            "type": "string"
         },
         "default": {},
         "title": "Env",
         "type": "object"
      },
      "prepend_paths": {
         "default": [],
         "items": {
            "format": "path",
            "type": "string"
         },
         "title": "Prepend Paths",
         "type": "array"
      }
   },
   "additionalProperties": false,
   "required": [
      "command"
   ]
}

Config:
  • extra: str = forbid

Fields:
field command: str [Required]#

The command to run, will be split by shlex.split and is not an actual shell script.

field env: dict[str, str] = {}#

Environment variables to set before executing the command.

field prepend_paths: list[Path] = []#

A list of paths to prepend to the PATH variable before executing the command.

If multiple paths are specified, the first one will be the first in PATH, etc.

field type: Literal['exec'] = 'exec'#

Identifies the command type.

Docker Commands#

commands:
  process:
    type: "docker"
    image: "myapp:1.0.0"
    command: "/app/run.sh"
    env:
      APP_VERSION: "${app.version}"
    mounts:
      read_only:
        - ["/data/reference", "/app/reference"]
      writeable:
        - ["/data/results", "/app/results"]
pydantic model bfabric_app_runner.specs.app.commands_spec.CommandDocker#

Show JSON schema
{
   "title": "CommandDocker",
   "type": "object",
   "properties": {
      "type": {
         "const": "docker",
         "default": "docker",
         "title": "Type",
         "type": "string"
      },
      "image": {
         "title": "Image",
         "type": "string"
      },
      "command": {
         "title": "Command",
         "type": "string"
      },
      "entrypoint": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Entrypoint"
      },
      "engine": {
         "default": "docker",
         "enum": [
            "docker",
            "podman"
         ],
         "title": "Engine",
         "type": "string"
      },
      "env": {
         "additionalProperties": {
            "type": "string"
         },
         "default": {},
         "title": "Env",
         "type": "object"
      },
      "mac_address": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Mac Address"
      },
      "mounts": {
         "$ref": "#/$defs/MountOptions",
         "default": {
            "work_dir_target": null,
            "read_only": [],
            "writeable": [],
            "share_bfabric_config": true
         }
      },
      "hostname": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Hostname"
      },
      "custom_args": {
         "default": [],
         "items": {
            "type": "string"
         },
         "title": "Custom Args",
         "type": "array"
      }
   },
   "$defs": {
      "MountOptions": {
         "additionalProperties": false,
         "properties": {
            "work_dir_target": {
               "anyOf": [
                  {
                     "format": "path",
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Work Dir Target"
            },
            "read_only": {
               "default": [],
               "items": {
                  "maxItems": 2,
                  "minItems": 2,
                  "prefixItems": [
                     {
                        "format": "path",
                        "type": "string"
                     },
                     {
                        "format": "path",
                        "type": "string"
                     }
                  ],
                  "type": "array"
               },
               "title": "Read Only",
               "type": "array"
            },
            "writeable": {
               "default": [],
               "items": {
                  "maxItems": 2,
                  "minItems": 2,
                  "prefixItems": [
                     {
                        "format": "path",
                        "type": "string"
                     },
                     {
                        "format": "path",
                        "type": "string"
                     }
                  ],
                  "type": "array"
               },
               "title": "Writeable",
               "type": "array"
            },
            "share_bfabric_config": {
               "default": true,
               "title": "Share Bfabric Config",
               "type": "boolean"
            }
         },
         "title": "MountOptions",
         "type": "object"
      }
   },
   "additionalProperties": false,
   "required": [
      "image",
      "command"
   ]
}

Config:
  • extra: str = forbid

Fields:
field command: str [Required]#

The command to execute in the container.

field custom_args: list[str] = []#

Any custom CLI arguments to pass to the container engine.

field engine: Literal['docker', 'podman'] = 'docker'#

The container engine to use.

field entrypoint: str | None = None#

The entrypoint to use for the container (instead of the image’s default).

field env: dict[str, str] = {}#

Environment variables to set in the container.

field hostname: str | None = None#

The hostname to use for the container (instead of Docker’s default assignment).

field image: str [Required]#

The container image to run.

field mac_address: str | None = None#

The MAC address to use for the container (instead of Docker’s default assignment).

field mounts: MountOptions = MountOptions(work_dir_target=None, read_only=[], writeable=[], share_bfabric_config=True)#

Mount options for the container.

field type: Literal['docker'] = 'docker'#

Identifies the command type.

pydantic model bfabric_app_runner.specs.app.commands_spec.MountOptions#

Show JSON schema
{
   "title": "MountOptions",
   "type": "object",
   "properties": {
      "work_dir_target": {
         "anyOf": [
            {
               "format": "path",
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Work Dir Target"
      },
      "read_only": {
         "default": [],
         "items": {
            "maxItems": 2,
            "minItems": 2,
            "prefixItems": [
               {
                  "format": "path",
                  "type": "string"
               },
               {
                  "format": "path",
                  "type": "string"
               }
            ],
            "type": "array"
         },
         "title": "Read Only",
         "type": "array"
      },
      "writeable": {
         "default": [],
         "items": {
            "maxItems": 2,
            "minItems": 2,
            "prefixItems": [
               {
                  "format": "path",
                  "type": "string"
               },
               {
                  "format": "path",
                  "type": "string"
               }
            ],
            "type": "array"
         },
         "title": "Writeable",
         "type": "array"
      },
      "share_bfabric_config": {
         "default": true,
         "title": "Share Bfabric Config",
         "type": "boolean"
      }
   },
   "additionalProperties": false
}

Config:
  • extra: str = forbid

Fields:
field read_only: list[tuple[Path, Path]] = []#
field share_bfabric_config: bool = True#
field work_dir_target: Path | None = None#
field writeable: list[tuple[Path, Path]] = []#

Shell Commands (deprecated)#

The shell command type is deprecated in favor of exec. It splits the command string by spaces (not using shlex.split).

pydantic model bfabric_app_runner.specs.app.commands_spec.CommandShell#

Show JSON schema
{
   "title": "CommandShell",
   "type": "object",
   "properties": {
      "type": {
         "const": "shell",
         "default": "shell",
         "title": "Type",
         "type": "string"
      },
      "command": {
         "title": "Command",
         "type": "string"
      }
   },
   "additionalProperties": false,
   "required": [
      "command"
   ]
}

Config:
  • extra: str = forbid

Fields:
field command: str [Required]#

The command to run, will be split by spaces and is not an actual shell script.

field type: Literal['shell'] = 'shell'#

Identifies the command type.

Python Environment Commands#

Provisions a Python virtual environment from a pylock file and runs the command inside it. Environments can be cached for reuse across runs.

commands:
  process:
    type: "python_env"
    pylock: "requirements.pylock.toml"
    command: "python run.py"
    python_version: "3.12"
pydantic model bfabric_app_runner.specs.app.commands_spec.CommandPythonEnv#

Show JSON schema
{
   "title": "CommandPythonEnv",
   "type": "object",
   "properties": {
      "type": {
         "const": "python_env",
         "default": "python_env",
         "title": "Type",
         "type": "string"
      },
      "pylock": {
         "format": "path",
         "title": "Pylock",
         "type": "string"
      },
      "command": {
         "title": "Command",
         "type": "string"
      },
      "python_version": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Python Version"
      },
      "local_extra_deps": {
         "default": [],
         "items": {
            "format": "path",
            "type": "string"
         },
         "title": "Local Extra Deps",
         "type": "array"
      },
      "env": {
         "additionalProperties": {
            "type": "string"
         },
         "default": {},
         "title": "Env",
         "type": "object"
      },
      "prepend_paths": {
         "default": [],
         "items": {
            "format": "path",
            "type": "string"
         },
         "title": "Prepend Paths",
         "type": "array"
      },
      "refresh": {
         "default": false,
         "title": "Refresh",
         "type": "boolean"
      }
   },
   "additionalProperties": false,
   "required": [
      "pylock",
      "command"
   ]
}

Config:
  • extra: str = forbid

Fields:
field command: str [Required]#

The command to run, will be split by shlex.split and is not an actual shell script.

field env: dict[str, str] = {}#

Environment variables to set before executing the command.

field local_extra_deps: list[Path] = []#

Additional dependencies to install into the environment.

Each entry should be a path to a wheel, sdist, or local package directory. These will be installed into the environment with uv pip install –no-deps after the main requirements. No dependency resolution will be performed for these, so their dependencies should already be present in the environment (typically specified in the pylock file).

field prepend_paths: list[Path] = []#

A list of paths to prepend to the PATH variable before executing the command.

If multiple paths are specified, the first one will be the first in PATH, etc.

field pylock: Path [Required]#

Path to the Pylock file that specifies the environment to use.

field python_version: str | None = None#

The Python version to use.

field refresh: bool = False#

When True, forces provisioning of an ephemeral environment that doesn’t affect the cache.

This creates a temporary environment for the single execution and doesn’t modify or reuse the cached environment. This ensures that refresh operations always start from a clean state and don’t leave broken environments in the cache if provisioning fails.

field type: Literal['python_env'] = 'python_env'#

App Versions#

The AppVersionMultiTemplate class defines app versions in two ways:

  1. Single string: version: "1.0.0"

  2. List of strings: version: ["1.0.0", "1.0.1"]

Variables and Templating#

Available variables for Mako templates:

  • ${app.id}: Application ID (integer)

  • ${app.name}: Application name (alphanumeric, underscores, hyphens)

  • ${app.version}: Version string

Example:

image: "registry.example.com/${app.name}:${app.version}"
command: "/app/run.sh --app-id ${app.id}"

The interpolate_config_strings function processes all string values in the configuration after YAML loading.

Loading and Using App Specifications#

from pathlib import Path
from bfabric_app_runner.specs.app.app_spec import AppSpec

# Load from YAML
app_spec = AppSpec.load_yaml(Path("./app_spec.yaml"), app_id="123", app_name="MyApp")

# Check version
if "1.0.0" in app_spec:
    app_version = app_spec["1.0.0"]

Reference#

class bfabric_app_runner.specs.app.app_spec.AppSpec(*, bfabric: BfabricAppSpec, versions: list[AppVersion])#

Bases: BaseModel

Parsed app versions from the app spec file.

property available_versions: set[str]#

The available versions of the app.

bfabric: BfabricAppSpec#
classmethod load_yaml(app_yaml: Path, app_id: int | str, app_name: str) AppSpec#

Loads the app versions from the provided YAML file and evaluates the templates.

model_config: ClassVar[ConfigDict] = {}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod no_duplicate_versions(values: list[AppVersion]) list[AppVersion]#

Validates that there are no duplicate versions in the app spec.

versions: list[AppVersion]#
class bfabric_app_runner.specs.app.app_spec.AppSpecTemplate(*, bfabric: BfabricAppSpec, versions: list[AppVersionMultiTemplate])#

Bases: BaseModel

This model defines the app_spec definition in a file.

As the name suggests, this is a template that can be expanded to a concrete AppSpec instance. The main difference is that this may contain usages of Variables. TODO

bfabric: BfabricAppSpec#
evaluate(app_id: int, app_name: str) AppSpec#

Evaluates the template to a concrete AppSpec instance.

model_config: ClassVar[ConfigDict] = {}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

versions: list[AppVersionMultiTemplate]#
class bfabric_app_runner.specs.app.app_spec.BfabricAppSpec(*, app_runner: str, workflow_template_step_id: int | None = None)#

Bases: BaseModel

Contains the app specification information that is relevant to bfabric, and not exactly the app itself.

app_runner: str#

Specifies the app runner version to use for the app.

We support both a PyPI version (e.g. 0.0.17) as well as a git reference, which is a string in the format git+https://github.com/fgcz/bfabricPy@main#subdirectory=bfabric_app_runner where you can specify any git reference instead of main as needed.

model_config: ClassVar[ConfigDict] = {}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

workflow_template_step_id: int | None#

If specified, this indicates that a workflow step should be created for the workunit.

class bfabric_app_runner.specs.app.app_version.AppVersion(*, version: str = 'latest', commands: CommandsSpec, reuse_default_resource: bool = True)#

Bases: BaseModel

A concrete app version specification.

For a better separation of concerns, the submitter will not be resolved automatically.

commands: CommandsSpec#
model_config: ClassVar[ConfigDict] = {}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

reuse_default_resource: bool#
version: str#
class bfabric_app_runner.specs.app.app_version.AppVersionMultiTemplate(*, version: list[str], commands: CommandsSpec, reuse_default_resource: bool = True)#

Bases: BaseModel

commands: CommandsSpec#
expand_versions() list[AppVersionTemplate]#

Returns a list of individual AppVersionTemplate instances, expanding each template of multiple versions. If substitutions are used they will not be expanded yet but rather when converting the template to a concrete AppVersion.

model_config: ClassVar[ConfigDict] = {}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

reuse_default_resource: bool#
version: list[str]#
class bfabric_app_runner.specs.app.app_version.AppVersionTemplate(*, version: str, commands: CommandsSpec, reuse_default_resource: bool = True)#

Bases: BaseModel

commands: CommandsSpec#
evaluate(variables_app: VariablesApp) AppVersion#

Evaluates the template to a concrete AppVersion instance.

model_config = {}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

reuse_default_resource: bool#
version: str#