App Specification

Contents

App Specification#

Overview#

The bfabric-app-runner allows you to define and execute applications in various environments, by configuring the application steps in a YAML configuration files, the so-called app specification.

App Specification Structure#

The specification can be provided in a YAML file with the following structure:

bfabric:
  app_runner: "0.0.17"  # or a git reference
  workflow_template_step_id: null  # optional
versions:
  - version: "1.0.0"
    commands:
      dispatch: ...
      process: ...
      collect: ...  # Optional

The top-level bfabric section contains metadata relevant to B-Fabric integration (the runner version to use and an optional workflow template step ID). The versions list defines one or more application versions, each with its own set of commands.

Commands#

Each app defines these core commands:

  • dispatch: Prepares input data. Called with: $workunit_ref $work_dir

  • process: Executes main logic. Called with: $chunk_dir

  • collect: (Optional) Organizes results. Called with: $workunit_ref $chunk_dir

Commands are discriminated by their type field.

pydantic model bfabric_app_runner.specs.app.commands_spec.CommandsSpec#

Defines the commands that are required to execute an app.

Fields:
field collect: Annotated[CommandShell | CommandExec | CommandDocker | CommandPythonEnv, Discriminator(discriminator=type, custom_error_type=None, custom_error_message=None, custom_error_context=None)] | None = None#

The app collect command, can be omitted if your process command already creates an outputs.yml file.

It will be called with arguments: $workunit_ref $chunk_dir.

field dispatch: Annotated[CommandShell | CommandExec | CommandDocker | CommandPythonEnv, Discriminator(discriminator=type, custom_error_type=None, custom_error_message=None, custom_error_context=None)] [Required]#

The app dispatch command.

It will be called with arguments: $workunit_ref $work_dir.

Constraints:

  • discriminator = type

field process: Annotated[CommandShell | CommandExec | CommandDocker | CommandPythonEnv, Discriminator(discriminator=type, custom_error_type=None, custom_error_message=None, custom_error_context=None)] [Required]#

The app process command.

It will be called with arguments: $chunk_dir.

Constraints:

  • discriminator = type

Direct commands#

commands:
  dispatch:
    type: "exec"
    command: "python prepare_data.py"

The command string is split by spaces using shlex.split().

pydantic model bfabric_app_runner.specs.app.commands_spec.CommandExec#

A command run directly (no shell), with its string split via shlex.split.

Fields:
field command: str [Required]#

The command to run, will be split by shlex.split and is not an actual shell script.

field env: dict[str, str] = {}#

Environment variables to set before executing the command.

field prepend_paths: list[Path] = []#

A list of paths to prepend to the PATH variable before executing the command.

If multiple paths are specified, the first one will be the first in PATH, etc.

field type: Literal['exec'] = 'exec'#

Identifies the command type.

Docker Commands#

commands:
  process:
    type: "docker"
    image: "myapp:1.0.0"
    command: "/app/run.sh"
    env:
      APP_VERSION: "${app.version}"
    mounts:
      read_only:
        - ["/data/reference", "/app/reference"]
      writeable:
        - ["/data/results", "/app/results"]
pydantic model bfabric_app_runner.specs.app.commands_spec.CommandDocker#

A command run inside a container via Docker or Podman.

Fields:
field command: str [Required]#

The command to execute in the container.

field custom_args: list[str] = []#

Any custom CLI arguments to pass to the container engine.

field engine: Literal['docker', 'podman'] = 'docker'#

The container engine to use.

field entrypoint: str | None = None#

The entrypoint to use for the container (instead of the image’s default).

field env: dict[str, str] = {}#

Environment variables to set in the container.

field hostname: str | None = None#

The hostname to use for the container (instead of Docker’s default assignment).

field image: str [Required]#

The container image to run.

field mac_address: str | None = None#

The MAC address to use for the container (instead of Docker’s default assignment).

field mounts: MountOptions = MountOptions(work_dir_target=None, read_only=[], writeable=[], share_bfabric_config=True)#

Mount options for the container.

field type: Literal['docker'] = 'docker'#

Identifies the command type.

pydantic model bfabric_app_runner.specs.app.commands_spec.MountOptions#

Bind mounts to expose inside the container.

Fields:
field read_only: list[tuple[Path, Path]] = []#

Read-only bind mounts, each a (host_path, container_path) pair.

field share_bfabric_config: bool = True#

If True, mount the host’s ~/.bfabricpy.yml into the container (read-only).

field work_dir_target: Path | None = None#

Container path to mount the work directory at; None reuses the host work-directory path.

field writeable: list[tuple[Path, Path]] = []#

Writeable bind mounts, each a (host_path, container_path) pair.

Shell Commands (deprecated)#

The shell command type is deprecated in favor of exec. It splits the command string by spaces (not using shlex.split).

pydantic model bfabric_app_runner.specs.app.commands_spec.CommandShell#

A command run by splitting a string on spaces (no real shell).

Deprecated in favor of CommandExec.

Fields:
field command: str [Required]#

The command to run, will be split by spaces and is not an actual shell script.

field type: Literal['shell'] = 'shell'#

Identifies the command type.

Python Environment Commands#

Provisions a Python virtual environment from a pylock file and runs the command inside it. Environments can be cached for reuse across runs.

commands:
  process:
    type: "python_env"
    pylock: "requirements.pylock.toml"
    command: "python run.py"
    python_version: "3.12"
pydantic model bfabric_app_runner.specs.app.commands_spec.CommandPythonEnv#

A command run inside a uv-provisioned Python environment defined by a pylock file.

Fields:
field command: str [Required]#

The command to run, will be split by shlex.split and is not an actual shell script.

field env: dict[str, str] = {}#

Environment variables to set before executing the command.

field local_extra_deps: list[Path] = []#

Additional dependencies to install into the environment.

Each entry should be a path to a wheel, sdist, or local package directory. These will be installed into the environment with uv pip install –no-deps after the main requirements. No dependency resolution will be performed for these, so their dependencies should already be present in the environment (typically specified in the pylock file).

field prepend_paths: list[Path] = []#

A list of paths to prepend to the PATH variable before executing the command.

If multiple paths are specified, the first one will be the first in PATH, etc.

field pylock: Path [Required]#

Path to the Pylock file that specifies the environment to use.

field python_version: str = '3.13'#

The Python version to use for the environment (passed to uv venv -p).

field refresh: bool = False#

When True, forces provisioning of an ephemeral environment that doesn’t affect the cache.

This creates a temporary environment for the single execution and doesn’t modify or reuse the cached environment. This ensures that refresh operations always start from a clean state and don’t leave broken environments in the cache if provisioning fails.

field type: Literal['python_env'] = 'python_env'#

App Versions#

The AppVersionMultiTemplate class defines app versions in two ways:

  1. Single string: version: "1.0.0"

  2. List of strings: version: ["1.0.0", "1.0.1"]

Variables and Templating#

Available variables for Mako templates:

  • ${app.id}: Application ID (integer)

  • ${app.name}: Application name (alphanumeric, underscores, hyphens)

  • ${app.version}: Version string

Example:

image: "registry.example.com/${app.name}:${app.version}"
command: "/app/run.sh --app-id ${app.id}"

The interpolate_config_strings function processes all string values in the configuration after YAML loading.

Loading and Using App Specifications#

from pathlib import Path
from bfabric_app_runner.specs.app.app_spec import AppSpec

# Load from YAML
app_spec = AppSpec.load_yaml(Path("./app_spec.yaml"), app_id="123", app_name="MyApp")

# Check version
if "1.0.0" in app_spec:
    app_version = app_spec["1.0.0"]

Reference#

The top-level spec (AppSpecTemplate) is expanded into a concrete AppSpec at load time; the bfabric section is a BfabricAppSpec.

class bfabric_app_runner.specs.app.app_spec.AppSpec(*, bfabric: BfabricAppSpec, versions: list[AppVersion])#

Bases: BaseModel

Parsed app versions from the app spec file.

property available_versions: set[str]#

The available versions of the app.

bfabric: BfabricAppSpec#

B-Fabric-related settings shared by all versions (see BfabricAppSpec).

classmethod load_yaml(app_yaml: Path, app_id: int | str, app_name: str) → AppSpec#

Loads the app versions from the provided YAML file and evaluates the templates.

model_config = {}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod no_duplicate_versions(values: list[AppVersion]) → list[AppVersion]#

Validates that there are no duplicate versions in the app spec.

versions: list[AppVersion]#

The concrete, fully-resolved app versions.

class bfabric_app_runner.specs.app.app_spec.AppSpecTemplate(*, bfabric: BfabricAppSpec, versions: list[AppVersionMultiTemplate])#

Bases: BaseModel

This model defines the app_spec definition in a file.

As the name suggests, this is a template that can be expanded to a concrete AppSpec instance. The main difference is that this may contain usages of Variables. TODO

bfabric: BfabricAppSpec#

B-Fabric-related settings shared by all versions (see BfabricAppSpec).

evaluate(app_id: int, app_name: str) → AppSpec#

Evaluates the template to a concrete AppSpec instance.

model_config = {}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

versions: list[AppVersionMultiTemplate]#

App version templates; each may declare several versions and use Variables placeholders.

class bfabric_app_runner.specs.app.app_spec.BfabricAppSpec(*, app_runner: str, workflow_template_step_id: int | None = None)#

Bases: BaseModel

Contains the app specification information that is relevant to bfabric, and not exactly the app itself.

app_runner: str#

Specifies the app runner version to use for the app.

We support both a PyPI version (e.g. 0.0.17) as well as a git reference, which is a string in the format git+https://github.com/fgcz/bfabricPy@main#subdirectory=bfabric_app_runner where you can specify any git reference instead of main as needed.

model_config = {}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

workflow_template_step_id: int | None#

If specified, this indicates that a workflow step should be created for the workunit.

Each entry in versions is an AppVersionMultiTemplate (which may list several version strings), expanded into one concrete AppVersion per version.

class bfabric_app_runner.specs.app.app_version.AppVersion(*, version: str = 'latest', commands: CommandsSpec, reuse_default_resource: bool = True)#

Bases: BaseModel

A concrete app version specification.

For a better separation of concerns, the submitter will not be resolved automatically.

commands: CommandsSpec#

The dispatch, process, and (optional) collect commands that implement this version.

model_config = {}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

reuse_default_resource: bool#

Legacy flag: reuse the workunit’s auto-created default resource for the first copied output instead of creating a new resource entry.

version: str#

Version identifier of this app version (e.g. "1.2.0").

class bfabric_app_runner.specs.app.app_version.AppVersionMultiTemplate(*, version: list[str], commands: CommandsSpec, reuse_default_resource: bool = True)#

Bases: BaseModel

App version template that may declare several version strings sharing the same commands.

A single version string is also accepted and normalized to a one-element list.

commands: CommandsSpec#

The dispatch, process, and (optional) collect commands that implement these versions.

expand_versions() → list[AppVersionTemplate]#

Returns a list of individual AppVersionTemplate instances, expanding each template of multiple versions. If substitutions are used they will not be expanded yet but rather when converting the template to a concrete AppVersion.

model_config = {}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

reuse_default_resource: bool#

Legacy flag: reuse the workunit’s auto-created default resource for the first copied output instead of creating a new resource entry.

version: list[str]#

Version identifiers that share this definition; a single string is coerced to a one-element list.

Contents