Output specification

Output specification#

The outputs module provides a specification schema to define the outputs that were created by an app and should be registered. The file is usually called outputs.yml and lists the different output files, with information how to register them.

General structure#

Generally the structure is a yaml file containing a key outputs which is a list of dictionaries, each representing an output file. Each output has a type key which identifies the output type. This will allow us to extend this logic to different sources in the future.

An example file could look like:

outputs:
- type: bfabric_copy_resource
  local_path: /tmp/work/hello.txt
  store_entry_path: WU123456_hello.txt
- type: bfabric_dataset
  local_path: /tmp/work/hello.csv
  separator: ","
  name: Hello Dataset

Commands#

Validation#

The output file can be validated with the command:

bfabric-app-runner validate outputs-spec outputs.yml

Which on success will output a pretty-printed version of the outputs file. Validation will also be performed by all other commands, so this is not strictly necessary.

Register files#

To perform the registration to B-Fabric the following can be used:

bfabric-app-runner outputs register outputs.yml --workunit-id 1234

Please note:

  • The workunit ID needs to be specified, so the correct information can be retrieved. (TODO but instead of the workunit id it should also be possible to pass the ref)

  • Several actions might require a particular user to be possible, e.g. the bfabric_copy_resource will require a user with permission to create the particular file over SSH.

Reference#

pydantic model bfabric_app_runner.specs.outputs_spec.CopyResourceSpec#

Bases: BaseModel

Show JSON schema
{
   "title": "CopyResourceSpec",
   "type": "object",
   "properties": {
      "type": {
         "const": "bfabric_copy_resource",
         "default": "bfabric_copy_resource",
         "title": "Type",
         "type": "string"
      },
      "local_path": {
         "format": "path",
         "title": "Local Path",
         "type": "string"
      },
      "store_entry_path": {
         "format": "path",
         "title": "Store Entry Path",
         "type": "string"
      },
      "store_folder_path": {
         "anyOf": [
            {
               "format": "path",
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Store Folder Path"
      },
      "update_existing": {
         "$ref": "#/$defs/UpdateExisting",
         "default": "if_exists"
      },
      "protocol": {
         "const": "scp",
         "default": "scp",
         "title": "Protocol",
         "type": "string"
      }
   },
   "$defs": {
      "UpdateExisting": {
         "enum": [
            "no",
            "if_exists",
            "required"
         ],
         "title": "UpdateExisting",
         "type": "string"
      }
   },
   "additionalProperties": false,
   "required": [
      "local_path",
      "store_entry_path"
   ]
}

Config:

  • extra: str = forbid

Fields:
field local_path: Path [Required]#

The local path to the file to be copied.

field protocol: Literal['scp'] = 'scp'#
field store_entry_path: Path [Required]#

The path to the storage entry in the storage folder.

field store_folder_path: Path | None = None#

The storage folder will be determined by the default rule, but can be specified if needed.

field type: Literal['bfabric_copy_resource'] = 'bfabric_copy_resource'#
field update_existing: UpdateExisting = UpdateExisting.IF_EXISTS#
pydantic model bfabric_app_runner.specs.outputs_spec.OutputsSpec#

Bases: BaseModel

Show JSON schema
{
   "title": "OutputsSpec",
   "type": "object",
   "properties": {
      "outputs": {
         "items": {
            "discriminator": {
               "mapping": {
                  "bfabric_copy_resource": "#/$defs/CopyResourceSpec",
                  "bfabric_dataset": "#/$defs/SaveDatasetSpec"
               },
               "propertyName": "type"
            },
            "oneOf": [
               {
                  "$ref": "#/$defs/CopyResourceSpec"
               },
               {
                  "$ref": "#/$defs/SaveDatasetSpec"
               }
            ]
         },
         "title": "Outputs",
         "type": "array"
      }
   },
   "$defs": {
      "CopyResourceSpec": {
         "additionalProperties": false,
         "properties": {
            "type": {
               "const": "bfabric_copy_resource",
               "default": "bfabric_copy_resource",
               "title": "Type",
               "type": "string"
            },
            "local_path": {
               "format": "path",
               "title": "Local Path",
               "type": "string"
            },
            "store_entry_path": {
               "format": "path",
               "title": "Store Entry Path",
               "type": "string"
            },
            "store_folder_path": {
               "anyOf": [
                  {
                     "format": "path",
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Store Folder Path"
            },
            "update_existing": {
               "$ref": "#/$defs/UpdateExisting",
               "default": "if_exists"
            },
            "protocol": {
               "const": "scp",
               "default": "scp",
               "title": "Protocol",
               "type": "string"
            }
         },
         "required": [
            "local_path",
            "store_entry_path"
         ],
         "title": "CopyResourceSpec",
         "type": "object"
      },
      "SaveDatasetSpec": {
         "additionalProperties": false,
         "properties": {
            "type": {
               "const": "bfabric_dataset",
               "default": "bfabric_dataset",
               "title": "Type",
               "type": "string"
            },
            "local_path": {
               "format": "path",
               "title": "Local Path",
               "type": "string"
            },
            "separator": {
               "title": "Separator",
               "type": "string"
            },
            "name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Name"
            },
            "has_header": {
               "default": true,
               "title": "Has Header",
               "type": "boolean"
            },
            "invalid_characters": {
               "default": "",
               "title": "Invalid Characters",
               "type": "string"
            }
         },
         "required": [
            "local_path",
            "separator"
         ],
         "title": "SaveDatasetSpec",
         "type": "object"
      },
      "UpdateExisting": {
         "enum": [
            "no",
            "if_exists",
            "required"
         ],
         "title": "UpdateExisting",
         "type": "string"
      }
   },
   "additionalProperties": false,
   "required": [
      "outputs"
   ]
}

Config:

  • extra: str = forbid

Fields:
field outputs: list[Annotated[SpecType, Field(..., discriminator='type')]] [Required]#
classmethod read_yaml(path: Path) → list[CopyResourceSpec | SaveDatasetSpec]#
classmethod write_yaml(specs: list[CopyResourceSpec | SaveDatasetSpec], path: Path) → None#
pydantic model bfabric_app_runner.specs.outputs_spec.SaveDatasetSpec#

Bases: BaseModel

Show JSON schema
{
   "title": "SaveDatasetSpec",
   "type": "object",
   "properties": {
      "type": {
         "const": "bfabric_dataset",
         "default": "bfabric_dataset",
         "title": "Type",
         "type": "string"
      },
      "local_path": {
         "format": "path",
         "title": "Local Path",
         "type": "string"
      },
      "separator": {
         "title": "Separator",
         "type": "string"
      },
      "name": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Name"
      },
      "has_header": {
         "default": true,
         "title": "Has Header",
         "type": "boolean"
      },
      "invalid_characters": {
         "default": "",
         "title": "Invalid Characters",
         "type": "string"
      }
   },
   "additionalProperties": false,
   "required": [
      "local_path",
      "separator"
   ]
}

Config:

  • extra: str = forbid

Fields:
field has_header: bool = True#
field invalid_characters: str = ''#
field local_path: Path [Required]#
field name: str | None = None#
field separator: str [Required]#
field type: Literal['bfabric_dataset'] = 'bfabric_dataset'#
class bfabric_app_runner.specs.outputs_spec.UpdateExisting(*values)#

Bases: Enum

IF_EXISTS = 'if_exists'#
NO = 'no'#
REQUIRED = 'required'#
Contents