Input specification

Contents

Input specification#

The inputs module provides a specification schema to define the inputs required by an app. You can also use this functionality interactively while prototyping. The file is usually called inputs.yml and lists the different inputs, with information and how to retrieve them and the filename to save them as.

General structure#

Generally the structure is a yaml file containing a key inputs which is a list of dictionaries, each representing an input file. Each input has a type key which identifies the input type. This will allow us to extend this logic to different sources in the future.

In general the only other input key that will be available for all types is filename, which is the name of the file to save the input as. Fields like id might not be relevant for all types in the future, and depending on the type more specific options might exist.

An example file could look like this:

# file: inputs.yml
inputs:
  - type: bfabric_dataset
    id: 53706
    filename: test.csv
  - type: bfabric_resource
    id: 2700958
    filename: test.zip

HTTP transport (optional)#

By default a bfabric_resource is copied from its storage host over SSH (rsync/scp), which needs SSH/NFS access to that host. You can instead fetch it over HTTP with access: http:

inputs:
  - type: bfabric_resource
    id: 2700958
    filename: test.zip
    access: http        # default is "ssh"

HTTP is portable (works anywhere with web access, no SSH keys) but slower, so it is an add-on rather than a replacement. It requires an OAuth-backed client whose token carries the containers scope (the default bfabric-cli scope does not) — with an ordinary config-file (login+password) client there is no bearer token and access: http fails with a clear error. See the _http_input_transport design note for details and current limitations.

The generic file input also accepts an HTTP source, always fetched anonymously (the bearer token is only ever sent to storage-derived URLs):

inputs:
  - type: file
    source:
      http:
        url: https://example.org/data/reference.fasta
    filename: reference.fasta
    checksum: <md5>     # optional; verified after download

Commands#

Validation#

The input file can be validated with the command:

bfabric-app-runner validate inputs-spec inputs.yml

Which on success will output a pretty-printed version of the inputs file. Validation will also be performed by all other commands, so this is not strictly necessary.

For instance, in the above case this would print:

InputsSpec(
    inputs=[
        BfabricDatasetSpec(type='bfabric_dataset', id=53706, filename='test.csv', separator=',', format='csv'),
        BfabricResourceSpec(type='bfabric_resource', id=2700958, filename='test.zip', check_checksum=True, access='ssh')
    ]
)

Here you can also see all the extra parameters which were implicitly set.

Prepare files#

The prepare command downloads your files and requires two arguments. The first is the input file, and the second is the directory to save the files to. In general to download to the current directory simply use . as the second argument:

bfabric-app-runner inputs prepare inputs.yml .

If your files already exist and are up-to-date, it will not download them again.

List files#

You can list the files that are present or will be downloaded:

bfabric-app-runner inputs list inputs.yml .

If you also want to check whether the files are up-to-date, you can pass the --check flag:

bfabric-app-runner inputs list --check inputs.yml .

Reference#

The file parses into an InputsSpec wrapping a list of typed entries; each entry’s type field selects one of the models below.

pydantic model bfabric_app_runner.specs.inputs_spec.InputsSpec#
Fields:
field inputs: list[InputSpecType] [Required]#

B-Fabric-sourced inputs#

Fetched from B-Fabric by ID.

Download a single resource by ID:

pydantic model bfabric_app_runner.specs.inputs.bfabric_resource_spec.BfabricResourceSpec#

Downloads a single B-Fabric resource to a local file.

Fields:
field access: Literal['ssh', 'http'] = 'ssh'#

Transport used to fetch the file.

ssh copies it from the storage host over rsync/scp and requires SSH/NFS access. http streams it from the storage’s HTTP access endpoint; portable (works anywhere with web access) but slower, and requires an OAuth-backed client to provide a bearer token.

field check_checksum: bool = True#

Whether to check the checksum of the file, after downloading

field filename: RelativeFilePath | None = None#

Target filename to save to

field id: int [Required]#

B-Fabric resource ID

field type: Literal['bfabric_resource'] = 'bfabric_resource'#

Download a dataset as CSV or Parquet:

pydantic model bfabric_app_runner.specs.inputs.bfabric_dataset_spec.BfabricDatasetSpec#

Downloads a B-Fabric dataset and writes it to a local CSV or Parquet file.

Fields:
field filename: RelativeFilePath [Required]#

Target filename to save to

Constraints:

  • pattern = ^[^/][^:]*$

field format: Literal['csv', 'parquet'] = 'csv'#

Output file format, either "csv" or "parquet".

field id: int [Required]#

B-Fabric dataset ID

field separator: Literal[',', '\t'] = ','#

Separator for the CSV file (not relevant for Parquet)

field type: Literal['bfabric_dataset'] = 'bfabric_dataset'#

Download a resource that is an archive and extract it (with include/exclude globs):

pydantic model bfabric_app_runner.specs.inputs.bfabric_resource_archive_spec.BfabricResourceArchiveSpec#

Downloads a B-Fabric resource that is an archive and extracts it into a target directory.

Fields:
field check_checksum: bool = True#

Whether to check the checksum of the archive file, after downloading.

field exclude_patterns: list[str] = []#

Globs of files to exclude from the archive extraction (by default no files are excluded)

field extract: Literal['zip'] = 'zip'#

Extraction to perform, currently only ‘zip’ is supported

field filename: RelativeFilePath [Required]#

Target directory to save to

Constraints:

  • pattern = ^[^/][^:]*$

field id: int [Required]#

B-Fabric resource ID

field include_patterns: list[str] = []#

Globs of files to include in the archive extraction (by default all files are included)

field strip_root: bool = False#

If True, the root directory (if present) of the archive will be stripped during extraction.

field type: Literal['bfabric_resource_archive'] = 'bfabric_resource_archive'#

Download every resource referenced by a dataset into a folder, plus a metadata table:

pydantic model bfabric_app_runner.specs.inputs.bfabric_resource_dataset.BfabricResourceDatasetSpec#

Spec to download all resources listed in a B-Fabric dataset to a folder.

This requires a column (see column) referring to the resource IDs in B-Fabric.

The output will be saved to a folder (specified by filename), containing the selected files, as well as a parquet file (see output_dataset_filename) which contains the original dataset and an additional column (see output_dataset_file_column) which contains the file names to identify the files.

Fields:
field check_checksum: bool = True#

Whether to check the checksum of each resource file, after downloading.

field column: int | str = 'Resource'#

Column name or index containing the resource IDs. (case insensitive if string)

field exclude_patterns: list[str] = []#

Globs of files to exclude from the archive extraction (by default no files are excluded)

field filename: RelativeFilePath [Required]#

Target directory to save to.

Constraints:

  • pattern = ^[^/][^:]*$

field id: int [Required]#

B-Fabric dataset ID.

field include_patterns: list[str] = []#

Globs of files to include in the archive extraction (by default all files are included)

field output_dataset_file_column: str = 'File'#

Output name containing the file names (i.e. relative to the directory where the files get stored).

field output_dataset_filename: str = 'dataset.parquet'#

Filename to store the dataset metadata as a parquet file.

field output_dataset_only: bool = False#

Special flag which can be set to true for cases, where you only want the dataset but not the actual files.

field type: Literal['bfabric_resource_dataset'] = 'bfabric_resource_dataset'#

Write the FASTA sequence attached to an order (or a workunit’s order) to a file:

pydantic model bfabric_app_runner.specs.inputs.bfabric_order_fasta_spec.BfabricOrderFastaSpec#

Writes the FASTA sequence attached to a B-Fabric order to a file.

Fields:
field entity: Literal['workunit', 'order'] [Required]#

Whether id refers to an order directly, or to a workunit whose order is used.

field filename: RelativeFilePath [Required]#

Target filename (relative to the chunk directory) to write the FASTA sequence to.

Constraints:

  • pattern = ^[^/][^:]*$

field id: int [Required]#

ID of the entity to resolve the order from (see entity).

field required: bool = False#

If True, a missing order or FASTA sequence raises an error; otherwise an empty file is written.

field type: Literal['bfabric_order_fasta'] = 'bfabric_order_fasta'#

Discriminator marking this input as an order FASTA.

Write a B-Fabric annotation table (e.g. resource-sample mapping) to a file:

bfabric_app_runner.specs.inputs.bfabric_annotation_spec.BfabricAnnotationSpec#

alias of BfabricAnnotationResourceSampleSpec

Local / generic inputs#

Sourced from a path, URL, or inline content rather than by B-Fabric ID.

A generic file from a local path, an SSH host, or an HTTP URL:

pydantic model bfabric_app_runner.specs.inputs.file_spec.FileSpec#

Stages a single file from a local path, a remote host (SSH), or an HTTP(S) URL.

Fields:
field checksum: str | None = None#

Expected checksum to verify the staged file against; None skips verification.

field filename: RelativeFilePath | None = None#

Target filename (relative to the chunk directory); None derives it from the source.

If True, symlink the file instead of copying it; only allowed for a local source.

field source: FileSourceSsh | FileSourceLocal | FileSourceHttp [Required]#

Where the file comes from: a local path, a remote SSH location, or an HTTP(S) URL.

field type: Literal['file'] = 'file'#

Discriminator marking this input as a plain file.

Write inline text or binary content straight to a file:

pydantic model bfabric_app_runner.specs.inputs.static_file_spec.StaticFileSpec#

Writes inline text or binary content, provided in the spec itself, to a local file.

Fields:
field content: str | bytes [Required]#

The text or binary content to write.

field filename: str [Required]#

The target filename to write to.

field type: Literal['static_file'] = 'static_file'#

Write an inline YAML document straight to a file:

pydantic model bfabric_app_runner.specs.inputs.static_yaml_spec.StaticYamlSpec#

Writes an inline YAML document, provided in the spec itself, to a local file.

Fields:
field data: dict | list [Required]#

The YAML document content to write.

field filename: str [Required]#

The target filename to write to.

field type: Literal['static_yaml'] = 'static_yaml'#
Contents