Refgenie genome configuration file
Refgenie will read and write a genome configuration file in yaml format. In general, you shouldn't need to mess with the config file. You create one with
refgenie init -c genome_config.yaml, then you add assets using either
refgenie pull or
refgenie build. You can also add your own custom assets with
refgenie add, which is explained in using custom assets. Refgenie will use the config file to remember what assets are available and where they are.
Upgrading the configuration file
Refgenie is under active development and new features are added regularly. This sometimes necessitates changes in the refgenie configuration file format or asset directory structure. Starting with
refgenie v0.10.0 we introduced the
refgenie upgrade command, which will automatically detect the current configuration file version and will: 1) reformat the configuration file to the new version; and 2) make any necessary changes to the asset directory structure. To reformat the config, run from the command line:
refgenie upgrade --target-version 0.4 -c /path/to/old/cfg.yml
Or from within Python:
from refgenconf import upgrade_config upgrade_config(target_version="0.4", filepath="/path/to/old/cfg.yml")
Below is a CHANGELOG describing all changes introduced in configuration file versions.
Genome configuration file example
Here's how the config file works, in case you do need to edit some things by hand. Here's an example file which manages fasta and bowtie2_index assets for hg38 genome. Keep in mind that some of the keys in this config are optional:
config_version: 0.4 genome_folder: /path/to/genomes genome_archive_folder: /path/to/genome_archives genome_archive_config: /path/to/genome_archive/config.yaml remote_url_base: http://awspds.refgenie.databio.org/ genome_servers: ['http://refgenomes.databio.org'] genomes: 511fb1178275e7d529560d53b949dba40815f195623bce8e: aliases: - hg38 - human assets: fasta: tags: default: seek_keys: fasta: 511fb1178275e7d529560d53b949dba40815f195623bce8e.fa fai: 511fb1178275e7d529560d53b949dba40815f195623bce8e.fa.fai chrom_sizes: 511fb1178275e7d529560d53b949dba40815f195623bce8e.chrom.sizes asset_parents:  asset_path: fasta asset_digest: a3c46f201a3ce7831d85cf4a125aa334 asset_children: ['511fb1178275e7d529560d53b949dba40815f195623bce8e/bowtie2_index:default'] default_tag: default bowtie2_index: asset_description: Genome index for bowtie, produced with bowtie-build tags: default: asset_path: bowtie2_index seek_keys: bowtie2_index: . asset_digest: 0f9217d44264ae2188fcf8e469d2bb38 asset_parents: ['511fb1178275e7d529560d53b949dba40815f195623bce8e/fasta:default'] default_tag: default
Details of config attributes
- genome_folder: Path to parent folder refgenie-managed assets.
- genome_servers: URLs to a refgenieserver instances.
- genomes: A list of genomes, each genome has a list of assets. Any relative paths in the asset
pathattributes are considered relative to the genome folder in the config file (or the file itself if not folder path is specified), with the genome name as an intervening path component, e.g.
- aliases: A list of arbitrary strings that can be used to refer to the namespace
- tags: A collection of tags defined for the asset
- default_tag: A pointer to the tag that is currently defined as the default one
- asset_parents: A list of assets that were used to build the asset in question
- asset_children: A list of assets that required the asset in question for building
- seek_keys: A mapping of names and paths of the specific files within an asset
- asset_path: A path to the asset folder, relative to the genome config file
- asset_digest: A digest of the asset directory (more precisely, of the file contents within one) used to address the asset provenance issues when the assets are pulled or built.
Optional (used by refgenieserver)
- genome_archive_folder: Path to folder where asset archives will be stored.
- genome_archive_folder: Path to folder file asset archives config will be stored.
- remote_url_base: Path/URL to prepend to served asset archives, if non-local ones are to be served
For genomes that are managed by
refgenie (that is, they were built or pulled with
refgenie), these asset attributes will be automatically populated. You can edit them and refgenie will respect your edits (unless you re-build or re-pull the asset, which will overwrite those fields). You can also add your own assets and
refgenie won't touch them. For more info, see using custom assets.
Config file changelog
[0.4] - Unreleased; refgenie v0.10.0
Config format changes
- use sequence-derived unique genome identifiers instead of genome names everywhere
aliaseskey under each genome section to store the aliases that can be used to refer to the genomes easily
File tree structure changes
- use sequence-derived unique genome identifiers instead of genome names in every file name and directory name
- move all the contents from the refgenie directory to a new
- add an
aliasdirectory with contents corresponding to the aliases defined in the configuration file. The contents of the child directories are symbolic links to the asset files in the
[0.3] - 2019-10-21; refgenie v0.7.0
Config format changes
- Added seek keys, tags, asset digests, default tag pointer, asset description.
[0.2] - 2019-07-11; refgenie v0.5.0
Config format changes
- Added the
assetslevel in the config hierarchy.
- We moved the assets down a layer to accommodate other genome-level attributes we intend to use in the future (like a description, checksums, other provenance information). Earlier refgenie config files will need to be updated.
[0.1] - 2019-05-10; refgenie v0.3.0
- Initial version of the config file with the initial refgenie release.