Configuration

New in version 0.5.0.

Note

The configuration is only related to the precomputed and precomputed-local backends in regrid().

earthkit-regrid maintains a global configuration.

The configuration is automatically loaded from and saved into a yaml file located at ~/.config/earthkit/regrid/config.yaml. An alternative path can be specified via the EARTHKIT_REGRID_CONFIG_FILE environmental variable (it is only read at startup).

The configuration can be accessed and modified from Python. The configuration options can also be defined as environment variables, which take precedence over the config file.

See the following notebooks for examples:

Configuration

Configuration environment variables

Accessing configuration options

The earthkit-regrid configuration can be accessed using the python API:

import earthkit.regrid

# Access one of the config options
cache_path = earthkit.regrid.config.get("user-cache-directory")
print(cache_path)

# If this is the last line of a Notebook cell, this
# will display a table with all the current configuration
earthkit.regrid.config

Warning

When an environment variable is set, it takes precedence over the config parameter, and its value is returned from get().

Changing configuration

Note

It is recommended to restart your Jupyter kernels after changing or resetting config options.

The earthkit-regrid configuration can be modified using the python API:

import earthkit.regrid

# Change the location of the user defined cache:
earthkit.regrid.config.set("user-cache-directory", "/big-disk/earthkit-regrid-cache")

# Change the download timeout
earthkit.regrid.config.set("url-download-timeout", "1m")

# Multiple values can be set together. The argument list
# can be a dictionary:
earthkit.regrid.config.set({"url-download-timeout": "1m", "check-out-of-date-urls": True})

# Alternatively, we can use keyword arguments. However, because
# the “-” character is not allowed in variable names in Python we have
# to replace “-” with “_” in all the keyword arguments:
earthkit.regrid.config.set(url_download_timeout="1m", check_out_of_date_urls=True)

Warning

When an environment variable is set, the new value provided for set() is saved into the config file but get() wil still return the value of the environment variable. A warning is also generated.

Temporary configuration

We can create a temporary configuration (as a context manager) as a copy of the original configuration. We will still refer to it as “config”, but it is completely independent from the original object and changes are not saved into the yaml file (even when config.autosave is True).

import earthkit.regrid

print(earthkit.regrid.config.get("url-download-timeout"))

with earthkit.regrid.config.temporary():
    earthkit.regrid.config.set("url-download-timeout", 5)
    print(earthkit.regrid.config.get("url-download-timeout"))

# Temporary config can also be created with arguments:
with earthkit.regrid.config.temporary("url-download-timeout", 11):
    print(earthkit.regrid.config.get("url-download-timeout"))

Output:

30
5
11

Warning

When an environment variable is set, the same rules applies as for set().

Resetting configuration

Note

It is recommended to restart your Jupyter kernels after changing or resetting the configuration.

The earthkit-regrid configuration can be reset using the python API:

import earthkit.regrid

# Reset a named config option to its default value
earthkit.regrid.config.reset("user-cache-directory")

# Reset all the config options to their default values
earthkit.regrid.config.reset()

Warning

When an environment variable is set, the same rules applies as for set().

Environment variables

Each configuration parameter has a corresponding environment variable (see the full list here). When an environment variable is set, it takes precedence over the config parameter as the following examples show.

First, let us assume that the value of url-download-timeout is 30 in the config file and no environment variable is set.

>>> from earthkit.regrid import config
>>> config.get("url-download-timeout")
30

Then, set the environment variable EARTHKIT_REGRID_URL_DOWNLOAD_TIMEOUT.

export EARTHKIT_REGRID_URL_DOWNLOAD_TIMEOUT=5

>>> from earthkit.regrid import config
>>> config.get("url-download-timeout")
5
>>> config.env()
{'url-download-timeout': ('EARTHKIT_REGRID_URL_DOWNLOAD_TIMEOUT', '5')}
>>> config.set("url-download-timeout", 10)
UserWarning: Config option 'url-download-timeout' is also set by environment variable
'EARTHKIT_REGRID_URL_DOWNLOAD_TIMEOUT'.The environment variable takes precedence and
its value is returned when calling get(). Still, the value set here will be
saved to the config file.
>>> config.get("url-download-timeout")
5

Finally, unset the environment variable and check the config value again, which is now the value from the config file.

unset EARTHKIT_REGRID_URL_DOWNLOAD_TIMEOUT

>>> from earthkit.regrid import config
>>> config.get("url-download-timeout")
10

List of configuration parameters

This is the list of all the config parameters:

Name	Default	Description
cache‑policy	‘user’	Caching policy. Valid values: off, temporary and user. See Disk-based precomputed weights caching for more information.
check‑out‑of‑date‑urls	False	Perform a HTTP request to check if the remote version of a cache file has changed
download‑out‑of‑date‑urls	False	Re-download URLs when the remote version of a cached file as been changed
maximum‑cache‑disk‑usage	None	Disk usage threshold after which earthkit-regrid expires older cached entries (% of the full disk capacity). Can be set to None. See Disk-based precomputed weights caching for more information.
maximum‑cache‑size	‘5GB’	Maximum disk space used by the earthkit-regrid cache (e.g.: 100G or 2T). Can be set to None.
maximum‑weights‑memory‑cache‑size	‘500MB’	The maximum memory size of the in-memory precomputed weight cache in bytes. Only used when `weights-memory-cache-policy` is `"largest"` or `"lru"`. Can be set to None. See In-memory weights caching for more information.
temporary‑cache‑directory‑root	None	Parent of the cache directory when `cache-policy` is `temporary`. See Disk-based precomputed weights caching for more information.
temporary‑directory‑root	None	Parent of the temporary directory when `cache-policy` is `off`. See Disk-based precomputed weights caching for more information.
url‑download‑timeout	’30s’	Timeout when downloading from an url.
user‑cache‑directory	‘~/.cache/earthkit‑regrid’	Cache directory used when `cache-policy` is `user`. See Disk-based precomputed weights caching for more information.
weights‑memory‑cache‑policy	‘largest’	The in-memory precomputed weights cache policy. Valid values: off, unlimited, largest and lru. See In-memory weights caching for more information.
weights‑memory‑cache‑strict‑mode	False	Raise exception if the weights cannot be fitted into the in-memory cache. Only used when `weights-memory-cache-policy` is `"largest"` or `"lru"`. See In-memory weights caching for more information.

List of environment variables

This is the list of the config environment variables:

Config option name	Environment variable
cache‑policy	EARTHKIT_REGRID_CACHE_POLICY
check‑out‑of‑date‑urls	EARTHKIT_REGRID_CHECK_OUT_OF_DATE_URLS
download‑out‑of‑date‑urls	EARTHKIT_REGRID_DOWNLOAD_OUT_OF_DATE_URLS
maximum‑cache‑disk‑usage	EARTHKIT_REGRID_MAXIMUM_CACHE_DISK_USAGE
maximum‑cache‑size	EARTHKIT_REGRID_MAXIMUM_CACHE_SIZE
maximum‑weights‑memory‑cache‑size	EARTHKIT_REGRID_MAXIMUM_WEIGHTS_MEMORY_CACHE_SIZE
temporary‑cache‑directory‑root	EARTHKIT_REGRID_TEMPORARY_CACHE_DIRECTORY_ROOT
temporary‑directory‑root	EARTHKIT_REGRID_TEMPORARY_DIRECTORY_ROOT
url‑download‑timeout	EARTHKIT_REGRID_URL_DOWNLOAD_TIMEOUT
user‑cache‑directory	EARTHKIT_REGRID_USER_CACHE_DIRECTORY
weights‑memory‑cache‑policy	EARTHKIT_REGRID_WEIGHTS_MEMORY_CACHE_POLICY
weights‑memory‑cache‑strict‑mode	EARTHKIT_REGRID_WEIGHTS_MEMORY_CACHE_STRICT_MODE