jina.jaml package

Subpackages

Submodules

Module contents

class jina.jaml.JAML[source]

Bases: object

A Jina YAML parser supports loading and dumping and substituting variables.

To use it:

  1. from jina.jaml import JAML
  3. JAML.load(...)
  4. JAML.dump(...)
  6. class DummyClass:
  7.     pass
  9. JAML.register(DummyClass)

You can use expressions to programmatically set variables in YAML files and access contexts. An expression can be any combination of literal values, references to a context, or functions. You can combine literals, context references, and functions using operators.

You need to use specific syntax to tell Jina to evaluate an expression rather than treat it as a string, which is based on GitHub actions syntax, and looks like this:

  1. ${{ <expression> }}

This expression can be evaluated directly (i.e. substituted by the real value) when being loaded, by using load(substitute=True)()

JAML supports three different kinds of variables to be used as expressions: Environment variables (coming form the environment itself), context variables (being passed as a dict), and internal references (included in the .yaml file itself).

An environment variable var is accessed through the following syntax:

  1. ${{ env.var }}

Note the mandatory spaces before and after the variable denotation.

Context variables can be accessed using the following syntax:

  1. ${{ context_var }}

Or, if you want to be explicit:

  1. ${{ context.context_var }}

These context variables are passed as a dict:

  1. obj = JAML.load(fp, substitute=True,
  2.                     context={'context_var': 3.14,
  3.                             'context_var2': 'hello-world'})

Internal references point to other variables in the yaml file itself, and can be accessed using the following syntax:

  1. ${{root.path.to.var}}

Note omission of spaces in this syntax.

Note

BaseFlow, BaseExecutor, BaseDriver and all their subclasses have already implemented JAML interfaces, to load YAML config into objects, please use Flow.load_config(), BaseExecutor.load_config(), etc.

  • static load(stream, substitute=False, context=None)[source]

    Parse the first YAML document in a stream and produce the corresponding Python object.

    Note

    BaseFlow, BaseExecutor, BaseDriver and all their subclasses have already implemented JAML interfaces, to load YAML config into objects, please use Flow.load_config(), BaseExecutor.load_config(), etc.

    • Parameters

      • substitute (bool) – substitute environment, internal reference and context variables.

      • context (Optional[Dict[str, Any]]) – context replacement variables in a dict, the value of the dict is the replacement.

      • stream – the stream to load

      Returns

      the Python object

  • static escape(value, include_unknown_tags=True)[source]

    Escape the YAML content by replacing all customized tags ! to ``jtype: ``.

    • Parameters

      • value (str) – the original YAML content

      • include_unknown_tags (bool) – if to include unknown tags during escaping

      Return type

      str

      Returns

      escaped YAML

  • static unescape(value, include_unknown_tags=True, jtype_whitelist=None)[source]

    Unescape the YAML content by replacing all ``jtype: `` to tags.

    • Parameters

      • value (str) – the escaped YAML content

      • include_unknown_tags (bool) – if to include unknown tags during unescaping

      • jtype_whitelist (Optional[Tuple[str, …]]) – the list of jtype to be unescaped

      Return type

      str

      Returns

      unescaped YAML

  • static registered_tags()[source]

    Return a list of JAMLCompatible classes that have been registered.

    • Return type

      List[str]

      Returns

      tags

  • static registered_classes()[source]

    Return a dict of tags and JAMLCompatible classes that have been registered.

    • Return type

      Dict

      Returns

      tags and classes

  • static cls_from_tag(tag)[source]

    Fetch class from yaml tag

    • Parameters

      tag (str) – yaml tag

      Return type

      Optional[JAMLCompatible]

      Returns

      class object from tag

  • static load_no_tags(stream, \*kwargs*)[source]

    Load yaml object but ignore all customized tags, e.g. !Executor, !Driver, !Flow.

    • Parameters

      • stream – the output stream

      • kwargs – other kwargs

      Returns

      the Python object

  • static expand_dict(d, context=None, resolve_cycle_ref=True, resolve_passes=3)[source]

    Expand variables from YAML file.

    • Parameters

      • d (Dict) – yaml file loaded as python dict

      • context (Union[Dict, SimpleNamespace, None]) – context replacement variables in a dict, the value of the dict is the replacement.

      • resolve_cycle_ref – resolve internal reference if True.

      • resolve_passes (int) – number of rounds to resolve internal reference.

      Return type

      Dict[str, Any]

      Returns

      expanded dict.

  • static dump(data, stream=None, \*kwargs*)[source]

    Serialize a Python object into a YAML stream.

    If stream is None, return the produced string instead.

    • Parameters

      • data – the data to serialize

      • stream – the output stream

      • kwargs – other kwargs

      Returns

      the yaml output

  • static register(cls)[source]

    Register a class for dumping loading.

    • if it has attribute yaml_tag use that to register, else use class name

    • if it has methods to_yaml/from_yaml use those to dump/load else dump attributes as mapping

    • Parameters

      cls – the class to register

      Returns

      the registered class

class jina.jaml.JAMLCompatible[source]

Bases: object

JAMLCompatible is a mixin class designed to be used with multiple inheritance.

It will add to_yaml() and from_yaml() to the target class, making that class JAML-friendly.

Warning

For the sake of cooperative multiple inheritance, do NOT implement __init__() for this class

  • save_config(filename=None)[source]

    Save the object’s config into a YAML file.

    • Parameters

      filename (Optional[str]) – file path of the yaml file, if not given then config_abspath is used

  • classmethod load_config(source, **, allow_py_modules=True, substitute=True, context=None, uses_with=None, uses_metas=None, uses_requests=None, extra_search_paths=None, **kwargs*)[source]

    A high-level interface for loading configuration with features of loading extra py_modules, substitute env & context variables. Any class that implements JAMLCompatible mixin can enjoy this feature, e.g. BaseFlow, BaseExecutor, BaseDriver and all their subclasses.

    • Support substitutions in YAML:

      • Environment variables: ${{ ENV.VAR }} (recommended), $VAR (deprecated).

      • Context dict (context): ${{ CONTEXT.VAR }}``(recommended), ``${{ VAR }}.

      • Internal reference via this and root: ${{this.same_level_key}}, ${{root.root_level_key}}

  1. Substitutions are carried in the order and multiple passes to resolve variables with best effort.
  3. ```
  4. !BaseEncoder
  5. metas:
  6.     name: ${{VAR_A}}  # env or context variables
  7.     workspace: my-${{this.name}}  # internal reference
  8. ```
  10. ```
  11. # load Executor from yaml file
  12. BaseExecutor.load_config('a.yml')
  14. # load Executor from yaml file and substitute environment variables
  15. os.environ['VAR_A'] = 'hello-world'
  16. b = BaseExecutor.load_config('a.yml')
  17. assert b.name == hello-world
  19. # load Executor from yaml file and substitute variables from a dict
  20. b = BaseExecutor.load_config('a.yml', context={'VAR_A': 'hello-world'})
  21. assert b.name == hello-world
  23. # disable substitute
  24. b = BaseExecutor.load_config('a.yml', substitute=False)
  25. ```
  27. -   Parameters
  29.     -   **source** (`Union`\[`str`, `TextIO`, `Dict`\]) – the multi-kind source of the configs.
  31.     -   **allow\_py\_modules** (`bool`) – allow importing plugins specified by `py_modules` in YAML at any levels
  33.     -   **substitute** (`bool`) – substitute environment, internal reference and context variables.
  35.     -   **context** (`Optional`\[`Dict`\[`str`, `Any`\]\]) – context replacement variables in a dict, the value of the dict is the replacement.
  37.     -   **uses\_with** (`Optional`\[`Dict`\]) – dictionary of parameters to overwrite from the default config’s with field
  39.     -   **uses\_metas** (`Optional`\[`Dict`\]) – dictionary of parameters to overwrite from the default config’s metas field
  41.     -   **uses\_requests** (`Optional`\[`Dict`\]) – dictionary of parameters to overwrite from the default config’s requests field
  43.     -   **extra\_search\_paths** (`Optional`\[`List`\[`str`\]\]) – extra paths used when looking for executor yaml files
  45.     -   **kwargs** – kwargs for parse\_config\_source
  47.     Return type
  49.     [JAMLCompatible](#jina.jaml.JAMLCompatible "jina.jaml.JAMLCompatible")
  51.     Returns
  53.     [JAMLCompatible](#jina.jaml.JAMLCompatible "jina.jaml.JAMLCompatible") object

  • static is_valid_jaml(obj)[source]

    Verifies the yaml syntax of a given object by first serializing it and attempting to deserialize and catch parser errors :type obj: Dict :param obj: yaml object :rtype: bool :return: whether the syntax is valid or not