Model Runtimes

The actual work of calling the model runtime performs the model. This is because we use model runtimes as a way to provide a container for the model.

Our Model Runtimes are therefore similar to build packs. They represent a Linux environment which contains a defined set of libraries (OS and Python). This means that we can provide a consistent environment for the model to run in.The model runtime is also responsible for the actual execution of the model. It is the model runtime that will call the model and pass the document to it. The contract between the model and Kodexa is basically defined by the model runtime.

How do Models interact with the Model Runtime?

When you deploy a model into Kodexa you include the model runtime that you want to use. Today, all Kodexa model runtimes have the same interface, but this may change in the future. There are two primary forms of interaction between the model runtime and the model. The first is inference, the second is training. How the model runtime calls your model is based on how you have declared your model in the model.yml file.

Inference

The most common starting point with working with a model is learning how inference works. Let's take a simple example of a model.yml:

# A very simple first model that isn't trainable

slug: my-model
version: 1.0.0
orgSlug: kodexa
type: store
storeType: MODEL
name: My Model
metadata:
  atomic: true
  state: TRAINED
  modelRuntimeRef: kodexa/base-model-runtime
  type: model
  contents:
    - model/*

The key thing to note here is the modelRuntimeRef which is set to kodexa/base-model-runtime. This means that the Model Assistant will use the base model runtime to run the model. In fact, it will look up the model runtime with the ref kodexa/base-model-runtime. Then it will look at the model runtime to determine which action it uses for inference, and build a pipeline including that action. The platform will then schedule that pipeline and then model runtime action will be called.

When the model runtime is called it will be passed a document and also all the options that have been captured in the UI for the model. The model runtime will then look at the model.yml file and determine the entry point for the model. By default, the model runtime will expect a package called Model and then look for a function in that package called infer.

The model runtime will pass the document that we are processing to the model and then the model will return a document. The model runtime will then pass the document back to the platform for further processing.

Inference with Options

In the previous example, we saw how the model runtime would pass the document to the model. In this example, we will see how the model runtime will pass options to the model.

First, let's add some inference options to our model.yml file:

# A very simple first model that isn't trainable

slug: my-model
version: 1.0.0
orgSlug: kodexa
type: store
storeType: MODEL
name: My Model
metadata:
  atomic: true
  trainable: false
  modelRuntimeRef: kodexa/base-model-runtime
  type: model
  inferenceOptions:
    - name: my_option
      type: string
      default: "Hello World"
      description: "A simple option"
  contents:
    - model/*

Here we can see we have added an inference option to the model.yml file. This option will be displayed in the UI when the model is used. The user can then change the value of the option and that value will be passed to the model runtime. When we deploy this model update, we now can use that new option in our inference code.

import logging

logger = logging.getLogger(__name__)


def infer(document, my_option):
    logger.info(f"Hello from the model, the option is {my_option}")
    return document

As we can see, the option is passed to the model as a parameter.

Overriding Entry Points

If you want to override the entry point for your model, you can do so by specifying the modelRuntimeParameters property in the model.yml file.

modelRuntimeParameters:
  module: my_model
  function: custom_infer
  training_function: custom_train

Magic Parameters for Training and Inference

When you are training or inferring a model, you can pass in some magic parameters to the model runtime. If you include a parameter in either your train or infer function, it will be passed in by the model runtime.

Inference Options

While in training we pass all the training options as a dictionary, in inference we pass the options as individual parameters. This is because we want to make it easy to use the options in the inference code. Therefore, if you have an inference option called my_option then you will get a parameter called my_option passed to your inference function.

def infer(document, my_option):
    logger.info(f"Hello from the model, the option is {my_option}")
    return document

On this page

• Model Runtimes
• How do Models interact with the Model Runtime?
• Inference
• Inference with Options
• Overriding Entry Points
• Magic Parameters for Training and Inference
• Inference Options