evaluation-function-utils Package

• Error Reporting
• Schema validation
• Local testing

Errors

Submodule containing custom error and exception classes, which can be properly caught by the base evaluation layer, and return more detailed and appropriate errors.

class EvaluationException

This class extends the usual python Exception, with additional functionality. It can be used to package additional fields and values to errors thrown and returned by evaluation functions.

Example

If at some point in the execution of the evaluation_function, an error is thrown:

from evaluation_function_utils.errors import EvaluationException

if isinstance(input, str):
    raise EvaluationException(
        "The input must not be a string", 
        valid_types=["int", "float", "array"],
    )

Then the output generated by the lambda function will look like:

{
  "command": "eval",
  "error": {
    "message": "The input must not be a string",
    "valid_types": [
      "int", "float", "array"
    ]
  }
}

This class contains an error_dict property, which packages the additional arguments given to the Exception instance into a JSON-serializable object. It does so in an error-safe way, also reporting serialization errors if they occur.

Client

This submodule contains a custom EvaluationFunctionClient, which can be used to call other deployed evaluation functions.

class EvaluationFunctionClient

Client wrapped around the botocore.client.Lambda, for invoking deployed evaluation functions. On initialisation, it fetches credentials from environment variables "INVOKER_KEY", "INVOKER_ID" and "INVOKER_REGION", or from an optional environment file prescrived by env_path.

Example

from evaluation_function_utils.client import EvaluationFunctionClient
client = EvaluationFunctionClient()

def evaluation_function(response, answer, params): 
    return client.invoke('isExactEqual', response, answer, params)

In this example, the evaluation_function completely offloads grading to the deployed 'isExactEqual' function.

Note: The EvaluationFunctionClient.invoke method was designed to behave exactly as if the evaluation_function function defined in the targeted deployed function was called directly. This means that if errors are encountered an EvaluationException is raised.