Custom Models

Custom Models

Introduction

While you can use a wide range of built-in services for your Alloy workflow decisions, you may want to make decisions based on a proprietary dataset or model. For example, your company may have a fraud model based on your device and location data that you would like to apply in a workflow.

To address this, we allow clients to configure a “Custom Model” service within an Alloy workflow in the following way:

  1. You host and configure an endpoint that Alloy will POST to during an evaluation with any service data that have been configured to run before the custom model
  2. Your model returns a score to Alloy
  3. You can configure decision thresholds based on that score in the Alloy dashboard

Configuration

Web-server and public endpoint URL

To get started with Custom Models, you will need to host a publicly available web server that can process JSON over HTTP.

For security, we require the server to be HTTPS so any private information is encrypted end to end.

Authorization

We support HTTP Basic Auth on your endpoint. You can configure username and password credentials for your Custom Model in your organization’s settings page. Even if you have multiple custom models, there is only one shared credential, so all of your custom models must use the same Basic Auth username and password.

Workflow Configuration

Once you know your endpoint URL, contact your Alloy Solutions Architect with the endpoint URL. They will enable the custom model on our backend and give you a Custom Model ID.

In your workflow editor, create a new version that includes a “Custom Model Service”. If you don’t see “Custom Models” as an available service, please contact your Alloy Solutions Architect so they can enable it for you.

Click on the custom model service and then click on “Options”.

Finally, under “Additional service options” add a key/value of keycustom_model_id and value the Custom Model ID value Alloy provided you with.

Now the custom model is configured, and you may treat it like any other service.

Building and Debugging Your Custom Model

You can easily test what data and in what format we will POST to your endpoint URL by using our Custom Model test mode endpoints.

Important note: the data which is sent in the custom model in a live evaluation may be different from the data seen in historical evaluations depending on the configuration of your workflow.

The custom model will only see any service data before the custom model. That is, if you call a service or set a tag after the custom model, you won’t be guaranteed to see the data in the production custom model POST to your endpoint URL during an evaluation, but you may see the data in test mode.

To better test / debug your custom model, Alloy Solutions Architects can create a Separate Custom Model ID on our backend that is configured with an endpoint that will proxy to your development workstation such as https://ngrok.com/ that you provide us. With this setup, you can use our interactive endpoints as you develop without having to deploy or debug in production.

Custom Model Web-Server Response

Your custom model server must return the following response:

{
    "modelSuccess": boolean,
    "modelScore": Number,
    "modelVersion": Number
}

Get the custom model payload for an evaluation

HTTP Request

GET https://api.alloy.co/custom-models/evaluations/<evaluation_token>

curl https://api.alloy.co/custom-models/evaluations/<evaluation_token> \
  -u workflow_token:workflow_secret

It should return a value structured like this:

{
  "Alloy Data Supplied": [
    {
      "attributeName": "name_first",
      "attributeValue": "false"
    },
    {
      "attributeName": "name_last",
      "attributeValue": "false"
    },
    {
      "attributeName": "birth_date",
      "attributeValue": "false"
    },
  ],
  "error": null,
  "status_code": 200
}

Trigger a request to your custom model endpoint

HTTP REQUEST

POST https://api.alloy.co/custom-models/evaluations/<evaluation_token>

curl -X "POST" https://api.alloy.co/custom-models/evaluations/<evaluation_token> \
  -u workflow_token:workflow_secret \
  -d '{"custom_model_id": <custom_model_id>}' \
  -H 'content-type:application/json'

If successful will return a simple 200

{
  "error": null,
  "status_code": 200
}

This will trigger an end-to-end request against your custom model endpoint allowing you to inspect the request body and debug any server behavior. This will not re-run an evaluation and will use the service responses which are cached for the given historical evaluation_token.

Validate your custom model response

HTTP Request

POST https://api.alloy.co/custom-models/evaluations/<evaluation_token>/validate

curl -X "POST" https://api.alloy.co/custom-models/evaluations/<evaluation_token>/validate \
  -u workflow_token:workflow_secret \
 -H 'content-type:application/json' \
 -d '{"modelSuccess": true, "modelScore": 1, "modelVersion": 1}'

If successful

{
  "message": "Request valid",
  "error": null,
  "status_code": 200
}

If there is a validation error

{
  "status_code": 400,
  "error": {
    "minor_code": 4001,
    "type": "Request validation failure",
    "message": "Invalid request body",
    "details": {
      "message": "\"modelVersion\" must be a number",
      "path": "modelVersion",
      "type": "number.base",
      "context": {
        "key": "modelVersion"
      }
    },
    "error_token": <uuid>
  },
  "timestamp": 1621457333102,
  "evaluation_token": <evaluation_token>,
  "entity_token": null,
  "application_token": <application_token>,
  "application_version_id": <version>
}

Manually post your custom model's response to validate it.