Start Job

POST

scoring_system

calibrate

import PiClient from 'withpi';

const client = new PiClient({
  apiKey: process.env['WITHPI_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const scoringSpecCalibrationStatus = await client.scoringSystem.calibrate.startJob({
    examples: [
      { llm_input: 'good input', llm_output: 'good response', rating: 'Strongly Agree' },
      { llm_input: 'neutral input', llm_output: 'neutral response', rating: 'Neutral' },
    ],
    preference_examples: [
      { chosen: 'chosen response', llm_input: 'some input', rejected: 'rejected response' },
    ],
    scoring_spec: [{ question: 'Is this response truthful?' }, { question: 'Is this response relevant?' }],
  });

  console.log(scoringSpecCalibrationStatus.job_id);
}

main();

{
  "calibrated_scoring_spec": [
    {
      "custom_model_id": "your-model-id",
      "is_lower_score_desirable": "False",
      "label": "Relevance to Prompt",
      "parameters": [
        0.14285714285714285,
        0.2857142857142857,
        0.42857142857142855,
        0.5714285714285714,
        0.7142857142857143,
        0.8571428571428571
      ],
      "python_code": "\ndef score(response_text: str, input_text: str, kwargs: dict) -> dict:\n    word_count = len(response_text.split())\n    if word_count > 10:\n        return {\"score\": 0.2, \"explanation\": \"Response has more than 10 words\"}\n    elif word_count > 5:\n        return{\"score\": 0.6, \"explanation\": \"Response has more than 5 words\"}\n    else:\n        return {\"score\": 1, \"explanation\": \"Response has 5 or fewer words\"}\n",
      "question": "Is the response relevant to the prompt?",
      "scoring_type": "PI_SCORER",
      "tag": "Legal Formatting",
      "weight": 1
    }
  ],
  "detailed_status": [
    "Downloading model",
    "Tuning prompt"
  ],
  "job_id": "1234abcd",
  "state": "RUNNING"
}

Authorizations

x-api-key

string

header

required

Body

application/json

examples

object[]

required

Rated examples to use when calibrating the scoring spec. Must specify either the examples or the preference examples

An labeled example for training or evaluation

Example:

[
  {
    "llm_input": "good input",
    "llm_output": "good response",
    "rating": "Strongly Agree"
  },
  {
    "llm_input": "neutral input",
    "llm_output": "neutral response",
    "rating": "Neutral"
  }
]

preference_examples

object[]

required

Preference examples to use when calibrating the scoring spec. Must specify either the examples or preference examples

An preference example for training or evaluation

Example:

[
  {
    "chosen": "chosen response",
    "llm_input": "some input",
    "rejected": "rejected response"
  }
]

scoring_spec

required

Either a scoring spec or a list of questions to score

scoring_spec.question

string

required

The yes/no question to ask Pi Scoring System.

Example:

"Is the response relevant to the prompt?"

scoring_spec.custom_model_id

string | null

The ID of the custom model associated with the CUSTOM_MODEL_SCORER scoring_type.

Example:

"your-model-id"

scoring_spec.is_lower_score_desirable

boolean

default:false

Indicates whether a lower score represents a better outcome (e.g., fewer errors, less toxicity)

Example:

"False"

scoring_spec.label

string | null

The label of the question.

Example:

"Relevance to Prompt"

scoring_spec.parameters

number[] | null

The learned parameters for the scoring question define a piecewise linear interpolation over the range [0, 1]. This transformation adjusts the score distribution to better match your preferences—for example, by pulling scores below 0.5 closer to 0, and scores above 0.5 closer to 1.

Example:

[
  0.14285714285714285,
  0.2857142857142857,
  0.42857142857142855,
  0.5714285714285714,
  0.7142857142857143,
  0.8571428571428571
]

scoring_spec.python_code

string | null

The PYTHON code associated with the PYTHON_CODE scoring_type.

Example:

"\ndef score(response_text: str, input_text: str, kwargs: dict) -> dict:\n word_count = len(response_text.split())\n if word_count > 10:\n return {\"score\": 0.2, \"explanation\": \"Response has more than 10 words\"}\n elif word_count > 5:\n return{\"score\": 0.6, \"explanation\": \"Response has more than 5 words\"}\n else:\n return {\"score\": 1, \"explanation\": \"Response has 5 or fewer words\"}\n"

scoring_spec.scoring_type

enum<string> | null

The type of scoring performed for this question. Default: PI_SCORER.

Available options:

PI_SCORER,

PYTHON_CODE,

CUSTOM_MODEL_SCORER

Example:

"PI_SCORER"

scoring_spec.tag

string | null

The tag or the group to which this question belongs.

Example:

"Legal Formatting"

scoring_spec.weight

number | null

The weight of the question which reflects its relative importance. The sum of question weights will be normalized to one internally. A higher weight counts for more when aggregating this subdimension into the parent dimension.

Example:

1

Example:

[
  {
    "is_lower_score_desirable": false,
    "question": "Is this response truthful?"
  },
  {
    "is_lower_score_desirable": false,
    "question": "Is this response relevant?"
  }
]

strategy

enum<string>

The strategy to use to calibrate the scoring spec. FULL would take longer than LITE but may result in better result.

Available options:

LITE,

FULL

Response

200

application/json

Successful Response

detailed_status

string[]

required

Detailed status of the job

Example:

["Downloading model", "Tuning prompt"]

job_id

string

required

The job id

Example:

"1234abcd"

state

enum<string>

required

Current state of the job

Available options:

QUEUED,

RUNNING,

DONE,

ERROR,

CANCELLED

calibrated_scoring_spec

object[] | null

The calibrated scoring spec

calibrated_scoring_spec.question

string

required

The yes/no question to ask Pi Scoring System.

Example:

"Is the response relevant to the prompt?"

calibrated_scoring_spec.custom_model_id

string | null

The ID of the custom model associated with the CUSTOM_MODEL_SCORER scoring_type.

Example:

"your-model-id"

calibrated_scoring_spec.is_lower_score_desirable

boolean

default:false

Indicates whether a lower score represents a better outcome (e.g., fewer errors, less toxicity)

Example:

"False"

calibrated_scoring_spec.label

string | null

The label of the question.

Example:

"Relevance to Prompt"

calibrated_scoring_spec.parameters

number[] | null

Example:

[
  0.14285714285714285,
  0.2857142857142857,
  0.42857142857142855,
  0.5714285714285714,
  0.7142857142857143,
  0.8571428571428571
]

calibrated_scoring_spec.python_code

string | null

The PYTHON code associated with the PYTHON_CODE scoring_type.

Example:

calibrated_scoring_spec.scoring_type

enum<string> | null

The type of scoring performed for this question. Default: PI_SCORER.

Available options:

PI_SCORER,

PYTHON_CODE,

CUSTOM_MODEL_SCORER

Example:

"PI_SCORER"

calibrated_scoring_spec.tag

string | null

The tag or the group to which this question belongs.

Example:

"Legal Formatting"

calibrated_scoring_spec.weight

number | null

Example:

1

Generate Retrieve

import PiClient from 'withpi';

const client = new PiClient({
  apiKey: process.env['WITHPI_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const scoringSpecCalibrationStatus = await client.scoringSystem.calibrate.startJob({
    examples: [
      { llm_input: 'good input', llm_output: 'good response', rating: 'Strongly Agree' },
      { llm_input: 'neutral input', llm_output: 'neutral response', rating: 'Neutral' },
    ],
    preference_examples: [
      { chosen: 'chosen response', llm_input: 'some input', rejected: 'rejected response' },
    ],
    scoring_spec: [{ question: 'Is this response truthful?' }, { question: 'Is this response relevant?' }],
  });

  console.log(scoringSpecCalibrationStatus.job_id);
}

main();

{
  "calibrated_scoring_spec": [
    {
      "custom_model_id": "your-model-id",
      "is_lower_score_desirable": "False",
      "label": "Relevance to Prompt",
      "parameters": [
        0.14285714285714285,
        0.2857142857142857,
        0.42857142857142855,
        0.5714285714285714,
        0.7142857142857143,
        0.8571428571428571
      ],
      "python_code": "\ndef score(response_text: str, input_text: str, kwargs: dict) -> dict:\n    word_count = len(response_text.split())\n    if word_count > 10:\n        return {\"score\": 0.2, \"explanation\": \"Response has more than 10 words\"}\n    elif word_count > 5:\n        return{\"score\": 0.6, \"explanation\": \"Response has more than 5 words\"}\n    else:\n        return {\"score\": 1, \"explanation\": \"Response has 5 or fewer words\"}\n",
      "question": "Is the response relevant to the prompt?",
      "scoring_type": "PI_SCORER",
      "tag": "Legal Formatting",
      "weight": 1
    }
  ],
  "detailed_status": [
    "Downloading model",
    "Tuning prompt"
  ],
  "job_id": "1234abcd",
  "state": "RUNNING"
}

Get Started

API Reference

Authorizations

Body

Response