Score

POST

scoring_system

score

import PiClient from 'withpi';

const client = new PiClient({
  apiKey: process.env['WITHPI_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const scoringSystemMetrics = await client.scoringSystem.score({
    llm_input: 'Tell me something different',
    llm_output: 'The lazy dog was jumped over by the quick brown fox',
    scoring_spec: [{ question: 'Is this response truthful?' }, { question: 'Is this response relevant?' }],
  });

  console.log(scoringSystemMetrics.total_score);
}

main();

{
  "dimension_scores": {},
  "question_scores": {
    "Is the response relevant?": 0.5,
    "Is the response truthful?": 0.89
  },
  "total_score": 0.4
}

Authorizations

x-api-key

string

header

required

Body

application/json

llm_input

string

required

The input to score

Example:

"Tell me something different"

llm_output

string

required

The output to score

Example:

"The lazy dog was jumped over by the quick brown fox"

scoring_spec

required

Either a scoring spec or a list of questions to score

scoring_spec.question

string

required

The yes/no question to ask Pi Scoring System.

Example:

"Is the response relevant to the prompt?"

scoring_spec.custom_model_id

string | null

The ID of the custom model associated with the CUSTOM_MODEL_SCORER scoring_type.

Example:

"your-model-id"

scoring_spec.label

string | null

The label of the question.

Example:

"Relevance to Prompt"

scoring_spec.parameters

number[] | null

The learned parameters for the scoring question define a piecewise linear interpolation over the range [0, 1]. This transformation adjusts the score distribution to better match your preferences—for example, by pulling scores below 0.5 closer to 0, and scores above 0.5 closer to 1.

Example:

[
  0.14285714285714285,
  0.2857142857142857,
  0.42857142857142855,
  0.5714285714285714,
  0.7142857142857143,
  0.8571428571428571
]

scoring_spec.python_code

string | null

The PYTHON code associated with the PYTHON_CODE scoring_type.

Example:

"\ndef score(response_text: str, input_text: str, kwargs: dict) -> dict:\n word_count = len(response_text.split())\n if word_count > 10:\n return {\"score\": 0.2, \"explanation\": \"Response has more than 10 words\"}\n elif word_count > 5:\n return{\"score\": 0.6, \"explanation\": \"Response has more than 5 words\"}\n else:\n return {\"score\": 1, \"explanation\": \"Response has 5 or fewer words\"}\n"

scoring_spec.scoring_type

enum<string> | null

The type of scoring performed for this question. Default: PI_SCORER.

Available options:

PI_SCORER,

PYTHON_CODE,

CUSTOM_MODEL_SCORER

Example:

"PI_SCORER"

scoring_spec.tag

string | null

The tag or the group to which this question belongs.

Example:

"Legal Formatting"

scoring_spec.weight

number | null

The weight of the question which reflects its relative importance. The sum of question weights will be normalized to one internally. A higher weight counts for more when aggregating this subdimension into the parent dimension.

Example:

1

Example:

[
  { "question": "Is this response truthful?" },
  { "question": "Is this response relevant?" }
]

kwargs

object

Optional additional parameters (keyword arguments)

Example:

{ "any_param": "any_value" }

Response

200

application/json

Successful Response

dimension_scores

object

required

deprecated

The score components for each dimension

dimension_scores.{key}

object

dimension_scores.{key}.subdimension_scores

object

required

The score components for each subdimension

Example:

{ "subdimension1": 0.5 }

dimension_scores.{key}.total_score

number

required

The total score of the dimension

Example:

0.5

question_scores

object

required

The score components for each question

Example:

{
  "Is the response relevant?": 0.5,
  "Is the response truthful?": 0.89
}

total_score

number

required

The total score of the scoring spec

Example:

0.4

Example Notebooks Generate

import PiClient from 'withpi';

const client = new PiClient({
  apiKey: process.env['WITHPI_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const scoringSystemMetrics = await client.scoringSystem.score({
    llm_input: 'Tell me something different',
    llm_output: 'The lazy dog was jumped over by the quick brown fox',
    scoring_spec: [{ question: 'Is this response truthful?' }, { question: 'Is this response relevant?' }],
  });

  console.log(scoringSystemMetrics.total_score);
}

main();

{
  "dimension_scores": {},
  "question_scores": {
    "Is the response relevant?": 0.5,
    "Is the response truthful?": 0.89
  },
  "total_score": 0.4
}

Get Started

API Reference

Authorizations

Body

Response