Start Job

POST

prompt

optimize

import PiClient from 'withpi';

const client = new PiClient({
  apiKey: process.env['WITHPI_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const promptOptimizationStatus = await client.prompt.optimize.startJob({
    examples: [
      {
        llm_input: 'Tell me something different',
        llm_output: 'The lazy dog was jumped over by the quick brown fox',
      },
    ],
    initial_system_instruction: 'Write a great story around the given topic.',
    model_id: 'gpt-4o-mini',
    scoring_spec: {
      description: "Write a children's story communicating a simple life lesson.",
      dimensions: [
        {
          description: 'dimension1 description',
          label: 'dimension1',
          sub_dimensions: [
            { description: 'subdimension1 description', label: 'subdimension1', scoring_type: 'PI_SCORER' },
          ],
        },
      ],
      name: 'Sample Scoring Spec',
    },
    tuning_algorithm: 'DSPY',
  });

  console.log(promptOptimizationStatus.job_id);
}

main();

{
  "detailed_status": [
    "Downloading model",
    "Tuning prompt"
  ],
  "job_id": "1234abcd",
  "optimized_prompt_messages": [
    {
      "content": "Write a great story around the given topic.",
      "role": "system"
    },
    {
      "content": "{{ input }}",
      "role": "user"
    }
  ],
  "state": "RUNNING"
}

Authorizations

x-api-key

string

header

required

Body

application/json

examples

object[]

required

The examples (input-response pairs) to train and validate on

An example for training or evaluation

initial_system_instruction

string

required

The initial system instruction

Example:

"Write a great story around the given topic."

model_id

enum<string>

required

The model to use for generating responses

Available options:

gpt-4o-mini,

llama-3.1-8b,

mock-llm

scoring_spec

object

required

The scoring spec to optimize

scoring_spec.description

string

required

The application description

Example:

"Write a children's story communicating a simple life lesson."

scoring_spec.dimensions

object[]

required

The dimensions of the scoring spec

scoring_spec.dimensions.description

string

required

The description of the dimension

Example:

"Relevance of the response"

scoring_spec.dimensions.label

string

required

The label of the dimension

Example:

"Relevance"

scoring_spec.dimensions.sub_dimensions

object[]

required

The sub dimensions of the dimension

scoring_spec.dimensions.sub_dimensions.description

string

required

The description of the dimension

Example:

"Is the response relevant to the prompt?"

scoring_spec.dimensions.sub_dimensions.label

string

required

The label of the dimension

Example:

"Relevance to Prompt"

scoring_spec.dimensions.sub_dimensions.scoring_type

enum<string>

required

The type of scoring performed for this dimension

Available options:

PI_SCORER,

PYTHON_CODE,

CUSTOM_MODEL_SCORER

scoring_spec.dimensions.sub_dimensions.custom_model_id

string | null

The ID of the custom model to use for scoring. Only relevant for scoring_type of CUSTOM_MODEL_SCORER

Example:

"your-model-id"

scoring_spec.dimensions.sub_dimensions.parameters

number[] | null

The learned parameters for the scoring method. This represents piecewise linear interpolation between [0, 1].

Example:

[
  0.14285714285714285,
  0.2857142857142857,
  0.42857142857142855,
  0.5714285714285714,
  0.7142857142857143,
  0.8571428571428571
]

scoring_spec.dimensions.sub_dimensions.python_code

string | null

The PYTHON code associated the PYTHON_CODE DimensionScoringType.

Example:

"\ndef score(response_text: str, input_text: str, kwargs: dict) -> dict:\n word_count = len(response_text.split())\n if word_count > 10:\n return {\"score\": 0.2, \"explanation\": \"Response has more than 10 words\"}\n elif word_count > 5:\n return{\"score\": 0.6, \"explanation\": \"Response has more than 5 words\"}\n else:\n return {\"score\": 1, \"explanation\": \"Response has 5 or fewer words\"}\n"

scoring_spec.dimensions.sub_dimensions.weight

number | null

The weight of the subdimension. The sum of subdimension weights will be normalized to one internally. A higher weight counts for more when aggregating this subdimension into the parent dimension.

Example:

1

Example:

{
  "description": "subdimension1 description",
  "label": "subdimension1",
  "scoring_type": "PI_SCORER"
}

scoring_spec.dimensions.parameters

number[] | null

The learned parameters for the scoring method. This represents piecewise linear interpolation between [0, 1].

Example:

[
  0.14285714285714285,
  0.2857142857142857,
  0.42857142857142855,
  0.5714285714285714,
  0.7142857142857143,
  0.8571428571428571
]

scoring_spec.dimensions.weight

number | null

The weight of the dimension The sum of dimension weights will be normalized to one internally. A higher weight counts for more when aggregating this dimension is aggregated into the final score.

Example:

1

Example:

[
  {
    "description": "dimension1 description",
    "label": "dimension1",
    "sub_dimensions": [
      {
        "description": "subdimension1 description",
        "label": "subdimension1",
        "scoring_type": "PI_SCORER"
      }
    ]
  }
]

scoring_spec.name

string

required

The name of the scoring spec

Example:

"Sample Scoring Spec"

tuning_algorithm

enum<string>

required

The tuning algorithm to use

Available options:

DSPY,

PI

dspy_optimization_type

enum<string> | null

The DSPY teleprompter/optimizer to use. This only applies for the DSPY. Leave it as None if tuning_algorithm != DSPY.

Available options:

BOOTSTRAP_FEW_SHOT,

COPRO,

MIPROv2

Example:

"COPRO"

use_chain_of_thought

boolean

default:false

Decides if to use chain of thought or not. This only applies for the DSPY. Leave it as None if tuning_algorithm != DSPY.

Example:

false

Response

200

application/json

Successful Response

The optimized_prompt_messages field is an empty list unless the state is done.

detailed_status

string[]

required

Detailed status of the job

Example:

["Downloading model", "Tuning prompt"]

job_id

string

required

The job id

Example:

"1234abcd"

state

enum<string>

required

Current state of the job

Available options:

QUEUED,

RUNNING,

DONE,

ERROR,

CANCELLED

optimized_prompt_messages

object[]

The optimized prompt messages in the OpenAI message format with the jinja {{ input }} variable for the next user prompt

Example:

[
  {
    "content": "Write a great story around the given topic.",
    "role": "system"
  },
  { "content": "{{ input }}", "role": "user" }
]

List Retrieve

import PiClient from 'withpi';

const client = new PiClient({
  apiKey: process.env['WITHPI_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const promptOptimizationStatus = await client.prompt.optimize.startJob({
    examples: [
      {
        llm_input: 'Tell me something different',
        llm_output: 'The lazy dog was jumped over by the quick brown fox',
      },
    ],
    initial_system_instruction: 'Write a great story around the given topic.',
    model_id: 'gpt-4o-mini',
    scoring_spec: {
      description: "Write a children's story communicating a simple life lesson.",
      dimensions: [
        {
          description: 'dimension1 description',
          label: 'dimension1',
          sub_dimensions: [
            { description: 'subdimension1 description', label: 'subdimension1', scoring_type: 'PI_SCORER' },
          ],
        },
      ],
      name: 'Sample Scoring Spec',
    },
    tuning_algorithm: 'DSPY',
  });

  console.log(promptOptimizationStatus.job_id);
}

main();

{
  "detailed_status": [
    "Downloading model",
    "Tuning prompt"
  ],
  "job_id": "1234abcd",
  "optimized_prompt_messages": [
    {
      "content": "Write a great story around the given topic.",
      "role": "system"
    },
    {
      "content": "{{ input }}",
      "role": "user"
    }
  ],
  "state": "RUNNING"
}

Get Started

Client Libraries

API Reference

Authorizations

Body

Response