Starcoder Truss

This is a Truss for Starcoder. Starcoder is an open-source language model trained specifically for code auto-completions. It was trained on text from over 80 programming languages. Check out more info about this model here.

Before deploying this model, you'll need to:

1. Accept the terms of service of the Starcoder model here.
2. Retrieve your Huggingface token from the settings.
3. Set your Huggingface token as a Baseten secret here with the key hf_api_key. Note that you will not be able to successfully deploy Starcoder without doing this.

Deploying Starcoder

To deploy the Starcoder Truss, you'll need to follow these steps:

1. Prerequisites: Make sure you have a Baseten account and API key. You can sign up for a Baseten account here.

2. Install Truss and the Baseten Python client: If you haven't already, install the Baseten Python client and Truss in your development environment using:

pip install --upgrade baseten truss

3. Load the Starcoder Truss: Assuming you've cloned this repo, spin up an IPython shell and load the Truss into memory:

Note this assumes that you started the ipython shell from root of the repo.

import truss

starcoder_truss = truss.load(".")

4. Log in to Baseten: Log in to your Baseten account using your API key (key found here):

import baseten

baseten.login("PASTE_API_KEY_HERE")

5. Deploy the Starcoder Truss: Deploy the Starcoder Truss to Baseten with the following command:

baseten.deploy(starcoder_truss)

Once your Truss is deployed, you can start using the Starcoder model through the Baseten platform! Navigate to the Baseten UI to watch the model build and deploy and invoke it via the REST API.

Starcoder API documentation

Input

This deployment of Starcoder takes a dictionary as input, which requires the following key:

• prompt - the prompt for code auto-completion

It also supports all parameters detailed in the transformers GenerationConfig.

Output

The result will be a dictionary containing:

• status - either success or failed
• data - dictionary containing keys completion, which is the model result, and prompt, which is the prompt from the input.
• message - will contain details in the case of errors

{"status": "success",
 "data": {"completion": "code for fibonacci sequence: '))\n\ndef fibonacci(n):\n    if n == 0:\n        return 0\n    elif n == 1:\n        return 1\n    else:\n        return fibonacci(n-1) + fibonacci(n-2)\n\nprint(fibonacci(n))\n",
  "prompt": "code for fib"},
 "message": null}

Example usage

curl -X POST https://app.baseten.co/models/EqwKvqa/predict \
  -H 'Authorization: Api-Key {YOUR_API_KEY}' \
  -d '{"prompt": "def compute_fib(n):"}'