cookbook/liteLLM_Getting_Started.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "MZ01up0p7wOJ"
      },
      "source": [
        "## 🚅 liteLLM Quick Start Demo\n",
        "### TLDR: Call 50+ LLM APIs using chatGPT Input/Output format\n",
        "https://github.com/BerriAI/litellm\n",
        "\n",
        "liteLLM is package to simplify calling **OpenAI, Azure, Llama2, Cohere, Anthropic, Huggingface API Endpoints**. LiteLLM manages\n",
        "\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "RZtzCnQS7rW-"
      },
      "source": [
        "## Installation and setting Params"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rsrN5W-N7L8d"
      },
      "outputs": [],
      "source": [
        "!pip install litellm"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "ArrWyG5b7QAG"
      },
      "outputs": [],
      "source": [
        "from litellm import completion\n",
        "import os"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "bbhJRt34_NJ1"
      },
      "source": [
        "## Set your API keys\n",
        "- liteLLM reads your .env, env variables or key manager for Auth\n",
        "\n",
        "Set keys for the models you want to use below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "-h8Ga5cR7SvV"
      },
      "outputs": [],
      "source": [
        "# Only set keys for the LLMs you want to use\n",
        "os.environ['OPENAI_API_KEY'] = \"\" #@param\n",
        "os.environ[\"ANTHROPIC_API_KEY\"] = \"\" #@param\n",
        "os.environ[\"REPLICATE_API_KEY\"] = \"\" #@param\n",
        "os.environ[\"COHERE_API_KEY\"] = \"\" #@param\n",
        "os.environ[\"AZURE_API_BASE\"] = \"\" #@param\n",
        "os.environ[\"AZURE_API_VERSION\"] = \"\" #@param\n",
        "os.environ[\"AZURE_API_KEY\"] = \"\" #@param"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "fhqpKv6L8fBj"
      },
      "source": [
        "## Call chatGPT"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "speIkoX_8db4",
        "outputId": "331a6c65-f121-4e65-e121-bf8aaad05d9d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<OpenAIObject chat.completion id=chatcmpl-820kPkRwSLml4X6165fWbZlEDOedr at 0x12ff93630> JSON: {\n",
              "  \"id\": \"chatcmpl-820kPkRwSLml4X6165fWbZlEDOedr\",\n",
              "  \"object\": \"chat.completion\",\n",
              "  \"created\": 1695490221,\n",
              "  \"model\": \"gpt-3.5-turbo-0613\",\n",
              "  \"choices\": [\n",
              "    {\n",
              "      \"index\": 0,\n",
              "      \"message\": {\n",
              "        \"role\": \"assistant\",\n",
              "        \"content\": \"I'm sorry, but as an AI text-based model, I don't have real-time information. However, you can check the current weather in San Francisco by searching for \\\"weather in SF\\\" on any search engine or checking a weather website or app.\"\n",
              "      },\n",
              "      \"finish_reason\": \"stop\"\n",
              "    }\n",
              "  ],\n",
              "  \"usage\": {\n",
              "    \"prompt_tokens\": 13,\n",
              "    \"completion_tokens\": 51,\n",
              "    \"total_tokens\": 64\n",
              "  },\n",
              "  \"response_ms\": 2385.592\n",
              "}"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "completion(model=\"gpt-3.5-turbo\", messages=[{ \"content\": \"what's the weather in SF\",\"role\": \"user\"}])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Q3jV1Uxv8zNo"
      },
      "source": [
        "## Call Claude-2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V8yTWYzY8m9S",
        "outputId": "8b6dd32d-f9bf-4e89-886d-47cb8020f025"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<ModelResponse chat.completion id=chatcmpl-6d1a40c0-19c0-4bd7-9ca2-a91d8b8c2295 at 0x12ff85a40> JSON: {\n",
              "  \"object\": \"chat.completion\",\n",
              "  \"choices\": [\n",
              "    {\n",
              "      \"finish_reason\": \"stop_sequence\",\n",
              "      \"index\": 0,\n",
              "      \"message\": {\n",
              "        \"content\": \" Unfortunately I don't have enough context to know the exact location you are asking about when you say \\\"SF\\\". SF could refer to San Francisco, California, or potentially other cities that go by SF as an abbreviation. To get an accurate weather report, it would be helpful if you could provide the full city name and state/country. If you are looking for the weather in San Francisco, California, I would be happy to provide that forecast. Please let me know the specific location you want the weather for.\",\n",
              "        \"role\": \"assistant\",\n",
              "        \"logprobs\": null\n",
              "      }\n",
              "    }\n",
              "  ],\n",
              "  \"id\": \"chatcmpl-6d1a40c0-19c0-4bd7-9ca2-a91d8b8c2295\",\n",
              "  \"created\": 1695490260.983768,\n",
              "  \"response_ms\": 6351.544,\n",
              "  \"model\": \"claude-2\",\n",
              "  \"usage\": {\n",
              "    \"prompt_tokens\": 14,\n",
              "    \"completion_tokens\": 102,\n",
              "    \"total_tokens\": 116\n",
              "  }\n",
              "}"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "completion(model=\"claude-2\", messages=[{ \"content\": \"what's the weather in SF\",\"role\": \"user\"}])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "yu0LPDmW9PJa"
      },
      "source": [
        "## Call llama2 on replicate"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0GWV5mtO9Jbu",
        "outputId": "38538825-b271-406d-a437-f5cf0eb7e548"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<ModelResponse chat.completion id=chatcmpl-3151c2eb-b26f-4c96-89b5-ed1746b219e0 at 0x138b87e50> JSON: {\n",
              "  \"object\": \"chat.completion\",\n",
              "  \"choices\": [\n",
              "    {\n",
              "      \"finish_reason\": \"stop\",\n",
              "      \"index\": 0,\n",
              "      \"message\": {\n",
              "        \"content\": \" I'm happy to help! However, I must point out that the question \\\"what's the weather in SF\\\" doesn't make sense as \\\"SF\\\" could refer to multiple locations. Could you please clarify which location you are referring to? San Francisco, California or Sioux Falls, South Dakota? Once I have more context, I would be happy to provide you with accurate and reliable information.\",\n",
              "        \"role\": \"assistant\",\n",
              "        \"logprobs\": null\n",
              "      }\n",
              "    }\n",
              "  ],\n",
              "  \"id\": \"chatcmpl-3151c2eb-b26f-4c96-89b5-ed1746b219e0\",\n",
              "  \"created\": 1695490237.714101,\n",
              "  \"response_ms\": 12109.565,\n",
              "  \"model\": \"replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1\",\n",
              "  \"usage\": {\n",
              "    \"prompt_tokens\": 6,\n",
              "    \"completion_tokens\": 78,\n",
              "    \"total_tokens\": 84\n",
              "  },\n",
              "  \"ended\": 1695490249.821266\n",
              "}"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model = \"replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1\"\n",
        "completion(model=model, messages=[{ \"content\": \"what's the weather in SF\",\"role\": \"user\"}])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "HXdj5SEe9iLK"
      },
      "source": [
        "## Call Command-Nightly"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EaUq2xIx9fhr",
        "outputId": "55fe6f52-b58b-4729-948a-74dac4b431b2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<ModelResponse chat.completion id=chatcmpl-dc0d8ead-071d-486c-a111-78975b38794b at 0x1389725e0> JSON: {\n",
              "  \"object\": \"chat.completion\",\n",
              "  \"choices\": [\n",
              "    {\n",
              "      \"finish_reason\": \"stop\",\n",
              "      \"index\": 0,\n",
              "      \"message\": {\n",
              "        \"content\": \" As an AI model I don't have access to real-time data, so I can't tell\",\n",
              "        \"role\": \"assistant\",\n",
              "        \"logprobs\": null\n",
              "      }\n",
              "    }\n",
              "  ],\n",
              "  \"id\": \"chatcmpl-dc0d8ead-071d-486c-a111-78975b38794b\",\n",
              "  \"created\": 1695490235.936903,\n",
              "  \"response_ms\": 1022.6759999999999,\n",
              "  \"model\": \"command-nightly\",\n",
              "  \"usage\": {\n",
              "    \"prompt_tokens\": 6,\n",
              "    \"completion_tokens\": 19,\n",
              "    \"total_tokens\": 25\n",
              "  }\n",
              "}"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "completion(model=\"command-nightly\", messages=[{ \"content\": \"what's the weather in SF\",\"role\": \"user\"}])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "1g9hSgsL9soJ"
      },
      "source": [
        "## Call Azure OpenAI"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "For azure openai calls ensure to add the `azure/` prefix to `model`. If your deployment-id is `chatgpt-test` set `model` = `azure/chatgpt-test`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AvLjR-PF-lt0",
        "outputId": "deff2db3-b003-48cd-ea62-c03a68a4464a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<OpenAIObject chat.completion id=chatcmpl-820kZyCwbNvZATiLkNmXmpxxzvTKO at 0x138b84ae0> JSON: {\n",
              "  \"id\": \"chatcmpl-820kZyCwbNvZATiLkNmXmpxxzvTKO\",\n",
              "  \"object\": \"chat.completion\",\n",
              "  \"created\": 1695490231,\n",
              "  \"model\": \"gpt-35-turbo\",\n",
              "  \"choices\": [\n",
              "    {\n",
              "      \"index\": 0,\n",
              "      \"finish_reason\": \"stop\",\n",
              "      \"message\": {\n",
              "        \"role\": \"assistant\",\n",
              "        \"content\": \"Sorry, as an AI language model, I don't have real-time information. Please check your preferred weather website or app for the latest weather updates of San Francisco.\"\n",
              "      }\n",
              "    }\n",
              "  ],\n",
              "  \"usage\": {\n",
              "    \"completion_tokens\": 33,\n",
              "    \"prompt_tokens\": 14,\n",
              "    \"total_tokens\": 47\n",
              "  },\n",
              "  \"response_ms\": 1499.529\n",
              "}"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "completion(model=\"azure/chatgpt-v-2\", messages=[{ \"content\": \"what's the weather in SF\",\"role\": \"user\"}])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}