quickstarts/Audio.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0etRtS83RcWS"
      },
      "source": [
        "# Gemini API: Audio Quickstart\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Audio.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r1IzNLho-NqV"
      },
      "source": [
        "This notebook provides an example of how to prompt Gemini 1.5 Pro using an audio file. In this case, you'll use a [sound recording](https://www.jfklibrary.org/asset-viewer/archives/jfkwha-006) of President John F. Kennedy’s 1961 State of the Union address."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y6eH_Aq_NyNi"
      },
      "outputs": [],
      "source": [
        "!pip install -q -U google-generativeai"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LSe1pMEpR2L2"
      },
      "outputs": [],
      "source": [
        "import google.generativeai as genai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TXiv-NeZR5WA"
      },
      "source": [
        "## Configure your API key\n",
        "\n",
        "To run the following cell, your API key must be stored it in a Colab Secret named `GOOGLE_API_KEY`. If you don't already have an API key, or you're not sure how to create a Colab Secret, see [Authentication](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Authentication.ipynb) for an example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dm-iaNMGPdid"
      },
      "outputs": [],
      "source": [
        "from google.colab import userdata\n",
        "GOOGLE_API_KEY=userdata.get('GOOGLE_API_KEY')\n",
        "\n",
        "genai.configure(api_key=GOOGLE_API_KEY)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2YoxMrCdR7hf"
      },
      "source": [
        "## Upload an audio file with the File API\n",
        "\n",
        "To use an audio file in your prompt, you must first upload it using the [File API](https://github.com/google-gemini/cookbook/blob/main/quickstarts/File_API.ipynb).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OHvNLws4RRjx"
      },
      "outputs": [],
      "source": [
        "URL = \"https://storage.googleapis.com/generativeai-downloads/data/State_of_the_Union_Address_30_January_1961.mp3\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Cxq31LDwSFH6"
      },
      "outputs": [],
      "source": [
        "!wget -q $URL -O sample.mp3"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MAObE0BpaAwG"
      },
      "outputs": [],
      "source": [
        "your_file = genai.upload_file(path='sample.mp3')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m01XDoo4UQvN"
      },
      "source": [
        "## Use the file in your prompt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YmISEsqpafRb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "## Summary of President John F. Kennedy's 1961 State of the Union Address:\n",
            "\n",
            "**Main Theme:** The address focuses on the challenges and opportunities facing the United States both domestically and internationally, emphasizing the need for unity and action in the face of the Cold War and economic difficulties.\n",
            "\n",
            "**Key Points:**\n",
            "\n",
            "* **Economic Concerns:** Kennedy highlights the country's economic troubles, including a recession, high unemployment, and falling farm income. He proposes measures to address these issues, such as increased unemployment compensation, minimum wage increases, and tax incentives for investment.\n",
            "* **Balance of Payments Deficit:** Kennedy acknowledges the growing deficit but assures the nation that the dollar remains strong and pledges not to devalue it. He outlines steps to attract foreign investment, promote exports, and curb spending abroad.\n",
            "* **Unfinished Domestic Tasks:** Kennedy addresses several domestic issues needing attention, including urban decay, education, healthcare, and juvenile crime. He proposes programs for housing, education funding, and healthcare for the elderly.\n",
            "* **Foreign Policy and the Cold War:** Kennedy outlines the global challenges posed by the Cold War and the threat of communist expansion, particularly in Asia, Africa, and Latin America. He emphasizes the need to strengthen military capabilities and alliances while seeking peaceful competition with the Soviet Union and China. \n",
            "* **Alliance for Progress:** Kennedy proposes a new program to assist the economic and social development of Latin American countries, aiming for a \"free and prosperous Latin America.\"\n",
            "* **National Peace Corps:** Kennedy advocates for the creation of a National Peace Corps to utilize the skills of dedicated citizens to assist developing nations.\n",
            "* **Focus on Science and Diplomacy:** Kennedy calls for increased emphasis on science and diplomacy, proposing collaborations with other nations, including the Soviet Union, on projects like weather prediction, communication satellites, and space exploration.\n",
            "* **Strengthening the United Nations:** Kennedy emphasizes the importance of supporting and strengthening the United Nations as an instrument for peace and international cooperation.\n",
            "* **Call to Action:** Kennedy concludes by urging unity, dedication, and perseverance from all citizens to overcome the challenges facing the nation. He emphasizes the responsibility of the United States to lead the fight for freedom and world order. \n",
            "\n",
            "**Overall Tone:**  The address conveys a sense of urgency and determination while remaining optimistic about the nation's ability to overcome its challenges. Kennedy's call to action emphasizes shared responsibility and the need for collective effort in facing both domestic and international difficulties. \n",
            "\n"
          ]
        }
      ],
      "source": [
        "prompt = \"Listen carefully to the following audio file. Provide a brief summary.\"\n",
        "model = genai.GenerativeModel('models/gemini-1.5-pro-latest')\n",
        "response = model.generate_content([prompt, your_file])\n",
        "print(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WVFm2MOLWJO5"
      },
      "source": [
        "## Count audio tokens\n",
        "\n",
        "You can count the number of tokens in your audio file like this."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O0xk2-6CWLfC"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "total_tokens: 78330"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model.count_tokens([your_file])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zxxIUR8SV6dK"
      },
      "source": [
        "## Learning more"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zudj6gxEWR2Q"
      },
      "source": [
        "* Learn more about the [File API](https://github.com/google-gemini/cookbook/blob/main/quickstarts/File_API.ipynb) with the quickstart.\n",
        "\n",
        "* Learn more about prompting with [media files](https://ai.google.dev/tutorials/prompting_with_media) in the docs, including the supported formats and maximum length for audio files."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Audio.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}