generative-ai Documentation

Sample code and notebooks for Generative AI on Google Cloud, with Gemini Enterprise Agent Platform

17,481 stars Jupyter Notebook 1 file · ~9,869 tokens #agents#gcp#gemini#gemini-api#gen-ai#generative-ai#google#google-cloud
RAW Doc

1. Project Overview & Quickstart (GoogleCloudPlatform/generative-ai)

File: README.md

Generative AI on Google Cloud

The latest Gemini models are available! Try out Gemini 3.7 Flash.

Gemini Enterprise Agent Platform, the latest evolution of Vertex AI, has been released!

Check out the Google-Cloud-AI/agent-platform repository for a curated list of assets for agent building on Google Cloud.

This repository contains notebooks, code samples, sample apps, and other resources that demonstrate how to use, develop and manage generative AI workflows using Generative AI with Agent Platform.

Intro Video

Description
gemini/ Discover Gemini through starter notebooks, use cases, function calling, sample apps, and more.
search/ Use this folder if you're interested in using Agent Search, a Google-managed solution to help you rapidly build search engines for websites and across enterprise data. (Formerly known as Enterprise Search on Generative AI App Builder).
rag-grounding/ Use this folder for information on Retrieval Augmented Generation (RAG) and Grounding. This is an index of notebooks and samples across other directories focused on this topic.
vision/ Use this folder if you're interested in building your own solutions from scratch using features from Imagen and Veo.
audio/ Use this folder if you're interested in building your own solutions from scratch using features from Chirp, a version of Google's Universal Speech Model (USM).
setup-env/ Instructions on how to set up Google Cloud, the Gen AI Python SDK, and notebook environments on Google Colab and Workbench.
RESOURCES.md Learning resources (e.g. blogs, YouTube playlists) about Generative AI on Google Cloud.

Contributing

Contributions welcome! See the Contributing Guide.

Getting help

Please use the issues page to provide suggestions, feedback or submit a bug report.

Disclaimer

This repository itself is not an officially supported Google product. The code in this repository is for demonstrative purposes only.

File: audio/speech/README.md

Speech Recognition and Generation

This repository explores various use-cases and implementations of speech recognition and generation technologies with Google Cloud Speech-to-Text and Text-to-Speech

Contents

Here's a breakdown of the content available:

  • Getting Started:
    • get_started_with_chirp_3.ipynb: This notebook provides a basic introduction to speech synthesis using Chirp 3 with Google Cloud Text-to-Speech API.
    • get_started_with_chirp_2_sdk.ipynb: This Jupyter Notebook provides a basic introduction to performing speech recognition using the Chirp 2 model with Google Cloud Speech-to-Text API.
  • Use Cases:
    • Storytelling:
      • storytelling.ipynb: This notebook delves into a specific application of speech technology - crafting engaging stories. It likely leverages both speech recognition and generation to create interactive or automated storytelling experiences.

File: embeddings/README.md

Embeddings

This repository explores various techniques and use-cases for embedding in Machine Learning, with a particular focus on text embeddings and their applications.

Notebooks

Use Cases

Outlier Detection

File: gemini/multimodal-live-api/gradio-voice/README.md

Gemini Voice Chat Demo

A low-latency bidirectional streaming pure-python application that enables voice conversations with Gemini using Gradio. This demo showcases real-time audio streaming capabilities, allowing natural conversations where you can speak with Gemini and receive audio responses back.

Features

  • Real-time audio streaming using WebRTC
  • Voice-to-voice conversations with Gemini
  • Multiple voice options for Gemini's responses
  • Simple web interface built with Gradio
  • Low-latency bidirectional communication

Prerequisites

Before running the application, you need to:

  1. Enable Vertex AI in your Google Cloud Project

  2. Set up authentication

Installation

  1. Clone this repository:

    bash
    git clone [email protected]:GoogleCloudPlatform/generative-ai.git
cd generative-ai/gemini/multimodal-live-api/gradio-voice

  2. Install the required dependencies:

    bash
    pip install -r requirements.txt

The main dependencies are:

  • fastrtc
  • librosa
  • google-genai

Usage

  1. Run the application:

    bash
    python app.py

  2. Open your web browser and navigate to the local Gradio interface (typically http://localhost:7860)

  3. Configure the application:

    • Enter your Google Cloud Project ID
    • Select your preferred region (default: us-central1)
    • Choose a voice for Gemini (options: Puck, Charon, Kore, Fenrir, Aoede)
    • Click "Submit"

  4. Start the conversation:

    • Click the record button to start speaking
    • Listen to Gemini's responses through your speakers

Configuration Options

Regions

The application supports any Google Cloud regions, with presets for:

  • us-central1 (default)
  • us-east5
  • us-south1
  • us-west4
  • us-east4
  • us-east1
  • us-west1

Additional locations can be found in the Vertex AI documentation.

Voice Options

Gemini can respond using different voice personalities:

  • Puck (default)
  • Charon
  • Kore
  • Fenrir
  • Aoede

Technical Details

The application uses:

  • Gradio for the web interface
  • WebRTC for real-time audio streaming
  • Google Gen AI SDK for Gemini integration
  • Vertex AI for model hosting and inference

The audio streaming implementation uses:

  • 16kHz input sample rate
  • 24kHz output sample rate
  • PCM audio format
  • Base64 encoding for audio transmission

Deployment Notes

For deploying behind a firewall, you may need to modify the WebRTC configuration. See the Gradio WebRTC deployment documentation for more details.

File: gemini/multimodal-live-api/websocket-demo-app/README.md

Multimodal Live API Demo

In this tutorial, you will be building a web application that enables you to use your voice and camera to talk to Gemini 2.0 through the Multimodal Live API.

The Multimodal Live API is a low-latency bidirectional streaming API that supports audio and video streaming inputs and can output audio.

Architecture

  • Backend (Python WebSockets Server): Handles authentication and acts as an intermediary between your frontend and the Gemini API.
  • Frontend (HTML/JavaScript): Provides the user interface and interacts with the backend via WebSockets.

Pre-requisites

While some web development experience, particularly with localhost, port numbers, and the distinction between WebSockets and HTTP requests, can be beneficial for this tutorial, don't worry if you're not familiar with these concepts. We'll provide guidance along the way to ensure you can successfully follow along.

File Structure

  • backend/main.py: The Python backend code

  • backend/requirements.txt: Lists the required Python dependencies

  • frontend/index.html: The frontend HTML app

  • frontend/script.js: Main frontend JavaScript code

  • frontend/gemini-live-api.js: Script for interacting with the Gemini API

  • frontend/live-media-manager.js: Script for handling media input and output

  • frontend/pcm-processor.js: Script for processing PCM audio

  • frontend/cookieJar.js: Script for managing cookies

Setup instructions

You can set up this app locally or via Cloud Shell.

Setup locally

  1. Clone the repository and cd into the correct directory

    sh
    git clone https://github.com/GoogleCloudPlatform/generative-ai.git
cd generative-ai/gemini/multimodal-live-api/websocket-demo-app

  2. Create a new virtual environment and activate it:

    sh
    python3 -m venv env
source env/bin/activate

  3. Install dependencies:

    sh
    pip3 install -r backend/requirements.txt

  4. Start the Python WebSocket server:

    sh
    python3 backend/main.py

  5. Start the frontend:

    • Navigate to script.js on line 9, const PROXY_URL = "wss://[THE_URL_YOU_COPIED_WITHOUT_HTTP]"; and replace PROXY_URL value with ws://localhost:8000. It should look like: const PROXY_URL = "ws://localhost:8000";. Note the absence of the second "s" in "wss" as "ws" indicates a non-secure WebSocket connection.
    • Right below on line 10, update PROJECT_ID with your Google Cloud project ID.
    • Save the changes you've made to script.js
    • Now make sure to open a separate terminal window from the backend to run this command (keep the backend server running in the first terminal).
    sh
    cd frontend
python3 -m http.server

  6. Point your browser to the demo app UI based on the output of the terminal. (e.g., it may be http://localhost:8000, or it may use a different port.)

  7. Get your Google Cloud access token:
    Run the following command in a terminal with gcloud installed to set your project, and to retrieve your access token.

    sh
    gcloud components update
gcloud components install beta
gcloud config set project YOUR-PROJECT-ID
gcloud auth print-access-token

  8. Copy the access token from the previous step into the UI that you have open in your browser.

  9. Enter the model ID in the UI:
    Replace YOUR-PROJECT-ID in the input with your Google Cloud Project ID.

  10. Connect and interact with the demo:

    • After entering your Access Token and Model ID, press the connect button to connect your web app. Now you should be able to interact with Gemini 2.0 with the Multimodal Live API.

  11. To interact with the app, you can do the following:

    • Text input: You can write a text prompt to send to the model by entering your message in the box and pressing the send arrow. The model will then respond via audio (turn up your volume!).
    • Voice input: Press the microphone button to stop speaking. The model will respond via audio. If you would like to mute your microphone, press the button with a slash through the microphone.
    • Video input: The model will also capture your camera input and send it to Gemini. You can ask questions about current or previous video footage. For more details on how this works, visit the documentation page for the Multimodal Live API.

Setup in Cloud Shell

  1. Open Cloud Shell

  2. Upload the frontend and backend folders to your Cloud Shell Editor project. Alternatively, you can clone the repository and cd into the correct directory:

    sh
    git clone https://github.com/GoogleCloudPlatform/generative-ai.git
cd generative-ai/gemini/multimodal-live-api/websocket-demo-app

  3. Open two new terminal windows.

  4. Navigate to whichever folder in Cloud Shell you uploaded the code files to (i.e., using cd your_folder_name)

  5. Install dependencies: In one of the terminal windows run:

    sh
    pip3 install -r backend/requirements.txt

  6. Start the Python WebSocket server in one terminal.

    sh
    python3 backend/main.py

  7. In order for index.html to work properly, you will need to update the app URL inside script.js to point to the correct proxy server URL you just set up in the previous step. To do so:

    • Click on Web Preview (to the right of the Open Terminal button near the top)
    • Click "Preview on port 8080" (the port where you've setup the proxy server in the previous step)
    • Copy the URL, but make sure to discard everything at the end after "cloudshell.dev/"
    • Navigate to const PROXY_URL = "wss://your websocket server"; in frontend/script.js on line 8
    • Replace wss://your websocket server with wss://[THE_URL_YOU_COPIED_WITHOUT_HTTP]. For example, it should look like: const PROXY_URL = "wss://8080-cs-123456789-default.cs-us-central1-abcd.cloudshell.dev";
    • Replace your project id with your project ID on line 9, for the const PROJECT_ID
    • save the changes you've made to script.js

  8. Start the frontend:
    In the second terminal window, run the command below. Keep the backend server running in the first terminal.
    (Make sure you have navigated to the folder containing the code files, i.e. using cd frontend)

    sh
    cd frontend
python3 -m http.server

  9. Test the demo app:

    • Navigate to the Web Preview button again
    • Click on "Change port"
    • Change Preview Port to 8000, and then click on "Change and Preview". This should open up a new tab with the UI.

  10. Going back to the tab with the Cloud Shell Editor, connect to the application by running the following command in a new terminal window:

    sh
    gcloud config set project YOUR-PROJECT-ID
gcloud auth print-access-token
    • Copy your access token and paste it in the Access Token field in the UI.
    • In the second field of the UI, labeled Project ID, add your Google Cloud Project ID
    • Press the "Connect" button. Now you should be able to interact with Gemini 2.0 with the Multimodal Live API.

  11. To interact with the app, you can do the following:

    • Text input: You can write a text prompt to send to the model by entering your message in the box and pressing the send arrow. The model will then respond via audio (turn up your volume!).
    • Voice input: Press the pink microphone button and start speaking. The model will respond via audio. If you would like to mute your microphone, press the button with a slash through the microphone.
    • Video input: The model will also capture your camera input and send it to Gemini. You can ask questions about current or previous video footage. For more details on how this works, visit the documentation page for the Multimodal Live API.

Setup in Cloud Run

  1. Clone the repository and cd into the correct directory

    sh
    git clone https://github.com/GoogleCloudPlatform/generative-ai.git
cd generative-ai/gemini/multimodal-live-api/websocket-demo-app

  2. Modify the frontend code to point the WebSocket endpoint to the same container:

    • Navigate to the script.js file on line 9, const PROXY_URL = "wss://[THE_URL_YOU_COPIED_WITHOUT_HTTP]"; and replace PROXY_URL value with /ws. It should look like: const PROXY_URL = "/ws";. Note the absence of the second "s" in "wss" as "ws" indicates a non-secure WebSocket connection. And there is no host part as it will use the same container as the frontend and backend.
    • Right below on line 10, update PROJECT_ID with your Google Cloud project ID.
    • Save the changes you've made to script.js

  3. Deploy the code to Cloud Run using the following gcloud command:

    sh
    gcloud run deploy --project=YOUR-PROJECT-ID \
--region=us-central1 \
--source=./ \
--allow-unauthenticated \
--port=8000  \
gemini-live-demo

  4. Last step command will output a link for the deployment if it run successfully. Copy the link to your browser and navigate to the demo app UI.

  5. Get your Google Cloud access token: Run the following command in a terminal with gcloud installed to set your project, and to retrieve your access token.

    sh
    gcloud components update
gcloud components install beta
gcloud config set project YOUR-PROJECT-ID
gcloud auth print-access-token

  6. Copy the access token from the previous step into the UI that you have open in your browser.

  7. Enter the model ID in the UI:
    Replace YOUR-PROJECT-ID in the input with your Google Cloud Project ID.

  8. Connect and interact with the demo:

    • After entering your Access Token and Model ID, press the connect button to connect your web app. Now you should be able to interact with Gemini 2.0 with the Multimodal Live API.

  9. To interact with the app, you can do the following:

    • Text input: You can write a text prompt to send to the model by entering your message in the box and pressing the send arrow. The model will then respond via audio (turn up your volume!).
    • Voice input: Press the microphone button to stop speaking. The model will respond via audio. If you would like to mute your microphone, press the button with a slash through the microphone.
    • Video input: The model will also capture your camera input and send it to Gemini. You can ask questions about current or previous video footage. For more details on how this works, visit the documentation page for the Multimodal Live API.

File: gemini/multimodal-live-api/README.md

Multimodal Live API

The Multimodal Live API is a low-latency bidirectional streaming API that supports text, audio and video input, with audio and text output. This capability allows for natural, human-like conversations, giving you the option to interrupt the model at any point in the interaction. The model's abilities are further enhanced through video understanding, enabling you to share live video input and screencasts so that Gemini can reason across this information and respond to questions. You can also provide system instructions to better control the model's output.

Demo

The following video demonstrating the Multimodal Live API shows an example use case of conversing with the model. This demo app can be run locally or via Cloud Shell so that you can test your own use cases. Instructions to get this application running are in the websocket-demo-app directory.

Getting Started

Intro Notebooks

  1. Multimodal Live API: Directly access the Multimodal Live API. This notebook will demonstrate text-to-text generation, as well as text-to-audio generation.

  2. Multimodal Live API via Gen AI SDK: Use this tutorial to access the Multimodal Live API using the Google Gen AI SDK in Vertex AI. You'll see examples of text-to-text generation and text-to-audio generation.

Use Cases

  1. Interactive Loan Application Assistant: This notebook provides a comprehensive demonstration of how Gemini 2.0 can act as your personal file assistant across various storage platforms. It empowers users to seamlessly understand and interact with their loan documents.

  2. Real-time RAG for Retail: Users will learn to develop a real-time Retrieval Augmented Generation (RAG) system leveraging the Multimodal Live API for a retail use-case. This system will generate audio and text responses grounded in provided documents.

Demo Applications

  1. Getting Started Web App: A basic web application to get started with voice and camera interactions.
  2. React Demo App: A comprehensive React client featuring real-time streaming, tool use, and media handling.
  3. Plain JS Demo App: A simple, dependency-free JavaScript implementation for understanding core API mechanics.
  4. Plain JS + Python SDK Demo App: A demo using the Google Gen AI Python SDK for the backend and vanilla JS for the frontend.
  5. Real-time Advisor: A specialized advisor persona that can switch between silent and outspoken modes.
  6. Customer Support Agent: An advanced agent with emotion detection, multimodal input, and tool execution.
  7. Gaming Assistant: A gaming companion with persona switching and screen sharing capabilities.

Tools

  1. PCM Audio Debugger: A standalone tool for testing and debugging raw PCM audio streams and WebSocket connections.

File: gemini/mcp/adk_mcp_app/README.md

An ADK Agent integrated with MCP Client

This web application was developed using Google ADK (Agent Development Kit) and MCP (Model Context Protocol). Specifically, the Agent relies on the Google ADK. A local MCP server instance, established using custom server code designed for cocktail data management, facilitates data retrieval. The web application acts as an MCP client to fetch cocktail information via this local server.

Screenshot:

This example demonstrates how you can chat with the app to retrieve cocktail details from The Cocktail DB site using a local MCP server

bash
python -m venv .venv
source .venv/bin/activate

Install ADK

bash
pip install .

Project Structure

bash
your_project_folder/  # Project folder
|── adk_mcp_app
    ├── main.py
    ├── .env
    ├── mcp_server
    │   └── cocktail.py
    ├── README.md
    ├── pyproject.toml
    ├── uv.lock
    └── static
        └── index.html

Run the app

Start the Fast API: Run the following command within the adk_mcp_app folder

  • Create a .env file with the following contents:
bash
# Choose Model Backend: 0 -> Gemini Developer API, 1 -> Vertex AI
GOOGLE_GENAI_USE_VERTEXAI=1

# Gemini Developer API backend config
GOOGLE_API_KEY=YOUR_VALUE_HERE

# Vertex AI backend config
GOOGLE_CLOUD_PROJECT="<your project id>"
GOOGLE_CLOUD_LOCATION="us-central1"
  • Run the below command to start the app:
bash
uvicorn main:app --reload

2. Official Technical Reference & Guides (GoogleCloudPlatform/terraform-genai-doc-summarization)

File: README.md

Generative AI Document Summarization

Description

Tagline

Create summaries of a large corpus of documents using Generative AI.

Detailed

This solution showcases how to summarize a large corpus of documents using Generative AI. It provides an end-to-end demonstration of document summarization going all the way from raw documents, detecting text in the documents and summarizing the documents on-demand using Vertex AI LLM APIs, Document AI Optical Character Recognition (OCR), and BigQuery.

PreDeploy

To deploy this blueprint you must have an active billing account and billing permissions.

Architecture

  • User uploads a new document triggering the webhook Cloud Function.
  • Document AI extracts the text from the document file.
  • A Vertex AI Large Language Model summarizes the document text.
  • The document summaries are stored in BigQuery.

Documentation

Deployment Duration

Configuration: 1 mins
Deployment: 5 mins

Cost

Cost Details

Inputs

Name Description Type Default Required
disable_services_on_destroy Whether project services will be disabled when the resources are destroyed. bool false no
documentai_location Document AI location, see https://cloud.google.com/document-ai/docs/regions string "us" no
labels A set of key/value label pairs to assign to the resources deployed by this blueprint. map(string) {} no
project_id The Google Cloud project ID to deploy to string n/a yes
region The Google Cloud region to deploy to string "us-central1" no
unique_names Whether to use unique names for resources bool false no

Outputs

Name Description
bigquery_dataset_id The name of the BigQuery dataset created
bucket_docs_name The name of the docs bucket created
bucket_main_name The name of the main bucket created
documentai_processor_id The full Document AI processor path ID
neos_walkthrough_url The URL to launch the in-console tutorial for the Generative AI Document Summarization solution
unique_id The unique ID for this deployment

Requirements

These sections describe requirements for using this module.

Software

The following dependencies must be available:

Service Account

A service account with the following roles must be used to provision
the resources of this module:

  • Storage Admin: roles/storage.admin

APIs

A project with the following APIs enabled must be used to host the
resources of this module:

  • Google Cloud Storage JSON API: storage-api.googleapis.com

Contributing

Refer to the contribution guidelines for
information on contributing to this module.

Security Disclosures

Please see our security disclosure process.

File: docs/CODE_OF_CONDUCT.md

Code of Conduct

Our Pledge

In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, gender identity and expression, level of
experience, education, socio-economic status, nationality, personal appearance,
race, religion, or sexual identity and orientation.

Our Standards

Examples of behavior that contributes to creating a positive environment
include:

  • Using welcoming and inclusive language
  • Being respectful of differing viewpoints and experiences
  • Gracefully accepting constructive criticism
  • Focusing on what is best for the community
  • Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

  • The use of sexualized language or imagery and unwelcome sexual attention or
    advances
  • Trolling, insulting/derogatory comments, and personal or political attacks
  • Public or private harassment
  • Publishing others' private information, such as a physical or electronic
    address, without explicit permission
  • Other conduct which could reasonably be considered inappropriate in a
    professional setting

Our Responsibilities

Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, or to ban temporarily or permanently any
contributor for other behaviors that they deem inappropriate, threatening,
offensive, or harmful.

Scope

This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.

This Code of Conduct also applies outside the project spaces when the Project
Steward has a reasonable belief that an individual's behavior may have a
negative impact on the project or its community.

Conflict Resolution

We do not believe that all conflict is bad; healthy debate and disagreement
often yield positive results. However, it is never okay to be disrespectful or
to engage in behavior that violates the project’s code of conduct.

If you see someone violating the code of conduct, you are encouraged to address
the behavior directly with those involved. Many issues can be resolved quickly
and easily, and this gives people more control over the outcome of their
dispute. If you are unable to resolve the matter for any reason, or if the
behavior is threatening or harassing, report it. We are dedicated to providing
an environment where participants feel welcome and safe.

Reports should be directed to [PROJECT STEWARD NAME(s) AND EMAIL(s)], the
Project Steward(s) for [PROJECT NAME]. It is the Project Steward’s duty to
receive and address reported violations of the code of conduct. They will then
work with a committee consisting of representatives from the Open Source
Programs Office and the Google Open Source Strategy team. If for any reason you
are uncomfortable reaching out to the Project Steward, please email
[email protected].

We will investigate every complaint, but you may not receive a direct response.
We will use our discretion in determining when and how to follow up on reported
incidents, which may range from not taking action to permanent expulsion from
the project and project-sponsored spaces. We will notify the accused of the
report and provide them an opportunity to discuss it before any action is taken.
The identity of the reporter will be omitted from the details of the report
supplied to the accused. In potentially harmful situations, such as ongoing
harassment or threats to anyone's safety, we may take action without notice.

Attribution

This Code of Conduct is adapted from the Contributor Covenant, version 1.4,
available at
https://www.contributor-covenant.org/version/1/4/code-of-conduct/

File: docs/contributing.md

How to Contribute

We would love to accept your patches and contributions to this project.

Before you begin

Sign our Contributor License Agreement

Contributions to this project must be accompanied by a
Contributor License Agreement (CLA).
You (or your employer) retain the copyright to your contribution; this simply
gives us permission to use and redistribute your contributions as part of the
project.

If you or your current employer have already signed the Google CLA (even if it
was for a different project), you probably don't need to do it again.

Visit https://cla.developers.google.com/ to see your current agreements or to
sign a new one.

Review our Community Guidelines

This project follows Google's Open Source Community
Guidelines.

Contribution process

Code Reviews

All submissions, including submissions by project members, require review. We
use GitHub pull requests
for this purpose.

File: examples/simple_example/README.md

Simple Example

Inputs

Name Description Type Default Required
project_id GCP project for provisioning cloud resources. any n/a yes

Outputs

No outputs.

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#agents Browse agents catalog → #gcp Browse gcp catalog → #gemini Browse gemini catalog → #gemini-api Browse gemini-api catalog → #gen-ai Browse gen-ai catalog → #generative-ai Browse generative-ai catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →