moondream Documentation

Tiny vision language model

10,122 stars Python Python 8 files · ~6,528 tokens
RAW Doc

Technical Documentation: m87-labs/moondream

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

🌔 moondream

a tiny vision language model that kicks ass and runs anywhere

Website | Demo

Examples

Image Example
**What is the girl doing?**The girl is sitting at a table and eating a large hamburger.**What color is the girl's hair?**The girl's hair is white.
**What is this?**This is a computer server rack, which is a device used to store and manage multiple computer servers. The rack is filled with various computer servers, each with their own dedicated space and power supply. The servers are connected to the rack via multiple cables, indicating that they are part of a larger system. The rack is placed on a carpeted floor, and there is a couch nearby, suggesting that the setup is in a living or entertainment area.**What is behind the stand?**Behind the stand, there is a brick wall.

About

Moondream is a highly efficient open-source vision language model that combines powerful image understanding capabilities with a remarkably small footprint. It's designed to be versatile and accessible, capable of running on a wide range of devices and platforms.

The project offers two model variants:

  • Moondream 2B: The primary model with 2 billion parameters, offering robust performance for general-purpose image understanding tasks including captioning, visual question answering, and object detection.
  • Moondream 0.5B: A compact 500 million parameter model specifically optimized as a distillation target for edge devices, enabling efficient deployment on resource-constrained hardware while maintaining impressive capabilities.

How to use

Moondream can be run locally, or in the cloud. Please refer to the Getting Started page for details.

Special thanks

Overview (`recipes/gaze-detection-video/README.md`)

Gaze Detection Video Processor

⚠️ IMPORTANT: This project currently uses Moondream 2B (2025-01-09 release) via the Hugging Face Transformers library. We will migrate to the official Moondream client
libraries once they become available for this version.

Table of Contents

Overview

This project uses the Moondream 2B model to detect faces and their gaze directions in videos. It processes videos frame by frame, visualizing face detections and gaze directions.

Sample Output

Input Video Processed Output

Features

  • Face detection in video frames
  • Gaze direction tracking
  • Real-time visualization with:
    • Colored bounding boxes for faces
    • Gradient lines showing gaze direction
    • Gaze target points
  • Supports multiple faces per frame
  • Processes all common video formats (.mp4, .avi, .mov, .mkv)
  • Uses Moondream 2 (2025-01-09 release) via Hugging Face Transformers
    • Note: Will be migrated to official client libraries in future updates
    • No authentication required

Prerequisites

  1. Python 3.8 or later
  2. CUDA-capable GPU recommended (but CPU mode works too)
  3. FFmpeg installed on your system

Installation

Linux/macOS Installation

  1. Install system dependencies:

    bash
    # Ubuntu/Debian
sudo apt-get update && sudo apt-get install -y libvips42 libvips-dev ffmpeg

# CentOS/RHEL
sudo yum install vips vips-devel ffmpeg

# macOS
brew install vips ffmpeg

  2. Clone and setup the project:

    bash
    git clone https://github.com/vikhyat/moondream.git
cd moondream/recipes/gaze-detection-video
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Windows Installation

Windows setup requires a few additional steps for proper GPU support and libvips installation.

  1. Clone the repository:

    bash
    git clone [repository-url]
cd moondream/recipes/gaze-detection-video

  2. Create and activate virtual environment:

    bash
    python -m venv venv
.\venv\Scripts\activate

  3. Install PyTorch with CUDA support:

    bash
    # For NVIDIA GPUs
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

  4. Install libvips: Download the appropriate version based on your system architecture:

    Architecture VIPS Version to Download
    32-bit x86 vips-dev-w32-all-8.16.0.zip
    64-bit x64 vips-dev-w64-all-8.16.0.zip
    • Extract the ZIP file
    • Copy all DLL files from vips-dev-8.16\bin to either:
      • Your project's root directory (easier) OR
      • C:\Windows\System32 (requires admin privileges)
    • Add to PATH:
      1. Open System Properties → Advanced → Environment Variables
      2. Under System Variables, find PATH
      3. Add the full path to the vips-dev-8.16\bin directory

  5. Install FFmpeg:

  6. Install other dependencies:

    bash
    pip install -r requirements.txt

Usage

  1. Place your input videos in the input directory

    • Supported formats: .mp4, .avi, .mov, .mkv
    • The directory will be created automatically if it doesn't exist

  2. Run the script:

    bash
    python gaze-detection-video.py

  3. The script will:

    • Process all videos in the input directory
    • Show progress bars for each video
    • Save processed videos to the output directory with prefix 'processed_'

Output

  • Processed videos are saved as output/processed_[original_name].[ext]
  • Each frame in the output video shows:
    • Colored boxes around detected faces
    • Lines indicating gaze direction
    • Points showing where each person is looking

Troubleshooting

  1. CUDA/GPU Issues:

    • Ensure you have CUDA installed for GPU support
    • The script will automatically fall back to CPU if no GPU is available

  2. Memory Issues:

    • If processing large videos, ensure you have enough RAM
    • Consider reducing video resolution if needed

  3. libvips Errors:

    • Make sure libvips is properly installed for your OS
    • Check system PATH includes libvips

  4. Video Format Issues:

    • Ensure FFmpeg is installed and in your system PATH
    • Try converting problematic videos to MP4 format

Performance Notes

  • GPU processing is significantly faster than CPU
  • Processing time depends on:
    • Video resolution
    • Number of faces per frame
    • Frame rate
    • Available computing power

Dependencies

  • transformers (for Moondream 2 model access)
  • torch
  • opencv-python
  • pillow
  • matplotlib
  • numpy
  • tqdm
  • pyvips
  • accelerate
  • einops

Model Details

⚠️ IMPORTANT: This project currently uses Moondream 2 (2025-01-09 release) via the Hugging Face Transformers library. We will migrate to the official Moondream client
libraries once they become available for this version.

The model is loaded using:

Overview (`recipes/promptable-content-moderation/README.md`)

Promptable Content Moderation with Moondream

Welcome to the future of content moderation with Moondream 2B, a powerful and lightweight vision-language model that enables detection and moderation of video content using natural language prompts.

Try it now.

Features

  • Content moderation through natural language prompts
  • Multiple visualization styles
  • Intelligent scene detection and tracking:
    • DeepSORT tracking with scene-aware reset
    • Persistent moderation across frames
    • Smart tracker reset at scene boundaries
  • Optional grid-based detection for improved accuracy on complex scenes
  • Frame-by-frame processing with IoU-based merging
  • Web-compatible output format
  • Test mode (process only first X seconds)
  • Advanced moderation analysis with multiple visualization plots

Examples

Prompt Output
"white cigarette"
"gun"
"confederate flag"

Requirements

Python Dependencies

For Windows users, before installing other requirements, first install PyTorch with CUDA support:

bash
pip install torch==2.5.1+cu121 torchvision==0.20.1+cu121 --index-url https://download.pytorch.org/whl/cu121

Then install the remaining dependencies:

bash
pip install -r requirements.txt

System Requirements

  • FFmpeg (required for video processing)
  • libvips (required for image processing)

Installation by platform:

Installation

  1. Clone this repository and create a new virtual environment:
bash
git clone https://github.com/vikhyat/moondream/blob/main/recipes/promptable-video-redaction
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
  1. Install Python dependencies:
bash
pip install -r requirements.txt
  1. Install ffmpeg and libvips:
    • On Ubuntu/Debian: sudo apt-get install ffmpeg libvips
    • On macOS: brew install ffmpeg
    • On Windows: Download from ffmpeg.org

Downloading libvips for Windows requires some additional steps, see here

Usage

The easiest way to use this tool is through its web interface, which provides a user-friendly experience for video content moderation.

Web Interface

  1. Start the web interface:
bash
python app.py

  1. Open the provided URL in your browser (typically http://localhost:7860)

  2. Use the interface to:

    • Upload your video file
    • Specify content to moderate (e.g., "face", "cigarette", "gun")
    • Choose redaction style (default: obfuscated-pixel)
    • OPTIONAL: Configure advanced settings
      • Processing speed/quality
      • Grid size for detection
      • Test mode for quick validation (default: on, 3 seconds)
    • Process the video and download results
    • Analyze detection patterns with visualization tools

Output Files

The tool generates two types of output files in the outputs directory:

  1. Processed Videos:

    • Format: [style]_[content_type]_[original_filename].mp4
    • Example: censor_inappropriate_video.mp4

  2. Detection Data:

    • Format: [style]_[content_type]_[original_filename]_detections.json
    • Contains frame-by-frame detection information
    • Used for visualization and analysis

Technical Details

Scene Detection and Tracking

The tool uses advanced scene detection and object tracking:

  1. Scene Detection:

    • Powered by PySceneDetect's ContentDetector
    • Automatically identifies scene changes in videos
    • Configurable detection threshold (default: 30.0)
    • Helps maintain tracking accuracy across scene boundaries

  2. Object Tracking:

    • DeepSORT tracking for consistent object identification
    • Automatic tracker reset at scene changes
    • Maintains object identity within scenes
    • Prevents tracking errors across scene boundaries

  3. Integration Benefits:

    • More accurate object tracking
    • Better handling of scene transitions
    • Reduced false positives in tracking
    • Improved tracking consistency

Best Practices

  • Use test mode for initial configuration
  • Enable grid-based detection for complex scenes
  • Choose appropriate redaction style based on content type:
    • Censor: Complete content blocking
    • Blur styles: Less intrusive moderation
    • Bounding Box: Content review and analysis
  • Monitor system resources during processing
  • Use appropriate processing quality settings based on your needs

Notes

  • Processing time depends on video length, resolution, GPU availability, and chosen settings
  • GPU is strongly recommended for faster processing
  • Grid-based detection increases accuracy but requires more processing time (each grid cell is processed independently)
  • Test mode processes only first X seconds (default: 3 seconds) for quick validation

Overview (`recipes/promptable-video-redaction/README.md`)

Promptable Video Redaction with Moondream

This tool uses Moondream 2B, a powerful yet lightweight vision-language model, to detect and redact objects from videos. Moondream can recognize a wide variety of objects, people,
text, and more with high accuracy while being much smaller than traditional models.

Try it now.

About Moondream

Moondream is a tiny yet powerful vision-language model that can analyze images and answer questions about them. It's designed to be lightweight and efficient while maintaining high
accuracy. Some key features:

  • Only 2B parameters
  • Fast inference with minimal resource requirements
  • Supports CPU and GPU execution
  • Open source and free to use
  • Can detect almost anything you can describe in natural language

Links:

Features

  • Real-time object detection in videos using Moondream
  • Multiple visualization styles:
    • Censor: Black boxes over detected objects
    • Bounding Box: Traditional bounding boxes with labels
    • Hitmarker: Call of Duty style crosshair markers
  • Optional grid-based detection for improved accuracy
  • Flexible object type detection using natural language
  • Frame-by-frame processing with IoU-based merging
  • Batch processing of multiple videos
  • Web-compatible output format
  • User-friendly web interface
  • Command-line interface for automation

Requirements

  • Python 3.8+
  • OpenCV (cv2)
  • PyTorch
  • Transformers
  • Pillow (PIL)
  • tqdm
  • ffmpeg
  • numpy
  • gradio (for web interface)

Installation

  1. Clone this repository and create a new virtual environment
bash
git clone https://github.com/vikhyat/moondream/blob/main/recipes/promptable-video-redaction
python -m venv .venv
source .venv/bin/activate
  1. Install the required packages:
bash
pip install -r requirements.txt
  1. Install ffmpeg:
    • On Ubuntu/Debian: sudo apt-get install ffmpeg libvips
    • On macOS: brew install ffmpeg
    • On Windows: Download from ffmpeg.org

      Downloading libvips for Windows requires some additional steps, see here

Usage

Web Interface

  1. Start the web interface:
bash
python app.py

  1. Open the provided URL in your browser

  2. Use the interface to:

    • Upload your video
    • Specify what to censor (e.g., face, logo, text)
    • Adjust processing speed and quality
    • Configure grid size for detection
    • Process and download the censored video

Command Line Interface

  1. Create an inputs directory in the same folder as the script:
bash
mkdir inputs

  1. Place your video files in the inputs directory. Supported formats:

    • .mp4
    • .avi
    • .mov
    • .mkv
    • .webm

  2. Run the script:

bash
python main.py

Optional Arguments:

  • --test: Process only first 3 seconds of each video (useful for testing detection settings)
bash
python main.py --test
  • --preset: Choose FFmpeg encoding preset (affects output quality vs. speed)
bash
python main.py --preset ultrafast  # Fastest, lower quality
python main.py --preset veryslow   # Slowest, highest quality
  • --detect: Specify what object type to detect (using natural language)
bash
python main.py --detect person     # Detect people
python main.py --detect "red car"  # Detect red cars
python main.py --detect "person wearing a hat"  # Detect people with hats
  • --box-style: Choose visualization style
bash
python main.py --box-style censor     # Black boxes (default)
python main.py --box-style bounding-box       # Bounding box-style boxes with labels
python main.py --box-style hitmarker  # COD-style hitmarkers
  • --rows and --cols: Enable grid-based detection by splitting frames
bash
python main.py --rows 2 --cols 2   # Split each frame into 2x2 grid
python main.py --rows 3 --cols 3   # Split each frame into 3x3 grid

You can combine arguments:

bash
python main.py --detect "person wearing sunglasses" --box-style bounding-box --test --preset "fast" --rows 2 --cols 2

Visualization Styles

The tool supports three different visualization styles for detected objects:

  1. Censor (default)

    • Places solid black rectangles over detected objects
    • Best for privacy and content moderation
    • Completely obscures the detected region

  2. Bounding Box

    • Traditional object detection style
    • Red bounding box around detected objects
    • Label showing object type above the box
    • Good for analysis and debugging

  3. Hitmarker

    • Call of Duty inspired visualization
    • White crosshair marker at center of detected objects
    • Small label above the marker
    • Stylistic choice for gaming-inspired visualization

Choose the style that best fits your use case using the --box-style argument.

Output

Processed videos will be saved in the outputs directory with the format: [style]_[object_type]_[original_filename].mp4

For example:

  • censor_face_video.mp4
  • bounding-box_person_video.mp4
  • hitmarker_car_video.mp4

The output videos will include:

  • Original video content
  • Selected visualization style for detected objects
  • Web-compatible H.264 encoding

Notes

  • Processing time depends on video length, grid size, and GPU availability
  • GPU is strongly recommended for faster processing
  • Requires sufficient disk space for temporary files
  • Detection quality varies based on video quality and Moondream's ability to recognize the specified object
  • Grid-based detection impacts performance significantly - use only when needed
  • Web interface shows progress updates and errors
  • Choose visualization style based on your use case
  • Moondream can detect almost anything you can describe in natural language

Overview (`examples/quickstart/bash/README.md`)

Moondream Bash Quickstart

Get started with Moondream's vision AI in Bash in minutes.

Setup

  1. Choose your deployment:

  2. Tweak the main.sh code if you want to run it on cloud

  3. Run the examples to see Moondream in action!

    bash
    bash main.sh

Overview (`examples/quickstart/modal/README.md`)

Deploy Moondream Station on Modal Labs

Get Moondream Station running in the cloud with Modal Labs—no hassle, just fast, scalable AI you can access from anywhere. This quick guide shows you how.

Setting Up

**1. Create Modal account**

Go to modal.com to sign up.

**2. Install Modal Labs Python Library**:

To access Modal Labs deployment tools, you need to install the library:

bash
pip install modal

**3. Login**

To connect your computer to your Modal Labs account, run:

bash
modal token new

This will open an authorization page in your web browser where you can authenticate your system with a login token.

**4. Launch a Moondream Station Instance**

**1. Deploying your Instance**:

To launch Moondream Station on Modal Labs, first make sure you're in the correct repository:

text
cd quickstart/modal

You have two deployment options:

A) Production Deployment

bash
modal deploy deploy_moondream.py

This creates a permanent deployment on Modal Labs that continues running even after you close your terminal.

B) Development Mode

For development and testing with live logs in terminal:

text
modal serve deploy_moondream.py

This runs the app while streaming logs directly to your terminal. The deployment stops when you terminate the command Ctrl+C.

**2. Get your URL**

Modal will show your server URL at the start of execution. Make sure to copy this URL as you will need this to access the Moondream Station instance.

It may look something like:

text
https://<username>--moondream-server-dev.modal.run

or

text
https://<username>--moondream-server.modal.run

This is your permanent service endpoint.

**4. Stopping your Instance**:

To stop your instance, run:

text
modal app stop moondream

or if you ran it with modal serve you can exit by inputting CTRL+C on your keyboard.

You may also locate and stop the app from your Modal Labs dashboard.

Inference

Using Moondream Station CLI

To use Moondream Station from the Moondream Station CLI, launch Moondream Station locally and then set the inference URL to the endpoint. Make sure to add /v1 to the end of the url that you copied.

text
moondream> admin set-inference-url https://<username>--moondream-server-dev.modal.run/v1

Then continue to use Moondream Station normally:

text
moondream> query "What's in this image?" path/to/image.jpg

Using Python Client

You can also access the Modal Labs endpoint using our Python client.

You will need to install the required libraries first:

text
pip install moondream pillow

Then, you can use the call the python client. Here's an example of what that may look like:

python
# modal_infer.py
import moondream as md
from PIL import Image

# To use with the example code, make sure to add /v1 at the end.
model = md.vl(endpoint="https://<username>--moondream-server-dev.modal.run/v1")
image = Image.open("/path/to/image")

# query
answer = model.query(image, "What's in this image?")["answer"]
print(answer)

Check out the python demo for more examples here.

Overview (`examples/quickstart/node/README.md`)

Moondream Node Quickstart

Get started with Moondream's vision AI in Node in minutes.

Setup

  1. Choose your deployment:

  2. Install dependencies:

    bash
    npm install

  3. Tweak the main.js code if you want to run it on cloud.

  4. Run the examples to see Moondream in action!

    bash
    node main.py

Overview (`examples/quickstart/python/README.md`)

Moondream Python Quickstart

Get started with Moondream's vision AI in Python in minutes.

Setup

  1. Choose your deployment:

  2. Install dependencies:

    bash
    pip install -r requirements.txt

  3. Run Moondream Station locally or tweak the main.py or run on the cloud using an API key.

  4. Run the example to see Moondream in action!

    bash
    python main.py

  5. Try a full tutorial:

    Open the detect_cars.ipynb notebook in Jupyter and run all the cells to try caption, query, detect and point capabilities:

    bash
    jupyter notebook detect_cars.ipynb

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →