# Technical Documentation: m87-labs/moondream > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/m87-labs/moondream) ### Overview (`README.md`) # 🌔 moondream a tiny vision language model that kicks ass and runs anywhere [Website](https://moondream.ai/) | [Demo](https://moondream.ai/playground) ## Examples | Image | Example | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | **What is the girl doing?**
The girl is sitting at a table and eating a large hamburger.

**What color is the girl's hair?**
The girl's hair is white. | | | **What is this?**
This is a computer server rack, which is a device used to store and manage multiple computer servers. The rack is filled with various computer servers, each with their own dedicated space and power supply. The servers are connected to the rack via multiple cables, indicating that they are part of a larger system. The rack is placed on a carpeted floor, and there is a couch nearby, suggesting that the setup is in a living or entertainment area.

**What is behind the stand?**
Behind the stand, there is a brick wall. | ## About Moondream is a highly efficient open-source vision language model that combines powerful image understanding capabilities with a remarkably small footprint. It's designed to be versatile and accessible, capable of running on a wide range of devices and platforms. The project offers two model variants: - **Moondream 2B**: The primary model with 2 billion parameters, offering robust performance for general-purpose image understanding tasks including captioning, visual question answering, and object detection. - **Moondream 0.5B**: A compact 500 million parameter model specifically optimized as a distillation target for edge devices, enabling efficient deployment on resource-constrained hardware while maintaining impressive capabilities. ## How to use Moondream can be run locally, or in the cloud. Please refer to the [Getting Started](https://moondream.ai/c/docs/quickstart) page for details. ## Special thanks * [Modal](https://modal.com/?utm_source=github&utm_medium=github&utm_campaign=moondream) - Modal lets you run jobs in the cloud, by just writing a few lines of Python. Here's an [example of how to run Moondream on Modal](https://github.com/m87-labs/moondream-examples/tree/main/quickstart/modal). --- ### Overview (`recipes/gaze-detection-video/README.md`) # Gaze Detection Video Processor > **⚠️ IMPORTANT:** This project currently uses Moondream 2B (2025-01-09 release) via the Hugging Face Transformers library. We will migrate to the official Moondream client > libraries once they become available for this version. ## Table of Contents - [Overview](#overview) - [Sample Output](#sample-output) - [Features](#features) - [Prerequisites](#prerequisites) - [Installation](#installation) - [Linux/macOS Installation](#linuxmacos-installation) - [Windows Installation](#windows-installation) - [Usage](#usage) - [Output](#output) - [Troubleshooting](#troubleshooting) - [Performance Notes](#performance-notes) - [Dependencies](#dependencies) - [Model Details](#model-details) - [License](#license) ## Overview This project uses the Moondream 2B model to detect faces and their gaze directions in videos. It processes videos frame by frame, visualizing face detections and gaze directions. ## Sample Output | Input Video | Processed Output | | :-----------------------------------: | :-----------------------------------------: | | | | ## Features - Face detection in video frames - Gaze direction tracking - Real-time visualization with: - Colored bounding boxes for faces - Gradient lines showing gaze direction - Gaze target points - Supports multiple faces per frame - Processes all common video formats (.mp4, .avi, .mov, .mkv) - Uses Moondream 2 (2025-01-09 release) via Hugging Face Transformers - Note: Will be migrated to official client libraries in future updates - No authentication required ## Prerequisites 1. Python 3.8 or later 2. CUDA-capable GPU recommended (but CPU mode works too) 3. FFmpeg installed on your system ## Installation ### Linux/macOS Installation 1. Install system dependencies: ```bash # Ubuntu/Debian sudo apt-get update && sudo apt-get install -y libvips42 libvips-dev ffmpeg # CentOS/RHEL sudo yum install vips vips-devel ffmpeg # macOS brew install vips ffmpeg ``` 2. Clone and setup the project: ```bash git clone https://github.com/vikhyat/moondream.git cd moondream/recipes/gaze-detection-video python3 -m venv venv source venv/bin/activate pip install -r requirements.txt ``` ### Windows Installation Windows setup requires a few additional steps for proper GPU support and libvips installation. 1. Clone the repository: ```bash git clone [repository-url] cd moondream/recipes/gaze-detection-video ``` 2. Create and activate virtual environment: ```bash python -m venv venv .\venv\Scripts\activate ``` 3. Install PyTorch with CUDA support: ```bash # For NVIDIA GPUs pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 ``` 4. Install libvips: Download the appropriate version based on your system architecture: | Architecture | VIPS Version to Download | | ------------ | ------------------------ | | 32-bit x86 | vips-dev-w32-all-8.16.0.zip | | 64-bit x64 | vips-dev-w64-all-8.16.0.zip | - Extract the ZIP file - Copy all DLL files from `vips-dev-8.16\bin` to either: - Your project's root directory (easier) OR - `C:\Windows\System32` (requires admin privileges) - Add to PATH: 1. Open System Properties → Advanced → Environment Variables 2. Under System Variables, find PATH 3. Add the full path to the `vips-dev-8.16\bin` directory 5. Install FFmpeg: - Download from https://ffmpeg.org/download.html#build-windows - Extract and add the `bin` folder to your system PATH (similar to step 4) or to the project root directory 6. Install other dependencies: ```bash pip install -r requirements.txt ``` ## Usage 1. Place your input videos in the `input` directory - Supported formats: .mp4, .avi, .mov, .mkv - The directory will be created automatically if it doesn't exist 2. Run the script: ```bash python gaze-detection-video.py ``` 3. The script will: - Process all videos in the input directory - Show progress bars for each video - Save processed videos to the `output` directory with prefix 'processed\_' ## Output - Processed videos are saved as `output/processed_[original_name].[ext]` - Each frame in the output video shows: - Colored boxes around detected faces - Lines indicating gaze direction - Points showing where each person is looking ## Troubleshooting 1. CUDA/GPU Issues: - Ensure you have CUDA installed for GPU support - The script will automatically fall back to CPU if no GPU is available 2. Memory Issues: - If processing large videos, ensure you have enough RAM - Consider reducing video resolution if needed 3. libvips Errors: - Make sure libvips is properly installed for your OS - Check system PATH includes libvips 4. Video Format Issues: - Ensure FFmpeg is installed and in your system PATH - Try converting problematic videos to MP4 format ## Performance Notes - GPU processing is significantly faster than CPU - Processing time depends on: - Video resolution - Number of faces per frame - Frame rate - Available computing power ## Dependencies - transformers (for Moondream 2 model access) - torch - opencv-python - pillow - matplotlib - numpy - tqdm - pyvips - accelerate - einops ## Model Details > **⚠️ IMPORTANT:** This project currently uses Moondream 2 (2025-01-09 release) via the Hugging Face Transformers library. We will migrate to the official Moondream client > libraries once they become available for this version. The model is loaded using: --- ### Overview (`recipes/promptable-content-moderation/README.md`) # Promptable Content Moderation with Moondream Welcome to the future of content moderation with Moondream 2B, a powerful and lightweight vision-language model that enables detection and moderation of video content using natural language prompts. [Try it now.](https://huggingface.co/spaces/moondream/content-moderation) ## Features - Content moderation through natural language prompts - Multiple visualization styles - Intelligent scene detection and tracking: - DeepSORT tracking with scene-aware reset - Persistent moderation across frames - Smart tracker reset at scene boundaries - Optional grid-based detection for improved accuracy on complex scenes - Frame-by-frame processing with IoU-based merging - Web-compatible output format - Test mode (process only first X seconds) - Advanced moderation analysis with multiple visualization plots ## Examples | Prompt | Output | |--------|-----------------| | "white cigarette" | | | "gun" | | | "confederate flag" | | ## Requirements ### Python Dependencies For Windows users, before installing other requirements, first install PyTorch with CUDA support: ```bash pip install torch==2.5.1+cu121 torchvision==0.20.1+cu121 --index-url https://download.pytorch.org/whl/cu121 ``` Then install the remaining dependencies: ```bash pip install -r requirements.txt ``` ### System Requirements - FFmpeg (required for video processing) - libvips (required for image processing) Installation by platform: - Ubuntu/Debian: `sudo apt-get install ffmpeg libvips` - macOS: `brew install ffmpeg libvips` - Windows: - Download FFmpeg from [ffmpeg.org](https://ffmpeg.org/download.html) - Follow [libvips Windows installation guide](https://docs.moondream.ai/quick-start) ## Installation 1. Clone this repository and create a new virtual environment: ```bash git clone https://github.com/vikhyat/moondream/blob/main/recipes/promptable-video-redaction python -m venv .venv source .venv/bin/activate # On Windows: .venv\Scripts\activate ``` 2. Install Python dependencies: ```bash pip install -r requirements.txt ``` 3. Install ffmpeg and libvips: - On Ubuntu/Debian: `sudo apt-get install ffmpeg libvips` - On macOS: `brew install ffmpeg` - On Windows: Download from [ffmpeg.org](https://ffmpeg.org/download.html) > Downloading libvips for Windows requires some additional steps, see [here](https://docs.moondream.ai/quick-start) ## Usage The easiest way to use this tool is through its web interface, which provides a user-friendly experience for video content moderation. ### Web Interface 1. Start the web interface: ```bash python app.py ``` 2. Open the provided URL in your browser (typically ) 3. Use the interface to: - Upload your video file - Specify content to moderate (e.g., "face", "cigarette", "gun") - Choose redaction style (default: obfuscated-pixel) - OPTIONAL: Configure advanced settings - Processing speed/quality - Grid size for detection - Test mode for quick validation (default: on, 3 seconds) - Process the video and download results - Analyze detection patterns with visualization tools ## Output Files The tool generates two types of output files in the `outputs` directory: 1. Processed Videos: - Format: `[style]_[content_type]_[original_filename].mp4` - Example: `censor_inappropriate_video.mp4` 2. Detection Data: - Format: `[style]_[content_type]_[original_filename]_detections.json` - Contains frame-by-frame detection information - Used for visualization and analysis ## Technical Details ### Scene Detection and Tracking The tool uses advanced scene detection and object tracking: 1. Scene Detection: - Powered by PySceneDetect's ContentDetector - Automatically identifies scene changes in videos - Configurable detection threshold (default: 30.0) - Helps maintain tracking accuracy across scene boundaries 2. Object Tracking: - DeepSORT tracking for consistent object identification - Automatic tracker reset at scene changes - Maintains object identity within scenes - Prevents tracking errors across scene boundaries 3. Integration Benefits: - More accurate object tracking - Better handling of scene transitions - Reduced false positives in tracking - Improved tracking consistency ## Best Practices - Use test mode for initial configuration - Enable grid-based detection for complex scenes - Choose appropriate redaction style based on content type: - Censor: Complete content blocking - Blur styles: Less intrusive moderation - Bounding Box: Content review and analysis - Monitor system resources during processing - Use appropriate processing quality settings based on your needs ## Notes - Processing time depends on video length, resolution, GPU availability, and chosen settings - GPU is strongly recommended for faster processing - Grid-based detection increases accuracy but requires more processing time (each grid cell is processed independently) - Test mode processes only first X seconds (default: 3 seconds) for quick validation --- ### Overview (`recipes/promptable-video-redaction/README.md`) # Promptable Video Redaction with Moondream This tool uses Moondream 2B, a powerful yet lightweight vision-language model, to detect and redact objects from videos. Moondream can recognize a wide variety of objects, people, text, and more with high accuracy while being much smaller than traditional models. [Try it now.](https://huggingface.co/spaces/moondream/promptable-video-redaction) ## About Moondream Moondream is a tiny yet powerful vision-language model that can analyze images and answer questions about them. It's designed to be lightweight and efficient while maintaining high accuracy. Some key features: - Only 2B parameters - Fast inference with minimal resource requirements - Supports CPU and GPU execution - Open source and free to use - Can detect almost anything you can describe in natural language Links: - [GitHub Repository](https://github.com/vikhyat/moondream) - [Hugging Face](https://huggingface.co/vikhyatk/moondream2) - [Build with Moondream](http://docs.moondream.ai/) ## Features - Real-time object detection in videos using Moondream - Multiple visualization styles: - Censor: Black boxes over detected objects - Bounding Box: Traditional bounding boxes with labels - Hitmarker: Call of Duty style crosshair markers - Optional grid-based detection for improved accuracy - Flexible object type detection using natural language - Frame-by-frame processing with IoU-based merging - Batch processing of multiple videos - Web-compatible output format - User-friendly web interface - Command-line interface for automation ## Requirements - Python 3.8+ - OpenCV (cv2) - PyTorch - Transformers - Pillow (PIL) - tqdm - ffmpeg - numpy - gradio (for web interface) ## Installation 1. Clone this repository and create a new virtual environment ```bash git clone https://github.com/vikhyat/moondream/blob/main/recipes/promptable-video-redaction python -m venv .venv source .venv/bin/activate ``` 2. Install the required packages: ```bash pip install -r requirements.txt ``` 3. Install ffmpeg: - On Ubuntu/Debian: `sudo apt-get install ffmpeg libvips` - On macOS: `brew install ffmpeg` - On Windows: Download from [ffmpeg.org](https://ffmpeg.org/download.html) > Downloading libvips for Windows requires some additional steps, see [here](https://docs.moondream.ai/quick-start) ## Usage ### Web Interface 1. Start the web interface: ```bash python app.py ``` 2. Open the provided URL in your browser 3. Use the interface to: - Upload your video - Specify what to censor (e.g., face, logo, text) - Adjust processing speed and quality - Configure grid size for detection - Process and download the censored video ### Command Line Interface 1. Create an `inputs` directory in the same folder as the script: ```bash mkdir inputs ``` 2. Place your video files in the `inputs` directory. Supported formats: - .mp4 - .avi - .mov - .mkv - .webm 3. Run the script: ```bash python main.py ``` ### Optional Arguments: - `--test`: Process only first 3 seconds of each video (useful for testing detection settings) ```bash python main.py --test ``` - `--preset`: Choose FFmpeg encoding preset (affects output quality vs. speed) ```bash python main.py --preset ultrafast # Fastest, lower quality python main.py --preset veryslow # Slowest, highest quality ``` - `--detect`: Specify what object type to detect (using natural language) ```bash python main.py --detect person # Detect people python main.py --detect "red car" # Detect red cars python main.py --detect "person wearing a hat" # Detect people with hats ``` - `--box-style`: Choose visualization style ```bash python main.py --box-style censor # Black boxes (default) python main.py --box-style bounding-box # Bounding box-style boxes with labels python main.py --box-style hitmarker # COD-style hitmarkers ``` - `--rows` and `--cols`: Enable grid-based detection by splitting frames ```bash python main.py --rows 2 --cols 2 # Split each frame into 2x2 grid python main.py --rows 3 --cols 3 # Split each frame into 3x3 grid ``` You can combine arguments: ```bash python main.py --detect "person wearing sunglasses" --box-style bounding-box --test --preset "fast" --rows 2 --cols 2 ``` ### Visualization Styles The tool supports three different visualization styles for detected objects: 1. **Censor** (default) - Places solid black rectangles over detected objects - Best for privacy and content moderation - Completely obscures the detected region 2. **Bounding Box** - Traditional object detection style - Red bounding box around detected objects - Label showing object type above the box - Good for analysis and debugging 3. **Hitmarker** - Call of Duty inspired visualization - White crosshair marker at center of detected objects - Small label above the marker - Stylistic choice for gaming-inspired visualization Choose the style that best fits your use case using the `--box-style` argument. ## Output Processed videos will be saved in the `outputs` directory with the format: `[style]_[object_type]_[original_filename].mp4` For example: - `censor_face_video.mp4` - `bounding-box_person_video.mp4` - `hitmarker_car_video.mp4` The output videos will include: - Original video content - Selected visualization style for detected objects - Web-compatible H.264 encoding ## Notes - Processing time depends on video length, grid size, and GPU availability - GPU is strongly recommended for faster processing - Requires sufficient disk space for temporary files - Detection quality varies based on video quality and Moondream's ability to recognize the specified object - Grid-based detection impacts performance significantly - use only when needed - Web interface shows progress updates and errors - Choose visualization style based on your use case - Moondream can detect almost anything you can describe in natural language --- ### Overview (`examples/quickstart/bash/README.md`) # Moondream Bash Quickstart Get started with Moondream's vision AI in Bash in minutes. ## Setup 1. Choose your deployment: - **Local**: Install [Moondream Station](https://moondream.ai/station) - **Cloud**: Get an API key from [Moondream Cloud](https://moondream.ai/cloud) 2. Tweak the `main.sh` code if you want to run it on cloud 3. Run the examples to see Moondream in action! ```bash bash main.sh ``` --- ### Overview (`examples/quickstart/modal/README.md`) # Deploy Moondream Station on Modal Labs Get Moondream Station running in the cloud with Modal Labs—no hassle, just fast, scalable AI you can access from anywhere. This quick guide shows you how. ## Setting Up ### **1. Create Modal account** Go to [modal.com](https://modal.com) to sign up. ### **2. Install Modal Labs Python Library**: To access Modal Labs deployment tools, you need to install the library: ```bash pip install modal ``` ### **3. Login** To connect your computer to your Modal Labs account, run: ```bash modal token new ``` This will open an authorization page in your web browser where you can authenticate your system with a login token. ### **4. Launch a Moondream Station Instance** #### **1. Deploying your Instance**: To launch Moondream Station on Modal Labs, first make sure you're in the correct repository: ``` cd quickstart/modal ``` You have two deployment options: **A) Production Deployment** ```bash modal deploy deploy_moondream.py ``` This creates a permanent deployment on Modal Labs that continues running even after you close your terminal. **B) Development Mode** For development and testing with live logs in terminal: ``` modal serve deploy_moondream.py ``` This runs the app while streaming logs directly to your terminal. The deployment stops when you terminate the command `Ctrl+C`. #### **2. Get your URL** Modal will show your server URL at the start of execution. Make sure to copy this URL as you will need this to access the Moondream Station instance. It may look something like: ``` https://--moondream-server-dev.modal.run ``` or ``` https://--moondream-server.modal.run ``` This is your permanent service endpoint. #### **4. Stopping your Instance**: To stop your instance, run: ``` modal app stop moondream ``` or if you ran it with `modal serve` you can exit by inputting `CTRL+C` on your keyboard. You may also locate and stop the app from your Modal Labs dashboard. ## Inference ### Using Moondream Station CLI To use Moondream Station from the Moondream Station CLI, launch Moondream Station locally and then set the inference URL to the endpoint. **Make sure to add `/v1` to the end of the url that you copied.** ``` moondream> admin set-inference-url https://--moondream-server-dev.modal.run/v1 ``` Then continue to use Moondream Station normally: ``` moondream> query "What's in this image?" path/to/image.jpg ``` ### Using Python Client You can also access the Modal Labs endpoint using our Python client. You will need to install the required libraries first: ``` pip install moondream pillow ``` Then, you can use the call the python client. Here's an example of what that may look like: ```python # modal_infer.py import moondream as md from PIL import Image # To use with the example code, make sure to add /v1 at the end. model = md.vl(endpoint="https://--moondream-server-dev.modal.run/v1") image = Image.open("/path/to/image") # query answer = model.query(image, "What's in this image?")["answer"] print(answer) ``` Check out the python demo for more examples [here](modal_infer.py). --- ### Overview (`examples/quickstart/node/README.md`) # Moondream Node Quickstart Get started with Moondream's vision AI in Node in minutes. ## Setup 1. Choose your deployment: - **Local**: Install [Moondream Station](https://moondream.ai/station) - **Cloud**: Get an API key from [Moondream Cloud](https://moondream.ai/cloud) 2. Install dependencies: ```bash npm install ``` 3. Tweak the `main.js` code if you want to run it on cloud. 4. Run the examples to see Moondream in action! ```bash node main.py ``` --- ### Overview (`examples/quickstart/python/README.md`) # Moondream Python Quickstart Get started with Moondream's vision AI in Python in minutes. ## Setup 1. Choose your deployment: - **Local**: Install [Moondream Station](https://moondream.ai/station) - **Cloud**: Get an API key from [Moondream Cloud](https://moondream.ai/cloud) 2. Install dependencies: ```bash pip install -r requirements.txt ``` 3. Run Moondream Station locally or tweak the [main.py](quickstart/python/main.py) or run on the cloud using an [API key](https://moondream.ai/c/cloud/api-keys). 4. Run the example to see Moondream in action! ```bash python main.py ``` 5. Try a full tutorial: Open the `detect_cars.ipynb` notebook in Jupyter and run all the cells to try caption, query, detect and point capabilities: ```bash jupyter notebook detect_cars.ipynb ``` --- --- METRICS --- - Files Extracted: 8 - Estimated Token Budget: ~6447 tokens - Canonical Reference: https://codewiki.google/github.com/m87-labs/moondream