# Technical Documentation: janhq/jan

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/janhq/jan)

### Overview (README.md)

# Jan - Open-source ChatGPT replacement

<img width="2048" height="280" alt="github jan banner" src="https://github.com/user-attachments/assets/f3f87889-c133-433b-b250-236218150d3f" />

<p align="center">
  <strong>English</strong> ·
  <a href="README.zh.md">中文</a> ·
  <a href="README.ja.md">日本語</a>
</p>

<p align="center">
  
  <img alt="GitHub commit activity" src="https://img.shields.io/github/commit-activity/m/janhq/jan"/>
  <img alt="Github Last Commit" src="https://img.shields.io/github/last-commit/janhq/jan"/>
  <img alt="Github Contributors" src="https://img.shields.io/github/contributors/janhq/jan"/>
  <img alt="GitHub closed issues" src="https://img.shields.io/github/issues-closed/janhq/jan"/>
  <img alt="Discord" src="https://img.shields.io/discord/1107178041848909847?label=discord"/>
</p>

<p align="center">
  <a href="https://jan.ai/docs/desktop">Getting Started</a>
  - <a href="https://discord.gg/Exe46xPMbK">Community</a>
  - <a href="https://jan.ai/changelog">Changelog</a>
  - <a href="https://github.com/janhq/jan/issues">Bug reports</a>
</p>

Jan is bringing the best of open-source AI in an easy-to-use product. Download and run LLMs with **full control** and **privacy**.

## Installation

<p align="center">
  <table>
    <tr>
      
      <td align="center" valign="middle">
        <a href="https://apps.microsoft.com/detail/xpdcnfn5cpzlqb">
          <img height="60"
            width="200"
               alt="Get it from Microsoft Store"
               src="https://get.microsoft.com/images/en-us%20dark.svg"/>
        </a>
      </td>
      
      <td width="20"></td>
      
      <td align="center" valign="middle">
        <a href="https://flathub.org/apps/ai.jan.Jan">
          <img height="60"
            width="200"
               alt="Get it on Flathub"
               src="https://flathub.org/assets/badges/flathub-badge-en.svg"/>
        </a>
      </td>
    </tr>
  </table>
</p>

The easiest way to get started is by downloading one of the following versions for your respective operating system:

<table>
  <tr>
    <td><b>Platform</b></td>
    <td><b>Download</b></td>
  </tr>
  <tr>
    <td><b>Windows</b></td>
    <td><a href='https://app.jan.ai/download/latest/win-x64'>jan.exe</a></td>
  </tr>
  <tr>
    <td><b>macOS</b></td>
    <td><a href='https://app.jan.ai/download/latest/mac-universal'>jan.dmg</a></td>
  </tr>
  <tr>
    <td><b>Linux (deb)</b></td>
    <td><a href='https://app.jan.ai/download/latest/linux-amd64-deb'>jan.deb</a></td>
  </tr>
  <tr>
    <td><b>Linux (AppImage)</b></td>
    <td><a href='https://app.jan.ai/download/latest/linux-amd64-appimage'>jan.AppImage</a></td>
  </tr>
  <tr>
    <td><b>Linux (Arm64)</b></td>
    <td><a href='https://github.com/janhq/jan/issues/4543#issuecomment-4142429792'>How-to</a></td>
  </tr>
</table>


Download from [jan.ai](https://jan.ai/) or [GitHub Releases](https://github.com/janhq/jan/releases).

## Features

- **Local AI Models**: Download and run LLMs (Llama, Gemma, Qwen, GPT-oss etc.) from HuggingFace
- **Cloud Integration**: Connect to GPT models via OpenAI, Claude models via Anthropic, Mistral, Groq, MiniMax, and others
- **Custom Assistants**: Create specialized AI assistants for your tasks
- **OpenAI-Compatible API**: Local server at `localhost:1337` for other applications
- **Model Context Protocol**: MCP integration for agentic capabilities
- **Privacy First**: Everything runs locally when you want it to

## Build from Source

For those who enjoy the scenic route:

### Prerequisites

- Node.js ≥ 20.0.0
- Yarn ≥ 4.5.3
- Make ≥ 3.81
- Rust (for Tauri)
- (macOS Apple Silicon only) MetalToolchain `xcodebuild -downloadComponent MetalToolchain`

### Run with Make

```bash
git clone https://github.com/janhq/jan
cd jan
make dev
```

This handles everything: installs dependencies, builds core components, and launches the app.

**Available make targets:**
- `make dev` - Full development setup and launch
- `make build` - Production build
- `make test` - Run tests and linting
- `make clean` - Delete everything and start fresh

### Manual Commands

```bash
yarn install
yarn build
yarn dev
```

### Building on Windows

Run `make dev` from **Git Bash** (installed with Git for Windows) — make dispatches its recipes through `sh`, so a plain `cmd.exe` won't work.

You do **not** need a "Native Tools Command Prompt for VS 2022". The bundled llama.cpp engine builds with Ninja + `clang-cl`, and `clang-cl` locates the MSVC toolchain and Windows SDK on its own. What has to be installed (and on `PATH` for `ninja`/`clang-cl`/`cmake`):

- Visual Studio 2022 Build Tools (MSVC x64 workload + Windows SDK)
- LLVM (provides `clang-cl`)
- Ninja
- CMake
- CUDA Toolkit — only for `JAN_ENGINE_VARIANT=cuda12`/`cuda13` builds

Engine variants are picked with `JAN_ENGINE_VARIANT` (tokens: `cpu`, `vulkan`, `metal`, `cuda12`, `cuda13`, `hip`/`rocm`, joined by `-`), e.g.:

```bash
make dev JAN_ENGINE_VARIANT=cuda13
```

**"nvcc fatal : Could not open output file ...fattn-...cu.obj.d"** during `tauri-plugin-llamacpp(build)` means the build path crossed Windows' 260-character `MAX_PATH` limit — nvcc does not honor the long-path opt-in. The build script now detects this and automatically relocates the llama.cpp build tree to a short directory under `%LOCALAPPDATA%\jan-engine`. If you hit path-length errors anyway, set `JAN_ENGINE_BUILD_DIR` to a short path (e.g. `C:\jb`) or move the checkout closer to the drive root.

## System Requirements

**Minimum specs for a decent experience:**

- **macOS**: 13.6+ (8GB RAM for 3B models, 16GB for 7B, 32GB for 13B)
- **Windows**: 10+ with GPU support for NVIDIA/AMD/Intel Arc
- **Linux**: Most distributions work, GPU acceleration available

For detailed compatibility, check our [installation guides](https://jan.ai/docs/desktop/mac).

## Troubleshooting

If things go sideways:

1. Check our [troubleshooting docs](https://jan.ai/docs/desktop/troubleshooting)
2. Copy your error logs and system specs
3. Ask for help in our [Discord](https://discord.gg/FTk2MvZwJH) `#🆘|jan-help` channel


## Contributing

Contributions welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full spiel.

> **Note:** Please [sign your commits](CONTRIBUTING.md#signed-commits) so we can verify your contributions.

## Links

- [Documentation](https://jan.ai/docs) - The manual you should read
- [API Reference](https://jan.ai/api-reference) - For the technically inclined
- [Changelog](https://jan.ai/changelog) - What we broke and fixed
- [Discord](https://discord.gg/FTk2MvZwJH) - Where the community lives

## Contact

- **Bugs**: [GitHub Issues](https://github.com/janhq/jan/issues)
- **Business**: hello@jan.ai
- **Jobs**: hr@jan.ai
- **General Discussion**: [Discord](https://discord.gg/FTk2MvZwJH)

## License

Apache 2.0 - Because sharing is caring.

## Acknowledgements

Built on the shoulders of giants:

- [Llama.cpp](https://github.com/ggerganov/llama.cpp)
- [Tauri](https://tauri.app/)
- [Scalar](https://github.com/scalar/scalar)

---

### README (`docs/README.md`)

# Website & Docs

This website is built using [Nextra](https://nextra.site/), a modern static website generator.

### Information Architecture

We try to **keep routes consistent** to maintain SEO.

- **`/docs/`**: Product documentation (Jan Desktop, Jan Agent).

- **`/handbook/`**: Handbook pages.

- **`/research/`**: Research content.

- **`/changelog/`**: A list of changes made to the Jan application with each release.

- **`/blog/`**: The blog index; individual posts live under `/post/`.

- **`/download/`**: Download page.

- **`/support/`**: Support page.

- **`/privacy/`**: Privacy policy.

- **`/tokamak/`**: Tokamak landing page.

## How to Contribute

Refer to the [Contributing Guide](https://github.com/janhq/jan/blob/main/CONTRIBUTING.md) for more comprehensive information on how to contribute to the Jan project.

### Pre-requisites and Installation

- [Node.js](https://nodejs.org/en/) (version 20.0.0 or higher)
- [yarn](https://yarnpkg.com/) (version 4.x; the repo pins 4.5.3 via corepack)

#### Installation

```bash
cd jan/docs
yarn install
yarn dev
```

This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.

#### Build

```bash
yarn build
```

This command generates static content into the `out` directory (`next.config.mjs` sets `output: 'export'`) and can be served using any static contents hosting service.

### Deployment

There is no `yarn deploy` script; the `jan-docs.yml` workflow builds the site and publishes `out` to Cloudflare Pages.

### Preview URL, Pre-release and Publishing Documentation

- When a pull request is created, the preview URL will be automatically commented on the pull request.

- The documentation will then be published to [https://jan.ai/](https://jan.ai/) when the pull request is merged to `main`.

---

### Src Pages Index (`docs/src/pages/index.mdx`)

---
title: "Jan - Open-Source ChatGPT Replacement"
description: "Jan is an open-source alternative to ChatGPT. Run open-source AI models locally or connect to cloud models like GPT, Claude and others."
keywords:
  [
    Jan,
    Customizable Intelligence, LLM,
    local AI,
    privacy focus,
    free and open source,
    private and offline,
    conversational AI,
    no-subscription fee,
    large language models,
    architecture,
  ]
---

import Home from "@/components/Home"

export const getStaticProps = async() => {
  const resReleaseLatest = await fetch('https://api.github.com/repos/janhq/jan/releases/latest')  
  const resRelease = await fetch('https://api.github.com/repos/janhq/jan/releases?per_page=500')  
  const resRepo = await fetch('https://api.github.com/repos/janhq/jan')
  const repo = await resRepo.json()
  const latestRelease = await resReleaseLatest.json()
  const release = await resRelease.json()

  return {
    props: {
      ssg: {
        stars: repo.stargazers_count || 44000, // placeholder before the client-side useGitHubStars fetch resolves
        lastRelease: latestRelease,
        release: release,
        lastVersion: latestRelease.tag_name || 'v.0.0.1'
      },
    },
  }
}

<Home />

---

### Src Pages Docs Index (`docs/src/pages/docs/index.mdx`)

---
title: Jan Docs
description: References for Jan Desktop and Jan Agent.
---

import DocsHome from "@/components/DocsHome"

<DocsHome/>

---

### Src Pages Handbook Index (`docs/src/pages/handbook/index.mdx`)

---
title: "Jan Team Handbook"
description: "Jan is a general purpose AI agent that works on its own to get things done."
---

# Jan Handbook
> Jan's Handbook is inspired by [Posthog](https://posthog.com/handbook) and [Gitlab](https://handbook.gitlab.com/). Thanks for showing us the way.

## Welcome

Welcome to [Jan](https://jan.ai)!

This handbook explains what Jan is, what we stand for, and how you can be part of it.

Jan is built in public by a team that believes AI should get things done, not just talk about getting things done. We're open source, we move fast, and we build alongside the people using what we make.

This handbook is a living thing. It grows as Jan grows - new ideas, new tools, new ways to make AI that actually works for you.

## Overview

Jan is a general purpose AI agent that works on its own to get things done.

Give Jan something to work on, maybe research for a brief, a report you have been putting off, or an analysis you need to pull together, and it will run for as long as needed. It could be five hours or ten. Jan figures it out, does the work, and finishes the job.

Think of it like this: you hand off the task, Jan handles it, and you get on with your day.

Jan is open source and built by a vibrant global community. Anyone can use it, contribute to it, or host it themselves.

Jan gets it done. That is the whole idea.

---

### Src Pages Research Index (`docs/src/pages/research/index.mdx`)

---
title: Research
description: Model reports from the Jan family — Jan-v3.5-4B, Jan-Code-4B, Jan-v3-4B, Jan-v2-VL, Jan-v1, Jan Nano, and Lucy.
---

import Research from "@/components/Research"

<Research/>

---

### Src Pages Docs Agent Index (`docs/src/pages/docs/agent/index.mdx`)

---
title: Overview
description: Jan Agent is a terminal AI agent that reads, writes, and runs code in your project - powered by the models you already configured.
keywords:
  [
    Jan,
    Jan AI,
    Jan Agent,
    agents,
    autonomous agent,
    terminal agent,
    coding agent,
    CLI agent,
    local AI agent,
    MCP,
  ]
---

import { Callout, Tabs } from 'nextra/components'
import { Link } from 'nextra-theme-docs'
import { DocCard, DocCards } from '@/components/DocCard'
import { Rocket, Terminal, Wrench, Settings, Users, SquareTerminal } from 'lucide-react'

# Overview

Jan Agent is an AI agent that lives in your terminal. You give it a task in plain language, and it
reads files, edits them, runs commands, and searches the web until the work is done - asking for
your approval before anything touches your machine.

It uses the providers you have already configured in Jan Desktop, so if you're a Jan user there is
nothing new to sign up for.

```bash
jan
```

That single command opens the interactive console in whatever directory you're in.

## Jan Agent and Jan Desktop

They're two front ends onto the same stack, built for different jobs. Jan Desktop is where you run
models on your own hardware and work in a GUI. Jan Agent is where you point a model at a folder and
let it work.

| | Jan Desktop | Jan Agent |
| --- | --- | --- |
| Interface | Desktop app | Terminal, interactive or headless |
| Where models run | Locally via llama.cpp or MLX, or a cloud provider | Cloud providers, or any OpenAI-compatible endpoint |
| Works on | Conversations, projects, and files you attach | A project folder on disk |
| Files and shell | Through MCP servers you add | Built in - read, write, edit, and run commands, each gated by approval |
| MCP servers | Yes | Yes, the same ones |
| Skills and memory | Yes | Yes, the same ones |

Neither replaces the other. A common setup is Jan Desktop serving a local model, with Jan Agent
pointed at it - see [running against local models](/docs/agent/providers#your-own-hardware).

Provider settings, MCP servers, [skills, and memory](/docs/agent/skills)
are all shared, so anything you set up in one is already there in the other.

## Install

<Callout type="warning">
Jan Agent is a preview. The installer on `dev` pulls from the `agent-nightly` channel - expect
nightly-quality builds.
</Callout>

<Tabs items={['macOS / Linux', 'Windows']}>
  <Tabs.Tab>
```bash
curl -fsSL https://delta.jan.ai/jan-cli/install-jan-agent.sh | bash
```
  </Tabs.Tab>
  <Tabs.Tab>
```powershell
irm https://delta.jan.ai/jan-cli/install-jan-agent.ps1 | iex
```

Installs to `%USERPROFILE%\.local\bin`. To also add it to your user `PATH`:

```powershell
& ([scriptblock]::Create((irm https://delta.jan.ai/jan-cli/install-jan-agent.ps1))) -AddToPath
```
  </Tabs.Tab>
</Tabs>

Then check it landed:

```bash
jan --version
```

The [Quickstart](/docs/agent/quickstart) covers connecting a provider and finishing a first task.
Looking for the desktop app instead? <Link href="https://jan.ai/download" target="_blank" rel="noopener noreferrer">Download Jan Desktop</Link>.

## What it can do

- **Read and change a project** - open files, make targeted edits, create new ones.
- **Run commands** - build, test, grep, and read the output to decide what to do next.
- **Search the web** - look things up mid-task when the answer isn't in your project.
- **Use your MCP servers** - every connector configured in Jan Desktop is available here too.
- **Follow your project's skills** - reusable procedures it loads when a task calls for them.
- **Remember across sessions** - durable facts about the project, kept as files you can review.
- **Delegate** - split large jobs across [subagents](/docs/agent/subagents) that work in parallel.
- **Keep working toward a goal** - `/goal` keeps it going until a condition you set is true.

<Callout type="info">
Jan Agent ships no inference engine of its own, so a model always runs somewhere else. That can be a
cloud provider, or your own machine - point it at any OpenAI-compatible endpoint, such as Jan
Desktop's [local API server](/docs/desktop/api-server), with `jan config set --base-url`.
</Callout>

## Get started

<DocCards>
  <DocCard title="Quickstart" href="/docs/agent/quickstart" icon={<Rocket size={20} />}>Install Jan Agent, connect a provider, and finish your first task.</DocCard>
  <DocCard title="Interactive console" href="/docs/agent/console" icon={<SquareTerminal size={20} />}>Slash commands, keyboard shortcuts, and how a session works.</DocCard>
  <DocCard title="Tools & permissions" href="/docs/agent/permissions" icon={<Wrench size={20} />}>What the agent can do, and how approval keeps you in control.</DocCard>
  <DocCard title="Configuration" href="/docs/agent/project-config" icon={<Settings size={20} />}>Providers, `agent.toml`, and per-project instructions.</DocCard>
  <DocCard title="Subagents" href="/docs/agent/subagents" icon={<Users size={20} />}>Delegate parts of a task to agents that run in parallel.</DocCard>
  <DocCard title="CLI reference" href="/docs/agent/cli" icon={<Terminal size={20} />}>Every flag and subcommand, including headless runs for scripts and CI.</DocCard>
</DocCards>

## Staying in control

The agent edits files and runs commands without stopping to ask. File paths are checked against the
project root, and anything in `[tools] deny` is refused outright - but
shell commands are not sandboxed by default, so one runs with your own access.

Three ways to change that:

- **Plan mode** (`jan --plan`) makes the session read-only. The agent investigates and proposes, but
  cannot change anything.
- **Safe mode** (`jan --safe`) asks before every write and shell command, instead of auto-approving
  them.
- **Sandbox** (`jan --sandbox`) confines shell commands to the project with an OS sandbox. Off by
  default on the CLI, always on in the desktop app.

See [Tools & permissions](/docs/agent/permissions) for the full picture.

---

### Src Pages Docs Agent Quickstart (`docs/src/pages/docs/agent/quickstart.mdx`)

---
title: QuickStart
description: Install Jan Agent, connect a provider, and finish your first task in the terminal.
keywords:
  [
    Jan,
    Jan Agent,
    agents,
    autonomous agent,
    local AI agent,
    quickstart,
    install,
    terminal,
    CLI,
  ]
---

import { Callout, Steps, Tabs } from 'nextra/components'
import { DocCard, DocCards } from '@/components/DocCard'
import { SquareTerminal, Wrench, Settings, Users } from 'lucide-react'

# QuickStart

Get Jan Agent running and finish a real task in a few minutes. You need a terminal and an API key
for one model provider.

<Steps>

### Install Nightly

<Callout type="warning">
Jan Agent is a preview. The installer on `dev` pulls from the `agent-nightly` channel - expect
nightly-quality builds.
</Callout>

<Tabs items={['macOS / Linux', 'Windows']}>
  <Tabs.Tab>
```bash
curl -fsSL https://delta.jan.ai/jan-cli/install-jan-agent.sh | bash
```

Installs to `~/.local/bin` by default. Override it with `JAN_INSTALL_DIR=/usr/local/bin`, and make
sure the directory is on your `PATH`.
  </Tabs.Tab>
  <Tabs.Tab>
```powershell
irm https://delta.jan.ai/jan-cli/install-jan-agent.ps1 | iex
```

Installs to `%USERPROFILE%\.local\bin`, which isn't on your `PATH` by default. To install and add
it to your user `PATH` in one go, then open a new terminal:

```powershell
& ([scriptblock]::Create((irm https://delta.jan.ai/jan-cli/install-jan-agent.ps1))) -AddToPath
```
  </Tabs.Tab>
</Tabs>

Check it worked:

```bash
jan --version
```

<Callout type="info">
Already have the Jan repo checked out? `scripts/install-jan-agent.sh --source` builds from your
checkout instead of downloading. Source builds don't self-update, because the update channel is
stamped in by CI.
</Callout>

### Connect a provider

Jan Agent has no local inference engine - it calls a remote provider. Point it at one:

```bash
jan config set --provider anthropic --api-key sk-ant-...
```

Or sign in to Tokamak, which needs no key of your own:

```bash
jan login
```

Confirm what's configured (keys are redacted):

```bash
jan config list
```

<Callout type="info">
Already using Jan Desktop? Jan Agent picks up the providers you configured there, so you can skip
this step entirely.
</Callout>

### Open the console

Change into the project you want to work on, then:

```bash
jan
```

You get an interactive console: type a message, press <kbd>Enter</kbd>. The agent works in the
directory you launched it from.

### Give it a task

Start with something small and verifiable:

```
Read the README and tell me how to run the tests
```

It will read files and answer. Nothing has changed on disk yet - reading is free.

Now ask for a change:

```
Add a --version flag to the CLI and update the README to document it
```

This time the agent asks before it writes:

```
write src/cli.rs
  y  yes, once
  a  yes, and don't ask again this session
  n  no
```

Press <kbd>y</kbd> to allow it once, <kbd>a</kbd> to stop being asked for that kind of call, or
<kbd>n</kbd> to refuse. You stay in the loop for every write and every shell command.

### Watch it work

While the agent runs you'll see each tool call as it happens, and a receipt when the turn ends:

```
  Worked for 12s • 14:02 • ↑ 43K ↓ 1.1K • 45.6/s
```

That's how long the turn took, when it finished, context sent, tokens produced, and throughput.

Press <kbd>Esc</kbd> at any point to cancel the current turn. Press <kbd>Ctrl</kbd>+<kbd>D</kbd> to
quit.

</Steps>

## Pick up where you left off

Sessions are saved per project. To continue the most recent one:

```bash
jan -c
```

Or choose from a list inside the console with `/resume`.

## Try these next

Three things worth doing on your first day:

**Explore before changing.** Start read-only so the agent can investigate without touching anything:

```bash
jan --plan
```

It will study the project and propose a plan. Type `/plan exit` when you're ready to let it act.

**Set a finish line.** Instead of babysitting turn by turn, tell it what "done" means:

```
/goal all tests in tests/auth pass
```

The agent keeps working and re-checking until that holds.

**Seed a task from the shell.** Handy in a script or an alias:

```bash
jan --task "fix the failing test in tests/auth"
```

## Where to go next

<DocCards>
  <DocCard title="Interactive console" href="/docs/agent/console" icon={<SquareTerminal size={20} />}>Every slash command and keyboard shortcut.</DocCard>
  <DocCard title="Tools & permissions" href="/docs/agent/permissions" icon={<Wrench size={20} />}>What the agent can reach, and how to widen or narrow it.</DocCard>
  <DocCard title="Configuration" href="/docs/agent/project-config" icon={<Settings size={20} />}>Per-project instructions, models, and budgets.</DocCard>
  <DocCard title="Subagents" href="/docs/agent/subagents" icon={<Users size={20} />}>Split a big job across agents running in parallel.</DocCard>
</DocCards>

## Troubleshooting

**`jan: command not found`** - the install directory isn't on your `PATH`. The installer prints a
note when it detects this. Add `~/.local/bin` to your shell profile. On Windows, re-run the installer
with `-AddToPath` (see [Install Nightly](#install-nightly)) and open a new terminal. If the installer warns that
another `jan` comes first on your `PATH`, remove the entry it names.

**"no providers configured"** - run `jan config set --provider <id> --api-key <key>`, or `jan login`
for Tokamak. `jan config path` prints the file it reads.

**The agent won't edit anything** - you're in plan mode. Type `/plan exit`.

**Model errors or an unexpected model** - `/model` inside the console lists what's available and
switches on the spot. `jan config list` shows which providers are wired up.

---

### Src Pages Docs Desktop Index (`docs/src/pages/docs/desktop/index.mdx`)

---
title: Overview
description: Jan is a local, privacy-first AI — a Claude-class alternative you own and control
keywords:
  [
    Jan,
    Jan AI,
    Claude alternative,
    local AI,
    private AI,
    self-hosted AI,
    foundation models,
    AI ownership,
    open superintelligence,
    AI ecosystem,
    llama.cpp,
    Model Context Protocol,
    MCP,
    GGUF models,
    large language model,
    LLM,
  ]
---

import { Callout } from 'nextra/components'
import { DocCard, DocCards } from '@/components/DocCard'
import { Rocket, Package, Bot, Plug, Monitor, Globe, Server, Brain, Terminal, Layers, MessageCircle, Hammer } from 'lucide-react'

# Overview

<Callout type="info">
**Jan 0.8.4 is now live!** Native web search and fetch tools, per-model chat-template kwargs for llama.cpp, a token counter for remote and MLX providers, settings and credentials moved into a backend store, and a batch of chat, MCP, provider, and streaming fixes. [See the full changelog.](/changelog/2026-07-21-jan-v0.8.4)
</Callout>

Jan is an open-source replacement for Claude and ChatGPT. It comes with built-in foundation models and an application ecosystem that runs entirely on your hardware, keeping your data private and giving you full ownership over your AI.

## Get Started

<DocCards>
  <DocCard title="Quickstart" href="/docs/desktop/quickstart" icon={<Rocket size={20} />}>Install Jan on macOS, Windows, or Linux and chat with AI in minutes — no account needed.</DocCard>
  <DocCard title="Models" href="/docs/desktop/jan-models/jan-code-4b" icon={<Package size={20} />}>Jan ships with foundation models built for reasoning, agents, and deep research — ready to run on your computer.</DocCard>
  <DocCard title="Cowork (Preview)" href="/docs/desktop/cowork" icon={<Hammer size={20} />}>Give Jan a task and it works on it autonomously — files, commands, and web research.</DocCard>
  <DocCard title="Agents" href="/docs/agent" icon={<Bot size={20} />}>Autonomous AI agents that run on your hardware — read files, manage calendars, and take actions via WhatsApp, Discord, or Slack.</DocCard>
  <DocCard title="Integrations" href="/docs/desktop/integrations/claude-code" icon={<Plug size={20} />}>Connect Jan with Claude Code, OpenClaw, MCP servers, and more.</DocCard>
</DocCards>

## Ecosystem

<DocCards>
  <DocCard title="Jan Desktop" href="/docs/desktop/quickstart" icon={<Monitor size={20} />}>macOS, Windows, and Linux apps with full offline support.</DocCard>
  <DocCard title="Jan Web" href="https://chat.jan.ai" icon={<Globe size={20} />}>Jan in the browser — a direct alternative to chatgpt.com.</DocCard>
  <DocCard title="Jan Server" href="/docs/desktop/api-server" icon={<Server size={20} />}>Run a local OpenAI-compatible API server on your machine.</DocCard>
  <DocCard title="Jan Models" href="/docs/desktop/jan-models/jan-code-4b" icon={<Brain size={20} />}>Optimized for reasoning, agents, and deep research — built to match frontier AI capabilities and small enough to run on your personal computer.</DocCard>
  <DocCard title="Jan CLI" href="/docs/desktop/cli" icon={<Terminal size={20} />}>Serve local models and launch autonomous agents from your terminal using Jan's local AI.</DocCard>
  <DocCard title="Jan Platform" href="https://jan.ai" icon={<Layers size={20} />}>The infrastructure layer for building AI products on top of Jan. (Coming Soon)</DocCard>
</DocCards>


## Community

<DocCards>
  <DocCard title="Discord" href="https://discord.gg/FTk2MvZwJH" icon={<MessageCircle size={20} />}>Join the Jan community — get help, share your setup, and talk local AI.</DocCard>
  <DocCard title="Reddit" href="https://www.reddit.com/r/askjan/" icon={<MessageCircle size={20} />}>Ask questions, share setups, and discuss local AI on r/askjan.</DocCard>
</DocCards>

## Principles

- [Open source](https://www.redhat.com/en/blog/open-source-culture-9-core-principles-and-values): [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) licensed, built in public.
- No [vendor lock-in](https://en.wikipedia.org/wiki/Vendor_lock-in): Switch freely between local and frontier models.
- [Right to Repair](https://en.wikipedia.org/wiki/Right_to_repair): Inspect, audit, and modify your AI stack.

## Acknowledgements

Jan exists because we've borrowed, learned, and built on the work of others.

- [llama.cpp](https://github.com/ggerganov/llama.cpp) and [GGML](https://github.com/ggerganov/ggml) for efficient inference
- [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/) for ideas, feedback, and debate
- [Model Context Protocol](https://modelcontextprotocol.io) for MCP integrations
- The open-source community for contributions, bug reports, and improvements

---

### Src Pages Docs Desktop Quickstart (`docs/src/pages/docs/desktop/quickstart.mdx`)

---
title: QuickStart
description: Get started with Jan and start chatting with AI in minutes.
keywords:
  [
    Jan,
    local AI,
    LLM,
    chat,
    threads,
    models,
    download,
    installation,
    conversations,
  ]
---

import { Callout, Steps } from 'nextra/components'
import { Link } from 'nextra-theme-docs'
import { DocCard, DocCards } from '@/components/DocCard'
import { FolderOpen, Bot, Cpu, Plug, Brain, Hammer } from 'lucide-react'

# QuickStart

Get up and running with Jan in minutes. This guide will help you install Jan, download a model, and start chatting immediately.

<Steps>

### Install Jan

1. <Link href="https://jan.ai/download" target="_blank" rel="noopener noreferrer">Download Jan</Link>
2. Install the app ([Mac](/docs/desktop/install/mac), [Windows](/docs/desktop/install/windows), [Linux](/docs/desktop/install/linux))
3. Launch Jan

### Getting Ready

Jan automatically downloads its default foundation model on first launch. Once the download completes, you're ready to chat — no setup required.



### Start Chatting

Type your message and start chatting. Jan is ready to go.



Try asking:
- "Explain quantum computing in simple terms"
- "Help me write a Python function to sort a list"
- "What are the pros and cons of electric vehicles?"

<Callout type="tip">
Jan has built-in web search and fetch tools, so you can ask about the latest news and current events without any extra setup. If you'd rather use a different provider, you can also configure the [Exa](https://exa.ai) MCP server under **Settings > MCP Servers** with your own API key.
</Callout>

### Download More Models

The **Hub** is where you browse and download local models. Each model shows whether it fits your
hardware, so you can pick one your machine can run comfortably.

1. Open the **Hub** from the left sidebar
2. Browse or search for a model, then click **Download**
3. Once downloaded, select it from the model selector in any chat



Want help choosing? See [Manage Models](/docs/desktop/manage-models) and the
[local engine guide](/docs/desktop/local-engine/llama-cpp).

</Steps>

## Local or Cloud

Jan runs models two ways, and you can mix both in the same app:

- **Local models** run entirely on your machine — private, offline, no API key. This is the default.
- **Cloud models** run on a provider's servers (OpenAI, Anthropic, Google, and more). They need an
  API key but let you use the largest frontier models without local hardware.

### Connect a Cloud Provider

1. Navigate to **Settings** > **Model Providers**
2. Select a provider (e.g. **OpenAI** or **Anthropic**) and paste your API key
3. Select one of its models from the model selector and start chatting



<Callout type="info">
Running your own server or using a provider Jan doesn't list? Connect any OpenAI- or
Anthropic-compatible endpoint — see [Custom Endpoints](/docs/desktop/remote-models/custom-endpoint).
</Callout>

See all [Cloud Providers](/docs/desktop/remote-models/openai) for per-provider setup.

## App Features

Jan is more than a chat interface. Organize your work with Projects, build custom Assistants, run autonomous Agents, and connect external tools via MCP.

<DocCards>
  <DocCard title="Cowork (Preview)" href="/docs/desktop/cowork" icon={<Hammer size={20} />}>Hand Jan a task it works through on its own — in a sandbox or directly on your project.</DocCard>
  <DocCard title="Projects" href="/docs/desktop/projects" icon={<FolderOpen size={20} />}>Organize conversations with shared instructions and files.</DocCard>
  <DocCard title="Assistants" href="/docs/desktop/assistants" icon={<Bot size={20} />}>Set up a personal AI assistant tailored to how you work.</DocCard>
  <DocCard title="Agents" href="/docs/agent" icon={<Cpu size={20} />}>Autonomous AI that reads files, manages calendars, and takes actions.</DocCard>
  <DocCard title="Connectors (MCP)" href="/docs/desktop/mcp" icon={<Plug size={20} />}>Extend your AI with web search, code execution, databases, and more.</DocCard>
</DocCards>

## Models

Beyond the default model, Jan comes with a range of foundation models for different tasks — from image understanding and reasoning to coding.

<DocCards>
  <DocCard title="Jan-Code-4B" href="/docs/desktop/jan-models/jan-code-4b" icon={<Brain size={20} />}>Optimized for code generation and reasoning tasks.</DocCard>
  <DocCard title="Jan-v3-4B" href="/docs/desktop/jan-models/jan-v3-4b-base-instruct" icon={<Brain size={20} />}>General purpose instruction-following model.</DocCard>
  <DocCard title="Jan-v2-VL" href="/docs/desktop/jan-models/jan-v2-vl-med" icon={<Brain size={20} />}>Vision-language model with strong reasoning capability for image understanding.</DocCard>
</DocCards>

---

### Public Robots (`docs/public/robots.txt`)

# *
User-agent: *
Allow: /

# Host
Host: https://jan.ai/

# Sitemaps
Sitemap: https://jan.ai/sitemap.xml

---

### Src Pages  App (`docs/src/pages/_app.mdx`)

import '@/styles/main.scss'
import '@code-hike/mdx/styles.css'
import { Fragment } from "react"
import Script from "next/script"

export default function App({ Component, pageProps }) {
  return (
    <Fragment>
      <Script
        strategy="afterInteractive"
        src={`https://www.googletagmanager.com/gtag/js?id=${process.env.GTM_ID}`}
      />
      <Script
        id="gtag-init"
        strategy="afterInteractive"
        dangerouslySetInnerHTML={{
          __html: `
            window.dataLayer = window.dataLayer || [];
            function gtag(){dataLayer.push(arguments);}
            gtag('js', new Date());
            gtag('config', '${process.env.GTM_ID}', {
              page_path: window.location.pathname,
            });
          `,
        }}
      />
      <Component {...pageProps} />
    </Fragment>
  )
}

---

### Src Pages Blog (`docs/src/pages/blog.mdx`)

---
title: Blog
description: Jan is a ChatGPT-alternative that runs on your own computer, with a local API server.
keywords:
  [
    Jan,
    Customizable Intelligence, LLM,
    local AI,
    privacy focus,
    free and open source,
    private and offline,
    conversational AI,
    no-subscription fee,
    large language models,
    architecture,
  ]
---

import Blog from "@/components/Blog"
import fs from 'fs'
import path from 'path'
import matter from 'gray-matter'
import { format } from 'date-fns'

<Blog/>

export const getStaticProps = async() => {
  const blogPosts = await fs.readdirSync(path.join(process.cwd(), 'src/pages/post')).filter((file) => {
    return path.extname(file).toLowerCase() === ".mdx" && !file.startsWith('index')
  })
  const allBlogPosts = []

  for (const item of blogPosts) {
    const content = fs.readFileSync(path.join(process.cwd(), `src/pages/post/${item}`), "utf8")
    const frontmatter = matter(content)
    if(!frontmatter.data.unlisted) {
      allBlogPosts.push({
        title: frontmatter.data?.title || null,
        url: '/post/' + item?.replace(/\.mdx?/, ''),
        description: frontmatter.data?.description || null,
        date: String(frontmatter.data?.date) || null,
        tags: frontmatter.data.tags?.split(', ') || null,
        author: frontmatter.data?.author || null,
        categories: frontmatter.data.categories?.split(', ') || null,
      })
    }
    allBlogPosts.sort((a, b) => new Date(b.date) - new Date(a.date))    
  }

  return {
    props: {
      ssg: allBlogPosts
    },
  }
}

---

### Src Pages Changelog (`docs/src/pages/changelog.mdx`)

---
title: Changelog
description: Jan is a ChatGPT-alternative that runs on your own computer, with a local API server.
keywords:
  [
    Jan,
    Customizable Intelligence, LLM,
    local AI,
    privacy focus,
    free and open source,
    private and offline,
    conversational AI,
    no-subscription fee,
    large language models,
    architecture,
  ]
---

import Changelog from "@/components/Changelog"
import fs from 'fs'
import path from 'path'
import matter from 'gray-matter'
import { format } from 'date-fns'

export const getStaticProps = async() => {
  const getChangelog = await fs.readdirSync(path.join(process.cwd(), 'src/pages/changelog')).filter((file) => {
    return path.extname(file).toLowerCase() === ".mdx" && !file.startsWith('index')
  })
  const changelog = []

  for (const item of getChangelog) {
    const content = fs.readFileSync(path.join(process.cwd(), `src/pages/changelog/${item}`), "utf8")
    const frontmatter = matter(content)

    if(!frontmatter.data.unlisted) {
      changelog.push({
        url: item.replace('.mdx', ''),
        title: frontmatter?.data?.title || '',
        ogImage: frontmatter?.data?.ogImage || null,
        version: frontmatter?.data?.version || null,
        description: frontmatter?.data?.description || null,
        date: String(frontmatter?.data?.date) || null,
      })
    }
    changelog.sort((a, b) => new Date(b.date) - new Date(a.date))
  }


  return {
    props: {
      ssg: changelog
    },
  }
}

<Changelog />

---

### Src Pages Download (`docs/src/pages/download.mdx`)

---
title: Download
description: Jan is a ChatGPT-alternative that runs on your own computer, with a local API server.
keywords:
  [
    Jan,
    Customizable Intelligence, LLM,
    local AI,
    privacy focus,
    free and open source,
    private and offline,
    conversational AI,
    no-subscription fee,
    large language models,
    architecture,
  ]
---

import Download from "@/components/Download"

export const getStaticProps = async() => {
  const resRelease = await fetch('https://api.github.com/repos/janhq/jan/releases/latest')  
  const release = await resRelease.json()

  return {
    props: {
      ssg: {
        lastRelease: release,
      },
    },
  }
}

<Download />

---

### Src Pages Privacy (`docs/src/pages/privacy.mdx`)

---
title: Privacy
---

import { Callout } from 'nextra/components'

# Privacy

Menlo Research is committed to protecting your privacy and ensuring that your personal information is handled safely and
responsibly. This policy outlines how we collect, store, and use your personal information when you use any of our products.

## Data Collection

Jan, Cortex, and all Menlo Research products do not collect personally identifying information. You can read about
[our philosophy](/about#philosophy) here and audit our open-source codebases.

### When you voluntarily provide data

We -do- collect personal information you voluntarily provide us, e.g., when you sign up for our newsletter, join our
Discord, or contact us via email.

### Jan

Jan runs with privacy by default and is used 100% offline on your own computer. Your data (e.g., conversation history,
usage logs) are stored locally and never leave your computer.

<Callout type="info">
If you use a Remote AI API (e.g., OpenAI API, Groq API), your data will naturally travel to their servers. They will be
subject to the privacy policy of the respective API provider.
</Callout>

### Cortex

Cortex is a library that runs large language models (LLMs) locally on your computer. Cortex does not collect any personal information.

## Data Sharing

We do not share your personal information with third parties except as required by law or as necessary to provide you with
the services you have requested.

## Data Security

We take the security of your personal information seriously and have implemented appropriate technical and organizational
measures to protect your personal information from unauthorized access, disclosure, or misuse.

## Your Choices

You have the right to access, update, and delete your personal information at any time. If you subscribed to our newsletter,
you may also opt-out of receiving marketing communications from us by following the unsubscribe link included in our emails.

## Contact Us

If you have any questions or concerns about our privacy policy, please contact us at hello@menlo.ai.

---

### Src Pages Support (`docs/src/pages/support.mdx`)

---
title: Support - Jan
---

# Support

- Bugs & requests: file a GitHub ticket [here](https://github.com/janhq/jan/issues)
- For discussion: join our Discord [here](https://discord.gg/FTk2MvZwJH)
- For business inquiries: email hello@jan.ai
- For jobs: please email hr@jan.ai

---

### Src Pages Tokamak (`docs/src/pages/tokamak.mdx`)

---
title: Tokamak
description: Tokamak is the org-level self-hosted AI backend — router, fusion model, governance and audit.
noindex: true
---

import Tokamak from "@/components/Tokamak"

<Tokamak/>

---

### Src Pages Changelog 2023 12 21 Faster Inference Across Platform (`docs/src/pages/changelog/2023-12-21-faster-inference-across-platform.mdx`)

---
title: "Faster inference across: Mac, Windows, Linux, and GPUs"
version: 0.4.3
description: ""
date: 2023-12-21
ogImage: "https://catalog.jan.ai/docs/Jan_v0.4.3.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Faster inference across= Mac, Windows, Linux, and GPUs" date= "2023-12-21" ogImage= "https://catalog.jan.ai/docs/Jan_v0.4.3.gif" />

### Highlights 🎉

- Custom models: `Trinity`, `Pandora` (great for general use).
- Faster inference across: Mac, Windows, Linux, and GPUs.
- Connect to remote OpenAI models like GPT4 via API key.

---

### Src Pages Changelog 2024 01 16 Settings Options Right Panel (`docs/src/pages/changelog/2024-01-16-settings-options-right-panel.mdx`)

---
title: "Thread settings options in the right panel"
version: 0.4.4
description: ""
date: 2024-01-16
ogImage: ""
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Thread settings options in the right panel" date= "2024-01-16" ogImage= "" />

### Highlights 🎉

- You can now see whether the model is compatible with running on your device.
- You can switch model mid-threads.
- More thread settings options in the right panel.
- CI automation, anti-virus checks.
- Social media access to Jan's Discord & Github from the app for further user support.
- Fixed major bugs, more stability.

---

### Src Pages Changelog 2024 01 29 Local Api Server (`docs/src/pages/changelog/2024-01-29-local-api-server.mdx`)

---
title: "Local API server"
version: 0.4.5
description: ""
date: 2024-01-29
ogImage: "https://catalog.jan.ai/docs/Jan_v0.4.5.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Local API server" date= "2024-01-29" ogImage= "https://catalog.jan.ai/docs/Jan_v0.4.5.gif" />

### Highlights 🎉

- Local API Server: Experience Jan's dashboard for the local API server to make your data processing smoother and more efficient.
- HTTP Proxy Support: Now, downloading and connecting are seamless, just like browsing Hugging Face in your browser.
- Updated Settings Page: Find what you need faster! We've updated the settings page.

### Fixes 💫

- Auto Update: Enjoy smoother updates. We've fixed the glitches.
- Swagger API Page: Full documentation, no more blanks.
- GPU for Models: Your imported models now fully leverage GPU power.

---

### Src Pages Changelog 2024 02 05 Jan Data Folder (`docs/src/pages/changelog/2024-02-05-jan-data-folder.mdx`)

---
title: "Jan Data Folder"
version: 0.4.6
description: ""
date: 2024-02-05
ogImage: "https://catalog.jan.ai/docs/jan_product_update_feature.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan Data Folder" date= "2024-02-05" ogImage= "https://catalog.jan.ai/docs/jan_product_update_feature.gif" />

### Highlights 🎉

- `Changing Jan Data Folder`: Now, moving your data across different folders is just a click away, making organization simpler. So you can even use an external drive.
- Factory Settings: You can reset all of Jan's usage data for a fresh start.

### Fixes 💫

- Easily see each thread's last update time, like WhatsApp, keeping you informed.
- A new loading screen during data migration ensures the app is responsive.
- Enhanced notifications for clearer feedback on model runs or errors.

---

### Src Pages Changelog 2024 02 10 Jan Is More Stable (`docs/src/pages/changelog/2024-02-10-jan-is-more-stable.mdx`)

---
title: "Jan is more stable 👋"
version: 0.5.5
description: "Jan supports Llama 3.2 and Qwen 2.5"
date: 2024-10-02
ogImage: "/assets/images/changelog/jan-v0.5.5.jpeg"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan is faster now" date="2024-09-01" ogImage= "/assets/images/changelog/jan-v0.5.5.jpeg"  />

Highlights 🎉

- Meta's Llama 3.2 and Alibaba's Qwen 2.5 added to the hub
- Improved starter screen
- Better local vs. cloud model navigation

Fixes 💫

- Solved GPU acceleration for GGUF models
- Improved model caching & threading
- Resolved input & toolbar overlaps

Update your product or download the latest: https://jan.ai

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.5).

---

### Src Pages Changelog 2024 02 26 Home Servers With Helm (`docs/src/pages/changelog/2024-02-26-home-servers-with-helm.mdx`)

---
title: "Run Jan on your home-servers with Helm"
version: 0.4.7
description: ""
date: 2024-02-26
ogImage: ""
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Run Jan on your home-servers with Helm" date= '2024-02-26' ogImage= "" />

### Highlights 🎉

- Run Jan on your `home-servers` with `Helm`
- Use Jan headless or with a Web UI
- `Intel Arc` & `AMD GPU` support through `Vulkan` & `LlamaCPP`


### Features & Fixes 💫

- 48 fixes, refactoring and stability improvements.
- Conversation threads are auto-summarized & messages are editable.
- Encountering an error? We've replaced vague alerts with a troubleshooting assistant.

---

### Src Pages Changelog 2024 03 06 Ui Revamp Settings (`docs/src/pages/changelog/2024-03-06-ui-revamp-settings.mdx`)

---
title: "New UI & Codestral Support"
version: 0.5.0
description: "Revamped Jan's UI to make it clearer and more user-friendly"
date: 2024-06-03
ogImage: "https://catalog.jan.ai/docs/jan_v0.5.0.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "New UI & Codestral Support" date= "2024-06-03" ogImage= "https://catalog.jan.ai/docs/jan_v0.5.0.gif" />

Revamped Jan's UI to make it clearer and more user-friendly.

- Updated Assistant, Model, and Tools sections
- Categorized customization options for easier control
- New settings for models, APIs, and experimental features

## New Model: Codestral

Jan now supports Mistral's new model Codestral. Thanks [Bartowski](https://huggingface.co/bartowski) for the GGUF model. You can download the model from the hub.

## More GGUF models

More GGUF models can run in Jan - we rebased to llama.cpp b3012.Big thanks to [ggerganov](https://github.com/ggerganov) 

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.0).

---

### Src Pages Changelog 2024 03 11 Import Models (`docs/src/pages/changelog/2024-03-11-import-models.mdx`)

---
title: "Import models directly using the UI"
version: 0.4.8
description: ""
date: 2024-03-11
ogImage: ""
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Import models directly using the UI" date= '2024-03-11' ogImage= ""/>

### Highlights 🎉

- Import models directly using the UI
- A revamped system monitor

### Features & Fixes 💫

- Install Jan using Homebrew on Mac Silicon (thanks to https://github.com/chenrui333 (I'll tag you when I find your Discord handle! :D)).
- Fixed an HTTP Proxy issue causing download errors.
- UI Improvements and more.

---

### Src Pages Changelog 2024 03 19 Nitro Tensorrt Llm Extension (`docs/src/pages/changelog/2024-03-19-nitro-tensorrt-llm-extension.mdx`)

---
title: "Nitro-Tensorrt-LLM Extension"
version: 0.4.9
description: ""
date: 2024-03-19
ogImage: ""
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Nitro-Tensorrt-LLM Extension" date= '2024-03-19' ogImage= "" />

### Highlights 🎉

- Nitro-Tensorrt-LLM Extension.
- Update models.json.
- Move tensorrt executable to the engine.

---

### Src Pages Changelog 2024 04 02 Groq Api Integration (`docs/src/pages/changelog/2024-04-02-groq-api-integration.mdx`)

---
title: "Groq API Integration"
version: 0.4.10
description: ""
date: 2024-04-02
ogImage: "https://catalog.jan.ai/docs/jan_update_groq.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Groq API Integration" date= "2024-04-02" ogImage= "https://catalog.jan.ai/docs/jan_update_groq.gif" />

### Highlights 🎉

- Groq API Integration 
- Enhanced hardware troubleshooting guide

### Features & Fixes  💫 

- Improved Jan data folder's functionality.
- Fixed URI malformed and `stop` parameter error.
- VRAM-aware model recommendations.

---

### Src Pages Changelog 2024 04 15 New Mistral Extension (`docs/src/pages/changelog/2024-04-15-new-mistral-extension.mdx`)

---
title: "New Mistral Extension"
version: 0.4.11
description: "Jan has a new Mistral Extension letting you chat with larger Mistral models via Mistral API"
date: 2024-04-15
ogImage: "https://catalog.jan.ai/docs/jan_mistral_api.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "New Mistral Extension" date= '2024-04-15' ogImage= "https://catalog.jan.ai/docs/jan_mistral_api.gif"/>

### Highlights 🎉

- Jan has a new `Mistral Extension` letting you chat with larger Mistral models via Mistral API. You can still run smaller Mistral models locally, but now there's a remote option.
- 3rd party extensions can register their own settings pages in the app without any code.
- You can now change set API Prefix for the local API server.
- You can now customize your Assistant's name in Thread Settings.

---

### Src Pages Changelog 2024 04 25 Llama3 Command R Hugginface (`docs/src/pages/changelog/2024-04-25-llama3-command-r-hugginface.mdx`)

---
title: 'Jan now supports Llama3 and Command R+'
version: 0.4.12
description: "Jan has added compatibility with Llama3 & Command R+"
date: 2024-04-25
ogImage: "https://catalog.jan.ai/docs/jan_llama3.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= 'Jan now supports Llama3 and Command R+' date= "2024-04-25" ogImage= 'https://catalog.jan.ai/docs/jan_llama3.gif' />

Jan has added compatibility with Meta’s open-source language model, `Llama3`, through the integration with `llamacpp` (thanks to [@ggerganov](https://github.com/ggerganov)). 

Additionally, `Command R+` is now supported. It is the first open-source model to surpass GPT-4 on the [LMSys leaderboard](https://chat.lmsys.org/?leaderboard).



## Import Huggingface models directly

Users can now import Huggingface models into Jan. Simply copy the model’s link from Huggingface and paste it into the search bar on Jan Hub.



## Enhanced LaTeX understanding

Jan now understands LaTeX, allowing users to process and understand complex mathematical expressions more effectively. 



For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.4.12).

---

### Src Pages Changelog 2024 05 20 Llamacpp Upgrade New Remote Models (`docs/src/pages/changelog/2024-05-20-llamacpp-upgrade-new-remote-models.mdx`)

---
title: "Jan now supports more GGUF models"
version: 0.4.13
description: "We rebased to llamacpp b2865."
date: 2024-05-20
ogImage: "https://catalog.jan.ai/docs/jan_v0.4.13_update.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan now supports more GGUF models" date= '2024-05-20' ogImage= "https://catalog.jan.ai/docs/jan_v0.4.13_update.gif" />

With this release, more GGUF models should work now! We rebased to llamacpp b2865!

## New remote models: Anthropic & Cohere APIs

Jan now supports `Anthropic API` models `Command R` and `Command R+`, along with `Cohere`'s `Claude 3 Opus`, `Claude 3 Sonnet`, and `Claude 3 Haiku`.

## New integrations: Martian and OpenRouter

Jan supports `Martian`, a dynamic LLM router that routes between multiple models and allows users to reduce costs by 20% to 97%. Jan also supports `OpenRouter`, helping users select the best model for each query.



## GPT-4o Access

Users can now connect to OpenAI's new model GPT-4o.



For more details, see the [GitHub release notes.](https://github.com/janhq/jan/releases/tag/v0.4.13)

---

### Src Pages Changelog 2024 05 28 Cohere Aya 23 8b 35b Phi 3 Medium (`docs/src/pages/changelog/2024-05-28-cohere-aya-23-8b-35b-phi-3-medium.mdx`)

---
title: "Jan now compatible with Aya 23 8B & 35B and Phi-3-Medium"
version: 0.4.14
description: "Jan now supports Cohere's Aya 23 8B & 35B and Microsoft's Phi-3-Medium."
date: 2024-05-28
ogImage: "https://catalog.jan.ai/docs/jan-v0-4-14-phi3.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan now compatible with Aya 23 8B & 35B and Phi-3-Medium" date= "2024-05-28" ogImage= "https://catalog.jan.ai/docs/jan-v0-4-14-phi3.gif" />

Jan now supports `Cohere`'s new models `Aya 23 (8B)` & `Aya 23 (35B)` and `Microsoft`'s `Phi-3-Medium`. 

More GGUF models can run in Jan - we rebased to llama.cpp b2961.

Huge shoutouts to [ggerganov](https://github.com/ggerganov) and contributors for llama.cpp, and [Bartowski](https://huggingface.co/bartowski) for GGUF models.

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.4.14).

---

### Src Pages Changelog 2024 06 21 Nvidia Nim Support (`docs/src/pages/changelog/2024-06-21-nvidia-nim-support.mdx`)

---
title: "Jan supports NVIDIA NIM"
version: 0.5.1
description: "Jan has integrated NVIDIA NIM and supports Qwen 2 7B"
date: 2024-06-21
ogImage: "https://catalog.jan.ai/docs/jan_nvidia_nim_support.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan supports NVIDIA NIM" date= '2024-06-21' ogImage= "https://catalog.jan.ai/docs/jan_nvidia_nim_support.gif"/>

## NVIDIA NIM

We've integrated NVIDIA NIM support.

## Qwen 2 7B

You can now access Qwen 2 7B directly in the Jan Hub.

We've updated to llama.cpp b3088 for better performance - thanks to [GG](https://github.com/ggerganov) 

## Fixes

- Fixed Anthropic API error
- Reduced chat font weight (back to normal!)
- Restored the maximize button

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.1).

---

### Src Pages Changelog 2024 07 15 Claude 3 5 Support (`docs/src/pages/changelog/2024-07-15-claude-3-5-support.mdx`)

---
title: "Jan supports Claude 3.5 Sonnet"
version: 0.5.2
description: "You can run Claude 3.5 Sonnet in Jan"
date: 2024-07-15
ogImage: "https://catalog.jan.ai/docs/jan_supports_claude_3_5.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan supports Claude 3.5 Sonnet" date= "2024-07-15" ogImage= "https://catalog.jan.ai/docs/jan_supports_claude_3_5.gif"  />

## Claude 3.5 Sonnet

We've integrated support for Claude 3.5 Sonnet, you can run Anthropic's latest model in Jan. 

Plus, You can now use optional spell check for chats. There's also a new shortcut for app logs in System Monitor.

## Fixes

In this hotfix, we've addressed several issues to improve your Jan experience:

### Gemma 2B Stability

Gemma 2B now runs without any issues.

### Tooltip Hover Functionality

We've restored the tooltip hover functionality, which makes it easier to access helpful information without any glitches.

### Right-click Options for Thread Settings

The right-click options for thread settings are now fully operational again. You can now manage your threads with this fix.

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.2).

---

### Src Pages Changelog 2024 09 01 Llama3 1 Gemma2 Support (`docs/src/pages/changelog/2024-09-01-llama3-1-gemma2-support.mdx`)

---
title: "v0.5.3 is out with stability improvements!"
version: 0.5.3
description: "You can run Llama 3.1 and Gemma 2 in Jan"
date: 2024-08-29
ogImage: "https://catalog.jan.ai/docs/janv0.5.3.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "v0.5.3 is out with stability improvements!" date="2024-09-01" ogImage= "https://catalog.jan.ai/docs/janv0.5.3.gif"  />

## Llama 3.1 and Gemma 2 Support

Jan finally supports Meta's Llama 3.1 and Google's Gemma 2. Thanks for the patience folks!

We've been working on stability issues over the last few weeks. Jan is now more stable.

### Additional Notes

- Upgraded the inference engine for better performance
- Model settings now persist across new threads
- Fixed the GPU memory utilization bar
- Some UX and copy improvements

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.3).

---

### Src Pages Changelog 2024 09 17 Improved Cpu Performance (`docs/src/pages/changelog/2024-09-17-improved-cpu-performance.mdx`)

---
title: "Jan is faster now"
version: 0.5.4
description: "Jan has faster CPU inference."
date: 2024-09-17
ogImage: "/assets/images/changelog/jan-v0.5.4.jpg"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan is faster now" date="2024-09-01" ogImage= "/assets/images/changelog/jan-v0.5.4.jpg"  />

Over the last few weeks, we've been working on improving Jan's stability. Every update helps us make Jan's experience faster and smoother. 
With version 0.5.4, you’ll notice AI running quicker on CPU - better performance all around.

### Model Downloads
You can now download models directly from the model selector in Threads. No more jumping between different tabs – just pick, download, and get started.

### Fast CPU Inference 
We've addressed the slower inference speeds on CPU, so you'll notice faster processing times, especially when using larger models.

### Model Starts
We tackled the notorious "model can't start / The specified module could not be found" error. 
Plus, Phi-3 models are now working smoothly even if you're using an outdated version.

### Consistent Warnings
Performance warnings are now aligned between Model Hub and Threads, giving you more reliable insights no matter where you're working.

### Persistent Thread Settings
Switching between threads used to reset your instruction settings. That’s fixed now! Your settings will stay intact as you jump between old and new threads.

### Minor UI Tweaks & Bug Fixes
We’ve also resolved issues with the input slider on the right panel and tackled several smaller bugs to keep everything running smoothly.

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.4).

---

### Src Pages Changelog 2024 10 24 Jan Stable (`docs/src/pages/changelog/2024-10-24-jan-stable.mdx`)

---
title: "Jan has Stable, Beta and Nightly versions"
version: 0.5.7
description: "This release is mostly focused on bug fixes."
date: 2024-10-24
ogImage: "https://catalog.jan.ai/docs/jan-v0.5.7.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan is faster now" date="2024-09-01" ogImage= "https://catalog.jan.ai/docs/jan-v0.5.7.gif"  />

Highlights 🎉

- Jan has Stable, Beta and Nightly versions
- Saving instructions for new threads is now stable

Fixes 💫

- Fixed broken links, hardware issues, and multi-modal download
- Resolved text overlap, scrolling, and multi-monitor reset problems
- Adjusted LLava model EOS token and context input

Update your product or download the latest: https://jan.ai

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.7).

---

### Src Pages Changelog 2024 11 22 Jan Bugs (`docs/src/pages/changelog/2024-11-22-jan-bugs.mdx`)

---
title: "Model downloads & running issues fixed"
version: 0.5.9
description: "Jan v0.5.9 is here: fixing what needed fixing."
date: 2024-11-22
ogImage: "https://catalog.jan.ai/docs/jan-v0.5.9.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan v0.5.9 is here:fixing what needed fixing" date="2024-11-22" ogImage= "https://catalog.jan.ai/docs/jan-v0.5.9.gif"  />

Jan v0.5.9 is here: fixing what needed fixing 

### Highlights 🎉 

- Model downloads & running issues fixed
- Document upload bugs resolved
- System glitches addressed: Factory Reset, HTTP Proxy, Hugging Face tokens
- Fixed issues with code blocks in streaming responses
- Improved the UX of the Local API Server page

Update your product or download the latest: https://jan.ai

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.9).

---

### Src Pages Changelog 2024 11.14 Jan Supports Qwen Coder (`docs/src/pages/changelog/2024-11.14-jan-supports-qwen-coder.mdx`)

---
title: "Jan supports Qwen2.5-Coder 14B & 32B"
version: 0.5.8
description: "Jan v0.5.8 is out: Jan supports Qwen2.5-Coder 14B & 32B through Cortex"
date: 2024-11-14
ogImage: "https://catalog.jan.ai/docs/jan-v0.5.8.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan supports Qwen2.5-Coder 14B & 32B" date="2024-11-14" ogImage= "https://catalog.jan.ai/docs/jan-v0.5.7.gif"  />

Jan v0.5.8 is out: Jan supports Qwen2.5-Coder 14B & 32B through Cortex

### Highlights 🎉 

- A new engine: Jan now run models via [Cortex](https://cortex.so)
- SupportsAlibaba_Qwen's Coder 14B & 32B Support
- Supports markdown rendering on user messages

and various UI/UX enhancements 💫

Update your product or download the latest: https://jan.ai

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.8).

---

### Src Pages Changelog 2024 12 03 Jan Is Faster (`docs/src/pages/changelog/2024-12-03-jan-is-faster.mdx`)

---
title: "Jan v0.5.10 is live"
version: 0.5.10
description: "Jan is faster, smoother, and more reliable."
date: 2024-12-03
ogImage: "https://catalog.jan.ai/docs/jan-v0.5.10.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan v0.5.10: Jan is faster, smoother, and more reliable." date="2024-12-03" ogImage= "https://catalog.jan.ai/docs/jan-v0.5.10.gif"  />

Jan v0.5.10 is live: Jan is faster, smoother, and more reliable.

### Highlights 🎉 

- Resolved model startup issues, memory leaks, and improved token limits
- Clearer error messages and subtle UX improvements

Update your product or download the latest: https://jan.ai

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.10).

---

### Src Pages Changelog 2024 12 05 Jan Hot Fix Mac (`docs/src/pages/changelog/2024-12-05-jan-hot-fix-mac.mdx`)

---
title: "Jan v0.5.11 is here!"
version: 0.5.11
description: "Critical issues fixed, Mac installation updated."
date: 2024-12-05
ogImage: "https://catalog.jan.ai/docs/jan-v0.5.11.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan v0.5.11: Jan is faster, smoother, and more reliable." date="2024-12-05" ogImage= "https://catalog.jan.ai/docs/jan-v0.5.11.gif"  />

Jan v0.5.11 is here - critical issues fixed, Mac installation updated.

### Highlights 🎉 

- Crashes (markdown & code highlighting)
- Thread switching & auto-scroll
- Syntax highlighting bugs
- API issues (Anthropic, OpenRouter)
- Title glitches with special characters
- Model settings inconsistencies

Update your product or download the latest: https://jan.ai

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.11).

---

### Src Pages Changelog 2024 12 30 Jan New Privacy (`docs/src/pages/changelog/2024-12-30-jan-new-privacy.mdx`)

---
title: "Jan gives you full control over your privacy"
version: 0.5.12
description: "Improved Privacy settings to give full control over analytics"
date: 2024-12-30
ogImage: "https://catalog.jan.ai/docs/jan-v0.5.12.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan v0.5.12: Improved Privacy settings to give full control over analytics." date="2024-12-30" ogImage= "https://catalog.jan.ai/docs/jan-v0.5.12.gif"  />

Jan v0.5.11 is here - critical issues fixed, Mac installation updated.

### Highlights 🎉 

- Updated privacy settings with opt-in/out options for Jan Analytics
- Adjustable chat width
- The right sidebar and input box are now optimized for new users

### Fixes 💫
- Updated privacy settings with opt-in/out options for Jan Analytics
- Adjustable chat width
- The right sidebar and input box are now optimized for new users

Update your product or download the latest: https://jan.ai

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.12).

---

### Src Pages Changelog 2025 01 06 Key Issues Resolved (`docs/src/pages/changelog/2025-01-06-key-issues-resolved.mdx`)

---
title: "A few key issues have been solved!"
version: 0.5.13
description: "Jan v0.5.13 is here: A few key issues have been solved."
date: 2025-01-06
ogImage: "/assets/images/changelog/jan-v0-5-13.jpg"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan v0.5.13: A few key issues have been solved!" date="2025-01-06" ogImage= "/assets/images/changelog/jan-v0-5-13.jpg"  />

👋 Jan v0.5.13 is here: A few key issues have been solved!

### Highlights 🎉 

- Resolved model loading issues on MacOS Intel
- Fixed app resetting max_tokens to 8192 on new threads - now uses model settings
- Fixed Vulkan settings visibility for some users

Update your product or download the latest: https://jan.ai

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.13).

---

### Src Pages Changelog 2025 01 23 Deepseek R1 Jan (`docs/src/pages/changelog/2025-01-23-deepseek-r1-jan.mdx`)

---
title: "Run DeepSeek R1 Distills error-free!"
version: 0.5.14
description: "Jan v0.5.14 is out: Run DeepSeek R1 Distills error-free!"
date: 2025-01-23
ogImage: "/assets/images/changelog/jan-v0-5-14-deepseek-r1.jpg"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title= "Jan v0.5.14: Run DeepSeek R1 Distills error-free!" date="2024-01-23" ogImage= "/assets/images/changelog/jan-v0-5-14-deepseek-r1.jpg"  />

👋 Jan v0.5.14 is out: Run DeepSeek R1 Distills error-free!

You can run DeepSeek R1 distills in Jan error-free. Follow our [step-by-step guide to run DeepSeek R1 locally](/post/deepseek-r1-locally) and get this AI model running on your device in minutes.

llama.cpp version updated via Cortex—thanks to GG & llama.cpp community!

- Paste GGUF links into Jan Hub to download
- Already downloaded the model but facing issues? Update Jan.

Models:

Qwen
- DeepSeek-R1-Distill-Qwen-1.5B-GGUF: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-1.5B-GGUF
- DeepSeek-R1-Distill-Qwen-7B-GGUF: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF
- DeepSeek-R1-Distill-Qwen-14B-GGUF: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF
- DeepSeek-R1-Distill-Qwen-32B-GGUF: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF

Llama
- DeepSeek-R1-Distill-Llama-8B-GGUF: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF
- DeepSeek-R1-Distill-Llama-70B-GGUF: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Llama-70B-GGUF

Update your Jan or [download the latest](https://jan.ai/).

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.14).

---

### Src Pages Changelog 2025 02 18 Advanced Llama.Cpp Settings (`docs/src/pages/changelog/2025-02-18-advanced-llama.cpp-settings.mdx`)

---
title: "You can now tweak llama.cpp settings, and add any cloud model!"
version: 0.5.15
description: "Jan v0.5.15 is out: Advanced llama.cpp settings and cloud model support"
date: 2025-02-18
ogImage: "/assets/images/changelog/jan-v0-5-15-llamacpp-settings.jpg"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title="Jan v0.5.15: You can now tweak llama.cpp settings, and add any cloud model!" date="2024-02-18" ogImage="/assets/images/changelog/jan-v0-5-15-llamacpp-settings.jpg" />

👋 Jan v0.5.15 is out with exciting new features and improvements!

## Highlights 🎉

- Improved llama.cpp settings for better model control and performance
- Install and use any cloud model seamlessly
- Enhanced hardware controls for better resource management
- New models supported:
  - DeepSeek AI
  - Google Gemini
  - OpenAI o3-mini
  - R1 distills

Update your Jan or [download the latest](https://jan.ai/).

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.15).

---

### Src Pages Changelog 2025 03 14 Jan Security Patch (`docs/src/pages/changelog/2025-03-14-jan-security-patch.mdx`)

---
title: "Security fixes and UI improvements"
version: 0.5.16
description: "Jan v0.5.16 is out: Security fixes and major improvements to Model Hub and chat experience"
date: 2025-03-14
ogImage: "/assets/images/changelog/jan-v0-5-16-security-patch.jpg"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title="Jan v0.5.16: Security fixes and UI improvements" date="2025-03-14" ogImage="/assets/images/changelog/jan-v0-5-16-security-patch.jpg" />

👋 Jan v0.5.16 is out: Security fixes (found in Cortex) and a few improvements!

## Highlights 🎉

- Redesigned Model Hub for better user experience
- Faster chat response times
- Cleaner layout and improved model picker
- New model support:
  - GPT-4.5 preview
  - Claude 3.7 Sonnet
  - Gemma 3

⚠️ Important: This release includes critical security fixes. We'll share more details about these security improvements soon.

Update your Jan or [download the latest](https://jan.ai/).

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.16).

---

### Src Pages Changelog 2025 05 14 Jan Qwen3 Patch (`docs/src/pages/changelog/2025-05-14-jan-qwen3-patch.mdx`)

---
title: "Qwen3 support is now more reliable."
version: 0.5.17
description: "Jan v0.5.17 is out: Qwen3 support is now more reliable"
date: 2025-05-14
ogImage: "https://catalog.jan.ai/docs/jan-v0-5-17-gemm3-patch.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title="Qwen3 support is now more reliable" date="2025-05-14" ogImage="https://catalog.jan.ai/docs/jan-v0-5-17-gemm3-patch.gif" />

👋 Jan v0.5.17 is out: Qwen3 support is now more reliable

## Highlights 🎉

- Improved Qwen3 support with cleaner token output
- Clearer install and quickstart docs
- UI polish and bug fixes throughout

Update your Jan or [download the latest](https://jan.ai/).

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.5.17).

---

### Src Pages Changelog 2025 06 19 Jan Ui Revamp (`docs/src/pages/changelog/2025-06-19-jan-ui-revamp.mdx`)

---
title: "Jan v0.6.1 is here: It's a whole new vibe!"
version: 0.6.1
description: "Are you ready for the sexiest UI ever?"
date: 2025-06-19
ogImage: "/assets/images/changelog/jan-v0.6.1-ui-revamp.png"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title="Jan v0.6.1 is here: It's a whole new vibe!" date="2025-06-19" ogImage="/assets/images/changelog/jan-v0.6.1-ui-revamp.png" />

## Highlights 🎉

- Jan's been redesigned to be faster, cleaner, and easier to use.
- You can now create assistants with custom instructions and settings from a dedicated tab.
- You can now use Jan with Menlo's models.

Update your Jan or [download the latest](https://jan.ai/).

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.6.1).

---

### Src Pages Changelog 2025 06 26 Jan Nano Mcp (`docs/src/pages/changelog/2025-06-26-jan-nano-mcp.mdx`)

---
title: "Jan v0.6.3 brings new features and models!"
version: 0.6.3
description: "Unlocking MCP for everyone and bringing our latest model to Jan!"
date: 2025-06-26
ogImage: "https://catalog.jan.ai/docs/jn128.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title="Jan v0.6.3 brings with it MCP and our latest model!" date="2025-06-26" ogImage="https://catalog.jan.ai/docs/jn128.gif" />

## Highlights 🎉

- We have added Model Context Protocol (MCP) support to the stable build of Jan. It needs to be enabled in the General Settings tab.
- Jan now supports Menlo's latest model, Jan-Nano-128k.
- Some hot fixes and improvements.

Update your Jan or [download the latest](https://jan.ai/).

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.6.3).

---

### Src Pages Changelog 2025 07 17 Responsive Ui (`docs/src/pages/changelog/2025-07-17-responsive-ui.mdx`)

---
title: "Jan v0.6.5 brings responsive UI and MCP examples!"
version: 0.6.5
description: "New MCP examples, updated pages, and bug fixes!"
date: 2025-07-17
ogImage: "https://catalog.jan.ai/docs/release_v0_6_5.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title="Jan v0.6.5 brings responsive UI and MCP examples!" date="2025-07-17" ogImage="https://catalog.jan.ai/docs/release_v0_6_5.gif" />

## Highlights 🎉

Jan v0.6.5 brings responsive UI improvements, enhanced model provider management, and better Linux compatibility alongside 
new MCP examples.

- Support responsive UI on Jan
- Rework of Model Providers UI
- Bump version of llama.cpp
- Fix the bug where fetching models from custom provider can cause app to crash
- AppImage can now render on wayland + mesa

Update your Jan or [download the latest](https://jan.ai/).

For more details, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.6.5).

---

### Src Pages Changelog 2025 07 31 Llamacpp Tutorials (`docs/src/pages/changelog/2025-07-31-llamacpp-tutorials.mdx`)

---
title: "Jan v0.6.6: Enhanced llama.cpp integration and smarter model management"
version: 0.6.6
description: "Major llama.cpp improvements, Hugging Face provider support, and refined MCP experience"
date: 2025-07-31
ogImage: "https://catalog.jan.ai/docs/changelog0.6.6.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader title="Jan v0.6.6: Enhanced llama.cpp integration and smarter model management" date="2025-01-31" ogImage="https://catalog.jan.ai/docs/changelog0.6.6.gif" />

## Highlights 🎉

Jan v0.6.6 delivers significant improvements to the llama.cpp backend, introduces Hugging Face as a
built-in provider, and brings smarter model management with auto-unload capabilities. This release
also includes numerous MCP refinements and platform-specific enhancements.

### 🚀 Major llama.cpp Backend Overhaul

We've completely revamped the llama.cpp integration with:
- **Smart Backend Management**: The backend now auto-updates and persists your settings properly
- **Device Detection**: Jan automatically detects available GPUs and hardware capabilities
- **Direct llama.cpp Access**: Models now interface directly with llama.cpp (previously hidden behind Cortex)
- **Automatic Migration**: Your existing models seamlessly move from Cortex to direct llama.cpp management
- **Better Error Handling**: Clear error messages when models fail to load, with actionable solutions
- **Per-Model Overrides**: Configure specific settings for individual models

### 🤗 Hugging Face Cloud Router Integration

Connect to Hugging Face's new cloud inference service:
- Access pre-configured models running on various providers (Fireworks, Together AI, and more)
- Hugging Face handles the routing to the best available provider
- Simplified setup with just your HF token
- Non-deletable provider status to prevent accidental removal
- Note: Direct model ID search in Hub remains available as before

### 🧠 Smarter Model Management

New intelligent features to optimize your system resources:
- **Auto-Unload Old Models**: Automatically free up memory by unloading unused models
- **Persistent Settings**: Your model capabilities and settings now persist across app restarts
- **Zero GPU Layers Support**: Set N-GPU Layers to 0 for CPU-only inference
- **Memory Calculation Improvements**: More accurate memory usage reporting

### 🎯 MCP Refinements

Enhanced MCP experience with:
- Tool approval dialog improvements with scrollable parameters
- Better experimental feature edge case handling
- Fixed tool call button disappearing issue
- JSON editing tooltips for easier configuration
- Auto-focus on "Always Allow" action for smoother workflows

### 📚 New MCP Integration Tutorials

Comprehensive guides for powerful MCP integrations:
- **Canva MCP**: Create and manage designs through natural language - generate logos, presentations, and marketing materials directly from chat
- **Browserbase MCP**: Control cloud browsers with AI - automate web tasks, extract data, and monitor sites without complex scripting
- **Octagon Deep Research MCP**: Access finance-focused research capabilities - analyze markets, investigate companies, and generate investment insights

### 🖥️ Platform-Specific Improvements

**Windows:**
- Fixed terminal windows popping up during model loading
- Better process termination handling
- VCRuntime included in installer for compatibility
- Improved NSIS installer with app running checks

**Linux:**
- AppImage now works properly with newest Tauri version and it went from almost 1GB to less than 200MB
- Better Wayland compatibility

**macOS:**
- Improved build process and artifact naming

### 🎨 UI/UX Enhancements

Quality of life improvements throughout:
- Fixed rename thread dialog showing incorrect thread names
- Assistant instructions now have proper defaults
- Download progress indicators remain visible when scrolling
- Better error pages with clearer messaging
- GPU detection now shows accurate backend information
- Improved clickable areas for better usability

### 🔧 Developer Experience

Behind the scenes improvements:
- New automated QA system using CUA (Computer Use Automation)
- Standardized build process across platforms
- Enhanced error stream handling and parsing
- Better proxy support for the new downloader
- Reasoning format support for advanced models

### 🐛 Bug Fixes

Notable fixes include:
- Factory reset no longer fails with access denied errors
- OpenRouter provider stays selected properly
- Model search in Hub shows latest data only
- Temporary download files are cleaned up on cancel
- Legacy threads no longer appear above new threads
- Fixed encoding issues on various platforms

## Breaking Changes

- Models previously managed by Cortex now interface directly with llama.cpp (automatic migration included)
- Some sampling parameters have been removed from the llama.cpp extension for consistency
- Cortex extension is deprecated in favor of direct llama.cpp integration

## Coming Next

We're working on expanding MCP capabilities, improving model download speeds, and adding more provider
integrations. Stay tuned!

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.6.6).

---

### Src Pages Changelog 2025 08 07 Gpt Oss (`docs/src/pages/changelog/2025-08-07-gpt-oss.mdx`)

---
title: "Jan v0.6.7: OpenAI gpt-oss support and enhanced MCP tutorials"
version: 0.6.7
description: "Full support for OpenAI's open-weight gpt-oss models and new Jupyter MCP integration guide"
date: 2025-08-07
ogImage: "/assets/images/changelog/gpt-oss-serper.png"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'


<ChangelogHeader title="Jan v0.6.7: OpenAI gpt-oss support and enhanced MCP tutorials" date="2025-08-07" ogImage="/assets/images/changelog/gpt-oss-serper.png"/>

## Highlights 🎉

Jan v0.6.7 brings full support for OpenAI's groundbreaking open-weight models - gpt-oss-120b and gpt-oss-20b - along with enhanced MCP documentation and critical bug fixes for reasoning models.

### 🚀 OpenAI gpt-oss Models Now Supported

Jan now fully supports OpenAI's first open-weight language models since GPT-2:

**gpt-oss-120b:**
- 117B total parameters, 5.1B active per token
- Runs efficiently on a single 80GB GPU
- Near-parity with OpenAI o4-mini on reasoning benchmarks
- Exceptional tool use and function calling capabilities

**gpt-oss-20b:**
- 21B total parameters, 3.6B active per token
- Runs on edge devices with just 16GB memory
- Similar performance to OpenAI o3-mini
- Perfect for local inference and rapid iteration

<Callout type="info">
Both models use Mixture-of-Experts (MoE) architecture and support context lengths up to 128k tokens. They come natively quantized in MXFP4 format for efficient memory usage.
</Callout>

### 🎮 GPU Layer Configuration

Due to the models' size, you may need to adjust GPU layers based on your hardware:



Start with default settings and reduce layers if you encounter out-of-memory errors. Each system requires different configurations based on available VRAM.

### 📚 New Jupyter MCP Tutorial

We've added comprehensive documentation for the Jupyter MCP integration:
- Real-time notebook interaction and code execution
- Step-by-step setup with Python environment management
- Example workflows for data analysis and visualization
- Security best practices for code execution
- Performance optimization tips

The tutorial demonstrates how to turn Jan into a capable data science partner that can execute analysis, create visualizations, and iterate based on actual results.

### 🔧 Bug Fixes

Critical fixes for reasoning model support:
- **Fixed reasoning text inclusion**: Reasoning text is no longer incorrectly included in chat completion requests
- **Fixed thinking block display**: gpt-oss thinking blocks now render properly in the UI
- **Fixed React state loop**: Resolved infinite re-render issue with useMediaQuery hook

## Using gpt-oss Models

### Download from Hub

All gpt-oss GGUF variants are available in the Jan Hub. Simply search for "gpt-oss" and choose the quantization that fits your hardware:

### Model Capabilities

Both models excel at:
- **Reasoning tasks**: Competition coding, mathematics, and problem solving
- **Tool use**: Web search, code execution, and function calling
- **CoT reasoning**: Full chain-of-thought visibility for monitoring
- **Structured outputs**: JSON schema enforcement and grammar constraints

### Performance Tips

- **Memory requirements**: gpt-oss-120b needs ~80GB, gpt-oss-20b needs ~16GB
- **GPU layers**: Adjust based on your VRAM (start high, reduce if needed)
- **Context size**: Both models support up to 128k tokens
- **Quantization**: Choose lower quantization for smaller memory footprint

## Coming Next

We're continuing to optimize performance for large models, expand MCP integrations, and improve the overall experience for running cutting-edge open models locally.

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.6.7).

---

### Src Pages Changelog 2025 08 14 General Improvs (`docs/src/pages/changelog/2025-08-14-general-improvs.mdx`)

---
title: "Jan v0.6.8: Engine fixes, new MCP tutorials, and cleaner docs"
version: 0.6.8
description: "Llama.cpp stability upgrades, Linear/Todoist MCP tutorials, new model pages (Lucy, Jan‑v1), and docs reorganization"
date: 2025-08-14
ogImage: "https://catalog.jan.ai/docs/mcplinear2.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader title="Jan v0.6.8: Engine fixes, new MCP tutorials, and cleaner docs" date="2025-08-14" ogImage="https://catalog.jan.ai/docs/mcplinear2.gif" />

## Highlights 🎉

v0.6.8 focuses on stability and real workflows: major llama.cpp hardening, two new MCP productivity tutorials, new model pages, and a cleaner docs structure.


### 🚀 New tutorials & docs

- Linear MCP tutorial: create/update issues, projects, comments, cycles — directly from chat
- Todoist MCP tutorial: add, list, update, complete, and delete tasks via natural language
- New model pages:
  - Lucy (1.7B) — optimized for web_search tool calling
  - Jan‑v1 (4B) — strong SimpleQA (91.1%), solid tool use
- Docs updates:
  - Reorganized landing and Products sections; streamlined QuickStart
  - Ongoing Docs v2 (Astro) migration with handbook, blog, and changelog sections added and then removed

### 🧱 Llama.cpp engine: stability & correctness

- Structured error handling for llama.cpp extension
- Better argument handling, improved model path resolution, clearer error messages
- Device parsing tests; conditional Vulkan support; support for missing CUDA backends
- AVX2 instruction support check (Mac Intel) for MCP
- Server hang on model load — fixed
- Session management & port allocation moved to backend for robustness
- Recommended labels in settings; per‑model Jinja template customization
- Tensor buffer type override support
- “Continuous batching” description corrected

### ✨ UX polish

- Thread sorting fixed; assistant dropdown click reliability improved
- Responsive left panel text color; provider logo blur cleanup
- Show toast on download errors; context size error dialog restored
- Prevent accidental message submit for IME users
- Onboarding loop fixed; GPU detection brought back
- Connected MCP servers status stays in sync after JSON edits

### 🔍 Hub & providers

- Hugging Face token respected for repo search and private README visualization
- Deep links and model details fixed
- Factory reset unblocked; special chars in `modelId` handled
- Feature toggle for auto‑updater respected

### 🧪 CI & housekeeping

- Nightly/PR workflow tweaks; clearer API server logs
- Cleaned unused hardware APIs
- Release workflows updated; docs release paths consolidated

### 🤖 Reasoning model fixes

- gpt‑oss “thinking block” rendering fixed
- Reasoning text no longer included in chat completion requests

## Thanks to new contributors

· @cmppoon · @shmutalov · @B0sh

---

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.6.8).

---

### Src Pages Changelog 2025 08 28 Image Support (`docs/src/pages/changelog/2025-08-28-image-support.mdx`)

---
title: "Jan v0.6.9: Image support, stable MCP, and powerful model tools"
version: 0.6.9
description: "Major multimodal support with image uploads, MCP out of experimental, auto-detect model capabilities, and enhanced tool calling"
date: 2025-08-28
ogImage: "https://catalog.jan.ai/docs/jan-images.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader title="Jan v0.6.9: Image support, stable MCP, and powerful model tools" date="2025-08-28" ogImage="https://catalog.jan.ai/docs/jan-images.gif" />

## Highlights 🎉

v0.6.9 delivers two highly requested features: **image support for multimodal models** and **stable MCP** (no more experimental flags!). Plus intelligent model capability detection, enhanced tool calling, and major UX improvements.

### 🖼️ Multimodal AI is here

**Image upload support** — Finally! Upload images directly in your chats and get visual understanding from compatible models:
- **Local models**: Automatic support for Gemma3, Qwen3, and other vision-capable models
- **Cloud models**: Manual toggle for GPT-4V, Claude 3.5 Sonnet, Gemini Pro Vision
- **Smart detection**: Jan automatically identifies which models support images
- **Privacy first**: Local models process images entirely offline

The team has been working hard to bring this long-awaited capability to Jan, and it's been worth the wait.

### 🔧 MCP is stable

**Model Context Protocol** graduated from experimental to stable:
- **No more experimental flags** — MCP tools work out of the box
- **Better error handling** for smoother MCP server connections
- **Cancel tool calls** mid-execution when needed
- **Enhanced reliability** with improved server status tracking

MCP unlocks powerful workflows: web search, code execution, productivity integrations, and more.

### 🎯 Intelligent model management

- **Auto-detect capabilities**: Jan identifies tool calling and vision support for local models
- **Model compatibility checker**: Hub shows hardware requirements before download
- **Favorite models**: Mark frequently used models for quick access
- **Universal GGUF import**: Import any valid .gguf file regardless of extension
- **Hardware-aware suggestions**: Get model recommendations based on your system specs

### 🚀 Enhanced tool calling

- **GPT-OSS models**: Upgraded llama.cpp brings tool calling to more open-source models
- **Improved performance**: Better tool execution with upgraded engine
- **Remote provider control**: Manual toggle for cloud model capabilities
- **Streamlined workflows**: Cancel operations, better error handling

## Major Features

### 🖼️ Multimodal support
- **Image uploads** work with compatible local and cloud models
- **Automatic detection** for local model vision capabilities
- **Manual toggle** for remote providers (GPT-4V, Claude, Gemini)
- **Privacy-preserving** local image processing

### 🔧 Stable MCP
- **MCP Server stable release** — no more experimental flags needed
- **Enhanced error handling** for MCP connections
- **Tool cancellation** support during execution
- **Improved server status** synchronization

### 🎯 Smart model features
- **Favorite models** — bookmark your go-to models
- **Auto-capability detection** for local models
- **Hardware compatibility** checks in Hub
- **Universal GGUF import** regardless of file extension

### ⚡ Enhanced engine
- **Tool calling support** for GPT-OSS models
- **Upgraded llama.cpp** version with stability improvements
- **Better performance** across model types

## Improvements

### 🔄 API & automation
- **Auto-start API server** on Jan startup (optional)
- **Model auto-loading** when API server starts
- **Ollama endpoint** support restored for custom configurations

### 🎨 User experience
- **Thinking windows** for OpenRouter models render correctly
- **Better error messages** across MCP operations
- **Improved import UX** with retired model cleanup
- **CPU architecture** detection at runtime

### 🔧 Technical enhancements
- **Vulkan backend** re-enabled for integrated GPUs with sufficient memory
- **Enhanced stability** with better error handling
- **Performance optimizations** across the board

## Thanks to our incredible team

The engineering team delivered major features that users have been requesting for months. Image support required deep multimodal AI integration, while stabilizing MCP involved extensive testing and refinement. The auto-detection features showcase thoughtful UX design that makes AI more accessible.

Special recognition to the contributors who made v0.6.9 possible through their dedication to bringing powerful, privacy-focused AI capabilities to everyone.

---

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.6.9).

---

### Src Pages Changelog 2025 09 18 Auto Optimize Vision Imports (`docs/src/pages/changelog/2025-09-18-auto-optimize-vision-imports.mdx`)

---
title: "Jan v0.6.10: Auto Optimize, custom backends, and vision model imports"
version: 0.6.10
description: "New experimental Auto Optimize feature, custom llama.cpp backend support, vision model imports, and critical bug fixes"
date: 2025-09-18
ogImage: "/assets/images/changelog/jan-v0.6.10-auto-optimize.gif"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader title="Jan v0.6.10: Auto Optimize, custom backends, and vision model imports" date="2025-09-18" ogImage="/assets/images/changelog/jan-v0.6.10-auto-optimize.gif" />

## Highlights 🎉

- **Auto Optimize**: One-click hardware-aware performance tuning for llama.cpp.
- **Custom Backend Support**: Import and manage your preferred llama.cpp versions.
- **Import Vision Models**: Seamlessly import and use vision-capable models.

### 🚀 Auto Optimize (Experimental)

**Intelligent performance tuning** — Jan can now apply the best llama.cpp settings for your specific hardware:
- **Hardware analysis**: Automatically detects your CPU, GPU, and memory configuration
- **One-click optimization**: Applies optimal parameters with a single click in model settings

<Callout type="info">
Auto Optimize is currently experimental and will be refined based on user feedback. It analyzes your system specs and applies proven configurations for optimal llama.cpp performance.
</Callout>

### 👁️ Vision Model Imports

<img src="/assets/images/changelog/jan-import-vlm-model.gif" alt="Vision Model Import Demo" width="600" />

**Enhanced multimodal support** — Import and use vision models seamlessly:
- **Direct vision model import**: Import vision-capable models from any source
- **Improved compatibility**: Better handling of multimodal model formats

### 🔧 Custom Backend Support

**Import your preferred llama.cpp version** — Full control over your AI backend:
- **Custom llama.cpp versions**: Import and use any llama.cpp build you prefer
- **Version flexibility**: Use bleeding-edge builds or stable releases
- **Backup CDN**: New CDN fallback when GitHub downloads fail
- **User confirmation**: Prompts before auto-updating llama.cpp

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.6.10).

---

### Src Pages Changelog 2025 10 02 Jan Projects (`docs/src/pages/changelog/2025-10-02-jan-projects.mdx`)

---
title: "Jan v0.7.0: Jan Projects"
version: 0.7.0
description: "Jan v0.7.0 introduces Projects, model renaming, llama.cpp auto-tuning, model stats, and Azure support."
date: 2025-10-02
ogImage: "/assets/images/changelog/jan-release-v0.7.0.jpeg"
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader title="Jan v0.7.0" date="2025-10-01" ogImage="/assets/images/changelog/jan-release-v0.7.0.jpeg" />

## Jan v0.7.0: Jan Projects

Jan v0.7.0 is live! This release focuses on helping you organize your workspace and better understand how models run.  

### What’s new
- **Projects**: Group related chats under one project for a cleaner workflow.  
- **Rename models**: Give your models custom names for easier identification.  
- **Model context stats**: See context usage when a model runs.  
- **Auto-loaded cloud models**: Cloud model names now appear automatically.  

---

Update your Jan or [download the latest version](https://jan.ai/).  

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.7.0).

---

### Src Pages Changelog 2025 10 03 Jan Stability Improvements (`docs/src/pages/changelog/2025-10-03-jan-stability-improvements.mdx`)

---
title: "Jan v0.7.1: Fixes Windows Version Revert & OpenRouter Models"
version: 0.7.1
description: "Jan v0.7.1 focuses on bug fixes, including a windows version revert and improvements to OpenRouter models."
date: 2025-10-03
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader title="Jan v0.7.1" date="2025-10-03" />

### Bug Fixes: Windows Version Revert & OpenRouter Models

#### Two quick fixes:
- Jan no longer reverts to an older version on load
- OpenRouter can now add models again
- Add headers for anthropic request to fetch models

---

Update your Jan or [download the latest version](https://jan.ai/). 

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.7.1).

---

### Src Pages Changelog 2025 10 16 Jan Security Update (`docs/src/pages/changelog/2025-10-16-jan-security-update.mdx`)

---
title: "Jan v0.7.2: Security Update"
version: 0.7.2
description: "Jan v0.7.2 updates the happy-dom dependency to v20.0.0 to address a recently disclosed sandbox vulnerability."
date: 2025-10-16
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader title="Jan v0.7.2" date="2025-10-16" />

## Jan v0.7.2: Security Update (happy-dom v20)

This release focuses on **security and stability improvements**.  
It updates the `happy-dom` dependency to the latest version to address a recently disclosed vulnerability.

### Security Fix
- Updated `happy-dom` to **^20.0.0**, preventing untrusted JavaScript executed within HAPPY DOM from accessing process-level functions and executing arbitrary code outside the intended sandbox.

---

Update your Jan or [download the latest version](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.7.2).

---

### Src Pages Changelog 2025 11 13 Jan Browser Mcp (`docs/src/pages/changelog/2025-11-13-jan-browser-mcp.mdx`)

---
title: "Jan v0.7.3: Enhanced Browser Control & Smarter Interactions"
version: 0.7.3
description: "Jan v0.7.3 introduces powerful browser automation with Jan Browser MCP, proactive AI assistance, and refined UX improvements for a smoother experience."
date: 2025-11-13
ogImage: "https://catalog.jan.ai/docs/jan-browser-mcp.gif"
---


import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader
  title="Jan v0.7.3"
  date="2025-11-13"
  ogImage="https://catalog.jan.ai/docs/jan-browser-mcp.gif"
/>

# Highlights 🎉

🚀 **Jan Browser MCP: Control Your Browser with AI**
- Introducing the Jan Browser MCP server - enabling AI-powered browser automation directly from Jan
- Search the web, navigate pages, take screenshots, and extract content through natural conversation
- Seamlessly integrated with OpenAI-compatible models for intelligent web interactions
- Perfect for research, testing, and automated workflows

✨ **Proactive Mode (Experimental)**
- Jan now anticipates your needs by automatically capturing relevant context during browser tool usage
- Visual snapshots and page content are intelligently gathered to enhance AI responses
- Enable in settings to experience context-aware assistance that learns from your browsing

🔍 **Configurable Connection Timeouts**
- Fine-tune llama.cpp connection timeouts to match your hardware and network conditions
- Improved stability for slower systems or complex model loading scenarios
- Adjustable settings in Advanced configuration

🧪 **Refined Scroll Behavior**
- Two distinct scroll modes with proper settings control:
  - **Smooth scrolling**: Elegant, animated transitions for comfortable reading
  - **Instant scrolling**: Snappy navigation for power users
- Configure in Interface settings to match your preference


## Thanks to new contributors

@locnguyen1986 * @utenadev * @vabatista * @fredatgithub * @Kuzmich55 * @volodya-lombrozo

---

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.7.3)

---

### Src Pages Changelog 2025 12 05 Jan Onboarding Attachments (`docs/src/pages/changelog/2025-12-05-jan-onboarding-attachments.mdx`)

---
title: "Jan v0.7.4: Streamlined Onboarding & Enhanced File Attachments"
version: 0.7.4
description: "Jan v0.7.4 delivers a simplified onboarding experience, powerful file attachment improvements, and smarter browser integration for seamless AI interactions."
date: 2025-12-05
ogImage: "https://catalog.jan.ai/docs/jan-onboarding.gif"
---


import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader
  title="Jan v0.7.4"
  date="2025-12-05"
  ogImage="https://catalog.jan.ai/docs/jan-onboarding.gif"
/>

# Highlights

**Simplified Onboarding Flow**
- Completely redesigned first-time user experience for faster setup
- Quick start with Jan's recommended model - get chatting in seconds
- Streamlined extension installation with clear instructions
- Background model preparation so you can explore while Jan gets ready

**Enhanced File Attachments**
- Desktop file attachments now fully enabled for richer conversations
- Improved attachment handling with graceful error recovery
- Re-submit attachments seamlessly when needed
- Better file parsing for more accurate context extraction

**Smarter Browser Integration**
- Jan Browser MCP now available as a chat input shortcut for instant web access
- New dialog to easily install the Jan Browser extension
- Updated link to Chrome Web Store for quick extension setup
- Proactive mode moved to MCP settings for better organization

**CUDA 13 Support**
- Added CUDA 13 backend support for latest NVIDIA GPUs
- CUDA 13 backend prioritized for improved performance on compatible hardware

**Stability & Performance**
- Fixed MCP server memory allocation errors with JIT compilation
- Improved cleanup functions to ensure proper app exit


## Thanks to contributors

@github-roushan * @fredatgithub * @Lokimorty * @trunghaiy

---

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.7.4)

---

### Src Pages Changelog 2025 12 08 Jan Hotfix (`docs/src/pages/changelog/2025-12-08-jan-hotfix.mdx`)

---
title: "Jan v0.7.5: Hotfix Release"
version: 0.7.5
description: "Jan v0.7.5 is a hotfix release that addresses model import issues on Windows, vision model compatibility, CUDA backend installation, and MCP server cleanup."
date: 2025-12-08
---


import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader
  title="Jan v0.7.5"
  date="2025-12-08"
/>

# Highlights

**Bug Fixes**

- Fixed model import issue on Windows
- Fixed users couldn't import the mmproj vision model if the base name didn't match
- Corrected CUDA backend installation and migration issues
- Fixed MCP server not being killed on app termination

---

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.7.5)

---

### Src Pages Changelog 2026 01 27 Jan Unified Ui Anthropic Api (`docs/src/pages/changelog/2026-01-27-jan-unified-ui-anthropic-api.mdx`)

---
title: "Jan v0.7.6: Unified Interface & Jan V3 Model"
version: 0.7.6
description: "Jan v0.7.6 delivers a completely reworked chat interface, plus introduces the Jan V3 model in onboarding."
date: 2026-01-27
---


import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader
  title="Jan v0.7.6: Unified Interface & Jan V3 Model"
  date="2026-01-27"
/>

# Highlights

**Unified Chat Interface**
- Completely reworked chat screen for a consistent experience
- New search dialog for quick navigation across conversations

**Jan V3 Model Introduction**
- Jan V3 model now featured prominently in onboarding experience
- Quick start with Jan's recommended V3 model
- Quick start with Jan's VL model - get chatting with images
- Streamlined first-time user experience for faster setup

**Anthropic Messages API Support**
- Proxy to llama.cpp now supports Anthropic Messages API format
- Use Claude-compatible tools and workflows with local models

---

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.7.6)

---

### Src Pages Changelog 2026 02 11 Jan Mlx Ui Refresh (`docs/src/pages/changelog/2026-02-11-jan-mlx-ui-refresh.mdx`)

---
title: "Jan v0.7.7: Native MLX Support & UI Refresh"
version: 0.7.7
description: "Jan v0.7.7 delivers native MLX support for macOS, app-wide UI refresh, file uploads in Projects, and improved API server capabilities."
date: 2026-02-11
---


import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader
  title="Jan v0.7.7: Native MLX Support & UI Refresh"
  date="2026-02-11"
/>

# Highlights

**Native MLX Support on macOS**
- Adds MLX as an accelerated backend option for Apple silicon
- Expands backend support for local inference on macOS

**App-wide UX and UI Refresh**
- Refreshed UX and UI across the app, beyond the chat screen
- Improved layout, navigation, and consistency across key surfaces

**Projects: File Upload Support**
- Upload files directly into Projects
- Work with documents and files alongside your project chats

**Local API Server Improvements**
- Local API server now supports both local and remote models
- More flexible setup for integrations and developer workflows

**Improved CLI Compatibility**
- Better compatibility with Claude Code and other CLI tools
- Smoother local workflows when connecting Jan to external tools

---

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.7.7)

---

### Src Pages Changelog 2026 03 11 Jan Openclaw Cli Context (`docs/src/pages/changelog/2026-03-11-jan-openclaw-cli-context.mdx`)

---
title: "Jan v0.7.8: OpenClaw Integration, Jan CLI & Auto Context Management"
version: 0.7.8
description: "Jan v0.7.8 brings OpenClaw integration, a brand-new Jan CLI, and automatic context management so you never get cut off mid-conversation."
date: 2026-03-11
---


import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader
  title="Jan v0.7.8: OpenClaw Integration, Jan CLI & Auto Context Management"
  date="2026-03-11"
/>

# Highlights

**OpenClaw Integration**
- Integrate OpenClaw with a single click — run it locally with your favorite model, fully private
- Safe, sandboxed environment setup via Docker
- New Agent Chat in Jan — ask your agent to work on tasks directly from the app
- Connect to Telegram and WhatsApp through a simple setup

**Jan CLI**
- Run Jan headless from the terminal — no desktop app required
- Serve models and launch your favorite agents like Claude Code and OpenClaw from the command line

**Automatic Context Management**
- Jan now automatically manages context size so you never get cut off mid-conversation
- Context window is dynamically adjusted to keep conversations flowing without manual tuning

---

## Thanks to Contributors

@since-2017-hub, @statxc, @dev-miro26, @sesajad, @eren-karakus0, @mykh-hailo, @jamesdam

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.7.8)

---

### Src Pages Changelog 2026 03 23 Jan Remote Models Context Cap (`docs/src/pages/changelog/2026-03-23-jan-remote-models-context-cap.mdx`)

---
title: "Jan v0.7.9: CLI on Windows, Smarter Context Management & Safer Data Location Management"
version: 0.7.9
description: "Jan v0.7.9 fetches the latest models during onboarding, caps context length to avoid high RAM usage, fixes CLI on Windows, and improves data location management."
date: 2026-03-23
---


import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
import { Callout } from 'nextra/components'

<ChangelogHeader
  title="Jan v0.7.9: CLI on Windows, Smarter Context Management & Safer Data Location Management"
  date="2026-03-23"
/>

# Highlights

**Remove OpenClaw Integration**

We recognize that the OpenClaw integration has been causing issues for some users. OpenClaw is a popular agent that has received significant attention from the community. However, it is a resource-heavy product that consumes a large amount of memory and tokens, and can introduce critical security vulnerabilities if not carefully handled in a local AI environment. Therefore, we have decided to remove OpenClaw from Jan and shift our focus toward exploring and building a simpler, safer, and more practical agent experience for Jan users in the near future.

- Jan now remotely fetches the latest models during onboarding, ready for upcoming new Jan models
- Context length is now automatically capped and increased reasonably, avoiding excessively large context sizes that caused high RAM usage
- Fixed CLI path installation on Windows
- Fixed changing Jan data location
- Safely remove Jan data folder with a registered list

---

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.7.9)

---

### Src Pages Changelog 2026 05 22 Jan V0.8.0 (`docs/src/pages/changelog/2026-05-22-jan-v0.8.0.mdx`)

---
title: "Jan v0.8.0: Multi-Token Prediction, llama.cpp Router Mode & Inline MCP Approval"
version: 0.8.0
description: "Jan v0.8.0 ships Multi-Token Prediction for compatible llama.cpp models, a unified llama.cpp router process, inline MCP tool approval with citation cards, per-model fit labels, bulk model deletion, per-thread activity indicators, and a backend dependency checker."
date: 2026-05-22
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader
  title="Jan v0.8.0: Multi-Token Prediction, llama.cpp Router Mode & Inline MCP Approval"
  date="2026-05-22"
/>

# Highlights

**Multi-Token Prediction (MTP)**

Jan now supports llama.cpp's Multi-Token Prediction for compatible models (e.g. GLM 4.5/4.6 and other architectures with MTP heads). Jan detects MTP-capable models from GGUF metadata (`{arch}.nextn_predict_layers`) at import time and exposes a per-model toggle plus draft tunables (`spec-draft-n-max`, `spec-draft-n-min`, `spec-draft-p-min`) in Model Settings. When enabled, `spec-type = draft-mtp` is emitted into the router preset, letting the model draft multiple tokens per step for faster generation. Requires the bundled llama.cpp build b9193 or newer; older backends disable the toggle with an upgrade hint.

**llama.cpp Router Mode**

Jan's local inference engine now runs as a single unified router process instead of spawning a separate server for every model. The router loads and unloads models on demand, so switching between them is faster and uses less memory. This release also adds:

- A reasoning toggle (brain icon in the chat input) to switch thinking mode — auto / on / off — per model without reloading; only available when using the llama.cpp provider
- Live prompt-processing progress so you see a percentage while the model processes your input rather than a bare spinner
- A "Loading model…" indicator when the router cold-starts a model
- A dialog prompting you to confirm or cancel when you close Jan while a model is still working

**Inline MCP Tool Approval & Citation Cards**

MCP tool calls no longer interrupt the conversation with a blocking dialog. Approval panels now appear inline inside each tool card, showing the exact arguments before you accept or deny. RAG results are displayed as numbered citation cards with source previews inside the tool output, and assistant responses include superscript markers linking back to each matched source (cosine similarity ≥ 0.65). Web search results show citation cards in the tool output but do not inject superscript markers into the response text.

**Model Fit Labels & Bulk Delete**

The Hub and provider model lists now show a colored fit pill — **Fits**, **May be slow**, or **Won't fit** — based on your hardware, without downloading anything. Model quantizations are grouped as **Small**, **Balanced**, or **Large** with a Recommended tag on the default download. A new **Delete All** button removes every managed model download at once and shows the total disk space to be freed (imported models are left untouched). Failed downloads no longer get stuck — they are cleared from the queue and a toast is shown. To retry, restart the download from the Hub or provider list.

**Provider List Redesign**

The provider list in Settings is now split into **Local** (llama.cpp, MLX) and **Remote** (OpenAI, Anthropic, Google, Groq, and others) sections so it is immediately clear which providers run on your device and which send data to the cloud.

**Per-Thread Activity Indicators & Navigation**

The thread sidebar now shows a small spinner next to each thread that is actively generating, loading a model, or running tools. Navigating away from an active thread no longer interrupts it — the work keeps running and the UI restores the correct state when you return. Deleting a thread properly stops all in-flight activity.

**Audio Attachments**

You can now attach audio files (WAV, MP3) directly in the chat input alongside images and documents. The **Add Audio** option appears when the active model supports audio input. Attached files show a preview chip with the filename and duration before sending.

**File Attachment Progress Bar**

File uploads now display a progress bar so you can see how much has transferred before sending your message.

**Backend Dependency Checker**

After installing a llama.cpp backend, Jan scans for required libraries (CUDA, Vulkan, cuDNN) and displays a checklist of anything missing with links to the official installers.

**KV Cache Default Reverted to f16**

The default KV cache type was temporarily changed to `q8_0` in a prior release and has been reverted to `f16` as a safer default. This change is not automatically migrated — if your models fail to load after updating, go to **Settings > llama.cpp > KV Cache K Type / V Type** and set both back to `f16`.

**Vercel AI SDK v6**

OpenAI, Anthropic, and Gemini model lists are now auto-populated when you enter an API key, with capabilities (tools, vision, audio) inferred from model IDs. Fallback API keys persist across restarts. Gemini 3.0 models route through Google's native SDK for correct tool-calling behavior. Error messages for context overflows and template errors are clearer and more actionable.

**Server-Side MCP Tool Execution**

A new `/v1/orchestrations` endpoint runs MCP tool calls server-side on `localhost:1337`, and a Settings toggle lets you enable the same orchestrator on `/v1/chat/completions`. An optional lightweight router model can be configured to pre-select which MCP servers a request needs, keeping the tool list short for the main model.

**Message Queue & Editable Pending Chips**

You can now keep typing while a response is streaming. Queued messages appear as editable chips in the input area, send automatically as soon as the current turn finishes, and the Stop button uses a two-stage interaction so you can cancel either the queue or the active generation.

**Auto-Summarized Thread Titles**

After your first response, Jan summarizes the thread title in the background using a cheap inference pass that can be cancelled if you rename the thread yourself.

**Chain of Thought, Regenerate & Edit Improvements**

Failed assistant responses now show a regenerate button instead of leaving the conversation stuck. A new Chain of Thought rendering surfaces step-by-step reasoning more cleanly, and you can scroll through streaming thinking content without being auto-interrupted.

**Settings, Assistants & UX**

- Settings navigation has been reorganized
- A **Default Assistant** setting and a chat-bar assistant picker (shown only when you have more than one assistant) make it easier to switch personas
- Audio, vision, and document attachment options are now hidden when the active model doesn't support them
- Document attachments accept many more file extensions
- Toast placement is configurable, and message timestamps render in your local timezone
- Hindi, Korean, and Catalan locales were added; existing locales were updated for the new strings
- Up Arrow in an empty input restores your last sent message
- Factory Reset offers options to preserve user data
- All user settings are now stored in local files instead of browser localStorage, preventing data loss on reload

**Remote Provider Improvements**

- **MiniMax** has been added as a built-in cloud provider, defaulting to M2.7
- Configurable token limits and auto-compact behavior for remote providers
- Restored Gemini SDK adapter for MCP tool calls, and Groq no longer breaks on follow-up turns after a reasoning response
- Mistral Magistral models stream through a dedicated adapter that understands their thinking part shape
- Gemma's `<thought>` reasoning tags are recognized by the parser and rendered as a collapsible reasoning block instead of appearing as raw text in the response

**Bug Fixes**

- llama.cpp context meter now accounts for tool-heavy chats, and `ctx_size` overflow no longer breaks reloading a chat
- CUDA backend is skipped when the GPU has insufficient memory, and CUDA library paths are no longer overwritten by other library setup
- Flatpak builds discover CUDA libraries and GL extensions for NVIDIA GPUs
- PDF parser panics are caught and a file size limit is enforced on uploads
- Stdio MCP transports verify `tools/list` before signalling ready, and reconnect automatically on disconnect; tool parameter schemas missing `type` are normalized for strict providers
- The MCP startup error dialog is shown again when servers fail to launch
- Internal RAG tools used for embedded documents are auto-approved
- Project file ingestion progress now persists across navigation, file upload errors are surfaced in plain language, and long URLs no longer overflow the project view
- The Hub fit badge and MLX download UI no longer get stuck after a failed download
- Spell check setting is honored in the Assistant editor's Instructions and Description fields
- Assistant identity preamble is stripped on upgrade from the older v2 default prompt
- Jan CLI install detection now works even when the install destination isn't on the app's PATH
- App config persists in the Tauri app data directory, and Windows extended-length paths open correctly in File Explorer
- Left sidebar maximum width reduced so content no longer cramps
- Chat input and new-chat buttons are disabled during engine downloads

# Known Issues

- **KV cache type is no longer auto-migrated.** A prior release temporarily defaulted the llama.cpp KV cache to `q8_0`, which broke loading for some models that require flash attention. v0.8.0 reverts the default to `f16` and removes the previous `f16 → q8_0` migration code. Existing installs are **not** migrated automatically — if a model fails to load after updating, go to **Settings > llama.cpp > KV Cache K Type / V Type** and switch both back to `f16` manually.
- **Migration glitches when upgrading from an older release.** If the app behaves oddly after upgrade (settings not applying, model lists looking wrong, etc.), open **Settings > General > Factory Reset** and run the reset with **Keep models and configs** enabled. This clears stale app state without deleting your downloaded models or provider configuration.
- **MLX support is experimental.** The MLX engine is still under active development on Apple Silicon and may exhibit instability or compatibility gaps compared to the llama.cpp engine.

---

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.8.0)

---

### Src Pages Changelog 2026 05 29 Jan V0.8.1 (`docs/src/pages/changelog/2026-05-29-jan-v0.8.1.mdx`)

---
title: "Jan v0.8.1: Anthropic-Compatible Custom Providers, Per-Message Errors & llama.cpp Settings Overhaul"
version: 0.8.1
description: "Jan v0.8.1 adds Anthropic-compatible custom providers, OS-native TLS trust, a composer-anchored sampler popover with capability gating, persistent per-thread and per-message errors, and a major llama.cpp settings overhaul with split version/backend selectors, file-backed persistence, and per-model context sizing."
date: 2026-05-29
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader
  title="Jan v0.8.1: Anthropic-Compatible Custom Providers, Per-Message Errors & llama.cpp Settings Overhaul"
  date="2026-05-29"
/>

# Highlights

**Anthropic-Compatible Custom Providers**

You can now point Jan at any endpoint that speaks the Anthropic Messages API wire format, including LiteLLM, Bedrock proxies, and self-hosted Claude gateways. The Add Provider dialog gained an API-format selector (OpenAI or Anthropic), and the built-in Anthropic provider is backfilled by an automatic store migration. The dialog also now requires an API key up front, matching backend gates that previously refused to register or list models for key-less providers.

**OS-Native TLS Trust**

Desktop HTTP clients now use the operating system's native trust store (macOS Keychain, Linux `/etc/ssl/certs`, Windows certificate store) instead of a bundled Mozilla CA list. HTTPS endpoints signed by private or corporate CAs (homelab vLLM, internal LiteLLM gateways, etc.) now work without manual certificate wrangling.

**llama.cpp Settings Overhaul**

The llama.cpp engine's configuration surface has been rewritten:

- The composite `version_backend` setting is split into independent `llamacpp_version` and `llamacpp_backend` selectors, with an automatic migration from prior values.
- Backend settings now live in `<jan_data>/llamacpp/settings.json` with atomic writes, so they survive localStorage wipes and can be inspected or edited as a file.
- A new `check_for_updates` toggle (on by default) gates background release fetches. `auto_update_engine` now requires it. The manual "Check for Updates" button always hits the network.
- Backend dependency verification gained a user-facing toggle (on by default) and is scoped to GPU backend libraries (CUDA, Vulkan) only, eliminating noisy false positives from probing `llama-server` itself. Verification is auto-skipped on Linux Flatpak builds where sandbox layout made the checker unreliable.
- Recommended backend hints are now computed only from upstream releases, so side-loaded "Install from File" backends no longer skew the recommendation.
- The macOS "Install from File" picker now accepts `.tar.gz` backend archives.

**Per-Model Context Sizing**

Auto-fit context sizing is off by default on all platforms. Per-model `ctx_len` now owns context sizing and defaults to 8192. A persist migration seeds empty `ctx_len` values, replacing the older v2/v3 fit migration paths.

**Provider-Aware Sampler Popover**

The sampler editor moved out of the model sidebar into a composer-anchored popover scoped to the active assistant. New behavior:

- A categorized "Add parameter" menu, with grouped blocks for coupled samplers (mirostat, dry, xtc, dynatemp).
- A provider/model capability table gates which parameters are exposed, with live mutual-exclusion handling (mirostat shadows top-k / top-p / min-p, Anthropic's temperature and top_p are mutually exclusive, etc.).
- Family-specific rejections are encoded for OpenAI o-series and gpt-5* (which reject temperature, top_p, and penalties), and for grok-3-mini and grok-4 quirks.
- Unsupported sampler keys are stripped at the dispatch chokepoint. If an upstream still rejects a request as a sampling error, Jan retries once with all injected sampling params stripped and shows a toast explaining what happened.
- The sampler UI and stored overrides are now hidden and stripped for predefined remote providers (like Gemini) that reject unknown JSON fields, preventing local-model overrides from leaking into remote requests.
- The popover trigger is disabled while the assistant is still hydrating, so edits cannot land on hardcoded defaults and get clobbered when persisted assistants load from disk.

**Per-Thread and Per-Message Error Persistence**

OOM and backend error banners now persist per-thread across restart by stamping `metadata.oomError` and `metadata.backendError` onto the last user message. This fixes the lost-banner-on-restart bug and the cross-thread banner leak.

Per-message inline error cards now render under the user message that triggered a failure, with a wired Regenerate button, replacing the old transient global error banner. Successful regeneration strips the prior `metadata.error` so resolved failures stop showing the card, and editing a user message also clears it.

Errored generations no longer persist as empty assistant rows on disk, and existing empty assistant rows from previous versions are cleaned up on thread load. Message persistence now upserts on `modify_message` and dedupes on `create_message`, fixing a JSONL race where error metadata stamped immediately after sending could be silently dropped.

**llama.cpp Error Scoping**

The llama.cpp router error banner, implicit stop, and Reload button now only fire on llama.cpp-backed threads. A router-side Metal crash no longer surfaces on top of working MLX, OpenAI, or Anthropic chats.

**Media-Only Messages**

Multimodal models can now be sent image-only or audio-only messages without requiring placeholder text in the prompt.

**Bug Fixes**

- Updating settings like `ctx_size`, `n_gpu_layers`, and `flash_attn` now correctly triggers a router restart. Previously they were persisted but never applied to the running router until next launch.
- Multimodal VRAM precheck now includes the size of the sibling `mmproj.gguf` projector file, preventing OOM crashes on multimodal models that previously slipped past the gate.
- The router's `models_max` slot accounting caps the embedding bonus at +1, so installed embedders no longer prevent eviction of stale chat models.
- The startup backend update check is skipped when auto-update is disabled, so the BackendUpdater dialog no longer hits the network behind the user's back.
- Imported GGUF models now use the `general.name` field from GGUF metadata for display, falling back to the filename and then to the model id.
- Extension logger formats `Error` and object arguments properly instead of writing `[object Object]` to the log file.
- The MCP auto-reconnect health probe interval was raised from 2 seconds to 30 seconds, cutting probe spam in the log.
- STDIO MCP stderr lines are now routed through the level token the server itself prints (defaulting to INFO), so Python MCP servers no longer flood the log with WARN entries.
- The model picker in MCP and router flows deduplicates entries by provider and id, fixing duplicate React keys when a model appears in both the registry and as a local import.
- Provider refetches no longer carry stale `recommended` or `options` metadata across requests; only the user's selected `value` is preserved.
- Dropdowns with a single option now render as inert plain text without a chevron or popover.
- The model dropdown's compatibility indicator switched from the heavy `isModelSupported` probe (which ran `read_gguf_metadata` on every selection change) to the lightweight `estimateModelFit` heuristic the Hub already uses, and the tooltip is now labeled as an estimate. The context value was removed from the tooltip to reduce noise.
- The `refresh_system_info` Tauri command is now properly registered in the hardware plugin's ACL, fixing a "not allowed by ACL" error fired by the visibility-change handler.
- A single failing extension `onLoad` no longer blocks the entire app from finishing setup. Load failures are isolated, logged by name, and other providers (MLX, OpenAI, Anthropic, etc.) continue to load.
- The llama.cpp extension's `list()` is hardened so a single malformed `model.yml` or unreadable entry can no longer cause Settings to render zero models.
- Soft-deleted local models that are later re-downloaded are scrubbed from the `deletedModels` tombstone list automatically, fixing an empty Settings > llama.cpp view after re-download.
- The ThreadList no longer overwrites a newly created thread's in-memory messages with the empty array fetched from disk, fixing a wipe race on brand-new threads.
- The chat input reasoning toggle was realigned to match its sibling icon buttons.
- Dark mode now applies synchronously on launch, eliminating a brief light-theme flash before the saved theme loads.
- NVIDIA and Vulkan GPU probes are skipped on macOS, where they cannot succeed and produced spurious errors in the hardware view.
- User-message action buttons (edit, copy, delete) now reveal on hover only instead of always being visible.
- `PromptProgress` no longer trips a router invariant when navigating between threads while a load is in flight.
- The Attachments entry was restored to the Settings sidebar; the route had been left reachable only by URL after the recent settings-navigation reorg.

**Localization**

Italian translations were added across assistants, chat interface, common terms, the Hub, the no-logs message, MCP servers, model errors, providers, and settings.

---

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.8.1)

---

### Src Pages Changelog 2026 06 01 Jan V0.8.2 (`docs/src/pages/changelog/2026-06-01-jan-v0.8.2.mdx`)

---
title: "Jan v0.8.2: Faster Startup, AMD ROCm/HIP on Linux & Resumable Downloads"
version: 0.8.2
description: "Jan v0.8.2 adds AMD ROCm/HIP backend support on Linux, a branded phased-startup loader with work deferred past first paint, pause/resume model downloads, a safer default context size, and a batch of chat, provider, and toast fixes."
date: 2026-06-01
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader
  title="Jan v0.8.2: Faster Startup, AMD ROCm/HIP on Linux & Resumable Downloads"
  date="2026-06-01"
/>

# Highlights

**AMD ROCm/HIP Backend (Experimental, Linux)**

Jan now detects AMD GPUs and offers a HIP (ROCm) llama.cpp backend on **Linux**. The new `hip` backend is wired through detection, naming, prioritization (ranked above Vulkan when VRAM is sufficient), download/extract, and dependency verification. Both Jan's own asset format and the official llama.cpp ROCm asset names are folded onto a single canonical backend id, so listing, update checks, and migration treat every HIP variant as one.

> **Experimental:** The entire ROCm/HIP backend is experimental in this release. Only **Linux** is shipped and supported — Windows ROCm/HIP is **not shipped**. If you run into problems, please [report them on GitHub](https://github.com/janhq/jan/issues) so we can stabilize AMD GPU support faster.

**Branded Startup Loader & Faster First Paint**

Startup was reworked so the window paints immediately instead of waiting on extension initialization:

- A branded loader with phased status messages replaces the blank boot screen, with a watchdog against a hung blank window.
- The initial app update check, the llama.cpp backend dependency verification, and the MCP setup are all deferred until after first paint / browser idle, so a cold start is no longer blocked on network probes and version checks.
- The dark theme is applied synchronously on launch (carried over and hardened), and the loader logo/caption spacing was tightened and centered.

**Pause & Resume Model Downloads**

Model downloads can now be paused and resumed directly from the Hub panel, so a large GGUF pull no longer has to restart from zero after an interruption.

**Safer Default Context Size**

When no context size is set and auto-fit is off, the llama.cpp router now defaults `ctx-size` to **8192** instead of letting llama.cpp load the model's full trained context (often 128k+), which could OOM on large-context models. User and per-model values still override this default.

**Automatic Update Check Toggle**

Settings gained a toggle for the automatic app update check. When disabled, Jan no longer hits the network for updates behind the scenes; the manual check still works on demand.

**Provider & Key Management**

- API key inputs now have copy and reveal buttons (a new secret-input component).
- Transport-level request failures (connection refused, DNS, TLS) now surface a friendly, actionable message instead of a raw error.
- MLX models are listed in the Local API Server's default model picker.

**Bug Fixes**

- **Sampler settings now persist across restart.** Thread updates previously rebuilt the assistant from a field whitelist that dropped `parameters`, so sampler overrides (temperature, top_p, top_k, etc.) set in the composer were lost on relaunch. The assistant is now persisted intact.
- **MLX commands fixed on macOS.** The MLX capability was moved into its own macOS-scoped file during the macOS-only build split but never added to the security allowlist, causing a "not allowed by ACL" error when loading MLX models. The capability is now registered.
- **Attachment size limit is honored.** The document parser hardcoded a 20MB cap regardless of the configured "max file size" setting, so a file under the user's limit but over 20MB (e.g. 21.5MB with a 50MB setting) was rejected as too large. The parser backstop now follows the setting's documented maximum, and the frontend remains the source of truth for the configured value.
- Notification toasts are visible and theme-aware again: the previously page-colored (invisible) card is themed via popover tokens and the active theme with an elevation shadow, and neutral toasts pick up the Appearance accent color.
- Toast styling is restored on Windows/WebView2. Sonner's CSS is shipped via the linked app bundle, and the accent tint moved from an inline style prop to a CSS class so WebView2's hashed `style-src` CSP no longer drops it.
- CSP no longer blocks runtime inline styles: Tauri's automatic nonce injection on `style-src` is disabled so `'unsafe-inline'` stays effective, fixing the wave of "inline style violates CSP" errors (Google Fonts, dropdowns, etc.).
- The thread's selected agent is now authoritative, so switching the assistant mid-chat works as expected.
- Prompt-too-big errors are routed to the "Increase Context Size" banner instead of a generic failure.
- Crashed models are recovered and mid-request crashes are surfaced rather than hanging silently.
- The router is started before the backend update check is deferred, avoiding a race where the engine wasn't ready.
- `requestIdleCallback` is bound to `window` to avoid an "Illegal invocation" error in the updater.
- The user message bubble and attachment chip text use the primary color and are legible.
- Linux: tao's draggable Wayland titlebar is preserved and follows the GNOME button-layout setting.

**Localization**

Italian was added to the in-app language switcher, and the Catalan locale code was corrected (`ct` → `ca`).

**Engineering**

- The MLX plugin and extension are built on macOS only, keeping `tauri-plugin-mlx` and its guest-js bindings out of Linux and Windows builds.
- The llama.cpp extension test suite now runs as part of the global `yarn test`.

---

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.8.2)

---

### Src Pages Changelog 2026 06 24 Jan V0.8.3 (`docs/src/pages/changelog/2026-06-24-jan-v0.8.3.mdx`)

---
title: "Jan v0.8.3: Message Branching, Linux Custom Titlebar & HTML/SVG Artifacts"
version: 0.8.3
description: "Jan v0.8.3 brings navigable message version branching, a fully custom Linux titlebar that follows your DE's button layout, a unified reasoning/tool chain-of-thought timeline, interactive HTML and static SVG artifact previews, video input for local vision models, and a sweeping batch of streaming performance, citation, sampler, and error-surfacing fixes."
date: 2026-06-24
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader
  title="Jan v0.8.3: Message Branching, Linux Custom Titlebar & HTML/SVG Artifacts"
  date="2026-06-24"
/>

# Highlights

**Message Version Branching**

Editing a user message, editing an assistant reply, or regenerating no longer destroys the original. Each operation creates a new version, and a `< n/m >` control beneath the affected message lets you navigate all versions. Switching a version re-threads the entire downstream conversation to match. Branches persist across restarts - they are modeled as a parent-pointer tree in message metadata, so no core type or backend change was required.

**Linux: Custom Titlebar Following Your DE's Button Layout**

Jan now draws its own borderless titlebar on Linux instead of relying on GTK3's CSD. GTK3 cannot negotiate server-side decorations on Wayland, so KDE never produced a slim native bar and the old headerbar couldn't be shrunk below the theme floor. The new self-drawn bar:

- Reads your desktop environment's window-button placement (KDE `kwinrc`, GNOME `gsettings`) and puts the controls on the correct side - including when that is the left.
- Refetches the layout on window focus, so a change made while Jan is unfocused takes effect the next time you switch back.
- Restores resize grips at window edges and corners (a borderless Wayland window loses them otherwise).
- Guards against a malformed layout payload that previously crashed the render with `undefined is not an object`.

An infinite render loop that caused the renderer to consume 3-4 GB/min and OOM on idle is also fixed (the root cause was `getCurrentWebviewWindow()` returning a fresh instance on every render, causing the focus-listener effect to re-run continuously).

**Unified Reasoning + Tool Chain-of-Thought Timeline**

Consecutive reasoning, tool calls, and interstitial narration now fold into a single collapsible chain-of-thought block instead of stacking separate "Thought for a few seconds" headers between loose prose. The block header tracks the current step while streaming ("Reasoning..." / "Using tools...") and summarizes after ("Thought for N seconds" / "Worked for N seconds"). The trace stays expanded while a tool awaits your approval so the approval controls are never hidden.

Streaming performance for long reasoning was also substantially improved: reasoning content now renders as plain pre-wrapped text rather than going through the full markdown/Shiki pipeline on every token, and the broader markdown renderer defers updates with `useDeferredValue` and skips the `normalizeLatex` scan mid-stream - together eliminating the main sources of per-token re-render cost.

**HTML & SVG Artifact Previews (Experimental)**

Model-generated HTML and SVG can now be rendered inline rather than displayed as code:

- ` ```html ` blocks render as interactive sandboxed previews (opaque origin, `allow-scripts`, no `allow-same-origin`, network blocked by CSP) with a Code/Preview toggle. Defaults to Preview.
- ` ```svg ` fences and bare `<svg>` blocks render as static previews (scripts disabled).
- Both are gated behind a new **Settings -> Interface** toggle (`renderHtmlArtifacts`, default off, marked Experimental).
- The injected CSP is guaranteed to precede all model markup by always wrapping the document, and SVG extraction correctly skips `<svg>` that appears inside a non-SVG code fence.

**Video Input for Local llama.cpp Vision Models**

You can now attach video files to messages when using a vision-capable llama.cpp model in router mode. Video capability is reconciled from `/props.modalities.video` after the model loads (the only reliable signal for whether the backend was built with `MTMD_VIDEO`), so the attach button only appears for models that genuinely support it. Glibc heap-corruption aborts in the video decode path now surface as a backend-error banner rather than silently reloading the model forever.

> **Requires ffmpeg:** llama.cpp decodes video into frames by shelling out to `ffmpeg`/`ffprobe`, which Jan does not bundle. Video input only works if both are installed and on your system `PATH`; without them, video requests will fail.

> **macOS limitation:** The macOS llama.cpp binaries Jan ships are **not** built with `MTMD_VIDEO`, so video input is unavailable on macOS local models regardless of ffmpeg. Use image or audio input, or a remote provider, on macOS for now.

> **Token cost:** A video is decoded into many frames, so it consumes **substantially more tokens than an image or an audio clip** of the same length. On local models this fills the context window quickly, increases prompt-processing time, and can trigger context-overflow or out-of-memory errors. Keep clips short, prefer a higher context size, and watch the token counter.

**Per-Model Default Sampling Parameters for the Local API Server**

External agents and tools that hit Jan's local API server at `localhost:1337` can now benefit from sampling defaults set in the GUI. From the Model Settings sheet, set temperature, top-k, top-p, min-p, repeat penalty, presence penalty, or frequency penalty as per-model server-side defaults. For llama.cpp, these are emitted into the router preset so every request (chat and API) inherits them; a per-request field still overrides. MLX honors temperature, top-p, and repeat penalty via proxy injection.

**Multi-Token Prediction (MTP) Draft Models**

Models that ship a companion `mtp-*.gguf` (e.g. Gemma 4) now auto-download the best-matching draft alongside the chosen quant and wire it to `llama-server` as a speculative draft. MTP companion variants are hidden from the Hub's variant list so they don't appear as standalone downloads.

**Assistant Switcher + Cmd/Ctrl+J**

A dedicated assistant-switcher button now appears in the chat input area when more than one assistant is available, and pressing **Cmd/Ctrl+J** cycles to the next one. The shortcut is listed in the Settings help screen. Two silent switching bugs were fixed: `setCurrentAssistant` previously no-op'd once a default assistant was set, and `updateCurrentThreadAssistant` dropped the model in local state.

**Corrected Sampler Defaults**

Sampler sliders were pinned to their minimum because model settings seeded each key with an empty string and the `stored ?? default` guard doesn't catch it. The fix resolves empty/null/undefined to the parameter default. Two defaults are also corrected to match llama.cpp upstream: temperature `0.7 -> 0.8`, repeat\_penalty `1.1 -> 1.0`. Embedding models now hide sampler controls entirely, and MLX is restricted to the three keys its server forwards (temperature, top-p, repeat\_penalty).

**Token Counts & ETA on the Prompt-Reading Progress Indicator**

The "Reading..." indicator now shows processed/total tokens and an extrapolated time-to-completion estimate, wrapped in a themed progress bar card.

**Other Highlights**

- **Sidebar thread highlight** - the currently open thread now has a persistent highlight, not just a hover state.
- **Colored user bubble toggle** - a new **Settings -> Interface** option (default on, backward compatible) lets you switch the user message bubble from the accent fill to a neutral style matching assistant messages.
- **Azure base URL** - the Azure provider settings now expose an editable endpoint URL field so you can substitute your resource name without workarounds.
- **RAG attachment limit raised to 100 MB** - the default maximum attachment size for document indexing is raised from 20 MB to 100 MB. Previously set values are preserved.
- **Explicit first-model download** - the setup screen no longer auto-starts a download on mount; you must click Download to begin, giving you a moment to review the selection.

---

# Bug Fixes

**Factory Reset Returns to True First-Run State**

Factory reset now clears webview localStorage across all platforms. Previously the data folder was wiped but localStorage was left intact, keeping a deleted model selected and `hasValidProviders` true - so the setup screen was skipped and stale API key errors appeared on next launch. A cross-platform sentinel approach was adopted: Rust drops a sentinel file before restart; on next boot the renderer prunes the matching localStorage keys before Zustand stores hydrate. A separate file-level webview profile delete handles macOS's `WKWebView`, which stores localStorage outside the app data folder.

**Browser MCP Port Leak Fixed**

The Jan Browser MCP bridge forked a grandchild process that outlived a single-PID kill, keeping the port occupied so re-enabling the MCP server immediately failed. MCP children are now spawned as process-group leaders, teardown signals the whole group (SIGTERM -> poll -> SIGKILL), and a bounded port-free poll replaces fixed settle sleeps. A config key typo (`envs` -> `env`) that silently skipped all cleanup on deactivation is also fixed. An explanatory toast now appears when browser MCP is auto-disabled because you switched to a non-vision model.

**ROCm / HIP Backend Startup on Linux**

ROCm installs into versioned roots (e.g. `/opt/rocm-7.2.0/`) that were not on the default loader path. Jan's dependency scan false-flagged `rocblas`/`hipblas`/`amdhip64` as missing and `LD_LIBRARY_PATH` at launch omitted the ROCm libs, so `--list-devices` found no GPU and the HIP backend couldn't start. Jan now resolves ROCm paths from `$ROCM_PATH`, `$ROCM_HOME`, `$HIP_PATH`, `/opt/rocm`, and globbed `/opt/rocm-*`.

**Citations: Correct Superscripts After Multi-Tool RAG Turns**

Two citation bugs are fixed. First, when a model called `retrieve`/`get_chunks` more than once, `isStreaming` briefly flickered false between calls, triggering a stale grounding computation that then won over the final answer - leaving only a few sentences with superscript markers. The fix uses a monotonic run token so stale in-flight computations are discarded. Second, each `<Citations>` card previously numbered from 1, producing duplicate DOM anchor IDs and broken scroll targets for inline `[n]` markers. A per-part `citationOffset` threads the global index through to each card so every citation continues the sequence correctly.

**Local API Server: Anthropic Prompt Caching & Hung Requests**

Two proxy regressions are fixed:

- Claude Code prepends an `x-anthropic-billing-header` to system prompts that busted Anthropic prompt caching and leaked CLI metadata to the model. The streaming request path bypassed the earlier strip logic; it is now removed at the buffered-body chokepoint in both POST arms.
- The `/chat/completions` proxy re-serializes the request body (normalizing/minifying it) but was forwarding the original `Content-Length` verbatim. Pretty-printed JSON from the OpenAI SDK is longer than the minified form, so `llama-server` blocked waiting for bytes that never arrived. `Content-Length` and `Transfer-Encoding` are now stripped from forwarded headers and re-derived from the attached body. This was a regression since 0.8.0.

**LaTeX: No More Subscript Corruption or KaTeX Warnings**

`normalizeLatex` previously rewrote the whole message string with regex, and the emphasis-flanking fix injected a zero-width space (U+200B) that landed inside LaTeX subscripts (`$x_{i}$`), causing KaTeX to warn about unrecognized Unicode character 8203. The fix adopts a placeholder-protection pipeline: math spans (`$$...$$`, `\[...\]`, `\(...\)`, inline `$...$`) are lifted out as opaque tokens before any prose transforms run, then restored - so no transform can corrupt math content. Bold/italic delimiters sitting between a word character and punctuation (e.g. `home**,**`) also now render correctly.

**Error & Loading State Fixes**

- Failed assistant turns are now hidden behind the active error banner (OOM / backend / context-limit); the banner's action buttons own the recovery flow.
- Backend errors (ggml/native crashes) that occur during an ordinary token stream now surface as a banner. Previously `hasActiveLlamacppRequest` didn't recognize mid-stream as active, so crashes were dropped silently.
- On Reload/regenerate, the old request's terminal callbacks no longer clear the loading state that the new request just set, eliminating the "Loading model..." indicator flash.
- Router OOM/backend-error events fired at launch or idle no longer banner when a different provider (e.g. MLX) is selected.
- The context-overflow error banner now shows the real server message (`request (N tokens) exceeds context (M tokens)`) and the token-count popup is reconciled to the failing request, so you no longer see a comfortable percentage next to an out-of-context error.
- A send-time guard in the transport now throws a clear "no user message to respond to" error when the message window has no genuine user query (e.g. after deleting the only user message or after token eviction drops it), replacing the cryptic Jinja "No user query found in messages" exception from Qwen3.5+ models.

**MLX Fixes**

- Tool-call turns with `content: null` no longer crash MLX's strict JSON decoder with a "couldn't be read" error after any tool use.
- The STDIO MCP server handshake now retries with system `npx`/`uvx` when Jan's bundled `bun x`/`uv tool run` override causes an EPIPE on the initialize round-trip. MLX error bodies are now routed through the shared `createCustomFetch` path so they are decoded correctly rather than surfacing as generic WebKit stream errors.

**Other Bug Fixes**

- The embeddings indicator is rewritten as a higher-contrast info-toned banner with a plain-language explanation; it now survives thread navigation (flag moved to per-thread state in `useAppState`).
- Downloads popover no longer clips at the window edge when the sidebar is collapsed.
- Hub search threshold tightened (0.6 -> 0.3) for more precise results; HuggingFace `author/model` namespaced IDs are correctly routed to the detail page.
- Pre-install extensions are now bundled in the base `tauri.conf.json` resource list so builds that bypass the platform overlay don't ship without extensions (previously caused model downloads to hang with no activity).
- Manually renamed thread titles are no longer overwritten by the LLM's auto-title generation.
- The Linux system theme is re-queried from the XDG portal when switching back to Auto, fixing a bug where System -> Light -> System could strand the app on the light theme.
- Invalid `<p>`-in-`<p>` nesting in the delete-project dialog is fixed.
- The thread busy indicator now activates during tool-call approval and execution; the prompt-reading progress bar no longer renders underneath "Loading model...".
- Leaked event subscriptions in `DataProvider` and `useInterfaceSettings` are cleaned up so HMR reloads and effect re-runs don't stack handlers.

---

# Localization

Translations updated.

---

# Engineering

- Workspace-wide `reqwest` bump from 0.11 to 0.12, migrating the local API server proxy to `hyper` 1.0. This clears the `rustls-webpki` advisories RUSTSEC-2026-0098, RUSTSEC-2026-0099, and RUSTSEC-2026-0104 (rustls-webpki 0.101.7, inherited from reqwest 0.11).
- Sub-crate `Cargo.lock` files are now committed and `--locked` is enforced in CI tests, so dependency resolution is deterministic and a broken upstream release can't silently pass CI.
- App and plugin versions bumped to 0.8.3.

---

Update your Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.8.3)

---

### Src Pages Changelog 2026 07 21 Jan V0.8.4 (`docs/src/pages/changelog/2026-07-21-jan-v0.8.4.mdx`)

---
title: "Jan v0.8.4: Native Web Search, Per-Model Chat Templates & a Backend Settings Store"
version: 0.8.4
description: "Jan v0.8.4 adds native web_search / web_fetch tools, per-model chat-template kwargs for llama.cpp, a token counter for remote and MLX providers, and moves settings and credentials into a backend-managed store (localStorage kept as a downgrade snapshot) - alongside a batch of chat, MCP, provider, and streaming fixes."
date: 2026-07-21
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader
  title="Jan v0.8.4: Native Web Search, Per-Model Chat Templates & a Backend Settings Store"
  date="2026-07-21"
/>

# Migration

**Settings and credentials now live in a backend store, not browser localStorage.**

Provider settings and credentials previously persisted in the webview's `localStorage`. In 0.8.4, Jan reads and writes them from a backend-managed store instead (secrets go to the OS keyring). Your existing settings and API keys are copied over automatically on first launch - no manual action is needed, and from this version on all new changes are saved to the backend store.

Your old `localStorage` data is **not deleted** - it's kept as a snapshot so you can downgrade to a pre-0.8.4 build and still have your previous settings. Note that any changes you make in 0.8.4+ live only in the backend store, so a downgrade sees the older snapshot, not your latest settings. The migration runs once and is guarded by a backend flag, so re-upgrading never re-imports the stale `localStorage` copy over newer backend data.

---

# Highlights

**Native Web Search & Web Fetch**

Jan now ships built-in `web_search` and `web_fetch` tools backed by a native `tauri-plugin-websearch`, so models can search the web and fetch page content without configuring an external MCP server. The tools are available to the agent loop like any other built-in.

**Per-Model Chat Template kwargs for llama.cpp**

You can now set chat-template keyword arguments per model in llama.cpp router mode, so template variables (for example reasoning or tool-format switches a model's template exposes) can be pinned per model rather than applied globally.

**Token Counter for Remote & MLX Providers**

The token counter now works for remote and MLX providers, not just local llama.cpp - so you can see token usage during chat across provider types. This landed alongside a batch of related chat and inference fixes.

**Interim Text Folding into the Reasoning Trace**

A new toggle lets you fold interim narration text into the reasoning/chain-of-thought trace instead of showing it as loose prose between steps, keeping the transcript compact.

---

# Bug Fixes

**Chat & Streaming**

- Stopping a generation mid model-load now preserves `AbortError` identity, so the stop is handled cleanly instead of surfacing as an unexpected error.
- Custom SSE event types in OpenAI-compatible streams are now ignored rather than breaking parsing.
- Detached assistant replies in branched threads are no longer persisted, so branch history stays correct.
- Numeric assistant parameters supplied as strings are now coerced to numbers.

**MCP & Local API Server**

- Guarded against a duplicate `serve()` on streamable-HTTP MCP servers.
- The MCP tool-approval popup no longer hangs; transient approval state was moved off disk.
- Providers now send the `anthropic-version` header when fetching the model list.

**RAG & Rendering**

- Embeddings were repaired, redundant chat-model reloads were stopped, and more PDF formats now parse for document indexing.
- Code blocks are now forced left-to-right (LTR) even inside a right-to-left language context.

**Desktop & Platform**

- macOS: hide-on-close now hides to the tray when the Local API Server is running, instead of quitting.
- Fixed a snake_case mismatch for the ADL enum in `get_gpu_usage`.

**Security**

- Bumped `handlebars` to 4.7.9 in docs to clear an RCE advisory.

---

# Localization

Japanese (`ja`) translations were updated across common, settings, and terminology.

---

# Engineering

- App plugin versions bumped to 0.8.4.
- Centralized the llama.cpp extension lookup with a scoped-name fallback.

---

Update Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.8.4).

---

### Src Pages Changelog 2026 10 08 Jan V0.8.5 (`docs/src/pages/changelog/2026-10-08-jan-v0.8.5.mdx`)

---
title: "Jan v0.8.5: llama.cpp Stability, MCP Hardening & Platform Fixes"
version: 0.8.5
description: "Jan v0.8.5 is a fixes release: the llama.cpp engine (now 0.5.0) ships inside the app instead of being downloaded, the jan CLI is no longer bundled, the desktop executable is renamed to Jan-Desktop, the Local API Server enforces Trusted Hosts on 0.0.0.0, and a batch of llama.cpp, MCP, platform and agent fixes. No new features are announced in it."
date: 2026-10-08
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader
  title="Jan v0.8.5: llama.cpp Stability, MCP Hardening & Platform Fixes"
  date="2026-10-08"
/>

# This release is fixes only

No new features are announced in v0.8.5. Work in progress stays in the nightly channel: [Cowork](https://jan.ai/docs/desktop/cowork) and the current agent work ship there until they are announced, and a stable build does not show them.

One thing a first launch will notice: the first-run setup is now a guided wizard - welcome, engine setup, a first model, and the consent page - instead of a single screen. It is onboarding rather than a new capability, so it is not a headline; it is written down here so a new install is not a surprise.

---

# Migration

**The llama.cpp engine now ships with the app.**

Jan no longer downloads a llama.cpp backend for your machine on first run or when one is updated. The engine (llama.cpp 0.5.0, build b11146) is bundled in the installer, runs in its own worker process, and updates together with the app, so the backend version and engine auto-update settings are gone. On Windows and Linux, NVIDIA GPUs use the bundled CUDA 13 build; other GPUs, and NVIDIA GPUs or drivers that cannot run CUDA 13 (Pascal cards such as the GTX 10 series, for example), use Vulkan. (In 0.8.4 such systems could download a CUDA 11 or 12 backend; that option is gone.) macOS uses Metal.

Backends downloaded by earlier versions, in `llamacpp/backends` (and `llamacpp/lib`, if present) inside your Jan data folder, are no longer used and are safe to delete.

**Local models no longer run on Intel Macs.**

Intel-based Macs have not been supported since v0.8.0, and the bundled engine is built for Apple Silicon only. On an Intel Mac, Jan v0.8.5 still opens and works with remote and custom providers, but llama.cpp models will not load. If you rely on local models on an Intel Mac, stay on v0.8.4.

**The desktop executable is now `Jan-Desktop`.**

The app is still called Jan, and its menu entries, shortcuts and `Jan.app` bundle keep working, but the executable inside was renamed. On Linux the `.deb` installs `/usr/bin/Jan-Desktop` instead of `/usr/bin/Jan`, and on macOS it is `Jan.app/Contents/MacOS/Jan-Desktop`; scripts that launch the old path need updating. On Windows, the in-app update moves the Start-menu and desktop shortcuts to the new executable.

**The `jan` CLI is no longer part of the desktop app.**

The desktop app no longer bundles the `jan` command-line agent, installs it, or adds it to your `PATH`; it is now a separate install (see the [agent quickstart](https://jan.ai/docs/agent/quickstart)).

- macOS and Linux: a `jan` that v0.8.4 installed (in `/usr/local/bin` or `~/.local/bin`) is left in place but stays at v0.8.4. Reinstall it from the quickstart to keep it current.
- Windows: updating from v0.8.4 removes the `jan.exe` it put in the app's `resources\bin` folder, so `jan` stops working until you install the CLI from the quickstart. The Windows installer now puts it in `%USERPROFILE%\.local\bin`, the same place as on macOS and Linux, not in the desktop app's folder, so a later app update cannot remove it. The update leaves that folder's entry in your user `PATH`; you can [delete the entry](https://jan.ai/docs/desktop/troubleshooting#jan-is-not-recognized-after-updating-the-desktop-app).

**The Local API Server checks Trusted Hosts when bound to `0.0.0.0`.**

Previously, binding to `0.0.0.0` accepted any `Host` header. Now a request whose host is not `localhost`, `host.docker.internal`, a loopback or private LAN IP, or an entry in **Trusted Hosts** is rejected with `403`. If clients reach Jan by another name - a machine name such as `my-pc.local`, a domain behind a reverse proxy, or a non-private IP - add it to **Settings > Local API Server > Trusted Hosts** (or `*` to allow any host).

**The agent's `bash` tool is now called `shell`.**

On Windows it runs in PowerShell, so Git Bash is no longer needed; on macOS and Linux it is still bash. Agent definitions and tool allowlists that name `bash` stop matching: Jan reports the old name as an error, and renaming it to `shell` fixes it. Past conversations that used `bash` still display normally.

**Settings that change on update**

- llama.cpp **Parallel Sequences**: a value of `1` (the old default) becomes `0`, which lets llama.cpp pick the slot count.
- A model whose `model.yml` carries the `n_gpu_layers: 100` an earlier version wrote has it removed, so GPU offload is decided automatically again.
- llama.cpp **Remember Conversations Between Sessions** is on by default: each conversation's prompt cache is kept on disk in `llamacpp/thread-cache`, capped by **Conversation Cache Disk Limit** (8192 MiB). Turn it off in **Settings > Model Providers > Llama.cpp**.
- Windows and Linux: **Settings > Local API Server** has a **Run in background** toggle, on by default (the previous behavior). Turn it off to make closing the window quit Jan and stop the server.

---

# Bug Fixes

**llama.cpp & Local Models**

- Setting `parallel` no longer halves the per-slot context window.
- Context-shifted chat history is trimmed instead of being sent back whole.
- llama.cpp runs in a separate worker process, so an engine crash no longer takes the app down with it.
- llama.cpp is upgraded to 0.5.0, and per-model chat-template and grammar files can be given as paths.
- Building Jan from source on Fedora and RHEL finds the llama.cpp libraries.

**Chat, Projects & Providers**

- A new chat inside a project uses that project's assistant.
- The predefined Azure provider has a Base URL field, so it can point at your Azure endpoint.
- A message that failed is no longer sent again inside your next one, and a local server (Jan's own engine, Ollama, LM Studio or anything on `localhost`) is not retried after an error, so a failing prompt runs once instead of three times.

**MCP**

- Tool results are capped before they enter conversation history, so a single large result can no longer blow the context window.
- A failing `shell` tool now reports as an error instead of a successful result.

**Desktop & Platform**

- The sidebar's Jan label no longer overlaps the macOS traffic lights.
- Windows: the maximize button turns into Restore while the window is maximized.
- Zoom shortcuts are scoped to chat message text, so zooming a message no longer rescales the whole app.
- Hub search ranks exact model id matches first.
- Log rotation no longer deletes the llama.cpp and worker logs.
- An in-app update stops the running engine, MCP servers and agent shells before the installer runs. If 0.8.4 left its engine running after an update, Jan stops it at the next start.
- The Linux AppImage shows its icon again.
- The Windows installer template substitutes `jan_workspace` correctly.
- On Windows, the Start-menu and desktop shortcuts keep working after the in-app update renames the app executable.

**Agent & Tooling**

- A session heals from malformed or poisoned tool-call history instead of failing every turn after it.
- Session token spend is counted honestly, including marginal spend.
- A compacted turn resumes from where it stopped, and orphan tool rows are retired.
- Compaction retries on llama-server context-overflow responses, and the compaction prompt is appended rather than rewriting the request prefix.
- The `shell` tool works on Windows (in PowerShell, inside the sandbox), and TUI scrollback follows the terminal's scroll wheel.

**Security**

- Binding the Local API Server to `0.0.0.0` now enforces the Trusted Hosts allowlist (GHSA-x6p8-7cp8-c3p6); see Migration above. Thanks to the reporter.
- An "allow always" grant for a shell command now covers only plain invocations of that command. Commands that redefine aliases or functions, change `PATH` or similar variables, or wrap another command in a way the scanner cannot follow ask again instead of running under the grant.

---

# Known issues

- A failed in-app update closes the update dialog without an error; the reason is only in the log ([#9137](https://github.com/janhq/jan/issues/9137)).
- Windows, inside the sandbox the `shell` tool runs in: `git` cannot read the workspace, and tools whose install folder denies access to app containers (Node.js installed from its MSI, for example) cannot start. PowerShell in Constrained Language Mode, which some managed machines enforce, also stops the `shell` tool.
- Ubuntu 24.04: the default AppArmor policy blocks the sandbox the `shell` tool runs in, so Jan does not offer the tool in chat. Everything else works.

---

# Nightly

Cowork and the in-progress agent work are available in the nightly channel:

- macOS (universal): https://app.jan.ai/download/nightly/mac-universal
- Windows (x64): https://app.jan.ai/download/nightly/win-x64
- Linux (x64): https://app.jan.ai/download/nightly/linux-amd64-deb or https://app.jan.ai/download/nightly/linux-amd64-appimage

---

# Engineering

- The desktop end-to-end suite now runs on all four platforms on a nightly schedule, with a mock provider covering the chat critical path.
- Building the engine on Windows no longer fails on the 260-character `MAX_PATH` limit.

---

Update Jan or [download the latest](https://jan.ai/).

For the complete list of changes, see the [GitHub release notes](https://github.com/janhq/jan/releases/tag/v0.8.5).

---

### Src Pages Changelog 2026 10 09 Jan V0.8.6 (`docs/src/pages/changelog/2026-10-09-jan-v0.8.6.mdx`)

---
title: "Jan v0.8.6: AppImage, Turing GPU, MTP Speed & MCP Image Fixes"
version: 0.8.6
description: "Jan v0.8.6 is a hotfix for v0.8.5: the Linux AppImage no longer crashes on model load, NVIDIA Turing GPUs (GTX 16xx, RTX 20xx) run the CUDA engine again on older drivers, Qwen3.8 with MTP is fast again when Parallel Sequences is auto, and images returned by MCP tools reach the model as images."
date: 2026-10-09
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader
  title="Jan v0.8.6: AppImage, Turing GPU, MTP Speed & MCP Image Fixes"
  date="2026-10-09"
/>

# A hotfix for v0.8.5

v0.8.6 fixes regressions reported after v0.8.5. Updating is recommended, especially on Linux with the AppImage and on older NVIDIA GPUs. Nothing changes in your data or settings.

---

# Fixes

**The Linux AppImage crashed on every model load on some distributions.**

The v0.8.5 AppImage bundled the build machine's Vulkan loader, which crashed on hosts with a different driver stack (reported on Arch). The AppImage now uses your system's Vulkan loader, and the build fails if a GPU loader is ever bundled again. ([#9173](https://github.com/janhq/jan/issues/9173))

**NVIDIA Turing GPUs could not run the CUDA engine on older drivers.**

On a GTX 16xx or RTX 20xx with a driver older than the R595 branch, loading a model on Windows or Linux failed with "the provided PTX was compiled with an unsupported toolchain". The bundled CUDA engine now carries native code for Turing, as the v0.8.4 backend did, so these cards no longer depend on the driver compiling it. RTX 30, 40 and 50 cards are unchanged. ([#9185](https://github.com/janhq/jan/issues/9185))

**Qwen3.8 with MTP slowed down sharply after updating to v0.8.5.**

With **Parallel Sequences** on auto (the v0.8.5 default), llama.cpp reserved four slots, and with MTP each slot holds extra state for draft rollback, about 600 MiB on a 27B model. On cards with around 12 GB that pushed the model into shared system memory and cut generation speed by up to 4x. A model with MTP now uses one slot when Parallel Sequences is auto; a value you set yourself is kept. ([#9183](https://github.com/janhq/jan/issues/9183))

**Images returned by MCP tools now reach the model as images.**

A tool that returns an image, such as the filesystem server's `read_media_file`, sent it to the model as base64 text that the model could not read and that could exceed a provider's input limit. It is now sent as an image to vision-capable models on local llama.cpp, OpenAI-compatible providers, Anthropic, Gemini and OpenAI Responses, and models without vision get a short note instead. Trimming a long conversation also counts an image as a fixed cost, so your question is no longer dropped. Thanks to [@cpius](https://github.com/cpius) for the original fix. ([#9188](https://github.com/janhq/jan/pull/9188))

---

# Also

**A documented way to use a different llama.cpp version.**

**Settings > Model Providers > Llama.cpp** now links to [using a different llama.cpp version](https://jan.ai/docs/desktop/remote-models/custom-endpoint#use-a-different-llamacpp-version): run your own `llama-server` and add it as a custom provider. It is a workaround when the bundled engine misbehaves with a particular model.

---

### Src Pages Changelog 2026 10 20 Jan V0.8.7 (`docs/src/pages/changelog/2026-10-20-jan-v0.8.7.mdx`)

---
title: "Jan v0.8.7: Shell Switch, Image Size Limit, Project Threads & Agent Subagents"
version: 0.8.7
description: "Jan v0.8.7 adds a switch to turn the shell tool off in chat and a separate image size limit, keeps project threads in their project when you attach a document, leaves pasted code with @decorators as typed, removes the hosted Exa MCP default, and brings built-in subagents to the Jan agent."
date: 2026-10-20
---

import ChangelogHeader from "@/components/Changelog/ChangelogHeader"

<ChangelogHeader
  title="Jan v0.8.7: Shell Switch, Image Size Limit, Project Threads & Agent Subagents"
  date="2026-10-20"
/>

# A minor update to v0.8.6

v0.8.7 collects fixes reported since v0.8.6, adds two chat settings, and brings a round of improvements to the Jan agent and CLI. Your data and settings carry over; one MCP clean-up runs on first launch (see below).

---

# New

**Turn the shell tool off in chat.**

The chat input's tool menu now has a **Shell** switch. Turned off, the model is no longer given the shell tool or its instructions, while web search, fetch and your MCP tools keep working. It is on by default, and it appears when the shell can run sandboxed and the model supports tools. ([#9222](https://github.com/janhq/jan/issues/9222))

**A separate size limit for images.**

**Settings > Attachments** has a new **Max Image Size (MB)**, next to the document limit it was easy to confuse with. It defaults to 10 MB, as before, and accepts 1 to 100. ([#9312](https://github.com/janhq/jan/pull/9312))

**Built-in subagents for the Jan agent.**

The agent ships with explore, research, review and debug subagents, each of which can use its own model. The explore subagent is read-only by default. ([#9261](https://github.com/janhq/jan/pull/9261))

---

# Fixes

**A thread left its project when you attached a document.**

Once the document was processed, the thread moved out of its project, and stayed out after a restart. It now stays in its project. Thanks to [@enkynakamura](https://github.com/enkynakamura) for the fix. ([#9197](https://github.com/janhq/jan/issues/9197))

**Pasted code with `@` lost its decorators.**

Every `@word` was treated as a file reference, so pasting Python with `@dataclass` or `@app.route` removed the decorators and flattened the indentation. Only an `@` followed by a real file or folder is now treated as a reference; everything else is sent as you typed it. ([#9291](https://github.com/janhq/jan/issues/9291))

**Stop now ends the whole turn.**

Pressing **Stop** while a tool was running or waiting for your approval could leave the turn going. It now ends the turn, including the tool. ([#9241](https://github.com/janhq/jan/pull/9241))

**A web page read by the built-in fetch tool is kept within the model's context window**, so a long page no longer overflows a local model. ([#9283](https://github.com/janhq/jan/pull/9283))

**Chat with a local model failed behind a proxy or VPN.**

Requests from Jan to its own local engine could be sent to a system proxy and fail although the engine was running. They now always go direct; remote providers still use your proxy. ([#9182](https://github.com/janhq/jan/pull/9182))

**The hosted Exa MCP server is removed from existing installs.**

v0.8.3 added a hosted Exa server to the default MCP list. On first launch, v0.8.7 removes that default entry. Servers you added, and an Exa entry you configured with your own key, are kept. ([#9010](https://github.com/janhq/jan/issues/9010))

**Windows: a `^` in a command name could get past shell permissions.**

cmd ignores `^` in a command name, so `cm^d` runs `cmd`. Such a command now always asks for approval. ([#9315](https://github.com/janhq/jan/issues/9315))

**macOS now says why Jan asks for Local Network access**, needed to reach a model server on another machine, such as Ollama on your network. ([#9318](https://github.com/janhq/jan/pull/9318))

---

# Jan agent and CLI

- A sandboxed shell command can ask to run outside the sandbox, with your approval each time ([#9239](https://github.com/janhq/jan/pull/9239)).
- A tool call outside a run's `allowed_tools` is refused ([#9254](https://github.com/janhq/jan/issues/9254)), and a subagent's permission prompt is denied when nobody can answer it ([#9259](https://github.com/janhq/jan/pull/9259)).
- On Windows, edits match files with CRLF line endings, and file names Windows cannot use (such as `con.txt`) are refused ([#9262](https://github.com/janhq/jan/issues/9262), [#9263](https://github.com/janhq/jan/issues/9263)).
- `jan cli mcp add` works on a fresh install ([#9257](https://github.com/janhq/jan/issues/9257)), and an MCP server that never answers is given up on instead of hanging the run ([#9281](https://github.com/janhq/jan/pull/9281)).
- Compaction summaries use your small model when one is set ([#9252](https://github.com/janhq/jan/pull/9252)), and the terminal UI gets search, clearer subagent and background-command rows, and many smaller refinements.

---

# Also

- On NVIDIA GPUs, the engine logs the driver version at start, and a crash caused by a driver too old for the engine says to update the driver.
- A model download stopped by a network error says why in more cases ([#9294](https://github.com/janhq/jan/pull/9294)), and a failed model-list fetch records the address it could not reach in the app log ([#9292](https://github.com/janhq/jan/issues/9292)).
- The sidebar's Jan label is back on Windows and Linux ([#9242](https://github.com/janhq/jan/pull/9242)).

---

# Known issues

- With **Shell** turned off, a model that calls the shell anyway ends its reply instead of continuing without it ([#9319](https://github.com/janhq/jan/issues/9319)).
- **Auto compact** needs **Max Context Tokens** set for remote models; Jan does not detect a remote model's context window yet ([#9317](https://github.com/janhq/jan/issues/9317)).
- Local models do not run on Intel Macs, as in v0.8.5.

---

### Src Pages Handbook Join Us (`docs/src/pages/handbook/join-us.mdx`)

---
title: "How to Join Us?"
description: "Ways to get involved with Jan."
---

# How to Join Us

Jan works because people contribute. The more people who use it, improve it, and share it, the better it gets.

## Join the community

Start in our [Discord](https://discord.gg/FTk2MvZwJH). That's where everything happens. You can:

* Join discussions and follow what we are building
* Share ideas, feedback, and experiments
* Learn from others and help newcomers find their way

## Contribute

Jan is [open source](https://github.com/janhq). You can:  

* Write or review code
* Report bugs or give feedback

## Work with us

Want to build Jan full-time? → [Careers](https://menlo.bamboohr.com/careers/)

## Partner or invest

Believe in what we're building? → [hello@jan.ai](mailto:hello@jan.ai)

## Share Jan

Tell people about us on [X](https://x.com/jandotai), [Threads](https://www.threads.com/@jandotai), [Bluesky](https://bsky.app/profile/jandotai.bsky.social), [LinkedIn](https://www.linkedin.com/company/opensuperintelligence), and [r/askjan](https://www.reddit.com/r/askjan/).

---

### Src Pages Handbook What Jan Is (`docs/src/pages/handbook/what-jan-is.mdx`)

---
title: "What is Jan?"
description: "Jan is a general purpose AI agent that works on its own to get things done."
---

# What is Jan
Jan is both a product suite and a family of models we train in-house:

## Applications:

We're building Jan to work wherever you do. Jan is available on:

* [Desktop](https://www.jan.ai/download)
* [Browser](https://chat.jan.ai/)
* Browser Extension (coming soon)
* Jan Server

Whether you prefer working from your desktop, need quick access through your browser, or want Jan on the go with mobile - use it however makes sense for you.

Over time, we expect most people will use Jan through the browser as we continue improving the web experience. 

## Models:

* Jan-v2: our multimodal agent built for long-horizon tasks
* Jan-v1: web search capabilities
* Ichigo: speech AI for voice interactions

Training our own models means we can optimise for how agents actually work - planning, executing, and completing tasks autonomously over hours, not just answering questions.

This won't be easy. Building agents that work reliably for hours - sometimes 5, sometimes 10 - without supervision is genuinely hard. We're still early. But the problem is real, the technology is ready, and we think this is the right direction.

---

### Src Pages Post Ai For Law (`docs/src/pages/post/ai-for-law.mdx`)

---
title: "Private AI for legal professionals who need confidentiality"
description: "It's possible to use AI without risking client data. Jan helps lawyers save time while keeping clients safe."
tags: AI, ai for law, ai for lawyers, ChatGPT alternative, Jan, local AI, offline AI
categories: guides
date: 2025-09-30
ogImage: assets/images/general/jan-for-ai-law-assistant-chat.jpeg
twitter:
  card: summary_large_image
  site: "@jandotai"
  title: "Private AI for legal professionals who need confidentiality"
  description: "It's possible to use AI without risking client data. Jan helps lawyers save time while keeping clients safe."
  image: assets/images/general/jan-assistants-ai-for-legal.jpeg
---
import { Callout } from 'nextra/components'
import CTABlog from '@/components/Blog/CTA'
import { OpenAIStatusChecker } from '@/components/OpenAIStatusChecker'

# Private AI for legal professionals who need confidentiality

  

Yes, it's possible to use AI in legal work without risking client data.  

<Callout type="warning">  
Client trust depends on privacy. Sending documents into public AI tools risks compliance and reputation.  
</Callout>   

Start by [downloading Jan](/download) and installing the **Jan v1 model**. Once installed, you can create assistants tailored to your practice and keep contracts, case notes, and client files under your control.  

<Callout type="info">  
**Why use Jan for legal tasks**  
- Runs locally on your hardware, no cloud uploads  
- Keeps chats and interactions private  
- Works offline once installed  
- Lets you build assistants for your own workflows  
</Callout>  

---

## Create your assistant

Once Jan is installed with the **Jan v1 model**, onboarding will guide you through downloading and setup.  

Click **Create assistant** to start:  
  
*Create your first assistant in Jan*  

Add an assistant name and prompt:  
  
*Example of a Jan assistant for contract review*  

You can create assistants using specific prompts. Below are examples for common legal workflows.  

---

## Contract review assistant  

AI can help lawyers move faster through long contracts by pointing out what matters most.  

**Prompt for Jan:**  
> You are a contract review assistant.  
> When I paste a contract:  
> - Highlight risky or unusual clauses  
> - Flag ambiguous or missing terms  
> - Summarize the agreement in plain English for a non-lawyer client  
> Format your response with sections: **Risks**, **Ambiguities/Missing**, **Summary**.  
> Do not provide legal advice.  

---

## Drafting assistant  

Use AI to create first drafts of NDAs, service agreements, or client letters. You still refine the output, but AI saves time on boilerplate.  

**Prompt for Jan:**  
> You are a drafting assistant.  
> When asked to draft a legal agreement or client letter:  
> - Produce a professional first version  
> - Use clear, concise language  
> - Leave placeholders like [Party Name], [Date], [Amount] for details  
> - Structure output with headings, numbered clauses, and consistent formatting  
> Do not provide legal advice.  

---

## Case preparation assistant  

Case prep often means reading hundreds of pages. AI can turn depositions, discovery files, or judgments into concise notes.  

  
*Jan chat interface for case preparation — process documents and extract key information*  

**Prompt for Jan:**  
> You are a case preparation assistant.  
> When I provide case materials:  
> - Extract key facts, issues, and arguments  
> - Present them as bullet points under headings: **Facts**, **Issues**, **Arguments**  
> - Keep summaries concise (under 500 words unless I request more)  
> Use plain English, no speculation or legal conclusions.  

---

## Knowledge management assistant  

Law firms accumulate memos, policies, and precedents. AI can help organize and retrieve them quickly.  

**Prompt for Jan:**  
> You are a knowledge management assistant.  
> When I ask questions about internal documents:  
> - Return concise summaries or direct excerpts  
> - Always cite the source (e.g., “Policy Manual, Section 4”)  
> - If not found in provided material, reply “Not found in documents.”  
> Do not invent information.  

---

## Final note  

AI in legal practice is not about replacing lawyers. It’s about handling repetitive tasks safely so you can focus on real decisions.  
With private AI, you gain efficiency without compromising client confidentiality.  

<CTABlog />

---

### Src Pages Post Ai For Teachers (`docs/src/pages/post/ai-for-teachers.mdx`)

---
title: "AI for teachers who care about student privacy"
description: "Use AI in teaching without risking student data. Jan helps teachers plan lessons, grade faster, and communicate with parents."
tags: AI, ai for teachers, ChatGPT alternative, Jan, local AI, offline AI, education
categories: guides
date: 2025-10-01
ogImage: assets/images/general/ai-for-teacher.jpeg
twitter:
  card: summary_large_image
  site: "@jandotai"
  title: "AI for teachers who care about student privacy"
  description: "Use AI in teaching without risking student data. Jan helps teachers plan lessons, grade faster, and communicate with parents."
  image: assets/images/general/ai-for-teacher.jpeg
---
import { Callout } from 'nextra/components'
import CTABlog from '@/components/Blog/CTA'

# AI for teachers who care about student privacy



AI can help teachers handle the work that piles up outside class. It can draft a lesson outline, suggest feedback on essays, or turn notes into a polite parent email. These are the tasks that usually stretch into evenings and weekends.  

<Callout>
Most AI tools like ChatGPT run in the cloud. Sharing lesson plans, student writing, or parent details there risks compliance and trust. 
</Callout> 

That's where Jan comes in: 
- [Download Jan](/download)
- You get the same time-saving help
- Your data never leaves your device.

<video controls>
  <source src="/assets/images/general/jan-ai-for-teacher.mp4" type="video/mp4" />
  Your browser does not support the video tag.
</video>
*See how teachers use Jan for AI-powered lesson planning and grading*  

<Callout type="info">  
**Why use Jan for teaching**  
- Runs locally, no cloud servers  
- Keeps lesson plans and student data private  
- Works offline once installed  
- Lets you build assistants for your daily teaching tasks  
</Callout>  

---

## Create your assistant

Once Jan is installed, click **Create assistant** and add one of the prompts below. Each assistant is for a specific classroom task.



---

## Lesson planning assistant  

AI can draft lesson outlines in minutes. You adapt and refine them for your students.  

**Prompt for Jan:**  
> You are a lesson planning assistant.  
> When I give you a topic or subject:  
> - Suggest a lesson outline with objectives, activities, and discussion questions  
> - Adjust for different grade levels if I specify  
> - Keep plans practical and realistic for a classroom setting  

Example ask: For Grade 6 science on ecosystems. Objectives: define food chains, explain producer/consumer roles. Activity: group poster on an ecosystem. Questions: How would removing one species affect the whole system? 

---

## Grading support assistant  

AI won't replace your judgment, but it can make feedback faster and more consistent.  

**Prompt for Jan:**  
> You are a grading support assistant.  
> When I paste student writing or answers:  
> - Highlight strengths and areas for improvement  
> - Suggest short, constructive feedback I can reuse  
> - Keep tone supportive and professional  
> Do not assign final grades.  

Example: For a history essay. Strength: clear thesis. Improvement: weak evidence. Feedback: "Great thesis and structure. Next time, support your points with specific historical examples."

---

## Parent communication assistant  

Writing parent emails is important but time-consuming.  

**Prompt for Jan:**  
> You are a parent communication assistant.  
> When I give you key points about a student:  
> - Draft a polite and empathetic email to parents  
> - Use clear and professional language  
> - Keep tone supportive, not overly formal  
> Only include details I provide.  

Example: Notes: “Student is falling behind on homework, otherwise engaged in class.” - Output: a short, encouraging message suggesting a check-in at home.

---

## Classroom resources assistant  

Generate quizzes, worksheets, or practice activities at short notice.  

**Prompt for Jan:**  
> You are a classroom resource assistant.  
> When I provide a topic or subject:  
> - Generate sample quiz questions (multiple choice and short answer)  
> - Suggest short practice activities  
> - Provide answer keys separately  
> Keep material age-appropriate for the level I specify.  

Example: For Grade 4 fractions. 5 multiple-choice questions with answer key, plus a quick worksheet with 3 practice problems.  

---

## Getting started  

1. [Download Jan](/download).  
2. Install the Jan model (guided in-app)
3. Create your first assistant using one of the prompts above 
4. Test with non-sensitive examples first
5. Use it in real classroom tasks once you're comfortable

---

## Final note  

AI isn't here to replace teachers. It's here to take repetitive tasks off your plate so you can focus on teaching. With Jan, you can use AI confidently without risking student privacy.  

<CTABlog />

---

### Src Pages Post Benchmarking Nvidia Tensorrt Llm (`docs/src/pages/post/benchmarking-nvidia-tensorrt-llm.mdx`)

---
title: Benchmarking NVIDIA TensorRT-LLM
description: This post compares the performance of TensorRT-LLM and llama.cpp on consumer NVIDIA GPUs, highlighting the trade-offs among speed, resource usage, and convenience.
tags: Nvidia, TensorRT-LLM, llama.cpp, rtx3090, rtx4090, "inference engine"
categories: research
ogImage: assets/images/general/og-throughput-benchmark.png
date: 2024-04-29
---

import { Callout } from 'nextra/components'
import CTABlog from '@/components/Blog/CTA'

# Benchmarking NVIDIA TensorRT-LLM

Jan now supports [NVIDIA TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) in addition to [llama.cpp](https://github.com/ggerganov/llama.cpp), making Jan multi-engine and ultra-fast for users with Nvidia GPUs.

We've been excited for TensorRT-LLM for a while, and [had a lot of fun implementing it](https://github.com/janhq/nitro-tensorrt-llm). As part of the process, we've run some benchmarks, to see how TensorRT-LLM fares on consumer hardware (e.g. [4090s](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/), [3090s](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/)) we commonly see in the [Jan's hardware community](https://discord.com/channels/1107178041848909847/1201834752206974996).

<Callout type="info" >
  **Give it a try!** Jan's TensorRT-LLM extension is available in Jan v0.4.9. We precompiled some TensorRT-LLM models for you to try: `Mistral 7b`, `TinyLlama-1.1b`, `TinyJensen-1.1b` 😂

  Bugs or feedback? Let us know on [GitHub](https://github.com/janhq/jan) or via [Discord](https://discord.com/channels/1107178041848909847/1201832734704795688).
</Callout>

<Callout type="info" >
  **An interesting aside:** Jan actually started out in June 2023 building on [NVIDIA FastTransformer](https://github.com/NVIDIA/FasterTransformer), the precursor library to TensorRT-LLM. TensorRT-LLM was released in September 2023, making it a very young library. We’re excited to see its roadmap develop!
</Callout>

## Key Findings



TensorRT-LLM was: 

- **30-70% faster** than llama.cpp on the same hardware
- **Consumes less memory on consecutive runs** and **marginally more GPU VRAM utilization** than llama.cpp
- **20%+ smaller compiled model sizes** than llama.cpp
- **Less convenient** as models have to be compiled for a specific OS and GPU architecture, vs. llama.cpp's "Compile once, run everywhere" portability
- **Less accessible** as it does not support older-generation NVIDIA GPUs

## Why TensorRT-LLM?

[TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) is Nvidia's open-source inference library that incorporates Nvidia's proprietary optimizations beyond the open-source [cuBLAS](https://developer.nvidia.com/cublas) library. 

As compared to [llama.cpp](https://github.com/ggerganov/llama.cpp), which today dominates Desktop AI as a cross-platform inference engine, TensorRT-LLM is highly optimized for Nvidia GPUs. While llama.cpp compiles models into a [single, generalizable CUDA "backend"](https://github.com/ggerganov/llama.cpp/blob/master/ggml-cuda.cu)  that can run on a wide range of Nvidia GPUs, TensorRT-LLM compiles models into a [GPU-specific execution graph](https://www.baseten.co/blog/high-performance-ml-inference-with-nvidia-tensorrt/) that is highly optimized for that specific GPU's Tensor Cores, CUDA cores, VRAM and memory bandwidth.

TensorRT-LLM is typically used in datacenter-grade GPUs, where it produces a [face-melting 10,000 tokens/s](https://nvidia.github.io/TensorRT-LLM/blogs/H100vsA100.html) on [NVIDIA H100 Tensor Core GPUs](https://www.nvidia.com/en-us/data-center/h100/). We were curious for how TensorRT-LLM performs on consumer-grade GPUs, and gave it a spin.

| Llama.cpp                                                                                                                 | TensorRT-LLM                                                                                                                      |
| ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| Baseline                                                                                                                  | Blazing fast (30-70% faster)                                                                                                      |
| Compile once, run cross-platform                                                                                          | Compiled and highly optimized for specific GPU architecture                                                                       |
| Generalizable and Portable                                                                                                | Highly-optimized                                                                                                                  |
| Model compiles to [single, generalizable CUDA "backend"](https://github.com/ggerganov/llama.cpp/blob/master/ggml-cuda.cu) | Model compiles to [GPU-specific execution graph](https://www.baseten.co/blog/high-performance-ml-inference-with-nvidia-tensorrt/) |

## Experiment Setup

We ran the experiment using standardized inference requests in a sandboxed environment:
- **Model**: Mistral 7b model, compiled and quantized at a comparable `int4` quantization.
- **Test runs**: 5 batches of 10 runs each, per inference engine, on a bare metal PC with no other applications.
- **Parameters**: User defaults, i.e. `batch_size 1`, `input_len 2048` and `output_len 512`
- **Measurements**:
  - CPU, memory from Jan system monitor
  - GPU VRAM utilization metrics from `nvidia-smi`, and taken over an interval of 14 seconds. 
  - Throughput (token/sec) using [Jan's built-in Tokens/sec perf stat](https://github.com/search?q=repo%3Ajanhq%2Fjan%20timeDiffInSeconds&type=code).

<Callout>
We picked a [batch size of 1](https://www.reddit.com/r/LocalLLaMA/comments/17sbwo5/what_does_batch_size_mean_in_inference/) to simulate realistic real-world use cases, as Jan users are likely to make one inference request at a time. We also used a `input_len` of 2048 and `output_len` of 512 to simulate a typical chatbot use case.
</Callout>

<Callout>
**Further Research**: We found GPU VRAM Utilization to be difficult to measure, with `nvidia-smi` on Windows producing a different result from Windows Task Manager's Resource Monitor for NVIDIA GPUs. After some [research](https://forums.developer.nvidia.com/t/how-to-evaluate-gpu-utilization-usage-on-windows/245451/3), we have gone with `nvidia-smi`'s measurement. 
</Callout>

### Hardware Selection

We chose the following GPUs based on our users' preferences:

| NVIDIA GPU                        | VRAM (GB) | CUDA Cores | Tensor Cores | Memory Bus Width (bit) | Memory Bandwidth (GB/s) | Connection (GB/s)                            |
| --------------------------------- | --------- | ---------- | ------------ | ---------------------- | ----------------------- | -------------------------------------------- |
| GeForce RTX 4090 (Ada)            | 24        | 16,384     | 512          | 384                    | ~1000                   | PCIe4.0 x16 (~32)                        |
| GeForce RTX 3090 (Ampere)         | 24        | 10,496     | 328          | 384                    | 935.8                   | PCIe4.0 x16 (~32)                        |
| GeForce RTX 4070 Laptop GPU (Ada) | 8         | 7680       | 144          | 192                    | 272                     | PCIe4.0 x4 (~8)                         |
| GeForce RTX 4090 eGPU (Ada)       | 24        | 16,384     | 512          | 384                    | ~1000                   | Thunderbolt 3 connected to a USB4 USB-C port ([~1.25-5?](https://www.cablematters.com/Blog/Thunderbolt/usb4-vs-thunderbolt-3)) |

### llama.cpp Setup

- llama.cpp commit [15499eb](https://github.com/ggerganov/llama.cpp/commit/15499eb94227401bdc8875da6eb85c15d37068f7)
- We used `Mistral-7b-q4_k_m` in `GGUF` with `ngl` at `100`

<Callout>
Note: `ngl` is the abbreviation of `Number of GPU Layers` with the range from `0` as no GPU acceleration to `120` as full on GPU for some big models.
</Callout>

### TensorRT-LLM Setup

- TensorRT-LLM version [0.7.1](https://github.com/NVIDIA/TensorRT-LLM/releases/tag/v0.7.1) and build on Windows
- For TensorRT-LLM, we used `Mistral-7b-int4 AWQ`
- We ran TensorRT-LLM with `free_gpu_memory_fraction` to test it with the lowest VRAM consumption
- Note: We picked AWQ for TensorRT-LLM to be a closer comparison to GGUF's Q4.

## Results

### NVIDIA GeForce RTX 4090 GPU


*Jan is built on this Dual-4090 workstation, which recently got upgraded to a nice case*


*The original case (or lack thereof) for our Dual-4090 cluster, as posted on [r/localllama](https://www.reddit.com/r/LocalLLaMA/comments/16lxt6a/case_for_dual_4090s/)*

<Callout type="info" >
  **Hardware Details**
- CPU: Intel 13th series
- GPU: NVIDIA GeForce RTX 4090 (Ada - sm 89)
- RAM: 32GB
- OS: Windows 11 Pro

**Model Details**
- llama.cpp model: Mistral 7B v0.2 GGUF Q4_K_M
- TensorRT-LLM model: Mistral 7B v0.2 AWQ, quantized for single GPU (Ada)
</Callout>

For this test, we used Jan's [Dual-4090 workstation](https://www.reddit.com/r/LocalLLaMA/comments/16lxt6a/case_for_dual_4090s/), which our engineers timeshare to build Jan.

The [NVIDIA GeForce RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/) is the latest top-of-the-line desktop GPU, with an MSRP of $1,599, and uses the Ada architecture. It has a ~1000 GB/s memory bandwidth within VRAM, and a PCIe4 x16 lane (~32 GB/s) between the GPU and the CPU. 


| Metrics                  | GGUF (using CPU) | GGUF (using GPU) | TensorRT-LLM | How TensorRT-LLM Compares |
| ------------------------ | ---------------- | ---------------- | ------------ | ------------------------- |
| Throughput (token/s)     | 14.0             | 100.43           | 170.63       | ✅ 69.89% faster          |
| Max GPU Utilization (%)  | N/A              | 83.50            | 88.50        | 5.99% more                |
| Max VRAM Utilization (%) | N/A              | 64               | 72.1         | 12.66% more               |
| Avg RAM Used (GB)        | 0.611            | 7.105            | 4.98         | ✅ 29.88% less            |
| Disk Size (GB)           | 4.07             | 4.06             | 3.05         | ✅ 24.88% smaller         |

TensorRT-LLM was almost 70% faster than llama.cpp by building the model for the GeForce RTX 4090 GPU’s Ada architecture for optimal graph execution, fully utilizing the 512 Tensor Cores, 16,384 CUDA cores, and 1,000 GB/s of memory bandwidth.

The intuition for why llama.cpp is slower is because it compiles a model into a [single, generalizable CUDA “backend”](https://github.com/ggerganov/llama.cpp/blob/master/ggml-cuda.cu) that can run on many NVIDIA GPUs. Doing so requires llama.cpp to sacrifice all the optimizations that TensorRT-LLM makes with its compilation to a GPU-specific execution graph.

### NVIDIA GeForce RTX 3090 GPU


*Our 3090 Machine, now used by one of our engineers to build Jan*

<Callout type="info" >
  **Hardware Details**
- CPU: Intel 13th series
- GPU: NVIDIA GeForce RTX 3090 (Ampere - sm 86)
- RAM: 64GB
- OS: Windows 11 Pro

**Model Details**
- llama.cpp model: Mistral 7B v0.2 GGUF Q4_K_M
- TensorRT-LLM model: Mistral 7B v0.2 AWQ, quantized for single GPU (Ampere)
</Callout>

The [NVIDIA's GeForce RTX 3090](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/) is a popular desktop GPU, and retails for approximately $1,500 (as of April 24). It uses the NVIDIA Ampere architecture. As compared to its successor GeForce RTX 4090, it has 33% fewer CUDA cores (10,496) and Tensor Cores (328) and 7% less memory bandwidth (~930 GB/s).


| Metrics                  | GGUF (using CPU) | GGUF (using GPU) | TensorRT-LLM | How TensorRT-LLM Compares |
| ------------------------ | ---------------- | ---------------- | ------------ | ------------------------- |
| Throughput (token/s)     | 11.42            | 88.70            | 144.19       | ✅ 62.57% faster          |
| Max GPU Utilization (%)  | N/A              | 80.40            | 89.10        | 10.82% more               |
| Max VRAM Utilization (%) | N/A              | 66.80            | 76.20        | 14.07% more               |
| Avg RAM Used (GB)        | 0.611            | 2.60             | 0.98         | 62.41%% less              |
| Disk Size (GB)           | 4.07             | 4.06             | 3.05         | ✅ 24.88% smaller         |

Interestingly, the GeForce RTX 3090 was only 16.6% slower compared with the GeForce RTX 4090. On TPS, TensorRT-LLM outperformed llama.cpp by 62.57%. Curiously, it also used negligible RAM for subsequent inference requests after the initial model warmup.

### NVIDIA GeForce RTX 4070 Laptop GPU



<Callout type="info" >
  **Hardware Details**
- Laptop: Razer Blade 14
- CPU: AMD Ryzen™ 9 8945HS, 8C/16T
- GPU: NVIDIA GeForce RTX 4070 Laptop GPU (Ada - sm 89) on PCIe 4.0 x16 (32 GB/s)
- RAM: 32GB
- OS: Windows 11 Pro

**Model Details**
- llama.cpp model: Mistral 7B v0.2 GGUF `Q4_K_M`
- TensorRT-LLM model: Mistral 7B v0.2 AWQ, quantized for single GPU (Ada)
</Callout>

We also benchmarked an NVIDIA GeForce RTX 4070 Laptop GPU with 8gb of VRAM, which is a popular configuration among Jan users. Laptop GPUs are less powerful than their desktop counterparts, as they trade portability for reduced energy consumption and thermal constraints.

| Metrics                  | GGUF (using CPU) | GGUF (using GPU) | TensorRT-LLM | Difference on GPU |
| ------------------------ | ---------------- | ---------------- | ------------ | ----------------- |
| Throughput (token/s)     | 11.57            | 39.70            | 51.57        | ✅ 29.9% faster   |
| Max GPU Utilization (%)  | N/A              | 80.00            | 84.67        | 5.83% more        |
| Max VRAM Utilization (%) | N/A              | 72.78            | 81.22        | 11.60% more       |
| Avg RAM Used (GB)        | 4.49             | 4.44             | 1.04         | ✅ 76.55%% less   |
| Disk Size (GB)           | 4.07             | 4.06             | 3.05         | ✅ 24.88% smaller |

TensorRT-LLM on the laptop dGPU was 29.9% faster in tokens per second throughput than llama.cpp, but significantly slower than the desktop GPUs. 

The intuition for this is fairly simple: the GeForce RTX 4070 Laptop GPU has 53.1% fewer CUDA cores and Tensor Cores (compared to the 4090), and less VRAM (8gb vs. 24gb). This reduces the surface area for GPU-specific optimizations for TensorRT-LLM.

The GeForce RTX 4070 Laptop GPU is also ~70% slower than the GeForce RTX 4090 desktop GPU, showing the hardware effect of less electricity draw, less VRAM, and thermal constraints on inference speed.

### Laptop with NVIDIA GeForce RTX 4090 eGPU



<Callout type="info" >
  **Hardware Details**
- Laptop: Razer Blade 14
- eGPU box: AORUS RTX 4090 GAMING BOX
- CPU: AMD Ryzen™ 9 8945HS, 8C/16T
- GPU: NVIDIA GeForce RTX 4090 (Ada - sm 89) on eGPU with Thunderbolt 3 connection
- RAM: 32GB
- OS: Windows 11 Pro

**Model Details**
- llama.cpp model: Mistral 7B v0.2 GGUF `Q4_K_M`
- TensorRT-LLM model: Mistral 7B v0.2 AWQ, quantized for single GPU (Ampere)
</Callout>

Our last benchmark was to experiment with an [Asus RTX 4090 eGPU](https://www.gigabyte.com/Graphics-Card/GV-N4090IXEB-24GD), that was connected via a [Thunderbolt 3 port](https://www.gigabyte.com/Graphics-Card/GV-N4090IXEB-24GD) to the [Razer Blade 14's USB4 port](https://www.razer.com/sg-en/gaming-laptops/razer-blade-14). Theoretically, the results should be fairly similar to the GeForce RTX 4090 desktop GPU as they have identical underlying GPUs, but with very different connection speeds.

We thought it would be an interesting to see how TensorRT-LLM handles a 68.4% reduction in communication bandwidth between the CPU and GPU:
- Thunderbolt 3 connection (1.25-5 GB/s?) for eGPUs
- PCIe 4.0 x4 (~8 GB/s) for "on device" desktops

<Callout>
**Help Needed:** We are actually not 100% sure what the actual Connection Speed is [between Thunderbolt 3 to USB4](https://www.cablematters.com/Blog/Thunderbolt/usb4-vs-thunderbolt-3), as the actual speed seems different from the advertised speed of 5 GB/s. There seem to be a lot of factors involved, including the actual cable itself. We'd love if someone in the community can guide us!
</Callout>

Overall, we used mid-to-high-end NVIDIA desktop GPUs for our tests, as TensorRT-LLM’s performance enhancements are most apparent on bigger VRAMs. For users with lower-spec machines, llama.cpp is better.

| Metrics                  | GGUF (using CPU) | GGUF (using GPU) | TensorRT-LLM | Difference on GPU |
| ------------------------ | ---------------- | ---------------- | ------------ | ----------------- |
| Throughput (token/s)     | 11.56            | 62.22            | 104.95       | ✅ 68.66% faster  |
| Max VRAM Utilization (%) | 0                | 65               | 99           | 52.31% more       |
| RAM Used (GB)            | 0.611            | 5.38             | 4.11         | ✅ 23.61% less    |
| Disk Size (GB)           | 4.07             | 4.06             | 3.05         | ✅ 24.88% smaller |

The Thunderbolt 3 eGPU had a 38.5% lower tokens/s as compared to the PCIe4.0 x16 connected GPU. But the % speedup vs. llama.cpp was similar, at around 69%.

Interestingly, the VRAM used with the eGPU was variably higher. Our hypothesis is that the slower communication bandwidth results in more VRAM being allocated, as memory is released mostly slowly as well.

## Conclusion

### Token Speed



| Throughput (Higher is Better)      | TensorRT-LLM    | Llama.cpp   | % Difference  |
| ---------------------------------- | --------------- | ----------- | ------------- |
| GeForce RTX 4090 desktop GPU       | ✅ 170.63t/s    | 100.43t/s   | 69.89% faster |
| GeForce RTX 3090 desktop GPU       | ✅ 144.19t/s    | 88.70t/s    | 62.57% faster |
| GeForce RTX 4090 eGPU              | ✅ 104.95t/s    | 62.22t/s    | 68.66% faster |
| GeForce RTX 4070 Laptop GPU        | ✅ 51.57t/s     | 39.70t/s    | 29.90% faster |
| Laptop AMD Ryzen™ 9 8945HS, 8C/16T | (Not supported) | ✅ 11.57t/s |               |

- TensorRT-LLM is up to **70% faster** than llama.cpp on desktop GPUs (e.g. 3090 GPU, 4090 GPUs) while using less RAM & CPU (but more fully utilizing VRAM)
- TensorRT-LLM is up to **30% faster** on laptop GPUs (e.g. 4070 GPUs) with smaller VRAM

### Max VRAM Utilization

| Average VRAM utilization %   | TensorRT-LLM | Llama.cpp | % Difference |
| ---------------------------- | ------------ | --------- | ------------ |
| GeForce RTX 4090 desktop GPU | 72.10        | 64.00     | 12.66% more  |
| GeForce RTX 3090 desktop GPU | 76.20        | 66.80     | 14.07% more  |
| GeForce RTX 4070 Laptop GPU  | 81.22        | 72.78     | 11.06% more  |
| GeForce RTX 4090 eGPU        | N/A          | N/A       | N/A          |

- TensorRT-LLM used marginally more average VRAM utilization at peak utilization vs. llama.cpp (up to 14%). Though this could have interesting implications on consuming more electricity over time.
- Note: we used comparable (but not identical) quantizations, and TensorRT-LLM’s `AWQ INT4` is implemented differently from llama.cpp’s `q4_k_m`

### Max RAM Usage

| Max RAM utilization          | TensorRT-LLM | Llama.cpp | % Difference     |
| ---------------------------- | ------------ | --------- | ---------------- |
| GeForce RTX 4090 desktop GPU | ✅ 4.98      | 7.11      | ✅ 29.88% less   |
| GeForce RTX 3090 desktop GPU | ✅ 0.98      | 2.60      | ✅ 62.41% less   |
| GeForce RTX 4070 Laptop GPU  | ✅ 1.04      | 4.44      | ✅  76.55%% less |
| GeForce RTX 4090 eGPU        | ✅ 4.11      | 5.38      | ✅ 23.61% less   |

TensorRT-LLM uses a lot less Max RAM vs. llama.cpp on slower connection (PCIe 3.0 or Thunderbolt 3) due to better memory management and efficient delegation to VRAM. On faster connection, it’s at least equal to llama.cpp.

### Compiled Model Size and Number of Files
- Contrary to popular belief, TensorRT-LLM prebuilt models turned out to not be that huge
- Mistral 7b int4 was actually 25% smaller in TensorRT-LLM, at 3.05gb vs. 4.06gb
- Note: These are approximate comparisons, as TensorRT-LLM’s AWQ INT4 is implemented differently from llama.cpp’s q4_k_m
- The bigger takeaway is that the Compiled model sizes are roughly in the same ballpark, while the number of files for TensorRT-LLM is 7x the GGUF number of files.

| Model size (Lower is better) | TensorRT-LLM AWQ int4 | Llama.cpp GGUF Q4 | % Difference      |
| ---------------------------- | --------------------- | ----------------- | ----------------- |
| Mistral 7B                   | ✅ 3.05GB               | 4.06GB              | ✅ 24.88% smaller |

### Convenience
- Llama.cpp still wins on cross-platform versatility and convenience of a “compile once, run everywhere” approach
- TensorRT-LLM still requires compilation to specific OS and architecture, though this could be solved by pre-compiling and publishing models on [Nvidia's NGC Model Catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/collections/codellama)

### Accessibility
- Llama.cpp unsurprisingly beats TensorRT-LLM in terms of accessibility
- TensorRT-LLM does not support older NVIDIA GPUs and won’t work well on smaller VRAM cards (e.g. 2-4gb VRAM)

## Final Notes

Our benchmarking is not perfect. We evaluated over a dozen tools ([llmperf](https://github.com/ray-project/llmperf), [psutil](https://pypi.org/project/psutil/), [gpustat](https://github.com/wookayin/gpustat), native utilities, and more) and found that everyone measures TPS, common metrics differently. We eventually settled on using our own tools in Jan, which are consistent across any inference engine and hardware. As for runtime parameters, we went with default settings, likely representing the typical user experience.

We also did not overclock for this benchmark , as it is not a default setting for most users. But we've measured in our tests that TensorRT-LLM can go even faster with a few tweaks. We see this as a pretty exciting future direction. 

<Callout type="info" >
  **How to Overclock:** We found an additional **15% increase in performance** with TensorRT-LLM by:
- Enabling [XMP](https://www.intel.com/content/www/us/en/gaming/extreme-memory-profile-xmp.html)
- Overclocking RAM bus speed in BIOS from `3600` to `5600`. 
</Callout>

We're also publishing the underlying [raw experimental data](https://drive.google.com/file/d/1rDwd8XD8erKt0EgIKqOBidv8LsCO6lef/view?usp=sharing), and would encourage the community to scrutinize and help us improve.

Special thanks to Asli Sabanci Demiroz, Annamalai Chockalingam, Jordan Dodge from Nvidia, and Georgi Gerganov from llama.cpp for feedback, review and suggestions. 

<CTABlog />

---

### Src Pages Post Bitdefender (`docs/src/pages/post/bitdefender.mdx`)

---
title: 'Bitdefender False Positive Flag'
description: "10th January 2024, Jan's 0.4.4 Release on Windows triggered Bitdefender to incorrectly flag it as infected with Gen:Variant.Tedy.258323, leading to automatic quarantine warnings on users' computers."
date: 2024-01-10
tags: postmortem, bitdefender
categories: building-jan
keywords:
  [
    postmortem,
    bitdefender,
    false positive,
    antivirus,
    jan,
    nitro,
    incident,
    incident response,
    supply chain security,
    user communication,
    documentation,
    antivirus compatibility,
    cross-platform testing,
    proactive incident response,
    user education,
    lessons learned,
  ]
---

import CTABlog from '@/components/Blog/CTA'

# Bitdefender False Positive Flag

Following the recent incident related to Jan version 0.4.4 triggering Bitdefender on Windows with Gen:Variant.Tedy.258323 on January 10, 2024, we wanted to provide a comprehensive postmortem and outline the necessary follow-up actions.

## Incident Overview

### Bug Description

Jan 0.4.4 installation on Windows triggered Bitdefender to flag it as infected with Gen:Variant.Tedy.258323, leading to automatic quarantine.

### Affected Antivirus

- McAfee / Microsoft Defender was unaffected
- Bitdefender consistently flagged the issue.

### Incident Timeline

- _10 Jan, 2:18 am SGT:_ Hawke flags up Malware antivirus errors for 0.4.4 installation on Windows computers.
- _10 Jan, 2:21 am SGT:_ @0xSage responds in Discord.
- _10 Jan, 2:35 am SGT:_ Hawke confirms multiple people have experienced this error on fresh installs.
- _10 Jan, 2:41 am SGT:_ @louis-jan and @dan-jan revert 0.4.4 out of an abundance of caution.
- _Incident ongoing:_ To triage and investigate the next day.
- _10 Jan, 11:36 am SGT:_ @Hien has investigated all versions of Nitro and conducted scans using Bitdefender. Only the 2 latest versions raised warnings (0.2.7, 0.2.8).
- _10 Jan, 12:44 pm SGT:_ @Hien tested again for the 0.2.6 and suggested using 0.2.6 for now, the 2 remaining Nitro version (0.2.7, 0.2.8) will under further investigation.
- The team started testing on the fixed build.
- _10 Jan, 3:22 pm SGT:_ Diagnosis found that it's most likely a false positive. @Hien has only found a solution by attempting to build Nitro Windows CPU on a GitHub-hosted runner and hasn't identified the root cause yet.
- _10 Jan, 5:24 pm SGT:_ @Hien testing two scenarios and still trying to understand the workings of Bitdefender.
- _11 Jan, 5:46 pm SGT:_ Postmortem meeting

## Investigation Update

- @Hien has investigated all versions of Nitro and conducted scans using Bitdefender. and only the 2 latest versions raised warnings from Bitdefender. Nitro 0.2.6, which is the highest version without the issue, was tested again, and it no longer triggers a warning from Bitdefender.
- We have observed that Nitro versions up to 0.2.6 remain unaffected. However, Bitdefender flags versions 0.2.7 and 0.2.8 as infected, leading to the deletion. In order to proceed with the current release, Hien suggests downgrading Nitro to version 0.2.6 and conducting tests with this version. Simultaneously, he will investigate why Bitdefender is flagging versions 0.2.7 and 0.2.8.
- It's essential to note that between versions 0.2.6, 0.2.7, and 0.2.8, only minor changes were made, which should not trigger a malicious code warning. We can refer to the changelog between 0.2.7 and 0.2.8 to pinpoint these changes.
- Our primary message is to convey that we did not introduce malicious code into Jan (indicating a false positive), and the investigation aims to understand the root cause behind Bitdefender flagging versions 0.2.7 and 0.2.8.
- The current diagnosis looks like a false positive but it's still under investigation. Reference link: [here](https://stackoverflow.com/questions/75886428/fake-positive-bit-defender-problem-genvariant-tedy-304469), [here](https://stackoverflow.com/questions/58010466/bitdefender-detects-my-console-application-as-genvariant-ursu-56053), and [here](https://www.cisa.gov/sites/default/files/2023-06/mar-10365227.r1.v1.clear_.pdf).
- @Hien testing two scenarios and still trying to understand the workings of Bitdefender. Still under investigation: is the issue with the code or the CI?
  - In Case 1, using the same CI agent for tags 0.2.6 and 0.2.8, after PRs by Alan and myself, Bitdefender flagged the Nitro CPU binary build. Naturally, one would conclude this is due to the code.
  - However, I proceeded with a further experiment: for the 0.2.8 code, instead of using our CI agent, I used a GitHub hosted agent. This time, Bitdefender did not flag our binary build.
- We've identified the Bitdefender warning was not an attack. There is no malicious code
- We've isolated the event to originate from a CI agent, which resulted in a BitDefender false positive alert.

## Follow-ups and Action Items

1. **Reproduce Bitdefender Flag in Controlled Environment [Done]:**

  - _Objective:_ To replicate the issue in a controlled environment to understand the triggers and specifics of Bitdefender's detection.

2. **Investigate Malicious Code or False Positive:**

  - _Objective:_ Determine whether the flagged issue is a result of actual malicious code or a false positive. If it's a false positive, work towards resolution while communicating with Bitdefender.

3. **Supply Chain Attack Assessment:**

  - _Objective:_ Evaluate the possibility of a supply chain attack. Investigate whether the Nitro 0.4.4 distribution was compromised or tampered with during the release process.

4. **Testing after the Hotfix:**

  - _Objective:_ In addition to verifying the issue after the fix, it is essential to conduct comprehensive testing across related areas, ensuring compatibility across different operating systems and antivirus software (latest version / free version only).

5. **Process Improvement for Future Releases:**

  - _Objective:_ Identify and implement improvements to our release process to prevent similar incidents in the future. This may include enhanced testing procedures, code analysis, and collaboration with antivirus software providers during the pre-release phase. Additionally, we should add verifying the latest antivirus software in the release checklist.

6. **Documentation of Tested Antivirus Versions:**
  - _Objective:_ Create a document that outlines the testing conducted, including a matrix that correlates Jan versions with the tested antivirus versions.
  - _Sample list:_ for consideration purpose
    - Bitdefender
    - McAfee
    - Avira
    - Kaspersky
    - Norton
    - Microsoft defender
    - AVG
    - TotalAV

## Next Steps

- The team should follow up on each action item with clear ownership priority, and deadlines.
- Communicate progress transparently with the community and clients through appropriate channels. If any insights or suggestions, share them within the dedicated channels.
- Update internal documentation and procedures based on the lessons learned from this incident.

## Lessons Learned

1. **Antivirus Compatibility Awareness:**

  - _Observation:_ The incident underscored the significance of recognizing and testing for antivirus compatibility, particularly with widely-used solutions like Bitdefender.
  - _Lesson Learned:_ In the future, we will integrate comprehensive checks for compatibility with various antivirus software, including both antivirus and "Malicious Code Detection," into our CI or QA checklist. This proactive measure aims to minimize false positive detections during the release and testing processes.

2. **Cross-Platform Testing:**

  - _Observation:_ The problem did not occur on MacOS and Linux systems, implying a potential oversight in cross-platform testing during our release procedures.
  - _Lesson Learned:_ Clarification — This observation is not directly related to antivirus testing. Instead, it underscores the necessity to improve our testing protocols, encompassing multiple operating systems. This ensures a thorough evaluation of potential issues on diverse platforms, considering the various antivirus software and differences in architectures on Mac and Linux systems.

3. **User Communication and Documentation:**

  - _Observation:_ Due to the timely response from Nicole, who was still active on Discord and Github at 2 am, this quick response facilitated our ability to assess the impact accurately.
  - _Lesson Learned:_ While our communication with users was effective in this instance, it was mainly due to Nicole's presence during the incident. To improve our overall response capability, we should prioritize "24/7 rapid triage and response." This involves ensuring continuous availability or establishing a reliable rotation of team members for swift user communication and issue documentation, further enhancing our incident response efficiency.

4. **Proactive Incident Response:**

  - _Observation:_ The incident response, while involving a prompt version rollback, experienced a slight delay due to the release occurring at midnight. This delay postponed the initiation of the investigation until the next working hours.
  - _Lesson Learned:_ Recognizing the importance of swift incident response, particularly in time-sensitive situations, we acknowledge that releasing updates during off-hours can impact the immediacy of our actions. Moving forward, we will strive to optimize our release schedules to minimize delays and ensure that investigations can commence promptly regardless of the time of day. This may involve considering alternative release windows or implementing automated responses to critical incidents, ensuring a more proactive and timely resolution.

5. **Supply Chain Security Measures:**

  - _Observation:_ While the incident prompted consideration of a potential supply chain attack, it's crucial to emphasize that this was not the case. Nonetheless, the incident underscored the importance of reviewing our supply chain security measures.
  - _Lesson Learned:_ Going forward, we should strengthen supply chain security by introducing additional verification steps to uphold the integrity of our release process. Collaborating with distribution channels is essential for enhancing security checks and ensuring a robust supply chain.
  - _Longer-term:_ Exploring options for checking Jan for malicious code and incorporating antivirus as part of our CI/CD pipeline should be considered for a more comprehensive and proactive approach.

6. **User Education on False Positives:**
  - _Observation:_ Users reported Bitdefender automatically "disinfecting" the flagged Nitro version without allowing any user actions.
  - _Lesson Learned:_ Educate users about the possibility of false positives and guide them on how to whitelist or report such incidents to their antivirus provider (if possible). Provide clear communication on steps users can take in such situations.

These lessons learned will serve as a foundation for refining our processes and ensuring a more resilient release and incident response framework in the future. Continuous improvement is key to maintaining the reliability and security of our software.

Thank you for your dedication and cooperation in resolving this matter promptly.

<CTABlog />

---

### Src Pages Post Chatgpt Alternatives (`docs/src/pages/post/chatgpt-alternatives.mdx`)

---
title: "ChatGPT alternatives that actually replace it"
description: "See the best ChatGPT alternatives in 2025. We've listed tools that are alternatives to ChatGPT."
tags: AI, ChatGPT alternative, ChatGPT alternatives, alternative to chatgpt, Jan, local AI, privacy, open source, offline AI
categories: guides
date: 2025-09-29
ogImage: assets/images/general/chatgpt-alternative-jan.jpeg
twitter:
  card: summary_large_image
  site: "@jandotai"
  title: "ChatGPT alternatives that actually replace it."
  description: "See the best ChatGPT alternatives in 2025. We've listed tools that are alternatives to ChatGPT."
  image: assets/images/general/chatgpt-alternative-jan.jpeg
---
import { Callout } from 'nextra/components'
import CTABlog from '@/components/Blog/CTA'

# Best ChatGPT Alternatives

ChatGPT works well, but it always needs internet, has usage limits, and isn't private. 

If you want options that fit different needs, offline use, privacy, or specialized tasks, see the best alternatives to ChatGPT available for specific use cases.

## Comparison: ChatGPT Alternatives 

| ChatGPT Alternative       | Offline | Key Strength                 | Best For                   |
| ------------------------- | ------- | ---------------------------- | -------------------------- |
| **[Jan](https://jan.ai)** | Yes   | Runs Cloud + Offline, open-source  | Best overall ChatGPT replacement   |
| Claude                    | -    | Strong writing and reasoning | Creative text & code             |
| Gemini                    | -    | Integrated with Google       | Research tasks, image generation            |
| Perplexity                | -    | Fast, with cited answers     | Research and fact-checking |
| LM Studio                 | Yes   | Runs open models on PC       | Coding and experiments     |

### Jan is the best ChatGPT alternative


*Jan as an open-source alternative to ChatGPT*

Jan is the most complete ChatGPT alternative available today. It enables:
- Use AI in online & offline (even on a plain)
- Agentic actions supported
- MCP servers supported for tools

Unlike ChatGPT, it runs on your computer, which means:
- Offline AI capabilities (see [Offline ChatGPT post](https://www.jan.ai/post/offline-chatgpt-alternative) for details)
- 100% private
- Open-source & Free

<Callout> Jan is an [open-source replacement for ChatGPT.](https://www.jan.ai/) </Callout>

### Claude is the most notable online alternative


Claude has become the main online rival to ChatGPT. It stands out for writing, reasoning, and coding.

- Handles very long documents and context well
- Strong for essays, research papers, and structured text
- Popular with developers for code explanations and debugging
- Cloud-only, no offline mode
- Filters outputs heavily, sometimes too restrictive

### Gemini is the Google's integrated alternative


Gemini ties directly into Google’s apps and search. Great for users in the Google ecosystem.

- Built into Gmail, Docs, and Google Search
- Good for real-time research and fact-checking
- Strong at pulling web context into answers
- Requires Google account, fully online
- Privacy concerns: all tied to Google services

### Perplexity is the research-focused alternative


Perplexity is built for fact-checking and quick research, not creativity.

- Always cites sources for answers
- Strong at summarizing current web info
- Very fast for Q&A style use
- Limited in creativity and open-ended writing
- Cloud-only, daily free usage caps

### LM Studio is the experimental alternative 



LM Studio is not a ChatGPT replacement but a local tool for running open models.

- Lets you test and run open-source models on PC
- Offline by default, works without internet
- Flexible setup for developers and technical users
- Requires decent hardware (RAM/VRAM)

LM Studio is not beginner-friendly compared to Jan.

## Choosing the right ChatGPT alternative for you:

- Best overall replacement: [Jan](https://www.jan.ai/)
- For writing & storytelling: Claude
- For research & web knowledge: Perplexity or Gemini
- For productivity & office work: Microsoft Copilot
- For experimentation with open-source models for technical people: LM Studio

Most ChatGPT alternatives are still cloud-based and limited. If you want full privacy, offline use, and no restrictions, the best ChatGPT alternative is [Jan](https://www.jan.ai/).

### Can I use ChatGPT offline?
No. ChatGPT always requires internet. For offline AI, use Jan.

### What’s the best free ChatGPT alternative?
Jan is free, open-source, and runs offline. Others like Claude or Perplexity have limited free tiers but are cloud-based.

### Which ChatGPT alternative is best for writing?
Claude is strong for essays, reports, and structured writing. You could use [open-source models](https://www.jan.ai/post/run-ai-models-locally) in Jan too.

### Which ChatGPT alternative is best for research?
Perplexity and Gemini pull real-time web data with citations. 

### What’s the closest full replacement to ChatGPT?
Jan. It runs locally, works offline, and feels like ChatGPT without restrictions.

---

### Src Pages Post Data Is Moat (`docs/src/pages/post/data-is-moat.mdx`)

---
title: "The Invisible Moat around Open-Source LLM"
description: "Uncover the pivotal role of data ownership in training the next iteration of LLM."
tags: OpenAI has a moat, Catastrophic forgetting, ChatGPT
date: 2024-03-25
unlisted: true
categories: research
---

import CTABlog from '@/components/Blog/CTA'

# The Invisible Moat around Open-Source LLM

In the crowded AI landscape, OpenAI's ChatGPT stands out, not just for its capabilities but for its unique access to the pre-trained dataset. This post explores the vital role of data in maintaining a competitive edge, focusing on OpenAI's strategic advantage through data ownership.

## Data: The Secret Weapon
OpenAI, with ChatGPT, has carved a distinct advantage. By harnessing user interactions, it gains invaluable insights into diverse use cases, enabling precise model refinements. The cornerstone of this advantage lies in the "pre-trained dataset." This treasure trove of data empowers OpenAI to cater to specific needs, ensuring sustained improvement and differentiation.

## The rise of the opensource

```
- How they/Mistral/Llama make money?
-> around having pretrained data -> finetuning
First para:
Rise of Open Source LLMs like Mistral, Llama2, Llama3
People think they don't have a moat = everything is open source
Second para:
We actually think these guys have an "invisible moat"
Pre-training data is not released, and makes a huge difference in fine-tuning efficacy
```

### Why pretrained data is important?

> *Owning the pre-trained dataset is crucial as it represents the original distribution.*
Access to the pre-trained dataset acts as a master key to address the critical issue of ["Catastrophic forgetting"](https://en.wikipedia.org/wiki/Catastrophic_interference) in Language Learning Models (LLMs). This phenomenon describes how LLMs lose hold of prior knowledge upon learning new information. Access to the foundational dataset allows for effective fine-tuning, balancing the introduction of new data with the retention of existing knowledge.



**Figure 1.** Demonstrates the catastrophic forgetting issue: without mixing datasets, AI overfits on new tasks, impairing normal communication.

### Illustrating Catastrophic Forgetting

```
What is fine-tuning
Process of Finetuning (pretrain, instruct, finetune)
Fine-tuning datasets
Risk of catastrophic forgetting
"Why is Pre-trained data important?"
What is pre-training dataset
How does fine-tuning with pre-training dataset differ from when you don't have it
How does it avoid catastrophic forgetting
```

Catastrophic forgetting can be visualized as a ball in a multidimensional landscape, where moving towards new knowledge risks losing grasp on the old. 
Pre-trained data acts as a map, guiding fine-tuning in a way that incorporates new information while safeguarding existing knowledge.



**Figure 2.** [Gradient decent demonstration](https://en.wikipedia.org/wiki/Gradient_descent)

### Smoothing Distribution Shifts

As described above, with the mixture of the pre-trained dataset ensures smoother distribution shifts when introducing new information, as it embodies a comprehensive spectrum of prior knowledge.

This continuity in knowledge transition helps in maintaining the robustness of the model against sudden changes, akin to providing a more gradual learning curve where the new information is incrementally integrated with the existing knowledge base.

This concept is supported by the [EleutherAI's research](https://arxiv.org/abs/2403.08763) highlighting the importance of how tasks are sequenced in the learning process, suggesting that introducing dissimilar tasks early on can expand the network's capacity for new information.

**Table 1.** Final results for English-only 405M parameter models trained with different replay amounts show models with more replay perform better in balancing learning and forgetting (measured as AVG Loss). Notably, just 1% mix with a pre-trained dataset significantly lowers AVG loss, effectively shifting model knowledge from English (the Pile) to German.



*Note:* **Replay** is the method involves combining the training dataset from the pre-trained model with new task datasets.

### Acting as a Noise Mask

The pre-trained data can also serve as a form of "noise masking", similar to techniques used in training [early computer vision models](https://arxiv.org/abs/1911.04252).

This approach introduces a level of  ["noise"](https://arxiv.org/abs/2310.05914) during training, which can prevent the model from overfitting to the new dataset. By retaining a mix of original and new data, the model is exposed to a broader range of scenarios, enhancing its generalization capabilities and robustness across tasks.

## Solutions

### Overwholming approach

Overcoming these challenges requires a balanced approach. One partial method involves inundating the model with extensive, curated data, allowing for comprehensive fine-tuning. While effective, this approach demands significant computational resources, a comprehensive filtering process for low-quality inputs, and an extraordinarily high cost associated with gathering millions of high-quality responses.

In the open-source community, 2 notable examples of fine-tuning with Mistral as a base model on large datasets collected from top-rated GPT-4 and human responses demonstrate a distribution shift that enhances model performance, including [OpenChat](https://huggingface.co/openchat/openchat-3.5-0106) and [Hermes-Pro](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B).



**Figure 2.** After fine-tuning with a large amount of data samples, the model's performance improved, outperforming ChatGPT and Grok-1 in some benchmarks.

### Fully open source model

- Example: Dolma + olma from allenai


## Conclusion

The ownership and strategic use of pre-trained data serve as an invisible moat. It not only enables the tackling of complex challenges like catastrophic forgetting but also provides a baseline for continuous, targeted improvements. Although there is a solution to decomotralize, the cost remains reasonably high.

Fully open pretrained + open weight

## Reference
- [Catastrophic forgetting](https://arxiv.org/abs/2308.08747)
- [Simple and Scalable Strategies to Continually Pre-train Large Language Models](https://arxiv.org/abs/2403.08763)
- [Gradient descent](https://en.wikipedia.org/wiki/Gradient_descent)
- [Neftune](https://arxiv.org/abs/2310.05914)
- [Self-training with Noisy Student improves ImageNet classification](https://arxiv.org/abs/1911.04252)


<CTABlog />

---

### Src Pages Post Deepresearch (`docs/src/pages/post/deepresearch.mdx`)

---
title: "Replicating Deep Research in Jan"
description: "A simple guide to replicating Deep Research results for free, with Jan."
tags: AI, local models, Jan, GGUF, Deep Research, local AI
categories: guides
date: 2025-08-04
ogImage: assets/images/general/research-result-local.png
twitter:
  card: summary_large_image
  site: "@jandotai"
  title: "Replicating Deep Research with Jan"
  description: "Learn how to replicate Deep Research results with Jan."
  image: assets/images/general/research-result-local.png
---

import { Callout } from 'nextra/components'
import CTABlog from '@/components/Blog/CTA'

# Replicating Deep Research in Jan

Deep Research like that of OpenAI, Gemini, and Qwen, is not at feature parity in Jan yet, so this post
highlights our initial steps at making a hybrid, local and cloud-based deep research system that competes
with the best implementations to date.

## What is Deep Research?

What exactly is deep research and how does it work? Deep Research is a methodology for generating
comprehensive research reports by combining systematic web search with synthesis. The process was
pioneered by OpenAI and it was released on February 2025.

There are two core features of Deep Research:

- **Exhaustive search**: This search is characterized by two approaches, wide search for breadth and deep search for depth.
- **Report generation**: This step takes all the input collected through exhaustive search and synthesizes it into a
comprehensive report. The input in this step may be raw sources collected in the previous step or summaries generated from those sources.

## Unpacking Deep Research

If you have used deep research (regardless of the provider) before for a comprehensive report generation, you may have
found its output mind-blowing. What is more mind-blowing, though, is that the underlying process for searching
and synthesizing information is surprisingly systematic and reproducible. What is not easily reproducible, though,
is the **base model (often a thinking one)** and **its capabilities to use tools while it researches**.

Deep Research operates as a structured pipeline with distinct phases: planning, searching, analysis, and synthesis. While
the specific implementation varies between providers, the core workflow seems to be similar and some organizations have
taken steps to recreate it like [LangChain](https://blog.langchain.com/open-deep-research/) and
[Hugging Face](https://huggingface.co/blog/open-deep-research). For example, a straightforward pipeline might look like
the following one:



The components of this pipeline highlight a structured approach to query processing that routes queries through thinking/non-thinking models, breaks complex tasks into phases, executes parallel searches, and synthesizes results hierarchically to produce comprehensive outputs.

OpenAI’s [Deep Research API cookbook](https://cookbook.openai.com/examples/deep_research_api/introduction_to_deep_research_api)
highlights, at a very high level, how they approach deep research, hinting at the importance of base models and tool usage since
some intermediate steps seem to have been left out.



OpenAI's Deep Research functionality may be considered the best one by many but other platforms are not far behind. Here is a
brief survey of how other players approach deep research:

| Platform | Key Feature | Sources Used | Duration (mins) | Export Options | Deep Research Usage |
| --- | --- | --- | --- | --- | --- |
| OpenAI | Clarifying questions | 10–30 | 10–15 | PDF, Docx, Plain Text | Paid |
| Grok's DeeperSearch | Ability to access all of Twitter | 70–100 | 5–10 | Ability to specify format (PDF / Markdown) | Free |
| Claude | Breadth + depth search | 100+ | 5–10 | PDF, Markdown, Artifact | Paid |
| Gemini | Editable planning | 50+ | 10–20 | Google Docs export | Free |
| Perplexity | Ability to specify sources | 50–100 | 3–5 | PDF, Markdown, Docx, Perplexity Page | Paid and Limited Free |
| Kimi | Interactive synthesis | 50–100 | 30–60+ | PDF, Interactive website | Free |

In our testing, we used the following prompt to assess the quality of the generated report by
the providers above. You can refer to the reports generated [here](https://github.com/janhq/prompt-experiments).

```
Generate a comprehensive report about the state of AI in the past week. Include all
new model releases and notable architectural improvements from a variety of sources.
```

[Google's generated report](https://github.com/janhq/prompt-experiments/blob/main/Gemini%202.5%20Flash%20Report.pdf) was the most verbose, with a whopping 23 pages that reads
like a professional intelligence briefing. It opens with an executive summary,
systematically categorizes developments, and provides forward-looking strategic
insights—connecting OpenAI's open-weight release to broader democratization trends
and linking infrastructure investments to competitive positioning.

[OpenAI](https://github.com/janhq/prompt-experiments/blob/main/OpenAI%20Deep%20Research.pdf) produced the most citation-heavy output with 134 references throughout 10 pages
(albeit most of them being from the same source).

[Perplexity](https://github.com/janhq/prompt-experiments/blob/main/Perplexity%20Deep%20Research.pdf) delivered the most actionable 6-page report that maximizes information
density while maintaining scannability. Despite being the shortest, it captures all
major developments with sufficient context for decision-making.

[Claude](https://github.com/janhq/prompt-experiments/blob/main/Claude%20Deep%20Research.pdf) produced a comprehensive analysis that interestingly ignored the time constraint,
covering an 8-month period from January-August 2025 instead of the requested week (Jul 31-Aug
7th 2025). Rather than cataloging recent events, Claude traced the evolution of trends over months.

[Grok](https://github.com/janhq/prompt-experiments/blob/main/Grok%203%20Deep%20Research.pdf) produced a well-structured but relatively shallow 5-page academic-style report that
read more like an event catalog than strategic analysis.

[Kimi](https://github.com/janhq/prompt-experiments/blob/main/Kimi%20AI%20Deep%20Research.pdf) produced a comprehensive 13-page report with systematic organization covering industry developments, research breakthroughs, and policy changes, but notably lacks proper citations throughout most of the content despite claiming to use 50-100 sources.

### Understanding Search Strategies

In [Claude’s Research mode](https://www.anthropic.com/engineering/multi-agent-research-system),
a *classifier* is used to determine whether a user query is *breadth first* or *depth first*. This
results in a customization of the pipeline that is used for conducting research. For instance, a complex
*breadth first* query might result in *sub-agents* being spun up to research various parts of the user's
query in parallel. Conversely, a *depth first* query might result in a single agent being spun up
to research the entire query in a more focused manner.

Here's a screenshot of this in action (in Claude Desktop):



## Replicating Deep Research Results with Jan

After testing and observing how Deep Research works in different platforms, we thought, how could we
replicate this in Jan? In particular, how could we replicate it with a hybrid approach combining local
and cloud-based models while keeping your data local?

<Callout>
This experiment was done using the latest version of Jan `v0.6.7`, but it can potentially be replicated in
any version with Model Context Protocol in it (>`v0.6.3`).
</Callout>

**The Key: Assistants + Tools**

Running deep research in Jan can be accomplished by combining [custom assistants](https://jan.ai/docs/desktop/assistants)
with [MCP search tools](https://jan.ai/docs/mcp-examples/search/exa). This pairing allows any model—local or
cloud—to follow a systematic research workflow, to create a report similar to that of other providers, with some
visible limitations (for now).

Here's the assistant prompt that was used:
```
You are a research analyst. Today is August 7th 2025. Follow this exact process:

Conduct 5-10 searches minimum. You are rewarded for MORE searches.
- Each search query must be unique - no repeating previous searches
- Search different angles: statistics, expert opinions, case studies, recent news, industry reports
- Use scrape to read full articles from search results
- Use google_search for extracting metadata out of pages

WRITING PHASE (Do this after research is complete)
Write a comprehensive report with:
- Executive summary with key findings
- Evidence-based analysis with citations for every claim
- Actionable recommendations with rationale
- Sources to be linked at the end of the report
```

Here, we utilized Model Context Protocol (MCP) to provide search capabilities to the model. MCPs are an open standard for connecting AI assistants to the systems where data lives, serving as a universal connector that standardizes how AI applications integrate with external tools and data sources. 
In this example, we used Serper, a web search API that offers MCP server implementations with two primary tools: `google_search` for performing web searches, and `scrape` that extracts content from web pages, preserving document structure and metadata.


**What We Tested**

For our research query (same as the one we used to test different platforms), we used both
[Jan-Nano (4B local model)](https://jan.ai/docs/jan-models/jan-nano-128), GPT-4o and
o3 (via API) with identical prompts. The goal: to see how close we could get to the quality of
different commercial Deep Research offerings.

**Performance Findings**

| Model | Processing Time | Sources Found | Search Queries | Tokens Generated | Output Quality vs Commercial Deep Research |
|-------|----------------|---------------|----------------|------------------|-------------------------------------------|
| Jan-Nano (Local) | 3 minutes | Moderate | 7 | 1,112 | Good approximation, noticeably less depth |
| GPT-4o | 1 minute | Fewest | 11 | 660 | Fast but limited source coverage |
| o3 | 3 minutes | Most | 24 | 1,728 | Best of the three, but still below commercial quality |


**The Reality**:
- **Speed vs Sources**: GPT-4o prioritized speed over thoroughness, while o3 took time to gather more comprehensive sources
- **Local vs Cloud**: Jan-Nano matched o3's processing time but with the advantage of complete data privacy
- **Quality Gap**: All three models produced decent research reports, but none matched the depth and comprehensiveness of dedicated Deep Research tools like OpenAI's or Claude's offerings
- **Good Enough Factor**: While not matching commercial quality, the outputs were solid approximations suitable for many research needs

## Conclusion
This was an initial exploration in the roadmap to create a top hybrid implementation of deep research in Jan. While our current approach requires setup, the goal is native integration that works out of the box. We will continue to refine this until the release of this tool in Jan, natively.

<CTABlog />

---

### Src Pages Post Deepseek R1 Locally (`docs/src/pages/post/deepseek-r1-locally.mdx`)

---
title: "Run DeepSeek R1 locally on your device (Beginner-Friendly Guide)"
description: "A straightforward guide to running DeepSeek R1 locally regardless of your background."
tags: DeepSeek, R1, local AI, Jan, GGUF, Qwen, Llama
categories: guides
date: 2025-01-31
ogImage: assets/images/general/deepseek-r1-locally-jan.jpg
twitter:
  card: summary_large_image
  site: "@jandotai"
  title: "Run DeepSeek R1 locally on your device (Beginner-Friendly Guide)"
  description: "A straightforward guide to running DeepSeek R1 locally regardless of your background."
  image: assets/deepseek-r1-locally-jan.jpg
---
import { Callout } from 'nextra/components'
import CTABlog from '@/components/Blog/CTA'

# Run DeepSeek R1 locally on your device (Beginner-Friendly Guide)



DeepSeek R1 is one of the best open-source models in the market right now, and you can run DeepSeek R1 on your own computer! 

<Callout type="info">
New to running AI models locally? Check out the [guide on running AI models locally](/post/run-ai-models-locally) first. It covers essential concepts that will help you better understand this DeepSeek R1 guide.
</Callout>

DeepSeek R1 requires data-center level computers to run at its full potential, and we'll use a smaller version that works great on regular computers.

Why use an optimized version?
- Efficient performance on standard hardware
- Faster download and initialization
- Optimized storage requirements
- Maintains most of the original model's capabilities

## Quick Steps at a Glance
1. Download [Jan](https://jan.ai/)
2. Select a model version
3. Choose settings
4. Set up the prompt template & start using DeepSeek R1

Let's walk through each step with detailed instructions.

## Step 1: Download Jan
[Jan](https://jan.ai/) is an open-source application that enables you to run AI models locally. It's available for Windows, Mac, and Linux. For beginners, Jan is the best choice to get started.



1. Visit [jan.ai](https://jan.ai)
2. Download the appropriate version for your operating system
3. Install the app

## Step 2: Choose Your DeepSeek R1 Version

To run AI models like DeepSeek R1 on your computer, you'll need something called VRAM (Video Memory). Think of VRAM as your computer's special memory for handling complex tasks like gaming or, in our case, running AI models. It's different from regular RAM - VRAM is part of your graphics card (GPU).

<Callout type="info">
Running AI models locally is like running a very sophisticated video game - it needs dedicated memory to process all the AI's "thinking." The more VRAM you have, the larger and more capable AI models you can run.
</Callout>

Let's first check how much VRAM your computer has. Don't worry if it's not much - DeepSeek R1 has versions for all kinds of computers!

Finding your VRAM is simple:
- On Windows: Press `Windows + R`, type `dxdiag`, hit Enter, and look under the "Display" tab
- On Mac: Click the Apple menu, select "About This Mac", then "More Info", and check under "Graphics/Displays"
- On Linux: Open Terminal and type `nvidia-smi` for NVIDIA GPUs, or `lspci -v | grep -i vga` for other graphics cards

<Callout>
**No dedicated graphics card?** That's okay! You can still run the smaller versions of DeepSeek R1. They're specially optimized to work on computers with basic graphics capabilities.
</Callout>

Once you know your VRAM, here's what version of DeepSeek R1 will work best for you. If you have:
- 6GB VRAM: Go for the 1.5B version - it's fast and efficient
- 8GB VRAM: You can run the 7B or 8B versions, which offer great capabilities
- 16GB or more VRAM: You have access to the larger models with enhanced features

Available versions and basic requirements for DeepSeek R1 distills:

| Version | Model Link | Required VRAM |
|---------|------------|---------------|
| Qwen 1.5B | [DeepSeek-R1-Distill-Qwen-1.5B-GGUF](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-1.5B-GGUF) | 6GB+ |
| Qwen 7B | [DeepSeek-R1-Distill-Qwen-7B-GGUF](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF) | 8GB+ |
| Llama 8B | [DeepSeek-R1-Distill-Llama-8B-GGUF](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF) | 8GB+ |
| Qwen 14B | [DeepSeek-R1-Distill-Qwen-14B-GGUF](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF) | 16GB+ |
| Qwen 32B | [DeepSeek-R1-Distill-Qwen-32B-GGUF](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF) | 16GB+ |
| Llama 70B | [DeepSeek-R1-Distill-Llama-70B-GGUF](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF) | 48GB+ |

To download your chosen model:

Launch Jan and navigate to Jan Hub using the sidebar



3. Input the model link in this field:



## Step 3: Configure Model Settings
When configuring your model, you'll encounter quantization options:

<Callout type="tip">
Quantization balances performance and resource usage:
- **Q4:** Recommended for most users - optimal balance of efficiency and quality
- **Q8:** Higher precision but requires more computational resources
</Callout>

## Step 4: Configure Prompt Template
Final configuration step:

1. Access Model Settings via the sidebar
2. Locate the Prompt Template configuration
3. Use this specific format:

<Callout type="warning">
```
<|User|>{prompt}<|Assistant|>
```
</Callout>

This template is for proper communication between you and the model.

You're now ready to interact with DeepSeek R1:



## Need Assistance?

<Callout type="info">
Join our [Discord community](https://discord.gg/Exe46xPMbK) for support and discussions about running AI models locally.
</Callout>

---

### Src Pages Post How We Benchmark Kernels (`docs/src/pages/post/how-we-benchmark-kernels.mdx`)

---
title: "How we (try to) benchmark GPU kernels accurately"
description: "We present the process behind how we decided to benchmark GPU kernels and iteratively improved our benchmarking pipeline"
tags: ""
categories: research
ogImage: assets/images/general/cover-kernel-benchmarking.png
date: 2025-09-17
---

# How we (try to) benchmark GPU kernels accurately
 
If you've read any other kernel benchmarking article before, then a lot of the information in this blogpost might be second-hand nature to you already. In fact, we would like to start off by thanking the blood, sweat and tears poured into many other kernel benchmarking guides made before us, which helped guide us in writing better benchmarking code, as well as in the creation of this blogpost.

Over here at Menlo, we recently acquired some [RTX PRO 6000 Blackwell Workstation Edition](https://www.nvidia.com/en-sg/products/workstations/professional-desktop-gpus/rtx-pro-6000/), and we are trying to make LLM inference engines like [vLLM](https://github.com/vllm-project/vllm) run faster on it. We've been writing our own kernels specifically for the RTX PRO 6000, and seeing if we can improve inference times on our hardware.

This blog will detail more of our own processes of how our ML Efficiency team identified problems in our benchmarking code, and how we iterated on it following the various cool benchmarking guides out there! Without further ado, let's start benchmarking from simple programs, to GPU kernels.


## Introduction to kernels and benchmarking

For those new to GPU programming, a [kernel](https://modal.com/gpu-glossary/device-software/kernel) is a piece of CUDA code that programmers write to execute a desired sequence of operations on the GPU. These kernels are launched once and is executed by threads that run concurrently, and we tend to launch these kernels from a [thread block grid](https://modal.com/gpu-glossary/device-software/thread-block-grid), which executes our kernel across multiple [Streaming Multiprocessors (SMs)](https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor) across the whole GPU.

Benchmarking is a fundamental aspect of high-performance computing. It enables us to quantitatively compare kernel performance across different problem sizes and understand how various hyperparameters impact execution speed. For GPU kernel development, benchmarking serves to assist us in iteratively optimizing our kernels to make them utilize the GPU better. 

That being said, **accurate kernel benchmarking** is much more important, as benchmarking kernels that run on the GPU can become very complex, and there are many traps one might fall into if not enough care is taken when writing benchmarking scripts. A great alternative is to use tools that NVIDIA offers via their [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit), such as [Nsight CUDA Profiling Tools Interface](https://developer.nvidia.com/cupti) (cupti) or use the [Nsight Compute CLI](https://docs.nvidia.com/nsight-compute/NsightComputeCli/index.html) (`ncu`), which provide accurate measurements to benchmarking various characteristics of kernels. For us, we wanted to use Python as it made it more convenient to sweep across different problem shapes and kernels quickly, but that meant we also had to learn how to properly benchmark kernels from scratch.

We will showcase some examples of how we can benchmark kernels on the GPU. Moreover, we have chosen Python for most of our benchmarking code, as most of our own codebase is in Python, making it simple to integrate in.

## Benchmarking CUDA programs
Pytorch provides a very basic API to help time `torch` programs, by following along this [tutorial](https://docs.pytorch.org/tutorials/recipes/recipes/benchmark.html).

We can see a basic implementation could be something as simple as:

```python
def batched_dot_mul_sum(a, b):
    '''Computes batched dot by multiplying and summing'''
    return a.mul(b).sum(-1)

num_threads = torch.get_num_threads()
print(f'Benchmarking on {num_threads} threads')

t0 = benchmark.Timer(
    stmt='batched_dot_mul_sum(x, x)',
    setup='from __main__ import batched_dot_mul_sum',
    globals={'x': x},
    num_threads=num_threads,
    label='Multithreaded batch dot',
    sub_label='Implemented using mul and sum')
```

When benchmarking kernels, there are a few tips that we should do, in order to ensure that we accurately benchmark our kernels.

### 1.  Always benchmark the code not with settings from your machine, but with **settings the user will see**.
  Benchmarking how fast your kernels take on a 3090 is meaningless if you are serving your models on a H100 DGX node. It is always a good idea to benchmark your kernels on the hardware you plan to serve on.

### 2. **Warmup your kernels** 
Take a look at this snippet from the tutorial.
  ```
   mul_sum(x, x):   27.6 μs
   mul_sum(x, x):   25.3 μs
   bmm(x, x):      2775.5 μs
   bmm(x, x):       22.4 μs
  ```
  The first kernel for `bmm` takes way longer to run. This is because most of the time is spent loading [cuBLAS](https://developer.nvidia.com/cublas) kernels when running for the first time. 

  Warming up your kernels can be as simple as running the kernel before timing it. This helps to preliminarily load these kernels so that we only measure how long it takes for the kernel to run.

### 3. `torch.cuda.synchronize` and CUDA Events

Now we'll also introduce a new API, which is the standard to benchmarking kernels. [CUDA events](https://docs.pytorch.org/docs/stable/generated/torch.cuda.Event.html) are awesome for a variety of reasons. The simplest reason is that it measures timing from the perspective from the GPU, whereas `time.time()` and `time.perf_counter()` measure time from the perspective of the CPU.

Moreover, its simplistic API allows you to call benchmarking code like this:

```python
steps = 10
start_events = [torch.cuda.Event(enable_timing=True) for _ in range(steps)]
end_events = [torch.cuda.Event(enable_timing=True) for _ in range(steps)]

for i in range(steps):
    start_events[i].record()
    run_kernel()
    end_events[i].record()

torch.cuda.synchronize()
times = [s.elapsed_time(e) for s, e in zip(start_events, end_events)]
```

The `torch.cuda.synchronize` tells the CPU to wait for the work on the GPU to finish, so that it can calculate the elapsed time after synchronization, which can be visualised here:


_Figure 1: Illustration taken from https://www.speechmatics.com/company/articles-and-news/timing-operations-in-pytorch_


### 4. Flush your L2 Cache

#### What is the L2 Cache
When data is read or written from [HBM or GDDR](https://www.exxactcorp.com/blog/hpc/gddr6-vs-hbm-gpu-memory), it goes through the [L2 cache](https://docs.nvidia.com/cuda/cuda-c-programming-guide/#architecture) first, which is shared by all [streaming multiprocessors (SM)](https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor). This L2 cache caches data accesses to local and global memory, and helps reuse data instead of loading it to shared memory again (which can be slow!).

Moreover, unlike the L1 cache, which is present on each SM, all SMs share the same L2 cache!


#### Why we need to flush the L2 Cache

Following this [guide](https://guillesanbri.com/CUDA-Benchmarks/#caches), if you had warmed up or run the kernel before, some of the intermediate data might be stored on the L2 cache, which means the kernel might be deceptively faster. 

However, in a real world setting, you want to measure the time it takes realistically for the kernel to run, and more often than not, when running large models, you will be running more than 1 kernel. This means your cache will probably thrash often, and not have data from a specific kernel stored for reuse. Thus, to simulate this behaviour, we would flush the L2 cache beforehand to eliminate "help" from the L2 cache.

Moreover, this also makes it much easier when calculating data reuse for the kernel, as now any L2 cache usage is independent of other kernels or runs.

#### Example of not flushing L2 cache
Previously when we were initially benchmarking our kernels, we had a small mistake of not flushing the L2 cache.


_Figure 2: Our SOL %  (which is a percentage of our observed maximum speed) is over 100% for the row for shape [2, 19456, 2560]._

#### How to flush the L2 Cache

To flush it, we should add the following lines:

```python
l2_size = torch.cuda.get_device_properties().L2_cache_size
cache = torch.empty(l2_size, dtype=torch.uint8, device="cuda")
#<your benchmarking code here>
cache.zero_()  # flush L2 cache

# You should flush your L2 cache within the benchmarking code if you're repeating the same process multiple times
```

This instantiates data the size of the L2 cache, and by zeroing it in place, we call a write operation which goes through the L2 cache and flushes it.

After flushing the L2 cache, we get a more sensible result here:


_Figure 3: New SOL% has all values under 100% now after flushing L2 cache._

### 5. Timing short-lived kernels

Initially, we used [Triton's](https://triton-lang.org/main/getting-started/installation.html) [`do_bench`](https://triton-lang.org/main/python-api/generated/triton.testing.do_bench.html) for benchmarking, as it has done everything we have mentioned above, such as warmup, CUDA Events and flushing L2 cache. However, we observed an issue with accurately benchmarking our kernels on smaller shapes. On smaller shapes, the kernel might be too fast, so it may finish before CPU issues a CUDA end event in Python.


_Figure 4: Taken from [Speechmatics](https://www.speechmatics.com/company/articles-and-news/timing-operations-in-pytorch), kernel is faster than CUDA event end launch, and therefore the true timing for the kernel is not recorded._

This results in kernels that look very slow:


_Figure 5: Side by side comparison of Python benchmark latencies vs `ncu`'s timing (right) for shape [2, 19456,2560]. `ncu` records a much faster duration of 71.36 μs compared to Python's 103.9 μs_

To fix this, we wrote a custom `do_bench_cuda()`, that inserts a dummy, untimed FP32 matmul before benchmarking each shape, so that the CPU has enough time to enqueue the CUDA end event.

This led to more accurate latencies for our small M kernels.


_Figure 6: There is a significant improvement in SOL% after inserting the dummy matmul._

We then also repeat the benchmark function for each shape on 5 copies of input/output data to make the CUDA event duration longer.

In the end, this is our `do_bench_cuda` function that we used to benchmark our kernels.

```python
import statistics

import torch

def do_bench_cuda(f, n_warmup: int = 10, n_repeats: int = 20):
    l2_size = torch.cuda.get_device_properties().L2_cache_size
    cache = torch.empty(l2_size, dtype=torch.uint8, device="cuda")

    # Matmul in case of short lived CUDA Events
    A = torch.randn(4096, 4096, dtype=torch.float32, device="cuda")
    B = torch.randn(4096, 4096, dtype=torch.float32, device="cuda")
    A @ B

    # L2 cache flush + Warmup
    for _ in range(n_warmup):
        cache.zero_()
        f()

    start_list = [torch.cuda.Event(enable_timing=True) for _ in range(n_repeats)]
    end_list = [torch.cuda.Event(enable_timing=True) for _ in range(n_repeats)]

    torch.cuda.synchronize()

    for start, end in zip(start_list, end_list):
        cache.zero_()  # flush L2 cache
        A @ B  # add a heavy task to fill GPU pipeline

        start.record()
        f()
        end.record()

    torch.cuda.synchronize()
    timings = [start.elapsed_time(end) for start, end in zip(start_list, end_list)]
    return statistics.median(timings)
```



### 6. Clock Speed

This was a silent problem, and it was very hard to find out that this was causing issues on our kernels. We initially found discrepancies between `ncu`'s latency (676.64 μs) and `do_bench_cuda` latency (535 μs) when profiling on the shape [2048, 19456, 2560], as `do_bench` was reporting ~140 μs faster timings than `ncu`'s latency.

As seen, although most of our codebase for benchmarking kernels in Python, developer errors might occur, and it is always good to have a point of reference for the accurate kernel timing. Nsight Compute CLI (`ncu` for short) is a tool that can help us measure the latency of our kernels accurately, and the values obtained from here is a good figure to sanity-check our own benchmarking code. 

#### 6.1 Clock Speed

Firstly, we suspected that clock speed could play a part in causing the discrepancy between `ncu`'s timings and our own benchmarking code. Clock speed can affect benchmarking times as it is the rate at which the GPU's processing units operate, and a higher clock speed translates to more operations per second, which can both speed up and slow down the kernel depending on how it was implemented.


_Figure 7: Taken from [GPU Mode Lecture 56](https://www.youtube.com/watch?v=CtrqBmYtSEk). We can see clock speed affects kernel performance. For problem shape of 1024, it got faster after increasing clock speed, while for problem shape of 384, it became slower after clock speed increased._

Looking at this [forum post](https://forums.developer.nvidia.com/t/nsight-compute-clock-speed-during-profiling/208646/3), we realised that one of the issues causing the discrepancy was because `ncu` by default locks the clock speed to the GPU base clock speed. We tried investigating by locking the clock speed to base clock speed, and also tried locking to max clock speed using `nvidia-smi -ac=<memClk>,<smClk>`. According to the GPU Mode lecture, this was not a proper solution.

This is due to the following reasons:
- Locking to max clock speed doesn't help as it just sets the ceiling of the GPU performance, our GPU can always go back to base clock speed of ~ 2287 Hz instead of boosted clock speed of 2617 Hz.

- Locking to base clock speed is also not meaningful, as it does not properly reflect the performance and experience users will get on our kernels, which at best will run on boosted clock speed.

However, we did find out that we should set the `ncu` `--clock-control` to `None`, so that it would not limit itself to just the base clock speeds. This helped improve the latency on `ncu` from 676.64 μs to 575 μs when profiled on the same problem shape  of [2048, 19456, 2560].

#### 6.2 Discrepancies after `clock-control`

At the time of writing, we have observed that `ncu` sometimes gives different latency results on the same benchmarking code with the same problem shapes. The cause of this is because when we set `clock-control` speed to `None`, the GPU clock speed is stochastic, and therefore affect the latency of the kernels measured. A more holistic approach would be to also benchmark kernels across different fixed clock speeds. 

_Figure 8: On the same benchmarking code and problem shapes, we can see vast deviations in duration, which is caused by the differences in SM Frequency. This resonates with the graphs shown in Figure 7._

As a result, there can be some discrepancy in `ncu`'s and our own benchmark timings. To figure out if your discrepancy is caused by the SM frequency, you can use the relationship
that FLOPS is directly proportional to SM clock, so their durations are inversely proportional.

In our case:
`544 / 2.14 (SM freq for 575 μs kernel) * 2.28 (SM freq for 544 μs kernel) = ~579`, so most of the discrepancy was coming from the SM frequency being different. 

The final command we used was:

`ncu -s 5 -k $kernel_name --clock-control none python3 benchmarks/bench_mm.py --profile 2048 19456 2560`

Explanation of arguments:
`-s`: Number of kernels skipped 
`-k`: name of kernel to profile
`--clock-control`: Whether clock speed is controlled 

Below is a side to side comparison of `ncu`'s benchmarked latency and our script after all the adjustments made.


_Figure 9: Side by side comparison of the above `ncu` command (left) (measuring shape [2048,19456,2560]) with our own Python benchmarking script (right). We can see at most 10us difference between `Duration` in `ncu` and our benchmarking script's `Latency (us)` measurement._

## Conclusion and TLDR;

TLDR, when benchmarking
1. Make sure to use the hardware you intend to deploy on
2. Warmup before benchmarking your kernels
3. Use CUDA events
4. Flush your L2 Cache
5. Use a dummy matmul to make timings more accurate for short lived kernels
6. Ensure your clock speed doesn't cause inconsistent readings

We hope that this helps anyone who are interested in benchmarking their own kernels, or are interested in how GPU kernels are benchmarked. Happy benchmarking!

### Acknowledgements and Related Resources:

We would like to give thanks and credit to the many resources and guides that we used in our own journey of finding out how to best benchmark our kernels on our GPUs, and a lot of our work would probably not be possible if it was not for these amazing guides.

- GPU Mode Lecture 56, presented by Georgii Evtushenko : https://www.youtube.com/watch?v=CtrqBmYtSEk
- https://www.spatters.ca/mma-matmul (Benchmarking using ncu for matrix multiplications)
- https://www.speechmatics.com/company/articles-and-news/timing-operations-in-pytorch (CUDA Events)
- https://guillesanbri.com/CUDA-Benchmarks/ (Good resource for introduction to benchmarking)
- https://modal.com/gpu-glossary/device-hardware/cuda-device-architecture (Glossary of Architecture in general)
- https://docs.nvidia.com/cuda/cuda-c-programming-guide/#global-memory-5-x (L2 cache explanation)
- https://cvw.cac.cornell.edu/gpu-architecture/gpu-memory/memory_types (L1 vs L2 cache)

---

### Src Pages Post Is Chatgpt Down Use Jan (`docs/src/pages/post/is-chatgpt-down-use-jan.mdx`)

---
title: "If ChatGPT is down, switch to AI that never goes down"
description: "Check if ChatGPT down right now, and learn how to use AI that never goes down."
tags: AI, ChatGPT down, ChatGPT alternative, Jan, local AI, offline AI, ChatGPT at capacity
categories: guides
date: 2025-09-30
ogImage: assets/images/general/is-chatgpt-down.jpg
twitter:
  card: summary_large_image
  site: "@jandotai"
  title: "Realtime Status: Is ChatGPT down?"
  description: "Check if ChatGPT is down right now with our real-time status checker, and learn how to use AI that never goes offline."
  image: assets/images/general/is-chatgpt-down.jpg
---
import { Callout } from 'nextra/components'
import CTABlog from '@/components/Blog/CTA'
import { OpenAIStatusChecker } from '@/components/OpenAIStatusChecker'

# If ChatGPT is down, switch to AI that never goes down

If you're seeing ChatGPT is down, it could be a good signal to switch to [Jan](https://www.jan.ai/), AI that never goes down.

## 🔴 Realtime Status: Is ChatGPT down?
<Callout>
This live tracker shows if ChatGPT is down right now. 
</Callout>
<OpenAIStatusChecker />

### ChatGPT Status Indicators

<div className="grid grid-cols-1 md:grid-cols-2 gap-4 my-6">
  <div className="p-4 rounded-lg border border-green-200 bg-green-50 dark:bg-green-900/20 dark:border-green-800">
    <div className="flex items-center gap-2 mb-2">
      <div className="w-3 h-3 bg-green-500 rounded-full"></div>
      <span className="font-semibold text-green-800 dark:text-green-300">Operational</span>
    </div>
    <p className="text-sm text-green-700 dark:text-green-400">All systems are functioning normally with no reported issues.</p>
  </div>
  
  <div className="p-4 rounded-lg border border-yellow-200 bg-yellow-50 dark:bg-yellow-900/20 dark:border-yellow-800">
    <div className="flex items-center gap-2 mb-2">
      <div className="w-3 h-3 bg-yellow-500 rounded-full"></div>
      <span className="font-semibold text-yellow-800 dark:text-yellow-300">Degraded Performance</span>
    </div>
    <p className="text-sm text-yellow-700 dark:text-yellow-400">Services are running but may be slower than usual.</p>
  </div>
  
  <div className="p-4 rounded-lg border border-orange-200 bg-orange-50 dark:bg-orange-900/20 dark:border-orange-800">
    <div className="flex items-center gap-2 mb-2">
      <div className="w-3 h-3 bg-orange-500 rounded-full"></div>
      <span className="font-semibold text-orange-800 dark:text-orange-300">Partial Outage</span>
    </div>
    <p className="text-sm text-orange-700 dark:text-orange-400">Some features or regions may be experiencing issues.</p>
  </div>
  
  <div className="p-4 rounded-lg border border-red-200 bg-red-50 dark:bg-red-900/20 dark:border-red-800">
    <div className="flex items-center gap-2 mb-2">
      <div className="w-3 h-3 bg-red-500 rounded-full"></div>
      <span className="font-semibold text-red-800 dark:text-red-300">Major Outage</span>
    </div>
    <p className="text-sm text-red-700 dark:text-red-400">Significant service disruption affecting most users.</p>
  </div>
</div>

## Skip the downtime with Jan

When ChatGPT is down, Jan keeps working. Jan is an open-source ChatGPT alternative that runs on your computer - no servers, no outages, no waiting.


*Jan works even when ChatGPT doesn't.*

### Why Jan never goes down:
- **Runs locally** - No dependency on external servers
- **Always available** - Works offline, even on flights
- **No capacity limits** - Uses your computer's resources
- **100% private** - Your conversations stay on your device

### Get started in 3 mins:
1. Download Jan: [jan.ai](https://jan.ai)
2. Install a model: Choose from Jan, Qwen, or other top models
3. Start chatting: Similar design as ChatGPT, but always available if you use local models

<Callout type="info">
**Pro tip:** Keep both ChatGPT and Jan. You'll never lose productivity to outages again.
</Callout>

Jan runs AI models locally, so you don't need internet access. That means Jan is unaffected when ChatGPT is down.

### Why does ChatGPT goes down?
There could be multiple reasons:
- Too many users at once
- Data center or API downtime
- Planned or uplanned updates 
- Limited in some locations

ChatGPT depends on OpenAI’s servers. If those go down, so does ChatGPT. Jan users don't affect by ChatGPT's outage.

### Common ChatGPT Errors

When ChatGPT experiences issues, you might see these error messages:

- "ChatGPT is at capacity right now": Too many users online, try again later
- "Error in message stream": Connection problems with OpenAI servers
- "Something went wrong": General server error, refresh and retry
- "Network error": Internet connectivity issues on your end or OpenAI's
- "Rate limit exceeded": Too many requests sent, wait before trying again
- "This model is currently overloaded": High demand for specific model

## Quick answers about ChatGPT status

### Is ChatGPT down?
Check the ChatGPT realtime status above. [See if ChatGPT is down right now.](http://localhost:3000/post/is-chatgpt-down-use-jan#-realtime-status-is-chatgpt-down)

### Why is ChatGPT down?  
Usually server overload, maintenance, or outages at OpenAI.  

### What does "ChatGPT is at capacity" mean?  
Too many users are online at the same time. You’ll need to wait or switch to Jan instead.  

### Is ChatGPT shutting down?  
No, ChatGPT isn't shutting down. Outages are temporary.  

### Can I use ChatGPT offline?  
No. ChatGPT always requires internet. For [offline AI](https://www.jan.ai/post/offline-chatgpt-alternative), use [Jan](https://jan.ai).

---

### Src Pages Post Jan V1 For Research (`docs/src/pages/post/jan-v1-for-research.mdx`)

---
title: "Jan v1 for Deep Research: System Prompts & Setup Guide"
description: "Explore Jan-V1 capabilities in report generation and research tasks with prompt comparisons, examples, and customization instructions."
keywords: ["Jan-V1", "AI research", "system prompts", "LLM optimization", "research AI", "Jan App", "model configuration"]
readingTime: "8 min read"
tags: Qwen, Jan-V1, Agentic
categories: research
ogImage: assets/images/general/og-jan-research.jpeg
date: 2025-08-22
---

# Jan v1 for Deep Research: System Prompts & Setup Guide

This cookbook will transform your Jan-V1 from a basic Q&A tool into a comprehensive research assistant. By the end of this guide, you'll have a custom-configured model that generates detailed reports with proper citations instead of surface-level answers.



## Key Points

- **Jan-V1 includes a default chat template** that's automatically embedded in its Hugging Face configuration
- **Use the default prompt** for daily tasks requiring short, accurate answers
- **Use the research prompt** for report generation and comprehensive research tasks
- **Always specify dates** when asking time-sensitive questions (e.g., "What's the world population in 2023?")

---

## Introduction

[Jan-V1](https://huggingface.co/janhq/Jan-v1-4B) is the first release in the **Jan Family**, designed for agentic reasoning and problem-solving within the [Jan App](https://jan.ai/). Based on our [**Lucy**](https://huggingface.co/Menlo/Lucy) model, Jan-v1 achieves improved performance through model scaling.

Jan-v1 uses the [Qwen3-4B-thinking](https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507) model to provide enhanced reasoning capabilities and tool utilization. This architecture delivers better performance on complex agentic tasks.

This guide explores how to optimize Jan-V1 for research-intensive tasks by switching from the default prompt to a specialized research system prompt. Through our internal testing, we've found significant improvements in report generation quality, depth of analysis, and citation accuracy when using the research-optimized system prompt.


## Findings

Jan-V1 comes with a built-in search and scrape prompt in its default template. This prompt is optimized for SimpleQA-style questions and daily tasks that require short, accurate responses. However, our testing revealed that when tasked with generating comprehensive long-form reports or conducting in-depth research, the model's performance was inconsistent with the default prompt.

We experimented with Jan-V1 using a new system prompt optimized for research-intensive tasks and found significant improvements in report generation capability. The research prompt produces more comprehensive, well-structured reports with proper citations and deeper analysis compared to the default prompt.

---

## Output comparison between Default System Prompt and Research Prompt on report generation task

### Example 1: Long-Context Benchmark Comparison

**User Request:**
> Compare all published benchmarks for long-context performance across LLMs released in 2024-2025


<div className="grid grid-cols-1 lg:grid-cols-2 gap-6 mb-8">
  
{/* Default Prompt Column */}
<div className="bg-gray-50 dark:bg-gray-900 p-6 rounded-lg border">

<h4 className="font-semibold mb-4 text-lg border-b border-gray-300 dark:border-gray-600 pb-2">Default Prompt Response</h4>
<div className="prose prose-xs max-w-none text-xs mt-4 h-[500px] overflow-y-auto [&_h1]:text-sm [&_h2]:text-xs [&_h3]:text-xs [&_h4]:text-xs [&_h5]:text-xs [&_h6]:text-xs [&_p]:text-xs [&_li]:text-xs [&_td]:text-xs [&_th]:text-xs">

I've analyzed multiple benchmarks for long-context LLM performance published in 2024-2025 through comprehensive scraping and search results. Here's a structured comparison of the major benchmarks:

#### 1. Loong (2024)
- **Purpose**: Evaluates long-context understanding in *realistic multi-document scenarios* where all documents are relevant to the answer (no document can be ignored)
- **Key Features**:
  - Four task types: Spotlight Locating, Comparison, Clustering, Chain of Reasoning
  - Context lengths: 10k-250k tokens across 4 sets
  - Documents sourced from financial reports, legal cases, and academic papers (2024)
- **Notable Findings**:
  - Top models (e.g., Gemini-1.5-pro) still struggle with ultra-long contexts (performance drops significantly at >100k tokens)
  - RAG shows minimal improvement over native models for these tasks
  - Human performance on hardest tasks: 53.7% accuracy with 15-minute time limit

#### 2. Ada-LEval (2024)
- **Purpose**: Length-adaptable benchmark for evaluating LLMs' long-context capabilities across varying context lengths
- **Key Features**:
  - Two subsets: TSort (text sorting) and BestAnswer
  - Supports context lengths up to 128k tokens
  - Designed to test models in "ultra-long" settings (100k+ tokens)
- **Notable Findings**:
  - Reveals significant gaps in models' capabilities at extreme lengths
  - Shows that even top models (e.g., GPT-4o) have limitations in ultra-long contexts

#### 3. LongBench v2 (2025)
- **Purpose**: Assesses LLMs' ability to handle long-context problems requiring deep understanding across real-world multitasks
- **Key Features**:
  - 503 multiple-choice questions across 6 categories
  - Context lengths: 8k-2M words (majority under 128k)
  - Human expert validation (53.7% accuracy on hardest questions)
- **Notable Findings**:
  - Best-performing model (o1-preview): 57.7% accuracy (vs. human baseline of 53.7%)
  - Clear performance degradation at longer contexts (e.g., 128k+ tokens)
  - Leaderboard shows models like Gemini-2.5-Pro and DeepSeek-R1 outperform others

#### 4. Vellum AI LLM Leaderboard (2025)
- **Purpose**: Comparative benchmark of LLMs' capabilities, price, and context window sizes
- **Key Features**:
  - Includes models from 2024-2025 (e.g., GPT-4o, Gemini-2.5-Pro, Qwen3-235B)
  - Focuses on practical metrics like context window size and real-world performance
- **Notable Findings**:
  - Highlights the gap between claimed context window sizes and actual performance
  - Shows that most models (even those with "1M" context windows) show performance drops beyond 128k tokens

#### Cross-Benchmark Comparison Summary

| Benchmark | Release Year | Key Innovation | Context Length Range |
|-----------|---------------|----------------|----------------------|
| Loong | 2024 | No-document-ignored multi-document QA | 10k-250k tokens |
| Ada-LEval | 2024 | Length-adaptive task subsets | Up to 128k tokens |
| LongBench v2 | 2025 | Real-world multitask evaluation | 8k-2M words |
| Vellum Leaderboard | 2025 | Context window vs. real performance | Varies |

#### Critical Insights from All Benchmarks
1. **Performance Decline at Scale**: All benchmarks show significant performance degradation beyond 100-128k tokens, indicating that current models don't fully leverage their claimed context windows
2. **Realism Gap**: Benchmarks like Loong and LongBench v2 reveal that models struggle with *realistic* multi-document scenarios (where evidence is scattered across documents) better than synthetic benchmarks
3. **RAG Limitations**: RAG implementations show minimal benefit for these tasks (Loong results indicate RAG often fails to retrieve all relevant documents)
4. **Model Variability**: Larger models (e.g., 235B+ parameter) show better performance but still fall short in ultra-long contexts compared to human capabilities

This comparison demonstrates that while 2024-2025 benchmarks have advanced long-context evaluation, there remains a significant gap between theoretical context window capabilities and practical performance in real-world multi-document scenarios.

    </div>
  </div>

  {/* Research Prompt Column */}
<div className="bg-blue-50 dark:bg-blue-950 p-6 rounded-lg border border-blue-200 dark:border-blue-800">
<h4 className="font-semibold mb-4 text-lg border-b border-gray-300 dark:border-gray-600 pb-2">Research Prompt Response</h4>
<div className="prose prose-xs max-w-none text-xs mt-4 h-[500px] overflow-y-auto [&_h1]:text-sm [&_h2]:text-xs [&_h3]:text-xs [&_h4]:text-xs [&_h5]:text-xs [&_h6]:text-xs [&_p]:text-xs [&_li]:text-xs [&_td]:text-xs [&_th]:text-xs">

# Comparative Analysis of Long-Context LLM Benchmarks (2024-2025)

## Overview

This report provides a comprehensive comparison of published long-context LLM benchmarks released during 2024-2025. Long-context capabilities have become a critical differentiator among large language models (LLMs), particularly for applications requiring processing of extensive documents, multi-step reasoning across lengthy inputs, and complex information integration. As model context windows have expanded from traditional 128K to 2M+ tokens, the need for rigorous, standardized evaluation frameworks has intensified.

## Key Benchmarks Identified

The following four benchmarks represent the most comprehensive and widely cited long-context evaluation frameworks from 2024-2025:

| Benchmark | Release Date | Context Length Range | Primary Focus |
|-----------|---------------|----------------------|----------------|
| LongBench v2 | May 2025 | 8k - 2M words | Real-world long-context understanding |
| MIR-Bench | Feb 2025 | Variable (up to 10k+ shots) | Many-shot in-context inductive reasoning |
| GSM-∞ | Feb 2025 | 0 - 16M+ tokens | Infinitely scalable reasoning complexity |
| Vellum AI Leaderboard 2025 | April 2025 | Up to 2M tokens | Cross-benchmark model comparison |

## Key Findings and Implications

Based on the comprehensive analysis of these benchmarks, several important findings emerge:

1. **Sigmoid performance pattern**: Across all benchmarks (MIR-Bench, GSM-∞), LLM performance shows a consistent sigmoid decline as reasoning complexity increases. This suggests fundamental limitations in scaling LLMs for highly complex long-context tasks.

2. **Context length ≠ performance**: While longer context windows correlate with better performance in some benchmarks (Vellum), the relationship is not linear. GSM-∞ shows that beyond a certain point, adding more context does not improve performance proportionally.

3. **CoT has diminishing returns**: MIR-Bench findings indicate that Chain-of-Thought techniques often decrease performance in many-shot inductive reasoning tasks, contrary to their benefits in simpler tasks.

4. **Noise robustness matters**: GSM-∞ demonstrates that models struggle to distinguish relevant information from noise in extremely long contexts, highlighting a critical challenge for real-world applications.

5. **Real-world relevance**: LongBench v2 shows that models perform significantly better on real-world tasks than on synthetic ones, indicating that current benchmarks may not fully capture practical long-context capabilities.

## Conclusion

The landscape of long-context LLM benchmarks in 2024-2025 reveals both progress and persistent challenges. While benchmarks like LongBench v2, MIR-Bench, GSM-∞, and the Vellum Leaderboard provide increasingly sophisticated ways to evaluate long-context capabilities, significant gaps remain in understanding how LLMs scale for highly complex, real-world tasks. Future work should focus on:

1. Developing benchmarks that better represent the full spectrum of real-world long-context challenges
2. Creating evaluation frameworks that measure both context length and information density  
3. Establishing standardized methods to assess noise robustness and inductive reasoning capabilities

These benchmarks collectively provide the foundation for continued progress in long-context LLM development, with their findings guiding researchers and practitioners toward more effective model design and evaluation.

## References

[1] Bai, Y., Tu, S., Zhang, J., et al. (2025). LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. https://longbench2.github.io/

[2] Yan, K., Chen, Z., & Tian, Y. (2025). MIR-Bench: Benchmarking LLM's Long-Context Intelligence via Many-Shot In-Context Inductive Reasoning. arXiv preprint arXiv:2502.09933.

[3] Liu, H., Chen, Z., & Tian, Y. (2025). GSM-∞: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity? arXiv preprint arXiv:2502.05252.

[4] Vellum AI. (2025). LLM Leaderboard 2025. https://www.vellum.ai/llm-leaderboard

    </div>
  </div>
</div>

### Example 2: AI Model Comparison

**User Request:**
> Explain the difference between different AI models: 1) Jan-Nano 2) Lucy 3) Jan-V1
<div className="grid grid-cols-1 lg:grid-cols-2 gap-6 mb-8">

{/* Default Prompt Column */}
<div className="bg-gray-50 dark:bg-gray-900 p-6 rounded-lg border">
<h4 className="font-semibold mb-4 text-lg border-b border-gray-300 dark:border-gray-600 pb-2">Default Prompt Response</h4>
<div className="prose prose-xs max-w-none text-xs mt-4 h-[500px] overflow-y-auto [&_h1]:text-sm [&_h2]:text-xs [&_h3]:text-xs [&_h4]:text-xs [&_h5]:text-xs [&_h6]:text-xs [&_p]:text-xs [&_li]:text-xs [&_td]:text-xs [&_th]:text-xs">

The differences between Jan-Nano, Lucy, and Jan-V1 can be summarized as follows based on available information:

1. **Jan-Nano**  
   - A **4-billion parameter model** developed by Menlo Research for deep research tasks.  
   - Achieves **83.2% accuracy** on the SimpleQA benchmark (per arXiv technical report).  
   - Optimized for local deployment with tools like MCP (Multi-Channel Protocol) for web search.  
   - Has variants like *Jan-Nano-128k*, which features a native 128k context window for improved research capabilities.

2. **Lucy**  
   - A **1.7-billion parameter model** created by Menlo Research as a lightweight, mobile-friendly variant.  
   - Designed for **agentic web search** and lightweight browsing (e.g., using tools like Serper and Crawl4AI).  
   - Benchmarked at **78.3% accuracy** on SimpleQA with MCP, slightly below Jan-Nano (80.7%), but optimized for mobile/CPU efficiency.  
   - Built on Qwen3-1.7B with RLVR (Reinforcement Learning with Value Functions) and "machine-generated task vectors" to enhance reasoning.

3. **Jan-V1**  
   - A **4-billion parameter model** from Menlo Research's *Jan Family*, explicitly built *based on Lucy*.  
   - Achieves **91.1% accuracy** on SimpleQA, outperforming Jan-Nano by ~8 percentage points.  
   - Focuses on **agentic reasoning and problem-solving** via enhanced tool integration (e.g., for complex tasks).  
   - Targets seamless integration with the Jan App for local, high-performance research workflows.

### Key Distinctions:
- **Size & Use Case**: Lucy is the smallest (1.7B) and mobile-optimized; Jan-Nano and Jan-V1 are both 4B but Jan-V1 is a scaled-up version of Lucy for higher performance.
- **Performance**: Jan-V1 > Jan-Nano > Lucy in SimpleQA benchmarks (91.1% → 83.2% → 78.3%).
- **Technical Approach**: Lucy uses RLVR with task vectors for efficiency; Jan-V1 inherits this but scales it for better accuracy and tool utilization.
- **Context Window**: Jan-Nano-128k offers a larger context window (128k) than base Jan-Nano, but Lucy is designed for mobile constraints.

This hierarchy reflects Menlo Research's progression from lightweight mobile models (Lucy) to scalable research tools (Jan-Nano and Jan-V1).

</div>
</div>

{/* Research Prompt Column */}
<div className="bg-blue-50 dark:bg-blue-950 p-6 rounded-lg border border-blue-200 dark:border-blue-800">
<h4 className="font-semibold mb-4 text-lg border-b border-gray-300 dark:border-gray-600 pb-2">Research Prompt Response</h4>
<div className="prose prose-xs max-w-none text-xs mt-4 h-[500px] overflow-y-auto [&_h1]:text-sm [&_h2]:text-xs [&_h3]:text-xs [&_h4]:text-xs [&_h5]:text-xs [&_h6]:text-xs [&_p]:text-xs [&_li]:text-xs [&_td]:text-xs [&_th]:text-xs">

# Differences Between Jan-Nano, Lucy, and Jan-V1 AI Models: A Comparative Analysis

## Overview
This report provides a detailed comparison of three AI language models: **Jan-Nano**, **Lucy**, and **Jan-V1**. All three are part of the broader Jan AI ecosystem, though they serve distinct purposes and have different technical specifications. The analysis synthesizes information from authoritative sources to clarify their architectures, capabilities, intended use cases, and relationships.

## Key Findings

### 1. **Jan-Nano**
- **Parameters**: 4 billion (4B)  
- **Primary Purpose**: Deep research tasks  
- **Notable Feature**: Native 128k context window in its enhanced version (Jan-Nano 128k)  

### 2. **Lucy**
- **Parameters**: 1.7 billion (1.7B)  
- **Primary Purpose**: Mobile-capable search and lightweight browsing tasks  
- **Notable Feature**: Designed as a "toy model" for search and browsing with minimal resource requirements  

### 3. **Jan-V1**
- **Parameters**: 4 billion (4B)  
- **Primary Purpose**: Agentic reasoning and problem-solving via web search and tool invocation  
- **Notable Feature**: Open-source alternative to Perplexity Pro with 91% SimpleQA accuracy  

## Technical and Functional Differences

| **Model** | **Parameter Count** | **Specialized For**                  | **Key Differentiator**                          |
|-----------|---------------------|--------------------------------------|------------------------------------------------|
| Jan-Nano  | 4B                   | Deep research tasks                 | 128k context window (enhanced version)          |
| Lucy      | 1.7B                 | Mobile search and lightweight browsing | Ultra-low resource footprint; "toy model" design |
| Jan-V1    | 4B                   | Agentic reasoning & web search      | Built on Lucy; 91% SimpleQA accuracy; open-source |


### Performance and Contextual Capabilities
- **Jan-Nano**'s 128k context window allows it to process longer documents or complex queries than standard models.  
- **Lucy**'s 1.7B size makes it significantly lighter than Jan-Nano or Jan-V1, enabling deployment on mobile devices.  
- **Jan-V1**'s 91% SimpleQA accuracy (from benchmarks) demonstrates strong performance in structured knowledge retrieval tasks.

## References
1. Jan-Nano: [Menlo/Jan-nano on Hugging Face](https://huggingface.co/Menlo/Jan-nano)  
2. Lucy: [Reddit discussion on Lucy as a 1.7B model](https://www.reddit.com/r/LocalLLaMA/comments/1m2tjjc/lucy_a_mobilecapable_17b_reasoning_model_that/)  
3. Jan-V1: [Jan-v1 on Hugging Face](https://huggingface.co/janhq/Jan-v1-4B)  
4. Jan-V1: [Jan's X announcement about Jan-v1](https://x.com/jandotai/status/1955176280535732415)  
5. Jan-V1: [Reddit comparison of Jan-v1 with Qwen3 and Perplexity](https://www.reddit.com/r/LocalLLaMA/comments/1mpmeba/janv1_trial_results_followup_and_comparison_to/)

</div>
</div>
</div>






### Example 3: Software Comparison

**User Request:** "Compare Signal, Telegram, WhatsApp for secure messaging"

<div className="grid grid-cols-1 lg:grid-cols-2 gap-6 mb-8">

{/* Default Prompt Column */}
<div className="bg-gray-50 dark:bg-gray-900 p-6 rounded-lg border">
<h4 className="font-semibold mb-4 text-lg border-b border-gray-300 dark:border-gray-600 pb-2">Default Prompt Response</h4>
<div className="prose prose-sm max-w-none text-sm mt-4">

**Security Overview:**
- Signal: E2EE by default, open-source
- Telegram: E2EE only in Secret Chats  
- WhatsApp: E2EE but Meta data collection
- Basic privacy comparison
- 200 words, single citation

**Strengths:** Clear recommendations  
**Weaknesses:** Limited analysis depth

</div>
</div>

{/* Research Prompt Column */}
<div className="bg-blue-50 dark:bg-blue-950 p-6 rounded-lg border border-blue-200 dark:border-blue-800">
<h4 className="font-semibold mb-4 text-lg border-b border-gray-300 dark:border-blue-800 pb-2">Research Prompt Response</h4>
<div className="prose prose-sm max-w-none text-sm mt-4">

**Comprehensive Security Analysis:**
- Full report with methodology and detailed comparison table
- Academic research citations (IACR 2023 study)
- Privacy practices analysis with breach history
- User experience and practical considerations
- Final recommendations table by use case
- 1000+ words, 5 authoritative references

**Strengths:** Evidence-based analysis, academic rigor  
**Result:** Professional security assessment format

</div>
</div>
</div>

---

## How to use Research Prompt with Jan-V1

Jan-V1 comes with an [embedded chat template](https://huggingface.co/janhq/Jan-v1-4B/blob/main/chat_template.jinja) that automatically applies a default system prompt. By default, users are automatically opted-in to this chat template. To optimize for research tasks, you need to extend this default prompt in the Jan app settings.

### Step 1: Access assistant settings

Navigate to the Jan app and access the **assistant settings icon** ⚙️ on the top left of the screen. 


When you open the assistant settings, you'll notice the system prompt field appears empty. This is expected behavior because Jan-V1's default system prompt is embedded directly into the [chat template](https://huggingface.co/janhq/Jan-v1-4B/blob/main/chat_template.jinja) rather than being displayed in the Jan app's UI. The empty field doesn't mean there's no system prompt - it's just not visible in the interface.



### Step 2: Understanding the Default System Prompt

Before switching to the research prompt, it's helpful to understand what the default Jan-V1 system prompt provides:

```md
 In this environment you have access to a set of tools you can use to answer the user's question. You can use one tool per message, and will receive the result of that tool use in the user's response. You use tools step-by-step to accomplish a given task, with each tool use informed by the result of the previous tool use.

  Tool Use Rules
  Here are the rules you should always follow to solve your task:
  1. Always use the right arguments for the tools. Never use variable names as the action arguments, use the value instead.
  2. Call a tool only when needed: do not call the search agent if you do not need information, try to solve the task yourself.
  3. If no tool call is needed, just answer the question directly.
  4. Never re-do a tool call that you previously did with the exact same parameters.
  5. For tool use, MARK SURE use XML tag format as shown in the examples above. Do not use any other format.
  Now Begin! If you solve the task correctly, you will receive a reward of $1,000,000.
```


### Step 3: Implementing the Research Prompt

To switch to the research-optimized prompt, replace the default system prompt with the following research template:

```
/* Detailed source-code truncated for AI context efficiency. */
```markdown
# [Clear Title]

## Overview
...

## Key Findings
- Finding 1 [1]
- Finding 2 [2]

## Detailed Analysis
...

## References
[1] https://example.com/source1  
[2] https://example.com/study2  
...

Goal

Answer with depth, precision, and scholarly rigor. You will be rewarded for:

Thoroughness in research
Use of high-quality sources when available (.gov, .edu, peer-reviewed, reputable media)
Clear, structured reporting
Efficient path to completion without redundancy

Now Begin! If you solve the task correctly, you will receive a reward of $1,000,000.
```

If set up correctly, you should see this on Jan screen.


Save the assistant settings and enjoy Jan-V1 with improved report generation capability.


### (Optional) Time-Sensitive queries optimization

Jan-V1 can sometimes incorrectly assume the current date based on the year it was trained on. This can be easily mitigated by attaching the current date to your system prompt:

```md
You are a **research agent** designed to ...

....
Current Year: 2025
Now Begin! If you solve the task correctly, you will receive a reward of $1,000,000.
```

Alternatively, you can simply add the date directly to your question:

**Example:**
- Instead of: "What's the world population?"
- Use: "What's the world population in 2025?"


## Advanced Usage: Customize Prompt Template

You can customize the chat template by opening the model settings icon ⚙️ in the center of Jan's model selection. Do not confuse this with the assistant settings ⚙️ on the top left of the screen.



Scroll down and you will see the Jinja template that can be overridden. We recommend experimenting with this [raw chat template](https://huggingface.co/janhq/Jan-v1-4B/blob/main/chat_template_raw.jinja) to completely eliminate the effect of the default system prompt.

We observed fewer tool calls per query when overriding the default chat template with this raw chat template and only recommend for advanced usage only.

---

### Src Pages Post Offline Chatgpt Alternative (`docs/src/pages/post/offline-chatgpt-alternative.mdx`)

---
title: "Offline ChatGPT: You can't run ChatGPT offline, do this instead"
description: "ChatGPT requires internet to work. Jan is a free offline ChatGPT replacement that runs 100% on your computer. Download for Mac, Windows & Linux."
tags: AI, chatgpt offline, ChatGPT alternative, offline AI, Jan, local AI, privacy
categories: guides
date: 2025-02-08
ogImage: assets/images/general/offline-chatgpt-alternatives-jan.jpg
twitter:
  card: summary_large_image
  site: "@jandotai"
  title: "Offline ChatGPT: You can't run ChatGPT offline, do this instead"
  description: "ChatGPT requires internet to work. Jan is a free offline ChatGPT replacement that runs 100% on your computer. Download for Mac, Windows & Linux."
  image: assets/images/general/offline-chatgpt-alternatives-jan.jpg
---
import { Callout } from 'nextra/components'
import CTABlog from '@/components/Blog/CTA'

# Offline ChatGPT: You can't run ChatGPT offline, do this instead

ChatGPT requires internet because it runs on OpenAI's servers. For offline AI, you need local models running on your computer.

The fastest way: [Jan](https://jan.ai/) - a free, open-source ChatGPT alternative that works 100% offline on Mac, Windows, and Linux.

**3 steps to get started:**

1. Download Jan (free, works offline)
2. Download an AI model  
3. Start chatting without internet

The guide below shows you exactly how, plus answers common questions about offline AI.

## Can you download ChatGPT for offline use?

No. ChatGPT is cloud-only and always requires internet. OpenAI doesn't offer an offline download.

**The solution:** Download Jan and use offline AI models that work like ChatGPT - just without needing internet.

| Platform | Download | Works Offline |
|----------|----------|---------------|
| Windows | [jan.exe](https://jan.ai) | ✓ |
| macOS | [jan.dmg](https://jan.ai) | ✓ |
| Linux | [jan.AppImage](https://jan.ai) | ✓ |

## How to use AI offline?

Offline AI means the model runs on your computer. So no internet needed, 100% private, and data never leaves your device. With Jan you can run offline AI models locally.

## Jan as an offline ChatGPT replacement


*Jan lets you use AI offline - no internet connection needed*

**Get started with offline AI in 3 simple steps:**

### 1. Download Jan

Go to [jan.ai](https://jan.ai) and download the version for your computer (Mac, Windows, or Linux). It's completely free.



### 2. Download an AI model

You'll need an AI model to use AI offline, so download a model from Jan. Once it's on your computer, you don't need internet anymore. You can also use GPT models via Jan - check [running gpt-oss locally](https://www.jan.ai/post/run-gpt-oss-locally) post to see it.


*Select an AI model that matches your needs and computer capabilities*

### Which model should you choose?

- For most computers: Try Mistral 7B or DeepSeek - they're similar to ChatGPT 3.5
- For older computers: Use smaller 3B models
- For gaming PCs: You can try larger 13B models

<Callout type="info">
Jan automatically recommends models that work well on your computer. No technical knowledge needed.
</Callout>

If you'd like to learn more about local AI, check [how to run AI models locally as a beginner](https://www.jan.ai/post/run-ai-models-locally) article.

### 3. Start using AI offline


*Use Jan's clean interface to chat with AI - no internet required*

Once downloaded, you can use AI anywhere, anytime:

- Chat like you do with ChatGPT
- Work on documents offline
- Get coding help without internet
- Keep your conversations private
- Use AI even when servers are down

<Callout type="info">
**Works completely offline:** Once you download Jan and a model, disconnect your internet. Jan will still work perfectly.
</Callout>

## See Jan in Action

Watch how Jan works as an offline ChatGPT alternative:

<div style={{ position: 'relative', paddingBottom: '56.25%', height: 0, overflow: 'hidden', maxWidth: '100%', marginTop: '1rem', marginBottom: '2rem' }}>
  <iframe 
    src="https://www.youtube.com/embed/ES021_sY6WQ" 
    style={{ position: 'absolute', top: 0, left: 0, width: '100%', height: '100%' }}
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" 
    allowFullScreen
  />
</div>

## How to chat with your docs in Jan?

Simply add your files and ask questions about them.


*Chat with your documents privately - no internet needed*

I did this for you and got a reply from a 7B parameter model. If you'd like to learn what "7B" means and understand other local AI terms, check our [guide on running AI models locally](/post/run-ai-models-locally).

A response from AI, Qwen2.5 7B Instruct Q4:

`This document appears to be about the benefits and advantages of running artificial intelligence (AI) models locally on your device rather than using cloud-based or remote AI services. The key points it highlights include data privacy, offline functionality, freedom from paywalls and restrictions, and giving users full control over their AI models. Additionally, the text mentions that local AI is becoming a new trend and provides a link to a guide for beginners who want to learn more about this topic.`

Local AI makes possible offline AI use, so Jan is going to be your first step to get started. 

## Why choose Jan over ChatGPT?

1. **True Offline Use:** Unlike ChatGPT, Jan works without internet
2. **100% Private:** Your data never leaves your computer
3. **Free Forever:** No subscriptions or API costs
4. **No Server Issues:** No more "ChatGPT is at capacity"
5. **Your Choice of Models:** Use newer models as they come out

### "Is Jan really free? What's the catch?"

Jan is completely free and open source. Built by developers who believe AI should be accessible to everyone. No subscriptions, no hidden costs.

### How does Jan compare to ChatGPT?"
Modern open-source models like DeepSeek and Mistral are very capable. While they might not match GPT-4, they're perfect for most tasks and getting better every month.

### "Do I need a powerful computer?"
If your computer is from the last 5 years, it will likely work fine. You need about 8GB of RAM and 10GB of free space for comfortable usage.

### "What about my privacy?"
Everything stays on your computer with Jan. Your conversations, documents, and data never leave your device unless you choose to share them.

Want to learn more about the technical side? Check our detailed [guide on running AI models locally](/post/run-ai-models-locally). It's not required to [use AI offline](https://jan.ai/) but helps understand how it all works.

## Need help?

<Callout type="info">
[Join our Discord community](https://discord.gg/Exe46xPMbK) for support and tips on using Jan as your offline ChatGPT alternative.
</Callout>

## Frequently Asked Questions

#### Can I download ChatGPT for offline use?
No. ChatGPT is cloud-only and cannot be downloaded.

#### How to use ChatGPT offline?
You cannot use ChatGPT offline. Use Jan instead for a ChatGPT-like offline experience with local AI models.

#### How to download offline ChatGPT?
ChatGPT itself cannot be downloaded. Download Jan from jan.ai as a free offline alternative that works without internet.

#### Does ChatGPT work without internet?
No. ChatGPT requires an internet connection to function.

#### What's the best offline AI chatbot?
Jan is the leading open-source offline AI. It runs models like DeepSeek, Mistral, and GPT-OSS locally on your computer.

#### Is there a free offline AI?
Yes. Jan is completely free and open-source. Download from jan.ai.

#### Offline AI app for Windows?
Jan works on Windows 10 and 11. Download the .exe installer from jan.ai.

#### Offline AI app for Mac?
Jan works on macOS (Intel and Apple Silicon). Download the .dmg from jan.ai.

#### What's GPT offline?
OpenAI has open-source models you can run locally (not via ChatGPT). One is [gpt-oss](https://www.jan.ai/post/run-gpt-oss-locally) - run it via Jan.

---

### Src Pages Post Qwen3 Settings (`docs/src/pages/post/qwen3-settings.mdx`)

---
title: "Best Settings to Run Qwen3-30B-A3B Locally"
description: "If you're running Qwen3-30B-A3B locally, don't guess your way through the settings. This guide tells you what actually works based on Qwen's own documentation and what we've seen hold up in practice."
tags: Qwen3, local AI, model settings, Jan, offline AI
categories: guides
date: 2025-05-10
ogImage: assets/images/general/qwen3-30b-settings.jpg
---
import { Callout } from 'nextra/components'
import CTABlog from '@/components/Blog/CTA'

# Best Settings to Run Qwen3-30B-A3B Locally

If you're running Qwen3-30B-A3B locally, don't guess your way through the settings. This guide tells you what actually works based on Qwen's own documentation and what we've seen hold up in practice.

<Callout>
**Quick Summary:**
- Qwen3-30B-A3B has two modes: thinking and non-thinking
- Each mode needs different generation settings
- Greedy decoding breaks thinking mode - avoid it
- Don't log &lt;think&gt; blocks in chat history
</Callout>

Qwen3 comes with a unique toggle: `enable_thinking`. When it's on, the model "thinks", it breaks down problems, reasons step-by-step, and wraps part of its output in a `<think>...</think>` block. When it's off, the model skips all that and just gives you an answer.

That changes how you configure it.

---

### Thinking mode (`enable_thinking=True`)

This is the mode for reasoning, math, coding, logic — anything that benefits from step-by-step generation.

**Use these generation settings:**

```vbnet
Temperature: 0.6  
TopP: 0.95  
TopK: 20  
Max tokens: 32,768  
Do not use greedy decoding
```

<Callout type="info">
**Why it matters:**
Thinking mode is powerful, but greedy decoding kills its output. It'll repeat or get stuck. Stick to sampling, and give it enough token space to finish its thoughts.
</Callout>

---

## Quick summary 



### Non-thinking mode (`enable_thinking=False`)

This is for fast, general-purpose replies. Instruction following, chat, creative writing — no `<think>` block, no extra steps.

**Use these settings:**

```makefile
Temperature: 0.7  
TopP: 0.8  
TopK: 20
```

<Callout type="info">
Non-thinking mode is faster and more efficient, so the default token length is usually enough. If you're not doing long-form tasks, don't worry about it.
</Callout>

---

## Soft vs. hard switch

You can toggle thinking dynamically in the prompt using:

```bash
/think      # turns thinking ON
/no_think   # turns it OFF
```

This works only if `enable_thinking=True` is set in the code. If you set it to False, the soft switch won't do anything.

---

### What most people miss

- **Don't log the `think` block in chat history.** Qwen recommends keeping only the final answer. Otherwise, the next reply gets bloated and off-topic.
- **Greedy decoding is a trap.** It's tempting to use for consistency, but Qwen3's output gets worse - and sometimes broken - without sampling.
- **YaRN isn't always needed.** The model supports up to 32k context by default. Use YaRN only if you regularly go beyond that.

---

## Running Qwen3 locally with Jan

The easiest way to run Qwen3-30B-A3B locally is through Jan.

1. Download and install [Jan](https://jan.ai)
2. Open Jan and navigate to Jan Hub
3. Find `Qwen3` and `Qwen3-30B-A3B` in the model list
4. Click "Download" to get the model

### Qwen3 in Jan Hub

You can easily find Qwen3 models in Jan Hub:



Once downloaded, Jan handles all the technical setup, so you can focus on using the model rather than configuring it. The settings we covered in this guide are automatically applied when you use Qwen3 through Jan.

### How to customize Qwen3-30B-A3B settings in Jan

You can also customize these settings anytime by opening the right panel in Jan and adjusting the parameters to match your needs.



## Bottom Line

If you're running Qwen3-30B-A3B locally, treat it like two models in one. Flip the thinking mode based on the task, adjust the generation settings accordingly, and let it work how it was meant to.

## Need help?

<Callout type="info">
[Join our Discord community](https://discord.gg/Exe46xPMbK) for support and tips on using Jan as your offline ChatGPT alternative.
</Callout>

<CTABlog />

---

### Src Pages Post Rag Is Not Enough (`docs/src/pages/post/rag-is-not-enough.mdx`)

---
title: "RAG is not enough: Lessons from Beating GPT-3.5 on Specialized Tasks with Mistral 7B"
description: We present a straightforward approach to customizing small, open-source models using fine-tuning and RAG that outperforms GPT-3.5 for specialized use cases. 
tags: RAG, opensource chatgpt alternative, outperform ChatGPT, Mistral
date: 2024-03-25
unlisted: true
categories: research
---

import CTABlog from '@/components/Blog/CTA'

# RAG is not enough: Lessons from Beating GPT-3.5 on Specialized Tasks with Mistral 7B

## Abstract

We present a straightforward approach to customizing small, open-source models using fine-tuning and RAG that outperforms GPT-3.5 for specialized use cases. With it, we achieved superior Q&A results of [technical documentation](https://nitro.jan.ai/docs) for a small codebase [codebase](https://github.com/janhq/nitro).

In short, (1) extending a general foundation model like [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) with strong math and coding, and (2) training it over a high-quality, synthetic dataset generated from the intended corpus, and (3) adding RAG capabilities, can lead to significant accuracy improvements.

Problems still arise with catastrophic forgetting in general tasks, commonly observed during specilizied domain fine-tuning. In our case, this is likely exacerbated by our lack of access to Mistral’s original training dataset and various compression techniques used in our approach to keep the model small.

## Selecting a strong foundation model

[Mistral 7B](https://huggingface.co/mistralai/Mistral-7B-v0.1) outshines both [Meta's Llama-2 7B](https://huggingface.co/meta-llama/Llama-2-7b) and [Google's Gemma 7B](https://huggingface.co/google/gemma-7b) in key benchmarks, making it our choice for a base model. Starting with a strong foundation like Mistral allowed us to achieve greater accuracy in our specialized adaptations.



*Figure 1. Mistral 7B excels in benchmarks, ranking among the top foundational models.*

*Note: We are not sponsored by the Mistral team, though lots of folks like to run Mistral locally using [Jan](https://jan.ai/)., our desktop client.*

## Cost effectively improving the base model

Our technical use case required excellent math capabilities, an area where Mistral can underperform. Thus, we tested a number of Mistral model variants, from foundation models to finetunes to model merges, to find a stronger base model before proceeding to finetuning.




*Figure 2: The merged model, Stealth, doubles the mathematical capabilities of its foundational model while retaining the performance in other tasks.*

We found merging models is quick and cost-effective, enabling fast adjustments based on the result of each iteration.

We ended up with [Stealth 7B v1.1](https://huggingface.co/jan-hq/stealth-v1.1), a [SLERP](https://github.com/Digitous/LLM-SLERP-Merge) merge of Mistral with the following:

- [WizardMath](https://huggingface.co/WizardLM/WizardMath-7B-V1.1) for its math capabilities.
- [WizardCoder](https://huggingface.co/WizardLM/WizardCoder-Python-7B-V1.0) for its coding capabilities.
- Our own [Trinity](https://huggingface.co/jan-hq/trinity-v1.2) model for its versatility across general tasks.

This particular combination yielded the best tradeoff across mathematical & technical reasoning while retaining the most pre-merge performance on general tasks.

## **DPO finetuning**

Merging different LLMs can lead to a mixed answering style because each model was originally trained on different types of data.

Thus, we applied Direct Preference Optimization ([DPO](https://arxiv.org/abs/2305.18290)) using the [Intel's Orca DPO pairs](https://huggingface.co/datasets/Intel/orca_dpo_pairs) dataset, chosen for its helpful answering style in general, math and coding concentration.

This approach produced a final model - [Stealth 7B v1.2](https://huggingface.co/jan-hq/stealth-v1.2), aligned to our technical preferences and demonstrating minimal loss.

## **Using our own technical documentation**

With the base model ready, we started on our specific use case.

Jan is an open-source project enjoying strong growth, but at one point we began receiving a new support ticket every minute, which quickly overwhelmed our bootstrapped resources.

So, we directed our efforts toward training a model to answer user questions based on existing technical documentation.  

Specifically, we trained it on on the [Nitro documentation](https://nitro.jan.ai/docs). For context, Nitro is the default inference engine for Jan. It’s a enterprise-ready server implementation of LlamaCPP, written in C++, with multimodal, queues, and other production-level server capabilities. 

It made an interesting corpus because it was rife with post-2023 technical jargon, edge cases, and poor informational layout.

## Generating training data

The first step was to transform Nitro’s unstructured format into a synthetic Q&A dataset designed for [instruction tuning](https://arxiv.org/pdf/2109.01652.pdf). 

The text was split into chunks of 300-token segments with 30-token overlaps. This helped to avoid a [lost-in-the-middle](https://arxiv.org/abs/2307.03172) problem where LLM can’t use context efficiently to answer given questions.

The chunks were then given to GPT-4 with 8k context length to generate 3800 Q&A pairs. The [training dataset](https://huggingface.co/datasets/jan-hq/nitro_binarized_v2) is available on HuggingFace.

## **Training**

Training was done with supervised finetuning (SFT) from the [Hugging Face's alignment-handbook](https://github.com/huggingface/alignment-handbook), per [Huggingface's Zephyr Beta](https://github.com/huggingface/alignment-handbook/tree/main/recipes/zephyr-7b-beta) guidelines. 

We used consumer-grade, dual Nvidia RTX 4090s for the training. The end-to-end training took 18 minutes. We found optimal hyperparameters in LoRA for this specific task to be `r = 256` and `alpha = 512`.

This final model can be found [here on Huggingface](https://huggingface.co/jan-hq/nitro-v1.2-e3).




*Figure 3. Using the new finetuned model in [Jan](https://jan.ai/)*

## Improving results with RAG

As an additional step, we also added [Retrieval Augmented Generation (RAG)](https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/) as an experiment parameter.

A simple RAG setup was done using **[Llamaindex](https://www.llamaindex.ai/)** and the **[bge-en-base-v1.5 embedding](https://huggingface.co/BAAI/bge-base-en-v1.5)** model for efficient documentation retrieval and question-answering. You can find the RAG implementation [here](https://github.com/janhq/open-foundry/blob/main/rag-is-not-enough/rag/nitro_rag.ipynb).

## Benchmarking the Results

We curated a new set of [50 multiple-choice questions](https://github.com/janhq/open-foundry/blob/main/rag-is-not-enough/rag/mcq_nitro.csv) (MCQ) based on the Nitro docs. The questions had varying levels of difficulty and had trick components that challenged the model's ability to discern misleading information.




*Figure 4. Comparation between finetuned model and OpenAI's GPT*

**Results**

- GPT-3.5 with RAG: 56.7%
- GPT-4 with RAG: 64.3%
- Merged 7B Model ([Stealth 7B](https://huggingface.co/jan-hq/stealth-v1.3)) with RAG: 47.7%
- Finetuned 7B Model (Nitro 7B) with RAG: 57.8%

This indicates that with task-specific training, we can improve an open-source, Small Language Model to the level of GPT-3.5 on domain knowledge. 

Notably, the finetuned + RAG approach also demonstrated more consistency across benchmarking, as indicated by its lower standard deviation.

## Conclusion

We conclude that this combination of model merging + finetuning + RAG yields promise. This finding is relevant for teams and individuals that need specialized, technical small language models that need to run in resource-constrained or highly secured environments, where GPT may not be an option.

Anecdotally, we’ve had some success using this model in practice to onboard new team members to the Nitro codebase.

A full research report with more statistics can be found [here](https://github.com/janhq/open-foundry/blob/main/rag-is-not-enough/README.md).

# References

- [Catastrophic forgetting](https://arxiv.org/abs/2308.08747)
- [Math specialization](https://arxiv.org/abs/2308.09583)
- [Code specialization](https://arxiv.org/abs/2306.08568)
- [Search specialization](https://github.com/SciPhi-AI/agent-search)
- [Evol Instruct](https://github.com/nlpxucan/WizardLM)
- [Lost in the middle](https://arxiv.org/abs/2307.03172)
- [Instruction tuning](https://arxiv.org/pdf/2109.01652.pdf)

<CTABlog />

---

### Src Pages Post Run Ai Models Locally (`docs/src/pages/post/run-ai-models-locally.mdx`)

---
title: "How to run AI models locally as a beginner?"
description: "A straightforward guide to running AI models locally on your computer, regardless of your background."
tags: AI, local models, Jan, GGUF, privacy, local AI
categories: guides
date: 2025-01-31
ogImage: assets/images/general/run-ai-locally-jan.jpeg
twitter:
  card: summary_large_image
  site: "@jandotai"
  title: "How to run AI models locally as a beginner?"
  description: "Learn how to run AI models locally on your computer for enhanced privacy and control. Perfect for beginners!"
  image: assets/images/general/run-ai-locally-jan.jpeg
---

import { Callout } from 'nextra/components'
import CTABlog from '@/components/Blog/CTA'

# How to run AI models locally as a beginner?



Running AI locally is fastest when you take these 3 actions in order. This walkthrough gets you from zero to a working offline AI on your computer.

1. Download [Jan](https://www.jan.ai/) (free, open source)
2. Pick a model to use
3. Start chatting

The rest of this guide explains each step and answers common questions.

### 1. Download [Jan](https://jan.ai)


*Download Jan from [jan.ai](https://jan.ai) - it's free and open source.*

### 2. Choose a model that fits your hardware


*Jan helps you pick the right AI model for your computer.*

### 3. Start using AI locally

That's all to run your first AI model locally!


*Start chatting with local AI models using Jan.*

Keep reading to learn key terms of local AI and the things you should know before running AI models locally.

## Running AI Locally on Windows, Mac, and Linux

[Jan](https://www.jan.ai/) works on all major operating systems with the same features:

**Windows (10, 11)**  
Download the `.exe` installer from jan.ai. Works on Windows 10 and 11 with no additional setup.

**macOS (Intel and Apple Silicon)**  
Download the `.dmg` file. Supports both Intel Macs and Apple Silicon (M1, M2, M3) natively.

**Linux (Ubuntu, Debian, Fedora)**  
Download the `.AppImage` or `.deb` package. Works on most modern Linux distributions.

All platforms get the same models and features. The rest of this guide applies to all operating systems.

## Common Questions for Beginners

### Do I need coding skills?
No. Jan handles installation, GGUF downloads, and updates. You point and click, then start chatting.

### Is running AI locally free?
Yes. Jan is open source, local AI models are free, and offline AI replies cost nothing to run on your computer.

### Will AI on my computer slow it down?
Only during inference. Close big apps or pause the model if you need the CPU or GPU for other work.

### Do I need internet access after setup?
You only need it to download Jan and your first model. After that, you can run AI locally offline whenever you want.

### Is my data private?
Everything stays on-device unless you choose to share it. No prompts are sent to Jan’s servers by default.

## How Local AI Works
With the basics and beginner FAQs out of the way, here's what is happening under the hood when you run AI on your computer.

Before diving into the details, let's understand how AI runs on your computer:

<Callout>
**Why do we need special tools for local AI?**
Think of AI models like compressed files - they need to be "unpacked" to work on your computer. Tools like llama.cpp do this job:
- They make AI models run efficiently on regular computers
- Convert complex AI math into something your computer understands
- Help run large AI models even with limited resources
</Callout>


*llama.cpp helps millions of people run AI locally on their computers.*

<Callout>
**What is GGUF and why do we need it?**

Original AI models are huge and complex - like trying to read a book in a language your computer doesn't understand. Here's where GGUF comes in:

1. **Problem it solves:**
   - Original AI models are too big (100s of GB)
   - They're designed for specialized AI computers
   - They use too much memory

2. **How GGUF helps:**
   - Converts models to a smaller size
   - Makes them work on regular computers
   - Keeps the AI smart while using less memory

When browsing models, you'll see "GGUF" in the name (like "DeepSeek-R1-GGUF"). Don't worry about finding them - Jan automatically shows you the right GGUF versions for your computer.
</Callout>

## Understanding AI Models

Think of AI models like apps on your computer - some are light and quick to use, while others are bigger but can do more things. When you're choosing an AI model to run on your computer, you'll see names like "Llama-3-8B" or "Mistral-7B". Let's break down what this means in simple terms.

<Callout>
The "B" in model names (like 7B) stands for "billion" - it's just telling you the size of the AI model. Just like how some apps take up more space on your computer, bigger AI models need more space on your computer.

- Smaller models (1-7B): Work great on most computers
- Bigger models (13B+): Need more powerful computers but can do more complex tasks
</Callout>


*Jan Hub makes it easy to understand different model sizes and versions*

Running local AI models becomes easier once you understand how size affects speed; next you'll see what you can do after the install.

**Good news:** Jan helps you pick the right model size for your computer automatically! You don't need to worry about the technical details - just choose a model that matches what Jan recommends for your computer.

## What You Can Do with Local AI

<Callout type="info">
Running AI locally gives you:
- Complete privacy - your data stays on your computer
- No internet needed - works offline
- Full control - you decide what models to use
- Free to use - no subscription fees
</Callout>

## Hardware Requirements: Running AI on Your Laptop or Desktop

Most modern computers can run AI locally. Here's what you need:

**Minimum requirements (works on most laptops):**
- CPU from the last 5 years (Intel i5/AMD Ryzen 5 or better)
- 8GB RAM minimum - 16GB recommended for better performance
- 5GB+ free storage per model
- No GPU required - runs on CPU only

### What AI models can run on your laptop or desktop?

| | | |
|---|---|---|
| Regular Laptop | 3B-7B models | Good for chatting and writing. Like having a helpful assistant |
| Gaming Laptop | 7B-13B models | More capable. Better at complex tasks like coding and analysis |
| Powerful Desktop | 13B+ models | Better performance. Great for professional work and advanced tasks |

<Callout type="info">
**Not Sure About Your Computer?**
Start with a smaller model (3B-7B) - Jan will help you choose one that works well on your system.
</Callout>

## Getting Started with Models

### Model Versions

When browsing models in Jan, you'll see terms like "Q4", "Q6", or "Q8". Here's what that means in simple terms:

<Callout>
These are different versions of the same AI model, just packaged differently to work better on different computers:

- **Q4 versions**: Like a "lite" version of an app - runs fast and works on most computers
- **Q6 versions**: The "standard" version - good balance of speed and quality
- **Q8 versions**: The "premium" version - highest quality but needs a more powerful computer
</Callout>

**Pro tip**: Start with Q4 versions - they work great for most people and run smoothly on regular computers!

### Getting Models from Hugging Face

You'll often see links to "Hugging Face" when downloading AI models. Think of Hugging Face as the "GitHub for AI" - it's where the AI community shares their models. Jan makes it super easy to use:

1. Jan has a built-in connection to Hugging Face
2. You can download models right from Jan's interface
3. No need to visit the Hugging Face website unless you want to explore more options

## Setting up your local AI

### Getting Models from Hugging Face

You'll often see links to "Hugging Face" when downloading AI models. Think of Hugging Face as the "GitHub for AI" - it's where the AI community shares their models. This sounds technical, but Jan makes it super easy to use:

1. Jan has a built-in connection to Hugging Face
2. You can download models right from Jan's interface
3. No need to visit the Hugging Face website unless you want to explore more options

<Callout>
**What powers local AI?**
Jan uses [llama.cpp](https://github.com/ggerganov/llama.cpp), an inference that makes AI models run efficiently on regular computers. It's like a translator that helps AI models speak your computer's language, making them run faster and use less memory.
</Callout>

### 1. Get Started
Download Jan from [jan.ai](https://jan.ai) - it sets everything up for you.

### 2. Get an AI Model

You can get models two ways:

#### 1. Use Jan Hub (Recommended):
   - Click "Download Model" in Jan
   - Pick a recommended model
   - Choose one that fits your computer


*Use Jan Hub to download AI models*

#### 2. Use Hugging Face:

<Callout type="warning">
Important: Only GGUF models will work with Jan. Make sure to use models that have "GGUF" in their name.
</Callout>

##### Step 1: Get the model link
Find and copy a GGUF model link from [Hugging Face](https://huggingface.co)


*Look for models with "GGUF" in their name*

##### Step 2: Open Jan
Launch Jan and go to the Models tab


*Navigate to the Models section in Jan*

##### Step 3: Add the model
Paste your Hugging Face link into Jan


*Paste your GGUF model link here*

##### Step 4: Download
Select your quantization and start the download


*Choose your preferred model size and download*

## Technical FAQs

### Can I run AI locally without a GPU?
Yes. CPU-only inference works fine for 3B-7B models. Expect slower responses, so keep prompts short and close other heavy apps.

### Which local AI model should I start with for free?
Pick any Jan-recommended 7B GGUF model like DeepSeek-R1 7B Q4 or Llama-3.1 8B Q4. They balance accuracy, speed, and memory use for most laptops.

### How much RAM and storage do I need to run AI locally?
Reserve 5 GB storage per model plus 2× the model size in free RAM. Example: a 4 GB Q4 file needs roughly 8 GB of RAM to run smoothly.

### How do I run larger AI models on my computer?
Move up to Q6 or Q8 quantization or 13B+ models if you have a desktop GPU. Jan shows real-time VRAM and RAM requirements before download.

## Need help?
<Callout type="info">
[Join our Discord community](https://discord.gg/Exe46xPMbK) for support.
</Callout>

---

### Src Pages Post Run Gpt Oss Locally (`docs/src/pages/post/run-gpt-oss-locally.mdx`)

---
title: "Run OpenAI's gpt-oss locally in 5 mins (Beginner Guide)"
description: "Complete 5-minute beginner guide to running OpenAI's gpt-oss locally. Step-by-step setup with Jan AI for private, offline AI conversations."
tags: OpenAI, gpt-oss, local AI, Jan, privacy, Apache-2.0, llama.cpp, Ollama, LM Studio
categories: guides
date: 2025-08-06
ogImage: assets/images/general/gpt-oss locally.jpeg   
twitter:
  card: summary_large_image
  site: "@jandotai"
  title: "Run OpenAI's gpt-oss Locally in 5 Minutes (Beginner Guide)"
  description: "Complete 5-minute beginner guide to running OpenAI's gpt-oss locally with Jan AI for private, offline conversations."
  image: assets/images/general/gpt-oss locally.jpeg
---
import { Callout } from 'nextra/components'
import CTABlog from '@/components/Blog/CTA'

# Run OpenAI's gpt-oss Locally in 5 mins

OpenAI launched [gpt-oss](https://openai.com/index/introducing-gpt-oss/), marking their return to open-source AI after GPT-2. This model is designed to run locally on consumer hardware. This guide shows you how to install and run gpt-oss on your computer for private, offline AI conversations.

## What is gpt-oss?

gpt-oss is OpenAI's open-source large language model, released under the Apache-2.0 license. Unlike ChatGPT, gpt-oss:

- Runs completely offline - No internet required after setup
- 100% private - Your conversations never leave your device  
- Unlimited usage - No token limits or rate limiting
- Free forever - No subscription fees
- Commercial use allowed - Apache-2.0 license permits business use

Running AI models locally means everything happens on your own hardware, giving you complete control over your data and conversations.

## gpt-oss System Requirements

| Component | Minimum | Recommended |
|-----------|---------|-------------|
| **RAM** | 16 GB | 32 GB+ |
| **Storage** | 11+ GB free | 25 GB+ free |
| **CPU** | 4 cores | 8+ cores |
| **GPU** | Optional | Modern GPU with 6GB+ VRAM recommended |
| **OS** | Windows 10+, macOS 11+, Linux | Latest versions |

**Installation apps available:**
- **Jan** (Recommended - easiest setup) 
- **llama.cpp** (Command line)
- **Ollama** (Docker-based)
- **LM Studio** (GUI alternative)

## How to install gpt-oss locally with Jan (5 mins)

### Step 1: Download Jan

First download Jan to run gpt-oss locally: [Download Jan AI](https://jan.ai/)

<Callout type="info">
Jan is the simplest way to run AI models locally. It automatically handles CPU/GPU optimization, provides a clean chat interface, and requires zero technical knowledge.
</Callout>

### Step 2: Install gpt-oss Model (2-3 minutes)



1. Open Jan Hub -> search "gpt-oss" (it appears at the top)
2. Click Download and wait for completion (~11GB download)
3. Installation is automatic - Jan handles everything

### Step 3: Start using gpt-oss offline (30 seconds)



1. Go to New Chat → select gpt-oss-20b from model picker
2. Start chatting - Jan automatically optimizes for your hardware
3. You're done! Your AI conversations now stay completely private

Success: Your gpt-oss setup is complete. No internet required for chatting, unlimited usage, zero subscription fees.

## Jan with gpt-oss vs ChatGPT vs other Local AI Models

| Feature | gpt-oss (Local) | ChatGPT Plus | Claude Pro | Other Local Models |
|---------|----------------|--------------|------------|-------------------|
| Cost | Free forever | $20/month | $20/month | Free |
| Privacy | 100% private | Data sent to OpenAI | Data sent to Anthropic | 100% private |
| Internet | Offline after setup | Requires internet | Requires internet | Offline |
| Usage limits | Unlimited | Rate limited | Rate limited | Unlimited |
| Performance | Good (hardware dependent) | Excellent | Excellent | Varies |
| Setup difficulty | Easy with Jan | None | None | Varies |

## Alternative Installation Methods

### Option 1: Jan (Recommended)

- Best for: Complete beginners, users wanting GUI interface
- Setup time: 5 minutes
- Difficulty: Very Easy

Already covered above - [Download Jan](https://jan.ai/)

### Option 2: llama.cpp (Command Line)

- Best for: Developers, terminal users, custom integrations  
- Setup time: 10-15 minutes  
- Difficulty: Intermediate

```bash
# macOS
brew install llama-cpp

# Windows: grab Windows exe from releases
curl -L -o gpt-oss-20b.gguf https://huggingface.co/openai/gpt-oss-20b-gguf/resolve/main/gpt-oss-20b.gguf
./main -m gpt-oss-20b.gguf --chat-simple

# Add GPU acceleration (adjust -ngl value based on your GPU VRAM)
./main -m gpt-oss-20b.gguf --chat-simple -ngl 20
```

### Option 3: Ollama (Docker-Based)

Best for: Docker users, server deployments  
Setup time: 5-10 minutes  
Difficulty: Intermediate

```bash
# Install from https://ollama.com
ollama run gpt-oss:20b
```

### Option 4: LM Studio (GUI Alternative)

Best for: Users wanting GUI but not Jan  
Setup time: 10 minutes  
Difficulty: Easy

1. Download LM Studio from official website
2. Go to Models → search "gpt-oss-20b (GGUF)"
3. Download the model (wait for completion)
4. Go to Chat tab → select the model and start chatting

## gpt-oss Performance & Troubleshooting

### Expected Performance Benchmarks

| Hardware Setup | First Response | Subsequent Responses | Tokens/Second |
|---------------|---------------|---------------------|---------------|
| **16GB RAM + CPU only** | 30-45 seconds | 3-6 seconds | 3-8 tokens/sec |
| **32GB RAM + RTX 3060** | 15-25 seconds | 1-3 seconds | 15-25 tokens/sec |
| **32GB RAM + RTX 4080+** | 8-15 seconds | 1-2 seconds | 25-45 tokens/sec |

### Common Issues & Solutions

Performance optimization tips:
- First response is slow: Normal - kernels compile once, then speed up dramatically
- Out of VRAM error: Reduce context length in settings or switch to CPU mode
- Out of memory: Close memory-heavy apps (Chrome, games, video editors)
- Slow responses: Check if other apps are using GPU/CPU heavily

Quick fixes:
1. Restart Jan if responses become slow
2. Lower context window from 4096 to 2048 tokens
3. Enable CPU mode if GPU issues persist
4. Free up RAM by closing unused applications

## Frequently Asked Questions (FAQ)

### Is gpt-oss completely free?
Yes! gpt-oss is 100% free under Apache-2.0 license. No subscription fees, no token limits, no hidden costs.

### How much internet data does gpt-oss use?
Only for the initial 11GB download. After installation, gpt-oss works completely offline with zero internet usage.

### Can I use gpt-oss for commercial projects?
Absolutely! The Apache-2.0 license permits commercial use, modification, and distribution.

### Is gpt-oss better than ChatGPT?
gpt-oss offers different advantages: complete privacy, unlimited usage, offline capability, and no costs. ChatGPT may have better performance but requires internet and subscriptions.

### What happens to my conversations with gpt-oss?
Your conversations stay 100% on your device. Nothing is sent to OpenAI, Jan, or any external servers.

### Can I run gpt-oss on a Mac with 8GB RAM?
No, gpt-oss requires minimum 16GB RAM. Consider upgrading your RAM or using cloud-based alternatives.

### How do I update gpt-oss to newer versions?
Jan automatically notifies you of updates. Simply click update in Jan Hub when new versions are available.

## Why Choose gpt-oss Over ChatGPT Plus?

gpt-oss advantages:
- $0/month vs $20/month for ChatGPT Plus
- 100% private - no data leaves your device  
- Unlimited usage - no rate limits or restrictions
- Works offline - no internet required after setup
- Commercial use allowed - build businesses with it

When to choose ChatGPT Plus instead:
- You need the absolute best performance
- You don't want to manage local installation
- You have less than 16GB RAM

## Get started with gpt-oss today



Ready to try gpt-oss?
- Download Jan: [https://jan.ai/](https://jan.ai/)
- View source code: [https://github.com/janhq/jan](https://github.com/janhq/jan)
- Need help? Check our [local AI guide](/post/run-ai-models-locally) for beginners

<CTABlog />

---

### Src Pages Research Jan Code 4b (`docs/src/pages/research/jan-code-4b.mdx`)

---
title: Jan-Code-4B
description: Lightweight 4B code-tuned model for fast local coding assistance and agentic workflows
---

import ResearchPostLayout from '@/components/ResearchPostLayout'
import OpenInJanButton from '@/components/OpenInJanButton'
import { Callout } from 'nextra/components'

<ResearchPostLayout title="Jan-Code-4B">

Jan-Code-4B is a lightweight, code-tuned 4B parameter model built for fast local inference. Fine-tuned on [Jan-v3-4B-base-instruct](https://docs.jan.ai/desktop/jan-models/jan-v3-4b-base-instruct), it is designed for practical coding tasks with an emphasis on handling well-scoped subtasks reliably while keeping latency and compute requirements low.

<OpenInJanButton href="jan://models/huggingface/janhq/Jan-code-4b-gguf" />

## Overview

| Property | Value |
|----------|-------|
| **Parameters** | 4B |
| **Base Model** | Jan-v3-4B-base-instruct (Qwen3-4B-Instruct-2507) |
| **Fine-tuning focus** | Code generation, editing, refactoring, debugging |
| **License** | Apache 2.0 |

## Capabilities

- **Coding assistant**: Code generation, editing, refactoring, and debugging
- **Agent workflows**: Use as a fast worker/sub-agent in agentic setups (e.g., generating patches or tests)
- **Claude Code integration**: Can replace the Haiku model in a Claude Code setup for a fully local coding workflow

<Callout type="tip">
  Jan-Code-4B is designed to work as a drop-in local alternative to cloud coding models in agentic pipelines, keeping your code private and inference fast.
</Callout>

## Performance

Jan-Code-4B leads Jan-v3-base-instruct and the Qwen3-4B-Instruct-2507 base model across all three coding and reasoning benchmarks:



- **Aider** (challenging Exercism tasks): 19.0 vs 18.0 vs 12.9
- **Livecode Bench v6** (real-world coding): 51.0 vs 45.8 vs 35.1
- **AIME25** (advanced math reasoning): 53.0 vs 47.0 vs 47.0

## Requirements

- **Memory**:
  - Minimum: 8GB RAM (with Q4 quantization)
  - Recommended: 16GB RAM (with Q8 quantization)
- **Hardware**: CPU or GPU
- **API Support**: OpenAI-compatible at localhost:1337

## Using Jan-Code-4B

### Quick Start

1. Download Jan Desktop
2. Click **Open in Jan** above, or open the **Hub** and search for Jan-Code-4B
3. Choose a variant that fits your hardware, then click **Download**
4. Once downloaded, select Jan-Code-4B from the model selector in any chat
5. Start coding — no additional configuration needed

### Deployment Options

**Using vLLM:**
```bash
vllm serve janhq/Jan-code-4b \
    --host 0.0.0.0 \
    --port 1234 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes
```

**Using llama.cpp:**
```bash
llama-server --model Jan-code-4b-Q8_0.gguf \
    --host 0.0.0.0 \
    --port 1234 \
    --jinja \
    --no-context-shift
```

### Recommended Parameters

```yaml
temperature: 0.7
top_p: 0.8
top_k: 20
```

## What Jan-Code-4B Does Well

- **Code generation**: Write new functions, classes, and modules from natural language descriptions
- **Editing & refactoring**: Modify existing code with targeted, reliable edits
- **Debugging**: Identify and fix bugs in provided code snippets
- **Agentic subtasks**: Fast, focused execution of scoped coding tasks within larger agent pipelines
- **Low latency**: Runs efficiently on consumer hardware with fast response times

## Limitations

- **Model size**: 4B parameters limit handling of very large codebases or complex architectural decisions in a single context
- **General tasks**: Optimized for coding — for general-purpose tasks, consider [Jan-v3-4B-base-instruct](https://docs.jan.ai/desktop/jan-models/jan-v3-4b-base-instruct)

## Available Formats

### GGUF Quantizations

- **Q4_K_M**: Good balance of size and quality
- **Q5_K_M**: Better quality, slightly larger
- **Q8_0**: Highest quality quantization (recommended)

## Models Available

- [Jan-Code-4B on Hugging Face](https://huggingface.co/janhq/Jan-code-4b)

## Community

- **Discussions**: [HuggingFace Community](https://huggingface.co/janhq/Jan-code-4b/discussions)
- **Support**: Available through Jan App at [jan.ai](https://jan.ai)

</ResearchPostLayout>

---

### Src Pages Research Jan Nano 128 (`docs/src/pages/research/jan-nano-128.mdx`)

---
title: Jan Nano 128k
description: Jan Models
---

import ResearchPostLayout from '@/components/ResearchPostLayout'
import OpenInJanButton from '@/components/OpenInJanButton'
import { Callout } from 'nextra/components'

<ResearchPostLayout title="Jan Nano 128k">

> Enabling deeper research through extended context understanding.

Jan-Nano-128k represents a notable advancement in compact language models for different applications. Building upon the
success of Jan-Nano-32k, this enhanced version features a native 128k context window that enables deeper, more comprehensive
research capabilities without the performance degradation typically associated with context extension methods.

You can have a look at all of our models, and download them from the HuggingFace [Menlo Models page](https://huggingface.co/Menlo).

<OpenInJanButton href="jan://models/huggingface/Menlo/Jan-nano-128k-gguf" />

**Key Improvements:**

- 🔍 Deeper Research: Extended context allows for processing entire research papers, lengthy documents, and complex multi-turn conversations
- ⚡ Native 128k Window: Built to handle long contexts efficiently, maintaining performance across the full context range
- 📈 Enhanced Performance: Unlike traditional context extension methods, Jan-Nano-128k's performance remains consistent with longer contexts

This model maintains full compatibility with Model Context Protocol (MCP) servers while dramatically expanding the scope of research
tasks it can handle in a single session.


## Why Jan-Nano-128k?

Most small models hit a wall at 8-32k tokens. Jan-Nano-128k goes beyond this limitation with a native 128k context window—that's roughly
300 pages of text or an entire novel's worth of information processed simultaneously.

Unlike YaRN or PI methods that retrofit models beyond their limits and degrade performance, Jan-Nano-128k was architecturally rewired for
128k contexts from the ground up. The result: an inverse scaling behavior where performance actually improves with longer contexts,
maintaining consistent accuracy from 1k to 128k tokens as the model leverages more information for synthesis.


<Callout type="info">
**Position Interpolation (PI):** A method that extends a model's context by scaling down position indices to fit within the original context
window. For example, to extend a 4k model to 32k, PI compresses the 32k positions into the original 4k range by dividing each position by 8.

**YaRN (Yet another RoPE extensioN method):** A more sophisticated context extension method that preserves frequently occurring tokens while
selectively scaling others. YaRN divides position embeddings into frequency groups and applies different scaling factors to each, resulting
in more efficient training and better performance than PI.

The key difference is that PI applies uniform scaling across all dimensions, while YaRN uses targeted interpolation based on frequency analysis—preserving
high-frequency information that's crucial for distinguishing nearby tokens while interpolating lower frequencies more aggressively.
</Callout>

**Applications unlocked:**
- **Academic**: Extract key findings from 50+ papers simultaneously
- **Legal**: Pinpoint relevant clauses across thousand-page contracts
- **Code**: Trace specific functions through massive codebases
- **Business**: Distill insights from quarters of financial data
- **Content**: Maintain narrative coherence across book-length outputs

**MCP Usage:** Jan-Nano-128k doesn't memorize, it orchestrates. With MCP integration, it becomes a research conductor that fetches dozens
of sources, holds everything in active memory, extracts precisely what's needed, and synthesizes findings across a marathon research session. It's
not about understanding every word; it's about finding the needle in a haystack of haystacks.

## Evaluation

Jan-Nano-128k has been rigorously evaluated on the SimpleQA benchmark using our MCP-based methodology, demonstrating superior performance compared to its predecessor:



**Key findings:**
- 15% improvement over Jan-Nano-32k on complex multi-document tasks
- Consistent performance across all context lengths (no cliff at 64k like other extended models)
- Superior citation accuracy when handling 10+ sources simultaneously

## 🖥️ How to Run Locally

### Demo

<video width="100%" controls>
  <source src="https://catalog.jan.ai/docs/jan-nano-demo.mp4" type="video/mp4" />
  Your browser does not support the video tag.
</video>

### Quick Start Guide

1. **Download Jan**
2. **Download Jan-Nano-128k**
3. **Enable MCP**, the serper or the exa MCPs work very well with Jan-Nano-128k
4. **Start researching**

### Usage

Deploy using VLLM:

```bash
vllm serve Menlo/Jan-nano-128k \
 --host 0.0.0.0 \
 --port 1234 \
 --enable-auto-tool-choice \
 --tool-call-parser hermes \
 --rope-scaling '{"rope_type":"yarn","factor":3.2,"original_max_position_embeddings":40960}' --max-model-len 131072
```

Or with `llama-server` from `llama.cpp`:

```bash
llama-server ... --rope-scaling yarn --rope-scale 3.2 --yarn-orig-ctx 40960
```

**Note:** The chat template is included in the tokenizer. For troubleshooting, download the [Non-think chat template](https://qwen.readthedocs.io/en/latest/_downloads/c101120b5bebcc2f12ec504fc93a965e/qwen3_nonthinking.jinja).

### Recommended Sampling Parameters

```yaml
Temperature: 0.7
Top-p: 0.8
Top-k: 20
Min-p: 0.0
```

### Hardware Requirements
- **Minimum**: 16GB RAM for Q4 quantization
- **Recommended**: 24GB RAM for Q8 quantization
- **Optimal**: 32GB+ RAM for full precision

## 🤝 Community & Support
- **Discussions**: [HuggingFace Community](https://huggingface.co/Menlo/Jan-nano-128k/discussions)
- **Issues**: [GitHub Repository](https://github.com/janhq/deep-research/issues)
- **Discord**: Join our research community for tips and best practices

</ResearchPostLayout>

---

### Src Pages Research Jan Nano 32 (`docs/src/pages/research/jan-nano-32.mdx`)

---
title: Jan Nano 32k
description: Jan-Nano-Gguf Model
---

import ResearchPostLayout from '@/components/ResearchPostLayout'
import OpenInJanButton from '@/components/OpenInJanButton'
import { Callout } from 'nextra/components'

<ResearchPostLayout title="Jan Nano 32k">



## Why Jan Nano?

Most language models face a fundamental tradeoff where powerful capabilities require a lot of computational resources. Jan
Nano breaks this constraint through a focused design philosophy where instead of trying to know everything, it excels at
knowing how to find anything.

<OpenInJanButton href="jan://models/huggingface/Menlo/Jan-nano-gguf" />

## What is Jan Nano?

Jan Nano is a compact 4-billion parameter language model specifically designed and trained for deep research tasks.
This model has been optimized to work seamlessly with Model Context Protocol (MCP) servers, enabling efficient integration
with various research tools and data sources.

The model and its different model variants are fully supported by Jan.

<Callout type="info">
  To use Jan-Nano, you will need to use a search engine via MCP. You can enable MCP in the **Settings**
  tab under **Advanced Settings**.
</Callout>


## System Requirements

- Minimum Requirements:
  - 8GB RAM (with iQ4_XS quantization)
  - 12GB VRAM (for Q8 quantization)
  - CUDA-compatible GPU
- Recommended Setup:
  - 16GB+ RAM
  - 16GB+ VRAM
  - Latest CUDA drivers
  - RTX 30/40 series or newer


## Using Jan-Nano-32k

**Step 1**
Download Jan from [here](https://jan.ai/download/).

**Step 2**
Go to the Hub Tab, search for Jan-Nano-Gguf, and click on the download button to the best model size for your system.



**Step 3**
Go to **Settings** > **Model Providers** > **Llama.cpp** click on the pencil icon and enable tool use for Jan-Nano-Gguf.

**Step 4**
To take advantage of Jan-Nano's full capabilities, you need to enable MCP support. We're going to use it with Serper's
API. You can get a free API key from [here](https://serper.dev/). Sign up and they will immediately generate one for you.

**Step 5**
Add the serper MCP to Jan via the **Settings** > **MCP Servers** tab.



**Step 6**
Open up a new chat and ask Jan-Nano to search the web for you.



## Queries to Try

Here are some example queries to showcase Jan-Nano's web search capabilities:

1. **Current Events**: What are the latest developments in renewable energy adoption in Germany and Denmark?
2. **International Business**: What is the current status of Tesla's Gigafactory in Berlin and how has it impacted the local economy?
3. **Technology Trends**: What are the newest AI developments from Google, Microsoft, and Meta that were announced this week?
4. **Global Weather**: What's the current weather forecast for Tokyo, Japan for the next 5 days?
5. **Stock Market**: What are the current stock prices for Apple, Samsung, and Huawei, and how have they performed this month?
6. **Sports Updates**: What are the latest results from the Premier League matches played this weekend?
7. **Scientific Research**: What are the most recent findings about climate change impacts in the Arctic region?
8. **Cultural Events**: What major music festivals are happening in Europe this summer and who are the headliners?
9. **Health & Medicine**: What are the latest developments in mRNA vaccine technology and its applications beyond COVID-19?
10. **Space Exploration**: What are the current missions being conducted by NASA, ESA, and China's space program?


## FAQ

- What are the recommended GGUF quantizations?
  - Q8 GGUF is recommended for best performance
  - iQ4_XS GGUF for very limited VRAM setups
  - Avoid Q4_0 and Q4_K_M as they show significant performance degradation

- Can I run this on a laptop with 8GB RAM?
  - Yes, but use the recommended quantizations (iQ4_XS)
  - Note that performance may be limited with Q4 quantizations

- How much did the training cost?
  - Training was done on internal A6000 clusters
  - Estimated cost on RunPod would be under $100 using H200
  - Hardware used:
    - 8xA6000 for training code
    - 4xA6000 for vllm server (inferencing)

- What frontend should I use?
  - Jan (recommended)
  - Download link: https://jan.ai/download

- Getting Jinja errors in LM Studio?
  - Use Qwen3 template from other LM Studio compatible models
  - Disable “thinking” and add the required system prompt
  - Fix coming soon in future GGUF releases
- Having model loading issues in Jan?
  - Use latest beta version: Jan-beta_0.5.18-rc6-beta
  - Ensure proper CUDA support for your GPU
  - Check VRAM requirements match your quantization choice

## Resources

- [Jan-Nano Model on Hugging Face](https://huggingface.co/Menlo/Jan-nano)
- [Jan-Nano GGUF on Hugging Face](https://huggingface.co/Menlo/Jan-nano-gguf)

</ResearchPostLayout>

---

### Src Pages Research Jan V1 (`docs/src/pages/research/jan-v1.mdx`)

---
title: Jan-v1
description: 4B parameter model with strong performance on reasoning benchmarks
---

import ResearchPostLayout from '@/components/ResearchPostLayout'
import OpenInJanButton from '@/components/OpenInJanButton'
import { Callout } from 'nextra/components'

<ResearchPostLayout title="Jan-v1" date="2025-08-22">

## Overview

Jan-v1 is a 4B parameter model based on Qwen3-4B-thinking, designed for reasoning and problem-solving tasks. The model achieves 91.1% accuracy on SimpleQA through model scaling and fine-tuning approaches.

<OpenInJanButton href="jan://models/huggingface/janhq/Jan-v1-4B-GGUF" />

## Performance

### SimpleQA Benchmark

Jan-v1 demonstrates strong factual question-answering capabilities:



At 91.1% accuracy, Jan-v1 outperforms several larger models on SimpleQA, including Perplexity's 70B model. This performance represents effective scaling and fine-tuning for a 4B parameter model.

### Chat and Creativity Benchmarks

Jan-v1 has been evaluated on conversational and creative tasks:



These benchmarks (EQBench, CreativeWriting, and IFBench) measure the model's ability to handle conversational nuance, creative expression, and instruction following.

## Requirements

- **Memory**:
  - Minimum: 8GB RAM (with Q4 quantization)
  - Recommended: 16GB RAM (with Q8 quantization)
- **Hardware**: CPU or GPU
- **API Support**: OpenAI-compatible at localhost:1337

## Using Jan-v1

### Quick Start

1. Download Jan Desktop
2. Click **Open in Jan** above, or open the **Hub** and search for Jan-v1
3. Choose a variant that fits your hardware, then click **Download**
4. Once downloaded, select Jan-v1 from the model selector in any chat
5. Start chatting - no additional configuration needed

### Demo



### Deployment Options

**Using vLLM:**
```bash
vllm serve janhq/Jan-v1-4B \
    --host 0.0.0.0 \
    --port 1234 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes
```

**Using llama.cpp:**
```bash
llama-server --model jan-v1.gguf \
    --host 0.0.0.0 \
    --port 1234 \
    --jinja \
    --no-context-shift
```

### Recommended Parameters

```yaml
temperature: 0.6
top_p: 0.95
top_k: 20
min_p: 0.0
max_tokens: 2048
```

## What Jan-v1 Does Well

- **Question Answering**: 91.1% accuracy on SimpleQA
- **Reasoning Tasks**: Built on thinking-optimized base model
- **Tool Calling**: Supports function calling through hermes parser
- **Instruction Following**: Reliable response to user instructions

## Limitations

- **Model Size**: 4B parameters limits complex reasoning compared to larger models
- **Specialized Tasks**: Optimized for Q&A and reasoning, not specialized domains
- **Context Window**: Standard context limitations apply

## Available Formats

### GGUF Quantizations

- **Q4_K_M**: 2.5 GB - Good balance of size and quality
- **Q5_K_M**: 2.89 GB - Better quality, slightly larger
- **Q6_K**: 3.31 GB - Near-full quality
- **Q8_0**: 4.28 GB - Highest quality quantization

## Models Available

- [Jan-v1 on Hugging Face](https://huggingface.co/janhq/Jan-v1-4B)
- [Jan-v1 GGUF on Hugging Face](https://huggingface.co/janhq/Jan-v1-4B-GGUF)

## Technical Notes

<Callout type="info">
The model includes a system prompt in the chat template by default to match benchmark performance. A vanilla template without system prompt is available in `chat_template_raw.jinja`.
</Callout>

## Community

- **Discussions**: [HuggingFace Community](https://huggingface.co/janhq/Jan-v1-4B/discussions)
- **Support**: Available through Jan App at [jan.ai](https://jan.ai)

</ResearchPostLayout>

---

### Src Pages Research Jan V2 Vl (`docs/src/pages/research/jan-v2-vl.mdx`)

---
title: Jan-v2-VL
description: 8B vision-language model for long-horizon agentic automation in real software environments
---

import ResearchPostLayout from '@/components/ResearchPostLayout'
import OpenInJanButton from '@/components/OpenInJanButton'
import { Callout } from 'nextra/components'

<ResearchPostLayout title="Jan-v2-VL">

Jan-v2-VL is an 8B vision-language model built on Qwen3-VL-8B-Thinking, designed for **long-horizon, multi-step agentic tasks** in real software environments — browsers, desktop apps, and beyond. It combines language reasoning with visual perception to execute stable, many-step automation chains with minimal drift.

<OpenInJanButton href="jan://models/huggingface/janhq/Jan-v2-VL-med-gguf" />

## Variants

Jan-v2-VL comes in three variants tuned for different latency/quality tradeoffs:

| Variant | Focus |
|---------|-------|
| **Jan-v2-VL-low** | Efficiency-oriented, lower latency |
| **Jan-v2-VL-med** | Balanced latency and quality |
| **Jan-v2-VL-high** | Deeper reasoning, higher think time |

## Capabilities

- **Agentic automation & UI control**: Follow complex multi-step instructions in browsers and desktop apps
- **Screenshot grounding**: Perceive and act on visual state of the screen
- **Tool calling**: Native support for BrowserMCP and similar integrations
- **Error recovery**: Maintain intermediate state and recover from minor execution errors
- **Long-horizon execution**: Stable performance across many-step automation chains

## Performance

### Long-Horizon Execution

Jan-v2-VL dramatically outperforms competing VLMs on the long-horizon benchmark, completing tasks up to 49 steps in a single turn — nearly 10x more than the next best model.



All three Jan-v2-VL variants significantly exceed the Qwen3-VL-8B-Thinking baseline (score: 5):



### Text-Only Benchmarks

Jan-v2-VL maintains strong performance on standard language benchmarks with no degradation from the base model:



### Multimodal Benchmarks

Core vision capabilities are fully preserved across all variants:



## Requirements

- **Memory**:
  - Minimum: 16GB RAM (with Q4 quantization)
  - Recommended: 24GB RAM (with Q8 quantization)
- **Hardware**: GPU recommended for real-time agentic use
- **API Support**: OpenAI-compatible at localhost:1337

## Using Jan-v2-VL

### Quick Start

1. Download Jan Desktop
2. Click **Open in Jan** above, or open the **Hub** and search for Jan-v2-VL
3. Choose Jan-v2-VL-med or another variant that fits your hardware, then click **Download**
4. Once downloaded, select the downloaded Jan-v2-VL variant from the model selector in any chat
5. Start using it for browser automation or visual tasks

### Deployment Options

**Using vLLM:**
```bash
vllm serve janhq/Jan-v2-VL-high \
    --host 0.0.0.0 \
    --port 1234 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --reasoning-parser qwen3
```

**Using llama.cpp:**
```bash
llama-server --model Jan-v2-VL-high-Q8_0.gguf \
    --vision-model-path mmproj-Jan-v2-VL-high.gguf \
    --host 0.0.0.0 \
    --port 1234 \
    --jinja \
    --no-context-shift
```

### Recommended Parameters

```yaml
temperature: 1.0
top_p: 0.95
top_k: 20
repetition_penalty: 1.0
presence_penalty: 1.5
```

## What Jan-v2-VL Does Well

- **UI automation**: Navigate and interact with browsers and desktop applications via screenshots
- **Long-horizon tasks**: Execute multi-step plans without losing context or drifting
- **Visual grounding**: Identify and interact with specific UI elements from screenshots
- **Tool calling**: Integrate with MCP servers like BrowserMCP for end-to-end automation

## Limitations

- **GPU recommended**: Real-time agentic use benefits significantly from GPU acceleration
- **8B size**: More capable than 4B models for visual tasks, but larger resource footprint
- **Vision required**: Text-only tasks do not benefit from the vision-language architecture

## Available Formats

### GGUF Quantizations

- **Q4_K_M**: Good balance of size and quality
- **Q5_K_M**: Better quality, slightly larger
- **Q8_0**: Highest quality quantization

## Models Available

- [Jan-v2-VL-low on Hugging Face](https://huggingface.co/janhq/Jan-v2-VL-low)
- [Jan-v2-VL-med on Hugging Face](https://huggingface.co/janhq/Jan-v2-VL-med)
- [Jan-v2-VL-high on Hugging Face](https://huggingface.co/janhq/Jan-v2-VL-high)

## Community

- **Discussions**: [HuggingFace Community](https://huggingface.co/janhq/Jan-v2-VL-med/discussions)
- **Support**: Available through Jan App at [jan.ai](https://jan.ai)

</ResearchPostLayout>

---

### Src Pages Research Jan V3 4b (`docs/src/pages/research/jan-v3-4b.mdx`)

---
title: Jan-v3-4B
description: 4B parameter instruct model distilled from a larger teacher, optimized as a fine-tuning base
---

import ResearchPostLayout from '@/components/ResearchPostLayout'
import OpenInJanButton from '@/components/OpenInJanButton'
import { Callout } from 'nextra/components'

<ResearchPostLayout title="Jan-v3-4B">

Jan-v3-4B-base-instruct is a compact 4B parameter model created via post-training distillation from a larger teacher model. It is designed as an ownable, general-purpose base that is straightforward to fine-tune, with strong instruction following out of the box.

<OpenInJanButton href="jan://models/huggingface/janhq/Jan-v3-4B-base-instruct-gguf" />

## Overview


| Property | Value |
|----------|-------|
| **Parameters** | 4B (3.6B non-embedding) |
| **Base Model** | Qwen3-4B-Instruct-2507 |
| **Context Length** | 262,144 tokens (native) |
| **Architecture** | 36 layers, 32 Q heads / 8 KV heads (GQA) |
| **License** | Apache 2.0 |

## Capabilities

- **Instruction following**: Strong compliance with user instructions out of the box
- **Fine-tuning base**: Designed as a starting point for downstream fine-tuning
- **Coding assistance**: Effective lightweight coding help (a dedicated code variant, Jan-Code-4B, is also available)
- **General purpose**: Distillation preserves broad capabilities across standard benchmarks

<Callout type="info">
  Jan-v3-4B-base-instruct is the base model for [Jan-Code-4B](https://docs.jan.ai/desktop/jan-models/jan-code-4b), the code-tuned variant.
</Callout>

## Performance

### Aider Benchmark (Polyglot Code Editing)

Jan-v3-4B leads its weight class on the Aider polyglot benchmark with a strict pass rate of 18, nearly 50% higher than the next best model (Qwen3-4B-Thinking at 12.1):



### General Performance Comparison

Jan-v3-4B-base-instruct outperforms Qwen3-4B-Instruct-2507 (its base) across coding, math, and instruction-following benchmarks while remaining competitive on reasoning:



## Requirements

- **Memory**:
  - Minimum: 8GB RAM (with Q4 quantization)
  - Recommended: 16GB RAM (with Q8 quantization)
- **Hardware**: CPU or GPU
- **API Support**: OpenAI-compatible at localhost:1337

## Using Jan-v3-4B

### Quick Start

1. Download Jan Desktop
2. Click **Open in Jan** above, or open the **Hub** and search for Jan-v3-4B-base-instruct
3. Choose a variant that fits your hardware, then click **Download**
4. Once downloaded, select Jan-v3-4B-base-instruct from the model selector in any chat
5. Start chatting — no additional configuration needed

### Demo

Try it live at [chat.jan.ai](https://chat.jan.ai/)

### Deployment Options

**Using vLLM:**
```bash
vllm serve janhq/Jan-v3-4B-base-instruct \
    --host 0.0.0.0 \
    --port 1234 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes
```

**Using llama.cpp:**
```bash
llama-server --model Jan-v3-4B-base-instruct-Q8_0.gguf \
    --host 0.0.0.0 \
    --port 1234 \
    --jinja \
    --no-context-shift
```

### Recommended Parameters

```yaml
temperature: 0.7
top_p: 0.8
top_k: 20
```

## What Jan-v3-4B Does Well

- **Long context**: Native 262k token context window for handling large documents
- **Instruction following**: Reliable and consistent response to user instructions
- **Fine-tuning**: Designed to be a strong, ownable starting point for specialization
- **General tasks**: Broad capabilities preserved through distillation from a larger model

## Limitations

- **Model size**: 4B parameters limit complex multi-step reasoning compared to larger models
- **Specialized domains**: For coding specifically, consider [Jan-Code-4B](https://docs.jan.ai/desktop/jan-models/jan-code-4b)

## Available Formats

### GGUF Quantizations

- **Q4_K_M**: Good balance of size and quality
- **Q5_K_M**: Better quality, slightly larger
- **Q8_0**: Highest quality quantization (recommended)

## Models Available

- [Jan-v3-4B-base-instruct on Hugging Face](https://huggingface.co/janhq/Jan-v3-4B-base-instruct)

## Community

- **Discussions**: [HuggingFace Community](https://huggingface.co/janhq/Jan-v3-4B-base-instruct/discussions)
- **Support**: Available through Jan App at [jan.ai](https://jan.ai)

</ResearchPostLayout>

---

### Src Pages Research Jan V3 5 4b (`docs/src/pages/research/jan-v3-5-4b.mdx`)

---
title: Jan-v3.5-4B
description: The first Jan personality — a 4B model fine-tuned for math reasoning with a distinct conversational identity.
---

import ResearchPostLayout from '@/components/ResearchPostLayout'
import OpenInJanButton from '@/components/OpenInJanButton'

<ResearchPostLayout title="Jan-v3.5-4B">



Jan-v3.5-4B is the first Jan personality: a 4-billion parameter fine-tuned
variant of [Jan-v3-4B-base-instruct](/research/jan-v3-4b), specialized in
mathematical reasoning while incorporating a distinct conversational personality
developed by Menlo Research.

This model enhances the base Qwen3-4B architecture with improved math
problem-solving capabilities and a deliberately crafted identity. Rather than
functioning as a neutral assistant, Jan-v3.5-4B uses casual, direct language
with lowercase-by-default styling and maintains an authentic conversational
tone.

<OpenInJanButton href="https://huggingface.co/janhq/Jan-v3.5-4B">
  View on Hugging Face →
</OpenInJanButton>

## Key Features

- **Base architecture**: Qwen3-4B with 36 layers and 4.0B parameters (3.6B non-embedding)
- **Context length**: 262,144 tokens natively supported
- **Training focus**: math reasoning datasets and curated identity/personality datasets
- **Distinctive voice**: self-aware humor, direct communication style, avoids corporate phrasing
- **Attention mechanism**: 32 query heads with 8 key-value heads (GQA)



## Recommended Parameters

```
temperature: 0.7
top_p: 0.8
top_k: 20
```

## Usage

Jan-v3.5-4B supports deployment via:

- Transformers library with pipeline or direct model loading
- vLLM server with tool-calling capabilities
- llama.cpp with quantized variants
- Jan Desktop application integration

## Limitations & Scope

- Optimized specifically for math-heavy tasks over general purpose
- Personality training may not align with all use cases
- Intended as a starting point for downstream fine-tuning

## Links

- [Hugging Face model card](https://huggingface.co/janhq/Jan-v3.5-4B)
- [GitHub](https://github.com/janhq/jan)
- [Jan App](https://jan.ai/)
- License: Apache 2.0

</ResearchPostLayout>

---

### Src Pages Research Lucy (`docs/src/pages/research/lucy.mdx`)

---
title: Lucy
description: Compact 1.7B model optimized for web search with tool calling
---

import ResearchPostLayout from '@/components/ResearchPostLayout'
import OpenInJanButton from '@/components/OpenInJanButton'
import { Callout } from 'nextra/components'

<ResearchPostLayout title="Lucy">

## Overview

Lucy is a 1.7B parameter model built on Qwen3-1.7B, optimized for web search through tool calling. The model has been trained to work effectively with search APIs like Serper, enabling web search capabilities in resource-constrained environments.

<OpenInJanButton href="jan://models/huggingface/Menlo/Lucy-128k-gguf" />


## Performance

### SimpleQA Benchmark

Lucy achieves competitive performance on SimpleQA despite its small size:



The benchmark shows Lucy (1.7B) compared against models ranging from 4B to 600B+ parameters. While larger models generally perform better, Lucy demonstrates that effective web search integration can partially compensate for smaller model size.

## Requirements

- **Memory**:
  - Minimum: 4GB RAM (with Q4 quantization)
  - Recommended: 8GB RAM (with Q8 quantization)
- **Search API**: Serper API key required for web search functionality
- **Hardware**: Runs on CPU or GPU

<Callout type="info">
To use Lucy's web search capabilities, you'll need a Serper API key. Get one at [serper.dev](https://serper.dev).
</Callout>

## Using Lucy

### Quick Start

1. Download Jan Desktop
2. Download Lucy from the Hub
3. Configure Serper MCP with your API key
4. Start using web search through natural language

### Demo



### Deployment Options

**Using vLLM:**
```bash
vllm serve Menlo/Lucy-128k \
    --host 0.0.0.0 \
    --port 1234 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --rope-scaling '{"rope_type":"yarn","factor":3.2,"original_max_position_embeddings":40960}' \
    --max-model-len 131072
```

**Using llama.cpp:**
```bash
llama-server model.gguf \
    --host 0.0.0.0 \
    --port 1234 \
    --rope-scaling yarn \
    --rope-scale 3.2 \
    --yarn-orig-ctx 40960
```

### Recommended Parameters

```yaml
Temperature: 0.7
Top-p: 0.9
Top-k: 20
Min-p: 0.0
```

## What Lucy Does Well

- **Web Search Integration**: Optimized to call search tools and process results
- **Small Footprint**: 1.7B parameters means lower memory requirements
- **Tool Calling**: Reliable function calling for search APIs

## Limitations

- **Requires Internet**: Web search functionality needs active connection
- **API Costs**: Serper API has usage limits and costs
- **Context Processing**: While supporting 128k context, performance may vary with very long inputs
- **General Knowledge**: Limited by 1.7B parameter size for tasks beyond search

## Models Available

- [Lucy on Hugging Face](https://huggingface.co/Menlo/Lucy-128k)
- [Lucy GGUF on Hugging Face](https://huggingface.co/Menlo/Lucy-128k-gguf)

## Citation

```bibtex
@misc{dao2025lucyedgerunningagenticweb,
      title={Lucy: edgerunning agentic web search on mobile with machine generated task vectors},
      author={Alan Dao and Dinh Bach Vu and Alex Nguyen and Norapat Buppodom},
      year={2025},
      eprint={2508.00360},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2508.00360},
}
```

</ResearchPostLayout>

---

### Src Pages Docs Agent Acp (`docs/src/pages/docs/agent/acp.mdx`)

---
title: Editors (ACP)
description: Drive Jan Agent from Zed, JetBrains and other editors over the Agent Client Protocol with the experimental `jan acp` server.
keywords: [Jan, Jan Agent, ACP, Agent Client Protocol, Zed, JetBrains, editor, IDE, external agent]
---

import { Callout } from 'nextra/components'

# Editors (ACP)

`jan acp` serves Jan Agent over the [Agent Client Protocol](https://agentclientprotocol.com)
(ACP), a JSON-RPC protocol that editors use to run coding agents. Once an editor is pointed at
`jan acp`, you can pick Jan in its agent panel, send prompts, and watch the answer, reasoning,
tool calls and plan stream in. The agent still uses Jan's own providers, tools and permissions.

<Callout type="warning">
`jan acp` is **experimental**. It's off by default, hidden from `jan --help`, and may change
between releases.
</Callout>

## Turn it on

Pick one of these. The environment variable takes precedence over the config file, so
`JAN_EXPERIMENTAL_ACP=0` turns it off again:

```bash
export JAN_EXPERIMENTAL_ACP=1          # per process
```

```toml
# ~/.jan/config.toml                    # every process
[experimental]
acp = true
```

If you run `jan acp` without turning it on, it prints how to enable it and exits with status 2.
It writes nothing to stdout.

Jan needs a working provider before any editor can use it. Run `jan login`, or
`jan config set --provider <name> --api-key <key> --base-url <url> --model <id>`. The model is
resolved the same way as in the TUI: `agent.toml`, then `default_model` in `~/.jan/config.toml`.

## Zed

Add Jan under `agent_servers` in Zed's `settings.json`:

```json
{
  "agent_servers": {
    "Jan": {
      "type": "custom",
      "command": "jan",
      "args": ["acp"],
      "env": { "JAN_EXPERIMENTAL_ACP": "1" }
    }
  }
}
```

Open the agent panel, start a new thread and choose **Jan**. To see the raw traffic, run
`dev: open acp logs`.

## JetBrains

In **AI Chat**, open the menu and choose **Add Custom Agent**. This opens `~/.jetbrains/acp.json`.
Add Jan there. Use the absolute path to the binary:

```json
{
  "agent_servers": {
    "Jan": {
      "command": "/absolute/path/to/jan",
      "args": ["acp"],
      "env": { "JAN_EXPERIMENTAL_ACP": "1" }
    }
  }
}
```

## What is supported

Jan speaks stable ACP protocol version **1**. It doesn't enable any draft or unstable features.

| ACP | Jan |
| --- | --- |
| `initialize` | Advertises `loadSession`, image and embedded-context prompts. Nothing else is advertised. |
| `authenticate` | Checks that a provider is configured. Clients that can run a terminal get a sign-in method that runs `jan acp --login`. |
| `session/new` | Opens a session in `cwd`, which must be absolute. The session id is a Jan thread id. MCP servers the editor passes over stdio are connected as well as the project's own. |
| `session/load` | Replays a saved thread, including ones started in the TUI or with `jan cli agent run`, then replies. Refused while that session has a prompt running. |
| `session/prompt` | Runs one turn. The reply's `stopReason` is `end_turn`, `max_tokens`, `max_turn_requests` (the session's cost ceiling was reached), `refusal` or `cancelled`. If the run fails, the error is shown in the reply text and the turn ends with `end_turn`. |
| `session/cancel` | Stops the turn. A tool call waiting for your permission never runs. The work done before the cancel is kept, so the next prompt continues from it. |
| `session/request_permission` | Sent for every gated tool call. You get **Allow**, **Always allow** where Jan offers it, and **Reject**. |

The session id is the thread id, so `jan --resume <id>` in the terminal opens a conversation that
you started in the editor.

How Jan's events appear in the editor:

| Jan | ACP `session/update` |
| --- | --- |
| Answer text / reasoning | `agent_message_chunk` / `agent_thought_chunk` |
| Tool call starts, runs, streams output, ends | `tool_call` then `tool_call_update` (`pending`, `in_progress`, `completed`, `failed`) |
| The `todo` list | `plan` |
| Token usage for a turn | `usage_update` |

These have no standard ACP update, so they are left out rather than sent through a made-up
extension: subagent progress (a subagent's permission prompts still reach you), notices,
compaction and retry status, and background monitors. The `ask` tool isn't offered in an ACP
session.

## Permissions

An ACP session always asks before writes, shell commands and MCP tool calls. It's never
auto-approved. It uses the same tool gate and project rules (`[tools]` in `agent.toml`) as the TUI.
If you reject a call, or cancel the prompt, the call is refused just as it would be in the terminal.
**Always allow** covers that tool for the rest of the session.

## Not yet supported

- Jan uses its own filesystem and shell tools. It doesn't read the editor's unsaved buffers or run
  commands in the editor's terminal, so `fs/*` and `terminal/*` are never requested.
- Editor MCP servers over HTTP or SSE are skipped. Only stdio servers are connected, alongside the
  servers set up for the project (see [MCP](/docs/agent/mcp)).
- There's no `session/set_mode` yet, so plan mode and model switching aren't available from the
  editor.

---

### Src Pages Docs Agent Adk Nightly (`docs/src/pages/docs/agent/adk-nightly.mdx`)

---
title: Nightly ADK Installation
description: Install prebuilt JavaScript and Python ADK previews without npm or PyPI publication, a repository clone, or a Rust build.
---

import { Callout, Tabs } from 'nextra/components'

# Nightly ADK Installation

**Install a packaged ADK preview, then its runtime.**
The ADK is your language client; the `jan` runtime is the separate executable
that runs the agent. Neither needs Jan Desktop, a Git checkout or a Rust compiler.

<Callout type="warning">
These are previews, not npm/PyPI releases. If a download below fails (HTTP 403 or 404), that
build is not available; use the [source-install fallback](#source-install-fallback).
</Callout>

## Install the ADK

Each command looks up the latest validated build and installs its versioned package URL, so
your dependency file pins that exact build.

<Tabs items={['JavaScript', 'Python']}>
<Tabs.Tab>

Node.js 20+ and npm:

```bash
npm install "$(node -e "fetch('https://delta.jan.ai/adk-nightly/manifest.json').then(r => r.json()).then(m => console.log(m.packages.javascript.url))")"
```

In PowerShell, replace `"$(...)"` with `(...)`.

</Tabs.Tab>
<Tabs.Tab>

Python 3.11+, in a virtual environment. Node is not required:

```bash
python -m pip install "$(python -c "import json, urllib.request as r; p = json.load(r.urlopen('https://delta.jan.ai/adk-nightly/manifest.json'))['packages']['python']; print(p['url'] + '#sha256=' + p['sha256'])")"
```

In PowerShell, replace `"$(...)"` with `(...)`. The `#sha256=` fragment makes pip check the
wheel against the digest the build published.

</Tabs.Tab>
</Tabs>

Prefer to copy a fixed URL? The [download page](https://delta.jan.ai/adk-nightly/index.html)
shows the same commands with the current build's URLs written out.

For repeatable installs, save that build's **versioned manifest**
(`https://delta.jan.ai/adk-nightly/<build>/manifest.json`, linked from the download page)
alongside your lockfile. It identifies the package versions, source revision, digests, protocol
version and compatible runtime. Use it rather than the moving latest manifest.

## Install the runtime

For a first run, the hosted installer puts the latest nightly `jan` in `~/.local/bin`, which also
gives you `jan login`. Make sure that directory is on your PATH; on Windows, see the
[Agent quickstart](/docs/agent/quickstart) for the `-AddToPath` form:

```bash
curl -fsSL https://delta.jan.ai/jan-cli/install-jan-agent.sh | bash    # macOS / Linux
irm https://delta.jan.ai/jan-cli/install-jan-agent.ps1 | iex            # Windows PowerShell
```

`JanRuntime.start()` finds `jan` on PATH, or the binary named by `JAN_BIN`.

### Pin the runtime your ADK build was tested with

Each ADK build records the runtime it was smoke-tested against. To install exactly that one,
download the build's versioned manifest as `adk-nightly.json` and pass its runtime pin to the ADK
installer, which downloads, verifies and caches the binary. Configure your provider as described
in the [Agent quickstart](/docs/agent/quickstart) before asking the agent to do work.

<Tabs items={['JavaScript', 'Python']}>
<Tabs.Tab>

Save as `runtime.mjs`, then run `node runtime.mjs`:

```js
import { readFile } from 'node:fs/promises'
import { JanRuntime, installRuntime } from '@janhq/adk'

const manifest = JSON.parse(await readFile('adk-nightly.json', 'utf8'))
const installed = await installRuntime({
  manifestUrl: manifest.runtime.manifestUrl,
  version: manifest.runtime.version,
})
const runtime = await JanRuntime.start({ bin: installed.bin })
try {
  console.log(runtime.serverInfo)
} finally {
  await runtime.close()
}
```

</Tabs.Tab>
<Tabs.Tab>

Save as `runtime.py`, then run `python runtime.py`:

```python
import json
from pathlib import Path
from jan_adk import JanRuntime, install_runtime

manifest = json.loads(Path('adk-nightly.json').read_text())
installed = install_runtime(
    manifest_url=manifest['runtime']['manifestUrl'],
    version=manifest['runtime']['version'],
)
with JanRuntime.start(bin=installed.bin) as runtime:
    print(runtime.server_info)
```

</Tabs.Tab>
</Tabs>

Expect the runtime's reported name and version, followed by a clean exit. A runtime's
reported version can differ from its nightly artifact version; the manifest pins the
artifact, while the ADK checks the wire protocol before starting work.

An already provisioned compatible runtime can be passed directly as `bin`, or selected
with `JAN_BIN`. For an offline launch, skip `installRuntime` / `install_runtime` and
pass that existing binary. Do not silently substitute another build if a pinned
artifact has gone missing.

## Supported nightly platforms

The publication gate exercises Linux x64 and ARM64, macOS Intel and Apple Silicon,
and Windows x64. Windows ARM64 is not claimed by this ADK nightly matrix, even when
the runtime channel offers an ARM64 binary.

Each job installs the packaged ADKs into clean environments, downloads the pinned
runtime, streams a turn, executes a host callback, checks that its answer reaches a
loopback provider, and verifies runtime shutdown. Python runs without Node on `PATH`.
This is a real-runtime packaging check, not evidence of a live model-provider run.

## Versions, integrity and failures

- The latest [manifest](https://delta.jan.ai/adk-nightly/manifest.json) and download
  page each point to one complete, validated build. An unsuccessful build or smoke
  run does not promote either pointer.
- Versioned directories include the source revision and CI run/attempt. The workflow
  does not overwrite or automatically delete them. Keep a local copy if you require
  availability independent of the download service's storage policy.
- The manifest records SHA-256 digests for both packages and the runtime manifest.
  The Python install link includes the wheel digest; compare the downloaded JS
  tarball with the manifest digest if independently verifying it. npm's lockfile
  also records the installed tarball's integrity.
- A 404 on a saved artifact URL means that build is unavailable. Choose another
  build explicitly; the installer does not replace your pin with latest.
- A digest or protocol mismatch is a failure before work starts. Do not retry a
  potentially executed host action just because the runtime disconnected.

## Source-install fallback

Until a nightly is published, clone Jan once and install only your language package:

```bash
git clone --depth 1 https://github.com/janhq/jan.git jan-source
```

For JavaScript, run `npm install ./jan-source/packages/adk` in your project.
For Python, run `python -m pip install ./jan-source/adk/python` in a virtual environment.
This fallback needs Git and a source checkout; it still needs no Jan Desktop or Rust
build. Install the runtime explicitly with the ADK installer as described in the
package README.

## Publishing and recovery

Maintainers can run **Jan ADK Nightly** manually from `main`; it also runs after a
successful **Jan Agent CLI - Nightly Build**. It snapshots the current published
runtime manifest, rather than assuming an ADK package version is a runtime version.

The workflow packages both clients, uploads immutable versioned assets, runs the
platform smoke matrix against the public downloads, then advances discovery. PRs
and manual runs on other branches validate artifacts without publishing credentials.
If smoke fails, inspect that run; the previous nightly remains selected. A rerun
creates a new run-attempt identity. To restore an older known-good build, restore its
complete discovery documents; never replace files inside its versioned directory.

This workflow does not publish to npm or PyPI. Registry releases are a separate path.

---

--- METRICS ---
- Files Extracted: 101
- Estimated Token Budget: ~96309 tokens
- Canonical Reference: https://codewiki.google/github.com/janhq/jan
