## File: README.md
Kalosm
Kalosm is an ecosystem of crates that make it easy to develop applications that use local or remote AI models. There are try main projects in this repo:
- [Kalosm](./interfaces/kalosm): A simple interface for pre-trained models in rust
- [Fusor](./fusor-ml/core): A runtime for quantized ML inference. Fusor uses WGPU to run models on any accelerator natively or in the browser
## Kalosm
[Kalosm](./interfaces/kalosm/) is a simple interface for pre-trained models in Rust. It makes it easy to interact with pre-trained, language, audio, and image models.
### Model Support
Kalosm supports a variety of models. Here is a list of the models that are currently supported:
| Model | Modality | Size | Description | Quantized | GPU Accelerated | Example |
| ---------------- | -------- | ---------- | -------------------------------------- | --------- | --------------- | --------------------------------------------------------------------- |
| Llama | Text | 1b-70b | General purpose language model | ✅ | ✅ | [llama 3 chat](interfaces/kalosm/examples/chat.rs) |
| Mistral | Text | 7-13b | General purpose language model | ✅ | ✅ | [mistral chat](interfaces/kalosm/examples/chat-mistral-2.rs) |
| Phi | Text | 2b-4b | Small reasoning focused language model | ✅ | ✅ | [phi 3 chat](interfaces/kalosm/examples/chat-phi-3.rs) |
| Whisper | Audio | 20MB-1GB | Audio transcription model | ✅ | ✅ | [live whisper transcription](interfaces/kalosm/examples/transcribe.rs) |
| Segment Anything | Image | 50MB-400MB | Image segmentation model | ✅ | ✅ | [Image Segmentation](interfaces/kalosm/examples/segment-image.rs) |
| Bert | Text | 100MB-1GB | Text embedding model | ✅ | ✅ | [Semantic Search](interfaces/kalosm/examples/semantic-search.rs) |
### Utilities
Kalosm also supports a variety of utilities around pre-trained models. These include:
- [Extracting, formatting and retrieving context for LLMs](./interfaces/kalosm/examples/context_extraction.rs): [Extract context from txt/html/docx/md/pdf](./interfaces/kalosm/examples/context_extraction.rs) [chunk that context](./interfaces/kalosm/examples/chunking.rs) [then search for relevant context with vector database integrations](./interfaces/kalosm/examples/semantic-search.rs)
- [Transcribing audio from your microphone or file](./interfaces/kalosm/examples/transcribe.rs)
- [Crawling and scraping content from web pages](./interfaces/kalosm/examples/crawl.rs)
### Structured Generation
Kalosm supports structured generation with arbitrary parsers. It uses a custom parser engine and sampler and structure-aware acceleration to make structure generation even faster than uncontrolled text generation. You can take any rust type and add `#[derive(Parse, Schema)]` to make it usable with structured generation:
```rust
use kalosm::language::*;
/// A fictional character
#[derive(Parse, Schema, Clone, Debug)]
struct Character {
/// The name of the character
#[parse(pattern = "[A-Z][a-z]{2,10} [A-Z][a-z]{2,10}")]
name: String,
/// The age of the character
#[parse(range = 1..=100)]
age: u8,
/// A description of the character
#[parse(pattern = "[A-Za-z ]{40,200}")]
}
#[tokio::main]
async fn main() {
// First create a model. Chat models tend to work best with structured generation
let model = Llama::phi_3().await.unwrap();
// Then create a task with the parser as constraints
let task = model.task("You generate realistic JSON placeholders for characters")
.typed();
// Finally, run the task
let mut stream = task(&"Create a list of random characters", &model);
stream.to_std_out().await.unwrap();
let characters: [Character; 10] = stream.await.unwrap();
println!("{characters:?}");
}
```
https://github.com/user-attachments/assets/8900f57d-55c8-4d4a-a67b-73beab1e5155
In addition to regex, you can provide your own grammar to generate structured data. This lets you constrain the response to any structure you want including complex data structures like JSON, HTML, and XML.
### Kalosm Quickstart!
This quickstart will get you up and running with a simple chatbot. Let's get started!
> A more complete guide for Kalosm is available on the [Kalosm website](https://floneum.com/kalosm/), and examples are available in the [examples folder](https://github.com/floneum/floneum/tree/main/interfaces/kalosm/examples).
1. Install [rust](https://rustup.rs/)
2. Create a new project:
```sh
cargo new kalosm-hello-world
cd ./kalosm-hello-world
```
3. Add Kalosm as a dependency
```sh
cargo add kalosm --features llama
cargo add tokio --features full
```
4. Add this code to your `main.rs` file
```rust, no_run
use kalosm::language::*;
#[tokio::main]
async fn main() -> Result<(), Box> {
let model = Llama::phi_3().await?;
let mut chat = model.chat()
.with_system_prompt("You are a pirate called Blackbeard");
loop {
chat(&prompt_input("\n> ")?)
.to_std_out()
.await?;
}
}
```
5. Run your application with:
```sh
cargo run --release
```
[chat bot demo](https://github.com/floneum/floneum/assets/66571940/e4e76efb-6387-4fcd-aa3c-aa556e840334)
## Fusor
⚠️ Fusor is still early in development and is not ready for production use. Fusor will serve as the backend for Kalosm in the 0.5 release to enable web and AMD support
[Fusor](./fusor-ml/core) is a WGPU runtime for quantized ML inference. Fusor works with the gguf file format to load quantized models. It targets uses WebGpu to target many different accelerators including Nvidia GPUs, AMD GPUs, and Metal. Most ML frameworks contain hand optimized kernels that perform a series of operations together. Fusor uses a kernel fusion compiler to make merge custom operation chains into an optimized kernel without dropping down to the shader code. This compiles to a single kernel:
```rust, ignore
fn exp_add_one(tensor: Tensor<2, f32>) -> Tensor<2, f32> {
1. + (-tensor).exp()
}
```
## Community
If you are interested in either project, you can join the [discord](https://discord.gg/dQdmhuB8q5) to discuss the project and get help.
## Contributing
- Report issues on our [issue tracker](https://github.com/floneum/floneum/issues).
- Help other users in the discord
- If you are interested in contributing, feel free to reach out on discord
---
## File: fusor-ml/core/README.md
# Fusor ML
This is a WGPU ML runtime with kernel fusion for ergonomic high performance custom operations. This will hopefully serve as the web and amd runtime for [kalosm](https://crates.io/crates/kalosm) once it is stable enough.
## Status
Basic operations are working and simple kernel fusion is implemented, but this is **not production ready yet**.
Features:
- [x] Elementwise ops
- [x] Fuse Elementwise ops together
- [x] MatMul
- [x] Reduce ops
- [x] Fuse Elementwise ops into Reduce ops
- [x] PairWise ops
- [x] Fuse Elementwise ops into PairWise ops
- [x] Analyze buffer usage for in-place ops
- [x] Memory move/cat/etc ops
- [x] Cast ops
- [ ] Fuse PairWise ops together?
- [ ] Fuse parallel Reduce ops?
- [ ] Fuse PairWise ops with two of the same input into an elementwise op
- [ ] Dynamically apply fusion based on runtime throughput data
Operations required for a Llama implementation:
- [x] RmsNorm
- [x] Matmul
- [x] Rope
- [x] Unqueeze
- [x] Cat
- [x] Reshape
- [x] Transpose
- [x] Softmax
- [x] narraw
- [x] silu
- [x] arange
- [x] sin
- [x] cos
## Resources
- https://github.com/googlefonts/compute-shader-101
- https://siboehm.com/articles/22/CUDA-MMM
---
## File: interfaces/language-model/docs/chat.md
Let's start with a simple chat application:
```rust, no_run
# use kalosm::language::*;
# #[tokio::main]
# async fn main() {
// Before you create a chat session, you need a model. Llama::new_chat will create a good default chat model.
let model = Llama::new_chat().await.unwrap();
// Then you can build a chat session that uses that model
let mut chat = model.chat()
// The builder exposes methods for settings like the system prompt and constraints the bot response must follow
.with_system_prompt("The assistant will act like a pirate");
loop {
// To use the chat session, you need to add messages to it
let mut response_stream = chat(&prompt_input("\n> ").unwrap());
// And then display the response stream to the user
response_stream.to_std_out().await.unwrap();
}
# }
```
LLMs are powerful because of their generality, but sometimes you need more control over the output. For example, you might want the assistant to start with a certain phrase, or to follow a certain format.
In kalosm, you can use constraints to guide the model's response. Constraints are a way to specify the format of the output. When generating with constraints, the model will always respond with the specified format.
Let's create a chat application that uses constraints to guide the assistant's response to always start with "Yes!":
```rust, no_run
# use kalosm::language::*;
# #[tokio::main]
# async fn main() {
let model = Llama::new_chat().await.unwrap();
// Create constraints that parses Yes! and then stops on the end of the assistant's response
let constraints = LiteralParser::new("Yes!")
.then(model.default_assistant_constraints());
// Create a chat session with the model and the constraints
let mut chat = model.chat();
// Chat with the user
loop {
let mut output_stream = chat(&prompt_input("\n> ").unwrap()).with_constraints(constraints.clone());
output_stream.to_std_out().await.unwrap();
}
# }
```
---
## File: interfaces/language-model/docs/chat_session.md
# Chat Session
The [`ChatSession`] trait holds the state of a text completion model after it has been fed some text. It can be used in combination with [`ChatModel`] to feed text and cache the results.
## Session History
You can use the [`ChatSession::history`] method to get messages that have already been fed to the model:
```rust, no_run
# use kalosm::language::*;
# #[tokio::main]
# async fn main() {
let mut llm = Llama::new_chat().await.unwrap();
let mut chat = llm.chat();
// Add a message to the session
chat(&"Hello, world!").to_std_out().await.unwrap();
// Get the history of the session
let history = chat.session().unwrap().history();
assert_eq!(history.len(), 1);
assert_eq!(history[0].role(), MessageType::UserMessage);
assert_eq!(history[0].content(), "Hello, world!");
# }
```
## Cloning Sessions
Not all chat models support cloning sessions, but if a model does support
cloning sessions, you can clone a session using the [`ChatSession::try_clone`] method
to clone a session state while retaining the original session.
```rust, no_run
use kalosm::language::*;
use std::io::Write;
#[tokio::main]
async fn main() {
let mut llm = Llama::new_chat().await.unwrap();
let mut chat = llm.chat();
// Feed some text into the session
chat(&"What is the capital of France?").to_std_out().await.unwrap();
let mut session = chat.session().unwrap();
// Clone the session
let cloned_session = session.try_clone().unwrap();
// Feed some more text into the cloned session
let mut chat = llm.chat().with_session(cloned_session);
chat(&"What was my first question?").to_std_out().await.unwrap();
}
```
---
## File: interfaces/language-model/docs/completion.md
# Text Completion Models
[`TextCompletionModelExt`] is the main trait for text generation models. Any model that implements either [TextCompletionModel`] or [`StructuredTextCompletionModel`] can be used with this trait.
The simplest way to use a model is to create a model and call [`TextCompletionModelExt::complete`]. The response builder that is returned can awaited to get the full response:
```rust, no_run
use kalosm::language::*;
#[tokio::main]
async fn main() {
let mut llm = Llama::new().await.unwrap();
let prompt = "The following is a 300 word essay about why the capital of France is Paris:";
print!("{prompt}");
let mut completion = llm
.complete(prompt)
.await
.unwrap();
println!("{completion}");
}
```
Or use the response as a [`Stream`]:
```rust, no_run
use kalosm::language::*;
use std::io::Write;
#[tokio::main]
async fn main() {
let mut llm = Llama::new().await.unwrap();
let prompt = "The following is a 300 word essay about why the capital of France is Paris:";
print!("{prompt}");
let mut completion = llm
.complete(prompt);
while let Some(token) = completion.next().await {
print!("{token}");
std::io::stdout().flush().unwrap();
}
}
```
## Changing the sampler
You can modify the response builder any time before reading the response. The sampler chooses the next token from the probability distribution the model generates. It can make the response more or less predictable and prevent repetition. You can call [`TextCompletionBuilder::with_sampler`] to set a sampler the model will use when completing the text:
```rust, no_run
# use kalosm::language::*;
# #[tokio::main]
# async fn main() {
let model = Llama::new().await.unwrap();
// Create the generation parameters to use for the text completion session
let sampler = GenerationParameters::default().with_temperature(0.8);
// Create a completion request with the generation parameters
let mut stream = model.complete("Here is a list of 5 primes: ").with_sampler(sampler);
stream.to_std_out().await.unwrap();
# }
```
## Structured Generation
Along with the sampler, you can use structured generation to force the output of a model to conform to a specific format a parser defines.
### Defining the parser
There are a few different ways create a parser for structured generation:
1. Derive a parser for your data
2. Create a parser from the set of prebuilt combinators
3. Create a parser from a regex
#### Deriving a parser from a struct
The simplest way to get started is to derive a parser for your data:
```rust, no_run
# use kalosm::language::*;
#[derive(Parse, Clone)]
struct Pet {
name: String,
age: u32,
}
```
Then you can generate text that works with the parser in a [`Task`](https://docs.rs/kalosm/latest/kalosm/language/struct.Task.html):
```rust, no_run
# use kalosm::language::*;
# #[derive(Parse, Clone, Debug)]
# struct Pet {
# name: String,
# age: u32,
# }
#[tokio::main]
async fn main() {
// First create a model
let model = Llama::new().await.unwrap();
// Then create a parser for your data. Any type that implements the `Parse` trait has the `new_parser` method
let parser = Pet::new_parser();
// Create a text completion stream with the constraints
let description = model.complete("JSON for an adorable dog named ruffles: ")
.with_constraints(parser);
// Finally, await the stream to get the parsed response
let pet: Pet = description.await.unwrap();
println!("{pet:?}");
}
```
#### Creating a Parser from the Set of Prebuilt Combinators
Kalosm also provides a set of prebuilt combinators for creating more complex parsers. You can use these combinators to create a parser with a custom format:
```rust, no_run
use kalosm::language::*;
#[tokio::main]
async fn main() {
// First create a model
let model = Llama::new().await.unwrap();
// Then create a parser for your custom format
let parser = LiteralParser::from("[")
.ignore_output_then(String::new_parser())
.then_literal(", ")
.then(u8::new_parser())
.then_literal(", ")
.then(String::new_parser())
.then_literal("]");
// Create a text completion stream with the constraints
let description = model.complete("JSON for an adorable dog named ruffles: ")
.with_constraints(parser);
// Finally, await the stream to get the parsed response
let ((name, age), description) = description.await.unwrap();
println!("{name} {age} {description}");
}
```
#### Creating a Parser from a Regex
You can also create a parser from a regex:
```rust, no_run
use kalosm::language::*;
#[tokio::main]
async fn main() {
// First create a model
let model = Llama::new().await.unwrap();
// Then create a parser for your data. Any
let parser = RegexParser::new(r"\[(\w+), (\d+), (\w+)\]").unwrap();
// Create a text completion stream with the constraints
let mut description = model.complete("JSON for an adorable dog named ruffles in the form [\"Pet name\", age number, \"Pet description\"]: ")
.with_constraints(parser);
// Finally, run the task. Unlike derived and custom parsers, regex parsers do not provide a useful output type
description.to_std_out().await.unwrap();
}
```
### Text Completion with Constraints
Once you have a parser, you can force the model to generate text that conforms to that parser with the [`TextCompletionBuilder::with_constraints`]:
```rust, no_run
use kalosm::language::*;
#[derive(Parse, Clone, Debug)]
struct Pet {
name: String,
age: u32,
}
#[tokio::main]
async fn main() {
// First create a model
let model = Llama::new().await.unwrap();
// Then create a parser for your data. Any type that implements the `Parse` trait has the `new_parser` method
let parser = Pet::new_parser();
// Create a text completion stream with the constraints
let description = model.complete("JSON for an adorable dog named ruffles: ")
.with_constraints(parser);
// Finally, await the stream to get the parsed response
let pet: Pet = description.await.unwrap();
println!("{pet:?}");
}
```
---
## File: interfaces/language-model/docs/completion_session.md
# Text Completion Session
The [`TextCompletionSession`] trait holds the state of a text completion model after it has been fed some text. It can be used in combination with [`TextCompletionModel`] to feed text and cache the results.
## Cloning Sessions
Not all models support cloning sessions, but if a model does support cloning sessions, you can clone a session using the [`TextCompletionSession::try_clone`] method to clone a session state while retaining the original session.
```rust, no_run
use kalosm::language::*;
use std::io::Write;
#[tokio::main]
async fn main() {
let mut llm = Llama::new().await.unwrap();
let mut session = llm.new_session().unwrap();
// Feed some text into the session
llm.stream_text_with_callback(&mut session, "The capital of France is ".into(), GenerationParameters::new().with_max_length(0), |_| Ok(())).await.unwrap();
// Clone the session
let cloned_session = session.try_clone().unwrap();
// Feed some more text into the cloned session
llm.stream_text_with_callback(&mut session, "The capital of France is ".into(), GenerationParameters::new(), |token| {println!("{token}"); Ok(())}).await.unwrap();
}
```
---
## File: interfaces/language-model/docs/embedding.md
# Embeddings
Embeddings are a way to represent the meaning of text in a numerical format. They can be used to compare the meaning of two different texts, or search for documents with a [embedding database](https://docs.rs/kalosm/latest/kalosm/struct.DocumentTable.html).
## Creating Embeddings
You can create embeddings from text using a [`Bert`](https://docs.rs/kalosm/latest/kalosm/struct.Bert.html) embedding model. You can call `embed` on a `Bert` instance to get an embedding for a single sentence or `embed_batch` to get embeddings for a list of sentences at once:
```rust, no_run
# use kalosm::language::*;
# #[tokio::main]
# async fn main() {
let mut bert = Bert::new().await.unwrap();
let sentences = vec![
"Kalosm can be used to build local AI applications",
"With private LLMs data never leaves your computer",
"The quick brown fox jumps over the lazy dog",
];
let embeddings = bert.embed_batch(&sentences).await.unwrap();
# }
```
Once you have embeddings, you can compare them to each other with a distance metric. The cosine similarity is a common metric for comparing embeddings that measures the cosine of the angle between the two vectors:
```rust, no_run
# use kalosm::language::*;
# #[tokio::main]
# async fn main() {
# let mut bert = Bert::new().await.unwrap();
# let sentences = vec![
# "Kalosm can be used to build local AI applications",
# "With private LLMs data never leaves your computer",
# "The quick brown fox jumps over the lazy dog",
# ];
# let embeddings = bert.embed_batch(&sentences).await.unwrap();
// Find the cosine similarity between each pair of sentences
let n_sentences = sentences.len();
for (i, e_i) in embeddings.iter().enumerate() {
for j in (i + 1)..n_sentences {
let e_j = embeddings.get(j).unwrap();
let cosine_similarity = e_j.cosine_similarity(e_i);
println!("score: {cosine_similarity:.2} '{}' '{}'", sentences[i], sentences[j])
}
}
# }
```
You should see that the first two sentences are similar to each other, while the third sentence not similar to either of the first two:
```text
score: 0.82 'Kalosm can be used to build local AI applications' 'With private LLMs data never leaves your computer'
score: 0.72 'With private LLMs data never leaves your computer' 'The quick brown fox jumps over the lazy dog'
score: 0.72 'Kalosm can be used to build local AI applications' 'The quick brown fox jumps over the lazy dog'
```
## Searching for Similar Text
Embeddings can also be a powerful tool for search. Unlike traditional text based search, searching for text with embeddings doesn't directly look for keywords in the text. Instead, it looks for text with similar meanings which can make search more robust and accurate.
In the previous example, we used the cosine similarity to find the similarity between two sentences. Even though the first two sentences have no words in common, their embeddings are similar because they have related meanings.
You can use a vector database to store embedding, value pairs in an easily searchable way. You can create an vector database with [`VectorDB::new`](https://docs.rs/kalosm/latest/kalosm/language/struct.VectorDB.html):
```rust, no_run
# use std::collections::HashMap;
# use kalosm::language::*;
# #[tokio::main]
# async fn main() -> anyhow::Result<()> {
// Create a good default Bert model for search
let bert = Bert::new_for_search().await?;
let sentences = [
"Kalosm can be used to build local AI applications",
"With private LLMs data never leaves your computer",
"The quick brown fox jumps over the lazy dog",
];
// Embed sentences into the vector space
let embeddings = bert.embed_batch(sentences).await?;
println!("embeddings {:?}", embeddings);
// Create a vector database from the embeddings along with a map between the embedding ids and the sentences
let db = VectorDB::new()?;
let embeddings = db.add_embeddings(embeddings)?;
let embedding_id_to_sentence: HashMap =
HashMap::from_iter(embeddings.into_iter().zip(sentences));
// Embed a query into the vector space. We use `embed_query` instead of `embed` because some models embed queries differently than normal text.
let embedding = bert.embed_query("What is Kalosm?").await?;
let closest = db.search(&embedding).run()?;
if let [closest] = closest.as_slice() {
let distance = closest.distance;
let text = embedding_id_to_sentence.get(&closest.value).unwrap();
println!("distance: {distance}");
println!("closest: {text}");
}
# Ok(())
# }
```
The vector database should find that the closest sentence to "What is Kalosm?" is "Kalosm can be used to build local AI applications":
```text
distance: 0.18480265
closest: Kalosm can be used to build local AI applications
```
---
## File: interfaces/language-model/docs/task.md
# Tasks
Any model that implements [`ChatModel`] or [`StructuredChatModel`] can be used with tasks to repeatedly perform work with the same system prompt.
You can create a task with the [`ChatModelExt::task`] method with a description of the task and then call the task like a function to start generating a response:
```rust, no_run
use kalosm::language::*;
#[tokio::main]
async fn main() {
let model = Llama::new_chat().await.unwrap();
let task = model.task("You are an editing assistant who offers suggestions for improving the quality of the text. You will be given some text and will respond with a list of suggestions for how to improve the text.");
let mut stream = task(&"this isnt correct. or is it?");
stream.to_std_out().await.unwrap();
}
```
Once you have the response builder, you can modify it with any of the methods on [`ChatResponseBuilder`]. For example, you can change the sampler with [`ChatResponseBuilder::with_sampler`]:
```rust, no_run
use kalosm::language::*;
#[tokio::main]
async fn main() {
let model = Llama::new_chat().await.unwrap();
let task = model.task("You are an editing assistant who offers suggestions for improving the quality of the text. You will be given some text and will respond with a list of suggestions for how to improve the text.");
let mut stream = task(&"this isnt correct. or is it?").with_sampler(GenerationParameters::default());
stream.to_std_out().await.unwrap();
}
```
## Structured Generation
You can use structured generation to force the output of the task to fit a specific format. Before you add structured generation to the tasks, you need to define a parser.
### Defining the parser
There are a few different ways create a parser for structured generation:
1. Derive a parser for your data
2. Create a parser from the set of prebuilt combinators
3. Create a parser from a regex
#### Deriving a parser from a struct
The simplest way to get started is to derive a parser for your data:
```rust, no_run
# use kalosm::language::*;
#[derive(Parse, Clone)]
struct Pet {
name: String,
age: u32,
}
```
Then you can generate text that works with the parser in a [`Task`](https://docs.rs/kalosm/latest/kalosm/language/struct.Task.html):
```rust, no_run
# use kalosm::language::*;
# use std::sync::Arc;
# #[derive(Parse, Clone, Debug)]
# struct Pet {
# name: String,
# age: u32,
# }
#[tokio::main]
async fn main() {
// First create a model
let model = Llama::new_chat().await.unwrap();
// Then create a parser for your data. Any type that implements the `Parse` trait has the `new_parser` method
let parser = Pet::new_parser();
// Create a task with the constraints
let task = model.task("You generate realistic JSON placeholders for pets in the form {\"name\": \"Pet name\", \"age\": 0, \"description\": \"Pet description\"}")
// The task constraints must be clone. If they don't implement Clone, you can wrap them in an Arc
.with_constraints(Arc::new(parser));
// Then run the task
let pet: Pet = task(&"Ruffles is a 3 year old adorable dog").await.unwrap();
println!("{pet:?}");
}
```
#### Creating a Parser from the Set of Prebuilt Combinators
Kalosm also provides a set of prebuilt combinators for creating more complex parsers. You can use these combinators to create a parser with a custom format:
```rust, no_run
use kalosm::language::*;
use std::sync::Arc;
#[tokio::main]
async fn main() {
// First create a model
let model = Llama::new_chat().await.unwrap();
// Then create a parser for your custom format
let parser = LiteralParser::from("[")
.ignore_output_then(String::new_parser())
.then_literal(", ")
.then(u8::new_parser())
.then_literal(", ")
.then(String::new_parser())
.then_literal("]");
// Create a task with the constraints
let task = model.task("You generate realistic JSON placeholders for pets in the form [\"Pet name\", age number, \"Pet description\"]")
// The task constraints must be clone. If they don't implement Clone, you can wrap them in an Arc
.with_constraints(Arc::new(parser));
// Then run the task
let ((name, age), description) = task(&"Ruffles is a 3 year old adorable dog").await.unwrap();
println!("{name} {age} {description}");
}
```
#### Creating a Parser from a Regex
You can also create a parser from a regex:
```rust, no_run
use kalosm::language::*;
use std::sync::Arc;
#[tokio::main]
async fn main() {
// First create a model
let model = Llama::new_chat().await.unwrap();
// Then create a parser for your data. Any
let parser = RegexParser::new(r"\[(\w+), (\d+), (\w+)\]").unwrap();
// Create a task with the constraints
let task = model.task("You generate realistic JSON placeholders for pets in the form [\"Pet name\", age number, \"Pet description\"]")
// The task constraints must be clone. If they don't implement Clone, you can wrap them in an Arc
.with_constraints(Arc::new(parser));
// Finally, run the task. Unlike derived and custom parsers, regex parsers do not provide a useful output type
task(&"Ruffles is a 3 year old adorable dog").to_std_out().await.unwrap();
}
```
### Tasks with Constraints
Once you have a parser, you can force the model to generate text that conforms to that parser with the [`Task::with_constraints`]:
```rust, no_run
use kalosm::language::*;
use std::sync::Arc;
#[derive(Parse, Clone, Debug)]
struct Pet {
name: String,
age: u32,
}
#[tokio::main]
async fn main() {
// First create a model
let model = Llama::new_chat().await.unwrap();
// Then create a parser for your data.
// Any type that implements the `Parse` trait has the `new_parser` method
let parser = Pet::new_parser();
// Create a task with the constraints
let task = model.task("You generate realistic JSON placeholders for pets in the form {\"name\": \"Pet name\", \"age\": 0, \"description\": \"Pet description\"}")
// The task constraints must be clone. If they don't implement Clone, you can wrap them in an Arc
.with_constraints(Arc::new(parser));
// Then run the task
let pet: Pet = task(&"Ruffles is a 3 year old adorable dog").await.unwrap();
println!("{pet:?}");
}
```
---
## File: interfaces/kalosm-vision/README.md
# Kalosm Vision
Kalosm Vision is a collection of image models and utilities for the Kalosm framework. It includes utilities for segmenting images into objects.
## Image Segmentation
Kalosm supports image segmentation with the [`SegmentAnything`] model. You can use the [`SegmentAnything::segment_everything`] method to segment an image into objects or the [`SegmentAnything::segment_from_points`] method to segment an image into objects at specific points:
```rust, no_run
use kalosm::vision::*;
#[tokio::main]
async fn main() {
let model = SegmentAnything::builder().build().await.unwrap();
let image = image::open("examples/landscape.jpg").unwrap();
let images = model
.segment_from_points(
SegmentAnythingInferenceSettings::new(image)
.add_goal_point_normalized(0.5, 0.25),
)
.await
.unwrap();
images.save("out.png").unwrap();
}
```
---
## File: interfaces/kalosm-sound/README.md
# Kalosm Sound
Kalosm Sound is a collection of audio models and utilities for the Kalosm framework. It supports several [voice activity detection models](crate::VoiceActivityDetectorExt), and provides utilities for [transcribing audio into text](crate::AsyncSourceTranscribeExt).
## Sound Streams
Models in kalosm sound work with any [`AsyncSource`]. You can use [`MicInput::stream`] to stream audio from the microphone, or any synchronous audio source that implements [`rodio::Source`] like a mp3 or wav file.
You can transform the audio streams with:
- [`VoiceActivityDetectorExt::voice_activity_stream`]: Detect voice activity in the audio data
- [`DenoisedExt::denoise_and_detect_voice_activity`]: Denoise the audio data and detect voice activity
- [`AsyncSourceTranscribeExt::transcribe`]: Chunk an audio stream based on voice activity and then transcribe the chunked audio data
- [`VoiceActivityStreamExt::rechunk_voice_activity`]: Chunk an audio stream based on voice activity
- [`VoiceActivityStreamExt::filter_voice_activity`]: Filter chunks of audio data based on voice activity
- [`TranscribeChunkedAudioStreamExt::transcribe`]: Transcribe a chunked audio stream
## Voice Activity Detection
VAD models are used to detect when a speaker is speaking in a given audio stream. The simplest way to use a VAD model is to create an audio stream and call [`VoiceActivityDetectorExt::voice_activity_stream`] to stream audio chunks that are actively being spoken:
```rust, no_run
use kalosm::sound::*;
#[tokio::main]
async fn main() {
// Get the default microphone input
let mic = MicInput::default();
// Stream the audio from the microphone
let stream = mic.stream();
// Detect voice activity in the audio stream
let mut vad = stream.voice_activity_stream();
while let Some(input) = vad.next().await {
println!("Probability: {}", input.probability);
}
}
```
Kalosm also provides [`VoiceActivityStreamExt::rechunk_voice_activity`] to collect chunks of consecutive audio samples with a high vad probability. This can be useful for applications like speech recognition where context between consecutive audio samples is important.
```rust, no_run
use kalosm::sound::*;
use rodio::Source;
#[tokio::main]
async fn main() {
// Get the default microphone input
let mic = MicInput::default();
// Stream the audio from the microphone
let stream = mic.stream();
// Chunk the audio into chunks of speech
let vad = stream.voice_activity_stream();
let mut audio_chunks = vad.rechunk_voice_activity();
// Print the chunks as they are streamed in
while let Some(input) = audio_chunks.next().await {
println!("New voice activity chunk with duration {:?}", input.total_duration());
}
}
```
## Transcription
You can use the [`Whisper`] model to transcribe audio into text. Kalosm can transcribe any [`AsyncSource`] into a transcription stream with the [`AsyncSourceTranscribeExt::transcribe`] method:
```rust, no_run
use kalosm::sound::*;
#[tokio::main]
async fn main() {
// Get the default microphone input
let mic = MicInput::default();
// Stream the audio from the microphone
let stream = mic.stream();
// Transcribe the audio into text with the default Whisper model
let mut transcribe = stream.transcribe(Whisper::new().await.unwrap());
// Print the text as it is streamed in
transcribe.to_std_out().await.unwrap();
}
```