![Image][ten-framework-banner]
[![TEN Releases][ten-releases-badge]][ten-releases]
[![Coverage Status][coverage-badge]][coverage]
[![Release Date][release-date-badge]][ten-releases]
[![Commits][commits-badge]][commit-activity]
[![Issues closed][issues-closed-badge]][issues-closed]
[![Contributors][contributors-badge]][contributors]
[![GitHub license][license-badge]][license]
[![Ask DeepWiki][deepwiki-badge]][deepwiki]
[![ReadmeX][readmex-badge]][readmex]
[![README in English][lang-en-badge]][lang-en-readme]
[![简体中文操作指南][lang-zh-badge]][lang-zh-readme]
[![日本語のREADME][lang-jp-badge]][lang-jp-readme]
[![README in 한국어][lang-kr-badge]][lang-kr-readme]
[![README en Español][lang-es-badge]][lang-es-readme]
[![README en Français][lang-fr-badge]][lang-fr-readme]
[![README in Italiano][lang-it-badge]][lang-it-readme]
[Official Site][official-site] •
[Documentation][documentation] •
[Blog][blog]
Table of Contents
- [Welcome to TEN][welcome-to-ten]
- [Agent Examples][agent-examples-section]
- [Quick Start with Agent Examples][quick-start]
- [Localhost][localhost-section]
- [Codespaces][codespaces-section]
- [Agent Examples Self-Hosting][agent-examples-self-hosting]
- [Deploying with Docker][deploying-with-docker]
- [Deploying with other cloud services][deploying-with-other-cloud-services]
- [Stay Tuned][stay-tuned]
- [TEN Ecosystem][ten-ecosystem-anchor]
- [Questions][questions]
- [Contributing][contributing]
- [Code Contributors][code-contributors]
- [Contribution Guidelines][contribution-guidelines]
- [License][license-section]
## Welcome to TEN
TEN is an open-source framework for real-time multimodal conversational AI.
[TEN Ecosystem][ten-ecosystem-anchor] includes [TEN Framework][ten-framework], [Agent Examples][agent-examples-repo], [VAD][ten-vad], [Turn Detection][ten-turn-detection] and [Portal][ten-portal].
| Community Channel | Purpose |
| ---------------- | ------- |
| [![Follow on X][follow-on-x-badge]][follow-on-x] | Follow TEN Framework on X for updates and announcements |
| [![Discord TEN Community][discord-badge]][discord-invite] | Join our Discord community to connect with developers |
| [![Follow on LinkedIn][linkedin-badge]][linkedin] | Follow TEN Framework on LinkedIn for updates and announcements |
| [![Hugging Face Space][hugging-face-badge]][hugging-face] | Join our Hugging Face community to explore our spaces and models |
## Agent Examples
![Image][voice-assistant-image]
Multi-Purpose Voice Assistant — This low-latency, high-quality real-time assistant supports both RTC and [WebSocket][websocket-example] connections, and you can extend it with [Memory][memory-example], [VAD][voice-assistant-vad-example], [Turn Detection][voice-assistant-turn-detection-example], and other extensions.
See the [Example code][voice-assistant-example] for more details.
![divider][divider-light]
![divider][divider-dark]
![Image][doodler-image]
Doodler — A doodle board that turns spoken or typed prompts into simple hand-drawn sketches, complete with a crayon palette and real-time drawing.
[Example code][doodler-example]
![divider][divider-light]
![divider][divider-dark]
![Image][speaker-diarization-image]
Speaker Diarization — Real-time diarization that detects and labels speakers, the Who Likes What game shows an interactive use case.
[Example code][speechmatics-diarization-example]
![divider][divider-light]
![divider][divider-dark]
![Image][lip-sync-image]
Lip Sync Avatars — Works with multiple avatar vendors, the main character features Kei, an anime character with MotionSync-powered lip sync, and also supports realistic avatars from Trulience, HeyGen, and Tavus.
See the [Example code][voice-assistant-live2d-example] for different Live2D characters.
![divider][divider-light]
![divider][divider-dark]
![Image][sip-call-image]
SIP Call — SIP extension that enables phone calls powered by TEN.
[Example code][voice-assistant-sip-example]
![divider][divider-light]
![divider][divider-dark]
![Image][transcription-image]
Transcription — A transcription tool that transcribes audio to text.
[Example code][transcription-example]
![divider][divider-light]
![divider][divider-dark]
![Image][esp32-image]
ESP32-S3 Korvo V3 — Runs TEN agent example on the Espressif ESP32-S3 Korvo V3 development board to integrate LLM-powered communication with hardware.
See the [integration guide][esp32-guide] for more details.
[![][back-to-top]][readme-top]
## Quick Start with Agent Examples
### Localhost
#### Step ⓵ - Prerequisites
| Category | Requirements |
| --- | --- |
| **Keys** | • Agora [App ID][agora-app-id] and [App Certificate][agora-app-certificate] • [OpenAI][openai-api] API key • [Deepgram][deepgram] ASR • [ElevenLabs][elevenlabs] TTS |
| **Installation** | • [Docker][docker] / [Docker Compose][docker-compose] • [Node.js (LTS) v18][nodejs] |
| **Minimum System Requirements** | • CPU >= 2 cores • RAM >= 4 GB |
![divider][divider-light]
![divider][divider-dark]
#### Step ⓶ - Build agent examples in VM
##### 1. Clone the repo, `cd` into `ai_agents`, and create a `.env` file from `.env.example`
```bash
cd ai_agents
cp ./.env.example ./.env
```
##### 2. Set up the Agora App ID and App Certificate in `.env`
```bash
AGORA_APP_ID=
AGORA_APP_CERTIFICATE=
# Deepgram (required for speech-to-text)
DEEPGRAM_API_KEY=
# OpenAI (required for language model)
OPENAI_API_KEY=
# ElevenLabs (required for text-to-speech)
ELEVENLABS_TTS_KEY=
```
##### 3. Start agent development containers
```bash
docker compose up -d
```
##### 4. Enter the container
```bash
docker exec -it ten_agent_dev bash
```
##### 5. Build the agent with the default example (~5-8 min)
Check the `agents/examples` folder for additional samples.
Start with one of these defaults:
```bash
# use the chained voice assistant
cd agents/examples/voice-assistant
# or use the speech-to-speech voice assistant in real time
cd agents/examples/voice-assistant-realtime
```
##### 6. Start the web server
Run `task install` before the first launch and after changing dependencies or Go source code. It installs the TEN, Python, and frontend dependencies and builds the Go API server. Python-only source changes do not require reinstalling.
```bash
task install
task run
```
##### 7. Access the agent
Once the agent example is running, you can access the following interfaces:
| **localhost:49483** | **localhost:3000** |
| :-----------------: | :----------------: |
| ![Screenshot 1][localhost-49483-image] | ![Screenshot 2][localhost-3000-image] |
- TMAN Designer: [localhost:49483][localhost-49483]
- Agent Examples UI: [localhost:3000][localhost-3000]
![divider][divider-light]
![divider][divider-dark]
#### Step ⓷ - Customize your agent example
1. Open [localhost:49483][localhost-49483].
2. Right-click the STT, LLM, and TTS extensions.
3. Open their properties and enter the corresponding API keys.
4. Submit your changes, now you can see the updated Agent Example in [localhost:3000][localhost-3000].
![divider][divider-light]
![divider][divider-dark]
#### Run a transcriber app from TEN Manager without Docker (Beta)
TEN also provides a transcriber app that you can run from TEN Manager without using Docker.
Check the [quick start guide][quick-start-guide-ten-manager] for more details.
![divider][divider-light]
![divider][divider-dark]
### Codespaces
GitHub offers free Codespaces for each repository. You can run Agent Examples in Codespaces without using Docker. Codespaces typically start faster than local Docker environments.
[![][codespaces-shield]][codespaces-new]
Check out [this guide][codespaces-guide] for more details.
[![][back-to-top]][readme-top]
## Agent Examples Self-Hosting
### Deploying with Docker
Once you have customized your agent (either by using the TMAN Designer or editing `property.json` directly), you can deploy it by creating a release Docker image for your service.
##### Release as Docker image
**Note**: The following commands need to be executed outside of any Docker container.
###### Build image
```bash
cd ai_agents
docker build -f agents/examples//Dockerfile -t example-app .
```
###### Run
```bash
docker run --rm -it --env-file .env -p 3000:3000 example-app
```
![divider][divider-light]
![divider][divider-dark]
### Deploying with other cloud services
You can split the deployment into two pieces when you want to host TEN on providers such as [Vercel][vercel] or [Netlify][netlify].
1. Run the TEN backend on any container-friendly platform (a VM with Docker, Fly.io, Render, ECS, Cloud Run, or similar). Use the example Docker image without modifying it and expose port `8080` from that service.
2. Deploy only the frontend to Vercel or Netlify. Point the project root to `ai_agents/agents/examples//frontend`, run `pnpm install` (or `bun install`) followed by `pnpm build` (or `bun run build`), and keep the default `.next` output directory.
3. Configure environment variables in your hosting dashboard so that `AGENT_SERVER_URL` points to the backend URL, and add any `NEXT_PUBLIC_*` keys the UI needs (for example, Agora credentials you surface to the browser).
4. Ensure your backend accepts requests from the frontend origin — via open CORS or by using the built-in proxy middleware.
With this setup, the backend handles long-running worker processes, while the hosted frontend simply forwards API traffic to it.
[![][back-to-top]][readme-top]
## Stay Tuned
Get instant notifications for new releases and updates. Your support helps us grow and improve TEN!
![Image][stay-tuned-image]
[![][back-to-top]][readme-top]
## TEN Ecosystem
| Project | Preview |
| ------- | ------- |
| [**️TEN Framework**][ten-framework-link] Open-source framework for conversational AI Agents.
| ![][ten-agent-example-banner] |
| [**TEN Portal**][ten-portal-link] The official site of the TEN Framework with documentation and a blog.
![][ten-portal-shield] | ![][ten-portal-banner] |
[![][back-to-top]][readme-top]
## Questions
TEN Framework is available on these AI-powered Q&A platforms. They can help you find answers quickly and accurately in multiple languages, covering everything from basic setup to advanced implementation details.
| Service | Link |
| ------- | ---- |
| DeepWiki | [![Ask DeepWiki][deepwiki-badge]][deepwiki] |
| ReadmeX | [![ReadmeX][readmex-badge]][readmex] |
[![][back-to-top]][readme-top]
## Contributing
We welcome all forms of open-source collaboration! Whether you're fixing bugs, adding features, improving documentation, or sharing ideas, your contributions help advance personalized AI tools. Check out our GitHub Issues and Projects to find ways to contribute and show your skills. Together, we can build something amazing!
> [!TIP]
>
> **Welcome all kinds of contributions** 🙏
>
> Join us in building TEN better! Every contribution makes a difference, from code to documentation. Share your TEN Agent projects on social media to inspire others!
>
> Connect with one of the TEN maintainers [@elliotchen200][elliotchen200-x] on 𝕏 or [@cyfyifanchen][cyfyifanchen-github] on GitHub for project updates, discussions, and collaboration opportunities.
![divider][divider-light]
![divider][divider-dark]
### Code Contributors
[![TEN][contributors-image]][contributors]
### Contribution Guidelines
Contributions are welcome! Please read the [contribution guidelines][contribution-guidelines-doc] first.
![divider][divider-light]
![divider][divider-dark]
### License
1. The entire TEN framework (except for the folders explicitly listed below) is released pursuant the Apache License, Version 2.0, with additional restrictions. For details, please refer to the [LICENSE][license-file] file located in the root directory of the TEN framework.
2. The components within the `packages` directory are released under the Apache License, Version 2.0. For details, please refer to the `LICENSE` file located in each package's root directory.
3. The third-party libraries used by the TEN framework are listed and described in detail. For more information, please refer to the [third_party][third-party-folder] folder.
[![][back-to-top]][readme-top]
[back-to-top]: https://img.shields.io/badge/-Back_to_top-gray?style=flat-square
[readme-top]: #readme-top
[welcome-to-ten]: #welcome-to-ten
[agent-examples-section]: #agent-examples
[quick-start]: #quick-start-with-agent-examples
[localhost-section]: #localhost
[codespaces-section]: #codespaces
[agent-examples-self-hosting]: #agent-examples-self-hosting
[deploying-with-docker]: #deploying-with-docker
[deploying-with-other-cloud-services]: #deploying-with-other-cloud-services
[stay-tuned]: #stay-tuned
[ten-ecosystem-anchor]: #ten-ecosystem
[questions]: #questions
[contributing]: #contributing
[code-contributors]: #code-contributors
[contribution-guidelines]: #contribution-guidelines
[license-section]: #license
[ten-releases-badge]: https://img.shields.io/github/v/release/ten-framework/ten-framework?color=369eff&labelColor=gray&logo=github&style=flat-square
[ten-releases]: https://github.com/TEN-framework/ten-framework/releases
[coverage-badge]: https://coveralls.io/repos/github/TEN-framework/ten-framework/badge.svg?branch=main
[coverage]: https://coveralls.io/github/TEN-framework/ten-framework?branch=main
[release-date-badge]: https://img.shields.io/github/release-date/ten-framework/ten-framework?labelColor=gray&style=flat-square
[commits-badge]: https://img.shields.io/github/commit-activity/m/TEN-framework/ten-framework?labelColor=gray&color=pink
[commit-activity]: https://github.com/TEN-framework/ten-framework/graphs/commit-activity
[issues-closed-badge]: https://img.shields.io/github/issues-search?query=repo%3ATEN-framework%2Ften-framework%20is%3Aclosed&label=issues%20closed&labelColor=gray&color=green
[issues-closed]: https://github.com/TEN-framework/ten-framework/issues
[contributors-badge]: https://img.shields.io/github/contributors/ten-framework/ten-framework?color=c4f042&labelColor=gray&style=flat-square
[contributors]: https://github.com/TEN-framework/ten-framework/graphs/contributors
[license-badge]: https://img.shields.io/badge/License-Apache_2.0_with_certain_conditions-blue.svg?labelColor=%20%23155EEF&color=%20%23528bff
[license]: https://github.com/TEN-framework/ten-framework/blob/main/LICENSE
[deepwiki-badge]: https://deepwiki.com/badge.svg
[deepwiki]: https://deepwiki.com/TEN-framework/TEN-framework
[readmex-badge]: https://raw.githubusercontent.com/CodePhiliaX/resource-trusteeship/main/readmex.svg
[readmex]: https://readmex.com/TEN-framework/ten-framework
[trendshift-badge]: https://trendshift.io/api/badge/repositories/11978
[trendshift]: https://trendshift.io/repositories/11978
[lang-en-badge]: https://img.shields.io/badge/English-lightgrey
[lang-en-readme]: https://github.com/TEN-framework/ten-framework/blob/main/README.md
[lang-zh-badge]: https://img.shields.io/badge/简体中文-lightgrey
[lang-zh-readme]: https://github.com/TEN-framework/ten-framework/blob/main/docs/README-CN.md
[lang-jp-badge]: https://img.shields.io/badge/日本語-lightgrey
[lang-jp-readme]: https://github.com/TEN-framework/ten-framework/blob/main/docs/README-JP.md
[lang-kr-badge]: https://img.shields.io/badge/한국어-lightgrey
[lang-kr-readme]: https://github.com/TEN-framework/ten-framework/blob/main/docs/README-KR.md
[lang-es-badge]: https://img.shields.io/badge/Español-lightgrey
[lang-es-readme]: https://github.com/TEN-framework/ten-framework/blob/main/docs/README-ES.md
[lang-fr-badge]: https://img.shields.io/badge/Français-lightgrey
[lang-fr-readme]: https://github.com/TEN-framework/ten-framework/blob/main/docs/README-FR.md
[lang-it-badge]: https://img.shields.io/badge/Italiano-lightgrey
[lang-it-readme]: https://github.com/TEN-framework/ten-framework/blob/main/docs/README-IT.md
[official-site]: https://theten.ai
[documentation]: https://theten.ai/docs
[blog]: https://theten.ai/blog
[ten-framework]: https://github.com/ten-framework/ten-framework
[agent-examples-repo]: https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples
[ten-vad]: https://github.com/ten-framework/ten-vad
[ten-turn-detection]: https://github.com/ten-framework/ten-turn-detection
[ten-portal]: https://github.com/ten-framework/portal
[follow-on-x-badge]: https://img.shields.io/twitter/follow/TenFramework?logo=X&color=%20%23f5f5f5
[follow-on-x]: https://twitter.com/intent/follow?screen_name=TenFramework
[discord-badge]: https://img.shields.io/badge/Discord-Join%20TEN%20Community-5865F2?style=flat&logo=discord&logoColor=white
[discord-invite]: https://discord.gg/VnPftUzAMJ
[linkedin-badge]: https://custom-icon-badges.demolab.com/badge/LinkedIn-TEN_Framework-0A66C2?logo=linkedin-white&logoColor=fff
[linkedin]: https://www.linkedin.com/company/ten-framework
[hugging-face-badge]: https://img.shields.io/badge/Hugging%20Face-TEN%20Framework-yellow?style=flat&logo=huggingface
[hugging-face]: https://huggingface.co/TEN-framework
[wechat-badge]: https://img.shields.io/badge/TEN_Framework-WeChat_Group-%2307C160?logo=wechat&labelColor=darkgreen&color=gray
[wechat-discussion]: https://github.com/TEN-framework/ten-agent/discussions/170
[voice-assistant-image]: https://github.com/user-attachments/assets/dce3db80-fb48-4e2a-8ac7-33f50bcffa32
[websocket-example]: ai_agents/agents/examples/websocket-example
[memory-example]: ai_agents/agents/examples/voice-assistant-with-memU
[voice-assistant-vad-example]: ai_agents/agents/examples/voice-assistant-with-ten-vad
[voice-assistant-turn-detection-example]: ai_agents/agents/examples/voice-assistant-with-turn-detection
[voice-assistant-example]: ai_agents/agents/examples/voice-assistant
[divider-light]: https://github.com/user-attachments/assets/aec54c94-ced9-4683-ae58-0a5a7ed803bd#gh-light-mode-only
[divider-dark]: https://github.com/user-attachments/assets/d57fad08-4f49-4a1c-bdfc-f659a5d86150#gh-dark-mode-only
[lip-sync-image]: https://github.com/user-attachments/assets/51ab1504-b67c-49d4-8a7a-5582d9b254da
[voice-assistant-live2d-example]: ai_agents/agents/examples/voice-assistant-live2d
[speaker-diarization-image]: https://github.com/user-attachments/assets/f94b21b8-9dda-4efc-9274-b028cc01296a
[speechmatics-diarization-example]: ai_agents/agents/examples/speaker-diarization
[sip-call-image]: https://github.com/user-attachments/assets/6ed5b04d-945a-4a30-a1cc-f8014b602b38
[voice-assistant-sip-example]: ai_agents/agents/examples/voice-assistant-sip-twilio
[transcription-image]: https://github.com/user-attachments/assets/d793bc6c-c8de-4996-bd85-9ce88c69dd8d
[transcription-example]: ai_agents/agents/examples/transcription
[doodler-image]: https://github.com/user-attachments/assets/80c4eabd-de96-4971-8956-6b365d4fbd64
[doodler-example]: ai_agents/agents/examples/doodler
[esp32-image]: https://github.com/user-attachments/assets/3d60f1ff-0f82-4fe7-b5c2-ac03d284f60c
[esp32-guide]: ai_agents/esp32-client
[agora-app-id]: https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project
[agora-app-certificate]: https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project
[openai-api]: https://openai.com/index/openai-api/
[deepgram]: https://deepgram.com/
[elevenlabs]: https://elevenlabs.io/
[docker]: https://www.docker.com/
[docker-compose]: https://docs.docker.com/compose/
[nodejs]: https://nodejs.org/en
[quick-start-guide-ten-manager]: https://theten.ai/docs/ten_framework/getting-started/quick-start
[localhost-49483-image]: https://github.com/user-attachments/assets/191a7c0a-d8e6-48f9-866f-6a70c58f0118
[localhost-3000-image]: https://github.com/user-attachments/assets/13e482b6-d907-4449-a779-9454bb24c0b1
[localhost-49483]: http://localhost:49483
[localhost-3000]: http://localhost:3000
[codespaces-shield]: https://github.com/codespaces/badge.svg
[codespaces-new]: https://codespaces.new/ten-framework/ten-agent
[codespaces-guide]: https://theten.ai/docs/ten_agent_examples/setup_development_env/setting_up_development_inside_codespace
[vercel]: https://vercel.com
[netlify]: https://www.netlify.com
[stay-tuned-image]: https://github.com/user-attachments/assets/72c6cc46-a2a2-484d-82a9-f3079269c815
[ten-framework-shield]: https://img.shields.io/github/stars/ten-framework/ten-framework?color=ffcb47&labelColor=gray&style=flat-square&logo=github
[ten-framework-banner]: https://github.com/user-attachments/assets/a99e13c5-aca4-4b28-bedf-10c89eedcd86
[ten-framework-link]: https://github.com/ten-framework/ten-framework
[ten-vad-link]: https://github.com/ten-framework/ten-vad
[ten-vad-shield]: https://img.shields.io/github/stars/ten-framework/ten-vad?color=ffcb47&labelColor=gray&style=flat-square&logo=github
[ten-vad-banner]: https://github.com/user-attachments/assets/e504135e-67fd-4fa1-b0e4-d495358d8aa5
[ten-turn-detection-link]: https://github.com/ten-framework/ten-turn-detection
[ten-turn-detection-shield]: https://img.shields.io/github/stars/ten-framework/ten-turn-detection?color=ffcb47&labelColor=gray&style=flat-square&logo=github
[ten-turn-detection-banner]: https://github.com/user-attachments/assets/c72d82cc-3667-496c-8bd6-3d194a91c452
[ten-agent-example-link]: https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples
[ten-agent-example-banner]: https://github.com/user-attachments/assets/7f735633-c7f6-4432-b6b4-d2a2977ca588
[ten-portal-link]: https://github.com/ten-framework/portal
[ten-portal-shield]: https://img.shields.io/github/stars/ten-framework/portal?color=ffcb47&labelColor=gray&style=flat-square&logo=github
[ten-portal-banner]: https://github.com/user-attachments/assets/f56c75b9-722c-4156-902d-ae98ce2b3b5e
[elliotchen200-x]: https://x.com/elliotchen200
[cyfyifanchen-github]: https://github.com/cyfyifanchen
[contributors-image]: https://contrib.rocks/image?repo=TEN-framework/ten-framework
[contribution-guidelines-doc]: ./docs/code-of-conduct/contributing.md
[license-file]: ./LICENSE
[third-party-folder]: ./third_party/
---
### Third Party Curl Docs README (`third_party/curl/docs/README.md`)
# Documentation
you will find a mix of various documentation in this directory and
subdirectories, using several different formats. Some of them are not ideal
for reading directly in your browser.
If you would rather see the rendered version of the documentation, check out the
curl website's [documentation section](https://curl.se/docs/) for
general curl stuff or the [libcurl section](https://curl.se/libcurl/) for
libcurl related documentation.
---
### Third Party Nlohmann Json Docs README (`third_party/nlohmann_json/docs/README.md`)
# Documentation
## Generate documentation
Note on documentation: The source files contain links to the online documentation at https://json.nlohmann.me. This URL
contains the most recent documentation and should also be applicable to previous versions; documentation for deprecated
functions is not removed, but marked deprecated.
If you want to see the documentation for a specific tag or commit hash, you can generate it as follows (here for tag
`v3.10.2`):
```shell
git clone https://github.com/nlohmann/json.git
cd json
git checkout v3.10.2
make install_venv serve -C docs/mkdocs
```
Open URL in your browser. Replace from any URL from the source code `https://json.nlohmann.me`
with `http://127.0.0.1:8000` to see the documentation for your tag or commit hash.
---
### Ai L1 L2 Index (`docs/ai/L1/L2/_index.md`)
# Deep Dives Index
| Document | Summary | Load When |
| -------------------------------------------------------- | ------------------------------------------------ | ------------------------------------------------ |
| [extension_development.md](extension_development.md) | Full extension creation guide, base classes, test configs, pre-submission checklist | Creating a new TTS/ASR/LLM extension |
| [asr_plugin_design_review.md](asr_plugin_design_review.md) | Required ASR design rules, lifecycle pitfalls, and PR review checklist | Adding or modifying any ASR extension |
| [graph_configuration.md](graph_configuration.md) | Graph nodes, connections, routing, property.json | Modifying graphs or wiring extensions together |
| [testing.md](testing.md) | Core TTS and ASR guarder tests, pass criteria, config files, debugging | Running or debugging tests for an extension |
| [deployment.md](deployment.md) | Docker, Cloudflare, Nginx, Grafana monitoring | Deploying to production or setting up monitoring |
| [server_architecture.md](server_architecture.md) | Go server, property injection, worker lifecycle | Understanding server internals or debugging |
| [operations_restarts.md](operations_restarts.md) | Full restart procedures, port debugging, recovery| Restarting services, crash recovery, port conflicts|
---
### Third Party Curl Docs Examples README (`third_party/curl/docs/examples/README.md`)
# libcurl examples
This directory is for libcurl programming examples. They are meant to show
some simple steps on how you can build your own application to take full
advantage of libcurl.
If you end up with other small but still useful example sources, please mail
them for submission in future packages and on the website.
## Building
The `Makefile.example` is an example Makefile that could be used to build
these examples. Just edit the file according to your system and requirements
first.
Most examples should build fine using a command line like this:
`curl-config --cc --cflags --libs` -o example example.c
Some compilers do not like having the arguments in this order but instead
want you do reorganize them like:
`curl-config --cc` -o example example.c `curl-config --cflags --libs`
**Please** do not use the `curl.se` site as a test target for your libcurl
applications/experiments. Even if some of the examples use that site as a URL
at some places, it does not mean that the URLs work or that we expect you to
actually torture our website with your tests. Thanks.
## Examples
Each example source code file is designed to be and work stand-alone and
rather self-explanatory. The examples may at times lack the level of error
checks you need in a real world, but that is then only for the sake of
readability: to make the code smaller and easier to follow.
---
### Third Party Nlohmann Json Docs Docset README (`third_party/nlohmann_json/docs/docset/README.md`)
# docset
The folder contains the required files to create a [docset](https://kapeli.com/docsets) which can be used in
documentation browsers like [Dash](https://kapeli.com/dash), [Velocity](https://velocity.silverlakesoftware.com), or
[Zeal](https://zealdocs.org).
The docset can be created with
```sh
make nlohmann_json.docset
```
The generated folder `nlohmann_json.docset` can then be opened in the documentation browser.
A recent version is also part of the [Dash user contributions](https://github.com/Kapeli/Dash-User-Contributions/tree/master/docsets/JSON_for_Modern_C%2B%2B).
## Licenses
The [JSON logo](https://commons.wikimedia.org/wiki/File:JSON_vector_logo.svg) is public domain.
---
### Third Party Clingo Sys Clingo Third Party Catch Docs Readme (`third_party/clingo-sys/clingo/third_party/catch/docs/Readme.md`)
# Reference
To get the most out of Catch2, start with the [tutorial](tutorial.md#top).
Once you're up and running consider the following reference material.
**Writing tests:**
* [Assertion macros](assertions.md#top)
* [Matchers (asserting complex properties)](matchers.md#top)
* [Comparing floating point numbers](comparing-floating-point-numbers.md#top)
* [Logging macros](logging.md#top)
* [Test cases and sections](test-cases-and-sections.md#top)
* [Test fixtures](test-fixtures.md#top)
* [Explicitly skipping, passing, and failing tests at runtime](skipping-passing-failing.md#top)
* [Reporters (output customization)](reporters.md#top)
* [Event Listeners](event-listeners.md#top)
* [Data Generators (value parameterized tests)](generators.md#top)
* [Other macros](other-macros.md#top)
* [Micro benchmarking](benchmarks.md#top)
**Fine tuning:**
* [Supplying your own main()](own-main.md#top)
* [Compile-time configuration](configuration.md#top)
* [String Conversions](tostring.md#top)
**Running:**
* [Command line](command-line.md#top)
**Odds and ends:**
* [Frequently Asked Questions (FAQ)](faq.md#top)
* [Best practices and other tips](usage-tips.md#top)
* [CMake integration](cmake-integration.md#top)
* [Tooling integration (CI, test runners, other)](ci-and-misc.md#top)
* [Known limitations](limitations.md#top)
**Other:**
* [Why Catch2?](why-catch.md#top)
* [Migrating from v2 to v3](migrate-v2-to-v3.md#top)
* [Open Source Projects using Catch2](opensource-users.md#top)
* [Commercial Projects using Catch2](commercial-users.md#top)
* [Contributing](contributing.md#top)
* [Release Notes](release-notes.md#top)
* [Deprecations and incoming changes](deprecations.md#top)
---
### Third Party Nlohmann Json Docs Mkdocs Docs Integration Index (`third_party/nlohmann_json/docs/mkdocs/docs/integration/index.md`)
# Header only
[`json.hpp`](https://github.com/nlohmann/json/blob/develop/single_include/nlohmann/json.hpp) is the single required
file in `single_include/nlohmann` or [released here](https://github.com/nlohmann/json/releases). You need to add
```cpp
#include
// for convenience
using json = nlohmann::json;
```
to the files you want to process JSON and set the necessary switches to enable C++11 (e.g., `-std=c++11` for GCC and
Clang).
You can further use file
[`single_include/nlohmann/json_fwd.hpp`](https://github.com/nlohmann/json/blob/develop/single_include/nlohmann/json_fwd.hpp)
for forward declarations.
---
### Third Party Nlohmann Json Docs Mkdocs Docs Api Adl Serializer Index (`third_party/nlohmann_json/docs/mkdocs/docs/api/adl_serializer/index.md`)
# nlohmann::adl_serializer
```cpp
template
struct adl_serializer;
```
Serializer that uses ADL ([Argument-Dependent Lookup](https://en.cppreference.com/w/cpp/language/adl)) to choose
`to_json`/`from_json` functions from the types' namespaces.
It is implemented similar to
```cpp
template
struct adl_serializer {
template
static void to_json(BasicJsonType& j, const T& value) {
// calls the "to_json" method in T's namespace
}
template
static void from_json(const BasicJsonType& j, T& value) {
// same thing, but with the "from_json" method
}
};
```
## Member functions
- [**from_json**](from_json.md) - convert a JSON value to any value type
- [**to_json**](to_json.md) - convert any value type to a JSON value
## Version history
- Added in version 2.1.0.
---
### Third Party Nlohmann Json Docs Mkdocs Docs Api Basic Json Index (`third_party/nlohmann_json/docs/mkdocs/docs/api/basic_json/index.md`)
# nlohmann::basic_json
Defined in header ``
```cpp
template<
template class ObjectType = std::map,
template class ArrayType = std::vector,
class StringType = std::string,
class BooleanType = bool,
class NumberIntegerType = std::int64_t,
class NumberUnsignedType = std::uint64_t,
class NumberFloatType = double,
template class AllocatorType = std::allocator,
template class JSONSerializer = adl_serializer,
class BinaryType = std::vector,
class CustomBaseClass = void
>
class basic_json;
```
## Template parameters
| Template parameter | Description | Derived type |
|----------------------|---------------------------------------------------------------------------|---------------------------------------------|
| `ObjectType` | type for JSON objects | [`object_t`](object_t.md) |
| `ArrayType` | type for JSON arrays | [`array_t`](array_t.md) |
| `StringType` | type for JSON strings and object keys | [`string_t`](string_t.md) |
| `BooleanType` | type for JSON booleans | [`boolean_t`](boolean_t.md) |
| `NumberIntegerType` | type for JSON integer numbers | [`number_integer_t`](number_integer_t.md) |
| `NumberUnsignedType` | type for JSON unsigned integer numbers | [`number_unsigned_t`](number_unsigned_t.md) |
| `NumberFloatType` | type for JSON floating-point numbers | [`number_float_t`](number_float_t.md) |
| `AllocatorType` | type of the allocator to use | |
| `JSONSerializer` | the serializer to resolve internal calls to `to_json()` and `from_json()` | [`json_serializer`](json_serializer.md) |
| `BinaryType` | type for binary arrays | [`binary_t`](binary_t.md) |
| `CustomBaseClass` | extension point for user code | [`json_base_class_t`](json_base_class_t.md) |
## Specializations
- [**json**](../json.md) - default specialization
- [**ordered_json**](../ordered_json.md) - specialization that maintains the insertion order of object keys
## Iterator invalidation
Todo
## Requirements
The class satisfies the following concept requirements:
### Basic
- [DefaultConstructible](https://en.cppreference.com/w/cpp/named_req/DefaultConstructible): JSON values can be default
constructed. The result will be a JSON null value.
- [MoveConstructible](https://en.cppreference.com/w/cpp/named_req/MoveConstructible): A JSON value can be constructed
from an rvalue argument.
- [CopyConstructible](https://en.cppreference.com/w/cpp/named_req/CopyConstructible): A JSON value can be
copy-constructed from an lvalue expression.
- [MoveAssignable](https://en.cppreference.com/w/cpp/named_req/MoveAssignable): A JSON value can be assigned from an
rvalue argument.
- [CopyAssignable](https://en.cppreference.com/w/cpp/named_req/CopyAssignable): A JSON value can be copy-assigned from
an lvalue expression.
- [Destructible](https://en.cppreference.com/w/cpp/named_req/Destructible): JSON values can be destructed.
### Layout
- [StandardLayoutType](https://en.cppreference.com/w/cpp/named_req/StandardLayoutType): JSON values have
[standard layout](https://en.cppreference.com/w/cpp/language/data_members#Standard_layout): All non-static data
members are private and standard layout types, the class has no virtual functions or (virtual) base classes.
### Library-wide
- [EqualityComparable](https://en.cppreference.com/w/cpp/named_req/EqualityComparable): JSON values can be compared with
`==`, see [`operator==`](operator_eq.md).
- [LessThanComparable](https://en.cppreference.com/w/cpp/named_req/LessThanComparable): JSON values can be compared with
`<`, see [`operator<`](operator_le.md).
- [Swappable](https://en.cppreference.com/w/cpp/named_req/Swappable): Any JSON lvalue or rvalue of can be swapped with
any lvalue or rvalue of other compatible types, using unqualified function `swap`.
- [NullablePointer](https://en.cppreference.com/w/cpp/named_req/NullablePointer): JSON values can be compared against
`std::nullptr_t` objects which are used to model the `null` value.
### Container
- [Container](https://en.cppreference.com/w/cpp/named_req/Container): JSON values can be used like STL containers and
provide iterator access.
- [ReversibleContainer](https://en.cppreference.com/w/cpp/named_req/ReversibleContainer): JSON values can be used like
STL containers and provide reverse iterator access.
## Member types
- [**adl_serializer**](../adl_serializer/index.md) - the default serializer
- [**value_t**](value_t.md) - the JSON type enumeration
- [**json_pointer**](../json_pointer/index.md) - JSON Pointer implementation
- [**json_serializer**](json_serializer.md) - type of the serializer to for conversions from/to JSON
- [**error_handler_t**](error_handler_t.md) - type to choose behavior on decoding errors
- [**cbor_tag_handler_t**](cbor_tag_handler_t.md) - type to choose how to handle CBOR tags
- **initializer_list_t** - type for initializer lists of `basic_json` values
- [**input_format_t**](input_format_t.md) - type to choose the format to parse
- [**json_sax_t**](../json_sax/index.md) - type for SAX events
### Exceptions
- [**exception**](exception.md) - general exception of the `basic_json` class
- [**parse_error**](parse_error.md) - exception indicating a parse error
- [**invalid_iterator**](invalid_iterator.md) - exception indicating errors with iterators
- [**type_error**](type_error.md) - exception indicating executing a member function with a wrong type
- [**out_of_range**](out_of_range.md) - exception indicating access out of the defined range
- [**other_error**](other_error.md) - exception indicating other library errors
### Container types
| Type | Definition |
|--------------------------|-----------------------------------------------------------------------------------------------------------|
| `value_type` | `#!cpp basic_json` |
| `reference` | `#!cpp value_type&` |
| `const_reference` | `#!cpp const value_type&` |
| `difference_type` | `#!cpp std::ptrdiff_t` |
| `size_type` | `#!cpp std::size_t` |
| `allocator_type` | `#!cpp AllocatorType` |
| `pointer` | `#!cpp std::allocator_traits::pointer` |
| `const_pointer` | `#!cpp std::allocator_traits::const_pointer` |
| `iterator` | [LegacyBidirectionalIterator](https://en.cppreference.com/w/cpp/named_req/BidirectionalIterator) |
| `const_iterator` | constant [LegacyBidirectionalIterator](https://en.cppreference.com/w/cpp/named_req/BidirectionalIterator) |
| `reverse_iterator` | reverse iterator, derived from `iterator` |
| `const_reverse_iterator` | reverse iterator, derived from `const_iterator` |
| `iteration_proxy` | helper type for [`items`](items.md) function |
### JSON value data types
- [**array_t**](array_t.md) - type for arrays
- [**binary_t**](binary_t.md) - type for binary arrays
- [**boolean_t**](boolean_t.md) - type for booleans
- [**default_object_comparator_t**](default_object_comparator_t.md) - default comparator for objects
- [**number_float_t**](number_float_t.md) - type for numbers (floating-point)
- [**number_integer_t**](number_integer_t.md) - type for numbers (integer)
- [**number_unsigned_t**](number_unsigned_t.md) - type for numbers (unsigned)
- [**object_comparator_t**](object_comparator_t.md) - comparator for objects
- [**object_t**](object_t.md) - type for objects
- [**string_t**](string_t.md) - type for strings
### Parser callback
- [**parse_event_t**](parse_event_t.md) - parser event types
- [**parser_callback_t**](parser_callback_t.md) - per-element parser callback type
## Member functions
- [(constructor)](basic_json.md)
- [(destructor)](~basic_json.md)
- [**operator=**](operator=.md) - copy assignment
- [**array**](array_t.md) (_static_) - explicitly create an array
- [**binary**](binary.md) (_static_) - explicitly create a binary array
- [**object**](object_t.md) (_static_) - explicitly create an object
### Object inspection
Functions to inspect the type of a JSON value.
- [**type**](type.md) - return the type of the JSON value
- [**operator value_t**](operator_value_t.md) - return the type of the JSON value
- [**type_name**](type_name.md) - return the type as string
- [**is_primitive**](is_primitive.md) - return whether type is primitive
- [**is_structured**](is_structured.md) - return whether type is structured
- [**is_null**](is_null.md) - return whether value is null
- [**is_boolean**](is_boolean.md) - return whether value is a boolean
- [**is_number**](is_number.md) - return whether value is a number
- [**is_number_integer**](is_number_integer.md) - return whether value is an integer number
- [**is_number_unsigned**](is_number_unsigned.md) - return whether value is an unsigned integer number
- [**is_number_float**](is_number_float.md) - return whether value is a floating-point number
- [**is_object**](is_object.md) - return whether value is an object
- [**is_array**](is_array.md) - return whether value is an array
- [**is_string**](is_string.md) - return whether value is a string
- [**is_binary**](is_binary.md) - return whether value is a binary array
- [**is_discarded**](is_discarded.md) - return whether value is discarded
### Value access
Direct access to the stored value of a JSON value.
- [**get**](get.md) - get a value
- [**get_to**](get_to.md) - get a value and write it to a destination
- [**get_ptr**](get_ptr.md) - get a pointer value
- [**get_ref**](get_ref.md) - get a reference value
- [**operator ValueType**](operator_ValueType.md) - get a value
- [**get_binary**](get_binary.md) - get a binary value
### Element access
Access to the JSON value
- [**at**](at.md) - access specified element with bounds checking
- [**operator[]**](operator[].md) - access specified element
- [**value**](value.md) - access specified object element with default value
- [**front**](front.md) - access the first element
- [**back**](back.md) - access the last element
### Lookup
- [**find**](find.md) - find an element in a JSON object
- [**count**](count.md) - returns the number of occurrences of a key in a JSON object
- [**contains**](contains.md) - check the existence of an element in a JSON object
### Iterators
- [**begin**](begin.md) - returns an iterator to the first element
- [**cbegin**](cbegin.md) - returns a const iterator to the first element
- [**end**](end.md) - returns an iterator to one past the last element
- [**cend**](cend.md) - returns a const iterator to one past the last element
- [**rbegin**](rbegin.md) - returns an iterator to the reverse-beginning
- [**rend**](rend.md) - returns an iterator to the reverse-end
- [**crbegin**](crbegin.md) - returns a const iterator to the reverse-beginning
- [**crend**](crend.md) - returns a const iterator to the reverse-end
- [**items**](items.md) - wrapper to access iterator member functions in range-based for
### Capacity
- [**empty**](empty.md) - checks whether the container is empty
- [**size**](size.md) - returns the number of elements
- [**max_size**](max_size.md) - returns the maximum possible number of elements
### Modifiers
- [**clear**](clear.md) - clears the contents
- [**push_back**](push_back.md) - add a value to an array/object
- [**operator+=**](operator+=.md) - add a value to an array/object
- [**emplace_back**](emplace_back.md) - add a value to an array
- [**emplace**](emplace.md) - add a value to an object if key does not exist
- [**erase**](erase.md) - remove elements
- [**insert**](insert.md) - inserts elements
- [**update**](update.md) - updates a JSON object from another object, overwriting existing keys
- [**swap**](swap.md) - exchanges the values
### Lexicographical comparison operators
- [**operator==**](operator_eq.md) - comparison: equal
- [**operator!=**](operator_ne.md) - comparison: not equal
- [**operator<**](operator_lt.md) - comparison: less than
- [**operator>**](operator_gt.md) - comparison: greater than
- [**operator<=**](operator_le.md) - comparison: less than or equal
- [**operator>=**](operator_ge.md) - comparison: greater than or equal
- [**operator<=>**](operator_spaceship.md) - comparison: 3-way
### Serialization / Dumping
- [**dump**](dump.md) - serialization
### Deserialization / Parsing
- [**parse**](parse.md) (_static_) - deserialize from a compatible input
- [**accept**](accept.md) (_static_) - check if the input is valid JSON
- [**sax_parse**](sax_parse.md) (_static_) - generate SAX events
### JSON Pointer functions
- [**flatten**](flatten.md) - return flattened JSON value
- [**unflatten**](unflatten.md) - unflatten a previously flattened JSON value
### JSON Patch functions
- [**patch**](patch.md) - applies a JSON patch
- [**patch_inplace**](patch_inplace.md) - applies a JSON patch in place
- [**diff**](diff.md) (_static_) - creates a diff as a JSON patch
### JSON Merge Patch functions
- [**merge_patch**](merge_patch.md) - applies a JSON Merge Patch
## Static functions
- [**meta**](meta.md) - returns version information on the library
- [**get_allocator**](get_allocator.md) - returns the allocator associated with the container
### Binary formats
- [**from_bjdata**](from_bjdata.md) (_static_) - create a JSON value from an input in BJData format
- [**from_bson**](from_bson.md) (_static_) - create a JSON value from an input in BSON format
- [**from_cbor**](from_cbor.md) (_static_) - create a JSON value from an input in CBOR format
- [**from_msgpack**](from_msgpack.md) (_static_) - create a JSON value from an input in MessagePack format
- [**from_ubjson**](from_ubjson.md) (_static_) - create a JSON value from an input in UBJSON format
- [**to_bjdata**](to_bjdata.md) (_static_) - create a BJData serialization of a given JSON value
- [**to_bson**](to_bson.md) (_static_) - create a BSON serialization of a given JSON value
- [**to_cbor**](to_cbor.md) (_static_) - create a CBOR serialization of a given JSON value
- [**to_msgpack**](to_msgpack.md) (_static_) - create a MessagePack serialization of a given JSON value
- [**to_ubjson**](to_ubjson.md) (_static_) - create a UBJSON serialization of a given JSON value
## Non-member functions
- [**operator<<(std::ostream&)**](../operator_ltlt.md) - serialize to stream
- [**operator>>(std::istream&)**](../operator_gtgt.md) - deserialize from stream
- [**to_string**](to_string.md) - user-defined `to_string` function for JSON values
## Literals
- [**operator""_json**](../operator_literal_json.md) - user-defined string literal for JSON values
## Helper classes
- [**std::hash<basic_json>**](std_hash.md) - return a hash value for a JSON object
- [**std::swap<basic_json>**](std_swap.md) - exchanges the values of two JSON objects
## Examples
??? example
The example shows how the library is used.
```cpp
--8<-- "examples/README.cpp"
```
Output:
```json
--8<-- "examples/README.output"
```
## See also
- [RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format](https://tools.ietf.org/html/rfc8259)
## Version history
- Added in version 1.0.0.
---
### Third Party Nlohmann Json Docs Mkdocs Docs Api Byte Container With Subtype Index (`third_party/nlohmann_json/docs/mkdocs/docs/api/byte_container_with_subtype/index.md`)
# nlohmann::byte_container_with_subtype
```cpp
template
class byte_container_with_subtype : public BinaryType;
```
This type extends the template parameter `BinaryType` provided to [`basic_json`](../basic_json/index.md) with a subtype
used by BSON and MessagePack. This type exists so that the user does not have to specify a type themselves with a
specific naming scheme in order to override the binary type.
## Template parameters
`BinaryType`
: container to store bytes (`#!cpp std::vector` by default)
## Member types
- **container_type** - the type of the underlying container (`BinaryType`)
- **subtype_type** - the type of the subtype (`#!cpp std::uint64_t`)
## Member functions
- [(constructor)](byte_container_with_subtype.md)
- **operator==** - comparison: equal
- **operator!=** - comparison: not equal
- [**set_subtype**](subtype.md) - sets the binary subtype
- [**subtype**](subtype.md) - return the binary subtype
- [**has_subtype**](has_subtype.md) - return whether the value has a subtype
- [**clear_subtype**](clear_subtype.md) - clears the binary subtype
## Version history
- Added in version 3.8.0.
- Changed type of subtypes to `#!cpp std::uint64_t` in 3.10.0.
---
### Third Party Nlohmann Json Docs Mkdocs Docs Api Json Pointer Index (`third_party/nlohmann_json/docs/mkdocs/docs/api/json_pointer/index.md`)
# nlohmann::json_pointer
```cpp
template
class json_pointer;
```
A JSON pointer defines a string syntax for identifying a specific value within a JSON document. It can be used with
functions [`at`](../basic_json/at.md) and [`operator[]`](../basic_json/operator%5B%5D.md). Furthermore, JSON pointers
are the base for JSON patches.
## Template parameters
`RefStringType`
: the string type used for the reference tokens making up the JSON pointer
!!! warning "Deprecation"
For backwards compatibility `RefStringType` may also be a specialization of [`basic_json`](../basic_json/index.md)
in which case `string_t` will be deduced as [`basic_json::string_t`](../basic_json/string_t.md). This feature is
deprecated and may be removed in a future major version.
## Member types
- [**string_t**](string_t.md) - the string type used for the reference tokens
## Member functions
- [(constructor)](json_pointer.md)
- [**to_string**](to_string.md) - return a string representation of the JSON pointer
- [**operator string_t**](operator_string_t.md) - return a string representation of the JSON pointer
- [**operator==**](operator_eq.md) - compare: equal
- [**operator!=**](operator_ne.md) - compare: not equal
- [**operator/=**](operator_slasheq.md) - append to the end of the JSON pointer
- [**operator/**](operator_slash.md) - create JSON Pointer by appending
- [**parent_pointer**](parent_pointer.md) - returns the parent of this JSON pointer
- [**pop_back**](pop_back.md) - remove last reference token
- [**back**](back.md) - return last reference token
- [**push_back**](push_back.md) - append an unescaped token at the end of the pointer
- [**empty**](empty.md) - return whether pointer points to the root document
## Literals
- [**operator""_json_pointer**](../operator_literal_json_pointer.md) - user-defined string literal for JSON pointers
## See also
- [RFC 6901](https://datatracker.ietf.org/doc/html/rfc6901)
## Version history
- Added in version 2.0.0.
- Changed template parameter from `basic_json` to string type in version 3.11.0.
---
### Third Party Nlohmann Json Docs Mkdocs Docs Api Json Sax Index (`third_party/nlohmann_json/docs/mkdocs/docs/api/json_sax/index.md`)
# nlohmann::json_sax
```cpp
template
struct json_sax;
```
This class describes the SAX interface used by [sax_parse](../basic_json/sax_parse.md). Each function is called in
different situations while the input is parsed. The boolean return value informs the parser whether to continue
processing the input.
## Template parameters
`BasicJsonType`
: a specialization of [`basic_json`](../basic_json/index.md)
## Member types
- [**number_integer_t**](../basic_json/number_integer_t.md) - `BasicJsonType`'s type for numbers (integer)
- [**number_unsigned_t**](../basic_json/number_unsigned_t.md) - `BasicJsonType`'s type for numbers (unsigned)
- [**number_float_t**](../basic_json/number_float_t.md) - `BasicJsonType`'s type for numbers (floating-point)
- [**string_t**](../basic_json/string_t.md) - `BasicJsonType`'s type for strings
- [**binary_t**](../basic_json/binary_t.md) - `BasicJsonType`'s type for binary arrays
## Member functions
- [**binary**](binary.md) (_virtual_) - a binary value was read
- [**boolean**](boolean.md) (_virtual_) - a boolean value was read
- [**end_array**](end_array.md) (_virtual_) - the end of an array was read
- [**end_object**](end_object.md) (_virtual_) - the end of an object was read
- [**key**](key.md) (_virtual_) - an object key was read
- [**null**](null.md) (_virtual_) - a null value was read
- [**number_float**](number_float.md) (_virtual_) - a floating-point number was read
- [**number_integer**](number_integer.md) (_virtual_) - an integer number was read
- [**number_unsigned**](number_unsigned.md) (_virtual_) - an unsigned integer number was read
- [**parse_error**](parse_error.md) (_virtual_) - a parse error occurred
- [**start_array**](start_array.md) (_virtual_) - the beginning of an array was read
- [**start_object**](start_object.md) (_virtual_) - the beginning of an object was read
- [**string**](string.md) (_virtual_) - a string value was read
## Version history
- Added in version 3.2.0.
- Support for binary values (`binary_t`, `binary`) added in version 3.8.0.
---
### Third Party Nlohmann Json Docs Mkdocs Docs Api Macros Index (`third_party/nlohmann_json/docs/mkdocs/docs/api/macros/index.md`)
# Macros
Some aspects of the library can be configured by defining preprocessor macros **before** including the `json.hpp`
header. See also the [macro overview page](../../features/macros.md).
## Runtime assertions
- [**JSON_ASSERT(x)**](json_assert.md) - control behavior of runtime assertions
## Exceptions
- [**JSON_CATCH_USER(exception)** **JSON_THROW_USER(exception)** **JSON_TRY_USER**](json_throw_user.md) - control exceptions
- [**JSON_DIAGNOSTICS**](json_diagnostics.md) - control extended diagnostics
- [**JSON_NOEXCEPTION**](json_noexception.md) - switch off exceptions
## Language support
- [**JSON_HAS_CPP_11** **JSON_HAS_CPP_14** **JSON_HAS_CPP_17** **JSON_HAS_CPP_20**](json_has_cpp_11.md) - set supported C++ standard
- [**JSON_HAS_FILESYSTEM** **JSON_HAS_EXPERIMENTAL_FILESYSTEM**](json_has_filesystem.md) - control `std::filesystem` support
- [**JSON_HAS_RANGES**](json_has_ranges.md) - control `std::ranges` support
- [**JSON_HAS_THREE_WAY_COMPARISON**](json_has_three_way_comparison.md) - control 3-way comparison support
- [**JSON_NO_IO**](json_no_io.md) - switch off functions relying on certain C++ I/O headers
- [**JSON_SKIP_UNSUPPORTED_COMPILER_CHECK**](json_skip_unsupported_compiler_check.md) - do not warn about unsupported compilers
- [**JSON_USE_GLOBAL_UDLS**](json_use_global_udls.md) - place user-defined string literals (UDLs) into the global namespace
## Library version
- [**JSON_SKIP_LIBRARY_VERSION_CHECK**](json_skip_library_version_check.md) - skip library version check
- [**NLOHMANN_JSON_VERSION_MAJOR** **NLOHMANN_JSON_VERSION_MINOR** **NLOHMANN_JSON_VERSION_PATCH**](nlohmann_json_version_major.md)
\- library version information
## Library namespace
- [**NLOHMANN_JSON_NAMESPACE**](nlohmann_json_namespace.md) - full name of the `nlohmann` namespace
- [**NLOHMANN_JSON_NAMESPACE_BEGIN** **NLOHMANN_JSON_NAMESPACE_END**](nlohmann_json_namespace_begin.md) - open and
close the library namespace
- [**NLOHMANN_JSON_NAMESPACE_NO_VERSION**](nlohmann_json_namespace_no_version.md) - disable the version component of
the inline namespace
## Type conversions
- [**JSON_DISABLE_ENUM_SERIALIZATION**](json_disable_enum_serialization.md) - switch off default serialization/deserialization functions for enums
- [**JSON_USE_IMPLICIT_CONVERSIONS**](json_use_implicit_conversions.md) - control implicit conversions
## Comparison behavior
- [**JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON**](json_use_legacy_discarded_value_comparison.md) -
control comparison of discarded values
## Serialization/deserialization macros
- [**NLOHMANN_DEFINE_TYPE_INTRUSIVE(type, member...)** **NLOHMANN_DEFINE_TYPE_INTRUSIVE_WITH_DEFAULT(type, member...)**
**NLOHMANN_DEFINE_TYPE_INTRUSIVE_ONLY_SERIALIZE(type, member...)**][DefInt]
\- serialization/deserialization of types _with_ access to private variables
- [**NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(type, member...)** **NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_WITH_DEFAULT(type, member...)**
**NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_ONLY_SERIALIZE(type, member...)**][DefNonInt]
\- serialization/deserialization of types _without_ access to private variables
- [**NLOHMANN_JSON_SERIALIZE_ENUM(type, ...)**](nlohmann_json_serialize_enum.md) - serialization/deserialization of enum types
[DefInt]: nlohmann_define_type_intrusive.md
[DefNonInt]: nlohmann_define_type_non_intrusive.md
---
### Third Party Nlohmann Json Docs Mkdocs Docs Features Binary Formats Index (`third_party/nlohmann_json/docs/mkdocs/docs/features/binary_formats/index.md`)
# Binary Formats
Though JSON is a ubiquitous data format, it is not a very compact format suitable for data exchange, for instance over
a network. Hence, the library supports
- [BJData](bjdata.md) (Binary JData),
- [BSON](bson.md) (Binary JSON),
- [CBOR](cbor.md) (Concise Binary Object Representation),
- [MessagePack](messagepack.md), and
- [UBJSON](ubjson.md) (Universal Binary JSON)
to efficiently encode JSON values to byte vectors and to decode such vectors.
## Comparison
### Completeness
| Format | Serialization | Deserialization |
|-------------|-----------------------------------------------|----------------------------------------------|
| BJData | complete | complete |
| BSON | incomplete: top-level value must be an object | incomplete, but all JSON types are supported |
| CBOR | complete | incomplete, but all JSON types are supported |
| MessagePack | complete | complete |
| UBJSON | complete | complete |
### Binary values
| Format | Binary values | Binary subtypes |
|-------------|---------------|-----------------|
| BJData | not supported | not supported |
| BSON | supported | supported |
| CBOR | supported | supported |
| MessagePack | supported | supported |
| UBJSON | not supported | not supported |
See [binary values](../binary_values.md) for more information.
### Sizes
| Format | canada.json | twitter.json | citm_catalog.json | jeopardy.json |
|--------------------|-------------|--------------|-------------------|---------------|
| BJData | 53.2 % | 91.1 % | 78.1 % | 96.6 % |
| BJData (size) | 58.6 % | 92.1 % | 86.7 % | 97.4 % |
| BJData (size+tyoe) | 58.6 % | 92.1 % | 86.5 % | 97.4 % |
| BSON | 85.8 % | 95.2 % | 95.8 % | 106.7 % |
| CBOR | 50.5 % | 86.3 % | 68.4 % | 88.0 % |
| MessagePack | 50.5 % | 86.0 % | 68.5 % | 87.9 % |
| UBJSON | 53.2 % | 91.3 % | 78.2 % | 96.6 % |
| UBJSON (size) | 58.6 % | 92.3 % | 86.8 % | 97.4 % |
| UBJSON (size+type) | 55.9 % | 92.3 % | 85.0 % | 95.0 % |
Sizes compared to minified JSON value.
---
### Third Party Nlohmann Json Docs Mkdocs Docs Features Element Access Index (`third_party/nlohmann_json/docs/mkdocs/docs/features/element_access/index.md`)
# Element Access
There are many ways elements in a JSON value can be accessed:
- unchecked access via [`operator[]`](unchecked_access.md)
- checked access via [`at`](checked_access.md)
- access with default value via [`value`](default_value.md)
- iterators
- JSON pointers
---
### Third Party Nlohmann Json Docs Mkdocs Docs Features Parsing Index (`third_party/nlohmann_json/docs/mkdocs/docs/features/parsing/index.md`)
# Parsing
!!! note
This page is under construction.
## Input
## SAX vs. DOM parsing
## Exceptions
See [parsing and exceptions](parse_exceptions.md).
---
### Third Party Nlohmann Json Docs Mkdocs Docs Features Types Index (`third_party/nlohmann_json/docs/mkdocs/docs/features/types/index.md`)
# Types
This page gives an overview how JSON values are stored and how this can be configured.
## Overview
By default, JSON values are stored as follows:
| JSON type | C++ type |
|-----------|-----------------------------------------------|
| object | `std::map` |
| array | `std::vector` |
| null | `std::nullptr_t` |
| string | `std::string` |
| boolean | `bool` |
| number | `std::int64_t`, `std::uint64_t`, and `double` |
Note there are three different types for numbers - when parsing JSON text, the best fitting type is chosen.
## Storage
```plantuml
enum value_t {
null
object
array
string
boolean
number_integer
number_unsigned
number_float
binary
discarded
}
class json_value << (U,orchid) >> {
object_t* object
array_t* array
string_t* string
binary_t* binary
boolean_t boolean
number_integer_t number_integer
number_unsigned_t number_unsigned
number_float_t number_float
}
class basic_json {
-- type and value --
value_t m_type
json_value m_value
-- derived types --
+ typedef object_t
+ typedef array_t
+ typedef binary_t
+ typedef boolean_t
+ typedef number_integer_t
+ typedef number_unsigned_t
+ typedef number_float_t
}
basic_json .. json_value
basic_json .. value_t
```
## Template arguments
The data types to store a JSON value are derived from the template arguments passed to class `basic_json`:
```cpp
template<
template class ObjectType = std::map,
template class ArrayType = std::vector,
class StringType = std::string,
class BooleanType = bool,
class NumberIntegerType = std::int64_t,
class NumberUnsignedType = std::uint64_t,
class NumberFloatType = double,
template class AllocatorType = std::allocator,
template class JSONSerializer = adl_serializer,
class BinaryType = std::vector
>
class basic_json;
```
Type `json` is an alias for `basic_json<>` and uses the default types.
From the template arguments, the following types are derived:
```cpp
using object_comparator_t = std::less<>;
using object_t = ObjectType>>;
using array_t = ArrayType>;
using string_t = StringType;
using boolean_t = BooleanType;
using number_integer_t = NumberIntegerType;
using number_unsigned_t = NumberUnsignedType;
using number_float_t = NumberFloatType;
using binary_t = nlohmann::byte_container_with_subtype;
```
## Objects
[RFC 8259](https://tools.ietf.org/html/rfc8259) describes JSON objects as follows:
> An object is an unordered collection of zero or more name/value pairs, where a name is a string and a value is a string, number, boolean, null, object, or array.
### Default type
With the default values for *ObjectType* (`std::map`), *StringType* (`std::string`), and *AllocatorType* (`std::allocator`), the default value for `object_t` is:
```cpp
std::map<
std::string, // key_type
basic_json, // value_type
std::less<>, // key_compare
std::allocator> // allocator_type
>
```
### Behavior
The choice of `object_t` influences the behavior of the JSON class. With the default type, objects have the following behavior:
- When all names are unique, objects will be interoperable in the sense that all software implementations receiving that object will agree on the name-value mappings.
- When the names within an object are not unique, it is unspecified which one of the values for a given key will be chosen. For instance, `#!json {"key": 2, "key": 1}` could be equal to either `#!json {"key": 1}` or `#!json {"key": 2}`.
- Internally, name/value pairs are stored in lexicographical order of the names. Objects will also be serialized (see `dump`) in this order. For instance, both `#!json {"b": 1, "a": 2}` and `#!json {"a": 2, "b": 1}` will be stored and serialized as `#!json {"a": 2, "b": 1}`.
- When comparing objects, the order of the name/value pairs is irrelevant. This makes objects interoperable in the sense that they will not be affected by these differences. For instance, `#!json {"b": 1, "a": 2}` and `#!json {"a": 2, "b": 1}` will be treated as equal.
### Key order
The order name/value pairs are added to the object is *not* preserved by the library. Therefore, iterating an object may return name/value pairs in a different order than they were originally stored. In fact, keys will be traversed in alphabetical order as `std::map` with `std::less` is used by default. Please note this behavior conforms to [RFC 8259](https://tools.ietf.org/html/rfc8259), because any order implements the specified "unordered" nature of JSON objects.
### Limits
[RFC 8259](https://tools.ietf.org/html/rfc8259) specifies:
> An implementation may set limits on the maximum depth of nesting.
In this class, the object's limit of nesting is not explicitly constrained. However, a maximum depth of nesting may be introduced by the compiler or runtime environment. A theoretical limit can be queried by calling the `max_size` function of a JSON object.
### Storage
Objects are stored as pointers in a `basic_json` type. That is, for any access to object values, a pointer of type `object_t*` must be dereferenced.
## Arrays
[RFC 8259](https://tools.ietf.org/html/rfc8259) describes JSON arrays as follows:
> An array is an ordered sequence of zero or more values.
### Default type
With the default values for *ArrayType* (`std::vector`) and *AllocatorType* (`std::allocator`), the default value for `array_t` is:
```cpp
std::vector<
basic_json, // value_type
std::allocator // allocator_type
>
```
### Limits
[RFC 8259](https://tools.ietf.org/html/rfc8259) specifies:
> An implementation may set limits on the maximum depth of nesting.
In this class, the array's limit of nesting is not explicitly constrained. However, a maximum depth of nesting may be introduced by the compiler or runtime environment. A theoretical limit can be queried by calling the `max_size` function of a JSON array.
### Storage
Arrays are stored as pointers in a `basic_json` type. That is, for any access to array values, a pointer of type `array_t*` must be dereferenced.
## Strings
[RFC 8259](https://tools.ietf.org/html/rfc8259) describes JSON strings as follows:
> A string is a sequence of zero or more Unicode characters.
Unicode values are split by the JSON class into byte-sized characters during deserialization.
### Default type
With the default values for *StringType* (`std::string`), the default value for `string_t` is `#!cpp std::string`.
### Encoding
Strings are stored in UTF-8 encoding. Therefore, functions like `std::string::size()` or `std::string::length()` return the number of **bytes** in the string rather than the number of characters or glyphs.
### String comparison
[RFC 8259](https://tools.ietf.org/html/rfc8259) states:
> Software implementations are typically required to test names of object members for equality. Implementations that transform the textual representation into sequences of Unicode code units and then perform the comparison numerically, code unit by code unit, are interoperable in the sense that implementations will agree in all cases on equality or inequality of two strings. For example, implementations that compare strings with escaped characters unconverted may incorrectly find that `"a\\b"` and `"a\u005Cb"` are not equal.
This implementation is interoperable as it does compare strings code unit by code unit.
### Storage
String values are stored as pointers in a `basic_json` type. That is, for any access to string values, a pointer of type `string_t*` must be dereferenced.
## Booleans
[RFC 8259](https://tools.ietf.org/html/rfc8259) implicitly describes a boolean as a type which differentiates the two literals `true` and `false`.
### Default type
With the default values for *BooleanType* (`#!cpp bool`), the default value for `boolean_t` is `#!cpp bool`.
### Storage
Boolean values are stored directly inside a `basic_json` type.
## Numbers
See the [number handling](number_handling.md) article for a detailed discussion on how numbers are handled by this library.
[RFC 8259](https://tools.ietf.org/html/rfc8259) describes numbers as follows:
> The representation of numbers is similar to that used in most programming languages. A number is represented in base 10 using decimal digits. It contains an integer component that may be prefixed with an optional minus sign, which may be followed by a fraction part and/or an exponent part. Leading zeros are not allowed. (...) Numeric values that cannot be represented in the grammar below (such as Infinity and NaN) are not permitted.
This description includes both integer and floating-point numbers. However, C++ allows more precise storage if it is known whether the number is a signed integer, an unsigned integer or a floating-point number. Therefore, three different types, `number_integer_t`, `number_unsigned_t`, and `number_float_t` are used.
### Default types
With the default values for *NumberIntegerType* (`std::int64_t`), the default value for `number_integer_t` is `std::int64_t`.
With the default values for *NumberUnsignedType* (`std::uint64_t`), the default value for `number_unsigned_t` is `std::uint64_t`.
With the default values for *NumberFloatType* (`#!cpp double`), the default value for `number_float_t` is `#!cpp double`.
### Default behavior
- The restrictions about leading zeros is not enforced in C++. Instead, leading zeros in integer literals lead to an interpretation as octal number. Internally, the value will be stored as decimal number. For instance, the C++ integer literal `#!c 010` will be serialized to `#!c 8`. During deserialization, leading zeros yield an error.
- Not-a-number (NaN) values will be serialized to `#!json null`.
### Limits
[RFC 8259](https://tools.ietf.org/html/rfc8259) specifies:
> An implementation may set limits on the range and precision of numbers.
When the default type is used, the maximal integer number that can be stored is `#!c 9223372036854775807` (`INT64_MAX`) and the minimal integer number that can be stored is `#!c -9223372036854775808` (`INT64_MIN`). Integer numbers that are out of range will yield over/underflow when used in a constructor. During deserialization, too large or small integer numbers will be automatically be stored as `number_unsigned_t` or `number_float_t`.
When the default type is used, the maximal unsigned integer number that can be stored is `#!c 18446744073709551615` (`UINT64_MAX`) and the minimal integer number that can be stored is `#!c 0`. Integer numbers that are out of range will yield over/underflow when used in a constructor. During deserialization, too large or small integer numbers will be automatically be stored as `number_integer_t` or `number_float_t`.
[RFC 8259](https://tools.ietf.org/html/rfc8259) further states:
> Note that when such software is used, numbers that are integers and are in the range $[-2^{53}+1, 2^{53}-1]$ are interoperable in the sense that implementations will agree exactly on their numeric values.
As this range is a subrange of the exactly supported range [`INT64_MIN`, `INT64_MAX`], this class's integer type is interoperable.
[RFC 8259](https://tools.ietf.org/html/rfc8259) states:
> This specification allows implementations to set limits on the range and precision of numbers accepted. Since software that implements IEEE 754-2008 binary64 (double precision) numbers is generally available and widely used, good interoperability can be achieved by implementations that expect no more precision or range than these provide, in the sense that implementations will approximate JSON numbers within the expected precision.
This implementation does exactly follow this approach, as it uses double precision floating-point numbers. Note values smaller than `#!c -1.79769313486232e+308` and values greater than `#!c 1.79769313486232e+308` will be stored as NaN internally and be serialized to `#!json null`.
### Storage
Integer number values, unsigned integer number values, and floating-point number values are stored directly inside a `basic_json` type.
---
### README CN (`docs/README-CN.md`)
![Image][ten-framework-banner]
[![TEN Releases][ten-releases-badge]][ten-releases]
[![Coverage Status][coverage-badge]][coverage]
[![][release-date-badge]][ten-releases]
[![Discussion posts][discussion-badge]][discussions]
[![Commits][commits-badge]][commit-activity]
[![Issues closed][issues-closed-badge]][issues-closed]
[![][contributors-badge]][contributors]
[![GitHub license][license-badge]][license]
[![Ask DeepWiki][deepwiki-badge]][deepwiki]
[![ReadmeX][readmex-badge]][readmex]
[Sitio oficial][official-site]
•
[Documentación][documentation]
•
[Blog][blog]
[![README en inglés][lang-en-badge]][lang-en-readme]
[![Guía en chino simplificado][lang-zh-badge]][lang-zh-readme]
[![README en japonés][lang-jp-badge]][lang-jp-readme]
[![README en coreano][lang-kr-badge]][lang-kr-readme]
[![README en español][lang-es-badge]][lang-es-readme]
[![README en francés][lang-fr-badge]][lang-fr-readme]
[![README en italiano][lang-it-badge]][lang-it-readme]
Tabla de contenido
- [Bienvenido a TEN][welcome-to-ten]
- [Ejemplos de agente][agent-examples-section]
- [Inicio rápido con los ejemplos de agente][quick-start]
- [Entorno local][localhost-section]
- [Codespaces][codespaces-section]
- [Auto-hospedaje de los ejemplos de agente][agent-examples-self-hosting]
- [Implementar con Docker][deploying-with-docker]
- [Implementar en otros servicios en la nube][deploying-with-other-cloud-services]
- [Mantente al día][stay-tuned]
- [Ecosistema TEN][ten-ecosystem-anchor]
- [Preguntas][questions]
- [Cómo contribuir][contributing]
- [Personas contribuidoras][code-contributors]
- [Guía de contribución][contribution-guidelines]
- [Licencia][license-section]
## Bienvenido a TEN
TEN es un marco de código abierto para agentes conversacionales de voz impulsados por IA.
El [ecosistema TEN][ten-ecosystem-anchor] incluye [TEN Framework][ten-framework-link], [Ejemplos de agente][ten-agent-example-link], [VAD][ten-vad-link], [Turn Detection][ten-turn-detection-link] y [Portal][ten-portal-link].
| Canal de la comunidad | Propósito |
| ---------------- | ------- |
| [![Follow on X][follow-on-x-badge]][follow-on-x] | Sigue TEN Framework en X para enterarte de novedades y anuncios |
| [![Discord TEN Community][discord-badge]][discord-invite] | Únete a la comunidad de Discord y conecta con otras personas desarrolladoras |
| [![Follow on LinkedIn][linkedin-badge]][linkedin] | Sigue TEN Framework en LinkedIn para recibir actualizaciones y noticias |
| [![Hugging Face Space][hugging-face-badge]][hugging-face] | Explora nuestros espacios y modelos en la comunidad de Hugging Face |
| [![WeChat][wechat-badge]][wechat-discussion] | Únete al grupo de WeChat para conversar con la comunidad china |
## Ejemplos de agente
![Image][voice-assistant-image]
Asistente de voz multipropósito — Un asistente en tiempo real, de baja latencia y alta calidad que puedes ampliar con [memoria][memory-example], [VAD][voice-assistant-vad-example], [detección de turnos][voice-assistant-turn-detection-example] y otras extensiones.
Consulta el [código de ejemplo][voice-assistant-example] para obtener más detalles.
![divider][divider-light]
![divider][divider-dark]
![Image][doodler-image]
Doodler — Un tablero de garabatos que convierte indicaciones habladas o escritas en bocetos sencillos a mano, con una paleta de crayones y dibujo en tiempo real.
[Código de ejemplo][doodler-example]
![divider][divider-light]
![divider][divider-dark]
![Image][lip-sync-image]
Avatares con sincronización labial — Compatible con múltiples proveedores de avatares. La demo incluye a Kei, un personaje anime con sincronización labial gracias a Live2D, y pronto añadirá avatares realistas de Trulience, HeyGen y Tavus.
Revisa el [código de ejemplo][voice-assistant-live2d-example] para Live2D.
![divider][divider-light]
![divider][divider-dark]
![Image][speech-diarization-image]
Diarización de voz — Detección y etiquetado de hablantes en tiempo real. El juego "Who Likes What" muestra un caso de uso interactivo.
[Código de ejemplo][speechmatics-diarization-example]
![divider][divider-light]
![divider][divider-dark]
![Image][sip-call-image]
Llamada SIP — Extensión SIP que habilita llamadas telefónicas impulsadas por TEN.
[Código de ejemplo][voice-assistant-sip-example]
![divider][divider-light]
![divider][divider-dark]
![Image][transcription-image]
Transcripción — Herramienta de transcripción que convierte audio en texto.
[Código de ejemplo][transcription-example]
![divider][divider-light]
![divider][divider-dark]
![Image][esp32-image]
ESP32-S3 Korvo V3 — Ejecuta el ejemplo de TEN Agent en la placa de desarrollo Espressif ESP32-S3 Korvo V3 para integrar comunicación impulsada por LLM con hardware.
Consulta la [guía de integración][esp32-guide] para conocer más.
[![][back-to-top]][readme-top]
## Inicio rápido con los ejemplos de agente
### Entorno local
#### Paso ⓵ - Requisitos previos
| Categoría | Requisitos |
| --- | --- |
| **Credenciales** | • Agora [App ID][agora-app-certificate] y [App Certificate][agora-app-certificate] (minutos gratuitos mensuales) • Clave de API de [OpenAI][openai-api] (cualquier LLM compatible con el protocolo de OpenAI) • ASR de [Deepgram][deepgram] (créditos gratuitos al registrarte) • TTS de [ElevenLabs][elevenlabs] (créditos gratuitos al registrarte) |
| **Instalación** | • [Docker][docker] / [Docker Compose][docker-compose] • [Node.js (LTS) v18][nodejs] |
| **Requisitos mínimos del sistema** | • CPU ≥ 2 núcleos • RAM ≥ 4 GB |
![divider][divider-light]
![divider][divider-dark]
#### Paso ⓶ - Compila los ejemplos dentro de una VM
##### 1. Clona el repositorio, entra en `ai_agents` y crea un `.env` a partir de `.env.example`
```bash
cd ai_agents
cp ./.env.example ./.env
```
##### 2. Configura el Agora App ID y App Certificate en `.env`
```bash
AGORA_APP_ID=
AGORA_APP_CERTIFICATE=
# Ejecutar el ejemplo predeterminado del asistente de voz
# Deepgram (requerido para STT)
DEEPGRAM_API_KEY=your_deepgram_api_key_here
# OpenAI (requerido para el modelo de lenguaje)
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-4o
# ElevenLabs (requerido para TTS)
ELEVENLABS_TTS_KEY=your_elevenlabs_api_key_here
```
##### 3. Inicia los contenedores de desarrollo del agente
```bash
docker compose up -d
```
##### 4. Entra en el contenedor
```bash
docker exec -it ten_agent_dev bash
```
##### 5. Compila el agente con el ejemplo predeterminado (≈5‑8 min)
En la carpeta `agents/examples` encontrarás más muestras.
Empieza con una de estas opciones:
```bash
# Usa el asistente de voz encadenado
cd agents/examples/voice-assistant
# O usa el asistente voz-a-voz en tiempo real
cd agents/examples/voice-assistant-realtime
```
##### 6. Inicia el servidor web
Ejecuta `task install` antes del primer inicio y después de cambiar las dependencias o el código fuente de Go. Este comando instala las dependencias de TEN, Python y frontend, y compila el servidor API de Go. Los cambios que solo afectan a Python no requieren reinstalación.
```bash
task install
task run
```
##### 7. Accede al agente
Cuando el ejemplo esté en marcha podrás usar estas interfaces:
localhost:49483
localhost:3000
- TMAN Designer:
- UI de ejemplos de agente:
![divider][divider-light]
![divider][divider-dark]
#### Paso ⓷ - Personaliza tu ejemplo de agente
1. Abre [localhost:49483][localhost-49483].
2. Haz clic derecho en las extensiones STT, LLM y TTS.
3. Rellena sus propiedades con las API keys correspondientes.
4. Envía los cambios; el ejemplo actualizado aparecerá en [localhost:3000][localhost-3000].
![divider][divider-light]
![divider][divider-dark]
#### Ejecuta una app de transcripción desde TEN Manager sin Docker (Beta)
TEN también ofrece una app de transcripción que puedes ejecutar en TEN Manager sin usar Docker.
Consulta la [guía de inicio rápido][quick-start-guide-ten-manager] para más detalles.
![divider][divider-light]
![divider][divider-dark]
### Codespaces
GitHub ofrece Codespaces gratuitos para cada repositorio. Puedes ejecutar los ejemplos de agente allí sin usar Docker, y normalmente inicia más rápido que un entorno local con contenedores.
[codespaces-shield]:
[![][codespaces-shield]][codespaces-new]
Consulta [esta guía][codespaces-guide] para obtener más detalles.
[![][back-to-top]][readme-top]
## Auto-hospedaje de los ejemplos de agente
### Implementar con Docker
Cuando personalices tu agente (ya sea con TMAN Designer o editando `property.json`), crea una imagen de Docker lista para producción y despliega tu servicio.
##### Publicar como imagen de Docker
**Nota**: Ejecuta estos comandos fuera de cualquier contenedor Docker.
###### Compilar la imagen
```bash
cd ai_agents
docker build -f agents/examples//Dockerfile -t example-app .
```
###### Ejecutar
```bash
docker run --rm -it --env-file .env -p 3000:3000 example-app
```
![divider][divider-light]
![divider][divider-dark]
### Implementar en otros servicios en la nube
Puedes dividir el despliegue en dos partes si deseas alojar TEN en proveedores como [Vercel][vercel] o [Netlify][netlify].
1. Ejecuta el backend de TEN en cualquier plataforma preparada para contenedores (una VM con Docker, Fly.io, Render, ECS, Cloud Run, etc.). Usa la imagen de ejemplo sin modificar y expón el puerto `8080`.
2. Despliega solo el frontend en Vercel o Netlify. Apunta la raíz del proyecto a `ai_agents/agents/examples//frontend`, ejecuta `pnpm install` (o `bun install`), luego `pnpm build` (o `bun run build`) y conserva el directorio de salida `.next` predeterminado.
3. Configura las variables de entorno en el panel de tu hosting. `AGENT_SERVER_URL` debe apuntar al backend y añade cualquier clave `NEXT_PUBLIC_*` necesaria (por ejemplo, credenciales de Agora visibles para el navegador).
4. Asegúrate de que tu backend acepte solicitudes desde el origen del frontend, ya sea mediante CORS abierto o usando el middleware proxy incluido.
Con esta arquitectura, el backend gestiona los procesos de larga duración y el frontend alojado solo reenvía el tráfico al backend.
[![][back-to-top]][readme-top]
## Mantente al día
Recibe notificaciones instantáneas sobre nuevas versiones y actualizaciones. Tu apoyo nos ayuda a seguir mejorando TEN.
![Image][stay-tuned-image]
[![][back-to-top]][readme-top]
## Ecosistema TEN
| Proyecto | Vista previa |
| ------- | ------- |
| [**️TEN Framework**][ten-framework-link] Marco de código abierto para agentes conversacionales.
![][ten-framework-shield] | ![][ten-framework-banner] |
| [**TEN VAD**][ten-vad-link] Detector de actividad de voz (VAD) en streaming, ligero y de baja latencia.
![][ten-vad-shield] | ![][ten-vad-banner] |
| [**️TEN Turn Detection**][ten-turn-detection-link] Permite diálogo full-duplex mediante detección de turnos.
![][ten-turn-detection-shield] | ![][ten-turn-detection-banner] |
| [**TEN Agent Examples**][ten-agent-example-link] Casos de uso impulsados por TEN.
| ![][ten-agent-example-banner] |
| [**TEN Portal**][ten-portal-link] Sitio oficial con documentación y blog.
![][ten-portal-shield] | ![][ten-portal-banner] |
[![][back-to-top]][readme-top]
## Preguntas
TEN Framework también está disponible en plataformas de preguntas y respuestas impulsadas por IA. Ofrecen soporte multilingüe para todo, desde la configuración básica hasta la implementación avanzada.
| Servicio | Enlace |
| ------- | ---- |
| DeepWiki | [![Ask DeepWiki][deepwiki-badge]][deepwiki] |
| ReadmeX | [![ReadmeX][readmex-badge]][readmex] |
[![][back-to-top]][readme-top]
## Cómo contribuir
¡Toda forma de colaboración de código abierto es bienvenida! Ya sea que corrijas bugs, agregues funciones, mejores la documentación o compartas ideas, tus aportes ayudan a impulsar herramientas de IA personalizadas. Revisa los Issues y Projects de GitHub para encontrar oportunidades y mostrar tus habilidades. ¡Construyamos juntas y juntos algo increíble!
> [!TIP]
>
> **Agradecemos todo tipo de contribuciones** 🙏
>
> Acompáñanos a mejorar TEN. Cada PR, issue o guía suma. Comparte tus proyectos con TEN Agent en redes sociales para inspirar a la comunidad.
>
> Ponte en contacto con la persona mantenedora [@elliotchen200][elliotchen200-x] en 𝕏 o [@cyfyifanchen][cyfyifanchen-github] en GitHub para recibir novedades, debatir ideas y explorar colaboraciones.
![divider][divider-light]
![divider][divider-dark]
### Personas contribuidoras
[![TEN][contributors-image]][contributors]
### Guía de contribución
¡Contribuye cuando quieras! Lee primero la [guía de contribución][contribution-guidelines-doc].
![divider][divider-light]
![divider][divider-dark]
### Licencia
1. Todo TEN Framework, salvo los directorios listados más abajo, se publica bajo la licencia Apache 2.0 con restricciones adicionales. Consulta el archivo [LICENSE][license-file] en la raíz del proyecto.
2. Los componentes dentro de `packages` se liberan bajo Apache License 2.0. Cada paquete contiene su propio archivo `LICENSE` con los detalles.
3. Las dependencias de terceros que usa TEN Framework se describen en la carpeta [third_party][third-party-folder].
![Image][ten-framework-banner]
[![TEN Releases][ten-releases-badge]][ten-releases]
[![Coverage Status][coverage-badge]][coverage]
[![][release-date-badge]][ten-releases]
[![Discussion posts][discussion-badge]][discussions]
[![Commits][commits-badge]][commit-activity]
[![Issues closed][issues-closed-badge]][issues-closed]
[![][contributors-badge]][contributors]
[![GitHub license][license-badge]][license]
[![Ask DeepWiki][deepwiki-badge]][deepwiki]
[![ReadmeX][readmex-badge]][readmex]
[Site officiel][official-site]
•
[Documentation][documentation]
•
[Blog][blog]
[![README en anglais][lang-en-badge]][lang-en-readme]
[![Guide en chinois simplifié][lang-zh-badge]][lang-zh-readme]
[![README en japonais][lang-jp-badge]][lang-jp-readme]
[![README en coréen][lang-kr-badge]][lang-kr-readme]
[![README en espagnol][lang-es-badge]][lang-es-readme]
[![README en français][lang-fr-badge]][lang-fr-readme]
[![README en italien][lang-it-badge]][lang-it-readme]
Table des matières
- [Bienvenue chez TEN][welcome-to-ten]
- [Exemples d’agents][agent-examples-section]
- [Démarrage rapide avec les exemples d’agents][quick-start]
- [En local][localhost-section]
- [Codespaces][codespaces-section]
- [Auto-hébergement des exemples d’agents][agent-examples-self-hosting]
- [Déployer avec Docker][deploying-with-docker]
- [Déployer sur d’autres services cloud][deploying-with-other-cloud-services]
- [Restez informé·e][stay-tuned]
- [Écosystème TEN][ten-ecosystem-anchor]
- [Questions][questions]
- [Contribuer][contributing]
- [Contributrices et contributeurs][code-contributors]
- [Guide de contribution][contribution-guidelines]
- [Licence][license-section]
## Bienvenue chez TEN
TEN est un framework open source pour créer des agents conversationnels vocaux pilotés par l’IA.
L’[écosystème TEN][ten-ecosystem-anchor] comprend [TEN Framework][ten-framework-link], les [Exemples d’agents][ten-agent-example-link], [VAD][ten-vad-link], [Turn Detection][ten-turn-detection-link] et [Portal][ten-portal-link].
| Canal communautaire | Objectif |
| ---------------- | ------- |
| [![Follow on X][follow-on-x-badge]][follow-on-x] | Suivez TEN Framework sur X pour connaître les nouveautés et annonces |
| [![Discord TEN Community][discord-badge]][discord-invite] | Rejoignez notre communauté Discord pour échanger avec d’autres développeurs |
| [![Follow on LinkedIn][linkedin-badge]][linkedin] | Abonnez-vous sur LinkedIn afin de recevoir nos actualités |
| [![Hugging Face Space][hugging-face-badge]][hugging-face] | Découvrez nos espaces et modèles sur Hugging Face |
| [![WeChat][wechat-badge]][wechat-discussion] | Rejoignez le groupe WeChat pour discuter avec la communauté chinoise |
## Exemples d’agents
![Image][voice-assistant-image]
Assistant vocal polyvalent — Assistant temps réel, basse latence et haute qualité, extensible avec des modules de [mémoire][memory-example], de [VAD][voice-assistant-vad-example], de [détection de tours][voice-assistant-turn-detection-example], etc.
Consultez le [code d’exemple][voice-assistant-example] pour en savoir plus.
![divider][divider-light]
![divider][divider-dark]
![Image][doodler-image]
Doodler — Un tableau de gribouillage qui transforme des prompts parlés ou saisis en croquis simples dessinés à la main, avec une palette de crayons et un dessin en temps réel.
[Code d’exemple][doodler-example]
![divider][divider-light]
![divider][divider-dark]
![Image][lip-sync-image]
Avatars avec synchronisation labiale — Compatible avec plusieurs fournisseurs d’avatars. La démo met en scène Kei, un personnage animé avec synchronisation labiale Live2D, et proposera bientôt des avatars réalistes de Trulience, HeyGen et Tavus.
Voir le [code d’exemple Live2D][voice-assistant-live2d-example].
![divider][divider-light]
![divider][divider-dark]
![Image][speech-diarization-image]
Diarisation vocale — Détection et étiquetage des locuteurs en temps réel. Le jeu "Who Likes What" illustre un cas d’usage interactif.
[Code d’exemple][speechmatics-diarization-example]
![divider][divider-light]
![divider][divider-dark]
![Image][sip-call-image]
Appels SIP — Extension SIP qui permet d’effectuer des appels téléphoniques propulsés par TEN.
[Code d’exemple][voice-assistant-sip-example]
![divider][divider-light]
![divider][divider-dark]
![Image][transcription-image]
Transcription — Outil de transcription qui convertit la voix en texte.
[Code d’exemple][transcription-example]
![divider][divider-light]
![divider][divider-dark]
![Image][esp32-image]
ESP32-S3 Korvo V3 — Fait tourner un exemple TEN Agent sur la carte de développement Espressif ESP32-S3 Korvo V3 pour relier communication LLM et matériel.
Voir le [guide d’intégration][esp32-guide] pour plus d’informations.
[![][back-to-top]][readme-top]
## Démarrage rapide avec les exemples d’agents
### En local
#### Étape ⓵ - Prérequis
| Catégorie | Exigences |
| --- | --- |
| **Clés** | • Agora [App ID][agora-app-certificate] et [App Certificate][agora-app-certificate] (minutes gratuites chaque mois) • Clé API de [OpenAI][openai-api] (n’importe quel LLM compatible OpenAI) • ASR [Deepgram][deepgram] (crédits offerts à l’inscription) • TTS [ElevenLabs][elevenlabs] (crédits offerts à l’inscription) |
| **Installation** | • [Docker][docker] / [Docker Compose][docker-compose] • [Node.js (LTS) v18][nodejs] |
| **Configuration minimale** | • CPU ≥ 2 cœurs • RAM ≥ 4 Go |
![divider][divider-light]
![divider][divider-dark]
#### Étape ⓶ - Compiler les exemples dans une VM
##### 1. Clonez le dépôt, placez-vous dans `ai_agents` et créez `.env` à partir de `.env.example`
```bash
cd ai_agents
cp ./.env.example ./.env
```
##### 2. Configurez Agora App ID et App Certificate dans `.env`
```bash
AGORA_APP_ID=
AGORA_APP_CERTIFICATE=
# Exécuter l’exemple d’assistant vocal par défaut
# Deepgram (requis pour la transcription)
DEEPGRAM_API_KEY=your_deepgram_api_key_here
# OpenAI (requis pour le modèle de langage)
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-4o
# ElevenLabs (requis pour la synthèse vocale)
ELEVENLABS_TTS_KEY=your_elevenlabs_api_key_here
```
##### 3. Lancez les conteneurs de développement
```bash
docker compose up -d
```
##### 4. Entrez dans le conteneur
```bash
docker exec -it ten_agent_dev bash
```
##### 5. Compilez l’agent avec l’exemple par défaut (~5-8 min)
D’autres exemples sont disponibles dans `agents/examples`.
Commencez par l’une des options suivantes :
```bash
# Assistant vocal chaîné
cd agents/examples/voice-assistant
# Assistant voix-à-voix temps réel
cd agents/examples/voice-assistant-realtime
```
##### 6. Démarrez le serveur web
Exécutez `task install` avant le premier lancement et après toute modification des dépendances ou du code source Go. Cette commande installe les dépendances TEN, Python et frontend, puis compile le serveur API Go. Les modifications portant uniquement sur Python ne nécessitent pas de nouvelle installation.
```bash
task install
task run
```
##### 7. Accédez à l’agent
Une fois l’exemple démarré, ces interfaces sont disponibles :
localhost:49483
localhost:3000
- TMAN Designer :
- Interface des exemples :
![divider][divider-light]
![divider][divider-dark]
#### Étape ⓷ - Personnaliser votre exemple
1. Ouvrez [localhost:49483][localhost-49483].
2. Cliquez droit sur les extensions STT, LLM et TTS.
3. Renseignez les clés API correspondantes.
4. Validez : la mise à jour apparaît sur [localhost:3000][localhost-3000].
![divider][divider-light]
![divider][divider-dark]
#### Exécuter une application de transcription depuis TEN Manager sans Docker (Beta)
TEN propose aussi une application de transcription que vous pouvez lancer dans TEN Manager sans utiliser Docker.
Consultez le [guide de démarrage rapide][quick-start-guide-ten-manager] pour en savoir plus.
![divider][divider-light]
![divider][divider-dark]
### Codespaces
GitHub fournit des Codespaces gratuits par dépôt. Vous pouvez exécuter les exemples d’agents sans Docker, avec des temps de démarrage souvent plus courts qu’en local.
[codespaces-shield]:
[![][codespaces-shield]][codespaces-new]
Consultez [ce guide][codespaces-guide] pour plus d’informations.
[![][back-to-top]][readme-top]
## Auto-hébergement des exemples d’agents
### Déployer avec Docker
Après avoir personnalisé votre agent (via TMAN Designer ou en modifiant `property.json`), générez une image Docker prête pour la production et déployez votre service.
##### Publier en image Docker
**Remarque** : exécutez ces commandes hors de tout conteneur Docker.
###### Construire l’image
```bash
cd ai_agents
docker build -f agents/examples//Dockerfile -t example-app .
```
###### Exécuter
```bash
docker run --rm -it --env-file .env -p 3000:3000 example-app
```
![divider][divider-light]
![divider][divider-dark]
### Déployer sur d’autres services cloud
Divisez le déploiement en deux parties pour héberger TEN sur des plateformes comme [Vercel][vercel] ou [Netlify][netlify].
1. Exécutez le backend TEN sur une plateforme compatible conteneurs (VM Docker, Fly.io, Render, ECS, Cloud Run, etc.). Utilisez l’image fournie et exposez le port `8080`.
2. Déployez uniquement le frontend sur Vercel ou Netlify. Pointez la racine du projet vers `ai_agents/agents/examples//frontend`, lancez `pnpm install` (ou `bun install`) puis `pnpm build` (ou `bun run build`) et conservez le répertoire `.next` par défaut.
3. Dans le tableau de bord d’hébergement, définissez `AGENT_SERVER_URL` vers l’URL du backend et ajoutez les variables `NEXT_PUBLIC_*` nécessaires (comme les identifiants Agora côté navigateur).
4. Autorisez le frontend à contacter le backend via CORS ouvert ou le middleware proxy intégré.
Ainsi, le backend gère les workers longue durée tandis que le frontend hébergé achemine simplement les requêtes.
[![][back-to-top]][readme-top]
## Restez informé·e
Recevez instantanément les nouvelles versions et les mises à jour. Votre soutien nous aide à faire grandir TEN.
![Image][stay-tuned-image]
[![][back-to-top]][readme-top]
## Écosystème TEN
| Projet | Aperçu |
| ------- | ------- |
| [**️TEN Framework**][ten-framework-link] Framework open source pour agents conversationnels.
![][ten-framework-shield] | ![][ten-framework-banner] |
| [**TEN VAD**][ten-vad-link] Détecteur d’activité vocale (VAD) léger et à faible latence.
![][ten-vad-shield] | ![][ten-vad-banner] |
| [**️TEN Turn Detection**][ten-turn-detection-link] Permet des dialogues full-duplex grâce à la détection de tours.
![][ten-turn-detection-shield] | ![][ten-turn-detection-banner] |
| [**TEN Agent Examples**][ten-agent-example-link] Cas d'usage construits avec TEN.
| ![][ten-agent-example-banner] |
| [**TEN Portal**][ten-portal-link] Site officiel avec documentation et blog.
![][ten-portal-shield] | ![][ten-portal-banner] |
[![][back-to-top]][readme-top]
## Questions
TEN Framework est présent sur des plateformes de questions/réponses alimentées par l’IA. Elles fournissent des réponses multilingues, de la configuration de base aux cas avancés.
| Service | Lien |
| ------- | ---- |
| DeepWiki | [![Ask DeepWiki][deepwiki-badge]][deepwiki] |
| ReadmeX | [![ReadmeX][readmex-badge]][readmex] |
[![][back-to-top]][readme-top]
## Contribuer
Nous accueillons toute forme de collaboration open source ! Corrections de bugs, nouvelles fonctionnalités, documentation ou idées : vos contributions font progresser les outils d’IA personnalisés. Consultez les Issues et Projects GitHub pour trouver des sujets sur lesquels intervenir et montrer votre expertise. Ensemble, faisons grandir TEN !
> [!TIP]
>
> **Toutes les contributions comptent** 🙏
>
> Aidez-nous à améliorer TEN. Du code à la doc, chaque partage est précieux. Publiez vos projets TEN Agent sur les réseaux pour inspirer la communauté.
>
> Contactez un mainteneur, [@elliotchen200][elliotchen200-x] sur 𝕏 ou [@cyfyifanchen][cyfyifanchen-github] sur GitHub, pour suivre les actualités, échanger et collaborer.
![divider][divider-light]
![divider][divider-dark]
### Contributrices et contributeurs
[![TEN][contributors-image]][contributors]
### Guide de contribution
Les contributions sont les bienvenues ! Lisez d’abord le [guide de contribution][contribution-guidelines-doc].
![divider][divider-light]
![divider][divider-dark]
### Licence
1. L’ensemble de TEN Framework (hors dossiers listés ci-dessous) est publié sous licence Apache 2.0 avec restrictions additionnelles. Voir le fichier [LICENSE][license-file] à la racine.
2. Les composants du dossier `packages` sont publiés sous Apache 2.0. Référez-vous au fichier `LICENSE` propre à chaque package.
3. Les bibliothèques tierces utilisées par TEN Framework sont référencées dans le dossier [third_party][third-party-folder].
![Image][ten-framework-banner]
[![TEN Releases][ten-releases-badge]][ten-releases]
[![Coverage Status][coverage-badge]][coverage]
[![][release-date-badge]][ten-releases]
[![Discussion posts][discussion-badge]][discussions]
[![Commits][commits-badge]][commit-activity]
[![Issues closed][issues-closed-badge]][issues-closed]
[![][contributors-badge]][contributors]
[![GitHub license][license-badge]][license]
[![Ask DeepWiki][deepwiki-badge]][deepwiki]
[![ReadmeX][readmex-badge]][readmex]
[Sito ufficiale][official-site]
•
[Documentazione][documentation]
•
[Blog][blog]
[![README in inglese][lang-en-badge]][lang-en-readme]
[![Guida in cinese semplificato][lang-zh-badge]][lang-zh-readme]
[![README in giapponese][lang-jp-badge]][lang-jp-readme]
[![README in coreano][lang-kr-badge]][lang-kr-readme]
[![README in spagnolo][lang-es-badge]][lang-es-readme]
[![README in francese][lang-fr-badge]][lang-fr-readme]
[![README in italiano][lang-it-badge]][lang-it-readme]
Indice
- [Benvenuto in TEN][welcome-to-ten]
- [Esempi di agenti][agent-examples-section]
- [Guida rapida agli esempi di agenti][quick-start]
- [Ambiente locale][localhost-section]
- [Codespaces][codespaces-section]
- [Auto-hosting degli esempi][agent-examples-self-hosting]
- [Distribuire con Docker][deploying-with-docker]
- [Distribuire su altri servizi cloud][deploying-with-other-cloud-services]
- [Rimani aggiornato][stay-tuned]
- [Ecosistema TEN][ten-ecosystem-anchor]
- [Domande][questions]
- [Contribuire][contributing]
- [Contributor del codice][code-contributors]
- [Linee guida per contribuire][contribution-guidelines]
- [Licenza][license-section]
## Benvenuto in TEN
TEN è un framework open source per creare agenti vocali conversazionali.
L’[ecosistema TEN][ten-ecosystem-anchor] comprende [TEN Framework][ten-framework-link], [Esempi di agenti][ten-agent-example-link], [VAD][ten-vad-link], [Turn Detection][ten-turn-detection-link] e [Portal][ten-portal-link].
| Canale della community | Scopo |
| ---------------- | ------- |
| [![Follow on X][follow-on-x-badge]][follow-on-x] | Segui TEN Framework su X per aggiornamenti e annunci |
| [![Discord TEN Community][discord-badge]][discord-invite] | Unisciti alla community Discord per confrontarti con altri sviluppatori |
| [![Follow on LinkedIn][linkedin-badge]][linkedin] | Segui TEN Framework su LinkedIn per non perdere nessuna novità |
| [![Hugging Face Space][hugging-face-badge]][hugging-face] | Esplora i nostri spazi e modelli su Hugging Face |
| [![WeChat][wechat-badge]][wechat-discussion] | Entra nel gruppo WeChat per parlare con la community cinese |
## Esempi di agenti
![Image][voice-assistant-image]
Assistente vocale multiuso — Assistente in tempo reale, a bassa latenza e alta qualità, estendibile con [memoria][memory-example], [VAD][voice-assistant-vad-example], [rilevamento dei turni][voice-assistant-turn-detection-example] e altre estensioni.
Consulta il [codice di esempio][voice-assistant-example] per maggiori dettagli.
![divider][divider-light]
![divider][divider-dark]
![Image][doodler-image]
Doodler — Una lavagna di scarabocchi che trasforma prompt vocali o digitati in semplici schizzi a mano, con una palette di pastelli e disegno in tempo reale.
[Codice di esempio][doodler-example]
![divider][divider-light]
![divider][divider-dark]
![Image][lip-sync-image]
Avatar con lip sync — Supporta diversi provider di avatar. La demo mostra Kei, un personaggio anime con sincronizzazione labiale Live2D, e presto includerà avatar realistici di Trulience, HeyGen e Tavus.
Guarda il [codice di esempio Live2D][voice-assistant-live2d-example].
![divider][divider-light]
![divider][divider-dark]
![Image][speech-diarization-image]
Diarizzazione vocale — Rilevamento e etichettatura dei parlanti in tempo reale. Il gioco "Who Likes What" mostra un caso d’uso interattivo.
[Codice di esempio][speechmatics-diarization-example]
![divider][divider-light]
![divider][divider-dark]
![Image][sip-call-image]
Chiamata SIP — Estensione SIP che abilita chiamate telefoniche gestite da TEN.
[Codice di esempio][voice-assistant-sip-example]
![divider][divider-light]
![divider][divider-dark]
![Image][transcription-image]
Trascrizione — Strumento che trascrive l’audio in testo.
[Codice di esempio][transcription-example]
![divider][divider-light]
![divider][divider-dark]
![Image][esp32-image]
ESP32-S3 Korvo V3 — Esegue un esempio di TEN Agent sulla scheda di sviluppo Espressif ESP32-S3 Korvo V3 per portare comunicazioni basate su LLM sull’hardware.
Consulta la [guida di integrazione][esp32-guide] per ulteriori informazioni.
[![][back-to-top]][readme-top]
## Guida rapida agli esempi di agenti
### Ambiente locale
#### Passaggio ⓵ - Prerequisiti
| Categoria | Requisiti |
| --- | --- |
| **Chiavi** | • Agora [App ID][agora-app-certificate] e [App Certificate][agora-app-certificate] (minuti gratuiti ogni mese) • Chiave API [OpenAI][openai-api] (qualsiasi LLM compatibile con OpenAI) • ASR [Deepgram][deepgram] (crediti gratuiti alla registrazione) • TTS [ElevenLabs][elevenlabs] (crediti gratuiti alla registrazione) |
| **Installazione** | • [Docker][docker] / [Docker Compose][docker-compose] • [Node.js (LTS) v18][nodejs] |
| **Requisiti minimi** | • CPU ≥ 2 core • RAM ≥ 4 GB |
![divider][divider-light]
![divider][divider-dark]
#### Passaggio ⓶ - Compila gli esempi nella VM
##### 1. Clona il repo, entra in `ai_agents` e crea `.env` da `.env.example`
```bash
cd ai_agents
cp ./.env.example ./.env
```
##### 2. Configura Agora App ID e App Certificate in `.env`
```bash
AGORA_APP_ID=
AGORA_APP_CERTIFICATE=
# Esegui l’esempio predefinito dell’assistente vocale
# Deepgram (necessario per STT)
DEEPGRAM_API_KEY=your_deepgram_api_key_here
# OpenAI (necessario per il modello linguistico)
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-4o
# ElevenLabs (necessario per TTS)
ELEVENLABS_TTS_KEY=your_elevenlabs_api_key_here
```
##### 3. Avvia i container di sviluppo
```bash
docker compose up -d
```
##### 4. Entra nel container
```bash
docker exec -it ten_agent_dev bash
```
##### 5. Compila l’agente con l’esempio predefinito (≈5-8 min)
Trovi altre demo nella cartella `agents/examples`.
Inizia da una di queste:
```bash
# Assistente vocale concatenato
cd agents/examples/voice-assistant
# Assistente voce-a-voce in tempo reale
cd agents/examples/voice-assistant-realtime
```
##### 6. Avvia il server web
Esegui `task install` prima del primo avvio e dopo aver modificato le dipendenze o il codice sorgente Go. Questo comando installa le dipendenze TEN, Python e frontend e compila il server API Go. Le modifiche al solo codice Python non richiedono una nuova installazione.
```bash
task install
task run
```
##### 7. Accedi all’agente
Quando l’esempio è in esecuzione puoi usare queste interfacce:
localhost:49483
localhost:3000
- TMAN Designer:
- UI degli esempi:
![divider][divider-light]
![divider][divider-dark]
#### Passaggio ⓷ - Personalizza l’esempio
1. Apri [localhost:49483][localhost-49483].
2. Fai clic con il tasto destro sulle estensioni STT, LLM e TTS.
3. Inserisci le relative API key.
4. Dopo aver salvato, la versione aggiornata sarà visibile su [localhost:3000][localhost-3000].
![divider][divider-light]
![divider][divider-dark]
#### Esegui un’app di trascrizione da TEN Manager senza Docker (Beta)
TEN offre anche un’app di trascrizione che puoi eseguire in TEN Manager senza usare Docker.
Consulta la [guida rapida][quick-start-guide-ten-manager] per ulteriori dettagli.
![divider][divider-light]
![divider][divider-dark]
### Codespaces
GitHub offre Codespaces gratuiti per ogni repository. Puoi eseguire gli esempi senza Docker e, in genere, l’avvio è più rapido rispetto all’ambiente locale basato su container.
[codespaces-shield]:
[![][codespaces-shield]][codespaces-new]
Consulta [questa guida][codespaces-guide] per i dettagli.
[![][back-to-top]][readme-top]
## Auto-hosting degli esempi
### Distribuire con Docker
Dopo aver personalizzato l’agente (con TMAN Designer o modificando `property.json`), crea un’immagine Docker di release e distribuiscila.
##### Pubblicare come immagine Docker
**Nota**: esegui i comandi al di fuori di qualsiasi container.
###### Build dell’immagine
```bash
cd ai_agents
docker build -f agents/examples//Dockerfile -t example-app .
```
###### Esecuzione
```bash
docker run --rm -it --env-file .env -p 3000:3000 example-app
```
![divider][divider-light]
![divider][divider-dark]
### Distribuire su altri servizi cloud
Puoi dividere il deployment in due parti quando ospiti TEN su piattaforme come [Vercel][vercel] o [Netlify][netlify].
1. Esegui il backend TEN su una piattaforma compatibile con container (VM Docker, Fly.io, Render, ECS, Cloud Run, ecc.). Usa l’immagine di esempio senza modificarla ed esponi la porta `8080`.
2. Distribuisci solo il frontend su Vercel o Netlify. Imposta la radice del progetto su `ai_agents/agents/examples//frontend`, esegui `pnpm install` (o `bun install`) e poi `pnpm build` (o `bun run build`), mantenendo la cartella di output `.next` predefinita.
3. Configura le variabili di ambiente dal pannello del provider: `AGENT_SERVER_URL` deve puntare al backend e aggiungi le chiavi `NEXT_PUBLIC_*` necessarie (per esempio le credenziali Agora esposte al browser).
4. Consenti al backend di accettare richieste dall’origine del frontend tramite CORS aperto o usando il middleware proxy incluso.
In questo modo il backend gestisce i processi di lunga durata e il frontend hostato instrada semplicemente il traffico API.
[![][back-to-top]][readme-top]
## Rimani aggiornato
Ricevi notifiche immediate su nuove release e aggiornamenti. Il tuo supporto ci aiuta a migliorare TEN!
![Image][stay-tuned-image]
[![][back-to-top]][readme-top]
## Ecosistema TEN
| Progetto | Anteprima |
| ------- | ------- |
| [**️TEN Framework**][ten-framework-link] Framework open source per agenti conversazionali.
![][ten-framework-shield] | ![][ten-framework-banner] |
| [**TEN VAD**][ten-vad-link] Rilevatore di attività vocale in streaming, leggero e a bassa latenza.
![][ten-vad-shield] | ![][ten-vad-banner] |
| [**️TEN Turn Detection**][ten-turn-detection-link] Abilita dialoghi full-duplex tramite rilevamento dei turni.
![][ten-turn-detection-shield] | ![][ten-turn-detection-banner] |
| [**TEN Agent Examples**][ten-agent-example-link] Casi d'uso costruiti con TEN.
| ![][ten-agent-example-banner] |
| [**TEN Portal**][ten-portal-link] Sito ufficiale con documentazione e blog.
![][ten-portal-shield] | ![][ten-portal-banner] |
[![][back-to-top]][readme-top]
## Domande
TEN Framework è presente anche su piattaforme di Q&A alimentate dall’IA. Offrono risposte multilingue, dalla configurazione di base agli scenari avanzati.
| Servizio | Link |
| ------- | ---- |
| DeepWiki | [![Ask DeepWiki][deepwiki-badge]][deepwiki] |
| ReadmeX | [![ReadmeX][readmex-badge]][readmex] |
[![][back-to-top]][readme-top]
## Contribuire
Accogliamo qualsiasi forma di collaborazione open source! Bugfix, nuove funzionalità, documentazione o idee: ogni contributo aiuta a far crescere strumenti di IA personalizzati. Dai un’occhiata a Issues e Projects su GitHub per trovare attività su cui lavorare e mostrare le tue competenze. Costruiamo TEN insieme!
> [!TIP]
>
> **Ogni contributo è importante** 🙏
>
> Aiutaci a migliorare TEN: dal codice alla documentazione, tutto conta. Condividi i tuoi progetti TEN Agent sui social per ispirare altre persone.
>
> Contatta un maintainer — [@elliotchen200][elliotchen200-x] su 𝕏 o [@cyfyifanchen][cyfyifanchen-github] su GitHub — per aggiornamenti, discussioni e collaborazioni.
![divider][divider-light]
![divider][divider-dark]
### Contributor del codice
[![TEN][contributors-image]][contributors]
### Linee guida per contribuire
Le contribuzioni sono benvenute! Leggi prima le [linee guida per contribuire][contribution-guidelines-doc].
![divider][divider-light]
![divider][divider-dark]
### Licenza
1. L’intero TEN Framework (esclusi i folder elencati qui sotto) è rilasciato sotto Apache License 2.0 con restrizioni aggiuntive. Consulta il file [LICENSE][license-file] nella root del progetto.
2. I componenti nella directory `packages` sono distribuiti sotto Apache License 2.0. Ogni package contiene il proprio file `LICENSE`.
3. Le librerie di terze parti utilizzate da TEN Framework sono elencate nella cartella [third_party][third-party-folder].
목차
- [TEN 소개][welcome-to-ten]
- [에이전트 예시][agent-examples-section]
- [에이전트 예시 빠른 시작][quick-start]
- [로컬 환경][localhost-section]
- [Codespaces][codespaces-section]
- [에이전트 예시 셀프 호스팅][agent-examples-self-hosting]
- [Docker 배포][deploying-with-docker]
- [기타 클라우드 제공업체 배포][deploying-with-other-cloud-services]
- [소식 받기][stay-tuned]
- [TEN 에코시스템][ten-ecosystem-anchor]
- [질문][questions]
- [기여하기][contributing]
- [코드 기여자][code-contributors]
- [기여 가이드][contribution-guidelines]
- [라이선스][license-section]
## TEN 소개
TEN은 음성 대화형 AI 에이전트를 위한 오픈소스 프레임워크입니다.
[TEN 에코시스템][ten-ecosystem-anchor]에는 [TEN Framework][ten-framework-link], [에이전트 예시][ten-agent-example-link], [VAD][ten-vad-link], [Turn Detection][ten-turn-detection-link], [Portal][ten-portal-link] 이 포함됩니다.
| 커뮤니티 채널 | 설명 |
| ---------------- | ------- |
| [![Follow on X][follow-on-x-badge]][follow-on-x] | X에서 TEN Framework 소식을 받아보세요 |
| [![Discord TEN Community][discord-badge]][discord-invite] | Discord 커뮤니티에 참여해 개발자들과 교류하세요 |
| [![Follow on LinkedIn][linkedin-badge]][linkedin] | LinkedIn에서 업데이트와 공지를 확인하세요 |
| [![Hugging Face Space][hugging-face-badge]][hugging-face] | Hugging Face 커뮤니티에서 스페이스와 모델을 살펴보세요 |
| [![WeChat][wechat-badge]][wechat-discussion] | 중국 커뮤니티와 대화할 수 있는 WeChat 그룹 |
## 에이전트 예시
![Image][voice-assistant-image]
멀티 퍼포즈 보이스 어시스턴트 — 저지연·고품질 실시간 어시스턴트로, [메모리][memory-example], [VAD][voice-assistant-vad-example], [턴 감지][voice-assistant-turn-detection-example] 등으로 확장할 수 있습니다.
자세한 내용은 [예시 코드][voice-assistant-example]를 확인하세요.
![divider][divider-light]
![divider][divider-dark]
![Image][doodler-image]
Doodler — 음성 또는 텍스트 프롬프트를 손그림 스케치로 바꾸는 낙서 보드로, 크레용 팔레트와 실시간 드로잉을 제공합니다.
[예시 코드][doodler-example]
![divider][divider-light]
![divider][divider-dark]
![Image][lip-sync-image]
립싱크 아바타 — 다양한 아바타 공급업체를 지원합니다. 데모에서는 Live2D 립싱크를 갖춘 애니메이션 캐릭터 Kei를 소개하며, 곧 Trulience, HeyGen, Tavus의 실사 아바타도 지원할 예정입니다.
[Live2D 예시 코드][voice-assistant-live2d-example]
![divider][divider-light]
![divider][divider-dark]
![Image][speech-diarization-image]
음성 화자 분리 — 실시간으로 화자를 감지하고 라벨링합니다. "Who Likes What" 게임에서 인터랙티브한 사용 사례를 확인할 수 있습니다.
[예시 코드][speechmatics-diarization-example]
![divider][divider-light]
![divider][divider-dark]
![Image][sip-call-image]
SIP 통화 — TEN 기반 전화 통화를 가능하게 하는 SIP 확장입니다.
[예시 코드][voice-assistant-sip-example]
![divider][divider-light]
![divider][divider-dark]
![Image][transcription-image]
전사(Transcription) — 오디오를 텍스트로 변환하는 전사 도구입니다.
[예시 코드][transcription-example]
![divider][divider-light]
![divider][divider-dark]
![Image][esp32-image]
ESP32-S3 Korvo V3 — Espressif ESP32-S3 Korvo V3 개발 보드에서 TEN Agent 예시를 실행해 LLM 기반 커뮤니케이션을 하드웨어와 통합합니다.
[통합 가이드][esp32-guide]를 참고하세요.
[![][back-to-top]][readme-top]
## 에이전트 예시 빠른 시작
### 로컬 환경
#### 단계 ⓵ - 준비 사항
| 구분 | 요구 사항 |
| --- | --- |
| **키/토큰** | • Agora [App ID][agora-app-certificate], [App Certificate][agora-app-certificate] (매월 무료 분 제공) • [OpenAI][openai-api] API 키 (OpenAI 호환 LLM) • [Deepgram][deepgram] ASR (회원가입 시 무료 크레딧) • [ElevenLabs][elevenlabs] TTS (회원가입 시 무료 크레딧) |
| **설치** | • [Docker][docker] / [Docker Compose][docker-compose] • [Node.js (LTS) v18][nodejs] |
| **최소 사양** | • CPU 2코어 이상 • RAM 4GB 이상 |
![divider][divider-light]
![divider][divider-dark]
#### 단계 ⓶ - VM 안에서 예시 빌드
##### 1. 저장소를 클론하고 `ai_agents`로 이동한 뒤 `.env.example`로부터 `.env` 생성
```bash
cd ai_agents
cp ./.env.example ./.env
```
##### 2. `.env`에 Agora App ID / App Certificate 설정
```bash
AGORA_APP_ID=
AGORA_APP_CERTIFICATE=
# 기본 보이스 어시스턴트 예시 실행
# Deepgram (STT 필수)
DEEPGRAM_API_KEY=your_deepgram_api_key_here
# OpenAI (LLM 필수)
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-4o
# ElevenLabs (TTS 필수)
ELEVENLABS_TTS_KEY=your_elevenlabs_api_key_here
```
##### 3. 개발용 컨테이너 시작
```bash
docker compose up -d
```
##### 4. 컨테이너 접속
```bash
docker exec -it ten_agent_dev bash
```
##### 5. 기본 예시로 에이전트 빌드 (약 5~8분)
`agents/examples` 폴더에서 다른 샘플도 확인할 수 있습니다.
다음 중 하나부터 시작하세요:
```bash
# 체이닝형 보이스 어시스턴트
cd agents/examples/voice-assistant
# 실시간 음성-음성 어시스턴트
cd agents/examples/voice-assistant-realtime
```
##### 6. 웹 서버 시작
처음 실행하기 전과 종속성 또는 Go 소스 코드를 변경한 후에는 `task install`을 실행하세요. 이 명령은 TEN, Python, 프런트엔드 종속성을 설치하고 Go API 서버를 빌드합니다. Python 소스 코드만 변경한 경우에는 다시 설치할 필요가 없습니다.
```bash
task install
task run
```
##### 7. 에이전트 접속
예시가 실행되면 아래 인터페이스를 이용할 수 있습니다.
localhost:49483
localhost:3000
- TMAN Designer:
- 에이전트 예시 UI:
![divider][divider-light]
![divider][divider-dark]
#### 단계 ⓷ - 예시 커스터마이징
1. [localhost:49483][localhost-49483]에 접속합니다.
2. STT, LLM, TTS 확장을 우클릭합니다.
3. 속성 창에서 해당 API 키를 입력합니다.
4. 저장 후 [localhost:3000][localhost-3000]에서 변경된 결과를 확인합니다.
![divider][divider-light]
![divider][divider-dark]
#### Docker 없이 TEN Manager에서 전사 앱 실행 (Beta)
TEN은 Docker 없이 TEN Manager에서 바로 실행할 수 있는 전사 앱도 제공합니다.
자세한 내용은 [빠른 시작 가이드][quick-start-guide-ten-manager]를 확인하세요.
![divider][divider-light]
![divider][divider-dark]
### Codespaces
GitHub는 저장소마다 무료 Codespaces를 제공합니다. Docker 없이도 에이전트 예시를 실행할 수 있으며, 일반적으로 로컬 Docker 환경보다 빠르게 시작됩니다.
[codespaces-shield]:
[![][codespaces-shield]][codespaces-new]
자세한 내용은 [이 가이드][codespaces-guide]를 참고하세요.
[![][back-to-top]][readme-top]
## 에이전트 예시 셀프 호스팅
### Docker 배포
TMAN Designer 또는 `property.json` 수정으로 에이전트를 커스터마이징했다면, 서비스용 Docker 릴리스 이미지를 만들어 배포할 수 있습니다.
##### Docker 이미지로 배포
**주의**: 아래 명령은 Docker 컨테이너 밖에서 실행하세요.
###### 이미지 빌드
```bash
cd ai_agents
docker build -f agents/examples//Dockerfile -t example-app .
```
###### 실행
```bash
docker run --rm -it --env-file .env -p 3000:3000 example-app
```
![divider][divider-light]
![divider][divider-dark]
### 기타 클라우드 제공업체 배포
[TEN을 Vercel][vercel]이나 [Netlify][netlify] 같은 플랫폼에 호스팅할 때는 백엔드와 프런트엔드를 분리할 수 있습니다.
1. Docker를 지원하는 플랫폼(도커 가능한 VM, Fly.io, Render, ECS, Cloud Run 등)에서 TEN 백엔드를 실행합니다. 예시 이미지를 그대로 사용하고 포트 `8080`을 개방하세요.
2. 프런트엔드는 Vercel 또는 Netlify에만 배포합니다. 프로젝트 루트를 `ai_agents/agents/examples//frontend`로 지정하고 `pnpm install`(또는 `bun install`) 후 `pnpm build`(또는 `bun run build`)를 실행하며 기본 `.next` 출력 디렉터리를 유지합니다.
3. 호스팅 대시보드에서 `AGENT_SERVER_URL`을 백엔드 주소로 설정하고, 브라우저에서 필요한 `NEXT_PUBLIC_*` 환경 변수를 추가합니다(예: Agora 자격 증명).
4. 백엔드가 프런트엔드 오리진을 허용하도록 CORS를 열거나 기본 제공 프록시 미들웨어를 사용하세요.
이 구조에서는 백엔드가 장기 실행 작업을 처리하고, 호스팅된 프런트엔드는 API 요청을 전달하기만 하면 됩니다.
[![][back-to-top]][readme-top]
## 소식 받기
새 릴리스와 업데이트를 즉시 받아보세요. 여러분의 응원이 TEN을 성장시킵니다!
![Image][stay-tuned-image]
[![][back-to-top]][readme-top]
## TEN 에코시스템
| 프로젝트 | 미리보기 |
| ------- | ------- |
| [**️TEN Framework**][ten-framework-link] 대화형 AI 에이전트를 위한 오픈소스 프레임워크.
![][ten-framework-shield] | ![][ten-framework-banner] |
| [**TEN VAD**][ten-vad-link] 저지연·경량·고성능 스트리밍 음성 활동 감지.
![][ten-vad-shield] | ![][ten-vad-banner] |
| [**️TEN Turn Detection**][ten-turn-detection-link] 풀 듀플렉스 대화를 지원하는 턴 감지.
![][ten-turn-detection-shield] | ![][ten-turn-detection-banner] |
| [**TEN Agent Examples**][ten-agent-example-link] TEN 기반 활용 사례 모음.
| ![][ten-agent-example-banner] |
| [**TEN Portal**][ten-portal-link] 문서와 블로그를 제공하는 공식 사이트.
![][ten-portal-shield] | ![][ten-portal-banner] |
[![][back-to-top]][readme-top]
## 질문
TEN Framework는 AI 기반 Q&A 플랫폼에서도 만나볼 수 있습니다. 멀티랭귀지를 지원하며 기본 설정부터 고급 구현까지 빠르게 답을 찾을 수 있습니다.
| 서비스 | 링크 |
| ------- | ---- |
| DeepWiki | [![Ask DeepWiki][deepwiki-badge]][deepwiki] |
| ReadmeX | [![ReadmeX][readmex-badge]][readmex] |
[![][back-to-top]][readme-top]
## 기여하기
버그 수정, 기능 추가, 문서 개선, 아이디어 공유 등 모든 형태의 오픈소스 협업을 환영합니다. GitHub Issues와 Projects에서 참여할 작업을 찾아 능력을 보여주세요. 함께 멋진 TEN을 만들어봅시다!
> [!TIP]
>
> **어떤 형태의 기여든 환영합니다** 🙏
>
> 코드부터 문서까지, 모든 기여가 TEN을 더 좋게 만듭니다. TEN Agent 프로젝트를 SNS에 공유해 다른 제작자에게 영감을 주세요.
>
> 𝕏의 [@elliotchen200][elliotchen200-x] 또는 GitHub의 [@cyfyifanchen][cyfyifanchen-github]에게 연락하면 업데이트, 토론, 협업 기회를 얻을 수 있습니다.
![divider][divider-light]
![divider][divider-dark]
### 코드 기여자
[![TEN][contributors-image]][contributors]
### 기여 가이드
기여를 환영합니다! 먼저 [기여 가이드][contribution-guidelines-doc]를 읽어 주세요.
![divider][divider-light]
![divider][divider-dark]
### 라이선스
1. 아래에 명시된 디렉터리를 제외한 TEN Framework 전체는 추가 제한이 포함된 Apache License 2.0으로 배포됩니다. 루트 디렉터리의 [LICENSE][license-file]를 참고하세요.
2. `packages` 디렉터리의 구성요소 역시 Apache License 2.0으로 배포되며, 각 패키지 루트의 `LICENSE` 파일에서 세부 내용을 확인할 수 있습니다.
3. TEN Framework에서 사용하는 서드파티 라이브러리는 [third_party][third-party-folder] 폴더에 정리되어 있습니다.
[![][back-to-top]][readme-top]
[back-to-top]: https://img.shields.io/badge/-Back_to_top-gray?style=flat-square
[readme-top]: #readme-top
[welcome-to-ten]: #welcome-to-ten
[agent-examples-section]: #agent-examples
[quick-start]: #quick-start-with-agent-examples
[localhost-section]: #localhost
[codespaces-section]: #codespaces
[agent-examples-self-hosting]: #agent-examples-self-hosting
[deploying-with-docker]: #deploying-with-docker
[deploying-with-other-cloud-services]: #deploying-with-other-cloud-services
[stay-tuned]: #stay-tuned
[ten-ecosystem-anchor]: #ten-ecosystem
[questions]: #questions
[contributing]: #contributing
[code-contributors]: #code-contributors
[contribution-guidelines]: #contribution-guidelines
[license-section]: #license
[discussion-badge]: https://img.shields.io/github/discussions/TEN-framework/ten_framework?labelColor=gray&color=%20%23f79009
[discussions]: https://github.com/TEN-framework/ten-framework/discussions/
[ten-releases-badge]: https://img.shields.io/github/v/release/ten-framework/ten-framework?color=369eff&labelColor=gray&logo=github&style=flat-square
[ten-releases]: https://github.com/TEN-framework/ten-framework/releases
[coverage-badge]: https://coveralls.io/repos/github/TEN-framework/ten-framework/badge.svg?branch=main
[coverage]: https://coveralls.io/github/TEN-framework/ten-framework?branch=main
[release-date-badge]: https://img.shields.io/github/release-date/ten-framework/ten-framework?labelColor=gray&style=flat-square
[commits-badge]: https://img.shields.io/github/commit-activity/m/TEN-framework/ten-framework?labelColor=gray&color=pink
[commit-activity]: https://github.com/TEN-framework/ten-framework/graphs/commit-activity
[issues-closed-badge]: https://img.shields.io/github/issues-search?query=repo%3ATEN-framework%2Ften-framework%20is%3Aclosed&label=issues%20closed&labelColor=gray&color=green
[issues-closed]: https://github.com/TEN-framework/ten-framework/issues
[contributors-badge]: https://img.shields.io/github/contributors/ten-framework/ten-framework?color=c4f042&labelColor=gray&style=flat-square
[contributors]: https://github.com/TEN-framework/ten-framework/graphs/contributors
[license-badge]: https://img.shields.io/badge/License-Apache_2.0_with_certain_conditions-blue.svg?labelColor=%20%23155EEF&color=%20%23528bff
[license]: https://github.com/TEN-framework/ten-framework/blob/main/LICENSE
[deepwiki-badge]: https://deepwiki.com/badge.svg
[deepwiki]: https://deepwiki.com/TEN-framework/TEN-framework
[readmex-badge]: https://raw.githubusercontent.com/CodePhiliaX/resource-trusteeship/main/readmex.svg
[readmex]: https://readmex.com/TEN-framework/ten-framework
[trendshift-badge]: https://trendshift.io/api/badge/repositories/11978
[trendshift]: https://trendshift.io/repositories/11978
[lang-en-badge]: https://img.shields.io/badge/English-lightgrey
[lang-en-readme]: https://github.com/TEN-framework/ten-framework/blob/main/README.md
[lang-zh-badge]: https://img.shields.io/badge/简体中文-lightgrey
[lang-zh-readme]: https://github.com/TEN-framework/ten-framework/blob/main/docs/README-CN.md
[lang-jp-badge]: https://img.shields.io/badge/日本語-lightgrey
[lang-jp-readme]: https://github.com/TEN-framework/ten-framework/blob/main/docs/README-JP.md
[lang-kr-badge]: https://img.shields.io/badge/한국어-lightgrey
[lang-kr-readme]: https://github.com/TEN-framework/ten-framework/blob/main/docs/README-KR.md
[lang-es-badge]: https://img.shields.io/badge/Español-lightgrey
[lang-es-readme]: https://github.com/TEN-framework/ten-framework/blob/main/docs/README-ES.md
[lang-fr-badge]: https://img.shields.io/badge/Français-lightgrey
[lang-fr-readme]: https://github.com/TEN-framework/ten-framework/blob/main/docs/README-FR.md
[lang-it-badge]: https://img.shields.io/badge/Italiano-lightgrey
[lang-it-readme]: https://github.com/TEN-framework/ten-framework/blob/main/docs/README-IT.md
[official-site]: https://theten.ai
[documentation]: https://theten.ai/docs
[blog]: https://theten.ai/blog
[ten-framework]: https://github.com/ten-framework/ten-framework
[agent-examples-repo]: https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples
[ten-vad]: https://github.com/ten-framework/ten-vad
[ten-turn-detection]: https://github.com/ten-framework/ten-turn-detection
[ten-portal]: https://github.com/ten-framework/portal
[follow-on-x-badge]: https://img.shields.io/twitter/follow/TenFramework?logo=X&color=%20%23f5f5f5
[follow-on-x]: https://twitter.com/intent/follow?screen_name=TenFramework
[discord-badge]: https://img.shields.io/badge/Discord-Join%20TEN%20Community-5865F2?style=flat&logo=discord&logoColor=white
[discord-invite]: https://discord.gg/VnPftUzAMJ
[linkedin-badge]: https://custom-icon-badges.demolab.com/badge/LinkedIn-TEN_Framework-0A66C2?logo=linkedin-white&logoColor=fff
[linkedin]: https://www.linkedin.com/company/ten-framework
[hugging-face-badge]: https://img.shields.io/badge/Hugging%20Face-TEN%20Framework-yellow?style=flat&logo=huggingface
[hugging-face]: https://huggingface.co/TEN-framework
[wechat-badge]: https://img.shields.io/badge/TEN_Framework-WeChat_Group-%2307C160?logo=wechat&labelColor=darkgreen&color=gray
[wechat-discussion]: https://github.com/TEN-framework/ten-agent/discussions/170
[voice-assistant-image]: https://github.com/user-attachments/assets/dce3db80-fb48-4e2a-8ac7-33f50bcffa32
[websocket-example]: ../ai_agents/agents/examples/websocket-example
[memory-example]: ../ai_agents/agents/examples/voice-assistant-with-memU
[voice-assistant-vad-example]: ../ai_agents/agents/examples/voice-assistant-with-ten-vad
[voice-assistant-turn-detection-example]: ../ai_agents/agents/examples/voice-assistant-with-turn-detection
[voice-assistant-example]: ../ai_agents/agents/examples/voice-assistant
[divider-light]: https://github.com/user-attachments/assets/aec54c94-ced9-4683-ae58-0a5a7ed803bd#gh-light-mode-only
[divider-dark]: https://github.com/user-attachments/assets/d57fad08-4f49-4a1c-bdfc-f659a5d86150#gh-dark-mode-only
[lip-sync-image]: https://github.com/user-attachments/assets/51ab1504-b67c-49d4-8a7a-5582d9b254da
[voice-assistant-live2d-example]: ../ai_agents/agents/examples/voice-assistant-live2d
[speech-diarization-image]: https://github.com/user-attachments/assets/f94b21b8-9dda-4efc-9274-b028cc01296a
[speechmatics-diarization-example]: ../ai_agents/agents/examples/speaker-diarization
[sip-call-image]: https://github.com/user-attachments/assets/6ed5b04d-945a-4a30-a1cc-f8014b602b38
[voice-assistant-sip-example]: ../ai_agents/agents/examples/voice-assistant-sip-twilio
[transcription-image]: https://github.com/user-attachments/assets/d793bc6c-c8de-4996-bd85-9ce88c69dd8d
[transcription-example]: ../ai_agents/agents/examples/transcription
[doodler-image]: https://github.com/user-attachments/assets/80c4eabd-de96-4971-8956-6b365d4fbd64
[doodler-example]: ../ai_agents/agents/examples/doodler
[esp32-image]: https://github.com/user-attachments/assets/3d60f1ff-0f82-4fe7-b5c2-ac03d284f60c
[esp32-guide]: ../ai_agents/esp32-client
[agora-app-id]: https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project
[agora-app-certificate]: https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project
[openai-api]: https://openai.com/index/openai-api/
[deepgram]: https://deepgram.com/
[elevenlabs]: https://elevenlabs.io/
[docker]: https://www.docker.com/
[docker-compose]: https://docs.docker.com/compose/
[nodejs]: https://nodejs.org/en
[quick-start-guide-ten-manager]: https://theten.ai/docs/ten_framework/getting-started/quick-start
[localhost-49483-image]: https://github.com/user-attachments/assets/191a7c0a-d8e6-48f9-866f-6a70c58f0118
[localhost-3000-image]: https://github.com/user-attachments/assets/13e482b6-d907-4449-a779-9454bb24c0b1
[localhost-49483]: http://localhost:49483
[localhost-3000]: http://localhost:3000
[codespaces-shield]: https://github.com/codespaces/badge.svg
[codespaces-new]: https://codespaces.new/ten-framework/ten-agent
[codespaces-guide]: https://theten.ai/docs/ten_agent_examples/setup_development_env/setting_up_development_inside_codespace
[vercel]: https://vercel.com
[netlify]: https://www.netlify.com
[stay-tuned-image]: https://github.com/user-attachments/assets/72c6cc46-a2a2-484d-82a9-f3079269c815
[ten-framework-shield]: https://img.shields.io/github/stars/ten-framework/ten-framework?color=ffcb47&labelColor=gray&style=flat-square&logo=github
[ten-framework-banner]: https://github.com/user-attachments/assets/a99e13c5-aca4-4b28-bedf-10c89eedcd86
[ten-framework-link]: https://github.com/ten-framework/ten-framework
[ten-vad-link]: https://github.com/ten-framework/ten-vad
[ten-vad-shield]: https://img.shields.io/github/stars/ten-framework/ten-vad?color=ffcb47&labelColor=gray&style=flat-square&logo=github
[ten-vad-banner]: https://github.com/user-attachments/assets/e504135e-67fd-4fa1-b0e4-d495358d8aa5
[ten-turn-detection-link]: https://github.com/ten-framework/ten-turn-detection
[ten-turn-detection-shield]: https://img.shields.io/github/stars/ten-framework/ten-turn-detection?color=ffcb47&labelColor=gray&style=flat-square&logo=github
[ten-turn-detection-banner]: https://github.com/user-attachments/assets/c72d82cc-3667-496c-8bd6-3d194a91c452
[ten-agent-example-link]: https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples
[ten-agent-example-banner]: https://github.com/user-attachments/assets/7f735633-c7f6-4432-b6b4-d2a2977ca588
[ten-portal-link]: https://github.com/ten-framework/portal
[ten-portal-shield]: https://img.shields.io/github/stars/ten-framework/portal?color=ffcb47&labelColor=gray&style=flat-square&logo=github
[ten-portal-banner]: https://github.com/user-attachments/assets/f56c75b9-722c-4156-902d-ae98ce2b3b5e
[elliotchen200-x]: https://x.com/elliotchen200
[cyfyifanchen-github]: https://github.com/cyfyifanchen
[contributors-image]: https://contrib.rocks/image?repo=TEN-framework/ten-framework
[contribution-guidelines-doc]: ./code-of-conduct/contributing.md
[license-file]: ../LICENSE
[third-party-folder]: ../third_party/
---
### Ai L0 Repo Card (`docs/ai/L0_repo_card.md`)
# TEN Framework — Repo Card
> Open-source platform for building real-time multimodal AI agents
## Identity
| Field | Value |
| ------------- | -------------------------------------------------------------------- |
| Repo | `TEN-framework/ten-framework` |
| Description | Open-source platform for building real-time multimodal AI agents |
| Type | `distributed-system` |
| Language | Python (extensions), Go (API server), TypeScript/React (playground) |
| Deploy Target | Docker container (`ten_agent_dev`), Taskfile-based build |
| Owner | TEN Framework team |
| Last Reviewed | 2026-06-29 |
## L1 Index
| File | Purpose | Audience |
| ---------------------------------------- | -------------------------------------------------------- | -------- |
| [01_setup](L1/01_setup.md) | Docker, .env, ports, health checks, restart procedures | Use & Maintain |
| [02_architecture](L1/02_architecture.md) | Extensions, graphs, connections, RTC-first design | Use & Maintain |
| [03_code_map](L1/03_code_map.md) | Directory tree, key files, base classes, 90+ extensions | Use & Maintain |
| [04_conventions](L1/04_conventions.md) | Naming, Pydantic configs, params pattern, formatting | Use & Maintain |
| [05_workflows](L1/05_workflows.md) | Create extension, modify graph, test, restart, deploy | Use & Maintain |
| [06_interfaces](L1/06_interfaces.md) | REST API, connection schemas, base class abstract methods | Use & Maintain |
| [07_gotchas](L1/07_gotchas.md) | Property tuples, signal handlers, zombies, .env timing | Use & Maintain |
| [08_security](L1/08_security.md) | API keys, .env, sensitive logging, git hooks | Use & Maintain |
---
### Api Log.Cn (`docs/api/log.cn.md`)
---
title: 日志
_portal_target: api/log.cn.md
---
TEN framework 允许使用不同语言开发的 extension 在同一进程中运行。这使得需要以统一的日志格式和信息在统一的日志中查看所有这些 extension 的日志，从而简化调试。
为了解决这个问题，`ten_env` 提供了一个日志 API。在 extension 的每个回调中，都可以访问 `ten_env` 的实例。使用此实例，您可以调用日志 API，将使用各种语言开发的 extension 的日志输出到统一的日志输出中。
## 日志级别
TEN framework 支持以下日志级别，按严重程度递增：
- **DEBUG** (1): 调试信息
- **INFO** (2): 一般信息
- **WARN** (3): 警告信息
- **ERROR** (4): 错误信息
## API 接口
### C++ 语言
```cpp
#include "ten_runtime/binding/cpp/ten.h"
class Extension {
public:
void on_cmd(ten::ten_env_t& ten_env, std::unique_ptr cmd) {
// 使用便捷宏
TEN_ENV_LOG_DEBUG(ten_env, "Processing command");
TEN_ENV_LOG_INFO(ten_env, "Command name: " + cmd->get_name());
TEN_ENV_LOG_WARN(ten_env, "Warning message");
TEN_ENV_LOG_ERROR(ten_env, "Error occurred");
// 直接调用方法，支持 category 和 fields 参数
ten_env.log(TEN_LOG_LEVEL_INFO, __func__, __FILE__, __LINE__,
"Direct log call", nullptr, nullptr);
}
};
```
C++ 提供了以下便捷宏：
- `TEN_ENV_LOG_DEBUG(ten_env, msg)`
- `TEN_ENV_LOG_INFO(ten_env, msg)`
- `TEN_ENV_LOG_WARN(ten_env, msg)`
- `TEN_ENV_LOG_ERROR(ten_env, msg)`
### Python 语言
```python
from ten_runtime import Extension, TenEnv
class MyExtension(Extension):
def on_cmd(self, ten_env: TenEnv, cmd):
# 使用不同级别的日志方法
ten_env.log_debug("调试信息")
ten_env.log_info("一般信息")
ten_env.log_warn("警告信息")
ten_env.log_error("错误信息")
# 支持格式化字符串和额外参数
test_value = "example"
ten_env.log_info(f"处理命令，测试值: {test_value}")
# 支持 category 和 fields 参数
ten_env.log_info("带分类的日志", category="my_extension")
```
Python 提供了以下方法：
- `log_debug(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
- `log_info(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
- `log_warn(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
- `log_error(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
### Go 语言
```go
package main
import "ten_runtime/ten"
type MyExtension struct {
ten.DefaultExtension
}
func (ext *MyExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
// 使用不同级别的日志方法
tenEnv.LogDebug("调试信息")
tenEnv.LogInfo("一般信息")
tenEnv.LogWarn("警告信息")
tenEnv.LogError("错误信息")
// 支持格式化
cmdName := cmd.GetName()
tenEnv.LogInfo("处理命令: " + cmdName)
// 使用完整的 Log 方法支持更多参数
category := "my_extension"
tenEnv.Log(ten.LogLevelInfo, "带分类的日志", &category, nil, nil)
}
```
Go 提供了以下方法：
- `LogDebug(msg string) error`
- `LogInfo(msg string) error`
- `LogWarn(msg string) error`
- `LogError(msg string) error`
- `Log(level LogLevel, msg string, category *string, fields *Value, option *TenLogOption) error`
### Node.js/TypeScript 语言
```typescript
import { Extension, TenEnv, Cmd } from "ten-runtime-nodejs";
class MyExtension extends Extension {
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise {
// 使用不同级别的日志方法
tenEnv.logDebug("调试信息");
tenEnv.logInfo("一般信息");
tenEnv.logWarn("警告信息");
tenEnv.logError("错误信息");
// 支持字符串拼接
const cmdName = cmd.getName();
tenEnv.logInfo("处理命令: " + cmdName);
// 支持 category 和 fields 参数
tenEnv.logInfo("带分类的日志", "my_extension");
}
}
```
Node.js 提供了以下方法：
- `logDebug(message: string, category?: string, fields?: Value): TenError | undefined`
- `logInfo(message: string, category?: string, fields?: Value): TenError | undefined`
- `logWarn(message: string, category?: string, fields?: Value): TenError | undefined`
- `logError(message: string, category?: string, fields?: Value): TenError | undefined`
## 使用示例
### 基本使用
```python
# Python 示例
def on_start(self, ten_env: TenEnv):
ten_env.log_info("extension 启动")
ten_env.on_start_done()
def on_cmd(self, ten_env: TenEnv, cmd):
cmd_name = cmd.get_name()
ten_env.log_debug(f"收到命令: {cmd_name}")
# 处理命令逻辑
if cmd_name == "hello":
ten_env.log_info("处理 hello 命令", category="command_handler")
# ... 处理逻辑
else:
ten_env.log_warn(f"未知命令: {cmd_name}")
```
### 在线程中使用
```python
import threading
import time
def log_routine(self, ten_env: TenEnv):
"""在单独线程中记录日志"""
while not self.stop_flag:
self.log_count += 1
ten_env.log_info(f"日志消息 {self.log_count}")
time.sleep(0.05)
def on_start(self, ten_env: TenEnv):
# 启动日志线程
self.log_thread = threading.Thread(
target=self.log_routine, args=(ten_env,)
)
self.log_thread.start()
ten_env.on_start_done()
```
### 错误处理
```python
def on_cmd(self, ten_env: TenEnv, cmd):
try:
# 业务逻辑
result = self.process_command(cmd)
ten_env.log_info("命令处理成功")
except ValueError as e:
ten_env.log_error(f"参数错误: {str(e)}", category="error_handling")
except Exception as e:
ten_env.log_error(f"严重错误: {str(e)}", category="error_handling")
raise
```
## 日志格式和输出
TEN framework 的日志系统会自动包含以下信息：
- **时间戳**: 日志记录的时间
- **日志级别**: DEBUG、INFO、WARN、ERROR
- **函数名**: 调用日志的函数名（如果可用）
- **文件名**: 调用日志的源文件名（如果可用）
- **行号**: 调用日志的代码行号（如果可用）
- **分类**: 可选的日志分类信息
- **消息内容**: 用户提供的日志消息
输出格式类似：
```bash
2025-01-01 12:00:00.123 [INFO] [function_name@file.py:123] [category] 用户日志消息
```
## 高级日志配置
TEN framework 支持通过 `property.json` 文件配置高级日志功能。这允许您自定义日志的输出格式、目标和过滤规则。
### 配置结构
在 `property.json` 文件中，您可以在 `ten.log` 部分定义高级日志配置：
```json
{
"ten": {
"log": {
"handlers": [
{
"matchers": [
{
"level": "debug",
"category": "my_extension"
}
],
"formatter": {
"type": "plain",
"colored": true
},
"emitter": {
"type": "console",
"config": {
"stream": "stdout"
}
}
}
]
}
}
}
```
### 配置组件说明
#### 1. 日志处理器 (Handlers)
每个处理器包含三个主要组件：
- **matchers**: 定义哪些日志消息会被此处理器处理
- **formatter**: 定义日志的输出格式
- **emitter**: 定义日志的输出目标
#### 2. 匹配器 (Matchers)
匹配器用于过滤日志消息：
```json
{
"level": "info", // 日志级别: "off", "debug", "info", "warn", "error"
"category": "auth" // 可选：日志分类（字符串）
}
```
支持的日志级别：
- `"off"`: 关闭日志
- `"debug"`: 调试级别
- `"info"`: 信息级别
- `"warn"`: 警告级别
- `"error"`: 错误级别
#### 3. 格式器 (Formatter)
控制日志的输出格式：
```json
{
"type": "plain", // 格式类型: "plain" 或 "json"
"colored": true // 可选：是否启用颜色输出（仅对 plain 格式有效）
}
```
- **plain**: 人类可读的纯文本格式
- **json**: JSON 结构化格式，便于日志分析工具处理
#### 4. 发射器 (Emitter)
定义日志的输出目标：
##### 控制台输出
```json
{
"type": "console",
"config": {
"stream": "stdout", // "stdout" 或 "stderr"
"encryption": { // 可选：加密配置
"enabled": true,
"algorithm": "AES-CTR",
"params": {
"key": "your-encryption-key",
"nonce": "your-nonce"
}
}
}
}
```
##### 文件输出
```json
{
"type": "file",
"config": {
"path": "/path/to/logfile.log", // 日志文件路径
"encryption": { // 可选：加密配置
"enabled": true,
"algorithm": "AES-CTR",
"params": {
"key": "your-encryption-key",
"nonce": "your-nonce"
}
}
}
}
```
##### OTLP 输出
OTLP (OpenTelemetry Protocol) 输出允许您将日志发送到支持 OpenTelemetry 的后端系统，如 Jaeger、Prometheus、Grafana 等。
```json
{
"type": "otlp",
"config": {
"endpoint": "http://localhost:4317", // OTLP 接收端点地址
"protocol": "grpc", // 可选：传输协议，"grpc"（默认）或 "http"
"headers": { // 可选：自定义 HTTP 头
"Authorization": "Bearer token",
"X-Custom-Header": "value"
},
"service_name": "my-service" // 可选：服务名称，用于标识日志来源
}
}
```
配置参数说明：
- **endpoint**: OTLP 接收端点的 URL（必需）
- **protocol**: 传输协议，支持 `"grpc"` 或 `"http"`，默认为 `"grpc"`
- **headers**: 自定义 HTTP 请求头，用于认证或其他元数据传递（可选）
- **service_name**: 服务名称，用于在日志聚合系统中标识不同的服务（可选）
### 配置示例
#### 示例 1: 多级别日志分离
```json
{
"ten": {
"log": {
"handlers": [
{
"matchers": [
{
"level": "error"
}
],
"formatter": {
"type": "json"
},
"emitter": {
"type": "file",
"config": {
"path": "/var/log/app/errors.log"
}
}
},
{
"matchers": [
{
"level": "info"
}
],
"formatter": {
"type": "plain",
"colored": true
},
"emitter": {
"type": "console",
"config": {
"stream": "stdout"
}
}
}
]
}
}
}
```
#### 示例 2: 按分类分离日志
```json
{
"ten": {
"log": {
"handlers": [
{
"matchers": [
{
"level": "debug",
"category": "database"
}
],
"formatter": {
"type": "json"
},
"emitter": {
"type": "file",
"config": {
"path": "/var/log/app/database.log"
}
}
},
{
"matchers": [
{
"level": "info",
"category": "api"
}
],
"formatter": {
"type": "plain"
},
"emitter": {
"type": "console",
"config": {
"stream": "stdout"
}
}
}
]
}
}
}
```
#### 示例 3: 加密日志输出
```json
{
"ten": {
"log": {
"handlers": [
{
"matchers": [
{
"level": "info"
}
],
"formatter": {
"type": "json"
},
"emitter": {
"type": "file",
"config": {
"path": "/secure/logs/app.log",
"encryption": {
"enabled": true,
"algorithm": "AES-CTR",
"params": {
"key": "your-32-character-encryption-key",
"nonce": "your-16-character-nonce"
}
}
}
}
}
]
}
}
}
```
#### 示例 4: OTLP 日志输出到 OpenTelemetry 收集器
```json
{
"ten": {
"log": {
"handlers": [
{
"matchers": [
{
"level": "info"
}
],
"formatter": {
"type": "json"
},
"emitter": {
"type": "otlp",
"config": {
"endpoint": "http://localhost:4317",
"protocol": "grpc",
"service_name": "ten-framework-app"
}
}
},
{
"matchers": [
{
"level": "error"
}
],
"formatter": {
"type": "json"
},
"emitter": {
"type": "otlp",
"config": {
"endpoint": "https://api.monitoring.example.com/v1/logs",
"protocol": "http",
"headers": {
"Authorization": "Bearer your-api-token",
"X-Environment": "production"
},
"service_name": "ten-framework-app"
}
}
}
]
}
}
}
```
### 配置注意事项
1. **多个处理器**: 可以定义多个处理器来实现复杂的日志路由策略
2. **匹配优先级**: 按照配置文件中的顺序匹配处理器
3. **加密安全**: 加密密钥和 nonce 应该安全存储，不要硬编码在配置文件中
4. **文件权限**: 确保日志文件路径具有适当的写入权限
5. **性能考虑**: JSON 格式可能会比 plain 格式产生更大的日志文件
6. **OTLP 端点**:
- 使用 gRPC 协议时，默认端口通常为 4317
- 使用 HTTP 协议时，默认端口通常为 4318
- 确保 OTLP 收集器端点可访问且配置正确
7. **认证**: 使用 OTLP 时，如果后端需要认证，请在 headers 中添加相应的认证信息
## 最佳实践
1. **选择合适的日志级别**：
- 使用 `DEBUG` 记录详细的调试信息
- 使用 `INFO` 记录正常的业务流程
- 使用 `WARN` 记录潜在问题
- 使用 `ERROR` 记录错误信息
2. **提供有意义的日志消息**：
- 包含上下文信息（如命令名称、参数值等）
- 使用描述性的消息而不是简单的标识符
- 合理使用 category 参数对日志进行分类
3. **避免过度日志记录**：
- 不要在高频循环中使用过多的日志
- 在生产环境中适当调整日志级别
- 使用高级日志配置的匹配器来过滤不必要的日志
4. **线程安全**：
- TEN framework 的日志 API 是线程安全的
- 可以在任何线程中安全调用日志方法
5. **高级配置建议**：
- 为不同的模块使用不同的 category 便于日志分析
- 在生产环境中考虑使用文件输出以便于日志持久化
- 对敏感日志使用加密功能
- 定期轮转日志文件以控制磁盘使用
- 使用 JSON 格式便于与日志分析工具集成
- 考虑使用 OTLP 输出将日志集成到现代可观测性平台（如 Grafana、Jaeger、Datadog 等）
- OTLP 输出与 OpenTelemetry 生态系统无缝集成，便于实现分布式追踪和日志关联
整体效果如下图所示：
---
### Api Log (`docs/api/log.md`)
---
title: Log
_portal_target: api/log.md
---
The TEN framework allows extensions developed in different languages to run within the same process. This creates a need to view logs from all these extensions in a unified log with a consistent format and information, making debugging easier.
To address this, the `ten_env` provides a logging API. Within each callback of an extension, an instance of `ten_env` can be accessed. Using this instance, you can call the logging API to output logs from extensions developed in various languages to a unified log output.
## Log Levels
The TEN framework supports the following log levels in increasing order of severity:
- **DEBUG** (1): Debug information
- **INFO** (2): General information
- **WARN** (3): Warning information
- **ERROR** (4): Error information
## API Interface
### C++ Language
```cpp
#include "ten_runtime/binding/cpp/ten.h"
class Extension {
public:
void on_cmd(ten::ten_env_t& ten_env, std::unique_ptr cmd) {
// Using convenience macros
TEN_ENV_LOG_DEBUG(ten_env, "Processing command");
TEN_ENV_LOG_INFO(ten_env, "Command name: " + cmd->get_name());
TEN_ENV_LOG_WARN(ten_env, "Warning message");
TEN_ENV_LOG_ERROR(ten_env, "Error occurred");
// Direct method call with category and fields support
ten_env.log(TEN_LOG_LEVEL_INFO, __func__, __FILE__, __LINE__,
"Direct log call", nullptr, nullptr);
}
};
```
C++ provides the following convenience macros:
- `TEN_ENV_LOG_DEBUG(ten_env, msg)`
- `TEN_ENV_LOG_INFO(ten_env, msg)`
- `TEN_ENV_LOG_WARN(ten_env, msg)`
- `TEN_ENV_LOG_ERROR(ten_env, msg)`
### Python Language
```python
from ten_runtime import Extension, TenEnv
class MyExtension(Extension):
def on_cmd(self, ten_env: TenEnv, cmd):
# Using different log level methods
ten_env.log_debug("Debug information")
ten_env.log_info("General information")
ten_env.log_warn("Warning information")
ten_env.log_error("Error information")
# Supports formatted strings and additional parameters
test_value = "example"
ten_env.log_info(f"Processing command, test value: {test_value}")
# Supports category and fields parameters
ten_env.log_info("Log with category", category="my_extension")
```
Python provides the following methods:
- `log_debug(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
- `log_info(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
- `log_warn(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
- `log_error(msg: str, category: str = None, fields: Value = None) -> Optional[TenError]`
### Go Language
```go
package main
import "ten_runtime/ten"
type MyExtension struct {
ten.DefaultExtension
}
func (ext *MyExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
// Using different log level methods
tenEnv.LogDebug("Debug information")
tenEnv.LogInfo("General information")
tenEnv.LogWarn("Warning information")
tenEnv.LogError("Error information")
// Supports formatting
cmdName := cmd.GetName()
tenEnv.LogInfo("Processing command: " + cmdName)
// Using the full Log method for additional parameters
category := "my_extension"
tenEnv.Log(ten.LogLevelInfo, "Log with category", &category, nil, nil)
}
```
Go provides the following methods:
- `LogDebug(msg string) error`
- `LogInfo(msg string) error`
- `LogWarn(msg string) error`
- `LogError(msg string) error`
- `Log(level LogLevel, msg string, category *string, fields *Value, option *TenLogOption) error`
### Node.js/TypeScript Language
```typescript
import { Extension, TenEnv, Cmd } from "ten-runtime-nodejs";
class MyExtension extends Extension {
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise {
// Using different log level methods
tenEnv.logDebug("Debug information");
tenEnv.logInfo("General information");
tenEnv.logWarn("Warning information");
tenEnv.logError("Error information");
// Supports string concatenation
const cmdName = cmd.getName();
tenEnv.logInfo("Processing command: " + cmdName);
// Supports category and fields parameters
tenEnv.logInfo("Log with category", "my_extension");
}
}
```
Node.js provides the following methods:
- `logDebug(message: string, category?: string, fields?: Value): TenError | undefined`
- `logInfo(message: string, category?: string, fields?: Value): TenError | undefined`
- `logWarn(message: string, category?: string, fields?: Value): TenError | undefined`
- `logError(message: string, category?: string, fields?: Value): TenError | undefined`
## Usage Examples
### Basic Usage
```python
# Python example
def on_start(self, ten_env: TenEnv):
ten_env.log_info("Extension starting")
ten_env.on_start_done()
def on_cmd(self, ten_env: TenEnv, cmd):
cmd_name = cmd.get_name()
ten_env.log_debug(f"Received command: {cmd_name}")
# Command processing logic
if cmd_name == "hello":
ten_env.log_info("Processing hello command", category="command_handler")
# ... processing logic
else:
ten_env.log_warn(f"Unknown command: {cmd_name}")
```
### Using in Threads
```python
import threading
import time
def log_routine(self, ten_env: TenEnv):
"""Log messages in a separate thread"""
while not self.stop_flag:
self.log_count += 1
ten_env.log_info(f"Log message {self.log_count}")
time.sleep(0.05)
def on_start(self, ten_env: TenEnv):
# Start logging thread
self.log_thread = threading.Thread(
target=self.log_routine, args=(ten_env,)
)
self.log_thread.start()
ten_env.on_start_done()
```
### Error Handling
```python
def on_cmd(self, ten_env: TenEnv, cmd):
try:
# Business logic
result = self.process_command(cmd)
ten_env.log_info("Command processed successfully")
except ValueError as e:
ten_env.log_error(f"Parameter error: {str(e)}", category="error_handling")
except Exception as e:
ten_env.log_error(f"Fatal error: {str(e)}", category="error_handling")
raise
```
## Log Format and Output
The TEN framework's logging system automatically includes the following information:
- **Timestamp**: Time when the log was recorded
- **Log Level**: DEBUG, INFO, WARN, ERROR
- **Function Name**: Name of the function that called the log (if available)
- **File Name**: Source file name where the log was called (if available)
- **Line Number**: Code line number where the log was called (if available)
- **Category**: Optional log category information
- **Message Content**: User-provided log message
The output format is similar to:
```bash
2025-01-01 12:00:00.123 [INFO] [function_name@file.py:123] [category] User log message
```
## Advanced Log Configuration
The TEN framework supports advanced logging functionality through the `property.json` file configuration. This allows you to customize the log output format, destination, and filtering rules.
### Configuration Structure
In the `property.json` file, you can define advanced log configuration in the `ten.log` section:
```json
{
"ten": {
"log": {
"handlers": [
{
"matchers": [
{
"level": "debug",
"category": "my_extension"
}
],
"formatter": {
"type": "plain",
"colored": true
},
"emitter": {
"type": "console",
"config": {
"stream": "stdout"
}
}
}
]
}
}
}
```
### Configuration Components
#### 1. Log Handlers
Each handler contains three main components:
- **matchers**: Define which log messages will be processed by this handler
- **formatter**: Define the log output format
- **emitter**: Define the log output destination
#### 2. Matchers
Matchers are used to filter log messages:
```json
{
"level": "info", // Log level: "off", "debug", "info", "warn", "error"
"category": "auth" // Optional: log category (string)
}
```
Supported log levels:
- `"off"`: Disable logging
- `"debug"`: Debug level
- `"info"`: Info level
- `"warn"`: Warning level
- `"error"`: Error level
#### 3. Formatter
Controls the log output format:
```json
{
"type": "plain", // Format type: "plain" or "json"
"colored": true // Optional: enable colored output (only for plain format)
}
```
- **plain**: Human-readable plain text format
- **json**: JSON structured format, convenient for log analysis tools
#### 4. Emitter
Defines the log output destination:
##### Console Output
```json
{
"type": "console",
"config": {
"stream": "stdout", // "stdout" or "stderr"
"encryption": { // Optional: encryption configuration
"enabled": true,
"algorithm": "AES-CTR",
"params": {
"key": "your-encryption-key",
"nonce": "your-nonce"
}
}
}
}
```
##### File Output
```json
{
"type": "file",
"config": {
"path": "/path/to/logfile.log", // Log file path
"encryption": { // Optional: encryption configuration
"enabled": true,
"algorithm": "AES-CTR",
"params": {
"key": "your-encryption-key",
"nonce": "your-nonce"
}
}
}
}
```
##### OTLP Output
OTLP (OpenTelemetry Protocol) output allows you to send logs to OpenTelemetry-compatible backend systems, such as Jaeger, Prometheus, Grafana, and more.
```json
{
"type": "otlp",
"config": {
"endpoint": "http://localhost:4317", // OTLP receiver endpoint URL
"protocol": "grpc", // Optional: transport protocol, "grpc" (default) or "http"
"headers": { // Optional: custom HTTP headers
"Authorization": "Bearer token",
"X-Custom-Header": "value"
},
"service_name": "my-service" // Optional: service name for identifying log source
}
}
```
Configuration parameters:
- **endpoint**: OTLP receiver endpoint URL (required)
- **protocol**: Transport protocol, supports `"grpc"` or `"http"`, defaults to `"grpc"`
- **headers**: Custom HTTP request headers for authentication or metadata (optional)
- **service_name**: Service name for identifying different services in log aggregation systems (optional)
### Configuration Examples
#### Example 1: Multi-level Log Separation
```json
{
"ten": {
"log": {
"handlers": [
{
"matchers": [
{
"level": "error"
}
],
"formatter": {
"type": "json"
},
"emitter": {
"type": "file",
"config": {
"path": "/var/log/app/errors.log"
}
}
},
{
"matchers": [
{
"level": "info"
}
],
"formatter": {
"type": "plain",
"colored": true
},
"emitter": {
"type": "console",
"config": {
"stream": "stdout"
}
}
}
]
}
}
}
```
#### Example 2: Category-based Log Separation
```json
{
"ten": {
"log": {
"handlers": [
{
"matchers": [
{
"level": "debug",
"category": "database"
}
],
"formatter": {
"type": "json"
},
"emitter": {
"type": "file",
"config": {
"path": "/var/log/app/database.log"
}
}
},
{
"matchers": [
{
"level": "info",
"category": "api"
}
],
"formatter": {
"type": "plain"
},
"emitter": {
"type": "console",
"config": {
"stream": "stdout"
}
}
}
]
}
}
}
```
#### Example 3: Encrypted Log Output
```json
{
"ten": {
"log": {
"handlers": [
{
"matchers": [
{
"level": "info"
}
],
"formatter": {
"type": "json"
},
"emitter": {
"type": "file",
"config": {
"path": "/secure/logs/app.log",
"encryption": {
"enabled": true,
"algorithm": "AES-CTR",
"params": {
"key": "your-32-character-encryption-key",
"nonce": "your-16-character-nonce"
}
}
}
}
}
]
}
}
}
```
#### Example 4: OTLP Log Output to OpenTelemetry Collector
```json
{
"ten": {
"log": {
"handlers": [
{
"matchers": [
{
"level": "info"
}
],
"formatter": {
"type": "json"
},
"emitter": {
"type": "otlp",
"config": {
"endpoint": "http://localhost:4317",
"protocol": "grpc",
"service_name": "ten-framework-app"
}
}
},
{
"matchers": [
{
"level": "error"
}
],
"formatter": {
"type": "json"
},
"emitter": {
"type": "otlp",
"config": {
"endpoint": "https://api.monitoring.example.com/v1/logs",
"protocol": "http",
"headers": {
"Authorization": "Bearer your-api-token",
"X-Environment": "production"
},
"service_name": "ten-framework-app"
}
}
}
]
}
}
}
```
### Configuration Notes
1. **Multiple Handlers**: You can define multiple handlers to implement complex log routing strategies
2. **Matching Priority**: Handlers are matched in the order they appear in the configuration file
3. **Encryption Security**: Encryption keys and nonces should be stored securely, not hardcoded in configuration files
4. **File Permissions**: Ensure log file paths have appropriate write permissions
5. **Performance Considerations**: JSON format may produce larger log files than plain format
6. **OTLP Endpoints**:
- When using gRPC protocol, the default port is typically 4317
- When using HTTP protocol, the default port is typically 4318
- Ensure the OTLP collector endpoint is accessible and properly configured
7. **Authentication**: When using OTLP, if the backend requires authentication, add appropriate credentials in the headers
## Best Practices
1. **Choose appropriate log levels**:
- Use `DEBUG` for detailed debugging information
- Use `INFO` for normal business processes
- Use `WARN` for potential issues
- Use `ERROR` for error information
2. **Provide meaningful log messages**:
- Include contextual information (such as command names, parameter values, etc.)
- Use descriptive messages rather than simple identifiers
- Use the category parameter appropriately to classify logs
3. **Avoid excessive logging**:
- Don't use too many logs in high-frequency loops
- Appropriately adjust log levels in production environments
- Use advanced log configuration matchers to filter unnecessary logs
4. **Thread safety**:
- The TEN framework's logging API is thread-safe
- Log methods can be safely called from any thread
5. **Advanced configuration recommendations**:
- Use different categories for different modules to facilitate log analysis
- Consider using file output in production environments for log persistence
- Use encryption functionality for sensitive logs
- Regularly rotate log files to control disk usage
- Use JSON format for easier integration with log analysis tools
- Consider using OTLP output to integrate logs into modern observability platforms (e.g., Grafana, Jaeger, Datadog)
- OTLP output seamlessly integrates with the OpenTelemetry ecosystem, enabling distributed tracing and log correlation
The overall effect is shown in the image below:
---
### Code Of Conduct Contributing (`docs/code-of-conduct/contributing.md`)
# Contributing to TEN Framework
Welcome to the TEN Framework community! We're excited to have you contribute to building the future of real-time conversational AI agents. This guide will help you get started with contributing to the project.
## 🌟 Ways to Contribute
We welcome all forms of contributions:
- **Code contributions**: Bug fixes, new features, performance improvements
- **Documentation**: Improving docs, tutorials, examples
- **Bug reports**: Identifying and reporting issues
- **Feature requests**: Suggesting new functionality
- **Community support**: Helping others in discussions and issues
- **Extensions**: Creating new extensions for the ecosystem
## 📋 Before You Start
### Required Reading
1. Read our [Code of Conduct](./CLA.md) - all contributors must follow our community guidelines
2. Sign the [Contributor License Agreement (CLA)](../../CLA.md) - required for all contributions
3. Review the project [README](../../README.md) to understand the architecture
### Prerequisites
Make sure you have the required tools installed:
- **Docker / Docker Compose** (for development environment)
- **Node.js v20+** (for frontend development)
- **Python 3.8+** (for AI agent development)
- **Go** (for backend server development)
- **Task** (task runner - install via `npm install @go-task/cli`)
## 🚀 Getting Started
### 1. Fork and Clone
```bash
# Fork the repository on GitHub first, then:
git clone https://github.com/YOUR_USERNAME/ten-framework.git
cd ten-framework
```
### 2. Set Up Development Environment
```bash
# Navigate to AI agents directory
cd ai_agents
# Copy environment template
cp .env.example .env
# Edit .env file with your API keys:
# - AGORA_APP_ID and AGORA_APP_CERTIFICATE
# - OpenAI, Deepgram, ElevenLabs API keys as needed
# Start development containers
docker compose up -d
# Enter the container
docker exec -it ten_agent_dev bash
# Build the agent (takes 5-8 minutes)
task use
# Build and run
task build
task run
```
### 3. Development Workflow
The project uses Task as the build system. Key commands:
```bash
# Core development commands
task build # Build the project
task test # Run all tests
task lint # Run linting
task format # Format code
task clean # Clean build artifacts
# AI Agents specific
task use AGENT=agents/examples/demo # Switch to different agent
task run-server # Run backend server
task run-gd-server # Run graph designer
```
## 🏗️ Project Structure
```
ten-framework/
├── ai_agents/ # AI Agent implementations
│ ├── agents/ # Agent examples and configurations
│ ├── playground/ # Frontend playground (Next.js)
│ ├── server/ # Backend server (Go)
│ └── Taskfile.yml # Build configuration
├── core/ # TEN Framework core
├── packages/ # Framework packages
├── docs/ # Documentation
├── tests/ # Test suites
└── third_party/ # Third-party dependencies
```
## 🔧 Making Changes
### Code Guidelines
1. **Follow existing patterns**: Review similar code before implementing
2. **Use existing libraries**: Check if the codebase already uses needed dependencies
3. **Security first**: Never expose secrets or credentials
4. **No hardcoded comments**: Don't add unnecessary code comments unless requested
### Language-Specific Guidelines
**Python (AI Agents)**
- Follow PEP 8 style guide
- Use Black formatter: `task format`
- Run pylint: `task lint`
- Add type hints where appropriate
**TypeScript/JavaScript (Playground)**
- Use existing ESLint configuration
- Follow React best practices
- Use TypeScript for new code
**Go (Server)**
- Follow Go conventions
- Use gofmt for formatting
- Add proper error handling
### Commit Guidelines
We use conventional commits:
```bash
feat: add new ASR integration
fix: resolve memory leak in audio processing
docs: update installation instructions
test: add integration tests for TTS module
refactor: improve error handling in server
```
## 🧪 Testing
### Running Tests
```bash
# Run all tests
task test
# Test specific extension
task test-extension EXTENSION=agents/ten_packages/extension/elevenlabs_tts_python
# Test server only
task test-server
# Test agent extensions
task test-agent-extensions
```
### Writing Tests
- Add unit tests for new functionality
- Include integration tests for complex features
- Ensure tests pass before submitting PRs
- Test configurations are in `tests/configs/` directories
## 📝 Documentation
- Update documentation for new features
- Include code examples where helpful
- Update README if adding new dependencies
- Document breaking changes clearly
## 🔄 Pull Request Process
### Before Submitting
1. **Test your changes**: Ensure all tests pass
2. **Format code**: Run `task format`
3. **Lint code**: Run `task lint`
4. **Update docs**: Include any necessary documentation updates
5. **Rebase on main**: Ensure your branch is up to date
### PR Guidelines
1. **Clear title**: Use descriptive, conventional commit style
2. **Detailed description**: Explain what, why, and how
3. **Link issues**: Reference related issues with "Fixes #123"
4. **Screenshots**: Include for UI changes
5. **Breaking changes**: Clearly document any breaking changes
### PR Template
```markdown
## Summary
Brief description of changes
## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Testing
- [ ] Tests added/updated
- [ ] All tests pass
- [ ] Manual testing completed
## Documentation
- [ ] Documentation updated
- [ ] Examples provided if needed
## Breaking Changes
List any breaking changes and migration steps
```
## 🐛 Reporting Issues
### Bug Reports
Use the issue template and include:
- **Environment**: OS, versions, Docker info
- **Steps to reproduce**: Clear, minimal reproduction steps
- **Expected vs actual**: What should happen vs what happens
- **Logs**: Include relevant error messages
- **Screenshots**: For UI issues
### Feature Requests
- **Use case**: Explain the problem you're solving
- **Proposed solution**: Describe your ideal solution
- **Alternatives**: Consider other approaches
- **Examples**: Reference similar implementations if possible
## 🏆 Recognition
Contributors are recognized in multiple ways:
- **Contributors graph**: Automatic GitHub recognition
- **Release notes**: Significant contributions mentioned
- **Social media**: Major features highlighted on [@TenFramework](https://twitter.com/TenFramework)
- **Community showcase**: Featured implementations
## 📞 Getting Help
### Community Channels
- **Discord**: [Join TEN Community](https://discord.gg/VnPftUzAMJ)
- **GitHub Discussions**: For questions and ideas
- **GitHub Issues**: For bugs and feature requests
- **WeChat**: Chinese community group
### Maintainer Contact
- **Twitter**: [@elliotchen100](https://x.com/elliotchen100)
- **GitHub**: [@cyfyifanchen](https://github.com/cyfyifanchen)
## 📜 License
By contributing to TEN Framework, you agree that your contributions will be licensed under the [Apache License v2.0 with additional conditions](../../LICENSE).
Your contributions must:
- Be your original work or properly attributed
- Not include any proprietary or confidential information
- Comply with the project's licensing terms
## ❤️ Thank You
Every contribution, no matter how small, helps make TEN Framework better for everyone. We appreciate your time and effort in making this project successful!
**Happy Contributing!** 🚀
---
### Development How To Develop With Ext.Cn (`docs/development/how_to_develop_with_ext.cn.md`)
---
title: TEN 扩展开发完整指南
_portal_target: development/how_to_develop_with_ext.cn.md
---
TEN Framework 提供了丰富的扩展模板，帮助开发者快速创建扩展并完成从开发到测试的完整流程。本指南将通过实际操作演示，详细介绍如何使用 C++，Go，Python，Node.js 四种语言进行扩展开发的全流程。
## 开发前准备
### 环境要求
在开始扩展开发之前，请确保您的开发环境已正确配置。通过以下命令验证安装：
```bash
tman --version
```
正常情况下，您应该看到类似以下的版本信息输出：
```bash
TEN Framework version:
```
> **重要提示**：请确保您使用的 `tman` 版本 >= 0.10.12，如果版本过低，请参考 [快速入门指南](https://theten.ai/cn/docs/ten_framework/getting-started/quick-start) 安装最新版本。
### 开发流程概览
无论使用哪种编程语言，TEN 扩展开发都遵循以下标准化流程：
1. **项目创建** - 使用官方模板生成扩展项目骨架
2. **依赖安装** - 配置运行环境并安装必要的依赖包
3. **核心开发** - 实现扩展的业务逻辑和功能代码
4. **构建测试** - 编译项目（如需要）并执行单元测试
5. **调试优化** - 使用专业调试工具定位并解决问题
---
## C++ 扩展开发
C++ 扩展适用于对性能要求极高的应用场景，如实时音视频处理、高频数据计算、底层系统操作等。
### 创建项目
使用 TEN 官方提供的 C++ 扩展模板快速创建新项目：
```bash
tman create extension my_example_ext_cpp --template default_extension_cpp
```
项目创建成功后，您会得到以下完整的项目结构：
```bash
my_example_ext_cpp/
├── BUILD.gn # GN 构建系统配置文件
├── manifest.json # 扩展元数据和配置信息
├── property.json # 扩展属性和参数配置
├── src/
│ └── main.cc # 扩展核心实现代码
├── tests/
│ ├── basic.cc # 基础功能测试用例
│ └── gtest_main.cc # 测试框架程序入口
├── include/ # 头文件存放目录
├── tools/ # 辅助工具和脚本
└── .vscode/
└── launch.json # VSCode 调试配置文件
```
### 环境配置验证
进入项目目录并验证构建工具是否正常工作：
```bash
cd my_example_ext_cpp
tgn --version
```
> **期望输出**：
```bash
0.1.0
```
如果命令无法执行，请参考 [快速入门指南](https://theten.ai/cn/docs/ten_framework/getting-started/quick-start) 检查 TEN Framework 开发环境是否正确安装。
### 依赖包安装
安装扩展运行时所需的全部依赖包：
```bash
tman install --standalone
```
安装成功后，您将看到类似以下的详细安装日志：
```bash
📦 Get all installed packages...
🔍 Filter compatible packages...
🔒 Creating manifest-lock.json...
📥 Installing packages...
[00:00:00] [########################################] 3/3 Done
🏆 Install successfully in 1 second
```
### 项目构建
TEN framework 提供了两种便捷的构建方式供您选择：
#### 方式一：手动分步构建
```bash
# 第一步：生成构建配置文件（启用独立测试模式）
tgn gen linux x64 debug -- ten_enable_standalone_test=true
# 第二步：执行项目构建
tgn build linux x64 debug
```
#### 方式二：使用 tman 快捷命令
```bash
tman run build
```
构建完成后，检查生成的可执行测试文件：
```bash
ls -la bin/
# 应该能看到：my_example_ext_cpp_test
```
### 运行测试
验证扩展功能是否正常工作：
#### 方式一：直接执行测试文件
```bash
./bin/my_example_ext_cpp_test
```
#### 方式二：使用 tman 统一命令
```bash
tman run test
```
测试执行成功时的输出示例：
```bash
Running main() from ../../../tests/gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] 1 test from Test
[ RUN ] Test.Basic
[ OK ] Test.Basic (20 ms)
[==========] 1 test from 1 test case ran. (79 ms total)
[ PASSED ] 1 test.
```
> **成功标志**：当您看到 `[ PASSED ] 1 test.` 时表示测试全部通过。
### 核心代码结构详解
#### 扩展主体实现（src/main.cc）
C++ 扩展的核心类需要继承自 `ten::extension_t` 基类，并实现完整的生命周期管理方法：
```cpp
class my_example_ext_cpp_t : public ten::extension_t {
public:
// 扩展初始化阶段 - 进行基础配置和资源预分配
void on_init(ten::ten_env_t &ten_env) override;
// 扩展启动阶段 - 正式开始处理业务逻辑
void on_start(ten::ten_env_t &ten_env) override;
// 命令处理器 - 处理来自其他扩展或应用的命令请求
void on_cmd(ten::ten_env_t &ten_env, std::unique_ptr cmd) override;
// 数据处理器 - 处理通用数据流
void on_data(ten::ten_env_t &ten_env, std::unique_ptr data) override;
// 音频帧处理器 - 处理实时音频流数据
void on_audio_frame(ten::ten_env_t &ten_env, std::unique_ptr frame) override;
// 视频帧处理器 - 处理实时视频流数据
void on_video_frame(ten::ten_env_t &ten_env, std::unique_ptr frame) override;
// 扩展停止阶段 - 清理资源和优雅关闭
void on_stop(ten::ten_env_t &ten_env) override;
};
```
#### 测试框架实现（tests/basic.cc）
使用 TEN 专用测试框架编写完整的单元测试：
```cpp
class my_example_ext_cpp_tester : public ten::extension_tester_t {
public:
void on_start(ten::ten_env_tester_t &ten_env) override {
// 创建测试用的命令对象
auto new_cmd = ten::cmd_t::create("foo");
// 发送命令到扩展并验证响应结果
ten_env.send_cmd(std::move(new_cmd), [](/* callback parameters */) {
// 在这里验证测试结果的正确性
});
}
};
```
### 调试环境配置
#### VSCode 集成调试
1. 安装 VSCode 的 **CodeLLDB** 扩展插件
2. 在源代码中设置断点
3. 选择 "standalone test (lldb, launch)" 调试配置
4. 按 F5 键启动调试会话
调试配置文件 `.vscode/launch.json` 的内容：
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "standalone test (lldb, launch)",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/bin/my_example_ext_cpp_test",
"cwd": "${workspaceFolder}"
}
]
}
```
#### 命令行调试
使用经典的 GDB 调试器进行命令行调试：
```bash
gdb ./bin/my_example_ext_cpp_test
(gdb) run
(gdb) bt # 查看完整的函数调用栈
```
---
## Go 扩展开发
Go 扩展在高性能和开发效率之间提供了很好的平衡，特别适合构建网络服务、并发处理、微服务架构等应用场景。
### 创建项目
使用 Go 扩展模板创建新项目：
```bash
tman create extension my_example_ext_go --template default_extension_go --template-data class_name_prefix=Example
```
项目创建成功后会显示：
```bash
🏆 Package 'extension:my_example_ext_go' created successfully in '/path/to/your/project' in 1 second.
```
完整的项目结构如下：
```bash
my_example_ext_go/
├── extension.go # 扩展核心实现代码
├── go.mod # Go 模块依赖管理配置
├── manifest.json # 扩展元数据信息
├── property.json # 扩展属性配置
├── README.md # 项目说明文档
├── tests/ # 测试相关文件
│ ├── basic_tester.go # 测试逻辑实现
│ ├── basic_tester_test.go # 测试用例定义
│ ├── main_test.go # 测试程序入口
│ └── bin/start # 测试启动脚本
└── .vscode/launch.json # VSCode 调试配置
```
### 依赖包安装
安装项目运行所需的依赖包：
```bash
tman install --standalone
```
### 运行测试
验证扩展功能的正确性：
#### 方式一：使用启动脚本
```bash
./tests/bin/start
```
#### 方式二：使用 tman 命令
```bash
tman run test
```
测试成功执行的输出示例：
```bash
=== RUN TestBasicExtensionTester
--- PASS: TestBasicExtensionTester (0.01s)
PASS
```
### 核心代码结构详解
#### 扩展实现（extension.go）
Go 扩展的核心实现结构：
```go
import (
ten "ten_framework/ten_runtime"
)
type ExampleExtension struct {
ten.DefaultExtension
}
// 扩展启动生命周期
func (e *ExampleExtension) OnStart(tenEnv ten.TenEnv) {
tenEnv.LogDebug("OnStart")
tenEnv.OnStartDone()
}
// 命令处理逻辑
func (e *ExampleExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
tenEnv.LogDebug("OnCmd")
cmdResult, _ := ten.NewCmdResult(ten.StatusCodeOk, cmd)
tenEnv.ReturnResult(cmdResult, nil)
}
// 扩展停止生命周期
func (e *ExampleExtension) OnStop(tenEnv ten.TenEnv) {
tenEnv.LogDebug("OnStop")
tenEnv.OnStopDone()
}
```
#### 测试框架（tests/basic_tester.go）
Go 扩展的测试框架实现：
```go
type BasicExtensionTester struct {
ten.DefaultExtensionTester
}
func (tester *BasicExtensionTester) OnStart(tenEnvTester ten.TenEnvTester) {
// 创建用于测试的命令对象
cmdTest, _ := ten.NewCmd("test")
// 发送命令并处理响应结果
tenEnvTester.SendCmd(cmdTest, func(tet ten.TenEnvTester, cr ten.CmdResult, err error) {
if err != nil {
panic(err)
}
statusCode, _ := cr.GetStatusCode()
if statusCode != ten.StatusCodeOk {
panic(statusCode)
}
tenEnvTester.StopTest(nil)
})
tenEnvTester.OnStartDone()
}
```
### 开发环境优化
为了提升开发体验和调试便利性，建议在扩展根目录创建 `go.work` 工作空间文件：
```go
go 1.18
use (
.
.ten/app/ten_packages/system/ten_runtime_go/interface
)
```
### 调试环境配置
#### VSCode 集成调试
确保已安装 Go 官方扩展，然后使用以下调试配置：
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "standalone test (go, launch)",
"type": "go",
"request": "launch",
"mode": "test",
"program": "${workspaceFolder}/tests/",
"env": {
"CGO_LDFLAGS": "-L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib -L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib -lten_runtime -lten_runtime_go",
"LD_LIBRARY_PATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib"
},
"args": ["-test.v"]
}
]
}
```
---
## Python 扩展开发
Python 扩展具有最高的开发效率，特别适合快速原型开发、AI/ML 应用集成、复杂业务逻辑实现等场景。
### 创建项目
使用 Python 异步扩展模板创建项目：
```bash
tman create extension my_example_ext_python --template default_async_extension_python --template-data class_name_prefix=Example
```
完整的项目结构：
```bash
my_example_ext_python/
├── extension.py # 扩展核心实现代码
├── addon.py # 扩展插件注册入口
├── __init__.py # Python 包初始化文件
├── requirements.txt # Python 依赖包清单
├── manifest.json # 扩展元数据配置
├── property.json # 扩展属性配置
├── tests/ # 测试相关文件
│ ├── test_basic.py # 基础测试用例
│ ├── conftest.py # pytest 配置文件
│ └── bin/start # 测试启动脚本
└── .vscode/launch.json # VSCode 调试配置
```
### 依赖包安装
安装项目所需的 Python 依赖包：
```bash
tman install --standalone
```
### 运行测试
验证 Python 扩展的功能：
#### 方式一：使用启动脚本
```bash
./tests/bin/start
```
#### 方式二：使用 tman 命令
```bash
tman run test
```
测试成功执行的输出示例：
```bash
============================================ test session starts ============================================
platform linux -- Python 3.10.17, pytest-8.3.4, pluggy-1.5.0
tests/test_basic.py . [100%]
============================================ 1 passed in 0.04s =======================================
```
### 核心代码结构详解
#### 扩展实现（extension.py）
Python 扩展推荐使用现代异步编程模式，以获得更好的性能和并发处理能力：
```python
from ten_runtime import (
AudioFrame, VideoFrame, AsyncExtension, AsyncTenEnv,
Cmd, StatusCode, CmdResult, Data
)
class ExampleExtension(AsyncExtension):
async def on_init(self, ten_env: AsyncTenEnv) -> None:
ten_env.log_debug("on_init")
async def on_start(self, ten_env: AsyncTenEnv) -> None:
ten_env.log_debug("on_start")
# TODO: 在这里读取配置文件，初始化必要的资源
async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
cmd_name = cmd.get_name()
ten_env.log_debug(f"on_cmd name {cmd_name}")
# TODO: 在这里实现具体的业务逻辑处理
cmd_result = CmdResult.create(StatusCode.OK, cmd)
await ten_env.return_result(cmd_result)
async def on_stop(self, ten_env: AsyncTenEnv) -> None:
ten_env.log_debug("on_stop")
# TODO: 在这里清理资源，进行优雅关闭
```
#### 插件注册入口（addon.py）
扩展插件的注册和创建逻辑：
```python
from ten_runtime import Addon, register_addon_as_extension, TenEnv
from .extension import ExampleExtension
@register_addon_as_extension("my_example_ext_python")
class ExampleExtensionAddon(Addon):
def on_create_instance(self, ten_env: TenEnv, name: str, context) -> None:
ten_env.log_info("on_create_instance")
ten_env.on_create_instance_done(ExampleExtension(name), context)
```
#### 测试实现（tests/test_basic.py）
完整的异步测试框架实现：
```python
from ten_runtime import (
AsyncExtensionTester, AsyncTenEnvTester, Cmd, StatusCode,
TenError, TenErrorCode
)
class ExtensionTesterBasic(AsyncExtensionTester):
async def on_start(self, ten_env: AsyncTenEnvTester) -> None:
# 创建用于测试的命令对象
new_cmd = Cmd.create("hello_world")
ten_env.log_debug("send hello_world")
result, err = await ten_env.send_cmd(new_cmd)
# 验证测试结果的正确性
if (err is not None or result is None
or result.get_status_code() != StatusCode.OK):
ten_env.stop_test(TenError.create(
TenErrorCode.ErrorCodeGeneric,
"Failed to send hello_world"
))
else:
ten_env.stop_test()
def test_basic():
tester = ExtensionTesterBasic()
tester.set_test_mode_single("my_example_ext_python")
err = tester.run()
if err is not None:
assert False, err.error_message()
```
### 调试环境配置
#### VSCode 集成调试
确保已安装 Python 扩展和 debugpy 调试器，使用以下配置进行调试：
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "standalone test (debugpy, launch)",
"type": "debugpy",
"request": "launch",
"python": "/usr/bin/python3",
"module": "pytest",
"args": ["-s", "${workspaceFolder}/tests/test_basic.py"],
"env": {
"TEN_ENABLE_PYTHON_DEBUG": "true",
"PYTHONPATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/interface:${workspaceFolder}"
},
"console": "integratedTerminal"
}
]
}
```
---
## Node.js 扩展开发
Node.js 扩展提供了现代 JavaScript/TypeScript 开发体验，特别适合 Web 应用集成、快速原型开发、前端技术栈扩展等场景。得益于 Node.js 的异步特性和丰富的生态系统，开发者可以轻松构建高效的实时应用。
### 创建项目
使用 TEN 官方提供的 Node.js 扩展模板快速创建新项目：
```bash
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example
```
项目创建成功后，您会得到以下完整的项目结构：
```bash
my_example_ext_nodejs/
├── manifest.json # 扩展元数据和配置信息
├── property.json # 扩展属性和参数配置
├── package.json # Node.js 依赖包管理配置
├── tsconfig.json # TypeScript 编译器配置
├── src/
│ └── index.ts # 扩展核心实现代码
├── tests/ # 测试相关文件
│ ├── src/
│ │ ├── index.ts # 测试器实现
│ │ ├── index.spec.ts # 测试用例定义
│ │ └── main.spec.ts # 测试框架配置
│ ├── bin/start # 测试启动脚本
│ ├── package.json # 测试依赖配置
│ └── tsconfig.json # 测试 TypeScript 配置
├── tools/ # 辅助工具和脚本
└── .vscode/
└── launch.json # VSCode 调试配置文件
```
### 环境配置验证
进入项目目录并验证开发环境：
```bash
cd my_example_ext_nodejs
node --version
npm --version
```
> **期望输出**：
```bash
v18.0.0 或更高版本
8.0.0 或更高版本
```
### 依赖包安装
安装扩展运行时所需的全部依赖包：
```bash
tman install --standalone
```
安装成功后，您将看到类似以下的详细安装日志：
```bash
📦 Get all installed packages...
🔍 Filter compatible packages...
🔒 Creating manifest-lock.json...
📥 Installing packages...
[00:00:00] [########################################] 2/2 Done
🏆 Install successfully in 1 second
```
> **重要提示**：`tman install --standalone` 会在项目目录下创建 `.ten/app/ten_packages/extension/my_example_ext_nodejs/` 目录，后续的构建和测试操作都需要在这个目录下进行。
### 项目构建
Node.js 扩展使用 TypeScript 进行开发，需要先安装独立模式依赖，然后编译为 JavaScript：
#### 方式一：手动构建
```bash
# 进入扩展的安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs
# 安装独立模式的 Node.js 依赖
npm run standalone-install
# 编译 TypeScript 代码
npm run build
```
#### 方式二：使用 tman 快捷命令
```bash
tman run build
```
构建完成后，检查生成的编译结果：
```bash
ls -la .ten/app/ten_packages/extension/my_example_ext_nodejs/build/
# 应该能看到：index.js 和相关映射文件
```
### 运行测试
验证扩展功能是否正常工作：
#### 方式一：使用启动脚本
```bash
# 进入扩展的安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs
# 运行测试
./tests/bin/start
```
#### 方式二：使用 tman 统一命令
```bash
tman run test
```
测试执行成功时的输出示例：
```bash
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
✓ case1 (1010ms)
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
✓ case2 (1010ms)
2 passing (2s)
```
> **成功标志**：当您看到所有测试用例显示 `✓` 并且最后显示 `passing` 时表示测试全部通过。
### 核心代码结构详解
#### 扩展主体实现（src/index.ts）
Node.js 扩展的核心类需要继承自 `Extension` 基类，并实现完整的生命周期管理方法：
```typescript
import {
Addon,
RegisterAddonAsExtension,
Extension,
TenEnv,
Cmd,
CmdResult,
StatusCode,
} from "ten-runtime-nodejs";
class DefaultExtension extends Extension {
constructor(name: string) {
super(name);
}
// 扩展配置阶段 - 进行配置参数的读取和验证
async onConfigure(_tenEnv: TenEnv): Promise {
console.log("DefaultExtension onConfigure");
}
// 扩展初始化阶段 - 进行基础配置和资源预分配
async onInit(_tenEnv: TenEnv): Promise {
console.log("DefaultExtension onInit");
}
// 扩展启动阶段 - 正式开始处理业务逻辑
async onStart(_tenEnv: TenEnv): Promise {
console.log("DefaultExtension onStart");
}
// 命令处理器 - 处理来自其他扩展或应用的命令请求
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise {
console.log("DefaultExtension onCmd", cmd.getName());
const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
cmdResult.setPropertyString("detail", "This is a demo");
tenEnv.returnResult(cmdResult);
}
// 扩展停止阶段 - 清理资源和优雅关闭
async onStop(_tenEnv: TenEnv): Promise {
console.log("DefaultExtension onStop");
}
// 扩展销毁阶段 - 最终清理和资源释放
async onDeinit(_tenEnv: TenEnv): Promise {
console.log("DefaultExtension onDeinit");
}
}
```
#### 插件注册入口
扩展插件的注册和创建逻辑：
```typescript
@RegisterAddonAsExtension("default_extension_nodejs")
class DefaultExtensionAddon extends Addon {
async onCreateInstance(
_tenEnv: TenEnv,
instanceName: string,
): Promise {
return new DefaultExtension(instanceName);
}
}
```
#### 测试框架实现（tests/src/index.ts）
使用 TEN 专用测试框架编写完整的单元测试：
```typescript
import { ExtensionTester, TenEnvTester } from "ten-runtime-nodejs";
export class MyExtensionTester extends ExtensionTester {
async onStart(tenEnvTester: TenEnvTester) {
console.log("MyExtensionTester onStart");
// 模拟异步操作和测试逻辑
new Promise((resolve) => {
setTimeout(() => {
resolve(true);
}, 1000);
}).then(() => {
// 停止测试并返回结果
tenEnvTester.stopTest();
});
}
async onStop(tenEnvTester: TenEnvTester) {
console.log("MyExtensionTester onStop");
}
async onDeinit(tenEnvTester: TenEnvTester) {
console.log("MyExtensionTester onDeinit");
}
}
```
#### 测试用例定义（tests/src/index.spec.ts）
使用 Mocha 测试框架编写具体的测试用例：
```typescript
import assert from "assert";
import { MyExtensionTester } from "./index.js";
const test_addon_name = "default_extension_nodejs";
describe("MyExtensionTester", () => {
it("case1", async () => {
const extensionTester = new MyExtensionTester();
extensionTester.setTestModeSingle(test_addon_name, "{}");
const result = await extensionTester.run();
assert(result === null, "result should be null");
console.log("deinit done");
});
it("case2", async () => {
const extensionTester = new MyExtensionTester();
extensionTester.setTestModeSingle(test_addon_name, "{}");
const result = await extensionTester.run();
assert(result === null, "result should be null");
console.log("deinit done");
});
});
```
### TypeScript 配置
Node.js 扩展使用现代 TypeScript 配置，支持最新的语言特性：
```json
{
"compilerOptions": {
"allowJs": false,
"composite": true,
"module": "NodeNext",
"target": "es6",
"moduleResolution": "NodeNext",
"outDir": "build",
"removeComments": false,
"sourceMap": true,
"noImplicitAny": true,
"noImplicitThis": true,
"noImplicitReturns": true,
"strictNullChecks": true,
"experimentalDecorators": true,
"emitDecoratorMetadata": true,
"rootDir": "src",
"strict": true,
"forceConsistentCasingInFileNames": true,
},
"exclude": [
"node_modules"
],
"include": [
"src/**/*"
]
}
```
### 调试环境配置
#### VSCode 集成调试
确保已安装 Node.js 官方扩展，然后使用以下调试配置：
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "standalone test (nodejs) (mocha, launch)",
"type": "node",
"request": "launch",
"program": "node_modules/mocha/bin/_mocha",
"stopOnEntry": true,
"args": [
"--no-timeouts",
"--package",
"package.json",
],
"cwd": "${workspaceFolder}/tests",
"env": {
"NODE_PATH": "../.ten/app/ten_packages/system/ten_runtime_nodejs/lib:$NODE_PATH",
},
"runtimeArgs": [
"--expose-gc",
"--loader",
"ts-node/esm",
"--no-warnings",
]
}
]
}
```
#### 命令行调试
使用 Node.js 内置调试器进行命令行调试：
```bash
# 进入扩展的安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs
# 启动调试模式
node --inspect-brk ./tests/bin/start
# 或者使用 Chrome DevTools
node --inspect ./tests/bin/start
```
### 完整开发流程总结
为了帮助开发者快速上手，这里提供一个完整的 Node.js 扩展开发流程总结：
```bash
# 1. 创建扩展项目
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example
# 2. 进入项目目录
cd my_example_ext_nodejs
# 3. 安装依赖
tman install --standalone
# 4. 进入扩展安装目录
cd .ten/app/ten_packages/extension/my_example_ext_nodejs
# 5. 安装独立模式依赖
npm run standalone-install
# 6. 构建项目
npm run build
# 7. 运行测试
./tests/bin/start
```
> **工作目录说明**：
>
> - 扩展源代码位于项目根目录的 `src/` 文件夹
> - 实际的构建、测试和运行操作都在 `.ten/app/ten_packages/extension/my_example_ext_nodejs/` 目录下进行
> - 这种设计确保了扩展的独立性和依赖管理的正确性
### 开发最佳实践
#### 异步编程
Node.js 扩展充分利用异步编程模式，提供更好的性能：
```typescript
// 推荐：使用 async/await
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise {
try {
const result = await processCommand(cmd);
const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
cmdResult.setPropertyString("result", result);
tenEnv.returnResult(cmdResult);
} catch (error) {
const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
cmdResult.setPropertyString("error", error.message);
tenEnv.returnResult(cmdResult);
}
}
```
#### 错误处理
实现完善的错误处理机制：
```typescript
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise {
try {
// 业务逻辑处理
const result = await this.handleBusinessLogic(cmd);
const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
cmdResult.setPropertyString("data", JSON.stringify(result));
tenEnv.returnResult(cmdResult);
} catch (error) {
console.error("Command processing error:", error);
const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
cmdResult.setPropertyString("error", error.message);
tenEnv.returnResult(cmdResult);
}
}
```
---
## 开发总结
通过遵循本指南提供的完整开发流程，您可以高效地进行 TEN 扩展的开发、测试和调试工作。无论选择 C++，Go，Python 还是 Node.js，TEN Framework 都为您提供了完整的工具链和最佳实践，帮助您充分发挥 TEN Framework 的强大功能，构建出高性能、高可靠性的扩展应用。
每种语言都有其独特的优势和适用场景：
- **C++**：适用于对性能要求极高的场景，如实时音视频处理、高频计算
- **Go**：在高性能和开发效率之间提供平衡，适合网络服务、并发处理
- **Python**：具有最高的开发效率，特别适合AI/ML应用、快速原型开发
- **Node.js**：提供现代 Web 开发体验，适用于前端技术栈扩展、实时应用
请根据您的具体需求和团队技术栈选择最合适的开发方案。在开发过程中，建议充分利用 TEN Framework 提供的调试工具和测试框架，以确保扩展的质量和稳定性。
---
### Development How To Develop With Ext (`docs/development/how_to_develop_with_ext.md`)
---
title: Complete Guide to TEN Extension Development
_portal_target: development/how_to_develop_with_ext.md
---
TEN Framework provides rich extension templates to help developers quickly create extensions and complete the entire process from development to testing. This guide will demonstrate in detail how to use C++, Go, Python, and Node.js for the complete extension development workflow through practical operations.
## Pre-development Preparation
### Environment Requirements
Before starting extension development, please ensure your development environment is properly configured. Verify the installation with the following command:
```bash
tman --version
```
Under normal circumstances, you should see version information output similar to the following:
```bash
TEN Framework version:
```
> **Important Note**: Please ensure you are using `tman` version >= 0.10.12. If your version is too low, please refer to the [Quick Start Guide](https://theten.ai/docs/ten_framework/getting-started/quick-start) to install the latest version.
### Development Workflow Overview
Regardless of which programming language you use, TEN extension development follows the following standardized workflow:
1. **Project Creation** - Generate extension project skeleton using official templates
2. **Dependency Installation** - Configure runtime environment and install necessary dependency packages
3. **Core Development** - Implement extension business logic and functional code
4. **Build and Test** - Compile project (if needed) and execute unit tests
5. **Debug and Optimize** - Use professional debugging tools to locate and solve problems
---
## C++ Extension Development
C++ extensions are suitable for application scenarios with extremely high performance requirements, such as real-time audio and video processing, high-frequency data computation, low-level system operations, etc.
### Create Project
Use the official C++ extension template provided by TEN to quickly create a new project:
```bash
tman create extension my_example_ext_cpp --template default_extension_cpp
```
After successful project creation, you will get the following complete project structure:
```bash
my_example_ext_cpp/
├── BUILD.gn # GN build system configuration file
├── manifest.json # Extension metadata and configuration information
├── property.json # Extension properties and parameter configuration
├── src/
│ └── main.cc # Extension core implementation code
├── tests/
│ ├── basic.cc # Basic functionality test cases
│ └── gtest_main.cc # Test framework program entry
├── include/ # Header file storage directory
├── tools/ # Auxiliary tools and scripts
└── .vscode/
└── launch.json # VSCode debugging configuration file
```
### Environment Configuration Verification
Enter the project directory and verify that the build tools are working properly:
```bash
cd my_example_ext_cpp
tgn --version
```
> **Expected Output**:
```bash
0.1.0
```
If the command cannot be executed, please refer to the [Quick Start Guide](https://theten.ai/docs/ten_framework/getting-started/quick-start) to check whether the TEN Framework development environment is correctly installed.
### Dependency Package Installation
Install all dependency packages required for extension runtime:
```bash
tman install --standalone
```
After successful installation, you will see detailed installation logs similar to the following:
```bash
📦 Get all installed packages...
🔍 Filter compatible packages...
🔒 Creating manifest-lock.json...
📥 Installing packages...
[00:00:00] [########################################] 3/3 Done
🏆 Install successfully in 1 second
```
### Project Build
TEN framework provides two convenient build methods for you to choose from:
#### Method 1: Manual Step-by-step Build
```bash
# Step 1: Generate build configuration files (enable standalone test mode)
tgn gen linux x64 debug -- ten_enable_standalone_test=true
# Step 2: Execute project build
tgn build linux x64 debug
```
#### Method 2: Use tman Shortcut Command
```bash
tman run build
```
After the build is complete, check the generated executable test file:
```bash
ls -la bin/
# You should see: my_example_ext_cpp_test
```
### Run Tests
Verify that the extension functionality is working properly:
#### Method 1: Execute Test File Directly
```bash
./bin/my_example_ext_cpp_test
```
#### Method 2: Use tman Unified Command
```bash
tman run test
```
Example output when tests execute successfully:
```bash
Running main() from ../../../tests/gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] 1 test from Test
[ RUN ] Test.Basic
[ OK ] Test.Basic (20 ms)
[==========] 1 test from 1 test case ran. (79 ms total)
[ PASSED ] 1 test.
```
> **Success Indicator**: When you see `[ PASSED ] 1 test.` it means all tests have passed.
### Core Code Structure Explanation
#### Extension Main Implementation (src/main.cc)
The core class of C++ extensions needs to inherit from the `ten::extension_t` base class and implement complete lifecycle management methods:
```cpp
class my_example_ext_cpp_t : public ten::extension_t {
public:
// Extension initialization phase - basic configuration and resource pre-allocation
void on_init(ten::ten_env_t &ten_env) override;
// Extension startup phase - officially start processing business logic
void on_start(ten::ten_env_t &ten_env) override;
// Command handler - handle command requests from other extensions or applications
void on_cmd(ten::ten_env_t &ten_env, std::unique_ptr cmd) override;
// Data handler - handle general data flow
void on_data(ten::ten_env_t &ten_env, std::unique_ptr data) override;
// Audio frame handler - handle real-time audio stream data
void on_audio_frame(ten::ten_env_t &ten_env, std::unique_ptr frame) override;
// Video frame handler - handle real-time video stream data
void on_video_frame(ten::ten_env_t &ten_env, std::unique_ptr frame) override;
// Extension stop phase - clean up resources and graceful shutdown
void on_stop(ten::ten_env_t &ten_env) override;
};
```
#### Test Framework Implementation (tests/basic.cc)
Use TEN-specific test framework to write complete unit tests:
```cpp
class my_example_ext_cpp_tester : public ten::extension_tester_t {
public:
void on_start(ten::ten_env_tester_t &ten_env) override {
// Create command object for testing
auto new_cmd = ten::cmd_t::create("foo");
// Send command to extension and verify response results
ten_env.send_cmd(std::move(new_cmd), [](/* callback parameters */) {
// Verify the correctness of test results here
});
}
};
```
### Debug Environment Configuration
#### VSCode Integrated Debugging
1. Install the **CodeLLDB** extension plugin for VSCode
2. Set breakpoints in source code
3. Select the "standalone test (lldb, launch)" debug configuration
4. Press F5 to start debugging session
Debug configuration file `.vscode/launch.json` content:
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "standalone test (lldb, launch)",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/bin/my_example_ext_cpp_test",
"cwd": "${workspaceFolder}"
}
]
}
```
#### Command Line Debugging
Use the classic GDB debugger for command-line debugging:
```bash
gdb ./bin/my_example_ext_cpp_test
(gdb) run
(gdb) bt # View complete function call stack
```
---
## Go Extension Development
Go extensions provide a good balance between high performance and development efficiency, particularly suitable for building network services, concurrent processing, microservice architecture and other application scenarios.
### Create Project
Create a new project using the Go extension template:
```bash
tman create extension my_example_ext_go --template default_extension_go --template-data class_name_prefix=Example
```
After successful project creation, it will display:
```bash
🏆 Package 'extension:my_example_ext_go' created successfully in '/path/to/your/project' in 1 second.
```
The complete project structure is as follows:
```bash
my_example_ext_go/
├── extension.go # Extension core implementation code
├── go.mod # Go module dependency management configuration
├── manifest.json # Extension metadata information
├── property.json # Extension property configuration
├── README.md # Project documentation
├── tests/ # Test-related files
│ ├── basic_tester.go # Test logic implementation
│ ├── basic_tester_test.go # Test case definition
│ ├── main_test.go # Test program entry
│ └── bin/start # Test startup script
└── .vscode/launch.json # VSCode debug configuration
```
### Dependency Package Installation
Install dependency packages required for project runtime:
```bash
tman install --standalone
```
### Run Tests
Verify the correctness of extension functionality:
#### Method 1: Use Startup Script
```bash
./tests/bin/start
```
#### Method 2: Use tman Command
```bash
tman run test
```
Example output when tests execute successfully:
```bash
=== RUN TestBasicExtensionTester
--- PASS: TestBasicExtensionTester (0.01s)
PASS
```
### Core Code Structure Explanation
#### Extension Implementation (extension.go)
Core implementation structure of Go extensions:
```go
import (
ten "ten_framework/ten_runtime"
)
type ExampleExtension struct {
ten.DefaultExtension
}
// Extension startup lifecycle
func (e *ExampleExtension) OnStart(tenEnv ten.TenEnv) {
tenEnv.LogDebug("OnStart")
tenEnv.OnStartDone()
}
// Command handling logic
func (e *ExampleExtension) OnCmd(tenEnv ten.TenEnv, cmd ten.Cmd) {
tenEnv.LogDebug("OnCmd")
cmdResult, _ := ten.NewCmdResult(ten.StatusCodeOk, cmd)
tenEnv.ReturnResult(cmdResult, nil)
}
// Extension stop lifecycle
func (e *ExampleExtension) OnStop(tenEnv ten.TenEnv) {
tenEnv.LogDebug("OnStop")
tenEnv.OnStopDone()
}
```
#### Test Framework (tests/basic_tester.go)
Go extension test framework implementation:
```go
type BasicExtensionTester struct {
ten.DefaultExtensionTester
}
func (tester *BasicExtensionTester) OnStart(tenEnvTester ten.TenEnvTester) {
// Create command object for testing
cmdTest, _ := ten.NewCmd("test")
// Send command and handle response results
tenEnvTester.SendCmd(cmdTest, func(tet ten.TenEnvTester, cr ten.CmdResult, err error) {
if err != nil {
panic(err)
}
statusCode, _ := cr.GetStatusCode()
if statusCode != ten.StatusCodeOk {
panic(statusCode)
}
tenEnvTester.StopTest(nil)
})
tenEnvTester.OnStartDone()
}
```
### Development Environment Optimization
To improve development experience and debugging convenience, it is recommended to create a `go.work` workspace file in the extension root directory:
```go
go 1.18
use (
.
.ten/app/ten_packages/system/ten_runtime_go/interface
)
```
### Debug Environment Configuration
#### VSCode Integrated Debugging
Make sure the official Go extension is installed, then use the following debug configuration:
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "standalone test (go, launch)",
"type": "go",
"request": "launch",
"mode": "test",
"program": "${workspaceFolder}/tests/",
"env": {
"CGO_LDFLAGS": "-L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib -L${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib -lten_runtime -lten_runtime_go",
"LD_LIBRARY_PATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_go/lib"
},
"args": ["-test.v"]
}
]
}
```
---
## Python Extension Development
Python extensions have the highest development efficiency, particularly suitable for rapid prototyping, AI/ML application integration, complex business logic implementation and other scenarios.
### Create Project
Create a project using the Python async extension template:
```bash
tman create extension my_example_ext_python --template default_async_extension_python --template-data class_name_prefix=Example
```
Complete project structure:
```bash
my_example_ext_python/
├── extension.py # Extension core implementation code
├── addon.py # Extension plugin registration entry
├── __init__.py # Python package initialization file
├── requirements.txt # Python dependency package list
├── manifest.json # Extension metadata configuration
├── property.json # Extension property configuration
├── tests/ # Test-related files
│ ├── test_basic.py # Basic test cases
│ ├── conftest.py # pytest configuration file
│ └── bin/start # Test startup script
└── .vscode/launch.json # VSCode debug configuration
```
### Dependency Package Installation
Install Python dependency packages required for the project:
```bash
tman install --standalone
```
### Run Tests
Verify Python extension functionality:
#### Method 1: Use Startup Script
```bash
./tests/bin/start
```
#### Method 2: Use tman Command
```bash
tman run test
```
Example output when tests execute successfully:
```bash
============================================ test session starts ============================================
platform linux -- Python 3.10.17, pytest-8.3.4, pluggy-1.5.0
tests/test_basic.py . [100%]
============================================ 1 passed in 0.04s =======================================
```
### Core Code Structure Explanation
#### Extension Implementation (extension.py)
Python extensions recommend using modern async programming patterns for better performance and concurrent processing capabilities:
```python
from ten_runtime import (
AudioFrame, VideoFrame, AsyncExtension, AsyncTenEnv,
Cmd, StatusCode, CmdResult, Data
)
class ExampleExtension(AsyncExtension):
async def on_init(self, ten_env: AsyncTenEnv) -> None:
ten_env.log_debug("on_init")
async def on_start(self, ten_env: AsyncTenEnv) -> None:
ten_env.log_debug("on_start")
# TODO: Read configuration files here, initialize necessary resources
async def on_cmd(self, ten_env: AsyncTenEnv, cmd: Cmd) -> None:
cmd_name = cmd.get_name()
ten_env.log_debug(f"on_cmd name {cmd_name}")
# TODO: Implement specific business logic processing here
cmd_result = CmdResult.create(StatusCode.OK, cmd)
await ten_env.return_result(cmd_result)
async def on_stop(self, ten_env: AsyncTenEnv) -> None:
ten_env.log_debug("on_stop")
# TODO: Clean up resources here, perform graceful shutdown
```
#### Plugin Registration Entry (addon.py)
Extension plugin registration and creation logic:
```python
from ten_runtime import Addon, register_addon_as_extension, TenEnv
from .extension import ExampleExtension
@register_addon_as_extension("my_example_ext_python")
class ExampleExtensionAddon(Addon):
def on_create_instance(self, ten_env: TenEnv, name: str, context) -> None:
ten_env.log_info("on_create_instance")
ten_env.on_create_instance_done(ExampleExtension(name), context)
```
#### Test Implementation (tests/test_basic.py)
Complete async test framework implementation:
```python
from ten_runtime import (
AsyncExtensionTester, AsyncTenEnvTester, Cmd, StatusCode,
TenError, TenErrorCode
)
class ExtensionTesterBasic(AsyncExtensionTester):
async def on_start(self, ten_env: AsyncTenEnvTester) -> None:
# Create command object for testing
new_cmd = Cmd.create("hello_world")
ten_env.log_debug("send hello_world")
result, err = await ten_env.send_cmd(new_cmd)
# Verify the correctness of test results
if (err is not None or result is None
or result.get_status_code() != StatusCode.OK):
ten_env.stop_test(TenError.create(
TenErrorCode.ErrorCodeGeneric,
"Failed to send hello_world"
))
else:
ten_env.stop_test()
def test_basic():
tester = ExtensionTesterBasic()
tester.set_test_mode_single("my_example_ext_python")
err = tester.run()
if err is not None:
assert False, err.error_message()
```
### Debug Environment Configuration
#### VSCode Integrated Debugging
Make sure you have installed the Python extension and debugpy debugger, use the following configuration for debugging:
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "standalone test (debugpy, launch)",
"type": "debugpy",
"request": "launch",
"python": "/usr/bin/python3",
"module": "pytest",
"args": ["-s", "${workspaceFolder}/tests/test_basic.py"],
"env": {
"TEN_ENABLE_PYTHON_DEBUG": "true",
"PYTHONPATH": "${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/lib:${workspaceFolder}/.ten/app/ten_packages/system/ten_runtime_python/interface:${workspaceFolder}"
},
"console": "integratedTerminal"
}
]
}
```
---
## Node.js Extension Development
Node.js extensions provide a modern JavaScript/TypeScript development experience, particularly suitable for Web application integration, rapid prototyping, frontend technology stack extensions and other scenarios. Thanks to Node.js's async characteristics and rich ecosystem, developers can easily build efficient real-time applications.
### Create Project
Use the official Node.js extension template provided by TEN to quickly create a new project:
```bash
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example
```
After successful project creation, you will get the following complete project structure:
```bash
my_example_ext_nodejs/
├── manifest.json # Extension metadata and configuration information
├── property.json # Extension properties and parameter configuration
├── package.json # Node.js dependency package management configuration
├── tsconfig.json # TypeScript compiler configuration
├── src/
│ └── index.ts # Extension core implementation code
├── tests/ # Test-related files
│ ├── src/
│ │ ├── index.ts # Tester implementation
│ │ ├── index.spec.ts # Test case definition
│ │ └── main.spec.ts # Test framework configuration
│ ├── bin/start # Test startup script
│ ├── package.json # Test dependency configuration
│ └── tsconfig.json # Test TypeScript configuration
├── tools/ # Auxiliary tools and scripts
└── .vscode/
└── launch.json # VSCode debug configuration file
```
### Environment Configuration Verification
Enter the project directory and verify the development environment:
```bash
cd my_example_ext_nodejs
node --version
npm --version
```
> **Expected Output**:
```bash
v18.0.0 or higher
8.0.0 or higher
```
### Dependency Package Installation
Install all dependency packages required for extension runtime:
```bash
tman install --standalone
```
After successful installation, you will see detailed installation logs similar to the following:
```bash
📦 Get all installed packages...
🔍 Filter compatible packages...
🔒 Creating manifest-lock.json...
📥 Installing packages...
[00:00:00] [########################################] 2/2 Done
🏆 Install successfully in 1 second
```
> **Important Note**: `tman install --standalone` will create a `.ten/app/ten_packages/extension/my_example_ext_nodejs/` directory in the project directory. Subsequent build and test operations need to be performed in this directory.
### Project Build
Node.js extensions are developed using TypeScript and need to install standalone mode dependencies first, then compile to JavaScript:
#### Method 1: Manual Build
```bash
# Enter the extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs
# Install standalone mode Node.js dependencies
npm run standalone-install
# Compile TypeScript code
npm run build
```
#### Method 2: Use tman Shortcut Command
```bash
tman run build
```
After the build is complete, check the generated compilation results:
```bash
ls -la .ten/app/ten_packages/extension/my_example_ext_nodejs/build/
# You should see: index.js and related mapping files
```
### Run Tests
Verify that the extension functionality is working properly:
#### Method 1: Use Startup Script
```bash
# Enter the extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs
# Run tests
./tests/bin/start
```
#### Method 2: Use tman Unified Command
```bash
tman run test
```
Example output when tests execute successfully:
```bash
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
✓ case1 (1010ms)
MyExtensionTester onStart
MyExtensionTester onStop
MyExtensionTester onDeinit
deinit done
✓ case2 (1010ms)
2 passing (2s)
```
> **Success Indicator**: When you see all test cases showing `✓` and finally displaying `passing`, it means all tests have passed.
### Core Code Structure Explanation
#### Extension Main Implementation (src/index.ts)
The core class of Node.js extensions needs to inherit from the `Extension` base class and implement complete lifecycle management methods:
```typescript
import {
Addon,
RegisterAddonAsExtension,
Extension,
TenEnv,
Cmd,
CmdResult,
StatusCode,
} from "ten-runtime-nodejs";
class DefaultExtension extends Extension {
constructor(name: string) {
super(name);
}
// Extension configuration phase - read and validate configuration parameters
async onConfigure(_tenEnv: TenEnv): Promise {
console.log("DefaultExtension onConfigure");
}
// Extension initialization phase - basic configuration and resource pre-allocation
async onInit(_tenEnv: TenEnv): Promise {
console.log("DefaultExtension onInit");
}
// Extension startup phase - officially start processing business logic
async onStart(_tenEnv: TenEnv): Promise {
console.log("DefaultExtension onStart");
}
// Command handler - handle command requests from other extensions or applications
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise {
console.log("DefaultExtension onCmd", cmd.getName());
const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
cmdResult.setPropertyString("detail", "This is a demo");
tenEnv.returnResult(cmdResult);
}
// Extension stop phase - clean up resources and graceful shutdown
async onStop(_tenEnv: TenEnv): Promise {
console.log("DefaultExtension onStop");
}
// Extension destruction phase - final cleanup and resource release
async onDeinit(_tenEnv: TenEnv): Promise {
console.log("DefaultExtension onDeinit");
}
}
```
#### Plugin Registration Entry
Extension plugin registration and creation logic:
```typescript
@RegisterAddonAsExtension("default_extension_nodejs")
class DefaultExtensionAddon extends Addon {
async onCreateInstance(
_tenEnv: TenEnv,
instanceName: string,
): Promise {
return new DefaultExtension(instanceName);
}
}
```
#### Test Framework Implementation (tests/src/index.ts)
Use TEN-specific test framework to write complete unit tests:
```typescript
import { ExtensionTester, TenEnvTester } from "ten-runtime-nodejs";
export class MyExtensionTester extends ExtensionTester {
async onStart(tenEnvTester: TenEnvTester) {
console.log("MyExtensionTester onStart");
// Simulate async operations and test logic
new Promise((resolve) => {
setTimeout(() => {
resolve(true);
}, 1000);
}).then(() => {
// Stop test and return results
tenEnvTester.stopTest();
});
}
async onStop(tenEnvTester: TenEnvTester) {
console.log("MyExtensionTester onStop");
}
async onDeinit(tenEnvTester: TenEnvTester) {
console.log("MyExtensionTester onDeinit");
}
}
```
#### Test Case Definition (tests/src/index.spec.ts)
Use Mocha test framework to write specific test cases:
```typescript
import assert from "assert";
import { MyExtensionTester } from "./index.js";
const test_addon_name = "default_extension_nodejs";
describe("MyExtensionTester", () => {
it("case1", async () => {
const extensionTester = new MyExtensionTester();
extensionTester.setTestModeSingle(test_addon_name, "{}");
const result = await extensionTester.run();
assert(result === null, "result should be null");
console.log("deinit done");
});
it("case2", async () => {
const extensionTester = new MyExtensionTester();
extensionTester.setTestModeSingle(test_addon_name, "{}");
const result = await extensionTester.run();
assert(result === null, "result should be null");
console.log("deinit done");
});
});
```
### TypeScript Configuration
Node.js extensions use modern TypeScript configuration, supporting the latest language features:
```json
{
"compilerOptions": {
"allowJs": false,
"composite": true,
"module": "NodeNext",
"target": "es6",
"moduleResolution": "NodeNext",
"outDir": "build",
"removeComments": false,
"sourceMap": true,
"noImplicitAny": true,
"noImplicitThis": true,
"noImplicitReturns": true,
"strictNullChecks": true,
"experimentalDecorators": true,
"emitDecoratorMetadata": true,
"rootDir": "src",
"strict": true,
"forceConsistentCasingInFileNames": true,
},
"exclude": [
"node_modules"
],
"include": [
"src/**/*"
]
}
```
### Debug Environment Configuration
#### VSCode Integrated Debugging
Make sure you have installed the official Node.js extension, then use the following debug configuration:
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "standalone test (nodejs) (mocha, launch)",
"type": "node",
"request": "launch",
"program": "node_modules/mocha/bin/_mocha",
"stopOnEntry": true,
"args": [
"--no-timeouts",
"--package",
"package.json",
],
"cwd": "${workspaceFolder}/tests",
"env": {
"NODE_PATH": "../.ten/app/ten_packages/system/ten_runtime_nodejs/lib:$NODE_PATH",
},
"runtimeArgs": [
"--expose-gc",
"--loader",
"ts-node/esm",
"--no-warnings",
]
}
]
}
```
#### Command Line Debugging
Use Node.js built-in debugger for command-line debugging:
```bash
# Enter the extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs
# Start debug mode
node --inspect-brk ./tests/bin/start
# Or use Chrome DevTools
node --inspect ./tests/bin/start
```
### Complete Development Process Summary
To help developers get started quickly, here is a complete Node.js extension development process summary:
```bash
# 1. Create extension project
tman create extension my_example_ext_nodejs --template default_extension_nodejs --template-data class_name_prefix=Example
# 2. Enter project directory
cd my_example_ext_nodejs
# 3. Install dependencies
tman install --standalone
# 4. Enter extension installation directory
cd .ten/app/ten_packages/extension/my_example_ext_nodejs
# 5. Install standalone mode dependencies
npm run standalone-install
# 6. Build project
npm run build
# 7. Run tests
./tests/bin/start
```
> **Working Directory Note**:
>
> - Extension source code is located in the `src/` folder at the project root directory
> - Actual build, test, and run operations are performed in the `.ten/app/ten_packages/extension/my_example_ext_nodejs/` directory
> - This design ensures extension independence and proper dependency management
### Development Best Practices
#### Async Programming
Node.js extensions fully utilize async programming patterns for better performance:
```typescript
// Recommended: Use async/await
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise {
try {
const result = await processCommand(cmd);
const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
cmdResult.setPropertyString("result", result);
tenEnv.returnResult(cmdResult);
} catch (error) {
const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
cmdResult.setPropertyString("error", error.message);
tenEnv.returnResult(cmdResult);
}
}
```
#### Error Handling
Implement comprehensive error handling mechanisms:
```typescript
async onCmd(tenEnv: TenEnv, cmd: Cmd): Promise {
try {
// Business logic processing
const result = await this.handleBusinessLogic(cmd);
const cmdResult = CmdResult.Create(StatusCode.OK, cmd);
cmdResult.setPropertyString("data", JSON.stringify(result));
tenEnv.returnResult(cmdResult);
} catch (error) {
console.error("Command processing error:", error);
const cmdResult = CmdResult.Create(StatusCode.ERROR, cmd);
cmdResult.setPropertyString("error", error.message);
tenEnv.returnResult(cmdResult);
}
}
```
---
## Development Summary
By following the complete development process provided in this guide, you can efficiently develop, test, and debug TEN extensions. Whether you choose C++, Go, Python, or Node.js, TEN Framework provides you with a complete toolchain and best practices to help you fully leverage the powerful features of TEN Framework and build high-performance, high-reliability extension applications.
Each language has its unique advantages and applicable scenarios:
- **C++**: Suitable for scenarios with extremely high performance requirements, such as real-time audio and video processing, high-frequency computation
- **Go**: Provides a balance between high performance and development efficiency, suitable for network services, concurrent processing
- **Python**: Has the highest development efficiency, particularly suitable for AI/ML applications, rapid prototyping
- **Node.js**: Provides modern Web development experience, suitable for frontend technology stack extensions, real-time applications
Please choose the most suitable development solution based on your specific needs and team technology stack. During the development process, it is recommended to fully utilize the debugging tools and testing frameworks provided by TEN Framework to ensure the quality and stability of your extensions.
---
### Getting Started Quick Start.Cn (`docs/getting-started/quick-start.cn.md`)
---
title: TEN Framework 快速入门指南
_portal_target: getting-started/quick-start.cn.md
---
## TEN Framework 快速入门指南
> 🎯 **目标**：5分钟内搭建开发环境并运行第一个 TEN 应用
## 系统要求
**支持的操作系统**：
- Linux (x64)
- Linux (arm64)
- macOS Intel (x64)
- macOS Apple Silicon (arm64)
- Windows (x64)
**必需的软件环境**：
- Python 3.10
- Go 1.20+
- Node.js / npm（用于安装和管理 JavaScript 依赖）
> 💡 **Windows 用户注意**：推荐使用 PowerShell 执行本指南中的命令。部分命令在 CMD 中可能不兼容。
## 第一步：检查环境
在开始之前，请确保你的系统已安装以下软件：
### Python 3.10
```bash
python3 --version
# 应显示: Python 3.10.x
```
> 💡 **重要**：TEN Framework 目前仅支持 Python 3.10。推荐使用 `pyenv` 或 `venv` 创建 Python 虚拟环境：
>
> **Linux / macOS：**
>
> ```bash
> # 使用 pyenv 安装和管理 Python 3.10（推荐）
> pyenv install 3.10.18
> pyenv local 3.10.18
>
> # 或使用 venv 创建虚拟环境
> python3.10 -m venv ~/ten-venv
> source ~/ten-venv/bin/activate
> ```
>
> **Windows：**
>
> 从 底部表格中下载 Windows installer，启动后安装 Python 3.10。
>
> 注意，点击Install Now前务必勾选 "Add Python to PATH"
>
> ```bash
> # Windows平台需要配置python3命令：
>
> # 首先，确认python的安装路径：
> where.exe python
> # 输出示例: C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe
>
> # 然后，以管理员身份打开 PowerShell，在python.exe同目录下创建symlink：
> New-Item -ItemType SymbolicLink -Path "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python3.exe" -Target "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe"
> # 请将上面的路径替换为 where.exe python 的实际输出路径
>
> # 验证：
> python3 --version
> ```
>
> ```powershell
> # 推荐安装后使用 venv 创建虚拟环境，在该环境中工作
> py -3.10 -m venv $env:USERPROFILE\ten-venv
> # 激活环境
> & "$env:USERPROFILE\ten-venv\Scripts\Activate.ps1"
>
> # 若有权限错误，关闭终端/IDE，右键选择“以管理员身份运行”重新打开
> # 或者改变执行策略来允许ps1脚本执行
> Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
> ```
### Go 1.20+
```bash
go version
# 应显示: go version go1.20 或更高版本
$env:CGO_ENABLED = "1"
# Windows下需要显式启用CGO
```
### Node.js / npm
```bash
node --version
npm --version
# 确保 node 和 npm 命令可用
```
### GCC (MinGW)
💡仅在windows上需要安装
``` bash
# 首先确定winget存在。
winget --version
# 若不存在则需要从 安装。
#（系统需要满足：Windows10 高于 1709 (Build 16299)，或者是Windows11）
#安装MinGW
winget install BrechtSanders.WinLibs.POSIX.MSVCRT
# 或查找后自行选择合适版本安装
winget search "mingw"
#检查安装
gcc --version
```
> 💡 **提示**：如果缺少上述环境，请先安装对应版本后再继续。
## 第二步：安装 TEN Manager (tman)
TEN Manager (tman) 是 TEN Framework 的命令行工具，用于创建项目、管理依赖和运行应用。
方式一：通过包管理器安装（推荐）
**Linux (Ubuntu/Debian):**
```bash
sudo add-apt-repository ppa:ten-framework/ten-framework
sudo apt update
sudo apt install tman
```
**macOS:**
```bash
brew install TEN-framework/ten-framework/tman
```
**Windows:**
```powershell
winget install TEN-framework.tman
```
方式二：通过安装脚本
**Linux / macOS：**
```bash
bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
```
**Windows：**
```powershell
irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.ps1 | iex
```
方式三：如果你已经克隆了仓库，使用仓库内的安装脚本
```bash
cd ten-framework
# Linux/MacOS
bash tools/tman/install_tman.sh
# Windows
& "C:\sw\ten-framework\tools\tman\install_tman.ps1"
```
> 💡 **提示**：如果系统中已经安装了 tman，安装脚本会询问是否重新安装/升级，输入 `y` 继续安装，输入 `n` 取消。
>
> **非交互式安装**（适用于自动化脚本或 CI 环境，Windows不可用）：
>
> ```bash
> # 远程安装
> yes y | bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
>
> # 本地安装
> yes y | bash tools/tman/install_tman.sh
> ```
**验证安装**：
```bash
tman --version
```
>💡 **提示**：如果出现类似
>
>- `tman: command not found`（Linux/macOS）
>- `无法将"tman"项识别为 cmdlet、函数、脚本文件或可运行程序的名称`（Windows）
>
>
> 的提示，请确保 tman 所在目录在你的 PATH 中：
>
> **Linux / macOS：**
>
> ```bash
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc # Linux
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.zshrc # macOS
> source ~/.bashrc # 或 source ~/.zshrc
> ```
>
> **Windows：**
>
> ```powershell
> # 检查 PATH 中是否包含 tman
> $env:Path -split ";" | Select-String "tman"
> # 如果不包含，请
> # 1. 通过 "系统属性 → 环境变量" 手动添加，然后重启终端以应用
> # 2. 运行以下命令来临时添加路径
> $env:PATH += ";$env:LOCALAPPDATA\tman"
> ```
## 第三步：创建并运行示例应用
### 1. 创建应用
```bash
# 创建一个新的 transcriber_demo 应用
tman install app transcriber_demo
cd transcriber_demo
```
### 2. 安装依赖
```bash
# 安装 TEN 包依赖
tman install
# 安装 Python 和 npm 包的依赖
tman run install_deps
```
> ⏱️ **预计时间**：1-2 分钟
### 3. 构建应用
```bash
tman run build
```
> ⏱️ **预计时间**：30 秒
### 4. 配置环境变量
在运行应用前，需要配置 ASR（语音识别）服务的密钥。当前示例使用 Azure ASR extension，你需要在 `transcriber_demo/.env` 文件中填写相关配置：
**Linux / macOS：**
```bash
# 创建 .env 文件
cat > .env << EOF
# Azure Speech Service 配置
AZURE_STT_KEY=your_azure_speech_api_key
AZURE_STT_REGION=your_azure_region # 例如：eastus
AZURE_STT_LANGUAGE=en-US # 根据你的音频语种或实时录音语种设置，如：zh-CN, ja-JP, ko-KR 等
EOF
```
**Windows（PowerShell）：**
```powershell
# 创建 .env 文件
@"
# Azure Speech Service 配置
AZURE_STT_KEY=your_azure_speech_api_key
AZURE_STT_REGION=your_azure_region
AZURE_STT_LANGUAGE=en-US
"@ | Out-File -Encoding utf8 .env
```
> 💡 **提示**：Windows 用户也可以直接用文本编辑器创建 `.env` 文件并填写上述配置。
>
> 💡 **提示**：如果你想使用其他 ASR extension（如 OpenAI Whisper、Google Speech 等），可以从云商店下载并替换，同样将相应的 API key 等环境变量配置在 `.env` 文件中。
### 5. 运行应用
```bash
tman run start
```
如果一切正常，你应该看到类似以下的输出：
```text
[web_audio_control_go] Web server started on port 8001
[audio_file_player_python] AudioFilePlayerExtension on_start
```
### 6. 体验 Demo
打开浏览器访问：
```text
http://localhost:8001
```
你应该能看到 Transcriber Demo 的 Web 界面，可以尝试：
- 点击麦克风按钮进行实时语音转录
- 上传音频文件进行转录
- 查看实时转录以及字幕结果
## 恭喜！🎉
你已经成功运行了第一个 TEN 应用！
### 了解应用架构
这个 `transcriber_demo` 应用展示了 TEN Framework 的多语言扩展能力，它由以下组件构成：
- **Go** - WebSocket 服务器扩展 (`web_audio_control_go`)
- **Python** - ASR 语音识别扩展 (`azure_asr_python`)
- **TypeScript** - VTT 字幕生成和音频录制扩展 (`vtt_nodejs`)
🎯 **你已经可以运行这些多语言插件了！**
### 下一步
现在你可以：
1. **从云商店探索和下载更多插件，设计和编排你的应用**
```bash
tman designer # 启动 TMAN Designer，在云商店中探索插件、下载插件并设计编排你的应用
```
2. **选择一个语言，开发自己的插件**
- 支持 Go、Python、TypeScript/JavaScript、C++ 等多种语言
- 查看 [TEN 扩展开发完整指南](https://theten.ai/cn/docs/ten_framework/development/how_to_develop_with_ext) 了解详情
## 进阶：开发和构建 C++ 插件
如果你想开发和使用 C++ 扩展，推荐安装 TEN 构建工具链（tgn）。以下是完整的步骤：
### 1. 安装 tgn 构建工具
tgn 是 TEN Framework 的 C/C++ 构建系统，基于 Google 的 GN。
方式一：一键安装（推荐）
**Linux / macOS：**
```bash
curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.sh | bash
```
**Windows：**
```powershell
irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.ps1 | iex
```
方式二：从克隆的仓库安装
**Linux / macOS：**
```bash
# 如果你已经克隆了 TEN Framework 仓库
cd ten-framework
bash tools/tgn/install_tgn.sh
```
**Windows：**
```powershell
# 如果你已经克隆了 TEN Framework 仓库
& ".\tools\tgn\install_tgn.ps1"
```
安装完成后，确认 tgn 已添加到 PATH：
**Linux / macOS：**
```bash
# 临时添加到当前会话
export PATH="/usr/local/ten_gn:$PATH"
# 或永久添加到 shell 配置（推荐）
echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.bashrc # Linux
echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.zshrc # macOS
source ~/.bashrc # 或 source ~/.zshrc
```
**Windows：**
```powershell
# 安装脚本会自动添加到用户 PATH，如需手动添加：
$env:Path += ";$env:LOCALAPPDATA\ten_gn"
```
验证安装：
```bash
tgn --help
```
### 2. 安装 C++ 扩展
以 WebRTC VAD（语音活动检测）扩展为例，从云商店安装 C++ 扩展：
```bash
cd transcriber_demo
tman install extension webrtc_vad_cpp
```
> 💡 **提示**：`webrtc_vad_cpp` 是一个用 C++ 实现的语音活动检测扩展，可以在实时语音识别场景中筛选出语音部分。
### 3. 编译 C++ 扩展
安装 C++ 扩展后，需要重新构建应用以编译 C++ 代码为动态库：
```bash
tman run build
```
> ⏱️ **预计时间**：首次编译 C++ 扩展可能需要 1-3 分钟，具体取决于你的机器性能。
### 4. 运行带有 VAD 功能的应用
```bash
tman run start_with_vad
```
如果一切正常，你应该看到：
```text
[web_audio_control_go] Web server started on port 8001
[vad] WebRTC VAD initialized with mode 2
[audio_file_player_python] AudioFilePlayerExtension on_start
```
现在打开浏览器访问 `http://localhost:8001`，进入麦克风实时转写页面，你会看到经过vad后的silence状态变化，当silence状态为true时，表示当前音频中没有语音。
### C++ 开发环境要求
开发和编译 C++ 扩展需要安装 C++ 编译器（gcc 或 clang）：
**Linux:**
```bash
# Ubuntu/Debian
sudo apt-get install gcc g++
# 或使用 clang
sudo apt-get install clang
```
**macOS:**
```bash
# 安装 Xcode Command Line Tools (包含 clang)
xcode-select --install
```
**Windows:**
```powershell
# 方式一：安装 Visual Studio Build Tools（推荐）
# 从 https://visualstudio.microsoft.com/visual-cpp-build-tools/ 下载安装
# 方式二：使用 winget 安装
winget install Microsoft.VisualStudio.2022.BuildTools --override "--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --passive"
# 💡注意：
# - 安装时选择 "使用 C++ 的桌面开发" 工作负载
# - 且务必勾选“适用于Windows的C++ Clang工具”
```
验证编译器安装：
```bash
# 检查 gcc（Linux/macOS/MSYS2）
gcc --version
g++ --version
# 或检查 clang（Linux/macOS）
clang --version
```
**Windows（使用 MSVC）：**
```powershell
# 在 "Developer PowerShell for VS" 中执行
clang-cl --version
```
### 常见问题（C++ 扩展）
1. tgn 命令找不到
确保已经执行安装脚本并将 tgn 添加到 PATH：
```bash
export PATH="/usr/local/ten_gn:$PATH"
```
2. 编译失败：找不到编译器
请参考上面的"C++ 开发环境要求"部分安装编译器。
### 了解更多
- [ten_gn 构建系统](https://github.com/TEN-framework/ten_gn) - TEN 的 C/C++ 构建工具
- [C++ 扩展开发指南](https://theten.ai/cn/docs/ten_framework/development/how_to_develop_with_ext) - 完整的 C++ 扩展开发文档
## 常见问题
### 1. macOS 上 Python 库加载失败
**问题**：运行应用时提示找不到 `libpython3.10.dylib`
**解决方案**：
```bash
export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
```
建议将这行添加到 `~/.zshrc` 或 `~/.bash_profile` 中。
### 2. tman 下载失败或速度很慢
**问题**：网络连接 GitHub 受限
**解决方案**：
- 手动下载：访问 [Releases 页面](https://github.com/TEN-framework/ten-framework/releases) 下载对应平台的 `tman` 二进制文件
### 3. 端口 8001 已被占用
**问题**：启动时提示端口冲突
**解决方案**：
- 查找占用端口的进程：
- Linux/macOS：`lsof -i :8001`
- Windows：`netstat -ano | findstr :8001`
- 杀掉该进程：
- Linux/macOS：`kill -9 `
- Windows：`taskkill /PID /F`
- 或修改应用配置文件（`transcriber_demo/ten_packages/extension/web_audio_control_go/property.json`）中的端口号
### 4. Go build 失败
**问题**：构建时提示 Go module 相关错误
**解决方案**：
```bash
# 设置代理
# Linux/macOS
export GOPROXY=https://goproxy.cn,direct
# Windows PowerShell
$env:GOPROXY = "https://goproxy.cn,direct"
# 清理 Go module 缓存
go clean -modcache
# 重新安装依赖
cd transcriber_demo
tman run build
```
### 5. Python 依赖安装失败
**问题**：pip 安装超时或失败
**解决方案**：使用国内镜像源
```bash
pip3 install --index-url https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt
```
### 6. Azure语音服务报错
**问题**：Azure语音服务相关错误，比如认证失败
**解决方案**：检查.env文件中的配置是否正确，确保 AZURE_STT_KEY 和 AZURE_STT_REGION 填写无误
### 7. Windows下权限问题
**问题**：Windows下在访问文件时报PermissionError
**解决方案**：右键点击PowerShell，选择“以管理员身份运行”
## 获取帮助
- **GitHub Issues**：
- **文档**：
- **贡献指南**：[contributing.md](../code-of-conduct/contributing.md)
---
### Getting Started Quick Start (`docs/getting-started/quick-start.md`)
---
title: TEN Framework Quick Start Guide
_portal_target: getting-started/quick-start.md
---
## TEN Framework Quick Start Guide
> 🎯 **Goal**: Set up your development environment and run your first TEN app in 5 minutes
## System Requirements
**Supported Operating Systems**:
- Linux (x64)
- Linux (arm64)
- macOS Intel (x64)
- macOS Apple Silicon (arm64)
- Windows (x64)
**Required Software**:
- Python 3.10
- Go 1.20+
- Node.js / npm (for managing JavaScript dependencies)
> 💡 **Windows users**: It's recommended to use PowerShell to run the commands in this guide. Some commands may not be compatible with CMD.
## Step 1: Check Your Environment
Before you begin, make sure the following software is installed on your system:
### Python 3.10
```bash
python3 --version
# Should display: Python 3.10.x
```
> 💡 **Important**: TEN Framework currently only supports Python 3.10. It's recommended to use `pyenv` or `venv` to manage your Python environment:
>
> **Linux / macOS:**
>
> ```bash
> # Install and manage Python 3.10 using pyenv (recommended)
> pyenv install 3.10.14
> pyenv local 3.10.14
>
> # Or create virtual environment using venv
> python3.10 -m venv ~/ten-venv
> source ~/ten-venv/bin/activate
> ```
>
> **Windows:**
>
> Download the Windows installer from the bottom table at and run it to install Python 3.10.
>
> Make sure to check "Add Python to PATH" before clicking Install Now.
>
> ```bash
> # On Windows, configure the python3 command:
>
> # First, find where python is installed:
> where.exe python
> # Example output: C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe
>
> # Then, open PowerShell as Administrator and create a symlink in the same directory:
> New-Item -ItemType SymbolicLink -Path "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python3.exe" -Target "C:\Users\YourName\AppData\Local\Programs\Python\Python310\python.exe"
> # Replace the paths above with the actual output from where.exe python
>
> # Verify:
> python3 --version
> ```
>
> ```powershell
> # It's recommended to create a venv after installation
> py -3.10 -m venv $env:USERPROFILE\ten-venv
> # Activate the environment
> & "$env:USERPROFILE\ten-venv\Scripts\Activate.ps1"
>
> # If you encounter a permission error, close the terminal/IDE, right-click and select "Run as Administrator" to reopen
> # Or change the execution policy to allow ps1 scripts
> Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
> ```
### Go 1.20+
```bash
go version
# Should display: go version go1.20 or higher
$env:CGO_ENABLED = "1"
# CGO must be explicitly enabled on Windows
```
### Node.js / npm
```bash
node --version
npm --version
# Ensure node and npm commands are available
```
### GCC (MinGW)
💡 Only required on Windows
```bash
# First, make sure winget is available.
winget --version
# If not, install it from
# (Requires Windows 10 version 1709 (Build 16299) or later, or Windows 11)
# Install MinGW
winget install BrechtSanders.WinLibs.POSIX.MSVCRT
# Or search and choose a suitable version to install
winget search "mingw"
# Verify installation
gcc --version
```
> 💡 **Tip**: If any of the above is missing, please install the required version before continuing.
## Step 2: Install TEN Manager (tman)
TEN Manager (tman) is the command-line tool for TEN Framework, used to create projects, manage dependencies, and run applications.
Option 1: Install via Package Manager (Recommended)
**Linux (Ubuntu/Debian):**
```bash
sudo add-apt-repository ppa:ten-framework/ten-framework
sudo apt update
sudo apt install tman
```
**macOS:**
```bash
brew install TEN-framework/ten-framework/tman
```
**Windows:**
```powershell
winget install TEN-framework.tman
```
Option 2: Install via Script
**Linux / macOS:**
```bash
bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
```
**Windows:**
```powershell
irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.ps1 | iex
```
Option 3: Install from cloned repository
```bash
cd ten-framework
# Linux/macOS
bash tools/tman/install_tman.sh
# Windows
& "C:\sw\ten-framework\tools\tman\install_tman.ps1"
```
> 💡 **Note**: If tman is already installed on your system, the installation script will ask whether you want to reinstall/upgrade it. Press `y` to continue or `n` to cancel.
>
> **Non-interactive Installation** (for automation scripts or CI environments, not available on Windows):
>
> ```bash
> # Remote installation
> yes y | bash <(curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tman/install_tman.sh)
>
> # Local installation
> yes y | bash tools/tman/install_tman.sh
> ```
**Verify Installation**:
```bash
tman --version
```
> 💡 **Tip**: If you see `tman: command not found` (Linux/macOS) or a similar unrecognized command error (Windows), make sure the tman directory is in your PATH:
>
> **Linux / macOS:**
>
> ```bash
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc # Linux
> echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.zshrc # macOS
> source ~/.bashrc # or source ~/.zshrc
> ```
>
> **Windows:**
>
> ```powershell
> # Check if tman is in PATH
> $env:Path -split ";" | Select-String "tman"
> # If not, either:
> # 1. Add it manually via "System Properties → Environment Variables", then restart your terminal
> # 2. Or temporarily add the path with:
> $env:PATH += ";$env:LOCALAPPDATA\tman"
> ```
## Step 3: Create and Run the Demo App
### 1. Create App
```bash
# Create a new transcriber_demo app
tman install app transcriber_demo
cd transcriber_demo
```
### 2. Install Dependencies
```bash
# Install TEN package dependencies
tman install
# Install Python and npm package dependencies
tman run install_deps
```
> ⏱️ **Estimated Time**: 1-2 minutes
### 3. Build the App
```bash
tman run build
```
> ⏱️ **Estimated Time**: 30 seconds
### 4. Configure Environment Variables
Before running the app, you need to configure the ASR (Automatic Speech Recognition) service credentials. The current example uses Azure ASR extension. You need to fill in the configuration in the `transcriber_demo/.env` file:
**Linux / macOS:**
```bash
# Create .env file
cat > .env << EOF
# Azure Speech Service Configuration
AZURE_STT_KEY=your_azure_speech_api_key
AZURE_STT_REGION=your_azure_region # e.g., eastus
AZURE_STT_LANGUAGE=en-US # Set according to your audio language or real-time recording language, e.g., zh-CN, ja-JP, ko-KR, etc.
EOF
```
**Windows (PowerShell):**
```powershell
# Create .env file
@"
# Azure Speech Service Configuration
AZURE_STT_KEY=your_azure_speech_api_key
AZURE_STT_REGION=your_azure_region
AZURE_STT_LANGUAGE=en-US
"@ | Out-File -Encoding utf8 .env
```
> 💡 **Tip**: Windows users can also create the `.env` file directly with a text editor and fill in the configuration above.
>
> 💡 **Tip**: If you want to use other ASR extensions (such as OpenAI Whisper, Google Speech, etc.), you can download and replace them from the cloud store. Similarly, configure the corresponding API keys and environment variables in the `.env` file.
### 5. Run the App
```bash
tman run start
```
If everything is working correctly, you should see output similar to:
```text
[web_audio_control_go] Web server started on port 8001
[audio_file_player_python] AudioFilePlayerExtension on_start
```
### 6. Experience the Demo
Open your browser and visit:
```text
http://localhost:8001
```
You should see the Transcriber Demo web interface. Try:
- Click the microphone button for real-time voice transcription
- Upload an audio file for transcription
- View real-time transcription and subtitle results
## Congratulations! 🎉
You've successfully run your first TEN application!
### Understanding the App Architecture
This `transcriber_demo` app showcases TEN Framework's multi-language extension capabilities, consisting of:
- **Go** - WebSocket server extension (`web_audio_control_go`)
- **Python** - ASR speech recognition extension (`azure_asr_python`)
- **TypeScript** - VTT subtitle generation and audio recording extension (`vtt_nodejs`)
🎯 **You can now run extensions in multiple languages!**
### Next Steps
Now you can:
1. **Explore and download more extensions from the cloud store, design and orchestrate your app**
```bash
tman designer # Launch TMAN Designer to explore extensions, download them, and design your app
```
2. **Choose a language and develop your own extension**
- Supports Go, Python, TypeScript/JavaScript, C++, and more
- Check out the [TEN Extension Development Guide](https://theten.ai/docs/ten_framework/development/how_to_develop_with_ext) for details
## Advanced: Developing and Building C++ Extensions
If you want to develop and use C++ extensions, you'll need to install the TEN build toolchain (tgn). Here's the complete process:
### 1. Install tgn Build Tool
tgn is TEN Framework's C/C++ build system, based on Google's GN.
Option 1: One-line Installation (Recommended)
**Linux / macOS:**
```bash
curl -fsSL https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.sh | bash
```
**Windows:**
```powershell
irm https://raw.githubusercontent.com/TEN-framework/ten-framework/main/tools/tgn/install_tgn.ps1 | iex
```
Option 2: Install from Cloned Repository
**Linux / macOS:**
```bash
# If you've already cloned the TEN Framework repository
cd ten-framework
bash tools/tgn/install_tgn.sh
```
**Windows:**
```powershell
# If you've already cloned the TEN Framework repository
& ".\tools\tgn\install_tgn.ps1"
```
After installation, ensure tgn is added to PATH:
**Linux / macOS:**
```bash
# Temporarily add to current session
export PATH="/usr/local/ten_gn:$PATH"
# Or permanently add to shell configuration (recommended)
echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.bashrc # Linux
echo 'export PATH="/usr/local/ten_gn:$PATH"' >> ~/.zshrc # macOS
source ~/.bashrc # or source ~/.zshrc
```
**Windows:**
```powershell
# The install script adds to user PATH automatically. To add manually:
$env:Path += ";$env:LOCALAPPDATA\ten_gn"
```
Verify installation:
```bash
tgn --help
```
### 2. Install C++ Extension
Using WebRTC VAD (Voice Activity Detection) extension as an example, install a C++ extension from the cloud store:
```bash
cd transcriber_demo
tman install extension webrtc_vad_cpp
```
> 💡 **Note**: `webrtc_vad_cpp` is a voice activity detection extension implemented in C++. It can filter out non-speech segments in real-time speech recognition scenarios.
### 3. Compile C++ Extension
After installing the C++ extension, rebuild the app to compile the C++ code into a dynamic library:
```bash
tman run build
```
> ⏱️ **Expected Time**: First-time compilation of C++ extensions may take 1-3 minutes, depending on your machine's performance.
### 4. Run App with VAD Functionality
```bash
tman run start_with_vad
```
If everything works correctly, you should see:
```text
[web_audio_control_go] Web server started on port 8001
[vad] WebRTC VAD initialized with mode 2
[audio_file_player_python] AudioFilePlayerExtension on_start
```
Now open your browser at `http://localhost:8001` and navigate to the microphone real-time transcription page. You'll see the silence state changes after VAD processing. When the silence state is true, it indicates there is no speech in the current audio.
### C++ Development Environment Requirements
Developing and compiling C++ extensions requires installing a C++ compiler (gcc or clang):
**Linux:**
```bash
# Ubuntu/Debian
sudo apt-get install gcc g++
# Or use clang
sudo apt-get install clang
```
**macOS:**
```bash
# Install Xcode Command Line Tools (includes clang)
xcode-select --install
```
**Windows:**
```powershell
# Option 1: Install Visual Studio Build Tools (recommended)
# Download from https://visualstudio.microsoft.com/visual-cpp-build-tools/
# Option 2: Install via winget
winget install Microsoft.VisualStudio.2022.BuildTools --override "--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --passive"
# 💡 Note:
# - Select the "Desktop development with C++" workload during installation
# - Make sure to check "C++ Clang tools for Windows"
```
Verify compiler installation:
```bash
# Check gcc (Linux/macOS/MSYS2)
gcc --version
g++ --version
# Or check clang (Linux/macOS)
clang --version
```
**Windows (MSVC):**
```powershell
# Run in "Developer PowerShell for VS"
clang-cl --version
```
### Troubleshooting (C++ Extensions)
1. tgn command not found
Ensure you've run the installation script and added tgn to PATH:
```bash
export PATH="/usr/local/ten_gn:$PATH"
```
2. Compilation failed: Compiler not found
Please refer to the "C++ Development Environment Requirements" section above to install the compiler.
### Learn More
- [ten_gn Build System](https://github.com/TEN-framework/ten_gn) - TEN's C/C++ build tool
- [C++ Extension Development Guide](https://theten.ai/docs/ten_framework/development/how_to_develop_with_ext) - Complete C++ extension development documentation
## Troubleshooting
### 1. Python Library Loading Failure on macOS
**Problem**: Error indicating `libpython3.10.dylib` cannot be found when running the app
**Solution**:
```bash
export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
```
Consider adding this line to your `~/.zshrc` or `~/.bash_profile`.
### 2. tman Download Failed or Slow
**Problem**: Network connection to GitHub is restricted
**Solution**:
- Manual download: Visit the [Releases page](https://github.com/TEN-framework/ten-framework/releases) to download the `tman` binary for your platform
### 3. Port 8001 Already in Use
**Problem**: Port conflict error when starting the app
**Solution**:
- Find the process using the port:
- Linux/macOS: `lsof -i :8001`
- Windows: `netstat -ano | findstr :8001`
- Kill the process:
- Linux/macOS: `kill -9 `
- Windows: `taskkill /PID /F`
- Or modify the port number in the app configuration file (`transcriber_demo/ten_packages/extension/web_audio_control_go/property.json`)
### 4. Go Build Failed
**Problem**: Go module related errors during build
**Solution**:
```
# Set proxy
# Linux/macOS
export GOPROXY=https://goproxy.cn,direct
# Windows PowerShell
$env:GOPROXY = "https://goproxy.cn,direct"
# Clean Go module cache
go clean -modcache
# Reinstall dependencies
cd transcriber_demo
tman run build
```
### 5. Python Dependencies Installation Failed
**Problem**: pip installation timeout or failure
**Solution**: Use a mirror source (for users in China)
```bash
pip3 install --index-url https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt
```
### 6. Azure Speech Service Error
**Problem**: Azure Speech Service related errors, such as authentication failure
**Solution**: Check that the configuration in your `.env` file is correct, and ensure `AZURE_STT_KEY` and `AZURE_STT_REGION` are filled in accurately
### 7. Permission Error on Windows
**Problem**: PermissionError when accessing files on Windows
**Solution**: Right-click PowerShell and select "Run as Administrator"
## Get Help
- **GitHub Issues**:
- **Documentation**:
- **Contributing Guide**: [contributing.md](../code-of-conduct/contributing.md)
---
### Getting Started Websocket Voice Assistant Quick Start.Cn (`docs/getting-started/websocket-voice-assistant-quick-start.cn.md`)
---
title: 基于WebSocket的语音助手快速入门
_portal_target: getting-started/websocket-voice-assistant-quick-start.cn.md
---
## 基于 WebSocket 的语音助手快速入门
在不到 10 分钟内构建并运行一个基于 WebSocket 的实时语音助手。
## 简介
本指南将帮助您快速运行一个完整的语音助手 Demo，支持实时语音对话和工具调用。该示例集成了：
| 组件 | 服务商 | 用途 |
| ---- | ---------- | ---------------- |
| ASR | Deepgram | 语音转文字 |
| LLM | OpenAI | 对话生成 |
| TTS | ElevenLabs | 文字转语音 |
| Tool | WeatherAPI | 天气查询（可选） |
## 环境要求
### 前置依赖
请先参考 [TEN Framework 快速入门指南](./quick-start.cn.md) 完成基础环境安装，确保以下命令通过检查：
```bash
tman check env
```
预期输出（至少 Python 和 Go 为 Ready）：
```text
✅ Operating System: Supported
✅ Python: Ready
✅ Go: Ready
```
### 额外工具
安装以下工具用于构建和运行：
#### Task（任务运行器）
**Linux / macOS：**
```bash
go install github.com/go-task/task/v3/cmd/task@latest
task --version # 验证安装
```
**Windows：**
```powershell
# 1. 使用winget安装（推荐）
# 从以下地址安装winget
# https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US
# 系统需要满足：Windows10 高于 1709 (Build 16299)，或者是Windows11
winget --version # 验证安装
winget install Task.Task
task --version # 验证安装
# 2. 使用scoop安装
irm get.scoop.sh | iex
scoop --version # 验证安装
scoop install task
task --version # 验证安装
```
#### Bun（JavaScript 包管理器）
**Linux / macOS：**
```bash
curl -fsSL https://bun.sh/install | bash
bun --version # 验证安装
```
**Windows：**
```powershell
powershell -c "irm bun.sh/install.ps1 | iex"
bun --version # 验证安装
```
#### uv（Python 包管理器）
**Linux / macOS：**
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv --version # 验证安装
```
**Windows：**
```powershell
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
uv --version # 验证安装
```
> 💡 安装后需要重新加载终端配置：
>
> - Linux/macOS：`source ~/.zshrc`
> - Windows：重新打开 PowerShell 终端
## 快速开始
### 1. 克隆仓库
```bash
git clone https://github.com/TEN-framework/ten-framework.git
cd ten-framework/ai_agents/agents/examples/websocket-example
```
### 2. 安装依赖
```bash
task install
```
> ⏱️ 预计耗时：2-3 分钟
该命令会自动完成：
- 安装 TEN 扩展包（45+ 个）
- 编译 Go 应用和 API 服务器
- 安装 Python 和前端依赖
### 3. 配置 API Keys
在 `ai_agents/.env` 文件中配置服务商密钥：
**Linux / macOS：**
```bash
cd ../../../ # 回到 ai_agents 目录
vim .env
```
**Windows：**
```powershell
cd ..\..\.. # 回到 ai_agents 目录
notepad .env
```
从.env.example复制所有内容后，填写以下配置：
```bash
# Deepgram - 语音识别
DEEPGRAM_API_KEY=your_deepgram_api_key
# OpenAI - 大语言模型
OPENAI_API_KEY=your_openai_api_key
OPENAI_MODEL=your_openai_model
# ElevenLabs - 语音合成
ELEVENLABS_TTS_KEY=your_elevenlabs_api_key
# WeatherAPI - 天气工具（可选）
WEATHERAPI_API_KEY=your_weatherapi_key
```
### 4. 启动服务
```bash
cd agents/examples/websocket-example
task run
```
服务启动后，自动运行：
- **前端 UI**：
- **API 服务器**：
- **TMAN Designer**：
### 5. 与助手对话
打开浏览器访问 [http://localhost:3000](http://localhost:3000)，点击麦克风按钮开始语音对话。
尝试以下对话：
- "你好，你是谁？"
- "今天北京的天气怎么样？"（需配置 WeatherAPI）
- "给我讲个笑话"
**🎉 恭喜！** 您已成功运行 TEN Agent 语音助手。
## 进阶配置
### 更换扩展
TEN Framework 支持灵活替换 ASR、LLM、TTS 等扩展。
#### 方式 1：使用 TMAN Designer（推荐）
访问 [http://localhost:49483](http://localhost:49483)，通过可视化界面：
1. 点击扩展节点查看详情
2. 选择替代扩展
3. 配置参数
4. 保存并重启
#### 方式 2：编辑配置文件
编辑 `tenapp/property.json`，修改 `predefined_graphs` 中的扩展配置。
**示例：更换 ASR 为 Azure**
```json
{
"type": "extension",
"name": "stt",
"addon": "azure_asr_python",
"extension_group": "stt",
"property": {
"params": {
"key": "${env:AZURE_STT_KEY}",
"region": "${env:AZURE_STT_REGION}",
"language": "en-US"
}
}
}
```
**示例：更换 TTS 为 Minimax**
```json
{
"type": "extension",
"name": "tts",
"addon": "minimax_tts_websocket_python",
"extension_group": "tts",
"property": {
"params": {
"api_key": "${env:MINIMAX_TTS_API_KEY|}",
"group_id": "${env:MINIMAX_TTS_GROUP_ID|}",
"model": "speech-02-turbo",
"audio_setting": {
"sample_rate": 16000
},
"voice_setting": {
"voice_id": "female-shaonv"
}
}
}
}
```
修改后重启服务（`Ctrl+C` 然后 `task run`）。
### 配置监控
使用 Grafana 监控应用的 Metrics 和 Logs。
#### 1. 启动监控栈
**Linux / macOS：**
```bash
cd tools/grafana-monitoring
docker compose -f docker-compose.push.yml up -d
```
**Windows：**
```powershell
cd tools\grafana-monitoring
docker compose -f docker-compose.push.yml up -d
# 若docker未安装，请通过https://apps.microsoft.com/detail/xp8cbj40xlbwkx 安装，并重启电脑后，启动docker desktop
```
启动的服务：
- **Grafana**（端口 3001）- 可视化界面
- **Prometheus**（端口 9091）- Metrics 存储
- **Loki**（端口 3100）- Logs 存储
- **OpenTelemetry Collector**（端口 4317/4318）- OTLP 数据接收
#### 2. 配置应用遥测
编辑 `agents/examples/websocket-example/tenapp/property.json`：
**配置 Logs 输出**
在 `ten.log.handlers` 数组中添加：
```json
{
"matchers": [{ "level": "info" }],
"formatter": { "type": "json", "colored": false },
"emitter": {
"type": "otlp",
"config": {
"endpoint": "http://localhost:4317",
"service_name": "websocket-example"
}
}
}
```
#### 配置 Metrics 输出
在 `ten.services` 中添加：
```json
{
"telemetry": {
"enabled": true,
"metrics": {
"enabled": true,
"exporter": {
"type": "otlp",
"config": {
"endpoint": "http://localhost:4317",
"service_name": "websocket-example"
}
}
}
}
}
```
#### 3. 重启并访问 Grafana
```bash
cd ../../../agents/examples/websocket-example
task run
```
访问 [http://localhost:3001](http://localhost:3001)（用户名/密码：`admin/admin`）
#### 查看 Metrics
进入 `Dashboards` → **"TEN Framework"** 查看：
- 扩展启动耗时
- Cmd 处理延迟
- 消息队列等待时间
#### 查看 Logs
进入 `Explore` → 选择 `Loki`，使用查询：
- `{service_name="websocket-example"}` - 所有日志
- `{service_name="websocket-example", ten_extension_name="stt"}` - ASR 日志
- `{service_name="websocket-example", ten_extension_name="llm"}` - LLM 日志
- `{service_name="websocket-example", ten_extension_name="tts"}` - TTS 日志
> 💡 使用 `ten_extension_name` 标签精确筛选扩展日志。
停止监控服务：
**Linux / macOS：**
```bash
cd tools/grafana-monitoring
docker compose -f docker-compose.push.yml down
```
**Windows：**
```powershell
cd tools\grafana-monitoring
docker compose -f docker-compose.push.yml down
```
## 常见问题
前端依赖安装失败
如果 `bun install` 报错版本不存在，切换到 npm 官方源：
```bash
cd frontend
echo "registry=https://registry.npmjs.org/" > .npmrc
rm bun.lock
bun install
```
macOS Python 库加载失败
设置 Python 动态库路径：
```bash
export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
# 添加到 ~/.zshrc 永久生效
```
Windows Python 库加载失败
确保 Python 3.10 安装路径在系统 PATH 中：
```powershell
# 检查 Python 路径
python -c "import sys; print(sys.prefix)"
# 将 Python DLL 目录添加到 PATH（根据实际路径调整）
$env:Path += ";C:\Python310;C:\Python310\DLLs"
```
如果安装 Python 时未勾选 "Add Python to PATH"，可以重新运行安装程序选择 "Modify" 来添加。
端口被占用
查找并终止占用端口的进程：
**Linux / macOS：**
```bash
lsof -i :3000 # 或 :8080
kill -9
```
**Windows：**
```powershell
netstat -ano | findstr :3000 # 或 :8080
taskkill /PID /F
```
Go module相关错误
```bash
# 设置代理
# Linux/macOS
export GOPROXY=https://goproxy.cn,direct
# Windows PowerShell
$env:GOPROXY = "https://goproxy.cn,direct"
# 清理 Go module 缓存
go clean -modcache
# 重新安装依赖
cd ai_agents/agents/examples/websocket-example
task install
```
## 下一步
跟随这些指南，将您的语音 AI 应用带入真实世界。
- **[扩展开发指南](https://theten.ai/cn/docs/ten_framework/development/how_to_develop_with_ext)** - 开发自定义扩展
- **[更多示例](https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples)** - 探索其他 Agent 示例
- **[GitHub Issues](https://github.com/TEN-framework/ten-framework/issues)** - 报告问题或请求功能
- **[文档中心](https://theten.ai/cn/docs)** - 完整的 TEN Framework 文档
---
### Getting Started Websocket Voice Assistant Quick Start (`docs/getting-started/websocket-voice-assistant-quick-start.md`)
---
title: WebSocket Voice Assistant Quickstart
_portal_target: getting-started/websocket-voice-assistant-quick-start.md
---
## WebSocket Voice Assistant Quickstart
Build and run a real-time WebSocket voice assistant in under 10 minutes.
## Overview
This guide helps you run a complete voice assistant demo locally. It supports real-time voice conversations and tool calling. The demo includes:
| Component | Vendor | Purpose |
| --------- | ---------- | ------------------------- |
| ASR | Deepgram | Speech-to-text |
| LLM | OpenAI | Response generation |
| TTS | ElevenLabs | Text-to-speech |
| Tool | WeatherAPI | Weather lookup (optional) |
## Requirements
### Prerequisites
Follow the [TEN Framework Quick Start](./quick-start.md) to install the base environment, then make sure the check below passes:
```bash
tman check env
```
Expected output (at least Python and Go are Ready):
```text
✅ Operating System: Supported
✅ Python: Ready
✅ Go: Ready
```
### Extra tools
Install the following tools for build & run:
#### Task (task runner)
**Linux / macOS:**
```bash
go install github.com/go-task/task/v3/cmd/task@latest
task --version # verify
```
**Windows:**
```powershell
# Option 1: Install via winget (recommended)
# Install winget from: https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=US
# Requires Windows 10 version 1709 (Build 16299) or higher, or Windows 11
winget --version # verify winget
winget install Task.Task
task --version # verify
# Option 2: Install via scoop
irm get.scoop.sh | iex
scoop --version # verify scoop
scoop install task
task --version # verify
```
#### Bun (JavaScript package manager)
**Linux / macOS:**
```bash
curl -fsSL https://bun.sh/install | bash
bun --version # verify
```
**Windows:**
```powershell
powershell -c "irm bun.sh/install.ps1 | iex"
bun --version # verify
```
#### uv (Python package manager)
**Linux / macOS:**
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv --version # verify
```
**Windows:**
```powershell
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
uv --version # verify
```
> 💡 After installation, reload your shell config:
>
> - Linux/macOS: `source ~/.zshrc`
> - Windows: reopen your PowerShell terminal
## Quickstart
### 1. Clone the repo
```bash
git clone https://github.com/TEN-framework/ten-framework.git
cd ten-framework/ai_agents/agents/examples/websocket-example
```
### 2. Install dependencies
```bash
task install
```
> ⏱️ Estimated time: 2–3 minutes
This command will:
- Install TEN extension packages (45+)
- Build the Go app and API server
- Install Python and frontend dependencies
### 3. Configure API keys
Configure vendor keys in `ai_agents/.env`:
**Linux / macOS:**
```bash
cd ../../../ # back to ai_agents
vim .env
```
**Windows:**
```powershell
cd ..\..\.. # back to ai_agents
notepad .env
```
Copy all content from .env.example, then fill in:
```bash
# Deepgram - ASR
DEEPGRAM_API_KEY=your_deepgram_api_key
# OpenAI - LLM
OPENAI_API_KEY=your_openai_api_key
OPENAI_MODEL=your_openai_model
# ElevenLabs - TTS
ELEVENLABS_TTS_KEY=your_elevenlabs_api_key
# WeatherAPI - Tool (optional)
WEATHERAPI_API_KEY=your_weatherapi_key
```
### 4. Start services
```bash
cd agents/examples/websocket-example
task run
```
After startup, the following services are available:
- **Frontend UI**:
- **API server**:
- **TMAN Designer**:
### 5. Talk to your assistant
Open [http://localhost:3000](http://localhost:3000), click the microphone button, and start speaking.
Try:
- "Hello, who are you?"
- "What's the weather in Beijing today?" (requires WeatherAPI)
- "Tell me a joke."
**Congratulations!** Your TEN Agent voice assistant is running.
## Advanced
### Swap extensions
TEN Framework lets you swap ASR/LLM/TTS extensions flexibly.
#### Option 1: Use TMAN Designer (recommended)
Open [http://localhost:49483](http://localhost:49483), then in the visual editor:
1. Click an extension node to inspect it
2. Choose a replacement extension
3. Configure parameters
4. Save and restart
#### Option 2: Edit configuration file
Edit `tenapp/property.json` and modify the extension configs under `predefined_graphs`.
##### Example: switch ASR to Azure
```json
{
"type": "extension",
"name": "stt",
"addon": "azure_asr_python",
"extension_group": "stt",
"property": {
"params": {
"key": "${env:AZURE_STT_KEY}",
"region": "${env:AZURE_STT_REGION}",
"language": "en-US"
}
}
}
```
##### Example: switch TTS to Minimax
```json
{
"type": "extension",
"name": "tts",
"addon": "minimax_tts_websocket_python",
"extension_group": "tts",
"property": {
"params": {
"api_key": "${env:MINIMAX_TTS_API_KEY|}",
"group_id": "${env:MINIMAX_TTS_GROUP_ID|}",
"model": "speech-02-turbo",
"audio_setting": {
"sample_rate": 16000
},
"voice_setting": {
"voice_id": "female-shaonv"
}
}
}
}
```
Restart after changes (`Ctrl+C` then `task run`).
### Monitoring (Grafana)
Use Grafana to observe application metrics and logs.
#### 1. Start monitoring stack
**Linux / macOS:**
```bash
cd tools/grafana-monitoring
docker-compose -f docker-compose.push.yml up -d
```
**Windows:**
```powershell
cd tools\grafana-monitoring
docker compose -f docker-compose.push.yml up -d
# If Docker is not installed, install it from https://apps.microsoft.com/detail/xp8cbj40xlbwkx, restart your PC, then launch Docker Desktop
```
Services:
- **Grafana** (port 3001) - UI
- **Prometheus** (port 9091) - metrics store
- **Loki** (port 3100) - log store
- **OpenTelemetry Collector** (ports 4317/4318) - OTLP receiver
#### 2. Configure app telemetry
Edit `agents/examples/websocket-example/tenapp/property.json`:
#### Logs (export)
Add to the `ten.log.handlers` array:
```json
{
"matchers": [{ "level": "info" }],
"formatter": { "type": "json", "colored": false },
"emitter": {
"type": "otlp",
"config": {
"endpoint": "http://localhost:4317",
"service_name": "websocket-example"
}
}
}
```
#### Metrics (export)
Add to `ten.services`:
```json
{
"telemetry": {
"enabled": true,
"metrics": {
"enabled": true,
"exporter": {
"type": "otlp",
"config": {
"endpoint": "http://localhost:4317",
"service_name": "websocket-example"
}
}
}
}
}
```
#### 3. Restart and open Grafana
```bash
cd ../../../agents/examples/websocket-example
task run
```
Open [http://localhost:3001](http://localhost:3001) (username/password: `admin/admin`).
#### Metrics (Dashboards)
Go to `Dashboards` → **"TEN Framework"** to view:
- Extension startup latency
- Cmd processing latency
- Message queue wait time
#### Logs (Explore/Loki)
Go to `Explore` → select `Loki`, then query:
- `{service_name="websocket-example"}` - all logs
- `{service_name="websocket-example", ten_extension_name="stt"}` - ASR logs
- `{service_name="websocket-example", ten_extension_name="llm"}` - LLM logs
- `{service_name="websocket-example", ten_extension_name="tts"}` - TTS logs
> 💡 Use the `ten_extension_name` label to filter logs per extension.
Stop the monitoring stack:
**Linux / macOS:**
```bash
cd tools/grafana-monitoring
docker-compose -f docker-compose.push.yml down
```
**Windows:**
```powershell
cd tools\grafana-monitoring
docker compose -f docker-compose.push.yml down
```
## Troubleshooting
Frontend dependency install fails
If `bun install` fails due to missing versions, switch to the official npm registry:
```bash
cd frontend
echo "registry=https://registry.npmjs.org/" > .npmrc
rm bun.lock
bun install
```
macOS Python library loading fails
Set the Python dynamic library path:
```bash
export DYLD_LIBRARY_PATH=/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib:$DYLD_LIBRARY_PATH
# Add to ~/.zshrc for persistence
```
Windows Python library loading fails
Make sure the Python 3.10 installation path is in your system PATH:
```powershell
# Check Python path
python -c "import sys; print(sys.prefix)"
# Add Python DLL directory to PATH (adjust to your actual path)
$env:Path += ";C:\Python310;C:\Python310\DLLs"
```
If you didn't check "Add Python to PATH" during installation, re-run the installer and choose "Modify" to add it.
Port already in use
Find and kill the process:
**Linux / macOS:**
```bash
lsof -i :3000 # or :8080
kill -9
```
**Windows:**
```powershell
netstat -ano | findstr :3000 # or :8080
taskkill /PID /F
```
Go module related errors
```bash
# Set proxy
# Linux/macOS
export GOPROXY=https://goproxy.cn,direct
# Windows PowerShell
$env:GOPROXY = "https://goproxy.cn,direct"
# Clean Go module cache
go clean -modcache
# Reinstall dependencies
cd ai_agents/agents/examples/websocket-example
task install
```
## Next steps
- **[Extension development guide](https://theten.ai/docs/ten_framework/development/how_to_develop_with_ext)** - build custom extensions
- **[More examples](https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples)** - explore other agent demos
- **[GitHub Issues](https://github.com/TEN-framework/ten-framework/issues)** - report bugs or request features
- **[Documentation hub](https://theten.ai/docs)** - full TEN Framework documentation
---
### Ai L1 01 Setup (`docs/ai/L1/01_setup.md`)
# 01 Setup
> Environment setup, local development, and quick commands for TEN Framework AI Agents.
## Prerequisites
| Requirement | Version / Notes |
| ----------------- | ------------------------------------------------------------ |
| Docker + Compose | Required for container-based development |
| Node.js | LTS v18+ on host; container has Node 22 |
| API Keys | Agora App ID, LLM provider, ASR provider, TTS provider |
| Hardware | 2+ CPU cores, 4 GB RAM minimum |
## Docker Container
```bash
cd /home/ubuntu/ten-framework/ai_agents
docker compose up -d
docker ps | grep ten_agent_dev # Verify running
```
Container image: `ghcr.io/ten-framework/ten_agent_build:0.7.14`
**Permissions:** if your user is not in the `docker` group, prefix every
`docker` / `docker exec` / `docker compose` command in this guide with `sudo`.
A bare `docker ps` failing with `permission denied while trying to connect to
the docker API at unix:///var/run/docker.sock` is the diagnostic — switch to
`sudo docker ps` and the rest works.
## Environment Variables
**Single .env file**: `ai_agents/.env` — the ONLY source of environment config.
| Variable | Purpose | Required |
| ---------------------------- | ---------------------------- | -------- |
| `AGORA_APP_ID` | Agora RTC app identifier | Yes |
| `AGORA_APP_CERTIFICATE` | Agora RTC certificate | No |
| `OPENAI_API_KEY` | LLM provider | Yes |
| `OPENAI_MODEL` | Model name (e.g., `gpt-4o`) | Yes |
| `DEEPGRAM_API_KEY` | ASR provider | Yes |
| `ELEVENLABS_TTS_KEY` | TTS provider | Yes |
| `LOG_PATH` | Container log directory bind | Yes |
| `LOG_STDOUT` | Worker log visibility | Yes (`true`) |
| `SERVER_PORT` | API server port | Yes (`8080`) |
| `WORKERS_MAX` | Max concurrent sessions | Yes (`100`) |
| `WORKER_QUIT_TIMEOUT_SECONDS`| Worker idle timeout | Yes (`60`) |
See `.env.example` for the complete list. Extensions may require different provider
keys (Deepgram, ElevenLabs, Cartesia, Azure, AWS, Rime, etc.) — check extension
README files.
## Install and Run
```bash
# 1. Install Python dependencies (NOT persisted across container restarts)
docker exec ten_agent_dev bash -c \
"cd /app/agents/examples//tenapp && \
bash scripts/install_python_deps.sh"
# 2. Build and install (5-8 minutes first time)
docker exec ten_agent_dev bash -c \
"cd /app/agents/examples/ && task install"
# 3. Start everything (API server + playground + TMAN Designer)
docker exec -d ten_agent_dev bash -c \
"cd /app/agents/examples/ && \
task run > /tmp/task_run.log 2>&1"
```
Typical choices:
- `voice-assistant` for standard vendor iteration and demo graphs
- `voice-assistant-advanced` for generated multi-graph setups
**CRITICAL**: Always use `task run` to start — never run `./bin/api` directly.
## Ports
| Port | Service |
| ----- | ---------------- |
| 8080 | Go API server |
| 3000 | Playground (Next.js) |
| 49483 | TMAN Designer |
## Health Checks
```bash
curl -s http://localhost:8080/health
# {"code":"0","data":null,"msg":"ok"}
curl -s http://localhost:8080/graphs | jq -r '.data[].name'
# voice_assistant, voice_assistant_oracle, etc.
```
## Restart Procedures
| What Changed | Container? | Server? | Frontend? |
| ------------------------------- | ---------- | ----------------- | ----------------- |
| `property.json` (graphs added) | No | Nuclear restart | Nuclear restart |
| `property.json` (config only) | No | No | No |
| `.env` file | Yes | Yes | No |
| Python extension code | No | Yes | No |
| Go server code | No | Yes + `task install` | No |
**Nuclear restart** (safest after graph changes):
```bash
sudo docker exec ten_agent_dev bash -c \
"pkill -9 -f 'bin/api'; pkill -9 -f bun; pkill -9 -f node; \
pkill -9 -f next-server; pkill -9 -f tman"
sudo docker exec ten_agent_dev bash -c "rm -f /app/playground/.next/dev/lock"
sleep 30
sudo docker exec -d ten_agent_dev bash -c \
"cd /app/agents/examples/ && task run > /tmp/task_run.log 2>&1"
```
**After container restart**: always reinstall Python deps, then `task run`.
**After .env changes**: `docker compose down && docker compose up -d`, reinstall deps, `task run`.
## Logs
```bash
# All logs (inside container)
docker exec ten_agent_dev tail -f /tmp/task_run.log
# Filter by extension or channel
docker exec ten_agent_dev tail -f /tmp/task_run.log | grep --line-buffered "deepgram"
# If grep reports binary/noisy output, inspect printable strings instead
docker exec ten_agent_dev bash -c "strings /tmp/task_run.log | tail -n 80"
```
## Related Deep Dives
- [Deployment](L2/deployment.md) — Docker Compose, Cloudflare tunnel, Nginx, Grafana monitoring
---
### Ai L1 02 Architecture (`docs/ai/L1/02_architecture.md`)
# 02 Architecture
> System design overview: extensions, graphs, connections, and the server-worker model.
## TEN Ecosystem
| Component | Purpose |
| --------------------- | ------------------------------------------------------ |
| TEN Framework | Core runtime (C/C++, Go, Python, Node.js bindings) |
| TEN Agent Examples | Pre-built agent configurations (this repo's `ai_agents/`) |
| TEN VAD | Low-latency voice activity detection |
| TEN Turn Detection | Full-duplex dialogue management |
| TEN Portal | Documentation and blog site |
## Extension System
Extensions are modular components that process data — ASR, TTS, LLM, tools, RTC, avatars, etc.
Each extension has a lifecycle:
```
on_init() → on_start() → [process messages] → on_stop() → on_deinit()
```
Every extension contains:
| File | Purpose |
| ----------------- | ------------------------------------------ |
| `addon.py` | Registration via `@register_addon_as_extension` |
| `extension.py` | Main logic, inherits from a base class |
| `manifest.json` | Metadata, dependencies, API interface |
| `property.json` | Default configuration values |
**Base classes** (in `ten_ai_base/interface/ten_ai_base/`):
| Base Class | Use For |
| ----------------------------- | ----------------- |
| `AsyncASRBaseExtension` | Speech-to-text |
| `AsyncTTS2BaseExtension` | Text-to-speech |
| `AsyncLLMBaseExtension` | Chat completion |
| `AsyncLLMToolBaseExtension` | LLM function tools|
| `AsyncExtension` | Generic / custom |
## Graph-Based Configuration
Agents are assembled by defining **graphs** in `property.json`. A graph specifies
which extensions run (nodes) and how data flows between them (connections).
```json
{
"ten": {
"predefined_graphs": [{
"name": "voice_assistant",
"auto_start": true,
"graph": {
"nodes": [
{"type": "extension", "name": "stt", "addon": "deepgram_asr_python", "property": {}},
{"type": "extension", "name": "llm", "addon": "openai_llm2_python", "property": {}},
{"type": "extension", "name": "tts", "addon": "elevenlabs_tts2_python", "property": {}}
],
"connections": [...]
}
}]
}
}
```
## Connection Types
| Type | Payload | Example |
| ------------- | -------------------- | -------------------------------------------- |
| `cmd` | Named commands | `tool_register`, `on_user_joined`, `flush` |
| `data` | Named data messages | `asr_result`, `text_data`, `tts_text_input` |
| `audio_frame` | PCM audio streams | `pcm_frame` (16-bit, mono, 16/24/48 kHz) |
| `video_frame` | Video streams | Raw video frames for vision/avatar |
## RTC-First Design
TEN uses Agora RTC (Real-Time Communication) as the default transport, not WebSockets.
| Aspect | RTC (default) | WebSocket |
| ---------------- | -------------------------------- | -------------------------- |
| Latency | 50-150ms (UDP-based) | Higher (TCP-based) |
| Codec support | Opus, VP8, VP9, AV1 | Raw PCM only |
| Bandwidth adapt | Built-in adaptation + FEC | Manual implementation |
| Use case | Real-time voice/video | Signaling, configuration |
WebSockets are used for signaling and configuration; RTC handles the media path.
## Server-Worker Model
```
┌─────────────────┐ ┌──────────────────┐
│ Go HTTP Server │────▶│ Worker Process │ (one per session)
│ (port 8080) │ │ (tman run start) │
│ │ │ │
│ /start → spawn │ │ Loads graph from │
│ /stop → kill │ │ property.json │
│ /ping → keep │ │ Runs extensions │
└─────────────────┘ └──────────────────┘
```
- **POST /start** spawns a worker process for a channel/session
- **POST /stop** terminates the worker
- **POST /ping** keeps the session alive (if timeout != -1)
## Property Injection
When `/start` is called, the server auto-injects dynamic values into the graph:
- `channel_name` → injected into every node that has a `"channel"` property
- `remote_stream_id`, `bot_stream_id`, `token` → injected via `startPropMap`
- `req.Properties[extensionName]` → merged into specific node properties
This is future-proof: any new extension with a "channel" property automatically
receives the dynamic channel value without code changes.
## Component Diagram
```
Client (Browser/Mobile)
│
▼
┌──────────────┐
│ Playground │ Next.js frontend (port 3000)
│ (UI) │
└──────┬───────┘
│ REST API
▼
┌──────────────┐ ┌──────────────────────────────────┐
│ Go Server │──spawn─▶│ Worker Process │
│ (port 8080) │ │ ┌─────┐ ┌─────┐ ┌─────┐ │
│ │ │ │ ASR │─▶│ LLM │─▶│ TTS │ │
│ │ │ └──┬──┘ └─────┘ └──┬──┘ │
└──────────────┘ │ │ │ │
│ ┌──┴───────────────────┴──┐ │
│ │ Agora RTC │ │
│ └─────────────────────────┘ │
└──────────────────────────────────┘
```
## Related Deep Dives
- [Server Architecture](L2/server_architecture.md) — Go server internals, property injection pipeline
- [Graph Configuration](L2/graph_configuration.md) — Node schema, connection wiring, parallel routing
---
### Ai L1 03 Code Map (`docs/ai/L1/03_code_map.md`)
# 03 Code Map
> Directory tree, module responsibilities, and key file locations.
## Top-Level Structure
All AI agent development happens inside `ai_agents/`:
```
ai_agents/
├── agents/
│ ├── ten_packages/
│ │ ├── extension/ # 90+ extensions (ASR, TTS, LLM, tools, avatar)
│ │ └── system/ # Core runtime packages
│ │ ├── ten_ai_base/ # Base classes and API interface definitions
│ │ ├── ten_runtime_python/
│ │ └── ten_runtime_go/
│ ├── examples/ # 24+ example agent configurations
│ │ ├── voice-assistant/
│ │ ├── voice-assistant-advanced/
│ │ ├── voice-assistant-realtime/
│ │ ├── voice-assistant-video/
│ │ ├── doodler/
│ │ └── ...
│ ├── integration_tests/ # Test frameworks
│ │ ├── asr_guarder/ # ASR integration tests
│ │ └── tts_guarder/ # TTS integration tests
│ └── scripts/ # Build and packaging scripts
├── server/ # Go API server
│ ├── main.go
│ └── internal/
│ ├── http_server.go # REST endpoints, property injection
│ └── config.go # Parameter mapping (startPropMap)
├── playground/ # Next.js frontend UI (port 3000)
│ └── src/ # React components
├── esp32-client/ # ESP32 hardware client
├── Taskfile.yml # Root-level build/test tasks
├── docker-compose.yml # Container config
├── .env # Environment variables (single source)
└── .env.example # Template with all variables
```
Other repo-root directories: `core/` (C runtime), `packages/` (example/core extensions),
`docs/` (framework docs), `tools/` (Grafana monitoring, profilers).
## Extension Categories
| Category | Count | Examples |
| --------- | ----- | ----------------------------------------------------------- |
| ASR | 10+ | `deepgram_asr_python`, `azure_asr_python`, `aws_asr_python` |
| TTS | 15+ | `deepgram_tts`, `elevenlabs_tts2_python`, `cartesia_tts` |
| LLM | 9+ | `openai_llm2_python`, `anthropic_llm2_python`, `gemini_llm2_python`, `bedrock_llm_python` |
| Avatar | 5+ | `heygen_avatar_python`, `anam_avatar_python` |
| Tools | 8+ | `bingsearch_tool_python`, `vision_tool_python` |
| Transport | 3+ | `agora_rtc`, `websocket_server`, `http_server_python` |
| Other | 10+ | `message_collector2`, `ten_vad_python`, `mcp_client_python` |
## Extension File Structure
Every extension follows this layout:
| File | Purpose |
| ------------------ | ---------------------------------------------- |
| `__init__.py` | Package marker |
| `addon.py` | `@register_addon_as_extension` registration |
| `extension.py` | Main logic, inherits from base class |
| `config.py` | Pydantic config model (optional but common) |
| `manifest.json` | Metadata, dependencies, API interface imports |
| `property.json` | Default config values with `${env:VAR}` syntax |
| `requirements.txt` | Python dependencies |
| `README.md` | Usage documentation (often multilingual) |
| `tests/` | Standalone tests with `bin/start` entry point |
## Base Classes
Located in example tenapp directories under `ten_packages/system/ten_ai_base/interface/ten_ai_base/`:
| File | Class | Purpose |
| ---------- | ---------------------------- | -------------------------- |
| `asr.py` | `AsyncASRBaseExtension` | Speech recognition |
| `tts.py` | `AsyncTTSBaseExtension` | Text-to-speech (basic) |
| `tts2.py` | `AsyncTTS2BaseExtension` | Text-to-speech (advanced) |
| `llm.py` | `AsyncLLMBaseExtension` | Language model completion |
| `llm2.py` | `AsyncLLM2BaseExtension` | Language model v2 |
| `llm_tool.py` | `AsyncLLMToolBaseExtension` | LLM function calling tools |
| `mllm.py` | `AsyncMLLMBaseExtension` | Multimodal LLM |
## API Interface Definitions
Standard interfaces in `ten_ai_base/api/`:
| File | Defines |
| ----------------------- | --------------------------------- |
| `asr-interface.json` | ASR data/cmd/audio_frame schemas |
| `tts-interface.json` | TTS data/cmd/audio_frame schemas |
| `llm-interface.json` | LLM data/cmd schemas |
| `mllm-interface.json` | Multimodal LLM schemas |
Extensions reference these via `manifest.json`:
```json
{"api": {"interface": [{"import_uri": "../../system/ten_ai_base/api/tts-interface.json"}]}}
```
## Key Files Quick Reference
| When working on... | Look at |
| -------------------------- | -------------------------------------------------- |
| New extension | Similar extension in `agents/ten_packages/extension/` |
| API interface changes | `ten_ai_base/api/*.json` |
| Graph configuration | `agents/examples/*/tenapp/property.json` |
| Server endpoints | `server/internal/http_server.go` |
| Build/test tasks | `Taskfile.yml` (root) and per-example |
| Test setup | `agents/ten_packages/extension/*/tests/bin/start` |
## Related Deep Dives
- [Extension Development](L2/extension_development.md) — Full creation guide with base class details
---
### Ai L1 04 Conventions (`docs/ai/L1/04_conventions.md`)
# 04 Conventions
> Coding patterns, naming, configuration, and formatting standards.
## Naming Conventions
| Item | Pattern | Example |
| --------------- | -------------------------------------- | -------------------------- |
| Extension dir | `__python` | `deepgram_asr_python` |
| Addon name | Same as directory name | `deepgram_asr_python` |
| Example dir | `voice-assistant-` | `voice-assistant-realtime` |
| Config class | `Config(BaseModel)` | `DeepgramTTSConfig` |
| Client class | `Client` | `DeepgramTTSClient` |
## Addon Registration
Every extension must register via decorator in `addon.py`:
```python
from ten_runtime import Addon, register_addon_as_extension, TenEnv
@register_addon_as_extension("deepgram_asr_python")
class DeepgramASRExtensionAddon(Addon):
def on_create_instance(self, ten: TenEnv, name: str, context) -> None:
ten.on_create_instance_done(DeepgramASRExtension(name), context)
```
The decorator name **must match** the `addon` field in `property.json` graph nodes.
## Base Class Selection
| Need | Base Class | Key Methods You Implement Most Often |
| ----------------------- | ----------------------------- | ------------------------------------------------------------- |
| Speech-to-text | `AsyncASRBaseExtension` | `vendor()`, `start_connection()`, `stop_connection()`, `send_audio()`, `finalize()`, `is_connected()`, `input_audio_sample_rate()`, `buffer_strategy()` |
| Text-to-speech (HTTP) | `AsyncTTS2HttpExtension` | `create_config()`, `create_client()`, `vendor()`, `synthesize_audio_sample_rate()` |
| Text-to-speech (WS) | `AsyncTTS2BaseExtension` | `vendor()`, `request_tts()`, `synthesize_audio_sample_rate()` |
| Chat completion | `AsyncLLMBaseExtension` | `on_call_chat_completion()`, `on_data_chat_completion()` |
| LLM function tool | `AsyncLLMToolBaseExtension` | `get_tool_metadata()`, `run_tool()` |
| Generic / custom | `AsyncExtension` | `on_cmd()`, `on_data()`, etc. |
## Pydantic Configuration
Extensions use Pydantic models for config validation:
```python
from pydantic import BaseModel, Field
class DeepgramTTSConfig(BaseModel):
params: dict[str, Any] = Field(default_factory=dict)
```
Config is loaded from property.json in `on_init()`:
```python
config_json, _ = await ten_env.get_property_to_json("")
self.config = DeepgramTTSConfig(**json.loads(config_json))
```
## Environment Variable Syntax
In `property.json`, reference env vars:
| Syntax | Behavior |
| --------------------- | --------------------------------------- |
| `${env:VAR_NAME}` | Required — error if missing |
| `${env:VAR_NAME\|}` | Optional — empty string if missing |
| `${env:VAR_NAME\|default}` | Optional — uses default if missing |
```json
{"params": {"api_key": "${env:DEEPGRAM_API_KEY}", "region": "${env:AZURE_REGION|}"}}
```
## Params Dict Pattern
Extensions using HTTP/WebSocket services store all config in a `params` dictionary:
1. **Store** `api_key` inside `params` dict in property.json and config
2. **Extract** for authentication headers in the client constructor
3. **Keep** convenience fields if needed, but preserve vendor params in config
4. **Strip** secrets only when creating the HTTP payload or WS query
```python
# In client constructor — extract for auth
self.api_key = config.params.get("api_key", "")
self.headers = {"Authorization": f"Bearer {self.api_key}"}
# In request method — strip before sending
vendor_params = {**self.config.params}
vendor_params.pop("api_key", None)
```
## Sensitive Data Logging
Implement `to_str()` to encrypt sensitive fields before logging:
```python
def to_str(self, sensitive_handling: bool = True) -> str:
if not sensitive_handling:
return f"{self}"
config = copy.deepcopy(self)
if config.params and "api_key" in config.params:
config.params["api_key"] = utils.encrypt(config.params["api_key"])
return f"{config}"
```
## Logging
- Use `ten_env.log_info()`, `ten_env.log_warn()`, `ten_env.log_error()`, `ten_env.log_debug()`
- Categories: `LOG_CATEGORY_KEY_POINT` (lifecycle events), `LOG_CATEGORY_VENDOR` (vendor status)
- Config log messages must use the `config:` prefix — QA automation matches on `(?:config|LOG_CATEGORY_KEY_POINT|KEYPOINT config):` to validate vendor config output:
```python
ten_env.log_info(f"config: {self.config.to_str(sensitive_handling=True)}", category=LOG_CATEGORY_KEY_POINT)
```
- All output goes to `/tmp/task_run.log` inside the container
## Error Classification
Use `ModuleErrorCode` consistently:
| Situation | Severity | Notes |
| --------- | -------- | ----- |
| Missing or invalid required config | `FATAL_ERROR` | Extension should not continue |
| Auth failure (`401`, `403`, invalid API key) | `FATAL_ERROR` | Include `vendor_info` when available |
| Transient vendor disconnect / timeout | `NON_FATAL_ERROR` | Retry or reconnect if supported |
| Retry ceiling reached | `FATAL_ERROR` | Surface terminal failure clearly |
| Invalid user input for a single request | `NON_FATAL_ERROR` | Request fails, extension keeps running |
Provider-originated failures should include `vendor_info` when possible:
```python
ModuleErrorVendorInfo(vendor=self.vendor(), code="401", message="Unauthorized")
```
Before changing corner-case behavior such as empty TTS input, check the guarder
tests first. Repo test expectations override generic assumptions.
## Import Convention
```python
# Correct (v0.11+)
from ten_runtime import Addon, register_addon_as_extension, TenEnv
# Wrong (old v0.8.x — will not work)
from ten import Addon
```
## Formatting
- **Black** formatter with `--line-length 80`
- Run: `task format` (from `ai_agents/`)
- Check: `task check`
- Excludes: `third_party/`, `http_server_python/`, `ten_packages/system`
> **Run before every push.** CI runs both `task check` (black format check)
> and `task lint` (pylint — any warning is fatal). Either failing blocks
> merge. If you commit from a host shell (outside the container), the
> pre-commit hook does not run and the PR fails CI.
> The reliable habit is:
>
> ```bash
> sudo docker exec ten_agent_dev bash -c \
> "cd /app && task format && task check && task lint"
> ```
>
> Run this after any change under `agents/ten_packages/extension/` or
> `agents/examples/` and before `git push`. `task lint` is strict — even
> a single `W0611: unused-import` warning fails CI.
## Commit Messages
A separate CI job — **`Lint Commit Messages / commitlint`** — runs
[`@commitlint/config-conventional`](https://github.com/conventional-changelog/commitlint)
against every commit in the PR (config: `.github/configs/commitlint.config.mjs`).
**There is no local commit-msg hook**, so a bad message is not caught until CI —
get it right when you write the commit.
Rules that actually fail CI (config-conventional defaults):
- **Conventional header:** `type(scope): subject` — e.g. `fix(tts): stream segments immediately`.
- **Type** must be one of: `feat`, `fix`, `docs`, `chore`, `test`, `refactor`,
`perf`, `build`, `ci`, `style`, `revert`.
- **Subject:** lowercase start, no trailing period.
- **Header line ≤ 100 chars.**
- **Body lines ≤ 100 chars each** — this is the one that bites: a long
paragraph written with `git commit -m "..."` is a single >100-char line and
fails `body-max-line-length`. **Hard-wrap the body** (≤72–80 cols is safe),
e.g. write it in a file and use `git commit -F msg.txt`.
- **Blank line between subject and body** (`body-leading-blank`).
Quick self-check before pushing (every commit added in the PR):
```bash
mb=$(git merge-base origin/main HEAD)
for c in $(git rev-list $mb..HEAD); do
git log -1 --format='%b' $c | awk -v c=$c 'length>100{print c": body line >100"}'
done
```
If you already pushed a long-body commit, reword/squash and force-push
(`git push --force-with-lease`) — commitlint checks the whole PR, so a new
commit on top will not clear the failure.
## Design Principles
- **YAGNI**: Only implement what is needed now, not what might be needed later
- **KISS**: Prefer simple solutions; three similar lines > premature abstraction
- **No git-ignored files**: Never modify auto-generated files (manifest-lock.json, out/, .ten/, bin/)
## Related Deep Dives
- [Extension Development](L2/extension_development.md) — Full creation guide with implementation walkthroughs
---
### Ai L1 05 Workflows (`docs/ai/L1/05_workflows.md`)
# 05 Workflows
> Step-by-step guides for common development tasks.
## Create a New TTS / ASR / LLM Extension
**Fastest path**: Copy a similar extension and adapt it.
| Type | Copy From | Base Class |
| ----------- | -------------------------- | --------------------------- |
| TTS (HTTP) | `rime_http_tts` | `AsyncTTS2HttpExtension` |
| TTS (WS) | `deepgram_tts` | `AsyncTTS2BaseExtension` |
| ASR | `deepgram_asr_python` | `AsyncASRBaseExtension` |
| LLM | `openai_llm2_python` | `AsyncLLMBaseExtension` |
```bash
cp -r agents/ten_packages/extension/ \
agents/ten_packages/extension/my_vendor_tts
```
Then:
1. Rename addon decorator, class names, `manifest.json` `name` field
2. Keep vendor config in `params`; extract secrets for auth and forward the rest to the vendor request layer
3. Implement the abstract methods for your vendor API
4. Create `tests/configs/` with required config files (see below)
5. Run guarder tests: `task tts-guarder-test EXTENSION=my_vendor_tts`
6. Run formatter: `task format`
**Typical required test config files** for TTS: `property_basic_audio_setting1.json`,
`property_basic_audio_setting2.json`, `property_dump.json`, `property_miss_required.json`,
`property_invalid.json`
Some extensions also keep `property.json` as a default valid config, but the exact
set depends on the extension template and guarder harness in use.
Optional / vendor-dependent TTS configs:
- `property_subtitle_alignment.json` if the vendor emits word/segment timestamps
**Required test config files** for ASR: `property_en.json`, `property_zh.json`,
`property_invalid.json`, `property_dump.json`
For full walkthrough with code and guarder expectations, see
[Extension Development](L2/extension_development.md) and [Testing](L2/testing.md).
### New ASR/TTS Extension Checklist
Use one recent strong example as the main template, not just any extension of
the same type.
For every new ASR extension or ASR behavior change, review against
[ASR Plugin Design and Review Guide](L2/asr_plugin_design_review.md). Its
`MUST` rules and PR checklist are the merge baseline for lifecycle, connection
state, buffering, finalize, reconnect, result shape, metrics, and tests.
| Type | Strong Template | Why |
| ---- | --------------- | --- |
| TTS (WS) | `deepgram_tts` | Better lifecycle and standalone test coverage |
| ASR | `openai_asr_python` | Stronger result-shape testing than thinner templates |
Minimum end-to-end steps:
1. Verify the vendor wire contract first: endpoint, event names, payload encoding, finalize primitive.
2. Copy a recent extension with the same transport shape.
3. Implement config loading, secret redaction, and `config:` logging.
4. Implement error classification and `vendor_info`.
5. Add standalone tests before running guarders.
6. Run guarders sequentially, not in parallel, inside the same container.
7. Add README and example graph wiring only after the extension tests are green.
## Add Extension to a Graph
1. **Add node** to `predefined_graphs[].graph.nodes[]` in the example's `tenapp/property.json`:
```json
{"type": "extension", "name": "my_tts", "addon": "my_tts_python",
"extension_group": "tts",
"property": {"params": {"api_key": "${env:MY_API_KEY}"}}}
```
2. **Add connections** — wire data flow between extensions:
```json
{"extension": "my_tts",
"data": [{"name": "tts_text_input", "source": [{"extension": "main"}]}],
"audio_frame": [{"name": "pcm_frame", "dest": [{"extension": "agora_rtc"}]}]}
```
3. **Add dependency** to example `tenapp/manifest.json` (local extensions use path):
```json
{"path": "../../../ten_packages/extension/my_tts_python"}
```
4. **Install** with `task install` (not bare `tman install` — the latter can wipe `bin/worker`).
5. **Nuclear restart** — required when graphs are added/removed. See
[Setup → Restart Procedures](01_setup.md#restart-procedures) for the exact
command sequence.
See [Graph Configuration](L2/graph_configuration.md) for connection types and
routing patterns. When adding a TTS vendor, copy a confirmed working voice graph and
preserve its routing model. Some shipped graphs rely on explicit message connections,
while others keep part of the orchestration inside `main_control`.
**For complex multi-graph setups** (A/B testing vendors, avatar variants), use
`rebuild_property.py` instead of hand-editing. See
[Generating property.json](L2/graph_configuration.md#generating-propertyjson-with-rebuild_propertypy).
## Customize the Main Extension
The "main" extension orchestrates agent behavior (greetings, tool routing, interruption).
Three implementation variants exist:
| Variant | File | Use Case |
| -------------------- | --------------------- | ------------------------------- |
| Python Cascade | `main_cascade_python` | ASR → LLM → TTS pipeline |
| Python Realtime V2V | `main_realtime_python`| OpenAI Realtime API (voice-to-voice) |
| Node.js Cascade | `main_nodejs` | TypeScript implementation |
Modify `on_data()` to change event routing, `on_cmd()` for tool handling.
## Run Tests
```bash
# Standalone (single extension, with install)
sudo docker exec ten_agent_dev bash -c \
"cd /app && task test-extension EXTENSION=agents/ten_packages/extension/"
# Integration guarder (ASR or TTS — run sequentially, not in parallel)
sudo docker exec ten_agent_dev bash -c \
"cd /app && task tts-guarder-test EXTENSION= CONFIG_DIR=tests/configs"
```
Other variants (`task test`, `task test-extension-no-install`, `task asr-guarder-test`) and
debugging guidance live in [Testing](L2/testing.md).
## Restart After Changes
| What Changed | Action |
| ------------------------------- | ---------------------------------------------------- |
| `property.json` (graphs added) | Nuclear restart (kill all, remove lock, task run) |
| `property.json` (config only) | No restart needed (loaded per session) |
| `.env` | `docker compose down && docker compose up -d` + deps |
| Python code | Restart server only |
| Go code | `task install` then restart server |
| Container restart | Reinstall Python deps, then `task run` |
## Build and Install
Always prefer `task install` over bare `tman install`:
```bash
sudo docker exec ten_agent_dev bash -c \
"cd /app/agents/examples/ && task install"
```
Python-deps-only and `tman install` variants are in
[Setup → Install and Run](01_setup.md#install-and-run).
## Update Extension Code in Running Container
See [Operations and Restarts](L2/operations_restarts.md) for `docker cp` syntax,
symlink verification, and restart steps.
## Pre-Commit / Pre-Push Checks
Run **all** of these before pushing — each maps to a CI job that blocks merge,
and there are no local hooks for most of them, so CI is the first thing that
catches a miss. For an extension change under
`agents/ten_packages/extension//`:
```bash
# 1. Format + static checks (CI: task check, task lint)
sudo docker exec ten_agent_dev bash -c \
"cd /app && task format && task check && task lint"
# 2. Standalone tests for the extension
sudo docker exec ten_agent_dev bash -c \
"cd /app && task test-extension EXTENSION=agents/ten_packages/extension/"
# 3. Guarder tests against the real vendor API (TTS shown; ASR uses asr-guarder-test)
sudo docker exec ten_agent_dev bash -c \
"cd /app && task tts-guarder-test EXTENSION= CONFIG_DIR=tests/configs"
```
Then verify **commit messages** — a separate CI job
(`Lint Commit Messages / commitlint`) runs `@commitlint/config-conventional`
on every commit in the PR, with **no local hook**. The usual trip-up is
`body-max-line-length` (≤100 chars per body line) — hard-wrap the body and use
`git commit -F msg.txt`. Full rules and a self-check script:
[Conventions → Commit Messages](04_conventions.md#commit-messages).
Checklist (all must be green before `git push`):
| Gate | Command / rule | CI job |
| ---- | -------------- | ------ |
| Black format | `task check` (`--line-length 80`) | format check |
| Pylint | `task lint` (any warning is fatal) | lint |
| Standalone tests | `task test-extension EXTENSION=...` | per-extension tests |
| Guarder tests | `task tts-guarder-test` / `task asr-guarder-test` | guarder |
| Commit messages | conventional + body lines ≤100 | commitlint |
> Note: a stale `.ten/` left in an extension dir (e.g. from a prior
> `tman install --standalone`) makes `task check` report spurious reformatting
> and breaks the next standalone install with a symlink error. Remove it
> (`rm -rf /.ten`) before re-running.
## Related Deep Dives
- [Extension Development](L2/extension_development.md) — Full extension creation with code examples
- [Graph Configuration](L2/graph_configuration.md) — Connection wiring and routing patterns
- [Testing](L2/testing.md) — Test infrastructure, guarder tests, debugging
- [Operations and Restarts](L2/operations_restarts.md) — Full restart procedures, recovery
---
### Ai L1 06 Interfaces (`docs/ai/L1/06_interfaces.md`)
# 06 Interfaces
> REST API contracts, graph connection schemas, and base class abstract methods.
## REST API Endpoints
The Go server (`server/internal/http_server.go`) exposes:
| Endpoint | Method | Purpose | Key Fields |
| -------------------- | ------ | ------------------------------------ | --------------------------------- |
| `/health` | GET | Health check | Returns `{"code":"0"}` |
| `/graphs` | GET | List available graphs | Returns `data[].name` |
| `/start` | POST | Start agent session | `graph_name`, `channel_name` |
| `/stop` | POST | Stop agent session | `channel_name` |
| `/ping` | POST | Keep session alive | `channel_name` |
| `/list` | GET | List active sessions | Returns worker list |
| `/token/generate` | POST | Generate Agora RTC token | `channel_name`, `uid` |
### POST /start Request Body
```json
{
"request_id": "uuid",
"channel_name": "test_channel",
"user_uid": 176573,
"graph_name": "voice_assistant",
"properties": {
"llm": {"model": "gpt-4o-mini"}
},
"timeout": 60
}
```
- `properties` — per-extension overrides merged into graph node properties
- `timeout` — seconds of inactivity before auto-stop (-1 = never)
## Graph Connection Types
Connections in `property.json` define data flow between extensions:
### Command Connections (`cmd`)
```json
{"extension": "main", "cmd": [
{"names": ["tool_register"], "dest": [{"extension": "llm"}]},
{"names": ["on_user_joined"], "source": [{"extension": "agora_rtc"}]}
]}
```
Common commands: `tool_register`, `on_user_joined`, `flush`, `chat_completion_call`,
`update_configs`
### Data Connections (`data`)
```json
{"extension": "llm", "data": [
{"name": "text_data", "source": [{"extension": "main"}]},
{"name": "text_data", "dest": [{"extension": "tts"}]}
]}
```
Common data: `asr_result`, `text_data`, `tts_text_input`, `tts_audio_start`,
`tts_audio_end`, `error`
### Audio Frame Connections (`audio_frame`)
```json
{"extension": "agora_rtc", "audio_frame": [
{"name": "pcm_frame", "dest": [{"extension": "stt"}]}
]}
```
### Video Frame Connections (`video_frame`)
```json
{"extension": "agora_rtc", "video_frame": [
{"name": "video_frame", "dest": [{"extension": "vision"}]}
]}
```
## Base Class Abstract Methods
### ASR (`AsyncASRBaseExtension`)
| Method | Returns | Purpose |
| ----------------------------------- | --------- | -------------------------------- |
| `vendor()` | `str` | Vendor name (e.g., "deepgram") |
| `start_connection()` | `None` | Connect to ASR service |
| `stop_connection()` | `None` | Disconnect |
| `send_audio(frame, session_id)` | `bool` | Send audio frame to service |
| `finalize(session_id)` | `None` | Drain pending audio |
| `is_connected()` | `bool` | Connection status check |
| `input_audio_sample_rate()` | `int` | Expected sample rate (e.g., 16000)|
| `buffer_strategy()` | `ASRBufferConfig` | Audio buffering behavior |
**Output helpers**: `send_asr_result()`, `send_asr_error()`, `send_asr_finalize_end()`,
`send_connect_delay_metrics()`, `send_vendor_metrics()`
### TTS (`AsyncTTS2BaseExtension`)
| Method | Returns | Purpose |
| -------------------------------- | -------- | ------------------------------------ |
| `vendor()` | `str` | Vendor name (e.g., "elevenlabs") |
| `request_tts(tts_text_input)` | `None` | Process queued text and emit audio via helpers |
| `cancel_tts()` | `None` | Handle flush/cancellation when overridden |
| `synthesize_audio_sample_rate()` | `int` | Output sample rate (e.g., 24000) |
| `synthesize_audio_channels()` | `int` | Channel count (default: 1) |
| `synthesize_audio_sample_width()`| `int` | Bytes per sample (default: 2) |
**Output helpers**: `send_tts_audio_data()`, `send_tts_audio_start()`, `send_tts_audio_end()`,
`send_tts_error()`, `send_tts_ttfb_metrics()`, `send_tts_text_result()`
**State machine**: QUEUED → PROCESSING → FINALIZING → COMPLETED (per request)
### TTS HTTP (`AsyncTTS2HttpExtension`)
| Method | Returns | Purpose |
| -------------------------------- | ------------------- | ------------------------------------ |
| `create_config(config_json_str)` | `AsyncTTS2HttpConfig` | Parse vendor config from property JSON |
| `create_client(config, ten_env)` | `AsyncTTS2HttpClient` | Build the vendor HTTP client |
| `vendor()` | `str` | Vendor name |
| `synthesize_audio_sample_rate()` | `int` | Output sample rate |
### LLM (`AsyncLLMBaseExtension`)
| Method | Returns | Purpose |
| ------------------------------- | ------- | -------------------------------- |
| `on_call_chat_completion()` | varies | Handle sync command requests |
| `on_data_chat_completion()` | varies | Handle stream-based data input |
| `on_tools_update(tool_metadata)`| `None` | Handle new tool registration |
**Tool flow**: Extensions register tools via `CMD_TOOL_REGISTER` → LLM stores in
`available_tools` → LLM calls tools during completion → results returned.
## Manifest API Interface
Extensions declare their API interface in `manifest.json`:
```json
{
"api": {
"interface": [
{"import_uri": "../../system/ten_ai_base/api/tts-interface.json"}
],
"property": {
"properties": {
"params": {
"type": "object",
"properties": {
"api_key": {"type": "string"},
"model": {"type": "string"},
"sample_rate": {"type": "int32"}
}
}
}
}
}
}
```
Interface JSON files define the standard cmd/data/audio_frame schemas for each extension type.
## Canonical Payload Notes
- ASR `asr_result`: keep `session_id` inside `metadata`, never as a top-level
field. Locked-interim vs final lives at `metadata.asr_info.locked`.
- Error payloads: include `vendor_info` (vendor name + vendor code + vendor
message) whenever the failure originated upstream.
- ASR `connect_delay` metrics are valid before `session_id` is known; other
metrics should wait.
See [Extension Development](L2/extension_development.md) for full example payloads.
## Portal References
- [API Events Reference](https://github.com/TEN-framework/portal/blob/main/content/docs/ten_agent_examples/api-reference/events.mdx) [EXTERNAL] — REST event types and payloads
- [API Schemas Reference](https://github.com/TEN-framework/portal/blob/main/content/docs/ten_agent_examples/api-reference/schemas.mdx) [EXTERNAL] — request/response schema definitions
## Related Deep Dives
- [Extension Development](L2/extension_development.md) — Implementing abstract methods
- [Server Architecture](L2/server_architecture.md) — Endpoint handlers and property injection
- [Graph Configuration](L2/graph_configuration.md) — Full connection wiring examples
---
### Ai L1 07 Gotchas (`docs/ai/L1/07_gotchas.md`)
# 07 Gotchas
> Critical pitfalls, tribal knowledge, and troubleshooting.
## CRITICAL: Property Getters Return Tuples
All `get_property_*()` methods return `(value, error_or_none)`, not the raw value.
```python
# WRONG — causes TypeError
threshold = await ten_env.get_property_float("threshold")
if threshold > 0.5: # TypeError: '>' not supported between 'float' and 'tuple'
# CORRECT — extract from tuple
threshold_result = await ten_env.get_property_float("threshold")
threshold = threshold_result[0] if isinstance(threshold_result, tuple) else threshold_result
```
This applies to `get_property_string()`, `get_property_int()`, `get_property_float()`,
`get_property_bool()`. Always extract `[0]`.
## CRITICAL: Signal Handlers Forbidden
Extensions run in worker threads. Signal handlers only work in the main thread.
```python
# WRONG — raises ValueError: signal only works in main thread
signal.signal(signal.SIGTERM, handler)
atexit.register(cleanup)
# CORRECT — use extension lifecycle
async def on_stop(self, ten_env):
await self.cleanup()
```
## CRITICAL: Always Use `task run`
Never start the server with `./bin/api` or `./bin/worker` directly.
`task run` sets the correct PYTHONPATH and starts all services together
(API server + playground + TMAN Designer).
## Zombie Worker Processes
Worker processes (`bin/worker`) can survive container and server restarts.
Always check for and kill zombies before restarting.
## .env Loaded at Container Startup Only
Editing `.env` while the container is running has **no effect**. You must
`docker compose down && docker compose up -d`, then reinstall Python deps.
## Next.js Lock File
After crashes, `.next/dev/lock` becomes stale, preventing restart. Delete it
and do a full restart. See [Operations and Restarts](L2/operations_restarts.md).
## Python Deps Not Persisted
Python dependencies are lost on container restart. Always reinstall after
`docker compose down && up`.
## tman install Can Wipe bin/worker
Running `tman install` when system dependencies have newer versions replaces
the runtime packages and **deletes `bin/worker`**. Use `task install` (full
rebuild) instead of bare `tman install`. Signs: Worker fails with
`bin/worker: No such file or directory` in logs.
## tman Install Creates Symlinks
Never manually `ln -s` for extensions. Use `tman install` which resolves
dependencies and creates correct links. If a symlink is missing after
`tman install`, create it manually as a fallback.
## docker cp Creates Nested Directories
When using `docker cp` to update extension code, trailing slashes create
nested directories. Use `docker cp ./ext/. container:/path/ext/` syntax.
Signs: `ModuleNotFoundError: No module named 'ten_packages.extension.X'`.
## Vendor "PCM" Is Not Always PCM16
The TEN `pcm_frame` contract expects **signed 16-bit (PCM16) mono**. A vendor's
`response_format: "pcm"` may be something else — e.g. Mistral/Voxtral streams
headerless **float32 LE** at 24 kHz. Sending those bytes through unconverted
produces noise. Convert in the client before yielding `RESPONSE`, and make
`synthesize_audio_sample_rate()` return the rate you actually emit (24000 for
Voxtral), not a default.
Two practical traps when converting a stream:
- **Don't split a sample across chunk boundaries.** float32 is 4 bytes; buffer a
trailing partial sample between `aiter_bytes()` chunks (see `Float32ToPcm16` in
`mistral_tts_python`).
- **Clamp before scaling** to `[-1, 1]` and map NaN to silence so one corrupt
sample can't crash the stream.
Requesting raw `pcm` (vs. a container like `wav`/`mp3`) also lowers
time-to-first-audio — there is no header to buffer before the first samples.
## Audio Routing: Split at Source Only
When routing audio to multiple destinations, the split must happen at the
source node (e.g., `agora_rtc`), not at intermediate nodes. Splitting from
intermediate nodes can cause crashes.
## Frontend Caches Graph List
The playground caches the `/graphs` API response. When adding or removing
graphs from `property.json`, a full restart is required — simple server
restart is not enough.
## Guarder Symlink Drift
Guarder apps create their own `ten_packages/extension/*` symlinks. These can
point at dead host paths and break imports after worktree changes or manual
copies.
Typical symptom:
`ModuleNotFoundError: No module named 'ten_packages.extension.'`
Reliable fix inside the container:
```bash
ln -sf /app/agents/ten_packages/extension/ \
/app/agents/integration_tests//ten_packages/extension/
```
## Run Guarders Sequentially
Do not run ASR and TTS guarders in parallel in the same container. Their build
scripts may collide on shared temp paths and fail with errors like:
`/tmp/test: Text file busy`
Run one guarder to completion, then start the other.
## Production Graph Picker 403
If the playground requests `/api/dev/v1/graphs` in production, the graph list
may be empty even though `/api/agents/graphs` is healthy. Check
`NEXT_PUBLIC_EDIT_GRAPH_MODE`; production deployments should typically set it to
`false`.
## Manifest Module Name Must Match
The `name` field in extension `manifest.json` must exactly match the `addon`
field used in graph nodes in `property.json`. Mismatches cause silent failures.
## next-server Holds Port 3000
Killing `node` and `bun` is not enough — `next-server` is a separate process
that holds port 3000. If port 3000 is occupied, Next.js silently starts on
3001+ which isn't Docker-exposed, making the frontend appear down.
## CI `lint` Runs Both Black and Pylint
The `lint` CI check on every PR runs two task targets:
- `task check` — `black --check` (formatting)
- `task lint` — `pylint` (any warning is fatal, including W0611 unused-import)
If you commit from a host shell rather than inside the container, the repo's
pre-commit hook does not run and either failure will surface only in CI.
Before every push:
```bash
sudo docker exec ten_agent_dev bash -c \
"cd /app && task format && task check && task lint"
```
Common foot-gun: leaving an unused import after a refactor. Pylint is strict
and a single W0611 fails the build. See [Formatting](04_conventions.md#formatting).
## Docker Permission Denied
`docker ps` fails with `permission denied while trying to connect to the
docker API at unix:///var/run/docker.sock` when the current user is not in
the `docker` group. The fix is `sudo docker …` for every invocation, not
adding the user to the group mid-session. Examples in this doc set are
inconsistent (some show `docker exec`, others `sudo docker exec`) — on a
host where the bare form fails, treat every example as needing `sudo`.
## Apple Silicon Docker
Docker containers may need Rosetta for x86 images on Apple Silicon Macs.
Enable in Docker Desktop: Settings > General > Use Rosetta.
## Windows Line Endings
Before cloning on Windows: `git config --global core.autocrlf false`
## Related Deep Dives
- [Operations and Restarts](L2/operations_restarts.md) — Full restart procedures, port debugging, recovery
- [Deployment](L2/deployment.md) — Production setup, persistent startup
- [Server Architecture](L2/server_architecture.md) — Worker lifecycle, session management
---
### Ai L1 08 Security (`docs/ai/L1/08_security.md`)
# 08 Security
> Secret management, input validation, and repository hygiene.
## API Key Management
- **Single source**: All API keys live in `ai_agents/.env` (git-ignored)
- **Never hardcode** keys in `property.json` — use `${env:VAR_NAME}` substitution
- **Persistent storage**: Keep a copy of keys outside the repo (e.g., `~/api_keys.txt`)
so branch switches don't lose them
- See `.env.example` for the complete variable catalog
## Environment Variable Substitution
In `property.json`, reference secrets via:
```json
{
"api_key": "${env:DEEPGRAM_API_KEY}",
"region": "${env:AZURE_REGION|eastus}"
}
```
| Syntax | Behavior |
| ------------------------- | ---------------------------- |
| `${env:VAR}` | Required — error if missing |
| `${env:VAR\|}` | Optional — empty if missing |
| `${env:VAR\|default}` | Optional — default if missing|
## Sensitive Data in Logs
Extensions must encrypt sensitive fields before logging:
```python
def to_str(self, sensitive_handling: bool = True) -> str:
config = copy.deepcopy(self)
if config.params and "api_key" in config.params:
config.params["api_key"] = utils.encrypt(config.params["api_key"])
return f"{config}"
```
Never log raw API keys, tokens, or credentials.
## Server-Side Protections
The Go server (`http_server.go`) implements:
- **Path traversal prevention**: Ignores client-requested `tenapp_dir`, always uses
the launch-configured directory
- **Channel name sanitization**: Validated before use in file operations
- **Safe type conversion**: Property values are type-checked during merge
- **Recursive property merge**: Prevents injection via nested config overrides
## Pre-Commit Hooks
| Hook | What It Checks |
| ------------- | ----------------------------------------------------------- |
| `pre-commit` | Scans staged files for API key patterns (`API_KEY.*=[A-Za-z0-9]{20,}`) |
| `pre-commit` | Black formatting compliance (line-length 80) |
| `commit-msg` | Conventional commit format, blocks AI tool name references |
## Git-Ignored Files
These are auto-generated — never modify or commit them:
| Pattern | Source |
| ---------------------- | ------------------------- |
| `manifest-lock.json` | `tman` dependency resolve |
| `compile_commands.json`| Build system |
| `BUILD.gn`, `.gn` | Build configuration |
| `out/`, `build/` | Build output |
| `.ten/` | TEN runtime files |
| `bin/main`, `bin/worker`| Compiled binaries |
| `.release/` | Release packaging |
| `node_modules/` | JS dependencies |
| `.env` | Environment secrets |
## Files That Should Never Be Committed
- `.env` (API keys and secrets)
- `*.pem` (certificates)
- `*.pcm` (audio dumps)
- Credential files, tokens, session data
## Related Deep Dives
- [Deployment](L2/deployment.md) — Production security considerations
- [Server Architecture](L2/server_architecture.md) — Server-side validation details
---
### Third Party Curl Docs ALTSVC (`third_party/curl/docs/ALTSVC.md`)
# Alt-Svc
curl features support for the Alt-Svc: HTTP header.
## Enable Alt-Svc in build
`./configure --enable-alt-svc`
(enabled by default since 7.73.0)
## Standard
[RFC 7838](https://datatracker.ietf.org/doc/html/rfc7838)
# Alt-Svc cache file format
This is a text based file with one line per entry and each line consists of nine
space separated fields.
## Example
h2 quic.tech 8443 h3-22 quic.tech 8443 "20190808 06:18:37" 0 0
## Fields
1. The ALPN id for the source origin
2. The host name for the source origin
3. The port number for the source origin
4. The ALPN id for the destination host
5. The host name for the destination host
6. The host number for the destination host
7. The expiration date and time of this entry within double quotes. The date format is "YYYYMMDD HH:MM:SS" and the time zone is GMT.
8. Boolean (1 or 0) if "persist" was set for this entry
9. Integer priority value (not currently used)
# TODO
- handle multiple response headers, when one of them says `clear` (should
override them all)
- using `Age:` value for caching age as per spec
- `CURLALTSVC_IMMEDIATELY` support
---
### Third Party Curl Docs BINDINGS (`third_party/curl/docs/BINDINGS.md`)
libcurl bindings
================
Creative people have written bindings or interfaces for various environments
and programming languages. Using one of these allows you to take advantage of
curl powers from within your favourite language or system.
This is a list of all known interfaces as of this writing.
The bindings listed below are not part of the curl/libcurl distribution
archives, but must be downloaded and installed separately.
[Ada95](https://web.archive.org/web/20070403105909/www.almroth.com/adacurl/index.html) Written by Andreas Almroth
[Basic](https://scriptbasic.com/) ScriptBasic bindings written by Peter Verhas
C++: [curlpp](https://github.com/jpbarrette/curlpp/) Written by Jean-Philippe Barrette-LaPierre,
[curlcpp](https://github.com/JosephP91/curlcpp) by Giuseppe Persico and [C++
Requests](https://github.com/libcpr/cpr) by Huu Nguyen
[Ch](https://chcurl.sourceforge.net/) Written by Stephen Nestinger and Jonathan Rogado
Cocoa: [BBHTTP](https://github.com/biasedbit/BBHTTP) written by Bruno de Carvalho
[curlhandle](https://github.com/karelia/curlhandle) Written by Dan Wood
Clojure: [clj-curl](https://github.com/lsevero/clj-curl) by Lucas Severo
[D](https://dlang.org/library/std/net/curl.html) Written by Kenneth Bogert
[Delphi](https://github.com/Mercury13/curl4delphi) Written by Mikhail Merkuryev
[Dylan](https://dylanlibs.sourceforge.net/) Written by Chris Double
[Eiffel](https://iron.eiffel.com/repository/20.11/package/ABEF6975-37AC-45FD-9C67-52D10BA0669B) Written by Eiffel Software
[Euphoria](https://web.archive.org/web/20050204080544/rays-web.com/eulibcurl.htm) Written by Ray Smith
[Falcon](http://www.falconpl.org/project_docs/curl/)
[Ferite](https://web.archive.org/web/20150102192018/ferite.org/) Written by Paul Querna
[Fortran](https://github.com/interkosmos/fortran-curl) Written by Philipp Engel
[Gambas](https://gambas.sourceforge.net/)
[glib/GTK+](https://web.archive.org/web/20100526203452/atterer.net/glibcurl) Written by Richard Atterer
Go: [go-curl](https://github.com/andelf/go-curl) by ShuYu Wang
[Guile](https://github.com/spk121/guile-curl) Written by Michael L. Gran
[Harbour](https://github.com/vszakats/hb/tree/main/contrib/hbcurl) Written by Viktor Szakats
[Haskell](https://hackage.haskell.org/package/curl) Written by Galois, Inc
[Hollywood](https://www.hollywood-mal.com/download.html) hURL by Andreas Falkenhahn
[Java](https://github.com/pjlegato/curl-java)
[Julia](https://github.com/JuliaWeb/LibCURL.jl) Written by Amit Murthy
[Kapito](https://github.com/puzza007/katipo) is an Erlang HTTP library around libcurl.
[Lisp](https://common-lisp.net/project/cl-curl/) Written by Liam Healy
Lua: [luacurl](https://web.archive.org/web/20201205052437/luacurl.luaforge.net/) by Alexander Marinov, [Lua-cURL](https://github.com/Lua-cURL) by Jürgen Hötzel
[Mono](https://web.archive.org/web/20070606064500/https://forge.novell.com/modules/xfmod/project/?libcurl-mono) Written by Jeffrey Phillips
[.NET](https://sourceforge.net/projects/libcurl-net/) libcurl-net by Jeffrey Phillips
[Nim](https://nimble.directory/pkg/libcurl) wrapper for libcurl
[node.js](https://github.com/JCMais/node-libcurl) node-libcurl by Jonathan Cardoso Machado
[Object-Pascal](https://web.archive.org/web/20020610214926/www.tekool.com/opcurl) Free Pascal, Delphi and Kylix binding written by Christophe Espern.
[OCaml](https://opam.ocaml.org/packages/ocurl/) Written by Lars Nilsson and ygrek
[Pascal](https://web.archive.org/web/20030804091414/houston.quik.com/jkp/curlpas/) Free Pascal, Delphi and Kylix binding written by Jeffrey Pohlmeyer.
Perl: [WWW::Curl](https://github.com/szbalint/WWW--Curl) Maintained by Cris
Bailiff and Bálint Szilakszi,
[perl6-net-curl](https://github.com/azawawi/perl6-net-curl) by Ahmad M. Zawawi
[NET::Curl](https://metacpan.org/pod/Net::Curl) by Przemyslaw Iskra
[PHP](https://php.net/curl) Originally written by Sterling Hughes
[PostgreSQL](https://github.com/pramsey/pgsql-http) - HTTP client for PostgreSQL
[PostgreSQL](https://github.com/RekGRpth/pg_curl) - cURL client for PostgreSQL
[PureBasic](https://www.purebasic.com/documentation/http/index.html) uses libcurl in its "native" HTTP subsystem
[Python](http://pycurl.io/) PycURL by Kjetil Jacobsen
[Q](https://q-lang.sourceforge.net/) The libcurl module is part of the default install
[R](https://cran.r-project.org/package=curl)
[Rexx](https://rexxcurl.sourceforge.net/) Written Mark Hessling
[Ring](https://ring-lang.sourceforge.io/doc1.3/libcurl.html) RingLibCurl by Mahmoud Fayed
RPG, support for ILE/RPG on OS/400 is included in source distribution
Ruby: [curb](https://github.com/taf2/curb) written by Ross Bamford,
[ruby-curl-multi](https://github.com/kball/curl_multi.rb) by Kristjan Petursson and Keith Rarick
[Rust](https://github.com/alexcrichton/curl-rust) curl-rust - by Carl Lerche
[Scheme](http://www.metapaper.net/lisovsky/web/curl/) Bigloo binding by Kirill Lisovsky
[Scilab](https://help.scilab.org/docs/current/fr_FR/getURL.html) binding by Sylvestre Ledru
[S-Lang](https://www.jedsoft.org/slang/modules/curl.html) by John E Davis
[Smalltalk](https://www.squeaksource.com/CurlPlugin/) Written by Danil Osipchuk
[SP-Forth](https://sourceforge.net/p/spf/spf/ci/master/tree/devel/~ac/lib/lin/curl/) Written by Andrey Cherezov
[SPL](https://web.archive.org/web/20210203022158/www.clifford.at/spl/spldoc/curl.html) Written by Clifford Wolf
[Tcl](https://web.archive.org/web/20160826011806/mirror.yellow5.com/tclcurl/) Tclcurl by Andrés García
[Visual Basic](https://sourceforge.net/projects/libcurl-vb/) libcurl-vb by Jeffrey Phillips
[Visual Foxpro](https://web.archive.org/web/20130730181523/www.ctl32.com.ar/libcurl.asp) by Carlos Alloatti
[wxWidgets](https://wxcode.sourceforge.net/components/wxcurl/) Written by Casey O'Donnell
[XBLite](https://web.archive.org/web/20060426150418/perso.wanadoo.fr/xblite/libraries.html) Written by David Szafranski
[Xojo](https://github.com/charonn0/RB-libcURL) Written by Andrew Lambert
---
### Third Party Curl Docs BUFQ (`third_party/curl/docs/BUFQ.md`)
# bufq
This is an internal module for managing I/O buffers. A `bufq` can be written
to and read from. It manages read and write positions and has a maximum size.
## read/write
Its basic read/write functions have a similar signature and return code handling
as many internal Curl read and write ones.
```
ssize_t Curl_bufq_write(struct bufq *q, const unsigned char *buf, size_t len, CURLcode *err);
- returns the length written into `q` or -1 on error.
- writing to a full `q` will return -1 and set *err to CURLE_AGAIN
ssize_t Curl_bufq_read(struct bufq *q, unsigned char *buf, size_t len, CURLcode *err);
- returns the length read from `q` or -1 on error.
- reading from an empty `q` will return -1 and set *err to CURLE_AGAIN
```
To pass data into a `bufq` without an extra copy, read callbacks can be used.
```
typedef ssize_t Curl_bufq_reader(void *reader_ctx, unsigned char *buf, size_t len,
CURLcode *err);
ssize_t Curl_bufq_slurp(struct bufq *q, Curl_bufq_reader *reader, void *reader_ctx,
CURLcode *err);
```
`Curl_bufq_slurp()` will invoke the given `reader` callback, passing it its own internal
buffer memory to write to. It may invoke the `reader` several times, as long as it has space
and while the `reader` always returns the length that was requested. There are variations of `slurp` that call the `reader` at most once or only read in a
maximum amount of bytes.
The analog mechanism for write out buffer data is:
```
typedef ssize_t Curl_bufq_writer(void *writer_ctx, const unsigned char *buf, size_t len,
CURLcode *err);
ssize_t Curl_bufq_pass(struct bufq *q, Curl_bufq_writer *writer, void *writer_ctx,
CURLcode *err);
```
`Curl_bufq_pass()` will invoke the `writer`, passing its internal memory and remove the
amount that `writer` reports.
## peek and skip
It is possible to get access to the memory of data stored in a `bufq` with:
```
bool Curl_bufq_peek(const struct bufq *q, const unsigned char **pbuf, size_t *plen);
```
On returning TRUE, `pbuf` will point to internal memory with `plen` bytes that one may read. This will only
be valid until another operation on `bufq` is performed.
Instead of reading `bufq` data, one may simply skip it:
```
void Curl_bufq_skip(struct bufq *q, size_t amount);
```
This will remove `amount` number of bytes from the `bufq`.
## lifetime
`bufq` is initialized and freed similar to the `dynbuf` module. Code using `bufq` will
hold a `struct bufq` somewhere. Before it uses it, it invokes:
```
void Curl_bufq_init(struct bufq *q, size_t chunk_size, size_t max_chunks);
```
The `bufq` is told how many "chunks" of data it shall hold at maximum and how large those
"chunks" should be. There are some variants of this, allowing for more options. How "chunks" are handled in a `bufq` is presented in the section about memory management.
The user of the `bufq` has the responsibility to call:
```
void Curl_bufq_free(struct bufq *q);
```
to free all resources held by `q`. It is possible to reset a `bufq` to empty via:
```
void Curl_bufq_reset(struct bufq *q);
```
## memory management
Internally, a `bufq` uses allocation of fixed size, e.g. the "chunk_size", up to a maximum number, e.g. "max_chunks". These chunks are allocated on demand, therefore writing to a `bufq` may return `CURLE_OUT_OF_MEMORY`. Once the max number of chunks are used, the `bufq` will report that it is "full".
Each chunks has a `read` and `write` index. A `bufq` keeps its chunks in a list. Reading happens always at the head chunk, writing always goes to the tail chunk. When the head chunk becomes empty, it is removed. When the tail chunk becomes full, another chunk is added to the end of the list, becoming the new tail.
Chunks that are no longer used are returned to a `spare` list by default. If the `bufq` is created with option `BUFQ_OPT_NO_SPARES` those chunks will be freed right away.
If a `bufq` is created with a `bufc_pool`, the no longer used chunks are returned to the pool. Also `bufq` will ask the pool for a chunk when it needs one. More in section "pools".
## empty, full and overflow
One can ask about the state of a `bufq` with methods such as `Curl_bufq_is_empty(q)`,
`Curl_bufq_is_full(q)`, etc. The amount of data held by a `bufq` is the sum of the data in all its chunks. This is what is reported by `Curl_bufq_len(q)`.
Note that a `bufq` length and it being "full" are only loosely related. A simple example:
* create a `bufq` with chunk_size=1000 and max_chunks=4.
* write 4000 bytes to it, it will report "full"
* read 1 bytes from it, it will still report "full"
* read 999 more bytes from it, and it will no longer be "full"
The reason for this is that full really means: *bufq uses max_chunks and the last one cannot be written to*.
So when you read 1 byte from the head chunk in the example above, the head still hold 999 unread bytes. Only when those are also read, can the head chunk be removed and a new tail be added.
There is another variation to this. If you initialized a `bufq` with option `BUFQ_OPT_SOFT_LIMIT`, it will allow writes **beyond** the `max_chunks`. It will report **full**, but one can **still** write. This option is necessary, if partial writes need to be avoided. But it means that you will need other checks to keep the `bufq` from growing ever larger and larger.
## pools
A `struct bufc_pool` may be used to create chunks for a `bufq` and keep spare ones around. It is initialized
and used via:
```
void Curl_bufcp_init(struct bufc_pool *pool, size_t chunk_size, size_t spare_max);
void Curl_bufq_initp(struct bufq *q, struct bufc_pool *pool, size_t max_chunks, int opts);
```
The pool gets the size and the mount of spares to keep. The `bufq` gets the pool and the `max_chunks`. It no longer needs to know the chunk sizes, as those are managed by the pool.
A pool can be shared between many `bufq`s, as long as all of them operate in the same thread. In curl that would be true for all transfers using the same multi handle. The advantages of a pool are:
* when all `bufq`s are empty, only memory for `max_spare` chunks in the pool is used. Empty `bufq`s will hold no memory.
* the latest spare chunk is the first to be handed out again, no matter which `bufq` needs it. This keeps the footprint of "recently used" memory smaller.
---
### Third Party Curl Docs BUFREF (`third_party/curl/docs/BUFREF.md`)
# bufref
This is an internal module for handling buffer references. A referenced
buffer is associated with its destructor function that is implicitly called
when the reference is invalidated. Once referenced, a buffer cannot be
reallocated.
A data length is stored within the reference for binary data handling
purposes; it is not used by the bufref API.
The `struct bufref` is used to hold data referencing a buffer. The members of
that structure **MUST NOT** be accessed or modified without using the dedicated
bufref API.
## `init`
```c
void Curl_bufref_init(struct bufref *br);
```
Initializes a `bufref` structure. This function **MUST** be called before any
other operation is performed on the structure.
Upon completion, the referenced buffer is `NULL` and length is zero.
This function may also be called to bypass referenced buffer destruction while
invalidating the current reference.
## `free`
```c
void Curl_bufref_free(struct bufref *br);
```
Destroys the previously referenced buffer using its destructor and
reinitializes the structure for a possible subsequent reuse.
## `set`
```c
void Curl_bufref_set(struct bufref *br, const void *buffer, size_t length,
void (*destructor)(void *));
```
Releases the previously referenced buffer, then assigns the new `buffer` to
the structure, associated with its `destructor` function. The latter can be
specified as `NULL`: this will be the case when the referenced buffer is
static.
if `buffer` is NULL, `length` must be zero.
## `memdup`
```c
CURLcode Curl_bufref_memdup(struct bufref *br, const void *data, size_t length);
```
Releases the previously referenced buffer, then duplicates the `length`-byte
`data` into a buffer allocated via `malloc()` and references the latter
associated with destructor `curl_free()`.
An additional trailing byte is allocated and set to zero as a possible string
null-terminator; it is not counted in the stored length.
Returns `CURLE_OK` if successful, else `CURLE_OUT_OF_MEMORY`.
## `ptr`
```c
const unsigned char *Curl_bufref_ptr(const struct bufref *br);
```
Returns a `const unsigned char *` to the referenced buffer.
## `len`
```c
size_t Curl_bufref_len(const struct bufref *br);
```
Returns the stored length of the referenced buffer.
---
### Third Party Curl Docs BUG BOUNTY (`third_party/curl/docs/BUG-BOUNTY.md`)
# The curl bug bounty
The curl project runs a bug bounty program in association with
[HackerOne](https://www.hackerone.com) and the [Internet Bug
Bounty](https://internetbugbounty.org).
## How does it work?
Start out by posting your suspected security vulnerability directly to [curl's
HackerOne program](https://hackerone.com/curl).
After you have reported a security issue, it has been deemed credible, and a
patch and advisory has been made public, you may be eligible for a bounty from
this program. See the [SECURITY-PROCESS](SECURITY-PROCESS.md) document for how
we work with security issues.
## What are the reward amounts?
The curl project offers monetary compensation for reported and published
security vulnerabilities. The amount of money that is rewarded depends on how
serious the flaw is determined to be.
Since 2021, the Bug Bounty is managed in association with the Internet Bug
Bounty and they will set the reward amounts. If it would turn out that they
set amounts that are way lower than we can accept, the curl project intends to
"top up" rewards.
In 2022, typical "Medium" rated vulnerabilities have been rewarded 2,400 USD
each.
## Who is eligible for a reward?
Everyone and anyone who reports a security problem in a released curl version
that has not already been reported can ask for a bounty.
Dedicated - paid for - security audits that are performed in collaboration
with curl developers are not eligible for bounties.
Vulnerabilities in features that are off by default and documented as
experimental are not eligible for a reward.
The vulnerability has to be fixed and publicly announced (by the curl project)
before a bug bounty will be considered.
Once the vulnerability has been published by curl, the researcher can request
their bounty from the [Internet Bug Bounty](https://hackerone.com/ibb).
Bounties need to be requested within twelve months from the publication of the
vulnerability.
## Product vulnerabilities only
This bug bounty only concerns the curl and libcurl products and thus their
respective source codes - when running on existing hardware. It does not
include curl documentation, curl websites, or other curl related
infrastructure.
The curl security team is the sole arbiter if a reported flaw is subject to a
bounty or not.
## How are vulnerabilities graded?
The grading of each reported vulnerability that makes a reward claim will be
performed by the curl security team. The grading will be based on the CVSS
(Common Vulnerability Scoring System) 3.0.
## How are reward amounts determined?
The curl security team gives the vulnerability a score or severity level, as
mentioned above. The actual monetary reward amount is decided and paid by the
Internet Bug Bounty..
## Regarding taxes, etc. on the bounties
In the event that the individual receiving a bug bounty needs to pay taxes on
the reward money, the responsibility lies with the receiver. The curl project
or its security team never actually receive any of this money, hold the money,
or pay out the money.
---
### Third Party Curl Docs BUGS (`third_party/curl/docs/BUGS.md`)
# BUGS
## There are still bugs
Curl and libcurl keep being developed. Adding features and changing code
means that bugs will sneak in, no matter how hard we try to keep them out.
Of course there are lots of bugs left. And lots of misfeatures.
To help us make curl the stable and solid product we want it to be, we need
bug reports and bug fixes.
## Where to report
If you cannot fix a bug yourself and submit a fix for it, try to report an as
detailed report as possible to a curl mailing list to allow one of us to have
a go at a solution. You can optionally also submit your problem in [curl's
bug tracking system](https://github.com/curl/curl/issues).
Please read the rest of this document below first before doing that.
If you feel you need to ask around first, find a suitable [mailing list](
https://curl.se/mail/) and post your questions there.
## Security bugs
If you find a bug or problem in curl or libcurl that you think has a security
impact, for example a bug that can put users in danger or make them
vulnerable if the bug becomes public knowledge, then please report that bug
using our security development process.
Security related bugs or bugs that are suspected to have a security impact,
should be reported on the [curl security tracker at
HackerOne](https://hackerone.com/curl).
This ensures that the report reaches the curl security team so that they
first can deal with the report away from the public to minimize the harm
and impact it will have on existing users out there who might be using the
vulnerable versions.
The curl project's process for handling security related issues is
[documented separately](https://curl.se/dev/secprocess.html).
## What to report
When reporting a bug, you should include all information that will help us
understand what is wrong, what you expected to happen and how to repeat the
bad behavior. You therefore need to tell us:
- your operating system's name and version number
- what version of curl you are using (`curl -V` is fine)
- versions of the used libraries that libcurl is built to use
- what URL you were working with (if possible), at least which protocol
and anything and everything else you think matters. Tell us what you expected
to happen, tell use what did happen, tell us how you could make it work
another way. Dig around, try out, test. Then include all the tiny bits and
pieces in your report. You will benefit from this yourself, as it will enable
us to help you quicker and more accurately.
Since curl deals with networks, it often helps us if you include a protocol
debug dump with your bug report. The output you get by using the `-v` or
`--trace` options.
If curl crashed, causing a core dump (in Unix), there is hardly any use to
send that huge file to anyone of us. Unless we have the same system setup as
you, we cannot do much with it. Instead, we ask you to get a stack trace and
send that (much smaller) output to us instead.
The address and how to subscribe to the mailing lists are detailed in the
`MANUAL.md` file.
## libcurl problems
When you have written your own application with libcurl to perform transfers,
it is even more important to be specific and detailed when reporting bugs.
Tell us the libcurl version and your operating system. Tell us the name and
version of all relevant sub-components like for example the SSL library
you are using and what name resolving your libcurl uses. If you use SFTP or
SCP, the libssh2 version is relevant etc.
Showing us a real source code example repeating your problem is the best way
to get our attention and it will greatly increase our chances to understand
your problem and to work on a fix (if we agree it truly is a problem).
Lots of problems that appear to be libcurl problems are actually just abuses
of the libcurl API or other malfunctions in your applications. It is advised
that you run your problematic program using a memory debug tool like valgrind
or similar before you post memory-related or "crashing" problems to us.
## Who will fix the problems
If the problems or bugs you describe are considered to be bugs, we want to
have the problems fixed.
There are no developers in the curl project that are paid to work on bugs.
All developers that take on reported bugs do this on a voluntary basis. We do
it out of an ambition to keep curl and libcurl excellent products and out of
pride.
Please do not assume that you can just lump over something to us and it will
then magically be fixed after some given time. Most often we need feedback
and help to understand what you have experienced and how to repeat a
problem. Then we may only be able to assist YOU to debug the problem and to
track down the proper fix.
We get reports from many people every month and each report can take a
considerable amount of time to really go to the bottom with.
## How to get a stack trace
First, you must make sure that you compile all sources with `-g` and that you
do not 'strip' the final executable. Try to avoid optimizing the code as well,
remove `-O`, `-O2` etc from the compiler options.
Run the program until it cores.
Run your debugger on the core file, like ` curl
core`. `` should be replaced with the name of your debugger, in
most cases that will be `gdb`, but `dbx` and others also occur.
When the debugger has finished loading the core file and presents you a
prompt, enter `where` (without quotes) and press return.
The list that is presented is the stack trace. If everything worked, it is
supposed to contain the chain of functions that were called when curl
crashed. Include the stack trace with your detailed bug report, it will help a
lot.
## Bugs in libcurl bindings
There will of course pop up bugs in libcurl bindings. You should then
primarily approach the team that works on that particular binding and see
what you can do to help them fix the problem.
If you suspect that the problem exists in the underlying libcurl, then please
convert your program over to plain C and follow the steps outlined above.
## Bugs in old versions
The curl project typically releases new versions every other month, and we
fix several hundred bugs per year. For a huge table of releases, number of
bug fixes and more, see: https://curl.se/docs/releases.html
The developers in the curl project do not have bandwidth or energy enough to
maintain several branches or to spend much time on hunting down problems in
old versions when chances are we already fixed them or at least that they have
changed nature and appearance in later versions.
When you experience a problem and want to report it, you really SHOULD
include the version number of the curl you are using when you experience the
issue. If that version number shows us that you are using an out-of-date curl,
you should also try out a modern curl version to see if the problem persists
or how/if it has changed in appearance.
Even if you cannot immediately upgrade your application/system to run the
latest curl version, you can most often at least run a test version or
experimental build or similar, to get this confirmed or not.
At times people insist that they cannot upgrade to a modern curl version, but
instead, they "just want the bug fixed". That is fine, just do not count on us
spending many cycles on trying to identify which single commit, if that is
even possible, that at some point in the past fixed the problem you are now
experiencing.
Security wise, it is almost always a bad idea to lag behind the current curl
versions by a lot. We keep discovering and reporting security problems
over time see you can see in [this
table](https://curl.se/docs/vulnerabilities.html)
# Bug fixing procedure
## What happens on first filing
When a new issue is posted in the issue tracker or on the mailing list, the
team of developers first needs to see the report. Maybe they took the day off,
maybe they are off in the woods hunting. Have patience. Allow at least a few
days before expecting someone to have responded.
In the issue tracker, you can expect that some labels will be set on the issue
to help categorize it.
## First response
If your issue/bug report was not perfect at once (and few are), chances are
that someone will ask follow-up questions. Which version did you use? Which
options did you use? How often does the problem occur? How can we reproduce
this problem? Which protocols does it involve? Or perhaps much more specific
and deep diving questions. It all depends on your specific issue.
You should then respond to these follow-up questions and provide more info
about the problem, so that we can help you figure it out. Or maybe you can
help us figure it out. An active back-and-forth communication is important
and the key for finding a cure and landing a fix.
## Not reproducible
We may require further work from you who actually see or experience the
problem if we cannot reproduce it and cannot understand it even after having
gotten all the info we need and having studied the source code over again.
## Unresponsive
If the problem have not been understood or reproduced, and there is nobody
responding to follow-up questions or questions asking for clarifications or
for discussing possible ways to move forward with the task, we take that as a
strong suggestion that the bug is unimportant.
Unimportant issues will be closed as inactive sooner or later as they cannot
be fixed. The inactivity period (waiting for responses) should not be shorter
than two weeks but may extend months.
## Lack of time/interest
Bugs that are filed and are understood can unfortunately end up in the
"nobody cares enough about it to work on it" category. Such bugs are
perfectly valid problems that *should* get fixed but apparently are not. We
try to mark such bugs as `KNOWN_BUGS material` after a time of inactivity and
if no activity is noticed after yet some time those bugs are added to the
`KNOWN_BUGS` document and are closed in the issue tracker.
## `KNOWN_BUGS`
This is a list of known bugs. Bugs we know exist and that have been pointed
out but that have not yet been fixed. The reasons for why they have not been
fixed can involve anything really, but the primary reason is that nobody has
considered these problems to be important enough to spend the necessary time
and effort to have them fixed.
The `KNOWN_BUGS` items are always up for grabs and we love the ones who bring
one of them back to life and offer solutions to them.
The `KNOWN_BUGS` document has a sibling document known as `TODO`.
## `TODO`
Issues that are filed or reported that are not really bugs but more missing
features or ideas for future improvements and so on are marked as
'enhancement' or 'feature-request' and will be added to the `TODO` document
and the issues are closed. We do not keep TODO items open in the issue
tracker.
The `TODO` document is full of ideas and suggestions of what we can add or
fix one day. You are always encouraged and free to grab one of those items and
take up a discussion with the curl development team on how that could be
implemented or provided in the project so that you can work on ticking it odd
that document.
If an issue is rather a bug and not a missing feature or functionality, it is
listed in `KNOWN_BUGS` instead.
## Closing off stalled bugs
The [issue and pull request trackers](https://github.com/curl/curl) only
hold "active" entries open (using a non-precise definition of what active
actually is, but they are at least not completely dead). Those that are
abandoned or in other ways dormant will be closed and sometimes added to
`TODO` and `KNOWN_BUGS` instead.
This way, we only have "active" issues open on GitHub. Irrelevant issues and
pull requests will not distract developers or casual visitors.
---
### Third Party Curl Docs CHECKSRC (`third_party/curl/docs/CHECKSRC.md`)
# checksrc
This is the tool we use within the curl project to scan C source code and
check that it adheres to our [Source Code Style guide](CODE_STYLE.md).
## Usage
checksrc.pl [options] [file1] [file2] ...
## Command line options
`-W[file]` skip that file and exclude it from being checked. Helpful
when, for example, one of the files is generated.
`-D[dir]` directory name to prepend to file names when accessing them.
`-h` shows the help output, that also lists all recognized warnings
## What does `checksrc` warn for?
`checksrc` does not check and verify the code against the entire style guide.
The script is an effort to detect the most common mistakes and syntax mistakes
that contributors make before they get accustomed to our code style. Heck,
many of us regulars do the mistakes too and this script helps us keep the code
in shape.
checksrc.pl -h
Lists how to use the script and it lists all existing warnings it has and
problems it detects. At the time of this writing, the existing `checksrc`
warnings are:
- `ASSIGNWITHINCONDITION`: Assignment within a conditional expression. The
code style mandates the assignment to be done outside of it.
- `ASTERISKNOSPACE`: A pointer was declared like `char* name` instead of the
more appropriate `char *name` style. The asterisk should sit next to the
name.
- `ASTERISKSPACE`: A pointer was declared like `char * name` instead of the
more appropriate `char *name` style. The asterisk should sit right next to
the name without a space in between.
- `BADCOMMAND`: There's a bad `checksrc` instruction in the code. See the
**Ignore certain warnings** section below for details.
- `BANNEDFUNC`: A banned function was used. The functions sprintf, vsprintf,
strcat, strncat, gets are **never** allowed in curl source code.
- `BRACEELSE`: '} else' on the same line. The else is supposed to be on the
following line.
- `BRACEPOS`: wrong position for an open brace (`{`).
- `BRACEWHILE`: more than once space between end brace and while keyword
- `COMMANOSPACE`: a comma without following space
- `COPYRIGHT`: the file is missing a copyright statement
- `CPPCOMMENTS`: `//` comment detected, that is not C89 compliant
- `DOBRACE`: only use one space after do before open brace
- `EMPTYLINEBRACE`: found empty line before open brace
- `EQUALSNOSPACE`: no space after `=` sign
- `EQUALSNULL`: comparison with `== NULL` used in if/while. We use `!var`.
- `EXCLAMATIONSPACE`: space found after exclamations mark
- `FOPENMODE`: `fopen()` needs a macro for the mode string, use it
- `INDENTATION`: detected a wrong start column for code. Note that this
warning only checks some specific places and will certainly miss many bad
indentations.
- `LONGLINE`: A line is longer than 79 columns.
- `MULTISPACE`: Multiple spaces were found where only one should be used.
- `NOSPACEEQUALS`: An equals sign was found without preceding space. We prefer
`a = 2` and *not* `a=2`.
- `NOTEQUALSZERO`: check found using `!= 0`. We use plain `if(var)`.
- `ONELINECONDITION`: do not put the conditional block on the same line as `if()`
- `OPENCOMMENT`: File ended with a comment (`/*`) still "open".
- `PARENBRACE`: `){` was used without sufficient space in between.
- `RETURNNOSPACE`: `return` was used without space between the keyword and the
following value.
- `SEMINOSPACE`: There was no space (or newline) following a semicolon.
- `SIZEOFNOPAREN`: Found use of sizeof without parentheses. We prefer
`sizeof(int)` style.
- `SNPRINTF` - Found use of `snprintf()`. Since we use an internal replacement
with a different return code etc, we prefer `msnprintf()`.
- `SPACEAFTERPAREN`: there was a space after open parenthesis, `( text`.
- `SPACEBEFORECLOSE`: there was a space before a close parenthesis, `text )`.
- `SPACEBEFORECOMMA`: there was a space before a comma, `one , two`.
- `SPACEBEFOREPAREN`: there was a space before an open parenthesis, `if (`,
where one was not expected
- `SPACESEMICOLON`: there was a space before semicolon, ` ;`.
- `TABS`: TAB characters are not allowed
- `TRAILINGSPACE`: Trailing whitespace on the line
- `TYPEDEFSTRUCT`: we frown upon (most) typedefed structs
- `UNUSEDIGNORE`: a `checksrc` inlined warning ignore was asked for but not
used, that is an ignore that should be removed or changed to get used.
### Extended warnings
Some warnings are quite computationally expensive to perform, so they are
turned off by default. To enable these warnings, place a `.checksrc` file in
the directory where they should be activated with commands to enable the
warnings you are interested in. The format of the file is to enable one
warning per line like so: `enable `
Currently these are the extended warnings which can be enabled:
- `COPYRIGHTYEAR`: the current changeset has not updated the copyright year in
the source file
- `STRERROR`: use of banned function strerror()
## Ignore certain warnings
Due to the nature of the source code and the flaws of the `checksrc` tool,
there is sometimes a need to ignore specific warnings. `checksrc` allows a few
different ways to do this.
### Inline ignore
You can control what to ignore within a specific source file by providing
instructions to `checksrc` in the source code itself. See examples below. The
instruction can ask to ignore a specific warning a specific number of times or
you ignore all of them until you mark the end of the ignored section.
Inline ignores are only done for that single specific source code file.
Example
/* !checksrc! disable LONGLINE all */
This will ignore the warning for overly long lines until it is re-enabled with:
/* !checksrc! enable LONGLINE */
If the enabling is not performed before the end of the file, it will be enabled
automatically for the next file.
You can also opt to ignore just N violations so that if you have a single long
line you just cannot shorten and is agreed to be fine anyway:
/* !checksrc! disable LONGLINE 1 */
... and the warning for long lines will be enabled again automatically after
it has ignored that single warning. The number `1` can of course be changed to
any other integer number. It can be used to make sure only the exact intended
instances are ignored and nothing extra.
### Directory wide ignore patterns
This is a method we have transitioned away from. Use inline ignores as far as
possible.
Make a `checksrc.skip` file in the directory of the source code with the
false positive, and include the full offending line into this file.
---
### Third Party Curl Docs CIPHERS (`third_party/curl/docs/CIPHERS.md`)
# Ciphers
With curl's options
[`CURLOPT_SSL_CIPHER_LIST`](https://curl.se/libcurl/c/CURLOPT_SSL_CIPHER_LIST.html)
and
[`--ciphers`](https://curl.se/docs/manpage.html#--ciphers)
users can control which ciphers to consider when negotiating TLS connections.
TLS 1.3 ciphers are supported since curl 7.61 for OpenSSL 1.1.1+, and since
curl 7.85 for Schannel with options
[`CURLOPT_TLS13_CIPHERS`](https://curl.se/libcurl/c/CURLOPT_TLS13_CIPHERS.html)
and
[`--tls13-ciphers`](https://curl.se/docs/manpage.html#--tls13-ciphers)
. If you are using a different SSL backend you can try setting TLS 1.3 cipher
suites by using the respective regular cipher option.
The names of the known ciphers differ depending on which TLS backend that
libcurl was built to use. This is an attempt to list known cipher names.
## OpenSSL
(based on [OpenSSL docs](https://www.openssl.org/docs/manmaster/man1/openssl-ciphers.html))
When specifying multiple cipher names, separate them with colon (`:`).
### SSL3 cipher suites
`NULL-MD5`
`NULL-SHA`
`RC4-MD5`
`RC4-SHA`
`IDEA-CBC-SHA`
`DES-CBC3-SHA`
`DH-DSS-DES-CBC3-SHA`
`DH-RSA-DES-CBC3-SHA`
`DHE-DSS-DES-CBC3-SHA`
`DHE-RSA-DES-CBC3-SHA`
`ADH-RC4-MD5`
`ADH-DES-CBC3-SHA`
### TLS v1.0 cipher suites
`NULL-MD5`
`NULL-SHA`
`RC4-MD5`
`RC4-SHA`
`IDEA-CBC-SHA`
`DES-CBC3-SHA`
`DHE-DSS-DES-CBC3-SHA`
`DHE-RSA-DES-CBC3-SHA`
`ADH-RC4-MD5`
`ADH-DES-CBC3-SHA`
### AES cipher suites from RFC3268, extending TLS v1.0
`AES128-SHA`
`AES256-SHA`
`DH-DSS-AES128-SHA`
`DH-DSS-AES256-SHA`
`DH-RSA-AES128-SHA`
`DH-RSA-AES256-SHA`
`DHE-DSS-AES128-SHA`
`DHE-DSS-AES256-SHA`
`DHE-RSA-AES128-SHA`
`DHE-RSA-AES256-SHA`
`ADH-AES128-SHA`
`ADH-AES256-SHA`
### SEED cipher suites from RFC4162, extending TLS v1.0
`SEED-SHA`
`DH-DSS-SEED-SHA`
`DH-RSA-SEED-SHA`
`DHE-DSS-SEED-SHA`
`DHE-RSA-SEED-SHA`
`ADH-SEED-SHA`
### GOST cipher suites, extending TLS v1.0
`GOST94-GOST89-GOST89`
`GOST2001-GOST89-GOST89`
`GOST94-NULL-GOST94`
`GOST2001-NULL-GOST94`
### Elliptic curve cipher suites
`ECDHE-RSA-NULL-SHA`
`ECDHE-RSA-RC4-SHA`
`ECDHE-RSA-DES-CBC3-SHA`
`ECDHE-RSA-AES128-SHA`
`ECDHE-RSA-AES256-SHA`
`ECDHE-ECDSA-NULL-SHA`
`ECDHE-ECDSA-RC4-SHA`
`ECDHE-ECDSA-DES-CBC3-SHA`
`ECDHE-ECDSA-AES128-SHA`
`ECDHE-ECDSA-AES256-SHA`
`AECDH-NULL-SHA`
`AECDH-RC4-SHA`
`AECDH-DES-CBC3-SHA`
`AECDH-AES128-SHA`
`AECDH-AES256-SHA`
### TLS v1.2 cipher suites
`NULL-SHA256`
`AES128-SHA256`
`AES256-SHA256`
`AES128-GCM-SHA256`
`AES256-GCM-SHA384`
`DH-RSA-AES128-SHA256`
`DH-RSA-AES256-SHA256`
`DH-RSA-AES128-GCM-SHA256`
`DH-RSA-AES256-GCM-SHA384`
`DH-DSS-AES128-SHA256`
`DH-DSS-AES256-SHA256`
`DH-DSS-AES128-GCM-SHA256`
`DH-DSS-AES256-GCM-SHA384`
`DHE-RSA-AES128-SHA256`
`DHE-RSA-AES256-SHA256`
`DHE-RSA-AES128-GCM-SHA256`
`DHE-RSA-AES256-GCM-SHA384`
`DHE-DSS-AES128-SHA256`
`DHE-DSS-AES256-SHA256`
`DHE-DSS-AES128-GCM-SHA256`
`DHE-DSS-AES256-GCM-SHA384`
`ECDHE-RSA-AES128-SHA256`
`ECDHE-RSA-AES256-SHA384`
`ECDHE-RSA-AES128-GCM-SHA256`
`ECDHE-RSA-AES256-GCM-SHA384`
`ECDHE-ECDSA-AES128-SHA256`
`ECDHE-ECDSA-AES256-SHA384`
`ECDHE-ECDSA-AES128-GCM-SHA256`
`ECDHE-ECDSA-AES256-GCM-SHA384`
`ADH-AES128-SHA256`
`ADH-AES256-SHA256`
`ADH-AES128-GCM-SHA256`
`ADH-AES256-GCM-SHA384`
`AES128-CCM`
`AES256-CCM`
`DHE-RSA-AES128-CCM`
`DHE-RSA-AES256-CCM`
`AES128-CCM8`
`AES256-CCM8`
`DHE-RSA-AES128-CCM8`
`DHE-RSA-AES256-CCM8`
`ECDHE-ECDSA-AES128-CCM`
`ECDHE-ECDSA-AES256-CCM`
`ECDHE-ECDSA-AES128-CCM8`
`ECDHE-ECDSA-AES256-CCM8`
### Camellia HMAC-Based cipher suites from RFC6367, extending TLS v1.2
`ECDHE-ECDSA-CAMELLIA128-SHA256`
`ECDHE-ECDSA-CAMELLIA256-SHA384`
`ECDHE-RSA-CAMELLIA128-SHA256`
`ECDHE-RSA-CAMELLIA256-SHA384`
### TLS 1.3 cipher suites
(Note these ciphers are set with `CURLOPT_TLS13_CIPHERS` and `--tls13-ciphers`)
`TLS_AES_256_GCM_SHA384`
`TLS_CHACHA20_POLY1305_SHA256`
`TLS_AES_128_GCM_SHA256`
`TLS_AES_128_CCM_8_SHA256`
`TLS_AES_128_CCM_SHA256`
## NSS
### Totally insecure
`rc4`
`rc4-md5`
`rc4export`
`rc2`
`rc2export`
`des`
`desede3`
### SSL3/TLS cipher suites
`rsa_rc4_128_md5`
`rsa_rc4_128_sha`
`rsa_3des_sha`
`rsa_des_sha`
`rsa_rc4_40_md5`
`rsa_rc2_40_md5`
`rsa_null_md5`
`rsa_null_sha`
`fips_3des_sha`
`fips_des_sha`
`fortezza`
`fortezza_rc4_128_sha`
`fortezza_null`
### TLS 1.0 Exportable 56-bit Cipher Suites
`rsa_des_56_sha`
`rsa_rc4_56_sha`
### AES ciphers
`dhe_dss_aes_128_cbc_sha`
`dhe_dss_aes_256_cbc_sha`
`dhe_rsa_aes_128_cbc_sha`
`dhe_rsa_aes_256_cbc_sha`
`rsa_aes_128_sha`
`rsa_aes_256_sha`
### ECC ciphers
`ecdh_ecdsa_null_sha`
`ecdh_ecdsa_rc4_128_sha`
`ecdh_ecdsa_3des_sha`
`ecdh_ecdsa_aes_128_sha`
`ecdh_ecdsa_aes_256_sha`
`ecdhe_ecdsa_null_sha`
`ecdhe_ecdsa_rc4_128_sha`
`ecdhe_ecdsa_3des_sha`
`ecdhe_ecdsa_aes_128_sha`
`ecdhe_ecdsa_aes_256_sha`
`ecdh_rsa_null_sha`
`ecdh_rsa_128_sha`
`ecdh_rsa_3des_sha`
`ecdh_rsa_aes_128_sha`
`ecdh_rsa_aes_256_sha`
`ecdhe_rsa_null`
`ecdhe_rsa_rc4_128_sha`
`ecdhe_rsa_3des_sha`
`ecdhe_rsa_aes_128_sha`
`ecdhe_rsa_aes_256_sha`
`ecdh_anon_null_sha`
`ecdh_anon_rc4_128sha`
`ecdh_anon_3des_sha`
`ecdh_anon_aes_128_sha`
`ecdh_anon_aes_256_sha`
### HMAC-SHA256 cipher suites
`rsa_null_sha_256`
`rsa_aes_128_cbc_sha_256`
`rsa_aes_256_cbc_sha_256`
`dhe_rsa_aes_128_cbc_sha_256`
`dhe_rsa_aes_256_cbc_sha_256`
`ecdhe_ecdsa_aes_128_cbc_sha_256`
`ecdhe_rsa_aes_128_cbc_sha_256`
### AES GCM cipher suites in RFC 5288 and RFC 5289
`rsa_aes_128_gcm_sha_256`
`dhe_rsa_aes_128_gcm_sha_256`
`dhe_dss_aes_128_gcm_sha_256`
`ecdhe_ecdsa_aes_128_gcm_sha_256`
`ecdh_ecdsa_aes_128_gcm_sha_256`
`ecdhe_rsa_aes_128_gcm_sha_256`
`ecdh_rsa_aes_128_gcm_sha_256`
### cipher suites using SHA384
`rsa_aes_256_gcm_sha_384`
`dhe_rsa_aes_256_gcm_sha_384`
`dhe_dss_aes_256_gcm_sha_384`
`ecdhe_ecdsa_aes_256_sha_384`
`ecdhe_rsa_aes_256_sha_384`
`ecdhe_ecdsa_aes_256_gcm_sha_384`
`ecdhe_rsa_aes_256_gcm_sha_384`
### chacha20-poly1305 cipher suites
`ecdhe_rsa_chacha20_poly1305_sha_256`
`ecdhe_ecdsa_chacha20_poly1305_sha_256`
`dhe_rsa_chacha20_poly1305_sha_256`
### TLS 1.3 cipher suites
`aes_128_gcm_sha_256`
`aes_256_gcm_sha_384`
`chacha20_poly1305_sha_256`
## GSKit
Ciphers are internally defined as [numeric
codes](https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/apis/gsk_attribute_set_buffer.htm). libcurl
maps them to the following case-insensitive names.
### SSL2 cipher suites (insecure: disabled by default)
`rc2-md5`
`rc4-md5`
`exp-rc2-md5`
`exp-rc4-md5`
`des-cbc-md5`
`des-cbc3-md5`
### SSL3 cipher suites
`null-md5`
`null-sha`
`rc4-md5`
`rc4-sha`
`exp-rc2-cbc-md5`
`exp-rc4-md5`
`exp-des-cbc-sha`
`des-cbc3-sha`
### TLS v1.0 cipher suites
`null-md5`
`null-sha`
`rc4-md5`
`rc4-sha`
`exp-rc2-cbc-md5`
`exp-rc4-md5`
`exp-des-cbc-sha`
`des-cbc3-sha`
`aes128-sha`
`aes256-sha`
### TLS v1.1 cipher suites
`null-md5`
`null-sha`
`rc4-md5`
`rc4-sha`
`exp-des-cbc-sha`
`des-cbc3-sha`
`aes128-sha`
`aes256-sha`
### TLS v1.2 cipher suites
`null-md5`
`null-sha`
`null-sha256`
`rc4-md5`
`rc4-sha`
`des-cbc3-sha`
`aes128-sha`
`aes256-sha`
`aes128-sha256`
`aes256-sha256`
`aes128-gcm-sha256`
`aes256-gcm-sha384`
## WolfSSL
`RC4-SHA`,
`RC4-MD5`,
`DES-CBC3-SHA`,
`AES128-SHA`,
`AES256-SHA`,
`NULL-SHA`,
`NULL-SHA256`,
`DHE-RSA-AES128-SHA`,
`DHE-RSA-AES256-SHA`,
`DHE-PSK-AES256-GCM-SHA384`,
`DHE-PSK-AES128-GCM-SHA256`,
`PSK-AES256-GCM-SHA384`,
`PSK-AES128-GCM-SHA256`,
`DHE-PSK-AES256-CBC-SHA384`,
`DHE-PSK-AES128-CBC-SHA256`,
`PSK-AES256-CBC-SHA384`,
`PSK-AES128-CBC-SHA256`,
`PSK-AES128-CBC-SHA`,
`PSK-AES256-CBC-SHA`,
`DHE-PSK-AES128-CCM`,
`DHE-PSK-AES256-CCM`,
`PSK-AES128-CCM`,
`PSK-AES256-CCM`,
`PSK-AES128-CCM-8`,
`PSK-AES256-CCM-8`,
`DHE-PSK-NULL-SHA384`,
`DHE-PSK-NULL-SHA256`,
`PSK-NULL-SHA384`,
`PSK-NULL-SHA256`,
`PSK-NULL-SHA`,
`HC128-MD5`,
`HC128-SHA`,
`HC128-B2B256`,
`AES128-B2B256`,
`AES256-B2B256`,
`RABBIT-SHA`,
`NTRU-RC4-SHA`,
`NTRU-DES-CBC3-SHA`,
`NTRU-AES128-SHA`,
`NTRU-AES256-SHA`,
`AES128-CCM-8`,
`AES256-CCM-8`,
`ECDHE-ECDSA-AES128-CCM`,
`ECDHE-ECDSA-AES128-CCM-8`,
`ECDHE-ECDSA-AES256-CCM-8`,
`ECDHE-RSA-AES128-SHA`,
`ECDHE-RSA-AES256-SHA`,
`ECDHE-ECDSA-AES128-SHA`,
`ECDHE-ECDSA-AES256-SHA`,
`ECDHE-RSA-RC4-SHA`,
`ECDHE-RSA-DES-CBC3-SHA`,
`ECDHE-ECDSA-RC4-SHA`,
`ECDHE-ECDSA-DES-CBC3-SHA`,
`AES128-SHA256`,
`AES256-SHA256`,
`DHE-RSA-AES128-SHA256`,
`DHE-RSA-AES256-SHA256`,
`ECDH-RSA-AES128-SHA`,
`ECDH-RSA-AES256-SHA`,
`ECDH-ECDSA-AES128-SHA`,
`ECDH-ECDSA-AES256-SHA`,
`ECDH-RSA-RC4-SHA`,
`ECDH-RSA-DES-CBC3-SHA`,
`ECDH-ECDSA-RC4-SHA`,
`ECDH-ECDSA-DES-CBC3-SHA`,
`AES128-GCM-SHA256`,
`AES256-GCM-SHA384`,
`DHE-RSA-AES128-GCM-SHA256`,
`DHE-RSA-AES256-GCM-SHA384`,
`ECDHE-RSA-AES128-GCM-SHA256`,
`ECDHE-RSA-AES256-GCM-SHA384`,
`ECDHE-ECDSA-AES128-GCM-SHA256`,
`ECDHE-ECDSA-AES256-GCM-SHA384`,
`ECDH-RSA-AES128-GCM-SHA256`,
`ECDH-RSA-AES256-GCM-SHA384`,
`ECDH-ECDSA-AES128-GCM-SHA256`,
`ECDH-ECDSA-AES256-GCM-SHA384`,
`CAMELLIA128-SHA`,
`DHE-RSA-CAMELLIA128-SHA`,
`CAMELLIA256-SHA`,
`DHE-RSA-CAMELLIA256-SHA`,
`CAMELLIA128-SHA256`,
`DHE-RSA-CAMELLIA128-SHA256`,
`CAMELLIA256-SHA256`,
`DHE-RSA-CAMELLIA256-SHA256`,
`ECDHE-RSA-AES128-SHA256`,
`ECDHE-ECDSA-AES128-SHA256`,
`ECDH-RSA-AES128-SHA256`,
`ECDH-ECDSA-AES128-SHA256`,
`ECDHE-RSA-AES256-SHA384`,
`ECDHE-ECDSA-AES256-SHA384`,
`ECDH-RSA-AES256-SHA384`,
`ECDH-ECDSA-AES256-SHA384`,
`ECDHE-RSA-CHACHA20-POLY1305`,
`ECDHE-ECDSA-CHACHA20-POLY1305`,
`DHE-RSA-CHACHA20-POLY1305`,
`ECDHE-RSA-CHACHA20-POLY1305-OLD`,
`ECDHE-ECDSA-CHACHA20-POLY1305-OLD`,
`DHE-RSA-CHACHA20-POLY1305-OLD`,
`ADH-AES128-SHA`,
`QSH`,
`RENEGOTIATION-INFO`,
`IDEA-CBC-SHA`,
`ECDHE-ECDSA-NULL-SHA`,
`ECDHE-PSK-NULL-SHA256`,
`ECDHE-PSK-AES128-CBC-SHA256`,
`PSK-CHACHA20-POLY1305`,
`ECDHE-PSK-CHACHA20-POLY1305`,
`DHE-PSK-CHACHA20-POLY1305`,
`EDH-RSA-DES-CBC3-SHA`,
## Schannel
Schannel allows the enabling and disabling of encryption algorithms, but not
specific cipher suites. They are
[defined](https://docs.microsoft.com/windows/desktop/SecCrypto/alg-id) by
Microsoft.
There is also the case that the selected algorithm is not supported by the
protocol or does not match the ciphers offered by the server during the SSL
negotiation. In this case curl will return error
`CURLE_SSL_CONNECT_ERROR (35) SEC_E_ALGORITHM_MISMATCH`
and the request will fail.
`CALG_MD2`,
`CALG_MD4`,
`CALG_MD5`,
`CALG_SHA`,
`CALG_SHA1`,
`CALG_MAC`,
`CALG_RSA_SIGN`,
`CALG_DSS_SIGN`,
`CALG_NO_SIGN`,
`CALG_RSA_KEYX`,
`CALG_DES`,
`CALG_3DES_112`,
`CALG_3DES`,
`CALG_DESX`,
`CALG_RC2`,
`CALG_RC4`,
`CALG_SEAL`,
`CALG_DH_SF`,
`CALG_DH_EPHEM`,
`CALG_AGREEDKEY_ANY`,
`CALG_HUGHES_MD5`,
`CALG_SKIPJACK`,
`CALG_TEK`,
`CALG_CYLINK_MEK`,
`CALG_SSL3_SHAMD5`,
`CALG_SSL3_MASTER`,
`CALG_SCHANNEL_MASTER_HASH`,
`CALG_SCHANNEL_MAC_KEY`,
`CALG_SCHANNEL_ENC_KEY`,
`CALG_PCT1_MASTER`,
`CALG_SSL2_MASTER`,
`CALG_TLS1_MASTER`,
`CALG_RC5`,
`CALG_HMAC`,
`CALG_TLS1PRF`,
`CALG_HASH_REPLACE_OWF`,
`CALG_AES_128`,
`CALG_AES_192`,
`CALG_AES_256`,
`CALG_AES`,
`CALG_SHA_256`,
`CALG_SHA_384`,
`CALG_SHA_512`,
`CALG_ECDH`,
`CALG_ECMQV`,
`CALG_ECDSA`,
`CALG_ECDH_EPHEM`,
As of curl 7.77.0, you can also pass `SCH_USE_STRONG_CRYPTO` as a cipher name
to [constrain the set of available ciphers as specified in the Schannel
documentation](https://docs.microsoft.com/en-us/windows/win32/secauthn/tls-cipher-suites-in-windows-server-2022).
Note that the supported ciphers in this case follow the OS version, so if you
are running an outdated OS you might still be supporting weak ciphers.
### TLS 1.3 cipher suites
(Note these ciphers are set with `CURLOPT_TLS13_CIPHERS` and `--tls13-ciphers`)
`TLS_AES_256_GCM_SHA384`
`TLS_AES_128_GCM_SHA256`
`TLS_CHACHA20_POLY1305_SHA256`
`TLS_AES_128_CCM_8_SHA256`
`TLS_AES_128_CCM_SHA256`
## BearSSL
BearSSL ciphers can be specified by either the OpenSSL name (`ECDHE-RSA-AES128-GCM-SHA256`) or the IANA name (`TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256`).
Since BearSSL 0.1:
`DES-CBC3-SHA`
`AES128-SHA`
`AES256-SHA`
`AES128-SHA256`
`AES256-SHA256`
`AES128-GCM-SHA256`
`AES256-GCM-SHA384`
`ECDH-ECDSA-DES-CBC3-SHA`
`ECDH-ECDSA-AES128-SHA`
`ECDH-ECDSA-AES256-SHA`
`ECDHE-ECDSA-DES-CBC3-SHA`
`ECDHE-ECDSA-AES128-SHA`
`ECDHE-ECDSA-AES256-SHA`
`ECDH-RSA-DES-CBC3-SHA`
`ECDH-RSA-AES128-SHA`
`ECDH-RSA-AES256-SHA`
`ECDHE-RSA-DES-CBC3-SHA`
`ECDHE-RSA-AES128-SHA`
`ECDHE-RSA-AES256-SHA`
`ECDHE-ECDSA-AES128-SHA256`
`ECDHE-ECDSA-AES256-SHA384`
`ECDH-ECDSA-AES128-SHA256`
`ECDH-ECDSA-AES256-SHA384`
`ECDHE-RSA-AES128-SHA256`
`ECDHE-RSA-AES256-SHA384`
`ECDH-RSA-AES128-SHA256`
`ECDH-RSA-AES256-SHA384`
`ECDHE-ECDSA-AES128-GCM-SHA256`
`ECDHE-ECDSA-AES256-GCM-SHA384`
`ECDH-ECDSA-AES128-GCM-SHA256`
`ECDH-ECDSA-AES256-GCM-SHA384`
`ECDHE-RSA-AES128-GCM-SHA256`
`ECDHE-RSA-AES256-GCM-SHA384`
`ECDH-RSA-AES128-GCM-SHA256`
`ECDH-RSA-AES256-GCM-SHA384`
Since BearSSL 0.2:
`ECDHE-RSA-CHACHA20-POLY1305`
`ECDHE-ECDSA-CHACHA20-POLY1305`
Since BearSSL 0.6:
`AES128-CCM`
`AES256-CCM`
`AES128-CCM8`
`AES256-CCM8`
`ECDHE-ECDSA-AES128-CCM`
`ECDHE-ECDSA-AES256-CCM`
`ECDHE-ECDSA-AES128-CCM8`
`ECDHE-ECDSA-AES256-CCM8`
---
### Third Party Curl Docs CMakeLists (`third_party/curl/docs/CMakeLists.txt`)
#***************************************************************************
# _ _ ____ _
# Project ___| | | | _ \| |
# / __| | | | |_) | |
# | (__| |_| | _ <| |___
# \___|\___/|_| \_\_____|
#
# Copyright (C) Daniel Stenberg, , et al.
#
# This software is licensed as described in the file COPYING, which
# you should have received as part of this distribution. The terms
# are also available at https://curl.se/docs/copyright.html.
#
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
# copies of the Software, and permit persons to whom the Software is
# furnished to do so, under the terms of the COPYING file.
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
# KIND, either express or implied.
#
# SPDX-License-Identifier: curl
#
###########################################################################
#add_subdirectory(examples)
add_subdirectory(libcurl)
add_subdirectory(cmdline-opts)
---
### Third Party Curl Docs CODE OF CONDUCT (`third_party/curl/docs/CODE_OF_CONDUCT.md`)
Contributor Code of Conduct
===========================
As contributors and maintainers of this project, we pledge to respect all
people who contribute through reporting issues, posting feature requests,
updating documentation, submitting pull requests or patches, and other
activities.
We are committed to making participation in this project a harassment-free
experience for everyone, regardless of level of experience, gender, gender
identity and expression, sexual orientation, disability, personal appearance,
body size, race, ethnicity, age, or religion.
Examples of unacceptable behavior by participants include the use of sexual
language or imagery, derogatory comments or personal attacks, trolling, public
or private harassment, insults, or other unprofessional conduct.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct. Project maintainers who do not
follow the Code of Conduct may be removed from the project team.
This code of conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community.
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by opening an issue or contacting one or more of the project
maintainers.
This Code of Conduct is adapted from the [Contributor
Covenant](https://contributor-covenant.org/), version 1.1.0, available at
[https://contributor-covenant.org/version/1/1/0/](https://contributor-covenant.org/version/1/1/0/)
---
### Third Party Curl Docs CODE REVIEW (`third_party/curl/docs/CODE_REVIEW.md`)
# How to do code reviews for curl
Anyone and everyone is encouraged and welcome to review code submissions in
curl. This is a guide on what to check for and how to perform a successful
code review.
## All submissions should get reviewed
All pull requests and patches submitted to the project should be reviewed by
at least one experienced curl maintainer before that code is accepted and
merged.
## Let the tools and tests take the first rounds
On initial pull requests, let the tools and tests do their job first and then
start out by helping the submitter understand the test failures and tool
alerts.
## How to provide feedback to author
Be nice. Ask questions. Provide examples or suggestions of improvements.
Assume the best intentions. Remember language barriers.
All first-time contributors can become regulars. Let's help them go there.
## Is this a change we want?
If this is not a change that seems to be aligned with the project's path
forward and as such cannot be accepted, inform the author about this sooner
rather than later. Do it gently and explain why and possibly what could be
done to make it more acceptable.
## API/ABI stability or changed behavior
Changing the API and the ABI may be fine in a change but it needs to be done
deliberately and carefully. If not, a reviewer must help the author to realize
the mistake.
curl and libcurl are similarly strict on not modifying existing behavior. API
and ABI stability is not enough, the behavior should also remain intact as far
as possible.
## Code style
Most code style nits are detected by checksrc but not all. Only leave remarks
on style deviation once checksrc does not find anymore.
Minor nits from fresh submitters can also be handled by the maintainer when
merging, in case it seems like the submitter is not clear on what to do. We
want to make the process fun and exciting for new contributors.
## Encourage consistency
Make sure new code is written in a similar style as existing code. Naming,
logic, conditions, etc.
## Are pointers always non-NULL?
If a function or code rely on pointers being non-NULL, take an extra look if
that seems to be a fair assessment.
## Asserts
Conditions that should never be false can be verified with `DEBUGASSERT()`
calls to get caught in tests and debugging easier, while not having an impact
on final or release builds.
## Memory allocation
Can the mallocs be avoided? Do not introduce mallocs in any hot paths. If
there are (new) mallocs, can they be combined into fewer calls?
Are all allocations handled in error paths to avoid leaks and crashes?
## Thread-safety
We do not like static variables as they break thread-safety and prevent
functions from being reentrant.
## Should features be `#ifdef`ed?
Features and functionality may not be present everywhere and should therefore
be `#ifdef`ed. Additionally, some features should be possible to switch on/off
in the build.
Write `#ifdef`s to be as little of a "maze" as possible.
## Does it look portable enough?
curl runs "everywhere". Does the code take a reasonable stance and enough
precautions to be possible to build and run on most platforms?
Remember that we live by C89 restrictions.
## Tests and testability
New features should be added in conjunction with one or more test cases.
Ideally, functions should also be written so that unit tests can be done to
test individual functions.
## Documentation
New features or changes to existing functionality **must** be accompanied by
updated documentation. Submitting that in a separate follow-up pull request is
not OK. A code review must also verify that the submitted documentation update
matches the code submission.
English is not everyone's first language, be mindful of this and help the
submitter improve the text if it needs a rewrite to read better.
## Code should not be hard to understand
Source code should be written to maximize readability and be easy to
understand.
## Functions should not be large
A single function should never be large as that makes it hard to follow and
understand all the exit points and state changes. Some existing functions in
curl certainly violate this ground rule but when reviewing new code we should
propose splitting into smaller functions.
## Duplication is evil
Anything that looks like duplicated code is a red flag. Anything that seems to
introduce code that we *should* already have or provide needs a closer check.
## Sensitive data
When credentials are involved, take an extra look at what happens with this
data. Where it comes from and where it goes.
## Variable types differ
`size_t` is not a fixed size. `time_t` can be signed or unsigned and have
different sizes. Relying on variable sizes is a red flag.
Also remember that endianness and >= 32 bit accesses to unaligned addresses
are problematic areas.
## Integer overflows
Be careful about integer overflows. Some variable types can be either 32 bit
or 64 bit. Integer overflows must be detected and acted on *before* they
happen.
## Dangerous use of functions
Maybe use of `realloc()` should rather use the dynbuf functions?
Do not allow new code that grows buffers without using dynbuf.
Use of C functions that rely on a terminating zero must only be used on data
that really do have a null-terminating zero.
## Dangerous "data styles"
Make extra precautions and verify that memory buffers that need a terminating
zero always have exactly that. Buffers *without* a null-terminator must not be
used as input to string functions.
# Commit messages
Tightly coupled with a code review is making sure that the commit message is
good. It is the responsibility of the person who merges the code to make sure
that the commit message follows our standard (detailed in the
[CONTRIBUTE](CONTRIBUTE.md) document). This includes making sure the PR
identifies related issues and giving credit to reporters and helpers.
---
### Third Party Curl Docs CODE STYLE (`third_party/curl/docs/CODE_STYLE.md`)
# curl C code style
Source code that has a common style is easier to read than code that uses
different styles in different places. It helps making the code feel like one
single code base. Easy-to-read is an important property of code and helps
making it easier to review when new things are added and it helps debugging
code when developers are trying to figure out why things go wrong. A unified
style is more important than individual contributors having their own personal
tastes satisfied.
Our C code has a few style rules. Most of them are verified and upheld by the
`scripts/checksrc.pl` script. Invoked with `make checksrc` or even by default
by the build system when built after `./configure --enable-debug` has been
used.
It is normally not a problem for anyone to follow the guidelines, as you just
need to copy the style already used in the source code and there are no
particularly unusual rules in our set of rules.
We also work hard on writing code that are warning-free on all the major
platforms and in general on as many platforms as possible. Code that obviously
will cause warnings will not be accepted as-is.
## Naming
Try using a non-confusing naming scheme for your new functions and variable
names. It does not necessarily have to mean that you should use the same as in
other places of the code, just that the names should be logical,
understandable and be named according to what they are used for. File-local
functions should be made static. We like lower case names.
See the [INTERNALS](https://curl.se/dev/internals.html#symbols) document on
how we name non-exported library-global symbols.
## Indenting
We use only spaces for indentation, never TABs. We use two spaces for each new
open brace.
```c
if(something_is_true) {
while(second_statement == fine) {
moo();
}
}
```
## Comments
Since we write C89 code, **//** comments are not allowed. They were not
introduced in the C standard until C99. We use only __/* comments */__.
```c
/* this is a comment */
```
## Long lines
Source code in curl may never be wider than 79 columns and there are two
reasons for maintaining this even in the modern era of large and high
resolution screens:
1. Narrower columns are easier to read than wide ones. There's a reason
newspapers have used columns for decades or centuries.
2. Narrower columns allow developers to easier show multiple pieces of code
next to each other in different windows. It allows two or three source
code windows next to each other on the same screen - as well as multiple
terminal and debugging windows.
## Braces
In if/while/do/for expressions, we write the open brace on the same line as
the keyword and we then set the closing brace on the same indentation level as
the initial keyword. Like this:
```c
if(age < 40) {
/* clearly a youngster */
}
```
You may omit the braces if they would contain only a one-line statement:
```c
if(!x)
continue;
```
For functions the opening brace should be on a separate line:
```c
int main(int argc, char **argv)
{
return 1;
}
```
## 'else' on the following line
When adding an **else** clause to a conditional expression using braces, we
add it on a new line after the closing brace. Like this:
```c
if(age < 40) {
/* clearly a youngster */
}
else {
/* probably grumpy */
}
```
## No space before parentheses
When writing expressions using if/while/do/for, there shall be no space
between the keyword and the open parenthesis. Like this:
```c
while(1) {
/* loop forever */
}
```
## Use boolean conditions
Rather than test a conditional value such as a bool against TRUE or FALSE, a
pointer against NULL or != NULL and an int against zero or not zero in
if/while conditions we prefer:
```c
result = do_something();
if(!result) {
/* something went wrong */
return result;
}
```
## No assignments in conditions
To increase readability and reduce complexity of conditionals, we avoid
assigning variables within if/while conditions. We frown upon this style:
```c
if((ptr = malloc(100)) == NULL)
return NULL;
```
and instead we encourage the above version to be spelled out more clearly:
```c
ptr = malloc(100);
if(!ptr)
return NULL;
```
## New block on a new line
We never write multiple statements on the same source line, even for short
if() conditions.
```c
if(a)
return TRUE;
else if(b)
return FALSE;
```
and NEVER:
```c
if(a) return TRUE;
else if(b) return FALSE;
```
## Space around operators
Please use spaces on both sides of operators in C expressions. Postfix **(),
[], ->, ., ++, --** and Unary **+, -, !, ~, &** operators excluded they should
have no space.
Examples:
```c
bla = func();
who = name[0];
age += 1;
true = !false;
size += -2 + 3 * (a + b);
ptr->member = a++;
struct.field = b--;
ptr = &address;
contents = *pointer;
complement = ~bits;
empty = (!*string) ? TRUE : FALSE;
```
## No parentheses for return values
We use the 'return' statement without extra parentheses around the value:
```c
int works(void)
{
return TRUE;
}
```
## Parentheses for sizeof arguments
When using the sizeof operator in code, we prefer it to be written with
parentheses around its argument:
```c
int size = sizeof(int);
```
## Column alignment
Some statements cannot be completed on a single line because the line would be
too long, the statement too hard to read, or due to other style guidelines
above. In such a case the statement will span multiple lines.
If a continuation line is part of an expression or sub-expression then you
should align on the appropriate column so that it's easy to tell what part of
the statement it is. Operators should not start continuation lines. In other
cases follow the 2-space indent guideline. Here are some examples from
libcurl:
```c
if(Curl_pipeline_wanted(handle->multi, CURLPIPE_HTTP1) &&
(handle->set.httpversion != CURL_HTTP_VERSION_1_0) &&
(handle->set.httpreq == HTTPREQ_GET ||
handle->set.httpreq == HTTPREQ_HEAD))
/* did not ask for HTTP/1.0 and a GET or HEAD */
return TRUE;
```
If no parenthesis, use the default indent:
```c
data->set.http_disable_hostname_check_before_authentication =
(0 != va_arg(param, long)) ? TRUE : FALSE;
```
Function invoke with an open parenthesis:
```c
if(option) {
result = parse_login_details(option, strlen(option),
(userp ? &user : NULL),
(passwdp ? &passwd : NULL),
NULL);
}
```
Align with the "current open" parenthesis:
```c
DEBUGF(infof(data, "Curl_pp_readresp_ %d bytes of trailing "
"server response left\n",
(int)clipamount));
```
## Platform dependent code
Use **#ifdef HAVE_FEATURE** to do conditional code. We avoid checking for
particular operating systems or hardware in the #ifdef lines. The HAVE_FEATURE
shall be generated by the configure script for unix-like systems and they are
hard-coded in the `config-[system].h` files for the others.
We also encourage use of macros/functions that possibly are empty or defined
to constants when libcurl is built without that feature, to make the code
seamless. Like this example where the **magic()** function works differently
depending on a build-time conditional:
```c
#ifdef HAVE_MAGIC
void magic(int a)
{
return a + 2;
}
#else
#define magic(x) 1
#endif
int content = magic(3);
```
## No typedefed structs
Use structs by all means, but do not typedef them. Use the `struct name` way
of identifying them:
```c
struct something {
void *valid;
size_t way_to_write;
};
struct something instance;
```
**Not okay**:
```c
typedef struct {
void *wrong;
size_t way_to_write;
} something;
something instance;
```
---
### Third Party Curl Docs CONNECTION FILTERS (`third_party/curl/docs/CONNECTION-FILTERS.md`)
# curl connection filters
Connection filters is a design in the internals of curl, not visible in its public API. They were added
in curl v7.xx.x. This document describes the concepts, its high level implementation and the motivations.
## Filters
A "connection filter" is a piece of code that is responsible for handling a range of operations
of curl's connections: reading, writing, waiting on external events, connecting and closing down - to name the most important ones.
The most important feat of connection filters is that they can be stacked on top of each other (or "chained" if you prefer that metaphor). In the common scenario that you want to retrieve a `https:` url with curl, you need 2 basic things to send the request and get the response: a TCP connection, represented by a `socket` and a SSL instance en- and decrypt over that socket. You write your request to the SSL instance, which encrypts and writes that data to the socket, which then sends the bytes over the network.
With connection filters, curl's internal setup will look something like this (cf for connection filter):
```
Curl_easy *data connectdata *conn cf-ssl cf-socket
+----------------+ +-----------------+ +-------+ +--------+
|https://curl.se/|----> | properties |----> | keys |---> | socket |--> OS --> network
+----------------+ +-----------------+ +-------+ +--------+
Curl_write(data, buffer)
--> Curl_cfilter_write(data, data->conn, buffer)
---> conn->filter->write(conn->filter, data, buffer)
```
While connection filters all do different things, they look the same from the "outside". The code in `data` and `conn` does not really know **which** filters are installed. `conn` just writes into the first filter, whatever that is.
Same is true for filters. Each filter has a pointer to the `next` filter. When SSL has encrypted the data, it does not write to a socket, it writes to the next filter. If that is indeed a socket, or a file, or an HTTP/2 connection is of no concern to the SSL filter.
And this allows the stacking, as in:
```
Direct:
http://localhost/ conn -> cf-socket
https://curl.se/ conn -> cf-ssl -> cf-socket
Via http proxy tunnel:
http://localhost/ conn -> cf-http-proxy -> cf-socket
https://curl.se/ conn -> cf-ssl -> cf-http-proxy -> cf-socket
Via https proxy tunnel:
http://localhost/ conn -> cf-http-proxy -> cf-ssl -> cf-socket
https://curl.se/ conn -> cf-ssl -> cf-http-proxy -> cf-ssl -> cf-socket
Via http proxy tunnel via SOCKS proxy:
http://localhost/ conn -> cf-http-proxy -> cf-socks -> cf-socket
```
### Connecting/Closing
Before `Curl_easy` can send the request, the connection needs to be established. This means that all connection filters have done, whatever they need to do: waiting for the socket to be connected, doing the TLS handshake, performing the HTTP tunnel request, etc. This has to be done in reverse order: the last filter has to do its connect first, then the one above can start, etc.
Each filter does in principle the following:
```
static CURLcode
myfilter_cf_connect(struct Curl_cfilter *cf,
struct Curl_easy *data,
bool *done)
{
CURLcode result;
if(cf->connected) { /* we and all below are done */
*done = TRUE;
return CURLE_OK;
}
/* Let the filters below connect */
result = cf->next->cft->connect(cf->next, data, blocking, done);
if(result || !*done)
return result; /* below errored/not finished yet */
/* MYFILTER CONNECT THINGS */ /* below connected, do out thing */
*done = cf->connected = TRUE; /* done, remember, return */
return CURLE_OK;
}
```
Closing a connection then works similar. The `conn` tells the first filter to close. Contrary to connecting,
the filter does its own things first, before telling the next filter to close.
### Efficiency
There are two things curl is concerned about: efficient memory use and fast transfers.
The memory footprint of a filter is relatively small:
```
struct Curl_cfilter {
const struct Curl_cftype *cft; /* the type providing implementation */
struct Curl_cfilter *next; /* next filter in chain */
void *ctx; /* filter type specific settings */
struct connectdata *conn; /* the connection this filter belongs to */
int sockindex; /* TODO: like to get rid off this */
BIT(connected); /* != 0 iff this filter is connected */
};
```
The filter type `cft` is a singleton, one static struct for each type of filter. The `ctx` is where a filter will hold its specific data. That varies by filter type. An http-proxy filter will keep the ongoing state of the CONNECT here, but free it after its has been established. The SSL filter will keep the `SSL*` (if OpenSSL is used) here until the connection is closed. So, this varies.
`conn` is a reference to the connection this filter belongs to, so nothing extra besides the pointer itself.
Several things, that before were kept in `struct connectdata`, will now go into the `filter->ctx` *when needed*. So, the memory footprint for connections that do *not* use an http proxy, or socks, or https will be lower.
As to transfer efficiency, writing and reading through a filter comes at near zero cost *if the filter does not transform the data*. An http proxy or socks filter, once it is connected, will just pass the calls through. Those filters implementations will look like this:
```
ssize_t Curl_cf_def_send(struct Curl_cfilter *cf, struct Curl_easy *data,
const void *buf, size_t len, CURLcode *err)
{
return cf->next->cft->do_send(cf->next, data, buf, len, err);
}
```
The `recv` implementation is equivalent.
## Filter Types
The (currently) existing filter types are: SOCKET, SOCKET-ACCEPT, SSL, HTTP-PROXY and SOCKS-PROXY. Vital to establishing and read/writing a connection. But filters are also a good way to implement tasks for *managing* a connection:
* **Statistics**: a filter that counts the number of bytes sent/received. Place one in front of SOCKET and one higher up and get the number of raw and "easy" bytes transferred. They may track the speed as well, or number of partial writes, etc.
* **Timeout**: enforce timeouts, e.g. fail if a connection cannot be established in a certain amount of time.
* **Progress**: report progress on a connection.
* **Pacing**: limit read/write rates.
* **Testing**: simulate network condition or failures.
As you see, filters are a good way to add functionality to curl's internal handling of transfers without impact on other code.
## Easy Filters?
Some things that curl needs to manage are not directly tied to a specific connection but the property of the `Curl_easy` handle, e.g. a particular transfer. When using HTTP/2 or HTTP/3, many transfers can use the same connection. If one wants to monitor of the transfer itself or restricting its speed alone, a connection filter is not the right place to do this.
So we might add "easy filters" one day. Who knows?
---
### Third Party Curl Docs CONTRIBUTE (`third_party/curl/docs/CONTRIBUTE.md`)
# Contributing to the curl project
This document is intended to offer guidelines on how to best contribute to the
curl project. This concerns new features as well as corrections to existing
flaws or bugs.
## Join the Community
Skip over to [https://curl.se/mail/](https://curl.se/mail/) and join
the appropriate mailing list(s). Read up on details before you post
questions. Read this file before you start sending patches. We prefer
questions sent to and discussions being held on the mailing list(s), not sent
to individuals.
Before posting to one of the curl mailing lists, please read up on the
[mailing list etiquette](https://curl.se/mail/etiquette.html).
We also hang out on IRC in #curl on libera.chat
If you are at all interested in the code side of things, consider clicking
'watch' on the [curl repo on GitHub](https://github.com/curl/curl) to be
notified of pull requests and new issues posted there.
## License and copyright
When contributing with code, you agree to put your changes and new code under
the same license curl and libcurl is already using unless stated and agreed
otherwise.
If you add a larger piece of code, you can opt to make that file or set of
files to use a different license as long as they do not enforce any changes to
the rest of the package and they make sense. Such "separate parts" can not be
GPL licensed (as we do not want copyleft to affect users of libcurl) but they
must use "GPL compatible" licenses (as we want to allow users to use libcurl
properly in GPL licensed environments).
When changing existing source code, you do not alter the copyright of the
original file(s). The copyright will still be owned by the original creator(s)
or those who have been assigned copyright by the original author(s).
By submitting a patch to the curl project, you are assumed to have the right
to the code and to be allowed by your employer or whatever to hand over that
patch/code to us. We will credit you for your changes as far as possible, to
give credit but also to keep a trace back to who made what changes. Please
always provide us with your full real name when contributing,
## What To Read
Source code, the man pages, the [INTERNALS
document](https://curl.se/dev/internals.html),
[TODO](https://curl.se/docs/todo.html),
[KNOWN_BUGS](https://curl.se/docs/knownbugs.html) and the [most recent
changes](https://curl.se/dev/sourceactivity.html) in git. Just lurking on
the [curl-library mailing
list](https://curl.se/mail/list.cgi?list=curl-library) will give you a
lot of insights on what's going on right now. Asking there is a good idea too.
## Write a good patch
### Follow code style
When writing C code, follow the
[CODE_STYLE](https://curl.se/dev/code-style.html) already established in
the project. Consistent style makes code easier to read and mistakes less
likely to happen. Run `make checksrc` before you submit anything, to make sure
you follow the basic style. That script does not verify everything, but if it
complains you know you have work to do.
### Non-clobbering All Over
When you write new functionality or fix bugs, it is important that you do not
fiddle all over the source files and functions. Remember that it is likely
that other people have done changes in the same source files as you have and
possibly even in the same functions. If you bring completely new
functionality, try writing it in a new source file. If you fix bugs, try to
fix one bug at a time and send them as separate patches.
### Write Separate Changes
It is annoying when you get a huge patch from someone that is said to fix 511
odd problems, but discussions and opinions do not agree with 510 of them - or
509 of them were already fixed in a different way. Then the person merging
this change needs to extract the single interesting patch from somewhere
within the huge pile of source, and that creates a lot of extra work.
Preferably, each fix that corrects a problem should be in its own patch/commit
with its own description/commit message stating exactly what they correct so
that all changes can be selectively applied by the maintainer or other
interested parties.
Also, separate changes enable bisecting much better for tracking problems
and regression in the future.
### Patch Against Recent Sources
Please try to get the latest available sources to make your patches against.
It makes the lives of the developers so much easier. The best is if you get
the most up-to-date sources from the git repository, but the latest release
archive is quite OK as well.
### Documentation
Writing docs is dead boring and one of the big problems with many open source
projects. But someone's gotta do it. It makes things a lot easier if you
submit a small description of your fix or your new features with every
contribution so that it can be swiftly added to the package documentation.
The documentation is always made in man pages (nroff formatted) or plain
ASCII files. All HTML files on the website and in the release archives are
generated from the nroff/ASCII versions.
### Test Cases
Since the introduction of the test suite, we can quickly verify that the main
features are working as they are supposed to. To maintain this situation and
improve it, all new features and functions that are added need to be tested
in the test suite. Every feature that is added should get at least one valid
test case that verifies that it works as documented. If every submitter also
posts a few test cases, it will not end up as a heavy burden on a single person!
If you do not have test cases or perhaps you have done something that is hard
to write tests for, do explain exactly how you have otherwise tested and
verified your changes.
## Submit Your Changes
### How to get your changes into the main sources
Ideally you file a [pull request on
GitHub](https://github.com/curl/curl/pulls), but you can also send your plain
patch to [the curl-library mailing
list](https://curl.se/mail/list.cgi?list=curl-library).
If you opt to post a patch on the mailing list, chances are someone will
convert it into a pull request for you, to have the CI jobs verify it proper
before it can be merged. Be prepared that some feedback on the proposed change
might then come on GitHub.
Your change will be reviewed and discussed and you will be expected to correct
flaws pointed out and update accordingly, or the change risks stalling and
eventually just getting deleted without action. As a submitter of a change,
you are the owner of that change until it has been merged.
Respond on the list or on GitHub about the change and answer questions and/or
fix nits/flaws. This is important. We will take lack of replies as a sign that
you are not anxious to get your patch accepted and we tend to simply drop such
changes.
### About pull requests
With GitHub it is easy to send a [pull
request](https://github.com/curl/curl/pulls) to the curl project to have
changes merged.
We strongly prefer pull requests to mailed patches, as it makes it a proper
git commit that is easy to merge and they are easy to track and not that easy
to lose in the flood of many emails, like they sometimes do on the mailing
lists.
Every pull request submitted will automatically be tested in several different
ways. [See the CI document for more
information](https://github.com/curl/curl/blob/master/tests/CI.md).
Sometimes the tests fail due to a dependency service temporarily being offline
or otherwise unavailable, e.g. package downloads. In this case you can just
try to update your pull requests to rerun the tests later as described below.
You can update your pull requests by pushing new commits or force-pushing
changes to existing commits. Force-pushing an amended commit without any
actual content changed also allows you to retrigger the tests for that commit.
When you adjust your pull requests after review, consider squashing the
commits so that we can review the full updated version more easily.
A pull request sent to the project might get labeled `needs-votes` by a
project maintainer. This label means that in addition to meeting all other
checks and qualifications this pull request must also receive more "votes" of
user support. More signs that people want this to happen. It could be in the
form of messages saying so, or thumbs-up reactions on GitHub.
### Making quality changes
Make the patch against as recent source versions as possible.
If you have followed the tips in this document and your patch still has not
been incorporated or responded to after some weeks, consider resubmitting it
to the list or better yet: change it to a pull request.
### Commit messages
A short guide to how to write git commit messages in the curl project.
---- start ----
[area]: [short line describing the main effect]
-- empty line --
[full description, no wider than 72 columns that describes as much as
possible as to why this change is made, and possibly what things
it fixes and everything else that is related, with unwieldy URLs replaced
with references like [0], [1], etc.]
-- empty line --
[[0] URL - Reference to a URL in the description, almost like Markdown;
the last numbered reference is followed by an -- empty line -- ]
[Follow-up to {shorthash} - if this fixes or continues a previous commit;
add a Ref: that commit's PR or issue if it's not a small, obvious fix;
followed by an -- empty line -- ]
[Bug: URL to the source of the report or more related discussion; use Fixes
for GitHub issues instead when that is appropriate]
[Approved-by: John Doe - credit someone who approved the PR; if you are
committing this for someone else using --author=... you don't need this
as you are implicitly approving it by committing]
[Authored-by: John Doe - credit the original author of the code; only use
this if you can't use "git commit --author=..."]
{Signed-off-by: John Doe - we don't use this, but don't bother removing it]
[whatever-else-by: credit all helpers, finders, doers; try to use one of
the following keywords if at all possible, for consistency:
Acked-by:, Assisted-by:, Co-authored-by:, Found-by:, Reported-by:,
Reviewed-by:, Suggested-by:, Tested-by:]
[Ref: #1234 - if this is related to a GitHub issue or PR, possibly one that
has already been closed]
[Ref: URL to more information about the commit; use Bug: instead for
a reference to a bug on another bug tracker]
[Fixes #1234 - if this closes a GitHub issue; GitHub will actually
close the issue once this commit is merged]
[Closes #1234 - if this closes a GitHub PR; GitHub will actually
close the PR once this commit is merged]
---- stop ----
The first line is a succinct description of the change:
- use the imperative, present tense: "change" not "changed" nor "changes"
- do not capitalize the first letter
- no period (.) at the end
The `[area]` in the first line can be `http2`, `cookies`, `openssl` or
similar. There's no fixed list to select from but using the same "area" as
other related changes could make sense.
Do not forget to use commit --author=... if you commit someone else's work, and
make sure that you have your own user and email setup correctly in git before
you commit.
Add whichever header lines as appropriate, with one line per person if more
than one person was involved. There's no need to credit yourself unless you are
using --author=... which hides your identity. Don't include people's e-mail
addresses in headers to avoid spam, unless they're already public from a
previous commit; saying `{userid} on github` is OK.
### Write Access to git Repository
If you are a frequent contributor, you may be given push access to the git
repository and then you will be able to push your changes straight into the git
repo instead of sending changes as pull requests or by mail as patches.
Just ask if this is what you would want. You will be required to have posted
several high quality patches first, before you can be granted push access.
### How To Make a Patch with git
You need to first checkout the repository:
git clone https://github.com/curl/curl.git
You then proceed and edit all the files you like and you commit them to your
local repository:
git commit [file]
As usual, group your commits so that you commit all changes at once that
constitute a logical change.
Once you have done all your commits and you are happy with what you see, you
can make patches out of your changes that are suitable for mailing:
git format-patch remotes/origin/master
This creates files in your local directory named `NNNN-[name].patch` for each
commit.
Now send those patches off to the curl-library list. You can of course opt to
do that with the 'git send-email' command.
### How To Make a Patch without git
Keep a copy of the unmodified curl sources. Make your changes in a separate
source tree. When you think you have something that you want to offer the
curl community, use GNU diff to generate patches.
If you have modified a single file, try something like:
diff -u unmodified-file.c my-changed-one.c > my-fixes.diff
If you have modified several files, possibly in different directories, you
can use diff recursively:
diff -ur curl-original-dir curl-modified-sources-dir > my-fixes.diff
The GNU diff and GNU patch tools exist for virtually all platforms, including
all kinds of Unixes and Windows.
### Useful resources
- [Webinar on getting code into cURL](https://www.youtube.com/watch?v=QmZ3W1d6LQI)
## Update copyright and license information
There is a CI job called **REUSE compliance / check** that will run on every
pull request and commit to verify that the *REUSE state* of all files are
still fine.
This means that all files need to have their license and copyright information
clearly stated. Ideally by having the standard curl source code header, with
an accurate copyright year range and the SPDX-License-Identifier included. If
the header does not work, you can use a smaller header or add the information
for a specific file to the `.reuse/dep5` file.
We update copyright year ranges to end on the year of the most recent change
of the individual file.
You can manually verify the copyright and compliance status by running the
`./scripts/copyright.pl` script in the root of the git repository.
---
### Third Party Curl Docs CURL DISABLE (`third_party/curl/docs/CURL-DISABLE.md`)
# Code defines to disable features and protocols
## `CURL_DISABLE_ALTSVC`
Disable support for Alt-Svc: HTTP headers.
## `CURL_DISABLE_COOKIES`
Disable support for HTTP cookies.
## `CURL_DISABLE_CRYPTO_AUTH`
Disable support for authentication methods using crypto.
## `CURL_DISABLE_DICT`
Disable the DICT protocol
## `CURL_DISABLE_DOH`
Disable DNS-over-HTTPS
## `CURL_DISABLE_FILE`
Disable the FILE protocol
## `CURL_DISABLE_FTP`
Disable the FTP (and FTPS) protocol
## `CURL_DISABLE_GETOPTIONS`
Disable the `curl_easy_options` API calls that lets users get information
about existing options to `curl_easy_setopt`.
## `CURL_DISABLE_GOPHER`
Disable the GOPHER protocol.
## `CURL_DISABLE_HEADERS_API`
Disable the HTTP header API.
## `CURL_DISABLE_HSTS`
Disable the HTTP Strict Transport Security support.
## `CURL_DISABLE_HTTP`
Disable the HTTP(S) protocols. Note that this then also disable HTTP proxy
support.
## `CURL_DISABLE_HTTP_AUTH`
Disable support for all HTTP authentication methods.
## `CURL_DISABLE_IMAP`
Disable the IMAP(S) protocols.
## `CURL_DISABLE_LDAP`
Disable the LDAP(S) protocols.
## `CURL_DISABLE_LDAPS`
Disable the LDAPS protocol.
## `CURL_DISABLE_LIBCURL_OPTION`
Disable the --libcurl option from the curl tool.
## `CURL_DISABLE_MIME`
Disable MIME support.
## `CURL_DISABLE_MQTT`
Disable MQTT support.
## `CURL_DISABLE_NETRC`
Disable the netrc parser.
## `CURL_DISABLE_NTLM`
Disable support for NTLM.
## `CURL_DISABLE_OPENSSL_AUTO_LOAD_CONFIG`
Disable the auto load config support in the OpenSSL backend.
## `CURL_DISABLE_PARSEDATE`
Disable date parsing
## `CURL_DISABLE_POP3`
Disable the POP3 protocol
## `CURL_DISABLE_PROGRESS_METER`
Disable the built-in progress meter
## `CURL_DISABLE_PROXY`
Disable support for proxies
## `CURL_DISABLE_RTSP`
Disable the RTSP protocol.
## `CURL_DISABLE_SHUFFLE_DNS`
Disable the shuffle DNS feature
## `CURL_DISABLE_SMB`
Disable the SMB(S) protocols
## `CURL_DISABLE_SMTP`
Disable the SMTP(S) protocols
## `CURL_DISABLE_SOCKETPAIR`
Disable the use of `socketpair()` internally to allow waking up and canceling
`curl_multi_poll()`.
## `CURL_DISABLE_TELNET`
Disable the TELNET protocol
## `CURL_DISABLE_TFTP`
Disable the TFTP protocol
## `CURL_DISABLE_VERBOSE_STRINGS`
Disable verbose strings and error messages.
---
### Third Party Curl Docs DEPRECATE (`third_party/curl/docs/DEPRECATE.md`)
# Items to be removed from future curl releases
If any of these deprecated features is a cause for concern for you, please
email the
[curl-library mailing list](https://lists.haxx.se/listinfo/curl-library)
as soon as possible and explain to us why this is a problem for you and
how your use case cannot be satisfied properly using a workaround.
## NSS
We remove support for building curl with the NSS TLS library in August 2023.
- There are few users left who use curl+NSS
- NSS has few users outside of curl as well (primarily Firefox)
- NSS is harder than ever to find documentation for
- NSS was always "best" used with Red Hat Linux when they provided additional
features on top of the regular NSS that is not shipped by the vanilla library
Starting in 7.82.0, building curl to use NSS configure requires the additional
flag `--with-nss-deprecated` in an attempt to highlight these plans.
## gskit
We remove support for building curl with the gskit TLS library in August 2023.
- This is a niche TLS library, only running on some IBM systems
- no regular curl contributors use this backend
- no CI builds use or verify this backend
- gskit, or the curl adaption for it, lacks many modern TLS features making it
an inferior solution
- build breakages in this code take weeks or more to get detected
- fixing gskit code is mostly done "flying blind"
## mingw v1
We remove support for building curl with the original legacy mingw version 1
in September 2023.
During the deprecation period you can enable the support with the configure
option `--with-mingw1-deprecated`.
mingw version 1 is old and deprecated software. There are much better and
still support build environments to use to build curl and other software. For
example [MinGW-w64](https://www.mingw-w64.org/).
## space-separated `NOPROXY` patterns
When specifying patterns/domain names for curl that should *not* go through a
proxy, the curl tool features the `--noproxy` command line option and the
library supports the `NO_PROXY` environment variable and the `CURLOPT_NOPROXY`
libcurl option.
They all set the same list of patterns. This list is documented to be a set of
**comma-separated** names, but can also be provided separated with just
space. The ability to just use spaces for this has never been documented but
some users may still have come to rely on this.
Several other tools and utilities also parse the `NO_PROXY` environment
variable but do not consider a space to be a valid separator. Using spaces for
separator is probably less portable and might cause more friction than commas
do. Users should use commas for this for greater portability.
curl will remove the support for space-separated names in July 2024.
## past removals
- Pipelining
- axTLS
- PolarSSL
- NPN
- Support for systems without 64 bit data types
---
### Third Party Curl Docs DYNBUF (`third_party/curl/docs/DYNBUF.md`)
# dynbuf
This is the internal module for creating and handling "dynamic buffers". This
means buffers that can be appended to, dynamically and grow to adapt.
There will always be a terminating zero put at the end of the dynamic buffer.
The `struct dynbuf` is used to hold data for each instance of a dynamic
buffer. The members of that struct **MUST NOT** be accessed or modified
without using the dedicated dynbuf API.
## `Curl_dyn_init`
```c
void Curl_dyn_init(struct dynbuf *s, size_t toobig);
```
This initializes a struct to use for dynbuf and it cannot fail. The `toobig`
value **must** be set to the maximum size we allow this buffer instance to
grow to. The functions below will return `CURLE_OUT_OF_MEMORY` when hitting
this limit.
## `Curl_dyn_free`
```c
void Curl_dyn_free(struct dynbuf *s);
```
Free the associated memory and clean up. After a free, the `dynbuf` struct can
be re-used to start appending new data to.
## `Curl_dyn_addn`
```c
CURLcode Curl_dyn_addn(struct dynbuf *s, const void *mem, size_t len);
```
Append arbitrary data of a given length to the end of the buffer.
If this function fails it calls `Curl_dyn_free` on `dynbuf`.
## `Curl_dyn_add`
```c
CURLcode Curl_dyn_add(struct dynbuf *s, const char *str);
```
Append a C string to the end of the buffer.
If this function fails it calls `Curl_dyn_free` on `dynbuf`.
## `Curl_dyn_addf`
```c
CURLcode Curl_dyn_addf(struct dynbuf *s, const char *fmt, ...);
```
Append a `printf()`-style string to the end of the buffer.
If this function fails it calls `Curl_dyn_free` on `dynbuf`.
## `Curl_dyn_vaddf`
```c
CURLcode Curl_dyn_vaddf(struct dynbuf *s, const char *fmt, va_list ap);
```
Append a `vprintf()`-style string to the end of the buffer.
If this function fails it calls `Curl_dyn_free` on `dynbuf`.
## `Curl_dyn_reset`
```c
void Curl_dyn_reset(struct dynbuf *s);
```
Reset the buffer length, but leave the allocation.
## `Curl_dyn_tail`
```c
CURLcode Curl_dyn_tail(struct dynbuf *s, size_t length);
```
Keep `length` bytes of the buffer tail (the last `length` bytes of the
buffer). The rest of the buffer is dropped. The specified `length` must not be
larger than the buffer length. To instead keep the leading part, see
`Curl_dyn_setlen()`.
## `Curl_dyn_ptr`
```c
char *Curl_dyn_ptr(const struct dynbuf *s);
```
Returns a `char *` to the buffer if it has a length, otherwise may return
NULL. Since the buffer may be reallocated, this pointer should not be trusted
or used anymore after the next buffer manipulation call.
## `Curl_dyn_uptr`
```c
unsigned char *Curl_dyn_uptr(const struct dynbuf *s);
```
Returns an `unsigned char *` to the buffer if it has a length, otherwise may
return NULL. Since the buffer may be reallocated, this pointer should not be
trusted or used anymore after the next buffer manipulation call.
## `Curl_dyn_len`
```c
size_t Curl_dyn_len(const struct dynbuf *s);
```
Returns the length of the buffer in bytes. Does not include the terminating
zero byte.
## `Curl_dyn_setlen`
```c
CURLcode Curl_dyn_setlen(struct dynbuf *s, size_t len);
```
Sets the new shorter length of the buffer in number of bytes. Keeps the
leftmost set number of bytes, discards the rest. To instead keep the tail part
of the buffer, see `Curl_dyn_tail()`.
---
### Third Party Curl Docs EARLY RELEASE (`third_party/curl/docs/EARLY-RELEASE.md`)
# How to determine if an early patch release is warranted
In the curl project we do releases every 8 weeks. Unless we break the cycle
and do an early patch release.
We do frequent releases partly to always have the next release "not too far
away".
## Bugfix
During the release cycle, and especially in the beginning of a new cycle (the
so-called "cool down" period), there are times when a bug is reported and
after it has been subsequently fixed correctly, the question might be asked:
is this bug and associated fix important enough for an early patch release?
The question can only be properly asked when a fix has been created and landed
in the git master branch.
## Early release
An early patch release means that we ship a new, complete and full release
called `major.minor.patch` where the `patch` part is increased by one since
the previous release. A curl release is a curl release. There is no small or
big and we never release just a patch. There is only "release".
## Questions to ask
- Is there a security advisory rated high or critical?
- Is there a data corruption bug?
- Did the bug cause an API/ABI breakage?
- Will the problem annoy a significant share of the user population?
If the answer is yes to one or more of the above, an early release might be
warranted.
More questions to ask ourselves when doing the assessment if the answers to
the three ones above are all 'no'.
- Does the bug cause curl to prematurely terminate?
- How common is the affected buggy option/feature/protocol/platform to get
used?
- How large is the estimated impacted user base?
- Does the bug block something crucial for applications or other adoption of
curl "out there" ?
- Does the bug cause problems for curl developers or others on "the curl
team" ?
- Is the bug limited to the curl tool only? That might have a smaller impact
than a bug also present in libcurl.
- Is there a (decent) workaround?
- Is it a regression? Is the bug introduced in this release?
- Can the bug be fixed "easily" by applying a patch?
- Does the bug break the build? Most users don't build curl themselves.
- How long is it until the already scheduled next release?
- Can affected users safely rather revert to a former release until the next
scheduled release?
- Is it a performance regression with no functionality side-effects? If so it
has to be substantial.
## If an early release is deemed necessary
Unless done for security or similarly important reasons, an early release
should not be done within a week of the previous release.
This, to enable us to collect and bundle more fixes into the same release to
make the release more worthwhile for everyone and to allow more time for fixes
to settle and things to get tested. Getting a release in shape and done in
style is work that should not be rushed.
---
### Third Party Curl Docs EXPERIMENTAL (`third_party/curl/docs/EXPERIMENTAL.md`)
# Experimental
Some features and functionality in curl and libcurl are considered
**EXPERIMENTAL**.
Experimental support in curl means:
1. Experimental features are provided to allow users to try them out and
provide feedback on functionality and API etc before they ship and get
"carved in stone".
2. You must enable the feature when invoking configure as otherwise curl will
not be built with the feature present.
3. We strongly advise against using this feature in production.
4. **We reserve the right to change behavior** of the feature without sticking
to our API/ABI rules as we do for regular features, as long as it is marked
experimental.
5. Experimental features are clearly marked so in documentation. Beware.
## Experimental features right now
- The Hyper HTTP backend
- HTTP/3 support and options
- The rustls backend
- WebSocket
---
### Third Party Curl Docs FEATURES (`third_party/curl/docs/FEATURES.md`)
# Features -- what curl can do
## curl tool
- config file support
- multiple URLs in a single command line
- range "globbing" support: [0-13], {one,two,three}
- multiple file upload on a single command line
- custom maximum transfer rate
- redirect stderr
- parallel transfers
## libcurl
- full URL syntax with no length limit
- custom maximum download time
- custom least download speed acceptable
- custom output result after completion
- guesses protocol from host name unless specified
- uses .netrc
- progress bar with time statistics while downloading
- "standard" proxy environment variables support
- compiles on win32 (reported builds on 70+ operating systems)
- selectable network interface for outgoing traffic
- IPv6 support on Unix and Windows
- happy eyeballs dual-stack connects
- persistent connections
- SOCKS 4 + 5 support, with or without local name resolving
- supports user name and password in proxy environment variables
- operations through HTTP proxy "tunnel" (using CONNECT)
- replaceable memory functions (malloc, free, realloc, etc)
- asynchronous name resolving (6)
- both a push and a pull style interface
- international domain names (10)
## HTTP
- HTTP/0.9 responses are optionally accepted
- HTTP/1.0
- HTTP/1.1
- HTTP/2, including multiplexing and server push (5)
- GET
- PUT
- HEAD
- POST
- multipart formpost (RFC1867-style)
- authentication: Basic, Digest, NTLM (9) and Negotiate (SPNEGO) (3)
to server and proxy
- resume (both GET and PUT)
- follow redirects
- maximum amount of redirects to follow
- custom HTTP request
- cookie get/send fully parsed
- reads/writes the Netscape cookie file format
- custom headers (replace/remove internally generated headers)
- custom user-agent string
- custom referrer string
- range
- proxy authentication
- time conditions
- via HTTP proxy, HTTPS proxy or SOCKS proxy
- retrieve file modification date
- Content-Encoding support for deflate and gzip
- "Transfer-Encoding: chunked" support in uploads
- automatic data compression (11)
## HTTPS (1)
- (all the HTTP features)
- HTTP/3 experimental support
- using client certificates
- verify server certificate
- via HTTP proxy, HTTPS proxy or SOCKS proxy
- select desired encryption
- select usage of a specific SSL version
## FTP
- download
- authentication
- Kerberos 5 (12)
- active/passive using PORT, EPRT, PASV or EPSV
- single file size information (compare to HTTP HEAD)
- 'type=' URL support
- dir listing
- dir listing names-only
- upload
- upload append
- upload via http-proxy as HTTP PUT
- download resume
- upload resume
- custom ftp commands (before and/or after the transfer)
- simple "range" support
- via HTTP proxy, HTTPS proxy or SOCKS proxy
- all operations can be tunneled through proxy
- customizable to retrieve file modification date
- no dir depth limit
## FTPS (1)
- implicit `ftps://` support that use SSL on both connections
- explicit "AUTH TLS" and "AUTH SSL" usage to "upgrade" plain `ftp://`
connection to use SSL for both or one of the connections
## SCP (8)
- both password and public key auth
## SFTP (7)
- both password and public key auth
- with custom commands sent before/after the transfer
## TFTP
- download
- upload
## TELNET
- connection negotiation
- custom telnet options
- stdin/stdout I/O
## LDAP (2)
- full LDAP URL support
## DICT
- extended DICT URL support
## FILE
- URL support
- upload
- resume
## SMB
- SMBv1 over TCP and SSL
- download
- upload
- authentication with NTLMv1
## SMTP
- authentication: Plain, Login, CRAM-MD5, Digest-MD5, NTLM (9), Kerberos 5
(4) and External.
- send emails
- mail from support
- mail size support
- mail auth support for trusted server-to-server relaying
- multiple recipients
- via http-proxy
## SMTPS (1)
- implicit `smtps://` support
- explicit "STARTTLS" usage to "upgrade" plain `smtp://` connections to use SSL
- via http-proxy
## POP3
- authentication: Clear Text, APOP and SASL
- SASL based authentication: Plain, Login, CRAM-MD5, Digest-MD5, NTLM (9),
Kerberos 5 (4) and External.
- list emails
- retrieve emails
- enhanced command support for: CAPA, DELE, TOP, STAT, UIDL and NOOP via
custom requests
- via http-proxy
## POP3S (1)
- implicit `pop3s://` support
- explicit `STLS` usage to "upgrade" plain `pop3://` connections to use SSL
- via http-proxy
## IMAP
- authentication: Clear Text and SASL
- SASL based authentication: Plain, Login, CRAM-MD5, Digest-MD5, NTLM (9),
Kerberos 5 (4) and External.
- list the folders of a mailbox
- select a mailbox with support for verifying the `UIDVALIDITY`
- fetch emails with support for specifying the UID and SECTION
- upload emails via the append command
- enhanced command support for: EXAMINE, CREATE, DELETE, RENAME, STATUS,
STORE, COPY and UID via custom requests
- via http-proxy
## IMAPS (1)
- implicit `imaps://` support
- explicit "STARTTLS" usage to "upgrade" plain `imap://` connections to use SSL
- via http-proxy
## MQTT
- Subscribe to and publish topics using URL scheme `mqtt://broker/topic`
## Footnotes
1. requires a TLS library
2. requires OpenLDAP or WinLDAP
3. requires a GSS-API implementation (such as Heimdal or MIT Kerberos) or
SSPI (native Windows)
4. requires a GSS-API implementation, however, only Windows SSPI is
currently supported
5. requires nghttp2
6. requires c-ares
7. requires libssh2, libssh or wolfSSH
8. requires libssh2 or libssh
9. requires OpenSSL, GnuTLS, mbedTLS, NSS, Secure Transport or SSPI
(native Windows)
10. requires libidn2 or Windows
11. requires libz, brotli and/or zstd
12. requires a GSS-API implementation (such as Heimdal or MIT Kerberos)
---
### Third Party Curl Docs GOVERNANCE (`third_party/curl/docs/GOVERNANCE.md`)
# Decision making in the curl project
A rough guide to how we make decisions and who does what.
## BDFL
This project was started by and has to some extent been pushed forward over
the years with Daniel Stenberg as the driving force. It matches a standard
BDFL (Benevolent Dictator For Life) style project.
This setup has been used due to convenience and the fact that it has worked
fine this far. It is not because someone thinks of it as a superior project
leadership model. It will also only continue working as long as Daniel manages
to listen in to what the project and the general user population wants and
expects from us.
## Legal entity
There is no legal entity. The curl project is just a bunch of people scattered
around the globe with the common goal to produce source code that creates
great products. We are not part of any umbrella organization and we are not
located in any specific country. We are totally independent.
The copyrights in the project are owned by the individuals and organizations
that wrote those parts of the code.
## Decisions
The curl project is not a democracy, but everyone is entitled to state their
opinion and may argue for their sake within the community.
All and any changes that have been done or will be done are eligible to bring
up for discussion, to object to or to praise. Ideally, we find consensus for
the appropriate way forward in any given situation or challenge.
If there is no obvious consensus, a maintainer who's knowledgeable in the
specific area will take an "executive" decision that they think is the right
for the project.
## Donations
Donating plain money to curl is best done to curl's [Open Collective
fund](https://opencollective.com/curl). Open Collective is a US based
non-profit organization that holds on to funds for us. This fund is then used
for paying the curl security bug bounties, to reimburse project related
expenses etc.
Donations to the project can also come in the form of server hosting, providing
services and paying for people to work on curl related code etc. Usually, such
donations are services paid for directly by the sponsors.
We grade sponsors in a few different levels and if they meet the criteria,
they can be mentioned on the Sponsors page on the curl website.
## Commercial Support
The curl project does not do or offer commercial support. It only hosts
mailing lists, runs bug trackers etc to facilitate communication and work.
However, Daniel works for wolfSSL and we offer commercial curl support there.
# Key roles
## User
Someone who uses or has used curl or libcurl.
## Contributor
Someone who has helped the curl project, who has contributed to bring it
forward. Contributing could be to provide advice, debug a problem, file a bug
report, run test infrastructure or writing code etc.
## Commit author
Sometimes also called 'committer'. Someone who has authored a commit in the
curl source code repository. Committers are recorded as `Author` in git.
## Maintainers
A maintainer in the curl project is an individual who has been given
permissions to push commits to one of the git repositories.
Maintainers are free to push commits to the repositories at their own will.
Maintainers are however expected to listen to feedback from users and any
change that is non-trivial in size or nature *should* be brought to the
project as a Pull-Request (PR) to allow others to comment/object before merge.
## Former maintainers
A maintainer who stops being active in the project will at some point get
their push permissions removed. We do this for security reasons but also to
make sure that we always have the list of maintainers as "the team that push
stuff to curl".
Getting push permissions removed is not a punishment. Everyone who ever worked
on maintaining curl is considered a hero, for all time hereafter.
## Security team members
We have a security team. That is the team of people who are subscribed to the
curl-security mailing list; the receivers of security reports from users and
developers. This list of people will vary over time but should be skilled
developers familiar with the curl project.
The security team works best when it consists of a small set of active
persons. We invite new members when the team seems to need it, and we also
expect to retire security team members as they "drift off" from the project or
just find themselves unable to perform their duties there.
## Server admins
We run a web server, a mailing list and more on the curl project's primary
server. That physical machine is owned and run by Haxx. Daniel is the primary
admin of all things curl related server stuff, but Björn Stenberg and Linus
Feltzing serve as backup admins for when Daniel is gone or unable.
The primary server is paid for by Haxx. The machine is physically located in a
server bunker in Stockholm Sweden, operated by the company Glesys.
The website contents are served to the web via Fastly and Daniel is the
primary curl contact with Fastly.
## BDFL
That is Daniel.
# Maintainers
A curl maintainer is a project volunteer who has the authority and rights to
merge changes into a git repository in the curl project.
Anyone can aspire to become a curl maintainer.
### Duties
There are no mandatory duties. We hope and wish that maintainers consider
reviewing patches and help merging them, especially when the changes are
within the area of personal expertise and experience.
### Requirements
- only merge code that meets our quality and style guide requirements.
- *never* merge code without doing a PR first, unless the change is "trivial"
- if in doubt, ask for input/feedback from others
### Recommendations
- we require two-factor authentication enabled on your GitHub account to
reduce risk of malicious source code tampering
- consider enabling signed git commits for additional verification of changes
### Merge advice
When you are merging patches/pull requests...
- make sure the commit messages follow our template
- squash patch sets into a few logical commits even if the PR did not, if
necessary
- avoid the "merge" button on GitHub, do it "manually" instead to get full
control and full audit trail (GitHub leaves out you as "Committer:")
- remember to credit the reporter and the helpers.
## Who are maintainers?
The [list of maintainers](https://github.com/orgs/curl/people). Be aware that
the level of presence and activity in the project vary greatly between
different individuals and over time.
### Become a maintainer?
If you think you can help making the project better by shouldering some
maintaining responsibilities, then please get in touch.
You will be expected to be familiar with the curl project and its ways of
working. You need to have gotten a few quality patches merged as a proof of
this.
### Stop being a maintainer
If you (appear to) not be active in the project anymore, you may be removed as
a maintainer. Thank you for your service!
---
### Third Party Curl Docs HELP US (`third_party/curl/docs/HELP-US.md`)
# How to get started helping out in the curl project
We are always in need of more help. If you are new to the project and are
looking for ways to contribute and help out, this document aims to give a few
good starting points.
You may subscribe to the [curl-library mailing
list](https://lists.haxx.se/listinfo/curl-library) to keep track of the
current discussion topics; or if you are registered on GitHub, you can use the
[Discussions section](https://github.com/curl/curl/discussions) on the main
curl repository.
## Scratch your own itch
One of the best ways is to start working on any problems or issues you have
found yourself or perhaps got annoyed at in the past. It can be a spelling
error in an error text or a weirdly phrased section in a man page. Hunt it
down and report the bug. Or make your first pull request with a fix for that.
## Smaller tasks
Some projects mark small issues as "beginner friendly", "bite-sized" or
similar. We do not do that in curl since such issues never linger around long
enough. Simple issues get handled fast.
If you are looking for a smaller or simpler task in the project to help out
with as an entry-point into the project, perhaps because you are a newcomer or
even maybe not a terribly experienced developer, here's our advice:
- Read through this document to get a grasp on a general approach to use
- Consider adding a test case for something not currently tested (correctly)
- Consider updating or adding documentation
- One way to get started gently in the project, is to participate in an
existing issue/PR and help out by reproducing the issue, review the code in
the PR etc.
## Help wanted
In the issue tracker we occasionally mark bugs with [help
wanted](https://github.com/curl/curl/labels/help%20wanted), as a sign that the
bug is acknowledged to exist and that there's nobody known to work on this
issue for the moment. Those are bugs that are fine to "grab" and provide a
pull request for. The complexity level of these will of course vary, so pick
one that piques your interest.
## Work on known bugs
Some bugs are known and have not yet received attention and work enough to get
fixed. We collect such known existing flaws in the
[KNOWN_BUGS](https://curl.se/docs/knownbugs.html) page. Many of them link
to the original bug report with some additional details, but some may also
have aged a bit and may require some verification that the bug still exists in
the same way and that what was said about it in the past is still valid.
## Fix autobuild problems
On the [autobuilds page](https://curl.se/dev/builds.html) we show a
collection of test results from the automatic curl build and tests that are
performed by volunteers. Fixing compiler warnings and errors shown there is
something we value greatly. Also, if you own or run systems or architectures
that are not already tested in the autobuilds, we also appreciate more
volunteers running builds automatically to help us keep curl portable.
## TODO items
Ideas for features and functions that we have considered worthwhile to
implement and provide are kept in the
[TODO](https://curl.se/docs/todo.html) file. Some of the ideas are
rough. Some are well thought out. Some probably are not really suitable
anymore.
Before you invest a lot of time on a TODO item, do bring it up for discussion
on the mailing list. For discussion on applicability but also for ideas and
brainstorming on specific ways to do the implementation etc.
## You decide
You can also come up with a completely new thing you think we should do. Or
not do. Or fix. Or add to the project. You then either bring it to the mailing
list first to see if people will shoot down the idea at once, or you bring a
first draft of the idea as a pull request and take the discussion there around
the specific implementation. Either way is fine.
## CONTRIBUTE
We offer [guidelines](https://curl.se/dev/contribute.html) that are
suitable to be familiar with before you decide to contribute to curl. If
you are used to open source development, you will probably not find many
surprises there.
---
### Third Party Curl Docs HISTORY (`third_party/curl/docs/HISTORY.md`)
How curl Became Like This
=========================
Towards the end of 1996, Daniel Stenberg was spending time writing an IRC bot
for an Amiga related channel on EFnet. He then came up with the idea to make
currency-exchange calculations available to Internet Relay Chat (IRC)
users. All the necessary data were published on the Web; he just needed to
automate their retrieval.
1996
----
On November 11, 1996 the Brazilian developer Rafael Sagula wrote and released
HttpGet version 0.1.
Daniel extended this existing command-line open-source tool. After a few minor
adjustments, it did just what he needed. The first release with Daniel's
additions was 0.2, released on December 17, 1996. Daniel quickly became the
new maintainer of the project.
1997
----
HttpGet 0.3 was released in January 1997 and now it accepted HTTP URLs on the
command line.
HttpGet 1.0 was released on April 8 1997 with brand new HTTP proxy support.
We soon found and fixed support for getting currencies over GOPHER. Once FTP
download support was added, the name of the project was changed and urlget 2.0
was released in August 1997. The http-only days were already passed.
Version 2.2 was released on August 14 1997 and introduced support to build for
and run on Windows and Solaris.
November 24 1997: Version 3.1 added FTP upload support.
Version 3.5 added support for HTTP POST.
1998
----
February 4: urlget 3.10
February 9: urlget 3.11
March 14: urlget 3.12 added proxy authentication.
The project slowly grew bigger. With upload capabilities, the name was once
again misleading and a second name change was made. On March 20, 1998 curl 4
was released. (The version numbering from the previous names was kept.)
(Unrelated to this project a company called Curl Corporation registered a US
trademark on the name "CURL" on May 18 1998. That company had then already
registered the curl.com domain back in November of the previous year. All this
was revealed to us much later.)
SSL support was added, powered by the SSLeay library.
August: first announcement of curl on freshmeat.net.
October: with the curl 4.9 release and the introduction of cookie support,
curl was no longer released under the GPL license. Now we are at 4000 lines of
code, we switched over to the MPL license to restrict the effects of
"copyleft".
November: configure script and reported successful compiles on several
major operating systems. The never-quite-understood -F option was added and
curl could now simulate quite a lot of a browser. TELNET support was added.
Curl 5 was released in December 1998 and introduced the first ever curl man
page. People started making Linux RPM packages out of it.
1999
----
January: DICT support added.
OpenSSL took over and SSLeay was abandoned.
May: first Debian package.
August: LDAP:// and FILE:// support added. The curl website gets 1300 visits
weekly. Moved site to curl.haxx.nu.
September: Released curl 6.0. 15000 lines of code.
December 28: added the project on Sourceforge and started using its services
for managing the project.
2000
----
Spring: major internal overhaul to provide a suitable library interface.
The first non-beta release was named 7.1 and arrived in August. This offered
the easy interface and turned out to be the beginning of actually getting
other software and programs to be based on and powered by libcurl. Almost
20000 lines of code.
June: the curl site moves to "curl.haxx.se"
August, the curl website gets 4000 visits weekly.
The PHP guys adopted libcurl already the same month, when the first ever third
party libcurl binding showed up. CURL has been a supported module in PHP since
the release of PHP 4.0.2. This would soon get followers. More than 16
different bindings exist at the time of this writing.
September: kerberos4 support was added.
November: started the work on a test suite for curl. It was later re-written
from scratch again. The libcurl major SONAME number was set to 1.
2001
----
January: Daniel released curl 7.5.2 under a new license again: MIT (or
MPL). The MIT license is extremely liberal and can be combined with GPL
in other projects. This would finally put an end to the "complaints" from
people involved in GPLed projects that previously were prohibited from using
libcurl while it was released under MPL only. (Due to the fact that MPL is
deemed "GPL incompatible".)
March 22: curl supports HTTP 1.1 starting with the release of 7.7. This
also introduced libcurl's ability to do persistent connections. 24000 lines of
code. The libcurl major SONAME number was bumped to 2 due to this overhaul.
The first experimental ftps:// support was added.
August: The curl website gets 8000 visits weekly. Curl Corporation contacted
Daniel to discuss "the name issue". After Daniel's reply, they have never
since got back in touch again.
September: libcurl 7.9 introduces cookie jar and `curl_formadd()`. During the
forthcoming 7.9.x releases, we introduced the multi interface slowly and
without many whistles.
September 25: curl (7.7.2) is bundled in Mac OS X (10.1) for the first time. It was
already becoming more and more of a standard utility of Linux distributions
and a regular in the BSD ports collections.
2002
----
June: the curl website gets 13000 visits weekly. curl and libcurl is
35000 lines of code. Reported successful compiles on more than 40 combinations
of CPUs and operating systems.
To estimate the number of users of the curl tool or libcurl library is next to
impossible. Around 5000 downloaded packages each week from the main site gives
a hint, but the packages are mirrored extensively, bundled with numerous OS
distributions and otherwise retrieved as part of other software.
October 1: with the release of curl 7.10 it is released under the MIT license
only.
Starting with 7.10, curl verifies SSL server certificates by default.
2003
----
January: Started working on the distributed curl tests. The autobuilds.
February: the curl site averages at 20000 visits weekly. At any given moment,
there's an average of 3 people browsing the website.
Multiple new authentication schemes are supported: Digest (May), NTLM (June)
and Negotiate (June).
November: curl 7.10.8 is released. 45000 lines of code. ~55000 unique visitors
to the website. Five official web mirrors.
December: full-fledged SSL for FTP is supported.
2004
----
January: curl 7.11.0 introduced large file support.
June: curl 7.12.0 introduced IDN support. 10 official web mirrors.
This release bumped the major SONAME to 3 due to the removal of the
`curl_formparse()` function
August: Curl and libcurl 7.12.1
Public curl release number: 82
Releases counted from the beginning: 109
Available command line options: 96
Available curl_easy_setopt() options: 120
Number of public functions in libcurl: 36
Amount of public website mirrors: 12
Number of known libcurl bindings: 26
2005
----
April: GnuTLS can now optionally be used for the secure layer when curl is
built.
April: Added the multi_socket() API
September: TFTP support was added.
More than 100,000 unique visitors of the curl website. 25 mirrors.
December: security vulnerability: libcurl URL Buffer Overflow
2006
----
January: We dropped support for Gopher. We found bugs in the implementation
that turned out to have been introduced years ago, so with the conclusion that
nobody had found out in all this time we removed it instead of fixing it.
March: security vulnerability: libcurl TFTP Packet Buffer Overflow
September: The major SONAME number for libcurl was bumped to 4 due to the
removal of ftp third party transfer support.
November: Added SCP and SFTP support
2007
----
February: Added support for the Mozilla NSS library to do the SSL/TLS stuff
July: security vulnerability: libcurl GnuTLS insufficient cert verification
2008
----
November:
Command line options: 128
curl_easy_setopt() options: 158
Public functions in libcurl: 58
Known libcurl bindings: 37
Contributors: 683
145,000 unique visitors. >100 GB downloaded.
2009
----
March: security vulnerability: libcurl Arbitrary File Access
April: added CMake support
August: security vulnerability: libcurl embedded zero in cert name
December: Added support for IMAP, POP3 and SMTP
2010
----
January: Added support for RTSP
February: security vulnerability: libcurl data callback excessive length
March: The project switched over to use git (hosted by GitHub) instead of CVS
for source code control
May: Added support for RTMP
Added support for PolarSSL to do the SSL/TLS stuff
August:
Public curl releases: 117
Command line options: 138
curl_easy_setopt() options: 180
Public functions in libcurl: 58
Known libcurl bindings: 39
Contributors: 808
Gopher support added (re-added actually, see January 2006)
2011
----
February: added support for the axTLS backend
April: added the cyassl backend (later renamed to WolfSSL)
2012
----
July: Added support for Schannel (native Windows TLS backend) and Darwin SSL
(Native Mac OS X and iOS TLS backend).
Supports Metalink
October: SSH-agent support.
2013
----
February: Cleaned up internals to always uses the "multi" non-blocking
approach internally and only expose the blocking API with a wrapper.
September: First small steps on supporting HTTP/2 with nghttp2.
October: Removed krb4 support.
December: Happy eyeballs.
2014
----
March: first real release supporting HTTP/2
September: Website had 245,000 unique visitors and served 236GB data
SMB and SMBS support
2015
----
June: support for multiplexing with HTTP/2
August: support for HTTP/2 server push
December: Public Suffix List
2016
----
January: the curl tool defaults to HTTP/2 for HTTPS URLs
December: curl 7.52.0 introduced support for HTTPS-proxy!
First TLS 1.3 support
2017
----
July: OSS-Fuzz started fuzzing libcurl
September: Added Multi-SSL support
The website serves 3100 GB/month
Public curl releases: 169
Command line options: 211
curl_easy_setopt() options: 249
Public functions in libcurl: 74
Contributors: 1609
October: SSLKEYLOGFILE support, new MIME API
October: Daniel received the Polhem Prize for his work on curl
November: brotli
2018
----
January: new SSH backend powered by libssh
March: starting with the 1803 release of Windows 10, curl is shipped bundled
with Microsoft's operating system.
July: curl shows headers using bold type face
October: added DNS-over-HTTPS (DoH) and the URL API
MesaLink is a new supported TLS backend
libcurl now does HTTP/2 (and multiplexing) by default on HTTPS URLs
curl and libcurl are installed in an estimated 5 *billion* instances
world-wide.
October 31: Curl and libcurl 7.62.0
Public curl releases: 177
Command line options: 219
curl_easy_setopt() options: 261
Public functions in libcurl: 80
Contributors: 1808
December: removed axTLS support
2019
----
March: added experimental alt-svc support
August: the first HTTP/3 requests with curl.
September: 7.66.0 is released and the tool offers parallel downloads
2020
----
curl and libcurl are installed in an estimated 10 *billion* instances
world-wide.
January: added BearSSL support
March: removed support for PolarSSL, added wolfSSH support
April: experimental MQTT support
August: zstd support
November: the website moves to curl.se. The website serves 10TB data monthly.
December: alt-svc support
2021
----
February 3: curl 7.75.0 ships with support for Hyper as an HTTP backend
March 31: curl 7.76.0 ships with support for rustls
July: HSTS is supported
2022
----
March: added --json, removed mesalink support
Public curl releases: 206
Command line options: 245
curl_easy_setopt() options: 295
Public functions in libcurl: 86
Contributors: 2601
The curl.se website serves 16,500 GB/month over 462M requests, the
official docker image has been pulled 4,098,015,431 times.
---
### Third Party Curl Docs HSTS (`third_party/curl/docs/HSTS.md`)
# HSTS support
HTTP Strict-Transport-Security. Added as experimental in curl
7.74.0. Supported "for real" since 7.77.0.
## Standard
[HTTP Strict Transport Security](https://datatracker.ietf.org/doc/html/rfc6797)
## Behavior
libcurl features an in-memory cache for HSTS hosts, so that subsequent
HTTP-only requests to a host name present in the cache will get internally
"redirected" to the HTTPS version.
## `curl_easy_setopt()` options:
- `CURLOPT_HSTS_CTRL` - enable HSTS for this easy handle
- `CURLOPT_HSTS` - specify file name where to store the HSTS cache on close
(and possibly read from at startup)
## curl command line options
- `--hsts [filename]` - enable HSTS, use the file as HSTS cache. If filename
is `""` (no length) then no file will be used, only in-memory cache.
## HSTS cache file format
Lines starting with `#` are ignored.
For each hsts entry:
[host name] "YYYYMMDD HH:MM:SS"
The `[host name]` is dot-prefixed if it includes subdomains.
The time stamp is when the entry expires.
## Possible future additions
- `CURLOPT_HSTS_PRELOAD` - provide a set of HSTS host names to load first
- ability to save to something else than a file
---
### Third Party Curl Docs HTTP COOKIES (`third_party/curl/docs/HTTP-COOKIES.md`)
# HTTP Cookies
## Cookie overview
Cookies are `name=contents` pairs that an HTTP server tells the client to
hold and then the client sends back those to the server on subsequent
requests to the same domains and paths for which the cookies were set.
Cookies are either "session cookies" which typically are forgotten when the
session is over which is often translated to equal when browser quits, or
the cookies are not session cookies they have expiration dates after which
the client will throw them away.
Cookies are set to the client with the Set-Cookie: header and are sent to
servers with the Cookie: header.
For a long time, the only spec explaining how to use cookies was the
original [Netscape spec from 1994](https://curl.se/rfc/cookie_spec.html).
In 2011, [RFC6265](https://www.ietf.org/rfc/rfc6265.txt) was finally
published and details how cookies work within HTTP. In 2016, an update which
added support for prefixes was
[proposed](https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-cookie-prefixes-00),
and in 2017, another update was
[drafted](https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-cookie-alone-01)
to deprecate modification of 'secure' cookies from non-secure origins. Both
of these drafts have been incorporated into a proposal to
[replace](https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-rfc6265bis-11)
RFC6265. Cookie prefixes and secure cookie modification protection has been
implemented by curl.
curl considers `http://localhost` to be a *secure context*, meaning that it
will allow and use cookies marked with the `secure` keyword even when done
over plain HTTP for this host. curl does this to match how popular browsers
work with secure cookies.
## Cookies saved to disk
Netscape once created a file format for storing cookies on disk so that they
would survive browser restarts. curl adopted that file format to allow
sharing the cookies with browsers, only to see browsers move away from that
format. Modern browsers no longer use it, while curl still does.
The Netscape cookie file format stores one cookie per physical line in the
file with a bunch of associated meta data, each field separated with
TAB. That file is called the cookie jar in curl terminology.
When libcurl saves a cookie jar, it creates a file header of its own in
which there is a URL mention that will link to the web version of this
document.
## Cookie file format
The cookie file format is text based and stores one cookie per line. Lines
that start with `#` are treated as comments. An exception is lines that
start with `#HttpOnly_`, which is a prefix for cookies that have the
`HttpOnly` attribute set.
Each line that specifies a single cookie consists of seven text fields
separated with TAB characters. A valid line must end with a newline
character.
### Fields in the file
Field number, what type and example data and the meaning of it:
0. string `example.com` - the domain name
1. boolean `FALSE` - include subdomains
2. string `/foobar/` - path
3. boolean `TRUE` - send/receive over HTTPS only
4. number `1462299217` - expires at - seconds since Jan 1st 1970, or 0
5. string `person` - name of the cookie
6. string `daniel` - value of the cookie
## Cookies with curl the command line tool
curl has a full cookie "engine" built in. If you just activate it, you can
have curl receive and send cookies exactly as mandated in the specs.
Command line options:
`-b, --cookie`
tell curl a file to read cookies from and start the cookie engine, or if it
is not a file it will pass on the given string. `-b name=var` works and so
does `-b cookiefile`.
`-j, --junk-session-cookies`
when used in combination with -b, it will skip all "session cookies" on load
so as to appear to start a new cookie session.
`-c, --cookie-jar`
tell curl to start the cookie engine and write cookies to the given file
after the request(s)
## Cookies with libcurl
libcurl offers several ways to enable and interface the cookie engine. These
options are the ones provided by the native API. libcurl bindings may offer
access to them using other means.
`CURLOPT_COOKIE`
Is used when you want to specify the exact contents of a cookie header to
send to the server.
`CURLOPT_COOKIEFILE`
Tell libcurl to activate the cookie engine, and to read the initial set of
cookies from the given file. Read-only.
`CURLOPT_COOKIEJAR`
Tell libcurl to activate the cookie engine, and when the easy handle is
closed save all known cookies to the given cookie jar file. Write-only.
`CURLOPT_COOKIELIST`
Provide detailed information about a single cookie to add to the internal
storage of cookies. Pass in the cookie as an HTTP header with all the
details set, or pass in a line from a Netscape cookie file. This option can
also be used to flush the cookies etc.
`CURLOPT_COOKIESESSION`
Tell libcurl to ignore all cookies it is about to load that are session
cookies.
`CURLINFO_COOKIELIST`
Extract cookie information from the internal cookie storage as a linked
list.
## Cookies with JavaScript
These days a lot of the web is built up by JavaScript. The web browser loads
complete programs that render the page you see. These JavaScript programs
can also set and access cookies.
Since curl and libcurl are plain HTTP clients without any knowledge of or
capability to handle JavaScript, such cookies will not be detected or used.
Often, if you want to mimic what a browser does on such websites, you can
record web browser HTTP traffic when using such a site and then repeat the
cookie operations using curl or libcurl.
---
### Third Party Curl Docs HTTP2 (`third_party/curl/docs/HTTP2.md`)
HTTP/2 with curl
================
[HTTP/2 Spec](https://www.rfc-editor.org/rfc/rfc7540.txt)
[http2 explained](https://daniel.haxx.se/http2/)
Build prerequisites
-------------------
- nghttp2
- OpenSSL, libressl, BoringSSL, NSS, GnuTLS, mbedTLS, wolfSSL or Schannel
with a new enough version.
[nghttp2](https://nghttp2.org/)
-------------------------------
libcurl uses this 3rd party library for the low level protocol handling
parts. The reason for this is that HTTP/2 is much more complex at that layer
than HTTP/1.1 (which we implement on our own) and that nghttp2 is an already
existing and well functional library.
We require at least version 1.12.0.
Over an http:// URL
-------------------
If `CURLOPT_HTTP_VERSION` is set to `CURL_HTTP_VERSION_2_0`, libcurl will
include an upgrade header in the initial request to the host to allow
upgrading to HTTP/2.
Possibly we can later introduce an option that will cause libcurl to fail if
not possible to upgrade. Possibly we introduce an option that makes libcurl
use HTTP/2 at once over http://
Over an https:// URL
--------------------
If `CURLOPT_HTTP_VERSION` is set to `CURL_HTTP_VERSION_2_0`, libcurl will use
ALPN to negotiate which protocol to continue with. Possibly introduce an
option that will cause libcurl to fail if not possible to use HTTP/2.
`CURL_HTTP_VERSION_2TLS` was added in 7.47.0 as a way to ask libcurl to prefer
HTTP/2 for HTTPS but stick to 1.1 by default for plain old HTTP connections.
ALPN is the TLS extension that HTTP/2 is expected to use.
`CURLOPT_SSL_ENABLE_ALPN` is offered to allow applications to explicitly
disable ALPN.
Multiplexing
------------
Starting in 7.43.0, libcurl fully supports HTTP/2 multiplexing, which is the
term for doing multiple independent transfers over the same physical TCP
connection.
To take advantage of multiplexing, you need to use the multi interface and set
`CURLMOPT_PIPELINING` to `CURLPIPE_MULTIPLEX`. With that bit set, libcurl will
attempt to re-use existing HTTP/2 connections and just add a new stream over
that when doing subsequent parallel requests.
While libcurl sets up a connection to an HTTP server there is a period during
which it does not know if it can pipeline or do multiplexing and if you add
new transfers in that period, libcurl will default to start new connections
for those transfers. With the new option `CURLOPT_PIPEWAIT` (added in 7.43.0),
you can ask that a transfer should rather wait and see in case there's a
connection for the same host in progress that might end up being possible to
multiplex on. It favors keeping the number of connections low to the cost of
slightly longer time to first byte transferred.
Applications
------------
We hide HTTP/2's binary nature and convert received HTTP/2 traffic to headers
in HTTP 1.1 style. This allows applications to work unmodified.
curl tool
---------
curl offers the `--http2` command line option to enable use of HTTP/2.
curl offers the `--http2-prior-knowledge` command line option to enable use of
HTTP/2 without HTTP/1.1 Upgrade.
Since 7.47.0, the curl tool enables HTTP/2 by default for HTTPS connections.
curl tool limitations
---------------------
The command line tool does not support HTTP/2 server push. It supports
multiplexing when the parallel transfer option is used.
HTTP Alternative Services
-------------------------
Alt-Svc is an extension with a corresponding frame (ALTSVC) in HTTP/2 that
tells the client about an alternative "route" to the same content for the same
origin server that you get the response from. A browser or long-living client
can use that hint to create a new connection asynchronously. For libcurl, we
may introduce a way to bring such clues to the application and/or let a
subsequent request use the alternate route automatically.
[Detailed in RFC 7838](https://datatracker.ietf.org/doc/html/rfc7838)
---
### Third Party Curl Docs HTTP3 (`third_party/curl/docs/HTTP3.md`)
# HTTP3 (and QUIC)
## Resources
[HTTP/3 Explained](https://http3-explained.haxx.se/en/) - the online free
book describing the protocols involved.
[quicwg.org](https://quicwg.org/) - home of the official protocol drafts
## QUIC libraries
QUIC libraries we are experimenting with:
[ngtcp2](https://github.com/ngtcp2/ngtcp2)
[quiche](https://github.com/cloudflare/quiche)
[msh3](https://github.com/nibanks/msh3) (with [msquic](https://github.com/microsoft/msquic))
## Experimental
HTTP/3 and QUIC support in curl is considered **EXPERIMENTAL** until further
notice. It needs to be enabled at build-time.
Further development and tweaking of the HTTP/3 support in curl will happen in
the master branch using pull-requests, just like ordinary changes.
To fix before we remove the experimental label:
- working multiplexing and GTFO handling
- fallback or another flexible way to go (back to) h1/h2 if h3 fails
- enough test cases to verify basic HTTP/3 functionality
- no "important" bugs left on HTTP/3
- it's fine to "leave" individual backends as experimental if necessary
# ngtcp2 version
Building curl with ngtcp2 involves 3 components: `ngtcp2` itself, `nghttp3` and a QUIC supporting TLS library. The supported TLS libraries are covered below.
For now, `ngtcp2` and `nghttp3` are still *experimental* which means their evolution bring breaking changes. Therefore, the proper version of both libraries need to be used when building curl. These are
* `ngtcp2`: v0.15.0
* `nghttp3`: v0.11.0
## Build with OpenSSL
Build (patched) OpenSSL
% git clone --depth 1 -b openssl-3.0.8+quic https://github.com/quictls/openssl
% cd openssl
% ./config enable-tls1_3 --prefix=
% make
% make install
Build nghttp3
% cd ..
% git clone -b v0.11.0 https://github.com/ngtcp2/nghttp3
% cd nghttp3
% autoreconf -fi
% ./configure --prefix= --enable-lib-only
% make
% make install
Build ngtcp2
% cd ..
% git clone -b v0.15.0 https://github.com/ngtcp2/ngtcp2
% cd ngtcp2
% autoreconf -fi
% ./configure PKG_CONFIG_PATH=/lib/pkgconfig:/lib/pkgconfig LDFLAGS="-Wl,-rpath,/lib" --prefix= --enable-lib-only
% make
% make install
Build curl
% cd ..
% git clone https://github.com/curl/curl
% cd curl
% autoreconf -fi
% LDFLAGS="-Wl,-rpath,/lib" ./configure --with-openssl= --with-nghttp3= --with-ngtcp2=
% make
% make install
For OpenSSL 3.0.0 or later builds on Linux for x86_64 architecture, substitute all occurrences of "/lib" with "/lib64"
## Build with GnuTLS
Build GnuTLS
% git clone --depth 1 https://gitlab.com/gnutls/gnutls.git
% cd gnutls
% ./bootstrap
% ./configure --prefix=
% make
% make install
Build nghttp3
% cd ..
% git clone -b v0.11.0 https://github.com/ngtcp2/nghttp3
% cd nghttp3
% autoreconf -fi
% ./configure --prefix= --enable-lib-only
% make
% make install
Build ngtcp2
% cd ..
% git clone -b v0.15.0 https://github.com/ngtcp2/ngtcp2
% cd ngtcp2
% autoreconf -fi
% ./configure PKG_CONFIG_PATH=/lib/pkgconfig:/lib/pkgconfig LDFLAGS="-Wl,-rpath,/lib" --prefix= --enable-lib-only --with-gnutls
% make
% make install
Build curl
% cd ..
% git clone https://github.com/curl/curl
% cd curl
% autoreconf -fi
% ./configure --with-gnutls= --with-nghttp3= --with-ngtcp2=
% make
% make install
## Build with wolfSSL
Build wolfSSL
% git clone https://github.com/wolfSSL/wolfssl.git
% cd wolfssl
% autoreconf -fi
% ./configure --prefix= --enable-quic --enable-session-ticket --enable-earlydata --enable-psk --enable-harden --enable-altcertchains
% make
% make install
Build nghttp3
% cd ..
% git clone -b v0.11.0 https://github.com/ngtcp2/nghttp3
% cd nghttp3
% autoreconf -fi
% ./configure --prefix= --enable-lib-only
% make
% make install
Build ngtcp2
% cd ..
% git clone -b v0.15.0 https://github.com/ngtcp2/ngtcp2
% cd ngtcp2
% autoreconf -fi
% ./configure PKG_CONFIG_PATH=/lib/pkgconfig:/lib/pkgconfig LDFLAGS="-Wl,-rpath,/lib" --prefix= --enable-lib-only --with-wolfssl
% make
% make install
Build curl
% cd ..
% git clone https://github.com/curl/curl
% cd curl
% autoreconf -fi
% ./configure --with-wolfssl= --with-nghttp3= --with-ngtcp2=
% make
% make install
# quiche version
Since the quiche build manages its dependencies, curl can be built against the latest version. You are *probably* able to build against their main branch, but in case of problems, we recommend their latest release tag.
## build
Build quiche and BoringSSL:
% git clone --recursive https://github.com/cloudflare/quiche
% cd quiche
% cargo build --package quiche --release --features ffi,pkg-config-meta,qlog
% mkdir quiche/deps/boringssl/src/lib
% ln -vnf $(find target/release -name libcrypto.a -o -name libssl.a) quiche/deps/boringssl/src/lib/
Build curl:
% cd ..
% git clone https://github.com/curl/curl
% cd curl
% autoreconf -fi
% ./configure LDFLAGS="-Wl,-rpath,$PWD/../quiche/target/release" --with-openssl=$PWD/../quiche/quiche/deps/boringssl/src --with-quiche=$PWD/../quiche/target/release
% make
% make install
If `make install` results in `Permission denied` error, you will need to prepend it with `sudo`.
# msh3 (msquic) version
## Build Linux (with quictls fork of OpenSSL)
Build msh3:
% git clone -b v0.6.0 --depth 1 --recursive https://github.com/nibanks/msh3
% cd msh3 && mkdir build && cd build
% cmake -G 'Unix Makefiles' -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
% cmake --build .
% cmake --install .
Build curl:
% git clone https://github.com/curl/curl
% cd curl
% autoreconf -fi
% ./configure LDFLAGS="-Wl,-rpath,/usr/local/lib" --with-msh3=/usr/local --with-openssl
% make
% make install
Run from `/usr/local/bin/curl`.
## Build Windows
Build msh3:
% git clone -b v0.6.0 --depth 1 --recursive https://github.com/nibanks/msh3
% cd msh3 && mkdir build && cd build
% cmake -G 'Visual Studio 17 2022' -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
% cmake --build . --config Release
% cmake --install . --config Release
**Note** - On Windows, Schannel will be used for TLS support by default. If
you with to use (the quictls fork of) OpenSSL, specify the `-DQUIC_TLS=openssl`
option to the generate command above. Also note that OpenSSL brings with it an
additional set of build dependencies not specified here.
Build curl (in [Visual Studio Command prompt](../winbuild/README.md#open-a-command-prompt)):
% git clone https://github.com/curl/curl
% cd curl/winbuild
% nmake /f Makefile.vc mode=dll WITH_MSH3=dll MSH3_PATH="C:/Program Files/msh3" MACHINE=x64
**Note** - If you encounter a build error with `tool_hugehelp.c` being missing,
rename `tool_hugehelp.c.cvs` in the same directory to `tool_hugehelp.c` and
then run `nmake` again.
Run in the `C:/Program Files/msh3/lib` directory, copy `curl.exe` to that
directory, or copy `msquic.dll` and `msh3.dll` from that directory to the
`curl.exe` directory. For example:
% C:\Program Files\msh3\lib> F:\curl\builds\libcurl-vc-x64-release-dll-ipv6-sspi-schannel-msh3\bin\curl.exe --http3 https://www.google.com
# `--http3`
Use only HTTP/3:
curl --http3-only https://nghttp2.org:4433/
Use HTTP/3 with fallback to HTTP/2 or HTTP/1.1 (see "HTTPS eyeballing" below):
curl --http3 https://nghttp2.org:4433/
Upgrade via Alt-Svc:
curl --alt-svc altsvc.cache https://quic.aiortc.org/
See this [list of public HTTP/3 servers](https://bagder.github.io/HTTP3-test/)
### HTTPS eyeballing
With option `--http3` curl will attempt earlier HTTP versions as well should the connect
attempt via HTTP/3 not succeed "fast enough". This strategy is similar to IPv4/6 happy
eyeballing where the alternate address family is used in parallel after a short delay.
The IPv4/6 eyeballing has a default of 200ms and you may override that via `--happy-eyeballs-timeout-ms value`.
Since HTTP/3 is still relatively new, we decided to use this timeout also for the HTTP eyeballing - with a slight twist.
The `happy-eyeballs-timeout-ms` value is the **hard** timeout, meaning after that time expired, a TLS connection is opened in addition to negotiate HTTP/2 or HTTP/1.1. At half of that value - currently - is the **soft** timeout. The soft timeout fires, when there has been **no data at all** seen from the server on the HTTP/3 connection.
So, without you specifying anything, the hard timeout is 200ms and the soft is 100ms:
* Ideally, the whole QUIC handshake happens and curl has an HTTP/3 connection in less than 100ms.
* When QUIC is not supported (or UDP does not work for this network path), no reply is seen and the HTTP/2 TLS+TCP connection starts 100ms later.
* In the worst case, UDP replies start before 100ms, but drag on. This will start the TLS+TCP connection after 200ms.
* When the QUIC handshake fails, the TLS+TCP connection is attempted right away. For example, when the QUIC server presents the wrong certificate.
The whole transfer only fails, when **both** QUIC and TLS+TCP fail to handshake or time out.
Note that all this happens in addition to IP version happy eyeballing. If the name resolution for the server gives more than one IP address, curl will try all those until one succeeds - just as with all other protocols. And if those IP addresses contain both IPv6 and IPv4, those attempts will happen, delayed, in parallel (the actual eyeballing).
## Known Bugs
Check out the [list of known HTTP3 bugs](https://curl.se/docs/knownbugs.html#HTTP3).
# HTTP/3 Test server
This is not advice on how to run anything in production. This is for
development and experimenting.
## Prerequisite(s)
An existing local HTTP/1.1 server that hosts files. Preferably also a few huge
ones. You can easily create huge local files like `truncate -s=8G 8GB` - they
are huge but do not occupy that much space on disk since they are just big
holes.
In a Debian setup you can install **apache2**. It runs on port 80 and has a
document root in `/var/www/html`. Download the 8GB file from apache with `curl
localhost/8GB -o dev/null`
In this description we setup and run an HTTP/3 reverse-proxy in front of the
HTTP/1 server.
## Setup
You can select either or both of these server solutions.
### nghttpx
Get, build and install **quictls**, **nghttp3** and **ngtcp2** as described
above.
Get, build and install **nghttp2**:
git clone https://github.com/nghttp2/nghttp2.git
cd nghttp2
autoreconf -fi
PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/home/daniel/build-quictls/lib/pkgconfig:/home/daniel/build-nghttp3/lib/pkgconfig:/home/daniel/build-ngtcp2/lib/pkgconfig LDFLAGS=-L/home/daniel/build-quictls/lib CFLAGS=-I/home/daniel/build-quictls/include ./configure --enable-maintainer-mode --prefix=/home/daniel/build-nghttp2 --disable-shared --enable-app --enable-http3 --without-jemalloc --without-libxml2 --without-systemd
make && make install
Run the local h3 server on port 9443, make it proxy all traffic through to
HTTP/1 on localhost port 80. For local toying, we can just use the test cert
that exists in curl's test dir.
CERT=$CURLSRC/tests/stunnel.pem
$HOME/bin/nghttpx $CERT $CERT --backend=localhost,80 \
--frontend="localhost,9443;quic"
### Caddy
[Install Caddy](https://caddyserver.com/docs/install). For easiest use, the binary
should be either in your PATH or your current directory.
Create a `Caddyfile` with the following content:
~~~
localhost:7443 {
respond "Hello, world! You're using {http.request.proto}"
}
~~~
Then run Caddy:
./caddy start
Making requests to `https://localhost:7443` should tell you which protocol is being used.
You can change the hard-coded response to something more useful by replacing `respond`
with `reverse_proxy` or `file_server`, for example: `reverse_proxy localhost:80`
---
### Third Party Curl Docs HYPER (`third_party/curl/docs/HYPER.md`)
# Hyper
Hyper is a separate HTTP library written in Rust. curl can be told to use this
library as a backend to deal with HTTP.
## Experimental!
Hyper support in curl is considered **EXPERIMENTAL** until further notice. It
needs to be explicitly enabled at build-time.
Further development and tweaking of the Hyper backend support in curl will
happen in the master branch using pull-requests, just like ordinary
changes.
## Hyper version
The C API for Hyper is brand new and is still under development.
## build curl with hyper
Since March 3 2022, hyper needs the nightly rustc to build, which you may need
to install first with:
% rustup toolchain install nightly
Then build hyper and enable its C API like this:
% git clone https://github.com/hyperium/hyper
% cd hyper
% RUSTFLAGS="--cfg hyper_unstable_ffi" cargo +nightly rustc --features client,http1,http2,ffi -Z unstable-options --crate-type cdylib
Build curl to use hyper's C API:
% git clone https://github.com/curl/curl
% cd curl
% autoreconf -fi
% ./configure --with-hyper=
% make
# using Hyper internally
Hyper is a low level HTTP transport library. curl itself provides all HTTP
headers and Hyper provides all received headers back to curl.
Therefore, most of the "header logic" in curl as in responding to and acting
on specific input and output headers are done the same way in curl code.
The API in Hyper delivers received HTTP headers as (cleaned up) name=value
pairs, making it impossible for curl to know the exact byte representation
over the wire with Hyper.
## Limitations
The hyper backend does not support
- `CURLOPT_IGNORE_CONTENT_LENGTH`
- `--raw` and disabling `CURLOPT_HTTP_TRANSFER_DECODING`
- RTSP
- hyper is much stricter about what HTTP header contents it allows
- HTTP/0.9
- HTTP/2 upgrade using HTTP:// URLs. Aka 'h2c'
## Remaining issues
This backend is still not feature complete with the native backend. Areas that
still need attention and verification include:
- multiplexed HTTP/2
- h2 Upgrade:
- pausing transfers
- receiving HTTP/1 trailers
- sending HTTP/1 trailers
---
### Third Party Curl Docs INSTALL (`third_party/curl/docs/INSTALL.md`)
# how to install curl and libcurl
## Installing Binary Packages
Lots of people download binary distributions of curl and libcurl. This
document does not describe how to install curl or libcurl using such a binary
package. This document describes how to compile, build and install curl and
libcurl from source code.
## Building using vcpkg
You can download and install curl and libcurl using the [vcpkg](https://github.com/Microsoft/vcpkg/) dependency manager:
git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg integrate install
vcpkg install curl[tool]
The curl port in vcpkg is kept up to date by Microsoft team members and
community contributors. If the version is out of date, please [create an issue
or pull request](https://github.com/Microsoft/vcpkg) on the vcpkg repository.
## Building from git
If you get your code off a git repository instead of a release tarball, see
the `GIT-INFO` file in the root directory for specific instructions on how to
proceed.
# Unix
A normal Unix installation is made in three or four steps (after you have
unpacked the source archive):
./configure --with-openssl [--with-gnutls --with-wolfssl]
make
make test (optional)
make install
(Adjust the configure line accordingly to use the TLS library you want.)
You probably need to be root when doing the last command.
Get a full listing of all available configure options by invoking it like:
./configure --help
If you want to install curl in a different file hierarchy than `/usr/local`,
specify that when running configure:
./configure --prefix=/path/to/curl/tree
If you have write permission in that directory, you can do 'make install'
without being root. An example of this would be to make a local install in
your own home directory:
./configure --prefix=$HOME
make
make install
The configure script always tries to find a working SSL library unless
explicitly told not to. If you have OpenSSL installed in the default search
path for your compiler/linker, you do not need to do anything special. If you
have OpenSSL installed in `/usr/local/ssl`, you can run configure like:
./configure --with-openssl
If you have OpenSSL installed somewhere else (for example, `/opt/OpenSSL`) and
you have pkg-config installed, set the pkg-config path first, like this:
env PKG_CONFIG_PATH=/opt/OpenSSL/lib/pkgconfig ./configure --with-openssl
Without pkg-config installed, use this:
./configure --with-openssl=/opt/OpenSSL
If you insist on forcing a build without SSL support, you can run configure
like this:
./configure --without-ssl
If you have OpenSSL installed, but with the libraries in one place and the
header files somewhere else, you have to set the `LDFLAGS` and `CPPFLAGS`
environment variables prior to running configure. Something like this should
work:
CPPFLAGS="-I/path/to/ssl/include" LDFLAGS="-L/path/to/ssl/lib" ./configure
If you have shared SSL libs installed in a directory where your runtime
linker does not find them (which usually causes configure failures), you can
provide this option to gcc to set a hard-coded path to the runtime linker:
LDFLAGS=-Wl,-R/usr/local/ssl/lib ./configure --with-openssl
## Static builds
To force a static library compile, disable the shared library creation by
running configure like:
./configure --disable-shared
The configure script is primarily done to work with shared/dynamic third party
dependencies. When linking with shared libraries, the dependency "chain" is
handled automatically by the library loader - on all modern systems.
If you instead link with a static library, you need to provide all the
dependency libraries already at the link command line.
Figuring out all the dependency libraries for a given library is hard, as it
might involve figuring out the dependencies of the dependencies and they vary
between platforms and change between versions.
When using static dependencies, the build scripts will mostly assume that you,
the user, will provide all the necessary additional dependency libraries as
additional arguments in the build. With configure, by setting `LIBS` or
`LDFLAGS` on the command line.
Building statically is not for the faint of heart.
## Debug
If you are a curl developer and use gcc, you might want to enable more debug
options with the `--enable-debug` option.
curl can be built to use a whole range of libraries to provide various useful
services, and configure will try to auto-detect a decent default. But if you
want to alter it, you can select how to deal with each individual library.
## Select TLS backend
These options are provided to select the TLS backend to use.
- AmiSSL: `--with-amissl`
- BearSSL: `--with-bearssl`
- GnuTLS: `--with-gnutls`.
- mbedTLS: `--with-mbedtls`
- NSS: `--with-nss`
- OpenSSL: `--with-openssl` (also for BoringSSL, AWS-LC, libressl, and quictls)
- rustls: `--with-rustls`
- Schannel: `--with-schannel`
- Secure Transport: `--with-secure-transport`
- wolfSSL: `--with-wolfssl`
You can build curl with *multiple* TLS backends at your choice, but some TLS
backends cannot be combined: if you build with an OpenSSL fork (or wolfSSL),
you cannot add another OpenSSL fork (or wolfSSL) simply because they have
conflicting identical symbol names.
When you build with multiple TLS backends, you can select the active one at
run-time when curl starts up.
## configure finding libs in wrong directory
When the configure script checks for third-party libraries, it adds those
directories to the `LDFLAGS` variable and then tries linking to see if it
works. When successful, the found directory is kept in the `LDFLAGS` variable
when the script continues to execute and do more tests and possibly check for
more libraries.
This can make subsequent checks for libraries wrongly detect another
installation in a directory that was previously added to `LDFLAGS` by another
library check.
# Windows
## Building Windows DLLs and C runtime (CRT) linkage issues
As a general rule, building a DLL with static CRT linkage is highly
discouraged, and intermixing CRTs in the same app is something to avoid at
any cost.
Reading and comprehending Microsoft Knowledge Base articles KB94248 and
KB140584 is a must for any Windows developer. Especially important is full
understanding if you are not going to follow the advice given above.
- [How To Use the C Run-Time](https://support.microsoft.com/help/94248/how-to-use-the-c-run-time)
- [Run-Time Library Compiler Options](https://docs.microsoft.com/cpp/build/reference/md-mt-ld-use-run-time-library)
- [Potential Errors Passing CRT Objects Across DLL Boundaries](https://docs.microsoft.com/cpp/c-runtime-library/potential-errors-passing-crt-objects-across-dll-boundaries)
If your app is misbehaving in some strange way, or it is suffering from memory
corruption, before asking for further help, please try first to rebuild every
single library your app uses as well as your app using the debug
multi-threaded dynamic C runtime.
If you get linkage errors read section 5.7 of the FAQ document.
## MinGW32
Make sure that MinGW32's bin directory is in the search path, for example:
```cmd
set PATH=c:\mingw32\bin;%PATH%
```
then run `mingw32-make mingw32` in the root dir. There are other
make targets available to build libcurl with more features, use:
- `mingw32-make mingw32-zlib` to build with Zlib support;
- `mingw32-make mingw32-ssl-zlib` to build with SSL and Zlib enabled;
- `mingw32-make mingw32-ssh2-ssl-zlib` to build with SSH2, SSL, Zlib;
- `mingw32-make mingw32-ssh2-ssl-sspi-zlib` to build with SSH2, SSL, Zlib
and SSPI support.
If you have any problems linking libraries or finding header files, be sure
to verify that the provided `Makefile.mk` files use the proper paths, and
adjust as necessary. It is also possible to override these paths with
environment variables, for example:
```cmd
set ZLIB_PATH=c:\zlib-1.2.12
set OPENSSL_PATH=c:\openssl-3.0.5
set LIBSSH2_PATH=c:\libssh2-1.10.0
```
It is also possible to build with other LDAP installations than MS LDAP;
currently it is possible to build with native Win32 OpenLDAP, or with the
*Novell CLDAP* SDK. If you want to use these you need to set these vars:
```cmd
set CPPFLAGS=-Ic:/openldap/include -DCURL_HAS_OPENLDAP_LDAPSDK
set LDFLAGS=-Lc:/openldap/lib
set LIBS=-lldap -llber
```
or for using the Novell SDK:
```cmd
set CPPFLAGS=-Ic:/openldapsdk/inc -DCURL_HAS_NOVELL_LDAPSDK
set LDFLAGS=-Lc:/openldapsdk/lib/mscvc
set LIBS=-lldapsdk -lldapssl -lldapx
```
If you want to enable LDAPS support then append `-ldaps` to the make target.
## Cygwin
Almost identical to the Unix installation. Run the configure script in the
curl source tree root with `sh configure`. Make sure you have the `sh`
executable in `/bin/` or you will see the configure fail toward the end.
Run `make`
## MS-DOS
Requires DJGPP in the search path and pointing to the Watt-32 stack via
`WATT_PATH=c:/djgpp/net/watt`.
Run `make -f Makefile.dist djgpp` in the root curl dir.
For build configuration options, please see the MinGW32 section.
Notes:
- DJGPP 2.04 beta has a `sscanf()` bug so the URL parsing is not done
properly. Use DJGPP 2.03 until they fix it.
- Compile Watt-32 (and OpenSSL) with the same version of DJGPP. Otherwise
things go wrong because things like FS-extensions and `errno` values have
been changed between releases.
## AmigaOS
Run `make -f Makefile.dist amiga` in the root curl dir.
For build configuration options, please see the MinGW32 section.
## Disabling Specific Protocols in Windows builds
The configure utility, unfortunately, is not available for the Windows
environment, therefore, you cannot use the various disable-protocol options of
the configure utility on this platform.
You can use specific defines to disable specific protocols and features. See
[CURL-DISABLE](CURL-DISABLE.md) for the full list.
If you want to set any of these defines you have the following options:
- Modify `lib/config-win32.h`
- Modify `lib/curl_setup.h`
- Modify `winbuild/Makefile.vc`
- Modify the "Preprocessor Definitions" in the libcurl project
Note: The pre-processor settings can be found using the Visual Studio IDE
under "Project -> Properties -> Configuration Properties -> C/C++ ->
Preprocessor".
## Using BSD-style lwIP instead of Winsock TCP/IP stack in Win32 builds
In order to compile libcurl and curl using BSD-style lwIP TCP/IP stack it is
necessary to make the definition of the preprocessor symbol `USE_LWIPSOCK`
visible to libcurl and curl compilation processes. To set this definition you
have the following alternatives:
- Modify `lib/config-win32.h` and `src/config-win32.h`
- Modify `winbuild/Makefile.vc`
- Modify the "Preprocessor Definitions" in the libcurl project
Note: The pre-processor settings can be found using the Visual Studio IDE
under "Project -> Properties -> Configuration Properties -> C/C++ ->
Preprocessor".
Once that libcurl has been built with BSD-style lwIP TCP/IP stack support, in
order to use it with your program it is mandatory that your program includes
lwIP header file `` (or another lwIP header that includes this)
before including any libcurl header. Your program does not need the
`USE_LWIPSOCK` preprocessor definition which is for libcurl internals only.
Compilation has been verified with lwIP 1.4.0.
This BSD-style lwIP TCP/IP stack support must be considered experimental given
that it has been verified that lwIP 1.4.0 still needs some polish, and libcurl
might yet need some additional adjustment.
## Important static libcurl usage note
When building an application that uses the static libcurl library on Windows,
you must add `-DCURL_STATICLIB` to your `CFLAGS`. Otherwise the linker will
look for dynamic import symbols.
## Legacy Windows and SSL
Schannel (from Windows SSPI), is the native SSL library in Windows. However,
Schannel in Windows <= XP is unable to connect to servers that
no longer support the legacy handshakes and algorithms used by those
versions. If you will be using curl in one of those earlier versions of
Windows you should choose another SSL backend such as OpenSSL.
# Apple Platforms (macOS, iOS, tvOS, watchOS, and their simulator counterparts)
On modern Apple operating systems, curl can be built to use Apple's SSL/TLS
implementation, Secure Transport, instead of OpenSSL. To build with Secure
Transport for SSL/TLS, use the configure option `--with-secure-transport`.
When Secure Transport is in use, the curl options `--cacert` and `--capath`
and their libcurl equivalents, will be ignored, because Secure Transport uses
the certificates stored in the Keychain to evaluate whether or not to trust
the server. This, of course, includes the root certificates that ship with the
OS. The `--cert` and `--engine` options, and their libcurl equivalents, are
currently unimplemented in curl with Secure Transport.
In general, a curl build for an Apple `ARCH/SDK/DEPLOYMENT_TARGET` combination
can be taken by providing appropriate values for `ARCH`, `SDK`, `DEPLOYMENT_TARGET`
below and running the commands:
```bash
# Set these three according to your needs
export ARCH=x86_64
export SDK=macosx
export DEPLOYMENT_TARGET=10.8
export CFLAGS="-arch $ARCH -isysroot $(xcrun -sdk $SDK --show-sdk-path) -m$SDK-version-min=$DEPLOYMENT_TARGET"
./configure --host=$ARCH-apple-darwin --prefix $(pwd)/artifacts --with-secure-transport
make -j8
make install
```
Above will build curl for macOS platform with `x86_64` architecture and `10.8` as deployment target.
Here is an example for iOS device:
```bash
export ARCH=arm64
export SDK=iphoneos
export DEPLOYMENT_TARGET=11.0
export CFLAGS="-arch $ARCH -isysroot $(xcrun -sdk $SDK --show-sdk-path) -m$SDK-version-min=$DEPLOYMENT_TARGET"
./configure --host=$ARCH-apple-darwin --prefix $(pwd)/artifacts --with-secure-transport
make -j8
make install
```
Another example for watchOS simulator for macs with Apple Silicon:
```bash
export ARCH=arm64
export SDK=watchsimulator
export DEPLOYMENT_TARGET=5.0
export CFLAGS="-arch $ARCH -isysroot $(xcrun -sdk $SDK --show-sdk-path) -m$SDK-version-min=$DEPLOYMENT_TARGET"
./configure --host=$ARCH-apple-darwin --prefix $(pwd)/artifacts --with-secure-transport
make -j8
make install
```
In all above, the built libraries and executables can be found in the
`artifacts` folder.
# Android
When building curl for Android it's recommended to use a Linux/macOS environment
since using curl's `configure` script is the easiest way to build curl
for Android. Before you can build curl for Android, you need to install the
Android NDK first. This can be done using the SDK Manager that is part of
Android Studio. Once you have installed the Android NDK, you need to figure out
where it has been installed and then set up some environment variables before
launching `configure`. On macOS, those variables could look like this to compile
for `aarch64` and API level 29:
```bash
export ANDROID_NDK_HOME=~/Library/Android/sdk/ndk/25.1.8937393 # Point into your NDK.
export HOST_TAG=darwin-x86_64 # Same tag for Apple Silicon. Other OS values here: https://developer.android.com/ndk/guides/other_build_systems#overview
export TOOLCHAIN=$ANDROID_NDK_HOME/toolchains/llvm/prebuilt/$HOST_TAG
export AR=$TOOLCHAIN/bin/llvm-ar
export AS=$TOOLCHAIN/bin/llvm-as
export CC=$TOOLCHAIN/bin/aarch64-linux-android21-clang
export CXX=$TOOLCHAIN/bin/aarch64-linux-android21-clang++
export LD=$TOOLCHAIN/bin/ld
export RANLIB=$TOOLCHAIN/bin/llvm-ranlib
export STRIP=$TOOLCHAIN/bin/llvm-strip
```
When building on Linux or targeting other API levels or architectures, you need
to adjust those variables accordingly. After that you can build curl like this:
./configure --host aarch64-linux-android --with-pic --disable-shared
Note that this will not give you SSL/TLS support. If you need SSL/TLS, you have
to build curl against a SSL/TLS layer, e.g. OpenSSL, because it's impossible for
curl to access Android's native SSL/TLS layer. To build curl for Android using
OpenSSL, follow the OpenSSL build instructions and then install `libssl.a` and
`libcrypto.a` to `$TOOLCHAIN/sysroot/usr/lib` and copy `include/openssl` to
`$TOOLCHAIN/sysroot/usr/include`. Now you can build curl for Android using
OpenSSL like this:
```bash
LIBS="-lssl -lcrypto -lc++" # For OpenSSL/BoringSSL. In general, you will need to the SSL/TLS layer's transitive dependencies if you are linking statically.
./configure --host aarch64-linux-android --with-pic --disable-shared --with-openssl="$TOOLCHAIN/sysroot/usr"
```
# IBM i
For IBM i (formerly OS/400), you can use curl in two different ways:
- Natively, running in the **ILE**. The obvious use is being able to call curl
from ILE C or RPG applications.
- You will need to build this from source. See `packages/OS400/README` for
the ILE specific build instructions.
- In the **PASE** environment, which runs AIX programs. curl will be built as
it would be on AIX.
- IBM provides builds of curl in their Yum repository for PASE software.
- To build from source, follow the Unix instructions.
There are some additional limitations and quirks with curl on this platform;
they affect both environments.
## Multi-threading notes
By default, jobs in IBM i will not start with threading enabled. (Exceptions
include interactive PASE sessions started by `QP2TERM` or SSH.) If you use
curl in an environment without threading when options like asynchronous DNS
were enabled, you will get messages like:
```
getaddrinfo() thread failed to start
```
Do not panic. curl and your program are not broken. You can fix this by:
- Set the environment variable `QIBM_MULTI_THREADED` to `Y` before starting
your program. This can be done at whatever scope you feel is appropriate.
- Alternatively, start the job with the `ALWMLTTHD` parameter set to `*YES`.
# Cross compile
Download and unpack the curl package.
`cd` to the new directory. (e.g. `cd curl-7.12.3`)
Set environment variables to point to the cross-compile toolchain and call
configure with any options you need. Be sure and specify the `--host` and
`--build` parameters at configuration time. The following script is an example
of cross-compiling for the IBM 405GP PowerPC processor using the toolchain on
Linux.
```bash
#! /bin/sh
export PATH=$PATH:/opt/hardhat/devkit/ppc/405/bin
export CPPFLAGS="-I/opt/hardhat/devkit/ppc/405/target/usr/include"
export AR=ppc_405-ar
export AS=ppc_405-as
export LD=ppc_405-ld
export RANLIB=ppc_405-ranlib
export CC=ppc_405-gcc
export NM=ppc_405-nm
./configure --target=powerpc-hardhat-linux
--host=powerpc-hardhat-linux
--build=i586-pc-linux-gnu
--prefix=/opt/hardhat/devkit/ppc/405/target/usr/local
--exec-prefix=/usr/local
```
You may also need to provide a parameter like `--with-random=/dev/urandom` to
configure as it cannot detect the presence of a random number generating
device for a target system. The `--prefix` parameter specifies where curl
will be installed. If `configure` completes successfully, do `make` and `make
install` as usual.
In some cases, you may be able to simplify the above commands to as little as:
./configure --host=ARCH-OS
# REDUCING SIZE
There are a number of configure options that can be used to reduce the size of
libcurl for embedded applications where binary size is an important factor.
First, be sure to set the `CFLAGS` variable when configuring with any relevant
compiler optimization flags to reduce the size of the binary. For gcc, this
would mean at minimum the -Os option, and potentially the `-march=X`,
`-mdynamic-no-pic` and `-flto` options as well, e.g.
./configure CFLAGS='-Os' LDFLAGS='-Wl,-Bsymbolic'...
Note that newer compilers often produce smaller code than older versions
due to improved optimization.
Be sure to specify as many `--disable-` and `--without-` flags on the
configure command-line as you can to disable all the libcurl features that you
know your application is not going to need. Besides specifying the
`--disable-PROTOCOL` flags for all the types of URLs your application will not
use, here are some other flags that can reduce the size of the library by
disabling support for some feature:
- `--disable-alt-svc` (HTTP Alt-Svc)
- `--disable-ares` (the C-ARES DNS library)
- `--disable-cookies` (HTTP cookies)
- `--disable-crypto-auth` (cryptographic authentication)
- `--disable-dateparse` (date parsing for time conditionals)
- `--disable-dnsshuffle` (internal server load spreading)
- `--disable-doh` (DNS-over-HTTP)
- `--disable-get-easy-options` (lookup easy options at runtime)
- `--disable-hsts` (HTTP Strict Transport Security)
- `--disable-http-auth` (all HTTP authentication)
- `--disable-ipv6` (IPv6)
- `--disable-libcurl-option` (--libcurl C code generation support)
- `--disable-manual` (built-in documentation)
- `--disable-netrc` (.netrc file)
- `--disable-ntlm-wb` (NTLM WinBind)
- `--disable-progress-meter` (graphical progress meter in library)
- `--disable-proxy` (HTTP and SOCKS proxies)
- `--disable-pthreads` (multi-threading)
- `--disable-socketpair` (socketpair for asynchronous name resolving)
- `--disable-threaded-resolver` (threaded name resolver)
- `--disable-tls-srp` (Secure Remote Password authentication for TLS)
- `--disable-unix-sockets` (UNIX sockets)
- `--disable-verbose` (eliminates debugging strings and error code strings)
- `--disable-versioned-symbols` (versioned symbols)
- `--enable-symbol-hiding` (eliminates unneeded symbols in the shared library)
- `--without-brotli` (Brotli on-the-fly decompression)
- `--without-libpsl` (Public Suffix List in cookies)
- `--without-nghttp2` (HTTP/2 using nghttp2)
- `--without-ngtcp2` (HTTP/2 using ngtcp2)
- `--without-zstd` (Zstd on-the-fly decompression)
- `--without-libidn2` (internationalized domain names)
- `--without-librtmp` (RTMP)
- `--without-ssl` (SSL/TLS)
- `--without-zlib` (on-the-fly decompression)
The GNU compiler and linker have a number of options that can reduce the
size of the libcurl dynamic libraries on some platforms even further.
Specify them by providing appropriate `CFLAGS` and `LDFLAGS` variables on
the configure command-line, e.g.
CFLAGS="-Os -ffunction-sections -fdata-sections
-fno-unwind-tables -fno-asynchronous-unwind-tables -flto"
LDFLAGS="-Wl,-s -Wl,-Bsymbolic -Wl,--gc-sections"
Be sure also to strip debugging symbols from your binaries after compiling
using 'strip' (or the appropriate variant if cross-compiling). If space is
really tight, you may be able to remove some unneeded sections of the shared
library using the -R option to objcopy (e.g. the .comment section).
Using these techniques it is possible to create a basic HTTP-only libcurl
shared library for i386 Linux platforms that is only 133 KiB in size
(as of libcurl version 7.80.0, using gcc 11.2.0).
You may find that statically linking libcurl to your application will result
in a lower total size than dynamically linking.
Note that the curl test harness can detect the use of some, but not all, of
the `--disable` statements suggested above. Use will cause tests relying on
those features to fail. The test harness can be manually forced to skip the
relevant tests by specifying certain key words on the `runtests.pl` command
line. Following is a list of appropriate key words for those configure options
that are not automatically detected:
- `--disable-cookies` !cookies
- `--disable-dateparse` !RETRY-AFTER !`CURLOPT_TIMECONDITION` !`CURLINFO_FILETIME` !`If-Modified-Since` !`curl_getdate` !`-z`
- `--disable-libcurl-option` !`--libcurl`
- `--disable-verbose` !verbose\ logs
# PORTS
This is a probably incomplete list of known CPU architectures and operating
systems that curl has been compiled for. If you know a system curl compiles
and runs on, that is not listed, please let us know!
## 92 Operating Systems
AIX, AmigaOS, Android, Aros, BeOS, Blackberry 10, Blackberry Tablet OS,
Cell OS, Chrome OS, Cisco IOS, Cygwin, DG/UX, Dragonfly BSD, DR DOS, eCOS,
FreeBSD, FreeDOS, FreeRTOS, Fuchsia, Garmin OS, Genode, Haiku, HardenedBSD,
HP-UX, Hurd, Illumos, Integrity, iOS, ipadOS, IRIX, Linux, Lua RTOS,
Mac OS 9, macOS, Mbed, Micrium, MINIX, MorphOS, MPE/iX, MS-DOS, NCR MP-RAS,
NetBSD, Netware, Nintendo Switch, NonStop OS, NuttX, Omni OS, OpenBSD,
OpenStep, Orbis OS, OS/2, OS/400, OS21, Plan 9, PlayStation Portable, QNX,
Qubes OS, ReactOS, Redox, RICS OS, RTEMS, Sailfish OS, SCO Unix, Serenity,
SINIX-Z, Solaris, SunOS, Syllable OS, Symbian, Tizen, TPF, Tru64, tvOS,
ucLinux, Ultrix, UNICOS, UnixWare, VMS, vxWorks, watchOS, WebOS,
Wii system software, Windows, Windows CE, Xbox System, Xenix, Zephyr,
z/OS, z/TPF, z/VM, z/VSE
## 26 CPU Architectures
Alpha, ARC, ARM, AVR32, CompactRISC, Elbrus, ETRAX, HP-PA, Itanium,
LoongArch, m68k, m88k, MicroBlaze, MIPS, Nios, OpenRISC, POWER, PowerPC,
RISC-V, s390, SH4, SPARC, Tilera, VAX, x86, Xtensa
---
### Third Party Curl Docs INTERNALS (`third_party/curl/docs/INTERNALS.md`)
# curl internals
The canonical libcurl internals documentation is now in the [everything
curl](https://everything.curl.dev/internals) book. This file lists supported
versions of libs and build tools.
## Portability
We write curl and libcurl to compile with C89 compilers on 32-bit and up
machines. Most of libcurl assumes more or less POSIX compliance but that is
not a requirement.
We write libcurl to build and work with lots of third party tools, and we
want it to remain functional and buildable with these and later versions
(older versions may still work but is not what we work hard to maintain):
## Dependencies
We aim to support these or later versions.
- OpenSSL 0.9.7
- GnuTLS 3.1.10
- zlib 1.1.4
- libssh2 1.0
- c-ares 1.16.0
- libidn2 2.0.0
- wolfSSL 2.0.0
- OpenLDAP 2.0
- MIT Kerberos 1.2.4
- GSKit V5R3M0
- NSS 3.14.x
- Heimdal ?
- nghttp2 1.12.0
- WinSock 2.2 (on Windows 95+ and Windows CE .NET 4.1+)
## Build tools
When writing code (mostly for generating stuff included in release tarballs)
we use a few "build tools" and we make sure that we remain functional with
these versions:
- GNU Libtool 1.4.2
- GNU Autoconf 2.59
- GNU Automake 1.7
- GNU M4 1.4
- perl 5.6
- roffit 0.5
- nroff any version that supports `-man [in] [out]`
- cmake 3.7
Library Symbols
===============
All symbols used internally in libcurl must use a `Curl_` prefix if they are
used in more than a single file. Single-file symbols must be made static.
Public ("exported") symbols must use a `curl_` prefix. Public API functions
are marked with `CURL_EXTERN` in the public header files so that all others
can be hidden on platforms where this is possible.
---
### Third Party Curl Docs MANUAL (`third_party/curl/docs/MANUAL.md`)
# curl tutorial
## Simple Usage
Get the main page from a web-server:
curl https://www.example.com/
Get a README file from an FTP server:
curl ftp://ftp.funet.fi/README
Get a web page from a server using port 8000:
curl http://www.weirdserver.com:8000/
Get a directory listing of an FTP site:
curl ftp://ftp.funet.fi
Get the all terms matching curl from a dictionary:
curl dict://dict.org/m:curl
Get the definition of curl from a dictionary:
curl dict://dict.org/d:curl
Fetch two documents at once:
curl ftp://ftp.funet.fi/ http://www.weirdserver.com:8000/
Get a file off an FTPS server:
curl ftps://files.are.secure.com/secrets.txt
or use the more appropriate FTPS way to get the same file:
curl --ftp-ssl ftp://files.are.secure.com/secrets.txt
Get a file from an SSH server using SFTP:
curl -u username sftp://example.com/etc/issue
Get a file from an SSH server using SCP using a private key (not
password-protected) to authenticate:
curl -u username: --key ~/.ssh/id_rsa scp://example.com/~/file.txt
Get a file from an SSH server using SCP using a private key
(password-protected) to authenticate:
curl -u username: --key ~/.ssh/id_rsa --pass private_key_password
scp://example.com/~/file.txt
Get the main page from an IPv6 web server:
curl "http://[2001:1890:1112:1::20]/"
Get a file from an SMB server:
curl -u "domain\username:passwd" smb://server.example.com/share/file.txt
## Download to a File
Get a web page and store in a local file with a specific name:
curl -o thatpage.html http://www.example.com/
Get a web page and store in a local file, make the local file get the name of
the remote document (if no file name part is specified in the URL, this will
fail):
curl -O http://www.example.com/index.html
Fetch two files and store them with their remote names:
curl -O www.haxx.se/index.html -O curl.se/download.html
## Using Passwords
### FTP
To ftp files using name and password, include them in the URL like:
curl ftp://name:passwd@machine.domain:port/full/path/to/file
or specify them with the `-u` flag like
curl -u name:passwd ftp://machine.domain:port/full/path/to/file
### FTPS
It is just like for FTP, but you may also want to specify and use SSL-specific
options for certificates etc.
Note that using `FTPS://` as prefix is the *implicit* way as described in the
standards while the recommended *explicit* way is done by using `FTP://` and
the `--ssl-reqd` option.
### SFTP / SCP
This is similar to FTP, but you can use the `--key` option to specify a
private key to use instead of a password. Note that the private key may itself
be protected by a password that is unrelated to the login password of the
remote system; this password is specified using the `--pass` option.
Typically, curl will automatically extract the public key from the private key
file, but in cases where curl does not have the proper library support, a
matching public key file must be specified using the `--pubkey` option.
### HTTP
Curl also supports user and password in HTTP URLs, thus you can pick a file
like:
curl http://name:passwd@machine.domain/full/path/to/file
or specify user and password separately like in
curl -u name:passwd http://machine.domain/full/path/to/file
HTTP offers many different methods of authentication and curl supports
several: Basic, Digest, NTLM and Negotiate (SPNEGO). Without telling which
method to use, curl defaults to Basic. You can also ask curl to pick the most
secure ones out of the ones that the server accepts for the given URL, by
using `--anyauth`.
**Note**! According to the URL specification, HTTP URLs can not contain a user
and password, so that style will not work when using curl via a proxy, even
though curl allows it at other times. When using a proxy, you _must_ use the
`-u` style for user and password.
### HTTPS
Probably most commonly used with private certificates, as explained below.
## Proxy
curl supports both HTTP and SOCKS proxy servers, with optional authentication.
It does not have special support for FTP proxy servers since there are no
standards for those, but it can still be made to work with many of them. You
can also use both HTTP and SOCKS proxies to transfer files to and from FTP
servers.
Get an ftp file using an HTTP proxy named my-proxy that uses port 888:
curl -x my-proxy:888 ftp://ftp.leachsite.com/README
Get a file from an HTTP server that requires user and password, using the
same proxy as above:
curl -u user:passwd -x my-proxy:888 http://www.get.this/
Some proxies require special authentication. Specify by using -U as above:
curl -U user:passwd -x my-proxy:888 http://www.get.this/
A comma-separated list of hosts and domains which do not use the proxy can be
specified as:
curl --noproxy localhost,get.this -x my-proxy:888 http://www.get.this/
If the proxy is specified with `--proxy1.0` instead of `--proxy` or `-x`, then
curl will use HTTP/1.0 instead of HTTP/1.1 for any `CONNECT` attempts.
curl also supports SOCKS4 and SOCKS5 proxies with `--socks4` and `--socks5`.
See also the environment variables Curl supports that offer further proxy
control.
Most FTP proxy servers are set up to appear as a normal FTP server from the
client's perspective, with special commands to select the remote FTP server.
curl supports the `-u`, `-Q` and `--ftp-account` options that can be used to
set up transfers through many FTP proxies. For example, a file can be uploaded
to a remote FTP server using a Blue Coat FTP proxy with the options:
curl -u "username@ftp.server Proxy-Username:Remote-Pass"
--ftp-account Proxy-Password --upload-file local-file
ftp://my-ftp.proxy.server:21/remote/upload/path/
See the manual for your FTP proxy to determine the form it expects to set up
transfers, and curl's `-v` option to see exactly what curl is sending.
## Piping
Get a key file and add it with `apt-key` (when on a system that uses `apt` for
package management):
curl -L https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
The '|' pipes the output to STDIN. `-` tells `apt-key` that the key file
should be read from STDIN.
## Ranges
HTTP 1.1 introduced byte-ranges. Using this, a client can request to get only
one or more sub-parts of a specified document. Curl supports this with the
`-r` flag.
Get the first 100 bytes of a document:
curl -r 0-99 http://www.get.this/
Get the last 500 bytes of a document:
curl -r -500 http://www.get.this/
Curl also supports simple ranges for FTP files as well. Then you can only
specify start and stop position.
Get the first 100 bytes of a document using FTP:
curl -r 0-99 ftp://www.get.this/README
## Uploading
### FTP / FTPS / SFTP / SCP
Upload all data on stdin to a specified server:
curl -T - ftp://ftp.upload.com/myfile
Upload data from a specified file, login with user and password:
curl -T uploadfile -u user:passwd ftp://ftp.upload.com/myfile
Upload a local file to the remote site, and use the local file name at the
remote site too:
curl -T uploadfile -u user:passwd ftp://ftp.upload.com/
Upload a local file to get appended to the remote file:
curl -T localfile -a ftp://ftp.upload.com/remotefile
Curl also supports ftp upload through a proxy, but only if the proxy is
configured to allow that kind of tunneling. If it does, you can run curl in a
fashion similar to:
curl --proxytunnel -x proxy:port -T localfile ftp.upload.com
### SMB / SMBS
curl -T file.txt -u "domain\username:passwd"
smb://server.example.com/share/
### HTTP
Upload all data on stdin to a specified HTTP site:
curl -T - http://www.upload.com/myfile
Note that the HTTP server must have been configured to accept PUT before this
can be done successfully.
For other ways to do HTTP data upload, see the POST section below.
## Verbose / Debug
If curl fails where it is not supposed to, if the servers do not let you in, if
you cannot understand the responses: use the `-v` flag to get verbose
fetching. Curl will output lots of info and what it sends and receives in
order to let the user see all client-server interaction (but it will not show you
the actual data).
curl -v ftp://ftp.upload.com/
To get even more details and information on what curl does, try using the
`--trace` or `--trace-ascii` options with a given file name to log to, like
this:
curl --trace trace.txt www.haxx.se
## Detailed Information
Different protocols provide different ways of getting detailed information
about specific files/documents. To get curl to show detailed information about
a single file, you should use `-I`/`--head` option. It displays all available
info on a single file for HTTP and FTP. The HTTP information is a lot more
extensive.
For HTTP, you can get the header information (the same as `-I` would show)
shown before the data by using `-i`/`--include`. Curl understands the
`-D`/`--dump-header` option when getting files from both FTP and HTTP, and it
will then store the headers in the specified file.
Store the HTTP headers in a separate file (headers.txt in the example):
curl --dump-header headers.txt curl.se
Note that headers stored in a separate file can be useful at a later time if
you want curl to use cookies sent by the server. More about that in the
cookies section.
## POST (HTTP)
It is easy to post data using curl. This is done using the `-d ` option.
The post data must be urlencoded.
Post a simple `name` and `phone` guestbook.
curl -d "name=Rafael%20Sagula&phone=3320780" http://www.where.com/guest.cgi
Or automatically [URL encode the data](https://everything.curl.dev/http/post/url-encode).
curl --data-urlencode "name=Rafael Sagula&phone=3320780" http://www.where.com/guest.cgi
How to post a form with curl, lesson #1:
Dig out all the `` tags in the form that you want to fill in.
If there is a normal post, you use `-d` to post. `-d` takes a full post
string, which is in the format
=&=&...
The variable names are the names set with `"name="` in the `` tags, and
the data is the contents you want to fill in for the inputs. The data *must*
be properly URL encoded. That means you replace space with + and that you
replace weird letters with `%XX` where `XX` is the hexadecimal representation
of the letter's ASCII code.
Example:
(page located at `http://www.formpost.com/getthis/`)
```html
```
We want to enter user `foobar` with password `12345`.
To post to this, you enter a curl command line like:
curl -d "user=foobar&pass=12345&id=blablabla&ding=submit"
http://www.formpost.com/getthis/post.cgi
While `-d` uses the application/x-www-form-urlencoded mime-type, generally
understood by CGI's and similar, curl also supports the more capable
multipart/form-data type. This latter type supports things like file upload.
`-F` accepts parameters like `-F "name=contents"`. If you want the contents to
be read from a file, use `@filename` as contents. When specifying a file, you
can also specify the file content type by appending `;type=` to the
file name. You can also post the contents of several files in one field. For
example, the field name `coolfiles` is used to send three files, with
different content types using the following syntax:
curl -F "coolfiles=@fil1.gif;type=image/gif,fil2.txt,fil3.html"
http://www.post.com/postit.cgi
If the content-type is not specified, curl will try to guess from the file
extension (it only knows a few), or use the previously specified type (from an
earlier file if several files are specified in a list) or else it will use the
default type `application/octet-stream`.
Emulate a fill-in form with `-F`. Let's say you fill in three fields in a
form. One field is a file name which to post, one field is your name and one
field is a file description. We want to post the file we have written named
`cooltext.txt`. To let curl do the posting of this data instead of your
favorite browser, you have to read the HTML source of the form page and find
the names of the input fields. In our example, the input field names are
`file`, `yourname` and `filedescription`.
curl -F "file=@cooltext.txt" -F "yourname=Daniel"
-F "filedescription=Cool text file with cool text inside"
http://www.post.com/postit.cgi
To send two files in one post you can do it in two ways:
Send multiple files in a single field with a single field name:
curl -F "pictures=@dog.gif,cat.gif" $URL
Send two fields with two field names
curl -F "docpicture=@dog.gif" -F "catpicture=@cat.gif" $URL
To send a field value literally without interpreting a leading `@` or `<`, or
an embedded `;type=`, use `--form-string` instead of `-F`. This is recommended
when the value is obtained from a user or some other unpredictable
source. Under these circumstances, using `-F` instead of `--form-string` could
allow a user to trick curl into uploading a file.
## Referrer
An HTTP request has the option to include information about which address
referred it to the actual page. curl allows you to specify the referrer to be
used on the command line. It is especially useful to fool or trick stupid
servers or CGI scripts that rely on that information being available or
contain certain data.
curl -e www.coolsite.com http://www.showme.com/
## User Agent
An HTTP request has the option to include information about the browser that
generated the request. Curl allows it to be specified on the command line. It
is especially useful to fool or trick stupid servers or CGI scripts that only
accept certain browsers.
Example:
curl -A 'Mozilla/3.0 (Win95; I)' http://www.nationsbank.com/
Other common strings:
- `Mozilla/3.0 (Win95; I)` - Netscape Version 3 for Windows 95
- `Mozilla/3.04 (Win95; U)` - Netscape Version 3 for Windows 95
- `Mozilla/2.02 (OS/2; U)` - Netscape Version 2 for OS/2
- `Mozilla/4.04 [en] (X11; U; AIX 4.2; Nav)` - Netscape for AIX
- `Mozilla/4.05 [en] (X11; U; Linux 2.0.32 i586)` - Netscape for Linux
Note that Internet Explorer tries hard to be compatible in every way:
- `Mozilla/4.0 (compatible; MSIE 4.01; Windows 95)` - MSIE for W95
Mozilla is not the only possible User-Agent name:
- `Konqueror/1.0` - KDE File Manager desktop client
- `Lynx/2.7.1 libwww-FM/2.14` - Lynx command line browser
## Cookies
Cookies are generally used by web servers to keep state information at the
client's side. The server sets cookies by sending a response line in the
headers that looks like `Set-Cookie: ` where the data part then
typically contains a set of `NAME=VALUE` pairs (separated by semicolons `;`
like `NAME1=VALUE1; NAME2=VALUE2;`). The server can also specify for what path
the cookie should be used for (by specifying `path=value`), when the cookie
should expire (`expire=DATE`), for what domain to use it (`domain=NAME`) and
if it should be used on secure connections only (`secure`).
If you have received a page from a server that contains a header like:
```http
Set-Cookie: sessionid=boo123; path="/foo";
```
it means the server wants that first pair passed on when we get anything in a
path beginning with `/foo`.
Example, get a page that wants my name passed in a cookie:
curl -b "name=Daniel" www.sillypage.com
Curl also has the ability to use previously received cookies in following
sessions. If you get cookies from a server and store them in a file in a
manner similar to:
curl --dump-header headers www.example.com
... you can then in a second connect to that (or another) site, use the
cookies from the `headers.txt` file like:
curl -b headers.txt www.example.com
While saving headers to a file is a working way to store cookies, it is
however error-prone and not the preferred way to do this. Instead, make curl
save the incoming cookies using the well-known Netscape cookie format like
this:
curl -c cookies.txt www.example.com
Note that by specifying `-b` you enable the cookie engine and with `-L` you
can make curl follow a `location:` (which often is used in combination with
cookies). If a site sends cookies and a location field, you can use a
non-existing file to trigger the cookie awareness like:
curl -L -b empty.txt www.example.com
The file to read cookies from must be formatted using plain HTTP headers OR as
Netscape's cookie file. Curl will determine what kind it is based on the file
contents. In the above command, curl will parse the header and store the
cookies received from www.example.com. curl will send to the server the stored
cookies which match the request as it follows the location. The file
`empty.txt` may be a nonexistent file.
To read and write cookies from a Netscape cookie file, you can set both `-b`
and `-c` to use the same file:
curl -b cookies.txt -c cookies.txt www.example.com
## Progress Meter
The progress meter exists to show a user that something actually is
happening. The different fields in the output have the following meaning:
% Total % Received % Xferd Average Speed Time Curr.
Dload Upload Total Current Left Speed
0 151M 0 38608 0 0 9406 0 4:41:43 0:00:04 4:41:39 9287
From left-to-right:
- `%` - percentage completed of the whole transfer
- `Total` - total size of the whole expected transfer
- `%` - percentage completed of the download
- `Received` - currently downloaded amount of bytes
- `%` - percentage completed of the upload
- `Xferd` - currently uploaded amount of bytes
- `Average Speed Dload` - the average transfer speed of the download
- `Average Speed Upload` - the average transfer speed of the upload
- `Time Total` - expected time to complete the operation
- `Time Current` - time passed since the invoke
- `Time Left` - expected time left to completion
- `Curr.Speed` - the average transfer speed the last 5 seconds (the first
5 seconds of a transfer is based on less time of course.)
The `-#` option will display a totally different progress bar that does not
need much explanation!
## Speed Limit
Curl allows the user to set the transfer speed conditions that must be met to
let the transfer keep going. By using the switch `-y` and `-Y` you can make
curl abort transfers if the transfer speed is below the specified lowest limit
for a specified time.
To have curl abort the download if the speed is slower than 3000 bytes per
second for 1 minute, run:
curl -Y 3000 -y 60 www.far-away-site.com
This can be used in combination with the overall time limit, so that the above
operation must be completed in whole within 30 minutes:
curl -m 1800 -Y 3000 -y 60 www.far-away-site.com
Forcing curl not to transfer data faster than a given rate is also possible,
which might be useful if you are using a limited bandwidth connection and you
do not want your transfer to use all of it (sometimes referred to as
*bandwidth throttle*).
Make curl transfer data no faster than 10 kilobytes per second:
curl --limit-rate 10K www.far-away-site.com
or
curl --limit-rate 10240 www.far-away-site.com
Or prevent curl from uploading data faster than 1 megabyte per second:
curl -T upload --limit-rate 1M ftp://uploadshereplease.com
When using the `--limit-rate` option, the transfer rate is regulated on a
per-second basis, which will cause the total transfer speed to become lower
than the given number. Sometimes of course substantially lower, if your
transfer stalls during periods.
## Config File
Curl automatically tries to read the `.curlrc` file (or `_curlrc` file on
Microsoft Windows systems) from the user's home dir on startup.
The config file could be made up with normal command line switches, but you
can also specify the long options without the dashes to make it more
readable. You can separate the options and the parameter with spaces, or with
`=` or `:`. Comments can be used within the file. If the first letter on a
line is a `#`-symbol the rest of the line is treated as a comment.
If you want the parameter to contain spaces, you must enclose the entire
parameter within double quotes (`"`). Within those quotes, you specify a quote
as `\"`.
NOTE: You must specify options and their arguments on the same line.
Example, set default time out and proxy in a config file:
# We want a 30 minute timeout:
-m 1800
# ... and we use a proxy for all accesses:
proxy = proxy.our.domain.com:8080
Whitespaces ARE significant at the end of lines, but all whitespace leading
up to the first characters of each line are ignored.
Prevent curl from reading the default file by using -q as the first command
line parameter, like:
curl -q www.thatsite.com
Force curl to get and display a local help page in case it is invoked without
URL by making a config file similar to:
# default url to get
url = "http://help.with.curl.com/curlhelp.html"
You can specify another config file to be read by using the `-K`/`--config`
flag. If you set config file name to `-` it will read the config from stdin,
which can be handy if you want to hide options from being visible in process
tables etc:
echo "user = user:passwd" | curl -K - http://that.secret.site.com
## Extra Headers
When using curl in your own programs, you may end up needing to pass on your
own custom headers when getting a web page. You can do this by using the `-H`
flag.
Example, send the header `X-you-and-me: yes` to the server when getting a
page:
curl -H "X-you-and-me: yes" www.love.com
This can also be useful in case you want curl to send a different text in a
header than it normally does. The `-H` header you specify then replaces the
header curl would normally send. If you replace an internal header with an
empty one, you prevent that header from being sent. To prevent the `Host:`
header from being used:
curl -H "Host:" www.server.com
## FTP and Path Names
Do note that when getting files with a `ftp://` URL, the given path is
relative to the directory you enter. To get the file `README` from your home
directory at your ftp site, do:
curl ftp://user:passwd@my.site.com/README
If you want the README file from the root directory of that same site, you
need to specify the absolute file name:
curl ftp://user:passwd@my.site.com//README
(I.e with an extra slash in front of the file name.)
## SFTP and SCP and Path Names
With sftp: and scp: URLs, the path name given is the absolute name on the
server. To access a file relative to the remote user's home directory, prefix
the file with `/~/` , such as:
curl -u $USER sftp://home.example.com/~/.bashrc
## FTP and Firewalls
The FTP protocol requires one of the involved parties to open a second
connection as soon as data is about to get transferred. There are two ways to
do this.
The default way for curl is to issue the PASV command which causes the server
to open another port and await another connection performed by the
client. This is good if the client is behind a firewall that does not allow
incoming connections.
curl ftp.download.com
If the server, for example, is behind a firewall that does not allow
connections on ports other than 21 (or if it just does not support the `PASV`
command), the other way to do it is to use the `PORT` command and instruct the
server to connect to the client on the given IP number and port (as parameters
to the PORT command).
The `-P` flag to curl supports a few different options. Your machine may have
several IP-addresses and/or network interfaces and curl allows you to select
which of them to use. Default address can also be used:
curl -P - ftp.download.com
Download with `PORT` but use the IP address of our `le0` interface (this does
not work on Windows):
curl -P le0 ftp.download.com
Download with `PORT` but use 192.168.0.10 as our IP address to use:
curl -P 192.168.0.10 ftp.download.com
## Network Interface
Get a web page from a server using a specified port for the interface:
curl --interface eth0:1 http://www.example.com/
or
curl --interface 192.168.1.10 http://www.example.com/
## HTTPS
Secure HTTP requires a TLS library to be installed and used when curl is
built. If that is done, curl is capable of retrieving and posting documents
using the HTTPS protocol.
Example:
curl https://www.secure-site.com
curl is also capable of using client certificates to get/post files from sites
that require valid certificates. The only drawback is that the certificate
needs to be in PEM-format. PEM is a standard and open format to store
certificates with, but it is not used by the most commonly used browsers. If
you want curl to use the certificates you use with your favorite browser, you
may need to download/compile a converter that can convert your browser's
formatted certificates to PEM formatted ones.
Example on how to automatically retrieve a document using a certificate with a
personal password:
curl -E /path/to/cert.pem:password https://secure.site.com/
If you neglect to specify the password on the command line, you will be
prompted for the correct password before any data can be received.
Many older HTTPS servers have problems with specific SSL or TLS versions,
which newer versions of OpenSSL etc use, therefore it is sometimes useful to
specify what TLS version curl should use.:
curl --tlv1.0 https://secure.site.com/
Otherwise, curl will attempt to use a sensible TLS default version.
## Resuming File Transfers
To continue a file transfer where it was previously aborted, curl supports
resume on HTTP(S) downloads as well as FTP uploads and downloads.
Continue downloading a document:
curl -C - -o file ftp://ftp.server.com/path/file
Continue uploading a document:
curl -C - -T file ftp://ftp.server.com/path/file
Continue downloading a document from a web server
curl -C - -o file http://www.server.com/
## Time Conditions
HTTP allows a client to specify a time condition for the document it requests.
It is `If-Modified-Since` or `If-Unmodified-Since`. curl allows you to specify
them with the `-z`/`--time-cond` flag.
For example, you can easily make a download that only gets performed if the
remote file is newer than a local copy. It would be made like:
curl -z local.html http://remote.server.com/remote.html
Or you can download a file only if the local file is newer than the remote
one. Do this by prepending the date string with a `-`, as in:
curl -z -local.html http://remote.server.com/remote.html
You can specify a plain text date as condition. Tell curl to only download the
file if it was updated since January 12, 2012:
curl -z "Jan 12 2012" http://remote.server.com/remote.html
curl accepts a wide range of date formats. You always make the date check the
other way around by prepending it with a dash (`-`).
## DICT
For fun try
curl dict://dict.org/m:curl
curl dict://dict.org/d:heisenbug:jargon
curl dict://dict.org/d:daniel:gcide
Aliases for `m` are `match` and `find`, and aliases for `d` are `define` and
`lookup`. For example,
curl dict://dict.org/find:curl
Commands that break the URL description of the RFC (but not the DICT
protocol) are
curl dict://dict.org/show:db
curl dict://dict.org/show:strat
Authentication support is still missing
## LDAP
If you have installed the OpenLDAP library, curl can take advantage of it and
offer `ldap://` support. On Windows, curl will use WinLDAP from Platform SDK
by default.
Default protocol version used by curl is LDAP version 3. Version 2 will be
used as a fallback mechanism in case version 3 fails to connect.
LDAP is a complex thing and writing an LDAP query is not an easy
task. Familiarize yourself with the exact syntax description elsewhere. One
such place might be: [RFC 2255, The LDAP URL
Format](https://curl.se/rfc/rfc2255.txt)
To show you an example, this is how to get all people from an LDAP server that
has a certain sub-domain in their email address:
curl -B "ldap://ldap.frontec.se/o=frontec??sub?mail=*sth.frontec.se"
You also can use authentication when accessing LDAP catalog:
curl -u user:passwd "ldap://ldap.frontec.se/o=frontec??sub?mail=*"
curl "ldap://user:passwd@ldap.frontec.se/o=frontec??sub?mail=*"
By default, if user and password are provided, OpenLDAP/WinLDAP will use basic
authentication. On Windows you can control this behavior by providing one of
`--basic`, `--ntlm` or `--digest` option in curl command line
curl --ntlm "ldap://user:passwd@ldap.frontec.se/o=frontec??sub?mail=*"
On Windows, if no user/password specified, auto-negotiation mechanism will be
used with current logon credentials (SSPI/SPNEGO).
## Environment Variables
Curl reads and understands the following environment variables:
http_proxy, HTTPS_PROXY, FTP_PROXY
They should be set for protocol-specific proxies. General proxy should be set
with
ALL_PROXY
A comma-separated list of host names that should not go through any proxy is
set in (only an asterisk, `*` matches all hosts)
NO_PROXY
If the host name matches one of these strings, or the host is within the
domain of one of these strings, transactions with that node will not be done
over proxy. When a domain is used, it needs to start with a period. A user can
specify that both www.example.com and foo.example.com should not use a proxy
by setting `NO_PROXY` to `.example.com`. By including the full name you can
exclude specific host names, so to make `www.example.com` not use a proxy but
still have `foo.example.com` do it, set `NO_PROXY` to `www.example.com`.
The usage of the `-x`/`--proxy` flag overrides the environment variables.
## Netrc
Unix introduced the `.netrc` concept a long time ago. It is a way for a user
to specify name and password for commonly visited FTP sites in a file so that
you do not have to type them in each time you visit those sites. You realize
this is a big security risk if someone else gets hold of your passwords,
therefore most Unix programs will not read this file unless it is only readable
by yourself (curl does not care though).
Curl supports `.netrc` files if told to (using the `-n`/`--netrc` and
`--netrc-optional` options). This is not restricted to just FTP, so curl can
use it for all protocols where authentication is used.
A simple `.netrc` file could look something like:
machine curl.se login iamdaniel password mysecret
## Custom Output
To better allow script programmers to get to know about the progress of curl,
the `-w`/`--write-out` option was introduced. Using this, you can specify what
information from the previous transfer you want to extract.
To display the amount of bytes downloaded together with some text and an
ending newline:
curl -w 'We downloaded %{size_download} bytes\n' www.download.com
## Kerberos FTP Transfer
Curl supports kerberos4 and kerberos5/GSSAPI for FTP transfers. You need the
kerberos package installed and used at curl build time for it to be available.
First, get the krb-ticket the normal way, like with the `kinit`/`kauth` tool.
Then use curl in way similar to:
curl --krb private ftp://krb4site.com -u username:fakepwd
There is no use for a password on the `-u` switch, but a blank one will make
curl ask for one and you already entered the real password to `kinit`/`kauth`.
## TELNET
The curl telnet support is basic and easy to use. Curl passes all data passed
to it on stdin to the remote server. Connect to a remote telnet server using a
command line similar to:
curl telnet://remote.server.com
And enter the data to pass to the server on stdin. The result will be sent to
stdout or to the file you specify with `-o`.
You might want the `-N`/`--no-buffer` option to switch off the buffered output
for slow connections or similar.
Pass options to the telnet protocol negotiation, by using the `-t` option. To
tell the server we use a vt100 terminal, try something like:
curl -tTTYPE=vt100 telnet://remote.server.com
Other interesting options for it `-t` include:
- `XDISPLOC=` Sets the X display location.
- `NEW_ENV=` Sets an environment variable.
NOTE: The telnet protocol does not specify any way to login with a specified
user and password so curl cannot do that automatically. To do that, you need to
track when the login prompt is received and send the username and password
accordingly.
## Persistent Connections
Specifying multiple files on a single command line will make curl transfer all
of them, one after the other in the specified order.
libcurl will attempt to use persistent connections for the transfers so that
the second transfer to the same host can use the same connection that was
already initiated and was left open in the previous transfer. This greatly
decreases connection time for all but the first transfer and it makes a far
better use of the network.
Note that curl cannot use persistent connections for transfers that are used
in subsequent curl invokes. Try to stuff as many URLs as possible on the same
command line if they are using the same host, as that will make the transfers
faster. If you use an HTTP proxy for file transfers, practically all transfers
will be persistent.
## Multiple Transfers With A Single Command Line
As is mentioned above, you can download multiple files with one command line
by simply adding more URLs. If you want those to get saved to a local file
instead of just printed to stdout, you need to add one save option for each
URL you specify. Note that this also goes for the `-O` option (but not
`--remote-name-all`).
For example: get two files and use `-O` for the first and a custom file
name for the second:
curl -O http://url.com/file.txt ftp://ftp.com/moo.exe -o moo.jpg
You can also upload multiple files in a similar fashion:
curl -T local1 ftp://ftp.com/moo.exe -T local2 ftp://ftp.com/moo2.txt
## IPv6
curl will connect to a server with IPv6 when a host lookup returns an IPv6
address and fall back to IPv4 if the connection fails. The `--ipv4` and
`--ipv6` options can specify which address to use when both are
available. IPv6 addresses can also be specified directly in URLs using the
syntax:
http://[2001:1890:1112:1::20]/overview.html
When this style is used, the `-g` option must be given to stop curl from
interpreting the square brackets as special globbing characters. Link local
and site local addresses including a scope identifier, such as `fe80::1234%1`,
may also be used, but the scope portion must be numeric or match an existing
network interface on Linux and the percent character must be URL escaped. The
previous example in an SFTP URL might look like:
sftp://[fe80::1234%251]/
IPv6 addresses provided other than in URLs (e.g. to the `--proxy`,
`--interface` or `--ftp-port` options) should not be URL encoded.
## Mailing Lists
For your convenience, we have several open mailing lists to discuss curl, its
development and things relevant to this. Get all info at
https://curl.se/mail/.
Please direct curl questions, feature requests and trouble reports to one of
these mailing lists instead of mailing any individual.
Available lists include:
### `curl-users`
Users of the command line tool. How to use it, what does not work, new
features, related tools, questions, news, installations, compilations,
running, porting etc.
### `curl-library`
Developers using or developing libcurl. Bugs, extensions, improvements.
### `curl-announce`
Low-traffic. Only receives announcements of new public versions. At worst,
that makes something like one or two mails per month, but usually only one
mail every second month.
### `curl-and-php`
Using the curl functions in PHP. Everything curl with a PHP angle. Or PHP with
a curl angle.
### `curl-and-python`
Python hackers using curl with or without the python binding pycurl.
---
### Third Party Curl Docs MQTT (`third_party/curl/docs/MQTT.md`)
# MQTT in curl
## Usage
A plain "GET" subscribes to the topic and prints all published messages.
Doing a "POST" publishes the post data to the topic and exits.
Example subscribe:
curl mqtt://host/home/bedroom/temp
Example publish:
curl -d 75 mqtt://host/home/bedroom/dimmer
## What does curl deliver as a response to a subscribe
It outputs two bytes topic length (MSB | LSB), the topic followed by the
payload.
## Caveats
Remaining limitations:
- Only QoS level 0 is implemented for publish
- No way to set retain flag for publish
- No TLS (mqtts) support
- Naive EAGAIN handling will not handle split messages
---
### Third Party Curl Docs NEW PROTOCOL (`third_party/curl/docs/NEW-PROTOCOL.md`)
# Adding a new protocol?
Every once in a while, someone comes up with the idea of adding support for yet
another protocol to curl. After all, curl already supports 25 something
protocols and it is the Internet transfer machine for the world.
In the curl project we love protocols and we love supporting many protocols
and doing it well.
So how do you proceed to add a new protocol and what are the requirements?
## No fixed set of requirements
This document is an attempt to describe things to consider. There is no
checklist of the twenty-seven things you need to cross off. We view the entire
effort as a whole and then judge if it seems to be the right thing - for
now. The more things that look right, fit our patterns and are done in ways
that align with our thinking, the better are the chances that we will agree
that supporting this protocol is a grand idea.
## Mutual benefit is preferred
curl is not here for your protocol. Your protocol is not here for curl. The
best cooperation and end result occur when all involved parties mutually see
and agree that supporting this protocol in curl would be good for everyone.
Heck, for the world.
Consider "selling us" the idea that we need an implementation merged in curl,
to be fairly important. *Why* do we want curl to support this new protocol?
## Protocol requirements
### Client-side
The protocol implementation is for a client's side of a "communication
session".
### Transfer oriented
The protocol itself should be focused on *transfers*. Be it uploads or
downloads or both. It should at least be possible to view the transfers as
such, like we can view reading emails over POP3 as a download and sending
emails over SMTP as an upload.
If you cannot even shoehorn the protocol into a transfer focused view, then
you are up for a tough argument.
### URL
There should be a documented URL format. If there is an RFC for it there is no
question about it but the syntax does not have to be a published RFC. It could
be enough if it is already in use by other implementations.
If you make up the syntax just in order to be able to propose it to curl, then
you are in a bad place. URLs are designed and defined for interoperability.
There should at least be a good chance that other clients and servers can be
implemented supporting the same URL syntax and work the same or similar way.
URLs work on registered 'schemes'. There is a register of [all officially
recognized
schemes](https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml). If
your protocol is not in there, is it really a protocol we want?
### Wide and public use
The protocol shall already be used or have an expectation of getting used
widely. Experimental protocols are better off worked on in experiments first,
to prove themselves before they are adopted by curl.
## Code
Of course the code needs to be written, provided, licensed agreeably and it
should follow our code guidelines and review comments have to be dealt with.
If the implementation needs third party code, that third party code should not
have noticeably lesser standards than the curl project itself.
## Tests
As much of the protocol implementation as possible needs to be verified by
curl test cases. We must have the implementation get tested by CI jobs,
torture tests and more.
We have experienced many times in the past how new implementations were brought
to curl and immediately once the code had been merged, the originator vanished
from the face of the earth. That is fine, but we need to take the necessary
precautions so when it happens we are still fine.
Our test infrastructure is powerful enough to test just about every possible
protocol - but it might require a bit of an effort to make it happen.
## Documentation
We cannot assume that users are particularly familiar with details and
peculiarities of the protocol. It needs documentation.
Maybe it even needs some internal documentation so that the developers who
will try to debug something five years from now can figure out functionality a
little easier!
The protocol specification itself should be freely available without requiring
a non-disclosure agreement or similar.
## Do not compare
We are constantly raising the bar and we are constantly improving the
project. A lot of things we did in the past would not be acceptable if done
today. Therefore, you might be tempted to use shortcuts or "hacks" you can
spot other - existing - protocol implementations have used, but there is
nothing to gain from that. The bar has been raised. Former "cheats" will not be
tolerated anymore.
---
### Third Party Curl Docs PARALLEL TRANSFERS (`third_party/curl/docs/PARALLEL-TRANSFERS.md`)
# Parallel transfers
curl 7.66.0 introduced support for doing multiple transfers simultaneously; in
parallel.
## -Z, --parallel
When this command line option is used, curl will perform the transfers given
to it at the same time. It will do up to `--parallel-max` concurrent
transfers, with a default value of 50.
## Progress meter
The progress meter that is displayed when doing parallel transfers is
completely different than the regular one used for each single transfer.
It shows:
o percent download (if known, which means *all* transfers need to have a
known size)
o percent upload (if known, with the same caveat as for download)
o total amount of downloaded data
o total amount of uploaded data
o number of transfers to perform
o number of concurrent transfers being transferred right now
o number of transfers queued up waiting to start
o total time all transfers are expected to take (if sizes are known)
o current time the transfers have spent so far
o estimated time left (if sizes are known)
o current transfer speed (the faster of upload/download speeds measured over
the last few seconds)
Example:
DL% UL% Dled Uled Xfers Live Qd Total Current Left Speed
72 -- 37.9G 0 101 30 23 0:00:55 0:00:34 0:00:22 2752M
## Behavior differences
Connections are shared fine between different easy handles, but the
"authentication contexts" are not. So for example doing HTTP Digest auth with
one handle for a particular transfer and then continue on with another handle
that reuses the same connection, the second handle cannot send the necessary
Authorization header at once since the context is only kept in the original
easy handle.
To fix this, the authorization state could be made possible to share with the
share API as well, as a context per origin + path (realm?) basically.
Visible in test 153, 1412 and more.
---
### Third Party Curl Docs RELEASE PROCEDURE (`third_party/curl/docs/RELEASE-PROCEDURE.md`)
curl release procedure - how to do a release
============================================
in the source code repo
-----------------------
- run `./scripts/copyright.pl` and correct possible omissions
- edit `RELEASE-NOTES` to be accurate
- update `docs/THANKS`
- make sure all relevant changes are committed on the master branch
- tag the git repo in this style: `git tag -a curl-7_34_0`. -a annotates the
tag and we use underscores instead of dots in the version number. Make sure
the tag is GPG signed (using -s).
- run `./maketgz 7.34.0` to build the release tarballs. It is important that
you run this on a machine with the correct set of autotools etc installed
as this is what then will be shipped and used by most users on \*nix like
systems.
- push the git commits and the new tag
- GPG sign the 4 tarballs as `maketgz` suggests
- upload the 8 resulting files to the primary download directory
in the curl-www repo
--------------------
- edit `Makefile` (version number and date),
- edit `_newslog.html` (announce the new release) and
- edit `_changes.html` (insert changes+bugfixes from RELEASE-NOTES)
- commit all local changes
- tag the repo with the same name as used for the source repo.
- make sure all relevant changes are committed and pushed on the master branch
(the website then updates its contents automatically)
on GitHub
---------
- edit the newly made release tag so that it is listed as the latest release
inform
------
- send an email to curl-users, curl-announce and curl-library. Insert the
RELEASE-NOTES into the mail.
celebrate
---------
- suitable beverage intake is encouraged for the festivities
curl release scheduling
=======================
Release Cycle
-------------
We normally do releases every 8 weeks on Wednesdays. If important problems
arise, we can insert releases outside the schedule or we can move the release
date.
Each 8 week (56 days) release cycle is divided into three distinct periods:
- During the first 10 calendar days after a release, we are in "cool down". We
do not merge features but only bug-fixes. If a regression is reported, we
might do a follow-up patch release.
- During the following 3 weeks (21 days) there is a feature window: we allow
new features and changes to curl and libcurl. If we accept any such changes,
we bump the minor number used for the next release.
- During the next 25 days we are in feature freeze. We do not merge any
features or changes, and we only focus on fixing bugs and polishing things
to make the pending release a solid one.
If a future release date happens to end up on a "bad date", like in the middle
of common public holidays or when the lead release manager is unavailable, the
release date can be moved forwards or backwards a full week. This is then
advertised well in advance.
Critical problems
-----------------
We can break the release cycle and do a patch release at any point if a
critical enough problem is reported. There is no exact definition of how to
assess such criticality, but if an issue is highly disturbing or has a
security impact on a large enough share of the user population it might
qualify.
If you think an issue qualifies, bring it to the curl-library mailing list and
push for it.
Coming dates
------------
Based on the description above, here are some planned release dates (at the
time of this writing):
- May 17, 2023
- July 19, 2023
- September 6, 2023
- November 1, 2023
- December 27, 2023
- February 21, 2024
- April 17, 2024
- June 12, 2024
---
### Third Party Curl Docs ROADMAP (`third_party/curl/docs/ROADMAP.md`)
# curl the next few years - perhaps
Roadmap of things Daniel Stenberg wants to work on next. It is intended to
serve as a guideline for others for information, feedback and possible
participation.
## "Complete" the HTTP/3 support
curl has experimental support for HTTP/3 since a good while back. There are
some functionality missing and once the final specs are published we want to
eventually remove the "experimental" label from this functionality.
## HTTPS DNS records
As a DNS version of alt-svc and also a pre-requisite for ECH (see below).
See: https://datatracker.ietf.org/doc/html/draft-ietf-dnsop-svcb-https-02
## ECH (Encrypted Client Hello - formerly known as ESNI)
See Daniel's post on [Support of Encrypted
SNI](https://curl.se/mail/lib-2019-03/0000.html) on the mailing list.
Initial work exists in [PR 4011](https://github.com/curl/curl/pull/4011)
---
### Third Party Curl Docs RUSTLS (`third_party/curl/docs/RUSTLS.md`)
# Rustls
[Rustls is a TLS backend written in Rust](https://docs.rs/rustls/). Curl can
be built to use it as an alternative to OpenSSL or other TLS backends. We use
the [rustls-ffi C bindings](https://github.com/rustls/rustls-ffi/). This
version of curl depends on version v0.9.2 of rustls-ffi.
# Building with rustls
First, [install Rust](https://rustup.rs/).
Next, check out, build, and install the appropriate version of rustls-ffi:
% cargo install cbindgen
% git clone https://github.com/rustls/rustls-ffi -b v0.9.2
% cd rustls-ffi
% make
% make DESTDIR=${HOME}/rustls-ffi-built/ install
Now configure and build curl with rustls:
% git clone https://github.com/curl/curl
% cd curl
% autoreconf -fi
% ./configure --with-rustls=${HOME}/rustls-ffi-built
% make
---
### Third Party Curl Docs SECURITY ADVISORY (`third_party/curl/docs/SECURITY-ADVISORY.md`)
# Anatomy of a curl security advisory
As described in the `SECURITY-PROCESS.md` document, when a security
vulnerability has been reported to the project and confirmed, we author an
advisory document for for the issue. It should ideally be written in
cooperation with the reporter to make sure all the angles and details of the
problem are gathered and described correctly and succinctly.
## New document
A security advisory for curl is created in the `docs/` folder in the
[curl-www](https://github.com/curl/curl-www) repository. It should be named
`$CVEID.md` where `$CVEID` is the full CVE Id that has been registered for the
flaw. Like `CVE-2016-0755`. The `.md` extension of course means that the
document is written using markdown.
The standard way to go about this is to first write the `VULNERABILITY`
section for the document, so that there is description of the flaw available,
then paste this description into the CVE Id request.
### `vuln.pm`
The new issue should be entered at the top of the list in the file `vuln.pm`
in the same directory. It holds a large array with all published curl
vulnerabilities. All fields should be filled in accordingly, separated by a
pipe character (`|`).
The eleven fields for each CVE in `vuln.pm` are, in order:
HTML page name, first vulnerable version, last vulnerable version, name of
the issue, CVE Id, announce date (`YYYYMMDD`), report to the project date
(`YYYYMMDD`), CWE, awarded reward amount (USD), area (single word), C-issue
(`-` if not a C issue at all, `OVERFLOW` , `OVERREAD`, `DOUBLE_FREE`,
`USE_AFTER_FREE`, `NULL_MISTAKE`, `UNINIT`)
### `Makefile`
The new CVE web page file name needs to be added in the `Makefile`'s `CVELIST`
macro.
When the markdown is in place and the `Makefile` and `vuln.pm` are updated,
all other files and metadata for all curl advisories and versions get
generated automatically using those files.
## Document format
The easy way is to start with a recent previously published advisory and just
blank out old texts and save it using a new name. Save the subtitles and
general layout.
Some details and metadata will be extracted from this document so it is
important to stick to the existing format.
The first list must be the title of the issue.
### VULNERABILITY
The first subtitle should be `VULNERABILITY`. That should then include a
through and detailed description of the flaw. Including how it can be
triggered and maybe something about what might happen if triggered or
exploited.
### INFO
The next section is `INFO` which adds meta data information about the flaw. It
specifically mentions the official CVE Id for the issue and it must list the
CWE Id, starting on its own line. We write CWE identifiers in advisories with
the full (official) explanation on the right side of a colon. Like this:
`CWE-305: Authentication Bypass by Primary Weakness`
### AFFECTED VERSIONS
The third section first lists what versions that are affected, then adds
clarity by stressing what versions that are *not* affected. A third line adds
information about which specific git commit that introduced the vulnerability.
The `Introduced-in` commit should be a full URL that displays the commit, but
should work as a stand-alone commit hash if everything up to the last slash is
cut out.
An example using the correct syntax:
~~~
- Affected versions: curl 7.16.1 to and including 7.88.1
- Not affected versions: curl < 7.16.1 and curl >= 8.0.0
- Introduced-in: https://github.com/curl/curl/commit/2147284cad
~~~
### THE SOLUTION
This section describes and discusses the fix. The only mandatory information
here is the link to the git commit that fixes the problem.
The `Fixed-in` value should be a full URL that displays the commit, but should
work as a stand-alone commit hash if everything up to the last slash is cut
out.
Example:
`- Fixed-in: https://github.com/curl/curl/commit/af369db4d3833272b8ed`
### RECOMMENDATIONS
This section lists the recommended actions for the users in a top to bottom
priority order and should ideally contain three items but no less than two.
The top two are almost always `upgrade curl to version XXX` and `apply the
patch to your local version`.
### TIMELINE
Detail when this report was received in the project. When package distributors
were notified (via the distros mailing list or similar)
When the advisory and fixed version are released.
### CREDITS
Mention the reporter and patch author at least, then everyone else involved
you think deserves a mention.
If you want to mention more than one name, separate the names with comma
(`,`).
~~~
- Reported-by: Full Name
- Patched-by: Full Name
~~~
---
### Third Party Curl Docs SECURITY PROCESS (`third_party/curl/docs/SECURITY-PROCESS.md`)
# curl security process
This document describes how security vulnerabilities should be handled in the
curl project.
## Publishing Information
All known and public curl or libcurl related vulnerabilities are listed on
[the curl website security page](https://curl.se/docs/security.html).
Security vulnerabilities **should not** be entered in the project's public bug
tracker.
## Vulnerability Handling
The typical process for handling a new security vulnerability is as follows.
No information should be made public about a vulnerability until it is
formally announced at the end of this process. That means, for example, that a
bug tracker entry must NOT be created to track the issue since that will make
the issue public and it should not be discussed on any of the project's public
mailing lists. Messages associated with any commits should not make any
reference to the security nature of the commit if done prior to the public
announcement.
- The person discovering the issue, the reporter, reports the vulnerability on
[HackerOne](https://hackerone.com/curl). Issues filed there reach a handful
of selected and trusted people.
- Messages that do not relate to the reporting or managing of an undisclosed
security vulnerability in curl or libcurl are ignored and no further action
is required.
- A person in the security team responds to the original report to acknowledge
that a human has seen the report.
- The security team investigates the report and either rejects it or accepts
it. See below for examples of problems that are not considered
vulnerabilities.
- If the report is rejected, the team writes to the reporter to explain why.
- If the report is accepted, the team writes to the reporter to let them
know it is accepted and that they are working on a fix.
- The security team discusses the problem, works out a fix, considers the
impact of the problem and suggests a release schedule. This discussion
should involve the reporter as much as possible.
- The release of the information should be "as soon as possible" and is most
often synchronized with an upcoming release that contains the fix. If the
reporter, or anyone else involved, thinks the next planned release is too
far away, then a separate earlier release should be considered.
- Write a security advisory draft about the problem that explains what the
problem is, its impact, which versions it affects, solutions or workarounds,
when the release is out and make sure to credit all contributors properly.
Figure out the CWE (Common Weakness Enumeration) number for the flaw. See
[SECURITY-ADVISORY](SECURITY-ADVISORY.md) for help on creating the advisory.
- Request a CVE number from
[HackerOne](https://docs.hackerone.com/programs/cve-requests.html)
- Update the "security advisory" with the CVE number.
- The security team commits the fix in a private branch. The commit message
should ideally contain the CVE number. If the severity level of the issue is
set to Low or Medium, the fix is allowed to get merged into the master
repository via a normal PR - but without mentioning it being a security
vulnerability.
- The monetary reward part of the bug-bounty is managed by the Internet Bug
Bounty team and the reporter is asked to request the reward from them after
the issue has been completely handled and published by curl.
- No more than 10 days before release, inform
[distros@openwall](https://oss-security.openwall.org/wiki/mailing-lists/distros)
to prepare them about the upcoming public security vulnerability
announcement - attach the advisory draft for information with CVE and
current patch. 'distros' does not accept an embargo longer than 14 days and
they do not care for Windows-specific flaws.
- No more than 48 hours before the release, the private branch is merged into
the master branch and pushed. Once pushed, the information is accessible to
the public and the actual release should follow suit immediately afterwards.
The time between the push and the release is used for final tests and
reviews.
- The project team creates a release that includes the fix.
- The project team announces the release and the vulnerability to the world in
the same manner we always announce releases. It gets sent to the
curl-announce, curl-library and curl-users mailing lists.
- The security web page on the website should get the new vulnerability
mentioned.
## security (at curl dot se)
This is a private mailing list for discussions on and about curl security
issues.
Who is on this list? There are a couple of criteria you must meet, and then we
might ask you to join the list or you can ask to join it. It really is not a
formal process. We basically only require that you have a long-term presence
in the curl project and you have shown an understanding for the project and
its way of working. You must have been around for a good while and you should
have no plans of vanishing in the near future.
We do not make the list of participants public mostly because it tends to vary
somewhat over time and a list somewhere will only risk getting outdated.
## Publishing Security Advisories
1. Write up the security advisory, using markdown syntax. Use the same
subtitles as last time to maintain consistency.
2. Name the advisory file after the allocated CVE id.
3. Add a line on the top of the array in `curl-www/docs/vuln.pm`.
4. Put the new advisory markdown file in the `curl-www/docs/` directory. Add it
to the git repository.
5. Run `make` in your local web checkout and verify that things look fine.
6. On security advisory release day, push the changes on the curl-www
repository's remote master branch.
## HackerOne
Request the issue to be disclosed. If there are sensitive details present in
the report and discussion, those should be redacted from the disclosure. The
default policy is to disclose as much as possible as soon as the vulnerability
has been published.
## Bug Bounty
See [BUG-BOUNTY](https://curl.se/docs/bugbounty.html) for details on the
bug bounty program.
# Severity levels
The curl project's security team rates security problems using four severity
levels depending how serious we consider the problem to be. We use **Low**,
**Medium**, **High** and **Critical**. We refrain from using numerical scoring
of vulnerabilities.
When deciding severity level on a particular issue, we take all the factors
into account: attack vector, attack complexity, required privileges, necessary
build configuration, protocols involved, platform specifics and also what
effects a possible exploit or trigger of the issue can lead do, including
confidentiality, integrity or availability problems.
## Low
This is a security problem that is truly hard or unlikely to exploit or
trigger. Due to timing, platform requirements or the fact that options or
protocols involved are rare etc. [Past
example](https://curl.se/docs/CVE-2022-43552.html)
## Medium
This is a security problem that is less hard than **Low** to exploit or
trigger. Less strict timing, wider platforms availability or involving more
widely used options or protocols. A problem that usually needs something else
to also happen to become serious. [Past
example](https://curl.se/docs/CVE-2022-32206.html)
## High
This issue in itself a serious problem with real world impact. Flaws that can
easily compromise the confidentiality, integrity or availability of resources.
Exploiting or triggering this problem is not hard. [Past
example](https://curl.se/docs/CVE-2019-3822.html)
## Critical
Easily exploitable by a remote unauthenticated attacker and lead to system
compromise (arbitrary code execution) without requiring user interaction, with
a common configuration on a popular platform. This issue has few restrictions
and requirements and can be exploited easily using most curl configurations.
No past curl vulnerability has had this severity level.
# Not security issues
This is an incomplete list of issues that are not considered vulnerabilities.
## Small memory leaks
We do not consider a small memory leak a security problem; even if the amount
of allocated memory grows by a small amount every now and then. Long-living
applications and services already need to have counter-measures and deal with
growing memory usage, be it leaks or just increased use. A small memory or
resource leak is then expected to *not* cause a security problem.
Of course there can be a discussion if a leak is small or not. A large leak
can be considered a security problem due to the DOS risk. If leaked memory
contains sensitive data it might also qualify as a security problem.
## Never-ending transfers
We do not consider flaws that cause a transfer to never end to be a security
problem. There are already several benign and likely reasons for transfers to
stall and never end, so applications that cannot deal with never-ending
transfers already need to have counter-measures established.
If the problem avoids the regular counter-measures when it causes a never-
ending transfer, it might be a security problem.
## Not practically possible
If the flaw or vulnerability cannot practically get executed on existing
hardware it is not a security problem.
## API misuse
If a reported issue only triggers by an application using the API in a way
that is not documented to work or even documented to not work, it is probably
not going to be considered a security problem. We only guarantee secure and
proper functionality when the APIs are used as expected and documented.
There can be a discussion about what the documentation actually means and how
to interpret the text, which might end up with us still agreeing that it is a
security problem.
## Local attackers already present
When an issue can only be attacked or misused by an attacker present on the
local system or network, the bar is raised. If a local user wrongfully has
elevated rights on your system enough to attack curl, they can probably
already do much worse harm and the problem is not really in curl.
## Experiments
Vulnerabilities in features which are off by default (in the build) and
documented as experimental, are not eligible for a reward and we do not
consider them security problems.
## URL inconsistencies
URL parser inconsistencies between browsers and curl are expected and are not
considered security vulnerabilities. The WHATWG URL Specification and RFC
3986+ (the plus meaning that it is an extended version) [are not completely
interoperable](https://github.com/bagder/docs/blob/master/URL-interop.md).
Obvious parser bugs can still be vulnerabilities of course.
## Visible command line arguments
The curl command blanks the contents of a number of command line arguments to
prevent them from appearing in process listings. It does not blank all
arguments even if some of them that are not blanked might contain sensitive
data. We consider this functionality a best-effort and omissions are not
security vulnerabilities.
- not all systems allow the arguments to be blanked in the first place
- since curl blanks the argument itself they will be readable for a short
moment no matter what
- virtually every argument can contain sensitive data, depending on use
- blanking all arguments would make it impractical for users to differentiate
curl command lines in process listings
## Busy-loops
Busy-loops that consume 100% CPU time but eventually end (perhaps due to a set
timeout value or otherwise) are not considered security problems. Applications
are supposed to already handle situations when the transfer loop legitimately
consumes 100% CPU time, so while a prolonged such busy-loop is a nasty bug, we
do not consider it a security problem.
---
### Third Party Curl Docs SSL PROBLEMS (`third_party/curl/docs/SSL-PROBLEMS.md`)
_ _ ____ _
___| | | | _ \| |
/ __| | | | |_) | |
| (__| |_| | _ <| |___
\___|\___/|_| \_\_____|
# SSL problems
First, let's establish that we often refer to TLS and SSL interchangeably as
SSL here. The current protocol is called TLS, it was called SSL a long time
ago.
There are several known reasons why a connection that involves SSL might
fail. This is a document that attempts to detail the most common ones and
how to mitigate them.
## CA certs
CA certs are used to digitally verify the server's certificate. You need a
"ca bundle" for this. See lots of more details on this in the `SSLCERTS`
document.
## CA bundle missing intermediate certificates
When using said CA bundle to verify a server cert, you will experience
problems if your CA store does not contain the certificates for the
intermediates if the server does not provide them.
The TLS protocol mandates that the intermediate certificates are sent in the
handshake, but as browsers have ways to survive or work around such
omissions, missing intermediates in TLS handshakes still happen that
browser-users will not notice.
Browsers work around this problem in two ways: they cache intermediate
certificates from previous transfers and some implement the TLS "AIA"
extension that lets the client explicitly download such certificates on
demand.
## Protocol version
Some broken servers fail to support the protocol negotiation properly that
SSL servers are supposed to handle. This may cause the connection to fail
completely. Sometimes you may need to explicitly select a SSL version to use
when connecting to make the connection succeed.
An additional complication can be that modern SSL libraries sometimes are
built with support for older SSL and TLS versions disabled!
All versions of SSL and the TLS versions before 1.2 are considered insecure
and should be avoided. Use TLS 1.2 or later.
## Ciphers
Clients give servers a list of ciphers to select from. If the list does not
include any ciphers the server wants/can use, the connection handshake
fails.
curl has recently disabled the user of a whole bunch of seriously insecure
ciphers from its default set (slightly depending on SSL backend in use).
You may have to explicitly provide an alternative list of ciphers for curl
to use to allow the server to use a weak cipher for you.
Note that these weak ciphers are identified as flawed. For example, this
includes symmetric ciphers with less than 128 bit keys and RC4.
Schannel in Windows XP is not able to connect to servers that no longer
support the legacy handshakes and algorithms used by those versions, so we
advise against building curl to use Schannel on really old Windows versions.
Reference: [Prohibiting RC4 Cipher
Suites](https://datatracker.ietf.org/doc/html/draft-popov-tls-prohibiting-rc4-01)
## Allow BEAST
BEAST is the name of a TLS 1.0 attack that surfaced 2011. When adding means
to mitigate this attack, it turned out that some broken servers out there in
the wild did not work properly with the BEAST mitigation in place.
To make such broken servers work, the --ssl-allow-beast option was
introduced. Exactly as it sounds, it re-introduces the BEAST vulnerability
but on the other hand it allows curl to connect to that kind of strange
servers.
## Disabling certificate revocation checks
Some SSL backends may do certificate revocation checks (CRL, OCSP, etc)
depending on the OS or build configuration. The --ssl-no-revoke option was
introduced in 7.44.0 to disable revocation checking but currently is only
supported for Schannel (the native Windows SSL library), with an exception
in the case of Windows' Untrusted Publishers block list which it seems cannot
be bypassed. This option may have broader support to accommodate other SSL
backends in the future.
References:
https://curl.se/docs/ssl-compared.html
---
### Third Party Curl Docs SSLCERTS (`third_party/curl/docs/SSLCERTS.md`)
SSL Certificate Verification
============================
SSL is TLS
----------
SSL is the old name. It is called TLS these days.
Native SSL
----------
If libcurl was built with Schannel or Secure Transport support (the native SSL
libraries included in Windows and Mac OS X), then this does not apply to
you. Scroll down for details on how the OS-native engines handle SSL
certificates. If you are not sure, then run "curl -V" and read the results. If
the version string says `Schannel` in it, then it was built with Schannel
support.
It is about trust
-----------------
This system is about trust. In your local CA certificate store you have certs
from *trusted* Certificate Authorities that you then can use to verify that
the server certificates you see are valid. They are signed by one of the
certificate authorities you trust.
Which certificate authorities do you trust? You can decide to trust the same
set of companies your operating system trusts, or the set one of the known
browsers trust. That is basically trust via someone else you trust. You should
just be aware that modern operating systems and browsers are setup to trust
*hundreds* of companies and in recent years several certificate authorities
have been found untrustworthy.
Certificate Verification
------------------------
libcurl performs peer SSL certificate verification by default. This is done
by using a CA certificate store that the SSL library can use to make sure the
peer's server certificate is valid.
If you communicate with HTTPS, FTPS or other TLS-using servers using
certificates in the CA store, you can be sure that the remote server really is
the one it claims to be.
If the remote server uses a self-signed certificate, if you do not install a CA
cert store, if the server uses a certificate signed by a CA that is not
included in the store you use or if the remote host is an impostor
impersonating your favorite site, and you want to transfer files from this
server, do one of the following:
1. Tell libcurl to *not* verify the peer. With libcurl you disable this with
`curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, FALSE);`
With the curl command line tool, you disable this with `-k`/`--insecure`.
2. Get a CA certificate that can verify the remote server and use the proper
option to point out this CA cert for verification when connecting. For
libcurl hackers: `curl_easy_setopt(curl, CURLOPT_CAINFO, cacert);`
With the curl command line tool: `--cacert [file]`
3. Add the CA cert for your server to the existing default CA certificate
store. The default CA certificate store can be changed at compile time with
the following configure options:
`--with-ca-bundle=FILE`: use the specified file as the CA certificate
store. CA certificates need to be concatenated in PEM format into this
file.
`--with-ca-path=PATH`: use the specified path as CA certificate store. CA
certificates need to be stored as individual PEM files in this directory.
You may need to run c_rehash after adding files there.
If neither of the two options is specified, configure will try to
auto-detect a setting. It's also possible to explicitly not set any
default store but rely on the built in default the crypto library may
provide instead. You can achieve that by passing both
`--without-ca-bundle` and `--without-ca-path` to the configure script.
If you use Internet Explorer, this is one way to get extract the CA cert
for a particular server:
- View the certificate by double-clicking the padlock
- Find out where the CA certificate is kept (Certificate>
Authority Information Access>URL)
- Get a copy of the crt file using curl
- Convert it from crt to PEM using the OpenSSL tool:
`openssl x509 -inform DES -in yourdownloaded.crt -out outcert.pem -text`
- Add the `outcert.pem` to the CA certificate store or use it stand-alone
as described below.
If you use the `openssl` tool, this is one way to get extract the CA cert
for a particular server:
- `openssl s_client -showcerts -servername server -connect server:443 > cacert.pem`
- type "quit", followed by the "ENTER" key
- The certificate will have "BEGIN CERTIFICATE" and "END CERTIFICATE"
markers.
- If you want to see the data in the certificate, you can do: `openssl
x509 -inform PEM -in certfile -text -out certdata` where `certfile` is
the cert you extracted from logfile. Look in `certdata`.
- If you want to trust the certificate, you can add it to your CA
certificate store or use it stand-alone as described. Just remember that
the security is no better than the way you obtained the certificate.
4. If you are using the curl command line tool, you can specify your own CA
cert file by setting the environment variable `CURL_CA_BUNDLE` to the path
of your choice.
If you are using the curl command line tool on Windows, curl will search
for a CA cert file named "curl-ca-bundle.crt" in these directories and in
this order:
1. application's directory
2. current working directory
3. Windows System directory (e.g. C:\windows\system32)
4. Windows Directory (e.g. C:\windows)
5. all directories along %PATH%
5. Get a better/different/newer CA cert bundle! One option is to extract the
one a recent Firefox browser uses by running 'make ca-bundle' in the curl
build tree root, or possibly download a version that was generated this
way for you: [CA Extract](https://curl.se/docs/caextract.html)
Neglecting to use one of the above methods when dealing with a server using a
certificate that is not signed by one of the certificates in the installed CA
certificate store, will cause SSL to report an error ("certificate verify
failed") during the handshake and SSL will then refuse further communication
with that server.
Certificate Verification with NSS
---------------------------------
If libcurl was built with NSS support, then depending on the OS distribution,
it is probably required to take some additional steps to use the system-wide
CA cert db. Red Hat ships with an additional module, libnsspem.so, which
enables NSS to read the OpenSSL PEM CA bundle. On openSUSE you can install
p11-kit-nss-trust which makes NSS use the system wide CA certificate
store. NSS also has a new [database
format](https://wiki.mozilla.org/NSS_Shared_DB).
Starting with version 7.19.7, libcurl automatically adds the `sql:` prefix to
the certificate database directory (either the set default `/etc/pki/nssdb` or
the directory configured with the `SSL_DIR` environment variable). To check
which certificate database format your distribution provides, examine the
default certificate database location: `/etc/pki/nssdb`; the new certificate
database format can be identified by the filenames `cert9.db`, `key4.db`,
`pkcs11.txt`; filenames of older versions are `cert8.db`, `key3.db`,
`secmod.db`.
Certificate Verification with Schannel and Secure Transport
-----------------------------------------------------------
If libcurl was built with Schannel (Microsoft's native TLS engine) or Secure
Transport (Apple's native TLS engine) support, then libcurl will still perform
peer certificate verification, but instead of using a CA cert bundle, it will
use the certificates that are built into the OS. These are the same
certificates that appear in the Internet Options control panel (under Windows)
or Keychain Access application (under OS X). Any custom security rules for
certificates will be honored.
Schannel will run CRL checks on certificates unless peer verification is
disabled. Secure Transport on iOS will run OCSP checks on certificates unless
peer verification is disabled. Secure Transport on OS X will run either OCSP
or CRL checks on certificates if those features are enabled, and this behavior
can be adjusted in the preferences of Keychain Access.
HTTPS proxy
-----------
Since version 7.52.0, curl can do HTTPS to the proxy separately from the
connection to the server. This TLS connection is handled separately from the
server connection so instead of `--insecure` and `--cacert` to control the
certificate verification, you use `--proxy-insecure` and `--proxy-cacert`.
With these options, you make sure that the TLS connection and the trust of the
proxy can be kept totally separate from the TLS connection to the server.
---
### Third Party Curl Docs TheArtOfHttpScripting (`third_party/curl/docs/TheArtOfHttpScripting.md`)
# The Art Of Scripting HTTP Requests Using Curl
## Background
This document assumes that you are familiar with HTML and general networking.
The increasing amount of applications moving to the web has made "HTTP
Scripting" more frequently requested and wanted. To be able to automatically
extract information from the web, to fake users, to post or upload data to
web servers are all important tasks today.
Curl is a command line tool for doing all sorts of URL manipulations and
transfers, but this particular document will focus on how to use it when
doing HTTP requests for fun and profit. This documents assumes that you know
how to invoke `curl --help` or `curl --manual` to get basic information about
it.
Curl is not written to do everything for you. It makes the requests, it gets
the data, it sends data and it retrieves the information. You probably need
to glue everything together using some kind of script language or repeated
manual invokes.
## The HTTP Protocol
HTTP is the protocol used to fetch data from web servers. It is a simple
protocol that is built upon TCP/IP. The protocol also allows information to
get sent to the server from the client using a few different methods, as will
be shown here.
HTTP is plain ASCII text lines being sent by the client to a server to
request a particular action, and then the server replies a few text lines
before the actual requested content is sent to the client.
The client, curl, sends an HTTP request. The request contains a method (like
GET, POST, HEAD etc), a number of request headers and sometimes a request
body. The HTTP server responds with a status line (indicating if things went
well), response headers and most often also a response body. The "body" part
is the plain data you requested, like the actual HTML or the image etc.
## See the Protocol
Using curl's option [`--verbose`](https://curl.se/docs/manpage.html#-v)
(`-v` as a short option) will display what kind of commands curl sends to the
server, as well as a few other informational texts.
`--verbose` is the single most useful option when it comes to debug or even
understand the curl<->server interaction.
Sometimes even `--verbose` is not enough. Then
[`--trace`](https://curl.se/docs/manpage.html#-trace) and
[`--trace-ascii`](https://curl.se/docs/manpage.html#--trace-ascii)
offer even more details as they show **everything** curl sends and
receives. Use it like this:
curl --trace-ascii debugdump.txt http://www.example.com/
## See the Timing
Many times you may wonder what exactly is taking all the time, or you just
want to know the amount of milliseconds between two points in a transfer. For
those, and other similar situations, the
[`--trace-time`](https://curl.se/docs/manpage.html#--trace-time) option
is what you need. It will prepend the time to each trace output line:
curl --trace-ascii d.txt --trace-time http://example.com/
## See the Response
By default curl sends the response to stdout. You need to redirect it
somewhere to avoid that, most often that is done with `-o` or `-O`.
# URL
## Spec
The Uniform Resource Locator format is how you specify the address of a
particular resource on the Internet. You know these, you have seen URLs like
https://curl.se or https://example.com a million times. RFC 3986 is the
canonical spec. And yeah, the formal name is not URL, it is URI.
## Host
The host name is usually resolved using DNS or your /etc/hosts file to an IP
address and that is what curl will communicate with. Alternatively you specify
the IP address directly in the URL instead of a name.
For development and other trying out situations, you can point to a different
IP address for a host name than what would otherwise be used, by using curl's
[`--resolve`](https://curl.se/docs/manpage.html#--resolve) option:
curl --resolve www.example.org:80:127.0.0.1 http://www.example.org/
## Port number
Each protocol curl supports operates on a default port number, be it over TCP
or in some cases UDP. Normally you do not have to take that into
consideration, but at times you run test servers on other ports or
similar. Then you can specify the port number in the URL with a colon and a
number immediately following the host name. Like when doing HTTP to port
1234:
curl http://www.example.org:1234/
The port number you specify in the URL is the number that the server uses to
offer its services. Sometimes you may use a proxy, and then you may
need to specify that proxy's port number separately from what curl needs to
connect to the server. Like when using an HTTP proxy on port 4321:
curl --proxy http://proxy.example.org:4321 http://remote.example.org/
## User name and password
Some services are setup to require HTTP authentication and then you need to
provide name and password which is then transferred to the remote site in
various ways depending on the exact authentication protocol used.
You can opt to either insert the user and password in the URL or you can
provide them separately:
curl http://user:password@example.org/
or
curl -u user:password http://example.org/
You need to pay attention that this kind of HTTP authentication is not what
is usually done and requested by user-oriented websites these days. They tend
to use forms and cookies instead.
## Path part
The path part is just sent off to the server to request that it sends back
the associated response. The path is what is to the right side of the slash
that follows the host name and possibly port number.
# Fetch a page
## GET
The simplest and most common request/operation made using HTTP is to GET a
URL. The URL could itself refer to a web page, an image or a file. The client
issues a GET request to the server and receives the document it asked for.
If you issue the command line
curl https://curl.se
you get a web page returned in your terminal window. The entire HTML document
that that URL holds.
All HTTP replies contain a set of response headers that are normally hidden,
use curl's [`--include`](https://curl.se/docs/manpage.html#-i) (`-i`)
option to display them as well as the rest of the document.
## HEAD
You can ask the remote server for ONLY the headers by using the
[`--head`](https://curl.se/docs/manpage.html#-I) (`-I`) option which
will make curl issue a HEAD request. In some special cases servers deny the
HEAD method while others still work, which is a particular kind of annoyance.
The HEAD method is defined and made so that the server returns the headers
exactly the way it would do for a GET, but without a body. It means that you
may see a `Content-Length:` in the response headers, but there must not be an
actual body in the HEAD response.
## Multiple URLs in a single command line
A single curl command line may involve one or many URLs. The most common case
is probably to just use one, but you can specify any amount of URLs. Yes
any. No limits. You will then get requests repeated over and over for all the
given URLs.
Example, send two GET requests:
curl http://url1.example.com http://url2.example.com
If you use [`--data`](https://curl.se/docs/manpage.html#-d) to POST to
the URL, using multiple URLs means that you send that same POST to all the
given URLs.
Example, send two POSTs:
curl --data name=curl http://url1.example.com http://url2.example.com
## Multiple HTTP methods in a single command line
Sometimes you need to operate on several URLs in a single command line and do
different HTTP methods on each. For this, you will enjoy the
[`--next`](https://curl.se/docs/manpage.html#-:) option. It is basically
a separator that separates a bunch of options from the next. All the URLs
before `--next` will get the same method and will get all the POST data
merged into one.
When curl reaches the `--next` on the command line, it will sort of reset the
method and the POST data and allow a new set.
Perhaps this is best shown with a few examples. To send first a HEAD and then
a GET:
curl -I http://example.com --next http://example.com
To first send a POST and then a GET:
curl -d score=10 http://example.com/post.cgi --next http://example.com/results.html
# HTML forms
## Forms explained
Forms are the general way a website can present an HTML page with fields for
the user to enter data in, and then press some kind of 'OK' or 'Submit'
button to get that data sent to the server. The server then typically uses
the posted data to decide how to act. Like using the entered words to search
in a database, or to add the info in a bug tracking system, display the
entered address on a map or using the info as a login-prompt verifying that
the user is allowed to see what it is about to see.
Of course there has to be some kind of program on the server end to receive
the data you send. You cannot just invent something out of the air.
## GET
A GET-form uses the method GET, as specified in HTML like:
```html
```
In your favorite browser, this form will appear with a text box to fill in
and a press-button labeled "OK". If you fill in '1905' and press the OK
button, your browser will then create a new URL to get for you. The URL will
get `junk.cgi?birthyear=1905&press=OK` appended to the path part of the
previous URL.
If the original form was seen on the page `www.example.com/when/birth.html`,
the second page you will get will become
`www.example.com/when/junk.cgi?birthyear=1905&press=OK`.
Most search engines work this way.
To make curl do the GET form post for you, just enter the expected created
URL:
curl "http://www.example.com/when/junk.cgi?birthyear=1905&press=OK"
## POST
The GET method makes all input field names get displayed in the URL field of
your browser. That is generally a good thing when you want to be able to
bookmark that page with your given data, but it is an obvious disadvantage if
you entered secret information in one of the fields or if there are a large
amount of fields creating a long and unreadable URL.
The HTTP protocol then offers the POST method. This way the client sends the
data separated from the URL and thus you will not see any of it in the URL
address field.
The form would look similar to the previous one:
```html
```
And to use curl to post this form with the same data filled in as before, we
could do it like:
curl --data "birthyear=1905&press=%20OK%20" http://www.example.com/when/junk.cgi
This kind of POST will use the Content-Type
`application/x-www-form-urlencoded` and is the most widely used POST kind.
The data you send to the server MUST already be properly encoded, curl will
not do that for you. For example, if you want the data to contain a space,
you need to replace that space with `%20`, etc. Failing to comply with this will
most likely cause your data to be received wrongly and messed up.
Recent curl versions can in fact url-encode POST data for you, like this:
curl --data-urlencode "name=I am Daniel" http://www.example.com
If you repeat `--data` several times on the command line, curl will
concatenate all the given data pieces - and put a `&` symbol between each
data segment.
## File Upload POST
Back in late 1995 they defined an additional way to post data over HTTP. It
is documented in the RFC 1867, why this method sometimes is referred to as
RFC1867-posting.
This method is mainly designed to better support file uploads. A form that
allows a user to upload a file could be written like this in HTML:
```html
```
This clearly shows that the Content-Type about to be sent is
`multipart/form-data`.
To post to a form like this with curl, you enter a command line like:
curl --form upload=@localfilename --form press=OK [URL]
## Hidden Fields
A common way for HTML based applications to pass state information between
pages is to add hidden fields to the forms. Hidden fields are already filled
in, they are not displayed to the user and they get passed along just as all
the other fields.
A similar example form with one visible field, one hidden field and one
submit button could look like:
```html
```
To POST this with curl, you will not have to think about if the fields are
hidden or not. To curl they are all the same:
curl --data "birthyear=1905&press=OK&person=daniel" [URL]
## Figure Out What A POST Looks Like
When you are about to fill in a form and send it to a server by using curl
instead of a browser, you are of course interested in sending a POST exactly
the way your browser does.
An easy way to get to see this, is to save the HTML page with the form on
your local disk, modify the 'method' to a GET, and press the submit button
(you could also change the action URL if you want to).
You will then clearly see the data get appended to the URL, separated with a
`?`-letter as GET forms are supposed to.
# HTTP upload
## PUT
Perhaps the best way to upload data to an HTTP server is to use PUT. Then
again, this of course requires that someone put a program or script on the
server end that knows how to receive an HTTP PUT stream.
Put a file to an HTTP server with curl:
curl --upload-file uploadfile http://www.example.com/receive.cgi
# HTTP Authentication
## Basic Authentication
HTTP Authentication is the ability to tell the server your username and
password so that it can verify that you are allowed to do the request you are
doing. The Basic authentication used in HTTP (which is the type curl uses by
default) is **plain text** based, which means it sends username and password
only slightly obfuscated, but still fully readable by anyone that sniffs on
the network between you and the remote server.
To tell curl to use a user and password for authentication:
curl --user name:password http://www.example.com
## Other Authentication
The site might require a different authentication method (check the headers
returned by the server), and then
[`--ntlm`](https://curl.se/docs/manpage.html#--ntlm),
[`--digest`](https://curl.se/docs/manpage.html#--digest),
[`--negotiate`](https://curl.se/docs/manpage.html#--negotiate) or even
[`--anyauth`](https://curl.se/docs/manpage.html#--anyauth) might be
options that suit you.
## Proxy Authentication
Sometimes your HTTP access is only available through the use of an HTTP
proxy. This seems to be especially common at various companies. An HTTP proxy
may require its own user and password to allow the client to get through to
the Internet. To specify those with curl, run something like:
curl --proxy-user proxyuser:proxypassword curl.se
If your proxy requires the authentication to be done using the NTLM method,
use [`--proxy-ntlm`](https://curl.se/docs/manpage.html#--proxy-ntlm), if
it requires Digest use
[`--proxy-digest`](https://curl.se/docs/manpage.html#--proxy-digest).
If you use any one of these user+password options but leave out the password
part, curl will prompt for the password interactively.
## Hiding credentials
Do note that when a program is run, its parameters might be possible to see
when listing the running processes of the system. Thus, other users may be
able to watch your passwords if you pass them as plain command line
options. There are ways to circumvent this.
It is worth noting that while this is how HTTP Authentication works, many
websites will not use this concept when they provide logins etc. See the Web
Login chapter further below for more details on that.
# More HTTP Headers
## Referer
An HTTP request may include a 'referer' field (yes it is misspelled), which
can be used to tell from which URL the client got to this particular
resource. Some programs/scripts check the referer field of requests to verify
that this was not arriving from an external site or an unknown page. While
this is a stupid way to check something so easily forged, many scripts still
do it. Using curl, you can put anything you want in the referer-field and
thus more easily be able to fool the server into serving your request.
Use curl to set the referer field with:
curl --referer http://www.example.come http://www.example.com
## User Agent
Similar to the referer field, all HTTP requests may set the User-Agent
field. It names what user agent (client) that is being used. Many
applications use this information to decide how to display pages. Silly web
programmers try to make different pages for users of different browsers to
make them look the best possible for their particular browsers. They usually
also do different kinds of JavaScript etc.
At times, you will see that getting a page with curl will not return the same
page that you see when getting the page with your browser. Then you know it
is time to set the User Agent field to fool the server into thinking you are
one of those browsers.
To make curl look like Internet Explorer 5 on a Windows 2000 box:
curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]
Or why not look like you are using Netscape 4.73 on an old Linux box:
curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]
## Redirects
## Location header
When a resource is requested from a server, the reply from the server may
include a hint about where the browser should go next to find this page, or a
new page keeping newly generated output. The header that tells the browser to
redirect is `Location:`.
Curl does not follow `Location:` headers by default, but will simply display
such pages in the same manner it displays all HTTP replies. It does however
feature an option that will make it attempt to follow the `Location:`
pointers.
To tell curl to follow a Location:
curl --location http://www.example.com
If you use curl to POST to a site that immediately redirects you to another
page, you can safely use
[`--location`](https://curl.se/docs/manpage.html#-L) (`-L`) and
`--data`/`--form` together. Curl will only use POST in the first request, and
then revert to GET in the following operations.
## Other redirects
Browsers typically support at least two other ways of redirects that curl
does not: first the html may contain a meta refresh tag that asks the browser
to load a specific URL after a set number of seconds, or it may use
JavaScript to do it.
# Cookies
## Cookie Basics
The way the web browsers do "client side state control" is by using
cookies. Cookies are just names with associated contents. The cookies are
sent to the client by the server. The server tells the client for what path
and host name it wants the cookie sent back, and it also sends an expiration
date and a few more properties.
When a client communicates with a server with a name and path as previously
specified in a received cookie, the client sends back the cookies and their
contents to the server, unless of course they are expired.
Many applications and servers use this method to connect a series of requests
into a single logical session. To be able to use curl in such occasions, we
must be able to record and send back cookies the way the web application
expects them. The same way browsers deal with them.
## Cookie options
The simplest way to send a few cookies to the server when getting a page with
curl is to add them on the command line like:
curl --cookie "name=Daniel" http://www.example.com
Cookies are sent as common HTTP headers. This is practical as it allows curl
to record cookies simply by recording headers. Record cookies with curl by
using the [`--dump-header`](https://curl.se/docs/manpage.html#-D) (`-D`)
option like:
curl --dump-header headers_and_cookies http://www.example.com
(Take note that the
[`--cookie-jar`](https://curl.se/docs/manpage.html#-c) option described
below is a better way to store cookies.)
Curl has a full blown cookie parsing engine built-in that comes in use if you
want to reconnect to a server and use cookies that were stored from a
previous connection (or hand-crafted manually to fool the server into
believing you had a previous connection). To use previously stored cookies,
you run curl like:
curl --cookie stored_cookies_in_file http://www.example.com
Curl's "cookie engine" gets enabled when you use the
[`--cookie`](https://curl.se/docs/manpage.html#-b) option. If you only
want curl to understand received cookies, use `--cookie` with a file that
does not exist. Example, if you want to let curl understand cookies from a
page and follow a location (and thus possibly send back cookies it received),
you can invoke it like:
curl --cookie nada --location http://www.example.com
Curl has the ability to read and write cookie files that use the same file
format that Netscape and Mozilla once used. It is a convenient way to share
cookies between scripts or invokes. The `--cookie` (`-b`) switch
automatically detects if a given file is such a cookie file and parses it,
and by using the `--cookie-jar` (`-c`) option you will make curl write a new
cookie file at the end of an operation:
curl --cookie cookies.txt --cookie-jar newcookies.txt \
http://www.example.com
# HTTPS
## HTTPS is HTTP secure
There are a few ways to do secure HTTP transfers. By far the most common
protocol for doing this is what is generally known as HTTPS, HTTP over
SSL. SSL encrypts all the data that is sent and received over the network and
thus makes it harder for attackers to spy on sensitive information.
SSL (or TLS as the current version of the standard is called) offers a set of
advanced features to do secure transfers over HTTP.
Curl supports encrypted fetches when built to use a TLS library and it can be
built to use one out of a fairly large set of libraries - `curl -V` will show
which one your curl was built to use (if any!). To get a page from an HTTPS
server, simply run curl like:
curl https://secure.example.com
## Certificates
In the HTTPS world, you use certificates to validate that you are the one
you claim to be, as an addition to normal passwords. Curl supports client-
side certificates. All certificates are locked with a pass phrase, which you
need to enter before the certificate can be used by curl. The pass phrase
can be specified on the command line or if not, entered interactively when
curl queries for it. Use a certificate with curl on an HTTPS server like:
curl --cert mycert.pem https://secure.example.com
curl also tries to verify that the server is who it claims to be, by
verifying the server's certificate against a locally stored CA cert
bundle. Failing the verification will cause curl to deny the connection. You
must then use [`--insecure`](https://curl.se/docs/manpage.html#-k)
(`-k`) in case you want to tell curl to ignore that the server cannot be
verified.
More about server certificate verification and ca cert bundles can be read in
the [`SSLCERTS` document](https://curl.se/docs/sslcerts.html).
At times you may end up with your own CA cert store and then you can tell
curl to use that to verify the server's certificate:
curl --cacert ca-bundle.pem https://example.com/
# Custom Request Elements
## Modify method and headers
Doing fancy stuff, you may need to add or change elements of a single curl
request.
For example, you can change the POST method to `PROPFIND` and send the data
as `Content-Type: text/xml` (instead of the default `Content-Type`) like
this:
curl --data "" --header "Content-Type: text/xml" \
--request PROPFIND example.com
You can delete a default header by providing one without content. Like you
can ruin the request by chopping off the `Host:` header:
curl --header "Host:" http://www.example.com
You can add headers the same way. Your server may want a `Destination:`
header, and you can add it:
curl --header "Destination: http://nowhere" http://example.com
## More on changed methods
It should be noted that curl selects which methods to use on its own
depending on what action to ask for. `-d` will do POST, `-I` will do HEAD and
so on. If you use the
[`--request`](https://curl.se/docs/manpage.html#-X) / `-X` option you
can change the method keyword curl selects, but you will not modify curl's
behavior. This means that if you for example use -d "data" to do a POST, you
can modify the method to a `PROPFIND` with `-X` and curl will still think it
sends a POST . You can change the normal GET to a POST method by simply
adding `-X POST` in a command line like:
curl -X POST http://example.org/
... but curl will still think and act as if it sent a GET so it will not send
any request body etc.
# Web Login
## Some login tricks
While not strictly just HTTP related, it still causes a lot of people
problems so here's the executive run-down of how the vast majority of all
login forms work and how to login to them using curl.
It can also be noted that to do this properly in an automated fashion, you
will most certainly need to script things and do multiple curl invokes etc.
First, servers mostly use cookies to track the logged-in status of the
client, so you will need to capture the cookies you receive in the
responses. Then, many sites also set a special cookie on the login page (to
make sure you got there through their login page) so you should make a habit
of first getting the login-form page to capture the cookies set there.
Some web-based login systems feature various amounts of JavaScript, and
sometimes they use such code to set or modify cookie contents. Possibly they
do that to prevent programmed logins, like this manual describes how to...
Anyway, if reading the code is not enough to let you repeat the behavior
manually, capturing the HTTP requests done by your browsers and analyzing the
sent cookies is usually a working method to work out how to shortcut the
JavaScript need.
In the actual `