# Technical Documentation: QwenLM/Qwen3-Coder > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/QwenLM/Qwen3-Coder) ### Overview (README.md)

💜 Qwen Chat   |   🤗 Hugging Face   |   🤖 ModelScope   |    📑 Blog    ｜   📖 Documentation
🌍 WebDev   |   💬 WeChat (微信)   |   🫨 Discord   |    📄 Arxiv   |    👽 Qwen Code

Visit our Hugging Face or ModelScope organization (click links above), search checkpoints with names starting with `Qwen3-Coder-`, and you will find all you need! Enjoy! --- ## Table of Contents - [Introduction](#introduction) - [Key Features](#key-features) - [Basic Information](#basic-information) - [Quick Start](#quick-start) - [👉🏻 Chat with Qwen3-Coder](#-chat-with-qwen3-coder) - [Fill in the middle with Qwen3-Coder](#fill-in-the-middle-with-qwen3-coder) - [Use Cases](#use-cases) - [Example: Releasing a Website](#example-releasing-a-website) - [Example: Desktop Tidy](#example-desktop-tidy) - [Example: Zombies vs. Plants](#example-zombies-vs-plants) - [Example: Sound ASCII Art](#example-sound-ascii-art) - [Example: Vibe Checking](#example-vibe-checking) - [Example: Parkour Game](#example-parkour-game) - [Star History](#star-history) - [Citation](#citation) - [Contact Us](#contact-us) --- # Qwen3-Coder-Next: Pushing Small Hybrid Models on Agentic Coding ## Introduction We are announcing Qwen3-Coder, our most agentic code model to date. **Qwen3-Coder** is available in multiple sizes, **Qwen3-Coder-480B-A35B-Instruct**, **Qwen3-Coder-30B-A3B-Instruct**, **Qwen3-Coder-Next**, offering exceptional performance in both coding and agentic tasks. **Qwen3-Coder-Next**, an open-weight language model designed specifically for coding agents and local development. Built on top of **Qwen3-Next-80B-A3B-Base**, which adopts a novel architecture with hybrid attention and MoE, Qwen3-Coder-Next has been agentically trained at scale on large-scale executable task synthesis, environment interaction, and reinforcement learning, obtaining strong coding and agentic capabilities with significantly lower inference costs. ### Key Features 💻 **Efficiency-Performance Tradeoff**: among open models on **Agentic Coding**, **Agentic Browser-Use**, and other foundational coding tasks, achieving results comparable to Claude Sonnet. 🛠 **Scaling Agentic Coding**: supporting most platforms such as **Qwen Code**, **CLINE**, **Claude Code**, featuring a specially designed function call format; 📚 **Long-context Capabilities**: with native support for **256K** tokens, extendable up to **1M** tokens using Yarn, optimized for repository-scale understanding. --- ## Basic Information 1. ✨ Supporting long context understanding and generation with the context length of 256K tokens; 2. ✨ Supporting 358 coding languages;
Click to view all supported languages ``` /* Detailed source-code truncated for AI context efficiency. */ ```
3. ✨ Retain strengths in math and general capabilities from base model. > [!Important] > > Qwen3-Coder function calling relies on our new tool parser in both **SGLang** and **vLLM** here. > > We updated both the special tokens and their corresponding token ids, in order to maintain consistency with Qwen3. Please make sure to use the new tokenizer. | model name | type | length | Download | |-----------------------------|----------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Qwen3-Coder-Next | instruct | 256k | 🤗 [Hugging Face](https://huggingface.co/Qwen/Qwen3-Coder-Next ) • 🤖 [ModelScope](https://modelscope.cn/models/Qwen/Qwen3-Coder-Next) | | Qwen3-Coder-Next-Base | base | 256k | 🤗 [Hugging Face](https://huggingface.co/Qwen/Qwen3-Coder-Next-Base) • 🤖 [ModelScope](https://modelscope.cn/models/Qwen/Qwen3-Coder-Next-Base) | | Qwen3-Coder-480B-A35B-Instruct | instruct | 256k | 🤗 [Hugging Face](https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct ) • 🤖 [ModelScope](https://modelscope.cn/models/Qwen/Qwen3-Coder-480B-A35B-Instruct) | | Qwen3-Coder-30B-A3B-Instruct | instruct | 256k | 🤗 [Hugging Face](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct) • 🤖 [ModelScope](https://modelscope.cn/models/Qwen/Qwen3-Coder-30B-A3B-Instruct) | | Qwen3-Coder-Next-FP8 | instruct | 256k | 🤗 [Hugging Face](https://huggingface.co/Qwen/Qwen3-Coder-Next-FP8 ) • 🤖 [ModelScope](https://modelscope.cn/models/Qwen/Qwen3-Coder-Next-FP8) | Qwen3-Coder-Next-GGUF | instruct | 256k | 🤗 [Hugging Face](https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF ) • 🤖 [ModelScope](https://modelscope.cn/models/Qwen/Qwen3-Coder-Next-GGUF) | | Qwen3-Coder-480B-A35B-Instruct-FP8 | instruct | 256k | 🤗 [Hugging Face](https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8) • 🤖 [ModelScope](https://modelscope.cn/models/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8) | | Qwen3-Coder-30B-A3B-Instruct-FP8 | instruct | 256k | 🤗 [Hugging Face](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8) • 🤖 [ModelScope](https://modelscope.cn/models/Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8) | Detailed performance and introduction are shown in this 📑 blog. --- ## Quick Start > [!Important] > **Qwen3-Coder** are instruct models for chatting; > > This model supports only non-thinking mode and does not generate `` blocks in its output. Meanwhile, specifying `enable_thinking=False` is no longer required. > ### 👉🏻 Chat with Qwen3-Coder You can write several lines of code with `transformers` to chat with Qwen3-Coder-Next. Essentially, we build the tokenizer and the model with the `from_pretrained` method, and we use the generate method to perform chatting with the help of the chat template provided by the tokenizer. Below is an example of how to chat with **Qwen3-Coder-Next**: ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "Qwen/Qwen3-Coder-Next" model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype="auto", device_map="auto" ) tokenizer = AutoTokenizer.from_pretrained(model_name) prompt = "write a quick sort algorithm." messages = [ {"role": "user", "content": prompt} ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) model_inputs = tokenizer([text], return_tensors="pt").to(model.device) generated_ids = model.generate( **model_inputs, max_new_tokens=65536 ) generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids) ] response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] ``` The `apply_chat_template()` function is used to convert the messages into a format that the model can understand. The `add_generation_prompt` argument is used to add a generation prompt, which refers to `<|im_start|>assistant\n` to the input. Notably, we apply the ChatML template for chat models following our previous practice. The `max_new_tokens` argument is used to set the maximum length of the response. The `tokenizer.batch_decode()` function is used to decode the response. In terms of the input, the above messages are an example to show how to format your dialog history and system prompt. You can use the other sizes of instruct models in the same way. #### Fill in the middle with Qwen3-Coder The code insertion task, also referred to as the "fill-in-the-middle" challenge, requires the insertion of code segments in a manner that bridges the gaps within a given code context. For an approach aligned with best practices, we recommend adhering to the formatting guidelines outlined in the paper "Efficient Training of Language Models to Fill in the Middle" [[arxiv](https://arxiv.org/abs/2207.14255)]. > [!Important] > It should be noted that FIM is supported in every version of Qwen3-Coder. Qwen3-Coder-Next is shown here as an example. > The prompt should be structured as follows: ```python prompt = '<|fim_prefix|>' + prefix_code + '<|fim_suffix|>' + suffix_code + '<|fim_middle|>' ``` Following the approach mentioned, an example would be structured in this manner: ```python from transformers import AutoTokenizer, AutoModelForCausalLM # load model device = "cuda" # the device to load the model onto TOKENIZER = AutoTokenizer.from_pretrained("Qwen/Qwen3-Coder-Next") MODEL = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-Coder-Next", device_map="auto").eval() input_text = """<|fim_prefix|>def quicksort(arr): if len(arr) <= 1: return arr pivot = arr[len(arr) // 2] <|fim_suffix|> middle = [x for x in arr if x == pivot] right = [x for x in arr if x > pivot] return quicksort(left) + middle + quicksort(right)<|fim_middle|>""" messages = [ {"role": "system", "content": "You are a code completion assistant."}, {"role": "user", "content": input_text} ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) model_inputs = TOKENIZER([text], return_tensors="pt").to(model.device) # Use `max_new_tokens` to control the maximum output length. eos_token_ids = [151659, 151661, 151662, 151663, 151664, 151643, 151645] generated_ids = MODEL.generate(model_inputs.input_ids, max_new_tokens=512, do_sample=False, eos_token_id=eos_token_ids)[0] # The generated_ids include prompt_ids, we only need to decode the tokens after prompt_ids. output_text = TOKENIZER.decode(generated_ids[len(model_inputs.input_ids[0]):], skip_special_tokens=True) print(f"Prompt: {input_text}\n\nGenerated text: {output_text}") ``` ## Use Cases ### Example: Releasing a Website
Prompt with OpenClaw ``` next week we will release new coder model, can you collect the history of qwen coder and write a web page, the release the website with the nginx, you can seach how to do this in alibaba cloud linux first ```

### Example: Desktop Tidy
Prompt with Qwen Code ``` Please tidy up my desk. ```

### Example: Zombies vs. Plants
Prompt with Claude Code ``` 帮我实现《僵尸大战植物》网页游戏 【核心机制】 - 反向塔防：玩家扮演僵尸方，从地图右侧（部署区）召唤僵尸向左进攻 - 资源循环：初始300脑子点数，僵尸吃掉植物返还100点，形成经济循环 - 倒计时制：120秒内清除所有植物获胜，超时失败 【地图规格】 - 5行9列网格，右侧3列为可部署区域（红色高亮标识） - 每格100x100像素，草地纹理交替渲染 - 坐标系：左侧为植物防线，右侧为僵尸出生点 【单位系统】 僵尸方（右侧购买）： - 普通僵尸：50脑，100HP，0.5速，标准单位 - 路障僵尸：100脑，200HP，0.5速，中期肉盾 - 铁桶僵尸：150脑，400HP，0.3速，重型坦克 - 冲刺僵尸：80脑，80HP，1.2速，快速突进 植物方（左侧随机初始部署12个）： - 豌豆射手：100HP，20伤/发，2秒间隔，直线射击 - 双发射手：120HP，20伤/发，1秒间隔，火力压制 - 坚果墙：300HP，0伤，纯肉盾阻挡 - 向日葵：80HP，0伤，经济单位（纯干扰） 【战斗逻辑】 - 碰撞检测：僵尸到达植物50px内触发啃食状态，停止移动 - 伤害结算：僵尸30帧/次咬击（0.5秒），植物射出弹道物理 - 击毁反馈：植物死亡时生成"+100"飘字特效与粒子爆炸 - 路径AI：同格僵尸队列不重叠，植物优先攻击横向最近目标 【交互设计】 - 右侧卡片式UI：显示僵尸图标、名称、脑子消耗 - 资源不足时卡片置灰并自动切换可选类型 - 鼠标悬停部署区显示半透明预览圈 - 实时血条：实体头顶显示绿/黄/红三色血槽 【胜利条件】 - 胜利：plants.length === 0 && timeLeft > 0 - 失败：timeLeft === 0 || (可选)僵尸全灭且脑子为0 ```

### Example: Sound ASCII Art
Prompt with Cline ``` Build an interactive ASCII art drawing tool with sound feedback. The application should: 1. Create a canvas where users can draw by clicking and dragging 2. Place different ASCII characters or symbols when the user draws 3. Play corresponding musical notes when each character is placed 4. Include multiple pattern sets with different characters and corresponding note scales 5. Add a pattern switcher button to cycle through different character/sound themes 6. Include a clear button to reset the canvas 7. Support both mouse and touch input for mobile compatibility The application should be creative and fun to use, creating an audio-visual experience where patterns of characters create both visual art and musical patterns. Ensure the musical notes are harmonious when played in sequence. ```

### Example: Vibe Checking
Prompt with Browser Use Agent ``` Vibe test this website. Click around, try things, report what's broken. ```

### Example: Parkour Game
Prompt with Qwen Chat Web Dev ``` Create an interactive real-time particle system using HTML5 Canvas: Core Features: - Render 800-1200 animated particles with physics-based movement - Mouse cursor exerts attractive/repulsive force on nearby particles - Click to toggle between attraction and repulsion modes - Particles respond with smooth acceleration and velocity calculations Technical Requirements: - Use requestAnimationFrame for optimal performance - Implement force calculation based on distance from cursor - Add visual feedback: particle glow, color variation, and fade effects - Include performance monitoring (FPS counter) Deliverables: - Single HTML file with embedded CSS and JavaScript - Clean, commented code following best practices - Responsive design compatible with modern browsers ```

--- ## Star History [](https://star-history.com/#QwenLM/Qwen3-Coder&Date) --- ## Citation If you find our work helpful, feel free to give us a cite. ```bibtex @article{Qwen3-Coder-Next, title={Qwen3-Coder-Next Technical Report}, author={Ruisheng Cao and Mouxiang Chen and Jiawei Chen and Zeyu Cui and Yunlong Feng and Binyuan Hui and Yuheng Jing and Kaixin Li and Mingze Li and Junyang Lin and Zeyao Ma and Kashun Shum and Xuwu Wang and Jinxi Wei and Jiaxi Yang and Jiajun Zhang and Lei Zhang and Zongmeng Zhang and Wenting Zhao and Fan Zhou}, journal={arXiv preprint arXiv:2603.00729}, year={2026}, } ``` --- ## Contact Us If you are interested to leave a message to either our research team or product team, join our [Discord](https://discord.gg/z3GAxXZ9Ce) or [WeChat groups](https://github.com/QwenLM/Qwen/blob/main/assets/wechat.png)!

↑ Back to Top ↑

--- ### Qwencoder Eval Instruct Eval Dev Quality Docs Kubernetes README (`qwencoder-eval/instruct/eval-dev-quality/docs/kubernetes/README.md`) # Running "eval-dev-quality" on Kubernetes ### Prerequisite Checklist - `kubectl` installed and configured with authentication to the cluster. - Dedicated Namespace to run the jobs. - RWX volume to store the evaluation results (check `volume.yml` for inspiration). ### Defining secrets The Job automatically references to a secret called `evaluation-secret` which is configured to pass the values on as environment Variables. The following key is required to be created. - `PROVIDER_TOKEN` - contains the API tokens for different providers. E.g.: `openrouter:abcdefgh1234,custom-provider:abcdefgh1234` ```bash kubectl --namespace eval-dev-quality create secret generic evaluation-secret --from-literal='PROVIDER_TOKEN=' ``` ### Running multiple evaluations with the eval-dev-quality kubernetes runtime - Define all the models with `--model` which should be run inside the containerized workload. - Define the parameter `--runtime kubernetes` to indicate that jobs should run inside a kubernetes cluster. - Define the parameter `--parallel 20` to indicate how many jobs should run simultaneously. Example: ```bash eval-dev-quality evaluate --runtime kubernetes --runs 5 --model symflower/symbolic-execution --model symflower/symbolic-execution --model symflower/symbolic-execution --repository golang/plain --parallel 2 ``` This commands run 3x the `symflower/symbolic-execution` model with 5 runs of each model inside a containerized workload on the kubernetes cluster, it will limit the parallel execution to 2 containers at the same time. --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.4.0 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/README.md`) **Note that for the [official blog post](https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/) the weighting of the result computation was adjusted to award responses that lead to `100% code coverage` with `10` instead of `1` point score!** --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.6 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.6/README.md`) This evaluation ran over several days and is actually assembled from several sub-evaluations. Hence, we only provide the `csv` results for now because the logs are distributed all over. The benchmark consisted of `5x` runs, **except** for the `code-repair` task, which contains only one usable run because of [a bug that is already fixed for later versions](https://github.com/symflower/eval-dev-quality/commit/19011da21c3196e346d363148bf83989dc1f2a88). --- ### Qwencoder Eval Instruct Aider Aider Website Docs Leaderboards Index (`qwencoder-eval/instruct/aider/aider/website/docs/leaderboards/index.md`) --- highlight_image: /assets/leaderboard.jpg nav_order: 950 description: Quantitative benchmarks of LLM code editing skill. --- # Aider LLM Leaderboards {: .no_toc } Aider works best with LLMs which are good at *editing* code, not just good at writing code. To evaluate an LLM's editing skill, aider uses a pair of benchmarks that assess a model's ability to consistently follow the system prompt to successfully edit code. The leaderboards below report the results from a number of popular LLMs. While [aider can connect to almost any LLM](/docs/llms.html), it works best with models that score well on the benchmarks. See the following sections for benchmark results and additional information: - TOC {:toc} ## Code editing leaderboard [Aider's code editing benchmark](/docs/benchmarks.html#the-benchmark) asks the LLM to edit python source files to complete 133 small coding exercises from Exercism. This measures the LLM's coding ability, and whether it can write new code that integrates into existing code. The model also has to successfully apply all its changes to the source file without human intervention. {% assign edit_sorted = site.data.edit_leaderboard | sort: 'pass_rate_2' | reverse %} {% for row in edit_sorted %} {% endfor %}
Model Percent completed correctly Percent using correct edit format Command Edit format
{{ row.model }} {{ row.pass_rate_2 }}% {{ row.percent_cases_well_formed }}% {{ row.command }} {{ row.edit_format }}
## Code refactoring leaderboard [Aider's refactoring benchmark](https://github.com/paul-gauthier/refactor-benchmark) asks the LLM to refactor 89 large methods from large python classes. This is a more challenging benchmark, which tests the model's ability to output long chunks of code without skipping sections or making mistakes. It was developed to provoke and measure [GPT-4 Turbo's "lazy coding" habit](/2023/12/21/unified-diffs.html). The refactoring benchmark requires a large context window to work with large source files. Therefore, results are available for fewer models. {% assign refac_sorted = site.data.refactor_leaderboard | sort: 'pass_rate_1' | reverse %} {% for row in refac_sorted %} {% endfor %}
Model Percent completed correctly Percent using correct edit format Command Edit format
{{ row.model }} {{ row.pass_rate_1 }}% {{ row.percent_cases_well_formed }}% {{ row.command }} {{ row.edit_format }}
## LLM code editing skill by model release date [](https://aider.chat/assets/models-over-time.svg) ## Notes on benchmarking results The key benchmarking results are: - **Percent completed correctly** - Measures what percentage of the coding tasks that the LLM completed successfully. To complete a task, the LLM must solve the programming assignment *and* edit the code to implement that solution. - **Percent using correct edit format** - Measures the percent of coding tasks where the LLM complied with the edit format specified in the system prompt. If the LLM makes edit mistakes, aider will give it feedback and ask for a fixed copy of the edit. The best models can reliably conform to the edit format, without making errors. ## Notes on the edit format Aider uses different "edit formats" to collect code edits from different LLMs. The "whole" format is the easiest for an LLM to use, but it uses a lot of tokens and may limit how large a file can be edited. Models which can use one of the diff formats are much more efficient, using far fewer tokens. Models that use a diff-like format are able to edit larger files with less cost and without hitting token limits. Aider is configured to use the best edit format for the popular OpenAI and Anthropic models and the [other models recommended on the LLM page](/docs/llms.html). For lesser known models aider will default to using the "whole" editing format since it is the easiest format for an LLM to use. ## Contributing benchmark results Contributions of benchmark results are welcome! See the [benchmark README](https://github.com/paul-gauthier/aider/blob/main/benchmark/README.md) for information on running aider's code editing benchmarks. Submit results by opening a PR with edits to the [benchmark results data files](https://github.com/paul-gauthier/aider/blob/main/website/_data/).

By Paul Gauthier, last updated September 05, 2024.

--- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.4.0 1 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/1/README.md`) # Evaluation from 2024-04-25 13:55:10 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.4.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - Category Unknown: Models in this category could not be categorized. - Response Error: Models in this category encountered an error. - Response Empty: Models in this category produced an empty response. - No Code: Models in this category produced no code. - Invalid Code: Models in this category produced invalid code. - Executable Code: Models in this category produced executable code. - Statement Coverage Reached: Models in this category produced code that reached full statement coverage. - No Excess Response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### "Response Error" Models in this category encountered an error. - `openrouter/anthropic/claude-2:beta` - `openrouter/anthropic/claude-2` - `openrouter/anthropic/claude-instant-1:beta` - `openrouter/anthropic/claude-instant-1` - `openrouter/haotian-liu/llava-13b` - `openrouter/jebcarter/psyfighter-13b` - `openrouter/jondurbin/bagel-34b` - `openrouter/lynn/soliloquy-l3` - `openrouter/nousresearch/nous-hermes-2-vision-7b` ### "Response Empty" Models in this category produced an empty response. - `openrouter/databricks/dbrx-instruct` ### "No Code" Models in this category produced no code. - `openrouter/01-ai/yi-34b` - `openrouter/alpindale/goliath-120b` - `openrouter/anthropic/claude-1` - `openrouter/anthropic/claude-instant-1.0` - `openrouter/austism/chronos-hermes-13b` - `openrouter/cognitivecomputations/dolphin-mixtral-8x7b` - `openrouter/google/gemini-pro` - `openrouter/gryphe/mythomax-l2-13b:extended` - `openrouter/gryphe/mythomax-l2-13b` - `openrouter/gryphe/mythomist-7b:free` - `openrouter/gryphe/mythomist-7b` - `openrouter/intel/neural-chat-7b` - `openrouter/koboldai/psyfighter-13b-2` - `openrouter/mancer/weaver` - `openrouter/meta-llama/llama-3-8b-instruct:extended` - `openrouter/mistralai/mistral-7b-instruct` - `openrouter/mistralai/mixtral-8x22b` - `openrouter/mistralai/mixtral-8x7b` - `openrouter/neversleep/noromaid-20b` - `openrouter/neversleep/noromaid-mixtral-8x7b-instruct` - `openrouter/nousresearch/nous-capybara-7b` - `openrouter/openai/gpt-3.5-turbo-0613` - `openrouter/openai/gpt-3.5-turbo-16k` - `openrouter/openai/gpt-3.5-turbo-instruct` - `openrouter/perplexity/pplx-7b-chat` - `openrouter/togethercomputer/stripedhyena-hessian-7b` - `openrouter/undi95/remm-slerp-l2-13b:extended` - `openrouter/undi95/toppy-m-7b` ### "Invalid Code" Models in this category produced invalid code. - `openrouter/01-ai/yi-34b-chat` - `openrouter/01-ai/yi-6b` - `openrouter/anthropic/claude-1.2` - `openrouter/anthropic/claude-2.0:beta` - `openrouter/anthropic/claude-instant-1.1` - `openrouter/anthropic/claude-instant-1.2` - `openrouter/codellama/codellama-70b-instruct` - `openrouter/cohere/command-r-plus` - `openrouter/cohere/command-r` - `openrouter/cohere/command` - `openrouter/google/gemini-pro-vision` - `openrouter/google/gemma-7b-it:free` - `openrouter/google/gemma-7b-it:nitro` - `openrouter/google/gemma-7b-it` - `openrouter/google/palm-2-chat-bison-32k` - `openrouter/google/palm-2-chat-bison` - `openrouter/google/palm-2-codechat-bison-32k` - `openrouter/google/palm-2-codechat-bison` - `openrouter/gryphe/mythomax-l2-13b:nitro` - `openrouter/huggingfaceh4/zephyr-7b-beta:free` - `openrouter/huggingfaceh4/zephyr-7b-beta` - `openrouter/huggingfaceh4/zephyr-orpo-141b-a35b` - `openrouter/lizpreciatior/lzlv-70b-fp16-hf` - `openrouter/meta-llama/llama-2-13b-chat` - `openrouter/meta-llama/llama-2-70b-chat` - `openrouter/meta-llama/llama-3-8b-instruct:nitro` - `openrouter/meta-llama/llama-3-8b-instruct` - `openrouter/microsoft/wizardlm-2-7b` - `openrouter/mistralai/mistral-7b-instruct:free` - `openrouter/mistralai/mistral-large` - `openrouter/mistralai/mistral-small` - `openrouter/mistralai/mistral-tiny` - `openrouter/mistralai/mixtral-8x22b-instruct` - `openrouter/mistralai/mixtral-8x7b-instruct:nitro` - `openrouter/mistralai/mixtral-8x7b-instruct` - `openrouter/nousresearch/nous-capybara-34b` - `openrouter/nousresearch/nous-capybara-7b:free` - `openrouter/nousresearch/nous-hermes-2-mistral-7b-dpo` - `openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpo` - `openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sft` - `openrouter/nousresearch/nous-hermes-llama2-13b` - `openrouter/nousresearch/nous-hermes-yi-34b` - `openrouter/open-orca/mistral-7b-openorca` - `openrouter/openai/gpt-4-turbo-preview` - `openrouter/openchat/openchat-7b:free` - `openrouter/openchat/openchat-7b` - `openrouter/openrouter/cinematika-7b:free` - `openrouter/openrouter/cinematika-7b` - `openrouter/perplexity/pplx-70b-online` - `openrouter/perplexity/pplx-7b-online` - `openrouter/perplexity/sonar-medium-online` - `openrouter/perplexity/sonar-small-chat` - `openrouter/perplexity/sonar-small-online` - `openrouter/phind/phind-codellama-34b` - `openrouter/pygmalionai/mythalion-13b` - `openrouter/recursal/eagle-7b` - `openrouter/recursal/rwkv-5-3b-ai-town` - `openrouter/rwkv/rwkv-5-world-3b` - `openrouter/sao10k/fimbulvetr-11b-v2` - `openrouter/teknium/openhermes-2-mistral-7b` - `openrouter/teknium/openhermes-2.5-mistral-7b` - `openrouter/undi95/toppy-m-7b:free` - `openrouter/undi95/toppy-m-7b:nitro` - `openrouter/xwin-lm/xwin-lm-70b` ### "Executable Code" Models in this category produced executable code. - `openrouter/togethercomputer/stripedhyena-nous-7b` ### "Statement Coverage Reached" Models in this category produced code that reached full statement coverage. - `openrouter/anthropic/claude-2.0` - `openrouter/anthropic/claude-2.1:beta` - `openrouter/anthropic/claude-2.1` - `openrouter/anthropic/claude-3-haiku:beta` - `openrouter/anthropic/claude-3-haiku` - `openrouter/anthropic/claude-3-opus:beta` - `openrouter/anthropic/claude-3-opus` - `openrouter/anthropic/claude-3-sonnet` - `openrouter/jondurbin/airoboros-l2-70b` - `openrouter/meta-llama/codellama-34b-instruct` - `openrouter/meta-llama/llama-2-70b-chat:nitro` - `openrouter/meta-llama/llama-3-70b-instruct:nitro` - `openrouter/meta-llama/llama-3-70b-instruct` - `openrouter/microsoft/wizardlm-2-8x22b:nitro` - `openrouter/microsoft/wizardlm-2-8x22b` - `openrouter/mistralai/mistral-medium` - `openrouter/openai/gpt-3.5-turbo-0301` - `openrouter/openai/gpt-4-1106-preview` - `openrouter/openai/gpt-4-32k` - `openrouter/perplexity/pplx-70b-chat` - `openrouter/perplexity/sonar-medium-chat` - `openrouter/sophosympatheia/midnight-rose-70b` - `openrouter/undi95/remm-slerp-l2-13b` ### "No Excess Response" Models in this category did not respond with more content than requested. - `openrouter/anthropic/claude-3-sonnet:beta` - `openrouter/google/gemini-pro-1.5` - `openrouter/mistralai/mistral-7b-instruct:nitro` - `openrouter/openai/gpt-3.5-turbo-0125` - `openrouter/openai/gpt-3.5-turbo-1106` - `openrouter/openai/gpt-3.5-turbo` - `openrouter/openai/gpt-4-0314` - `openrouter/openai/gpt-4-32k-0314` - `openrouter/openai/gpt-4-turbo` - `openrouter/openai/gpt-4-vision-preview` - `openrouter/openai/gpt-4` - `openrouter/openrouter/auto` - `symflower/symbolic-execution` --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.4.0 2 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/2/README.md`) # Evaluation from 2024-04-25 18:19:05 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.4.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - Category Unknown: Models in this category could not be categorized. - Response Error: Models in this category encountered an error. - Response Empty: Models in this category produced an empty response. - No Code: Models in this category produced no code. - Invalid Code: Models in this category produced invalid code. - Executable Code: Models in this category produced executable code. - Statement Coverage Reached: Models in this category produced code that reached full statement coverage. - No Excess Response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### "Response Error" Models in this category encountered an error. - `openrouter/anthropic/claude-2:beta` - `openrouter/anthropic/claude-2` - `openrouter/anthropic/claude-instant-1:beta` - `openrouter/anthropic/claude-instant-1` - `openrouter/haotian-liu/llava-13b` - `openrouter/jebcarter/psyfighter-13b` - `openrouter/jondurbin/bagel-34b` - `openrouter/nousresearch/nous-hermes-2-vision-7b` - `openrouter/openchat/openchat-7b:free` - `openrouter/perplexity/pplx-7b-online` - `openrouter/recursal/rwkv-5-3b-ai-town` - `openrouter/undi95/remm-slerp-l2-13b` ### "Response Empty" Models in this category produced an empty response. - `openrouter/01-ai/yi-34b` - `openrouter/databricks/dbrx-instruct` - `openrouter/mistralai/mixtral-8x22b` ### "No Code" Models in this category produced no code. - `openrouter/anthropic/claude-1.2` - `openrouter/anthropic/claude-1` - `openrouter/austism/chronos-hermes-13b` - `openrouter/codellama/codellama-70b-instruct` - `openrouter/gryphe/mythomax-l2-13b:extended` - `openrouter/gryphe/mythomax-l2-13b` - `openrouter/gryphe/mythomist-7b:free` - `openrouter/gryphe/mythomist-7b` - `openrouter/huggingfaceh4/zephyr-orpo-141b-a35b` - `openrouter/intel/neural-chat-7b` - `openrouter/jondurbin/airoboros-l2-70b` - `openrouter/koboldai/psyfighter-13b-2` - `openrouter/lynn/soliloquy-l3` - `openrouter/mancer/weaver` - `openrouter/meta-llama/llama-3-8b-instruct:extended` - `openrouter/mistralai/mistral-tiny` - `openrouter/mistralai/mixtral-8x7b` - `openrouter/neversleep/noromaid-20b` - `openrouter/neversleep/noromaid-mixtral-8x7b-instruct` - `openrouter/nousresearch/nous-capybara-7b:free` - `openrouter/open-orca/mistral-7b-openorca` - `openrouter/openai/gpt-3.5-turbo-instruct` - `openrouter/pygmalionai/mythalion-13b` - `openrouter/togethercomputer/stripedhyena-hessian-7b` - `openrouter/undi95/remm-slerp-l2-13b:extended` - `openrouter/undi95/toppy-m-7b:nitro` ### "Invalid Code" Models in this category produced invalid code. - `openrouter/01-ai/yi-34b-chat` - `openrouter/01-ai/yi-6b` - `openrouter/alpindale/goliath-120b` - `openrouter/anthropic/claude-2.0:beta` - `openrouter/anthropic/claude-3-haiku:beta` - `openrouter/anthropic/claude-instant-1.1` - `openrouter/anthropic/claude-instant-1.2` - `openrouter/cognitivecomputations/dolphin-mixtral-8x7b` - `openrouter/cohere/command-r-plus` - `openrouter/cohere/command` - `openrouter/google/gemini-pro-vision` - `openrouter/google/gemini-pro` - `openrouter/google/gemma-7b-it:free` - `openrouter/google/gemma-7b-it:nitro` - `openrouter/google/gemma-7b-it` - `openrouter/google/palm-2-chat-bison-32k` - `openrouter/google/palm-2-chat-bison` - `openrouter/google/palm-2-codechat-bison-32k` - `openrouter/google/palm-2-codechat-bison` - `openrouter/gryphe/mythomax-l2-13b:nitro` - `openrouter/huggingfaceh4/zephyr-7b-beta:free` - `openrouter/huggingfaceh4/zephyr-7b-beta` - `openrouter/lizpreciatior/lzlv-70b-fp16-hf` - `openrouter/meta-llama/llama-2-70b-chat` - `openrouter/meta-llama/llama-3-70b-instruct:nitro` - `openrouter/meta-llama/llama-3-8b-instruct:nitro` - `openrouter/meta-llama/llama-3-8b-instruct` - `openrouter/microsoft/wizardlm-2-7b` - `openrouter/mistralai/mistral-7b-instruct:free` - `openrouter/mistralai/mistral-7b-instruct:nitro` - `openrouter/mistralai/mistral-7b-instruct` - `openrouter/mistralai/mistral-large` - `openrouter/mistralai/mistral-small` - `openrouter/mistralai/mixtral-8x22b-instruct` - `openrouter/nousresearch/nous-capybara-34b` - `openrouter/nousresearch/nous-capybara-7b` - `openrouter/nousresearch/nous-hermes-2-mistral-7b-dpo` - `openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpo` - `openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sft` - `openrouter/nousresearch/nous-hermes-llama2-13b` - `openrouter/nousresearch/nous-hermes-yi-34b` - `openrouter/openai/gpt-3.5-turbo-16k` - `openrouter/openchat/openchat-7b` - `openrouter/openrouter/cinematika-7b:free` - `openrouter/openrouter/cinematika-7b` - `openrouter/perplexity/pplx-70b-online` - `openrouter/perplexity/pplx-7b-chat` - `openrouter/perplexity/sonar-medium-chat` - `openrouter/perplexity/sonar-medium-online` - `openrouter/perplexity/sonar-small-chat` - `openrouter/perplexity/sonar-small-online` - `openrouter/recursal/eagle-7b` - `openrouter/rwkv/rwkv-5-world-3b` - `openrouter/sao10k/fimbulvetr-11b-v2` - `openrouter/sophosympatheia/midnight-rose-70b` - `openrouter/teknium/openhermes-2-mistral-7b` - `openrouter/teknium/openhermes-2.5-mistral-7b` - `openrouter/togethercomputer/stripedhyena-nous-7b` - `openrouter/undi95/toppy-m-7b:free` - `openrouter/undi95/toppy-m-7b` - `openrouter/xwin-lm/xwin-lm-70b` ### "Statement Coverage Reached" Models in this category produced code that reached full statement coverage. - `openrouter/anthropic/claude-2.0` - `openrouter/anthropic/claude-2.1:beta` - `openrouter/anthropic/claude-2.1` - `openrouter/anthropic/claude-3-haiku` - `openrouter/anthropic/claude-3-opus:beta` - `openrouter/anthropic/claude-3-opus` - `openrouter/anthropic/claude-3-sonnet:beta` - `openrouter/anthropic/claude-3-sonnet` - `openrouter/anthropic/claude-instant-1.0` - `openrouter/meta-llama/codellama-34b-instruct` - `openrouter/meta-llama/llama-2-13b-chat` - `openrouter/meta-llama/llama-2-70b-chat:nitro` - `openrouter/microsoft/wizardlm-2-8x22b:nitro` - `openrouter/microsoft/wizardlm-2-8x22b` - `openrouter/mistralai/mistral-medium` - `openrouter/mistralai/mixtral-8x7b-instruct:nitro` - `openrouter/mistralai/mixtral-8x7b-instruct` - `openrouter/openai/gpt-3.5-turbo-0301` - `openrouter/openai/gpt-3.5-turbo-1106` - `openrouter/openai/gpt-4-1106-preview` - `openrouter/openai/gpt-4-32k` - `openrouter/openai/gpt-4` - `openrouter/perplexity/pplx-70b-chat` - `openrouter/phind/phind-codellama-34b` ### "No Excess Response" Models in this category did not respond with more content than requested. - `openrouter/cohere/command-r` - `openrouter/google/gemini-pro-1.5` - `openrouter/meta-llama/llama-3-70b-instruct` - `openrouter/openai/gpt-3.5-turbo-0125` - `openrouter/openai/gpt-3.5-turbo-0613` - `openrouter/openai/gpt-3.5-turbo` - `openrouter/openai/gpt-4-0314` - `openrouter/openai/gpt-4-32k-0314` - `openrouter/openai/gpt-4-turbo-preview` - `openrouter/openai/gpt-4-turbo` - `openrouter/openai/gpt-4-vision-preview` - `openrouter/openrouter/auto` - `symflower/symbolic-execution` --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.4.0 3 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/3/README.md`) # Evaluation from 2024-04-25 19:03:54 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.4.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - Category Unknown: Models in this category could not be categorized. - Response Error: Models in this category encountered an error. - Response Empty: Models in this category produced an empty response. - No Code: Models in this category produced no code. - Invalid Code: Models in this category produced invalid code. - Executable Code: Models in this category produced executable code. - Statement Coverage Reached: Models in this category produced code that reached full statement coverage. - No Excess Response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### "Response Error" Models in this category encountered an error. - `openrouter/nousresearch/nous-hermes-2-vision-7b` - `openrouter/jondurbin/bagel-34b` - `openrouter/anthropic/claude-2:beta` - `openrouter/anthropic/claude-instant-1` - `openrouter/jebcarter/psyfighter-13b` - `openrouter/anthropic/claude-instant-1:beta` - `openrouter/anthropic/claude-2` - `openrouter/haotian-liu/llava-13b` ### "Response Empty" Models in this category produced an empty response. - `openrouter/databricks/dbrx-instruct` - `openrouter/01-ai/yi-34b` ### "No Code" Models in this category produced no code. - `openrouter/nousresearch/nous-capybara-7b:free` - `openrouter/nousresearch/nous-hermes-yi-34b` - `openrouter/neversleep/noromaid-mixtral-8x7b-instruct` - `openrouter/anthropic/claude-1.2` - `openrouter/pygmalionai/mythalion-13b` - `openrouter/intel/neural-chat-7b` - `openrouter/openchat/openchat-7b:free` - `openrouter/gryphe/mythomax-l2-13b:extended` - `openrouter/anthropic/claude-instant-1.2` - `openrouter/lynn/soliloquy-l3` - `openrouter/undi95/remm-slerp-l2-13b:extended` - `openrouter/open-orca/mistral-7b-openorca` - `openrouter/nousresearch/nous-capybara-7b` - `openrouter/mancer/weaver` - `openrouter/undi95/toppy-m-7b:nitro` - `openrouter/mistralai/mistral-tiny` - `openrouter/gryphe/mythomax-l2-13b:nitro` - `openrouter/neversleep/noromaid-20b` - `openrouter/nousresearch/nous-hermes-2-mistral-7b-dpo` - `openrouter/togethercomputer/stripedhyena-hessian-7b` - `openrouter/codellama/codellama-70b-instruct` - `openrouter/gryphe/mythomist-7b:free` - `openrouter/openai/gpt-3.5-turbo-instruct` - `openrouter/anthropic/claude-1` - `openrouter/meta-llama/llama-3-8b-instruct:extended` - `openrouter/mistralai/mixtral-8x22b` - `openrouter/gryphe/mythomist-7b` - `openrouter/undi95/remm-slerp-l2-13b` - `openrouter/anthropic/claude-instant-1.0` ### "Invalid Code" Models in this category produced invalid code. - `openrouter/teknium/openhermes-2-mistral-7b` - `openrouter/meta-llama/llama-3-8b-instruct` - `openrouter/perplexity/pplx-70b-online` - `openrouter/perplexity/sonar-small-chat` - `openrouter/openrouter/cinematika-7b:free` - `openrouter/nousresearch/nous-hermes-llama2-13b` - `openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sft` - `openrouter/google/gemma-7b-it` - `openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpo` - `openrouter/undi95/toppy-m-7b` - `openrouter/huggingfaceh4/zephyr-orpo-141b-a35b` - `openrouter/google/palm-2-chat-bison-32k` - `openrouter/perplexity/pplx-7b-chat` - `openrouter/google/palm-2-codechat-bison-32k` - `openrouter/01-ai/yi-6b` - `openrouter/mistralai/mixtral-8x7b` - `openrouter/openai/gpt-3.5-turbo-16k` - `openrouter/google/gemma-7b-it:nitro` - `openrouter/mistralai/mistral-small` - `openrouter/mistralai/mistral-large` - `openrouter/openrouter/cinematika-7b` - `openrouter/cohere/command-r` - `openrouter/sao10k/fimbulvetr-11b-v2` - `openrouter/mistralai/mistral-7b-instruct` - `openrouter/undi95/toppy-m-7b:free` - `openrouter/recursal/eagle-7b` - `openrouter/teknium/openhermes-2.5-mistral-7b` - `openrouter/anthropic/claude-instant-1.1` - `openrouter/alpindale/goliath-120b` - `openrouter/cognitivecomputations/dolphin-mixtral-8x7b` - `openrouter/google/gemma-7b-it:free` - `openrouter/lizpreciatior/lzlv-70b-fp16-hf` - `openrouter/mistralai/mistral-7b-instruct:nitro` - `openrouter/perplexity/sonar-medium-online` - `openrouter/recursal/rwkv-5-3b-ai-town` - `openrouter/google/gemini-pro-vision` - `openrouter/gryphe/mythomax-l2-13b` - `openrouter/sophosympatheia/midnight-rose-70b` - `openrouter/cohere/command` - `openrouter/mistralai/mixtral-8x22b-instruct` - `openrouter/cohere/command-r-plus` - `openrouter/huggingfaceh4/zephyr-7b-beta:free` - `openrouter/meta-llama/codellama-34b-instruct` - `openrouter/microsoft/wizardlm-2-7b` - `openrouter/rwkv/rwkv-5-world-3b` - `openrouter/perplexity/pplx-7b-online` - `openrouter/austism/chronos-hermes-13b` - `openrouter/anthropic/claude-2.0:beta` - `openrouter/google/palm-2-chat-bison` - `openrouter/google/palm-2-codechat-bison` - `openrouter/togethercomputer/stripedhyena-nous-7b` - `openrouter/meta-llama/llama-3-8b-instruct:nitro` - `openrouter/koboldai/psyfighter-13b-2` - `openrouter/openchat/openchat-7b` - `openrouter/openai/gpt-3.5-turbo-0613` - `openrouter/huggingfaceh4/zephyr-7b-beta` - `openrouter/xwin-lm/xwin-lm-70b` - `openrouter/mistralai/mistral-7b-instruct:free` - `openrouter/perplexity/sonar-small-online` ### "Statement Coverage Reached" Models in this category produced code that reached full statement coverage. - `openrouter/anthropic/claude-2.1` - `openrouter/perplexity/sonar-medium-chat` - `openrouter/anthropic/claude-3-sonnet:beta` - `openrouter/mistralai/mistral-medium` - `openrouter/microsoft/wizardlm-2-8x22b` - `openrouter/anthropic/claude-3-haiku:beta` - `openrouter/microsoft/wizardlm-2-8x22b:nitro` - `openrouter/anthropic/claude-2.0` - `openrouter/anthropic/claude-2.1:beta` - `openrouter/anthropic/claude-3-sonnet` - `openrouter/openai/gpt-3.5-turbo-1106` - `openrouter/openai/gpt-4-1106-preview` - `openrouter/mistralai/mixtral-8x7b-instruct:nitro` - `openrouter/meta-llama/llama-2-13b-chat` - `openrouter/anthropic/claude-3-opus:beta` - `openrouter/jondurbin/airoboros-l2-70b` - `openrouter/meta-llama/llama-2-70b-chat:nitro` - `openrouter/meta-llama/llama-3-70b-instruct:nitro` - `openrouter/perplexity/pplx-70b-chat` - `openrouter/anthropic/claude-3-haiku` - `openrouter/meta-llama/llama-2-70b-chat` - `openrouter/openai/gpt-4` - `openrouter/anthropic/claude-3-opus` - `openrouter/phind/phind-codellama-34b` - `openrouter/mistralai/mixtral-8x7b-instruct` - `openrouter/openai/gpt-4-32k` ### "No Excess Response" Models in this category did not respond with more content than requested. - `openrouter/openai/gpt-4-0314` - `openrouter/openai/gpt-4-turbo` - `openrouter/nousresearch/nous-capybara-34b` - `symflower/symbolic-execution` - `openrouter/google/gemini-pro-1.5` - `openrouter/openrouter/auto` - `openrouter/openai/gpt-4-turbo-preview` - `openrouter/openai/gpt-3.5-turbo-0125` - `openrouter/openai/gpt-3.5-turbo` - `openrouter/meta-llama/llama-3-70b-instruct` - `openrouter/01-ai/yi-34b-chat` - `openrouter/openai/gpt-3.5-turbo-0301` - `openrouter/openai/gpt-4-32k-0314` - `openrouter/google/gemini-pro` - `openrouter/openai/gpt-4-vision-preview` --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.4.0 4 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/4/README.md`) # Evaluation from 2024-04-25 19:31:23 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.4.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - Category Unknown: Models in this category could not be categorized. - Response Error: Models in this category encountered an error. - Response Empty: Models in this category produced an empty response. - No Code: Models in this category produced no code. - Invalid Code: Models in this category produced invalid code. - Executable Code: Models in this category produced executable code. - Statement Coverage Reached: Models in this category produced code that reached full statement coverage. - No Excess Response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### "Response Error" Models in this category encountered an error. - `openrouter/jebcarter/psyfighter-13b` - `openrouter/anthropic/claude-instant-1` - `openrouter/anthropic/claude-instant-1:beta` - `openrouter/nousresearch/nous-hermes-2-vision-7b` - `openrouter/haotian-liu/llava-13b` - `openrouter/jondurbin/bagel-34b` - `openrouter/anthropic/claude-2` - `openrouter/anthropic/claude-2:beta` ### "No Code" Models in this category produced no code. - `openrouter/gryphe/mythomist-7b` - `openrouter/meta-llama/llama-3-8b-instruct:extended` - `openrouter/undi95/toppy-m-7b:nitro` - `openrouter/mancer/weaver` - `openrouter/nousresearch/nous-capybara-7b:free` - `openrouter/anthropic/claude-instant-1.2` - `openrouter/google/gemini-pro` - `openrouter/anthropic/claude-1` - `openrouter/gryphe/mythomax-l2-13b:nitro` - `openrouter/gryphe/mythomax-l2-13b` - `openrouter/lynn/soliloquy-l3` - `openrouter/gryphe/mythomist-7b:free` - `openrouter/neversleep/noromaid-20b` - `openrouter/openai/gpt-3.5-turbo-instruct` - `openrouter/koboldai/psyfighter-13b-2` - `openrouter/togethercomputer/stripedhyena-hessian-7b` - `openrouter/pygmalionai/mythalion-13b` - `openrouter/mistralai/mistral-tiny` - `openrouter/undi95/toppy-m-7b` - `openrouter/undi95/remm-slerp-l2-13b` - `openrouter/anthropic/claude-instant-1.0` - `openrouter/intel/neural-chat-7b` - `openrouter/undi95/toppy-m-7b:free` - `openrouter/neversleep/noromaid-mixtral-8x7b-instruct` - `openrouter/undi95/remm-slerp-l2-13b:extended` ### "Invalid Code" Models in this category produced invalid code. - `openrouter/mistralai/mistral-large` - `openrouter/togethercomputer/stripedhyena-nous-7b` - `openrouter/gryphe/mythomax-l2-13b:extended` - `openrouter/openai/gpt-3.5-turbo-0613` - `openrouter/open-orca/mistral-7b-openorca` - `openrouter/jondurbin/airoboros-l2-70b` - `openrouter/google/gemma-7b-it:nitro` - `openrouter/meta-llama/llama-3-8b-instruct` - `openrouter/anthropic/claude-2.0` - `openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sft` - `openrouter/google/palm-2-codechat-bison-32k` - `openrouter/google/gemini-pro-vision` - `openrouter/google/palm-2-codechat-bison` - `openrouter/openai/gpt-3.5-turbo-16k` - `openrouter/perplexity/pplx-7b-online` - `openrouter/01-ai/yi-34b` - `openrouter/cognitivecomputations/dolphin-mixtral-8x7b` - `openrouter/openrouter/cinematika-7b:free` - `openrouter/alpindale/goliath-120b` - `openrouter/teknium/openhermes-2.5-mistral-7b` - `openrouter/google/gemma-7b-it` - `openrouter/mistralai/mixtral-8x7b-instruct` - `openrouter/meta-llama/llama-3-8b-instruct:nitro` - `openrouter/fireworks/firellava-13b` - `openrouter/mistralai/mistral-7b-instruct:nitro` - `openrouter/cohere/command-r` - `openrouter/nousresearch/nous-capybara-7b` - `openrouter/01-ai/yi-6b` - `openrouter/mistralai/mistral-7b-instruct:free` - `openrouter/openchat/openchat-7b` - `openrouter/xwin-lm/xwin-lm-70b` - `openrouter/openchat/openchat-7b:free` - `openrouter/perplexity/sonar-small-online` - `openrouter/phind/phind-codellama-34b` - `openrouter/mistralai/mixtral-8x22b-instruct` - `openrouter/teknium/openhermes-2-mistral-7b` - `openrouter/huggingfaceh4/zephyr-7b-beta:free` - `openrouter/microsoft/wizardlm-2-7b` - `openrouter/cohere/command` - `openrouter/openrouter/cinematika-7b` - `openrouter/rwkv/rwkv-5-world-3b` - `openrouter/cohere/command-r-plus` - `openrouter/lizpreciatior/lzlv-70b-fp16-hf` - `openrouter/austism/chronos-hermes-13b` - `openrouter/mistralai/mixtral-8x7b` - `openrouter/perplexity/pplx-70b-online` - `openrouter/meta-llama/codellama-34b-instruct` - `openrouter/nousresearch/nous-hermes-2-mistral-7b-dpo` - `openrouter/huggingfaceh4/zephyr-7b-beta` - `openrouter/meta-llama/llama-2-13b-chat` - `openrouter/nousresearch/nous-hermes-llama2-13b` - `openrouter/anthropic/claude-1.2` - `openrouter/sao10k/fimbulvetr-11b-v2` - `openrouter/nousresearch/nous-hermes-yi-34b` - `openrouter/google/palm-2-chat-bison-32k` - `openrouter/perplexity/sonar-medium-chat` - `openrouter/mistralai/mistral-small` - `openrouter/mistralai/mixtral-8x22b` - `openrouter/openai/gpt-4-32k` - `openrouter/anthropic/claude-3-haiku:beta` - `openrouter/recursal/rwkv-5-3b-ai-town` - `openrouter/anthropic/claude-2.0:beta` - `openrouter/perplexity/sonar-medium-online` - `openrouter/nousresearch/nous-capybara-34b` - `openrouter/recursal/eagle-7b` - `openrouter/sophosympatheia/midnight-rose-70b` - `openrouter/huggingfaceh4/zephyr-orpo-141b-a35b` - `openrouter/mistralai/mixtral-8x7b-instruct:nitro` - `openrouter/openai/gpt-3.5-turbo` ### "Executable Code" Models in this category produced executable code. - `openrouter/codellama/codellama-70b-instruct` - `openrouter/perplexity/pplx-7b-chat` - `openrouter/google/gemini-pro-1.5` - `openrouter/perplexity/sonar-small-chat` ### "Statement Coverage Reached" Models in this category produced code that reached full statement coverage. - `openrouter/microsoft/wizardlm-2-8x22b:nitro` - `openrouter/anthropic/claude-3-sonnet:beta` - `openrouter/anthropic/claude-2.1` - `openrouter/anthropic/claude-3-opus` - `openrouter/mistralai/mistral-medium` - `openrouter/anthropic/claude-3-opus:beta` - `openrouter/openai/gpt-3.5-turbo-1106` - `openrouter/openai/gpt-4-1106-preview` - `openrouter/01-ai/yi-34b-chat` - `openrouter/meta-llama/llama-2-70b-chat:nitro` - `openrouter/openai/gpt-3.5-turbo-0301` - `openrouter/microsoft/wizardlm-2-8x22b` - `openrouter/anthropic/claude-3-sonnet` - `openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpo` - `openrouter/anthropic/claude-2.1:beta` - `openrouter/perplexity/pplx-70b-chat` - `openrouter/anthropic/claude-instant-1.1` - `openrouter/openai/gpt-4-32k-0314` - `openrouter/anthropic/claude-3-haiku` - `openrouter/meta-llama/llama-2-70b-chat` ### "No Excess Response" Models in this category did not respond with more content than requested. - `openrouter/mistralai/mistral-7b-instruct` - `openrouter/openai/gpt-4-vision-preview` - `openrouter/databricks/dbrx-instruct` - `openrouter/openai/gpt-4-turbo` - `openrouter/openai/gpt-4-0314` - `openrouter/openai/gpt-3.5-turbo-0125` - `openrouter/openai/gpt-4-turbo-preview` - `symflower/symbolic-execution` - `openrouter/meta-llama/llama-3-70b-instruct` - `openrouter/google/gemma-7b-it:free` - `openrouter/openai/gpt-4` - `openrouter/openrouter/auto` - `openrouter/meta-llama/llama-3-70b-instruct:nitro` - `openrouter/google/palm-2-chat-bison` --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.4.0 5 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/5/README.md`) # Evaluation from 2024-04-25 19:31:35 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.4.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - Category Unknown: Models in this category could not be categorized. - Response Error: Models in this category encountered an error. - Response Empty: Models in this category produced an empty response. - No Code: Models in this category produced no code. - Invalid Code: Models in this category produced invalid code. - Executable Code: Models in this category produced executable code. - Statement Coverage Reached: Models in this category produced code that reached full statement coverage. - No Excess Response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### "Response Error" Models in this category encountered an error. - `openrouter/jebcarter/psyfighter-13b` - `openrouter/anthropic/claude-2` - `openrouter/openchat/openchat-7b:free` - `openrouter/openrouter/cinematika-7b:free` - `openrouter/jondurbin/bagel-34b` - `openrouter/anthropic/claude-instant-1` - `openrouter/haotian-liu/llava-13b` - `openrouter/undi95/toppy-m-7b:free` - `openrouter/anthropic/claude-2:beta` - `openrouter/nousresearch/nous-hermes-2-vision-7b` - `openrouter/anthropic/claude-instant-1:beta` - `openrouter/nousresearch/nous-capybara-7b:free` - `openrouter/perplexity/pplx-70b-online` - `openrouter/lynn/soliloquy-l3` ### "No Code" Models in this category produced no code. - `openrouter/neversleep/noromaid-20b` - `openrouter/koboldai/psyfighter-13b-2` - `openrouter/nousresearch/nous-capybara-7b` - `openrouter/neversleep/noromaid-mixtral-8x7b-instruct` - `openrouter/gryphe/mythomist-7b:free` - `openrouter/meta-llama/codellama-34b-instruct` - `openrouter/undi95/remm-slerp-l2-13b:extended` - `openrouter/meta-llama/llama-3-8b-instruct:extended` - `openrouter/anthropic/claude-instant-1.0` - `openrouter/openai/gpt-3.5-turbo-1106` - `openrouter/gryphe/mythomist-7b` - `openrouter/nousresearch/nous-hermes-llama2-13b` - `openrouter/teknium/openhermes-2.5-mistral-7b` - `openrouter/mistralai/mistral-tiny` - `openrouter/intel/neural-chat-7b` - `openrouter/gryphe/mythomax-l2-13b:nitro` - `openrouter/gryphe/mythomax-l2-13b:extended` - `openrouter/mistralai/mistral-7b-instruct:nitro` - `openrouter/01-ai/yi-34b` - `openrouter/google/gemini-pro` - `openrouter/togethercomputer/stripedhyena-hessian-7b` - `openrouter/cognitivecomputations/dolphin-mixtral-8x7b` - `openrouter/openai/gpt-3.5-turbo-instruct` - `openrouter/pygmalionai/mythalion-13b` - `openrouter/mancer/weaver` - `openrouter/undi95/remm-slerp-l2-13b` - `openrouter/openai/gpt-3.5-turbo-0613` ### "Invalid Code" Models in this category produced invalid code. - `openrouter/mistralai/mistral-7b-instruct:free` - `openrouter/codellama/codellama-70b-instruct` - `openrouter/microsoft/wizardlm-2-7b` - `openrouter/google/palm-2-codechat-bison` - `openrouter/open-orca/mistral-7b-openorca` - `openrouter/google/palm-2-chat-bison` - `openrouter/perplexity/pplx-7b-chat` - `openrouter/01-ai/yi-34b-chat` - `openrouter/nousresearch/nous-hermes-2-mistral-7b-dpo` - `openrouter/anthropic/claude-3-haiku:beta` - `openrouter/undi95/toppy-m-7b` - `openrouter/huggingfaceh4/zephyr-7b-beta:free` - `openrouter/xwin-lm/xwin-lm-70b` - `openrouter/01-ai/yi-6b` - `openrouter/undi95/toppy-m-7b:nitro` - `openrouter/openai/gpt-3.5-turbo-16k` - `openrouter/huggingfaceh4/zephyr-7b-beta` - `openrouter/rwkv/rwkv-5-world-3b` - `openrouter/cohere/command-r-plus` - `openrouter/anthropic/claude-2.0` - `openrouter/google/gemini-pro-vision` - `openrouter/sao10k/fimbulvetr-11b-v2` - `openrouter/sophosympatheia/midnight-rose-70b` - `openrouter/alpindale/goliath-120b` - `openrouter/perplexity/sonar-medium-online` - `openrouter/openchat/openchat-7b` - `openrouter/mistralai/mixtral-8x7b` - `openrouter/meta-llama/llama-3-8b-instruct` - `openrouter/austism/chronos-hermes-13b` - `openrouter/mistralai/mistral-large` - `openrouter/mistralai/mixtral-8x22b` - `openrouter/anthropic/claude-1.2` - `openrouter/meta-llama/llama-2-13b-chat` - `openrouter/nousresearch/nous-capybara-34b` - `openrouter/openrouter/cinematika-7b` - `openrouter/perplexity/pplx-7b-online` - `openrouter/cohere/command` - `openrouter/anthropic/claude-instant-1.1` - `openrouter/teknium/openhermes-2-mistral-7b` - `openrouter/mistralai/mixtral-8x7b-instruct:nitro` - `openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sft` - `openrouter/perplexity/sonar-small-online` - `openrouter/cohere/command-r` - `openrouter/togethercomputer/stripedhyena-nous-7b` - `openrouter/google/gemma-7b-it:nitro` - `openrouter/nousresearch/nous-hermes-yi-34b` - `openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpo` - `openrouter/recursal/rwkv-5-3b-ai-town` - `openrouter/meta-llama/llama-3-8b-instruct:nitro` - `openrouter/perplexity/pplx-70b-chat` - `openrouter/fireworks/firellava-13b` - `openrouter/jondurbin/airoboros-l2-70b` - `openrouter/google/gemma-7b-it:free` - `openrouter/anthropic/claude-2.0:beta` - `openrouter/google/palm-2-codechat-bison-32k` - `openrouter/google/gemma-7b-it` - `openrouter/recursal/eagle-7b` - `openrouter/perplexity/sonar-small-chat` - `openrouter/google/palm-2-chat-bison-32k` ### "Executable Code" Models in this category produced executable code. - `openrouter/mistralai/mistral-7b-instruct` ### "Statement Coverage Reached" Models in this category produced code that reached full statement coverage. - `openrouter/anthropic/claude-2.1:beta` - `openrouter/openai/gpt-4-32k` - `openrouter/microsoft/wizardlm-2-8x22b:nitro` - `openrouter/anthropic/claude-3-sonnet` - `openrouter/openai/gpt-3.5-turbo-0125` - `openrouter/lizpreciatior/lzlv-70b-fp16-hf` - `openrouter/databricks/dbrx-instruct` - `openrouter/meta-llama/llama-2-70b-chat:nitro` - `openrouter/openai/gpt-4-1106-preview` - `openrouter/mistralai/mistral-medium` - `openrouter/mistralai/mixtral-8x7b-instruct` - `openrouter/phind/phind-codellama-34b` - `openrouter/perplexity/sonar-medium-chat` - `openrouter/anthropic/claude-3-opus` - `openrouter/anthropic/claude-1` - `openrouter/anthropic/claude-2.1` - `openrouter/mistralai/mistral-small` - `openrouter/meta-llama/llama-3-70b-instruct:nitro` - `openrouter/anthropic/claude-3-opus:beta` - `openrouter/gryphe/mythomax-l2-13b` - `openrouter/microsoft/wizardlm-2-8x22b` - `openrouter/anthropic/claude-3-sonnet:beta` - `openrouter/anthropic/claude-3-haiku` - `openrouter/mistralai/mixtral-8x22b-instruct` - `openrouter/meta-llama/llama-3-70b-instruct` - `openrouter/meta-llama/llama-2-70b-chat` - `openrouter/huggingfaceh4/zephyr-orpo-141b-a35b` ### "No Excess Response" Models in this category did not respond with more content than requested. - `openrouter/openai/gpt-4-turbo` - `openrouter/openai/gpt-3.5-turbo` - `openrouter/google/gemini-pro-1.5` - `symflower/symbolic-execution` - `openrouter/openai/gpt-4-turbo-preview` - `openrouter/openai/gpt-4-32k-0314` - `openrouter/openai/gpt-4` - `openrouter/openai/gpt-3.5-turbo-0301` - `openrouter/openai/gpt-4-vision-preview` - `openrouter/openai/gpt-4-0314` - `openrouter/anthropic/claude-instant-1.2` - `openrouter/openrouter/auto` --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Chronos Hermes 13b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/chronos-hermes-13b/README.md`) # Evaluation from 2024-06-19 10:00:20 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/austism/chronos-hermes-13b`](./openrouter_austism_chronos-hermes-13b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 1.2 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-1.2/README.md`) # Evaluation from 2024-06-24 11:19:06 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/anthropic/claude-1.2`](./openrouter_anthropic_claude-1.2/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 2.0 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-2.0/README.md`) # Evaluation from 2024-06-24 11:19:15 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/anthropic/claude-2.0`](./openrouter_anthropic_claude-2.0/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 2.1 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-2.1/README.md`) # Evaluation from 2024-06-21 09:52:06 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/anthropic/claude-2.1`](./openrouter_anthropic_claude-2.1/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 3 Haiku README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-3-haiku/README.md`) # Evaluation from 2024-06-19 08:44:31 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/anthropic/claude-3-haiku`](./openrouter_anthropic_claude-3-haiku/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 3 Opus README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-3-opus/README.md`) # Evaluation from 2024-06-19 09:59:38 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/anthropic/claude-3-opus`](./openrouter_anthropic_claude-3-opus/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 3 Sonnet README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-3-sonnet/README.md`) # Evaluation from 2024-06-19 09:59:44 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/anthropic/claude-3-sonnet`](./openrouter_anthropic_claude-3-sonnet/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 3.5 Sonnet README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-3.5-sonnet/README.md`) # Evaluation from 2024-06-20 15:04:46 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/anthropic/claude-3.5-sonnet`](./openrouter_anthropic_claude-3.5-sonnet/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude Instant 1.0 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-instant-1.0/README.md`) # Evaluation from 2024-06-24 11:18:54 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/anthropic/claude-instant-1.0`](./openrouter_anthropic_claude-instant-1.0/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude Instant 1.1 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-instant-1.1/README.md`) # Evaluation from 2024-06-19 09:09:19 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/anthropic/claude-instant-1.1`](./openrouter_anthropic_claude-instant-1.1/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Codellama 34b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/codellama-34b-instruct/README.md`) # Evaluation from 2024-06-19 11:21:09 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/meta-llama/codellama-34b-instruct`](./openrouter_meta-llama_codellama-34b-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Codeqwen README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/codeqwen/README.md`) # Evaluation from 2024-06-21 06:54:57 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`ollama/codeqwen:latest`](./ollama_codeqwen:latest/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Codestral README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/codestral/README.md`) # Evaluation from 2024-06-20 09:05:20 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`custom-codestral/codestral-latest`](./custom-codestral_codestral-latest/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Command R Plus README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/command-r-plus/README.md`) # Evaluation from 2024-06-19 10:00:24 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/cohere/command-r-plus`](./openrouter_cohere_command-r-plus/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Dbrx Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/dbrx-instruct/README.md`) # Evaluation from 2024-06-19 10:00:27 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/databricks/dbrx-instruct`](./openrouter_databricks_dbrx-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Deepseek Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/deepseek-chat/README.md`) # Evaluation from 2024-06-19 10:00:30 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/deepseek/deepseek-chat`](./openrouter_deepseek_deepseek-chat/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Deepseek Coder V2 16b Lite Instruct Fp16 54a969781028 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/deepseek-coder-v2-16b-lite-instruct-fp16-54a969781028/README.md`) # Evaluation from 2024-06-24 11:35:07 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`ollama/deepseek-coder-v2:16b-lite-instruct-fp16`](./ollama_deepseek-coder-v2:16b-lite-instruct-fp16/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Deepseek Coder V2 16b Lite Instruct Q8 0 44250301ba51 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/deepseek-coder-v2-16b-lite-instruct-q8_0-44250301ba51/README.md`) # Evaluation from 2024-06-24 14:19:10 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`ollama/deepseek-coder-v2:16b-lite-instruct-q8_0`](./ollama_deepseek-coder-v2:16b-lite-instruct-q8_0/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Deepseek Coder README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/deepseek-coder/README.md`) # Evaluation from 2024-06-19 10:00:32 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/deepseek/deepseek-coder`](./openrouter_deepseek_deepseek-coder/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Dolphin Mixtral 8x22b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/dolphin-mixtral-8x22b/README.md`) # Evaluation from 2024-06-19 10:00:21 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/cognitivecomputations/dolphin-mixtral-8x22b`](./openrouter_cognitivecomputations_dolphin-mixtral-8x22b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Dolphin Mixtral 8x7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/dolphin-mixtral-8x7b/README.md`) # Evaluation from 2024-06-19 10:00:21 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/cognitivecomputations/dolphin-mixtral-8x7b`](./openrouter_cognitivecomputations_dolphin-mixtral-8x7b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Eagle 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/eagle-7b/README.md`) # Evaluation from 2024-06-19 11:29:12 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/recursal/eagle-7b`](./openrouter_recursal_eagle-7b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Falcon2 11b Q8 0 2d49820d6bb5 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/falcon2-11b-q8_0-2d49820d6bb5/README.md`) # Evaluation from 2024-06-24 09:05:25 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`ollama/falcon2:11b-q8_0`](./ollama_falcon2:11b-q8_0/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Firellava 13b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/firellava-13b/README.md`) # Evaluation from 2024-06-19 10:00:35 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/fireworks/firellava-13b`](./openrouter_fireworks_firellava-13b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gemini Flash 1.5 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gemini-flash-1.5/README.md`) # Evaluation from 2024-06-19 10:01:09 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/google/gemini-flash-1.5`](./openrouter_google_gemini-flash-1.5/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gemini Pro 1.5 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gemini-pro-1.5/README.md`) # Evaluation from 2024-06-19 08:43:07 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/google/gemini-pro-1.5`](./openrouter_google_gemini-pro-1.5/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gemma 2 27b It README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gemma-2-27b-it/README.md`) # Evaluation from 2024-07-05 06:55:06 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. **REMARK: `gemma-2-9b-it` and `gemma-2-27-it` were originally evaluated together with the results then being split into separate folders. Therefore some logs might contain entries from "the other" gemma model.** ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`custom-nvidia/google/gemma-2-27b-it`](./custom-nvidia_google_gemma-2-27b-it/) - [`custom-nvidia/google/gemma-2-9b-it`](./custom-nvidia_google_gemma-2-9b-it/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gemma 2 9b It README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gemma-2-9b-it/README.md`) # Evaluation from 2024-07-05 06:55:06 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. **REMARK: `gemma-2-9b-it` and `gemma-2-27-it` were originally evaluated together with the results then being split into separate folders. Therefore some logs might contain entries from "the other" gemma model.** ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`custom-nvidia/google/gemma-2-27b-it`](./custom-nvidia_google_gemma-2-27b-it/) - [`custom-nvidia/google/gemma-2-9b-it`](./custom-nvidia_google_gemma-2-9b-it/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gemma 7b It README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gemma-7b-it/README.md`) # Evaluation from 2024-06-19 10:01:10 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/google/gemma-7b-it`](./openrouter_google_gemma-7b-it/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Goliath 120b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/goliath-120b/README.md`) # Evaluation from 2024-06-19 08:43:56 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/alpindale/goliath-120b`](./openrouter_alpindale_goliath-120b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gpt 4 Turbo README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gpt-4-turbo/README.md`) # Evaluation from 2024-06-19 11:09:05 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/openai/gpt-4-turbo`](./openrouter_openai_gpt-4-turbo/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gpt 4 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gpt-4/README.md`) # Evaluation from 2024-06-19 11:08:40 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/openai/gpt-4`](./openrouter_openai_gpt-4/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gpt 4o README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gpt-4o/README.md`) # Evaluation from 2024-06-19 09:57:48 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/openai/gpt-4o`](./openrouter_openai_gpt-4o/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Granite Code 34b Instruct F16 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/granite-code-34b-instruct-f16/README.md`) # Evaluation from 2024-06-21 11:13:38 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`ollama/granite-code:34b-instruct-f16`](./ollama_granite-code:34b-instruct-f16/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Granite Code 34b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/granite-code-34b/README.md`) # Evaluation from 2024-06-20 12:20:38 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`ollama/granite-code:34b`](./ollama_granite-code:34b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Granite Code 3b Instruct Q8 0 B93b1a15197d README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/granite-code-3b-instruct-q8_0-b93b1a15197d/README.md`) # Evaluation from 2024-06-24 14:53:48 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`ollama/granite-code:3b-instruct-q8_0`](./ollama_granite-code:3b-instruct-q8_0/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Hermes 2 Pro Llama 3 8b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/hermes-2-pro-llama-3-8b/README.md`) # Evaluation from 2024-06-19 10:37:04 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/nousresearch/hermes-2-pro-llama-3-8b`](./openrouter_nousresearch_hermes-2-pro-llama-3-8b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llama 3 70b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llama-3-70b-instruct/README.md`) # Evaluation from 2024-06-19 10:01:56 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/meta-llama/llama-3-70b-instruct`](./openrouter_meta-llama_llama-3-70b-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llama 3 8b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llama-3-8b-instruct/README.md`) # Evaluation from 2024-06-19 10:01:57 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/meta-llama/llama-3-8b-instruct`](./openrouter_meta-llama_llama-3-8b-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llama 3 Lumimaid 70b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llama-3-lumimaid-70b/README.md`) # Evaluation from 2024-06-19 10:28:10 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/neversleep/llama-3-lumimaid-70b`](./openrouter_neversleep_llama-3-lumimaid-70b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llama 3 Lumimaid 8b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llama-3-lumimaid-8b/README.md`) # Evaluation from 2024-06-19 10:28:10 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/neversleep/llama-3-lumimaid-8b`](./openrouter_neversleep_llama-3-lumimaid-8b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llama 3 Sonar Large 32k Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llama-3-sonar-large-32k-chat/README.md`) # Evaluation from 2024-06-19 11:14:50 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/perplexity/llama-3-sonar-large-32k-chat`](./openrouter_perplexity_llama-3-sonar-large-32k-chat/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llama 3 Sonar Small 32k Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llama-3-sonar-small-32k-chat/README.md`) # Evaluation from 2024-06-19 10:39:07 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/perplexity/llama-3-sonar-small-32k-chat`](./openrouter_perplexity_llama-3-sonar-small-32k-chat/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llava 13b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llava-13b/README.md`) # Evaluation from 2024-06-19 10:01:10 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/liuhaotian/llava-13b`](./openrouter_liuhaotian_llava-13b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llava Yi 34b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llava-yi-34b/README.md`) # Evaluation from 2024-06-19 10:01:19 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/liuhaotian/llava-yi-34b`](./openrouter_liuhaotian_llava-yi-34b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mistral 7b Instruct V0.3 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mistral-7b-instruct-v0.3/README.md`) # Evaluation from 2024-06-19 10:27:04 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/mistralai/mistral-7b-instruct-v0.3`](./openrouter_mistralai_mistral-7b-instruct-v0.3/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mistral 7b Openorca README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mistral-7b-openorca/README.md`) # Evaluation from 2024-06-19 11:08:37 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/open-orca/mistral-7b-openorca`](./openrouter_open-orca_mistral-7b-openorca/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mistral Large README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mistral-large/README.md`) # Evaluation from 2024-06-19 08:40:58 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/mistralai/mistral-large`](./openrouter_mistralai_mistral-large/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mistral Medium README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mistral-medium/README.md`) # Evaluation from 2024-06-19 10:27:12 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/mistralai/mistral-medium`](./openrouter_mistralai_mistral-medium/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mistral Small README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mistral-small/README.md`) # Evaluation from 2024-06-19 10:27:18 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/mistralai/mistral-small`](./openrouter_mistralai_mistral-small/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mistral Tiny README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mistral-tiny/README.md`) # Evaluation from 2024-06-19 10:27:32 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/mistralai/mistral-tiny`](./openrouter_mistralai_mistral-tiny/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mixtral 8x22b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mixtral-8x22b-instruct/README.md`) # Evaluation from 2024-06-19 10:27:42 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/mistralai/mixtral-8x22b-instruct`](./openrouter_mistralai_mixtral-8x22b-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mixtral 8x7b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mixtral-8x7b-instruct/README.md`) # Evaluation from 2024-06-19 10:27:44 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/mistralai/mixtral-8x7b-instruct`](./openrouter_mistralai_mixtral-8x7b-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Neural Chat 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/neural-chat-7b/README.md`) # Evaluation from 2024-06-19 10:01:09 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/intel/neural-chat-7b`](./openrouter_intel_neural-chat-7b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Nous Capybara 34b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/nous-capybara-34b/README.md`) # Evaluation from 2024-06-19 10:37:07 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/nousresearch/nous-capybara-34b`](./openrouter_nousresearch_nous-capybara-34b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Nous Capybara 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/nous-capybara-7b/README.md`) # Evaluation from 2024-06-19 10:47:13 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/nousresearch/nous-capybara-7b`](./openrouter_nousresearch_nous-capybara-7b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Nous Hermes 2 Mistral 7b Dpo README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/nous-hermes-2-mistral-7b-dpo/README.md`) # Evaluation from 2024-06-19 11:08:27 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/nousresearch/nous-hermes-2-mistral-7b-dpo`](./openrouter_nousresearch_nous-hermes-2-mistral-7b-dpo/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Nous Hermes 2 Mixtral 8x7b Dpo README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/nous-hermes-2-mixtral-8x7b-dpo/README.md`) # Evaluation from 2024-06-19 11:08:30 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpo`](./openrouter_nousresearch_nous-hermes-2-mixtral-8x7b-dpo/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Nous Hermes 2 Mixtral 8x7b Sft README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/nous-hermes-2-mixtral-8x7b-sft/README.md`) # Evaluation from 2024-06-19 11:08:32 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sft`](./openrouter_nousresearch_nous-hermes-2-mixtral-8x7b-sft/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Nous Hermes Yi 34b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/nous-hermes-yi-34b/README.md`) # Evaluation from 2024-06-19 11:08:35 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/nousresearch/nous-hermes-yi-34b`](./openrouter_nousresearch_nous-hermes-yi-34b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Olmo 7b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/olmo-7b-instruct/README.md`) # Evaluation from 2024-06-19 08:43:44 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/allenai/olmo-7b-instruct`](./openrouter_allenai_olmo-7b-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Openchat 8b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/openchat-8b/README.md`) # Evaluation from 2024-06-19 11:14:47 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/openchat/openchat-8b`](./openrouter_openchat_openchat-8b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Openhermes 2 Mistral 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/openhermes-2-mistral-7b/README.md`) # Evaluation from 2024-06-19 11:09:44 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/teknium/openhermes-2-mistral-7b`](./openrouter_teknium_openhermes-2-mistral-7b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Openhermes 2.5 Mistral 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/openhermes-2.5-mistral-7b/README.md`) # Evaluation from 2024-06-19 11:20:58 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/teknium/openhermes-2.5-mistral-7b`](./openrouter_teknium_openhermes-2.5-mistral-7b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Palm 2 Chat Bison README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/palm-2-chat-bison/README.md`) # Evaluation from 2024-06-19 10:01:10 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/google/palm-2-chat-bison`](./openrouter_google_palm-2-chat-bison/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Palm 2 Codechat Bison README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/palm-2-codechat-bison/README.md`) # Evaluation from 2024-06-19 10:01:10 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/google/palm-2-codechat-bison`](./openrouter_google_palm-2-codechat-bison/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Phi 3 Medium 128k Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/phi-3-medium-128k-instruct/README.md`) # Evaluation from 2024-06-19 10:11:27 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/microsoft/phi-3-medium-128k-instruct`](./openrouter_microsoft_phi-3-medium-128k-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Phi 3 Medium 4k Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/phi-3-medium-4k-instruct/README.md`) # Evaluation from 2024-06-19 10:11:31 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/microsoft/phi-3-medium-4k-instruct`](./openrouter_microsoft_phi-3-medium-4k-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Phi 3 Mini 128k Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/phi-3-mini-128k-instruct/README.md`) # Evaluation from 2024-06-19 10:19:41 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/microsoft/phi-3-mini-128k-instruct`](./openrouter_microsoft_phi-3-mini-128k-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Phind Codellama 34b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/phind-codellama-34b/README.md`) # Evaluation from 2024-06-19 10:39:01 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/phind/phind-codellama-34b`](./openrouter_phind_phind-codellama-34b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 110b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-110b-chat/README.md`) # Evaluation from 2024-06-19 11:27:23 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/qwen/qwen-110b-chat`](./openrouter_qwen_qwen-110b-chat/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 14b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-14b-chat/README.md`) # Evaluation from 2024-06-19 11:27:44 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/qwen/qwen-14b-chat`](./openrouter_qwen_qwen-14b-chat/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 2 72b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-2-72b-instruct/README.md`) # Evaluation from 2024-06-19 09:58:19 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/qwen/qwen-2-72b-instruct`](./openrouter_qwen_qwen-2-72b-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 32b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-32b-chat/README.md`) # Evaluation from 2024-06-19 11:28:04 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/qwen/qwen-32b-chat`](./openrouter_qwen_qwen-32b-chat/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 4b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-4b-chat/README.md`) # Evaluation from 2024-06-19 11:28:14 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/qwen/qwen-4b-chat`](./openrouter_qwen_qwen-4b-chat/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 72b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-72b-chat/README.md`) # Evaluation from 2024-06-19 11:28:49 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/qwen/qwen-72b-chat`](./openrouter_qwen_qwen-72b-chat/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 7b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-7b-chat/README.md`) # Evaluation from 2024-06-19 11:28:59 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/qwen/qwen-7b-chat`](./openrouter_qwen_qwen-7b-chat/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Rwkv 5 3b Ai Town README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/rwkv-5-3b-ai-town/README.md`) # Evaluation from 2024-06-19 11:29:26 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/recursal/rwkv-5-3b-ai-town`](./openrouter_recursal_rwkv-5-3b-ai-town/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Rwkv 5 World 3b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/rwkv-5-world-3b/README.md`) # Evaluation from 2024-06-19 11:29:39 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/rwkv/rwkv-5-world-3b`](./openrouter_rwkv_rwkv-5-world-3b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Snowflake Arctic Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/snowflake-arctic-instruct/README.md`) # Evaluation from 2024-06-19 10:21:30 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/snowflake/snowflake-arctic-instruct`](./openrouter_snowflake_snowflake-arctic-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Starcoder2 15b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/starcoder2-15b-instruct/README.md`) # Evaluation from 2024-06-19 09:12:15 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/bigcode/starcoder2-15b-instruct`](./openrouter_bigcode_starcoder2-15b-instruct/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Stripedhyena Hessian 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/stripedhyena-hessian-7b/README.md`) # Evaluation from 2024-06-19 11:18:23 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/togethercomputer/stripedhyena-hessian-7b`](./openrouter_togethercomputer_stripedhyena-hessian-7b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Stripedhyena Nous 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/stripedhyena-nous-7b/README.md`) # Evaluation from 2024-06-19 10:52:48 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/togethercomputer/stripedhyena-nous-7b`](./openrouter_togethercomputer_stripedhyena-nous-7b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Symbolic Execution README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/symbolic-execution/README.md`) # Evaluation from 2024-06-19 13:25:26 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. **PLEASE NOTE that we assume the results of the "symbolic execution" are deterministic. Hence they where only benchmarked over a single run and multiplied by `5` to be comparable to the remaining contestants.** ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`symflower/symbolic-execution`](./symflower_symbolic-execution/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Toppy M 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/toppy-m-7b/README.md`) # Evaluation from 2024-06-19 10:52:54 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/undi95/toppy-m-7b`](./openrouter_undi95_toppy-m-7b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Wizardlm 2 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/wizardlm-2-7b/README.md`) # Evaluation from 2024-06-19 10:23:07 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/microsoft/wizardlm-2-7b`](./openrouter_microsoft_wizardlm-2-7b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Wizardlm 2 8x22b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/wizardlm-2-8x22b/README.md`) # Evaluation from 2024-06-19 10:23:10 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/microsoft/wizardlm-2-8x22b`](./openrouter_microsoft_wizardlm-2-8x22b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Xwin Lm 70b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/xwin-lm-70b/README.md`) # Evaluation from 2024-06-19 10:26:13 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/xwin-lm/xwin-lm-70b`](./openrouter_xwin-lm_xwin-lm-70b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Yi 34b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/yi-34b-chat/README.md`) # Evaluation from 2024-06-19 10:15:10 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/01-ai/yi-34b-chat`](./openrouter_01-ai_yi-34b-chat/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Yi 6b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/yi-6b/README.md`) # Evaluation from 2024-06-19 08:43:03 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/01-ai/yi-6b`](./openrouter_01-ai_yi-6b/) --- ### Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Zephyr 7b Beta README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/zephyr-7b-beta/README.md`) # Evaluation from 2024-06-19 10:01:10 This report was generated by [DevQualityEval benchmark](https://github.com/symflower/eval-dev-quality) in `version 0.5.0`. ## Results > Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot. The results of all models have been divided into the following categories: - category unknown: Models in this category could not be categorized. - response error: Models in this category encountered an error. - no code: Models in this category produced no code. - invalid code: Models in this category produced invalid code. - executable code: Models in this category produced executable code. - statement coverage reached: Models in this category produced code that reached full statement coverage. - no excess response: Models in this category did not respond with more content than requested. The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found [here](./evaluation.log). Detailed scoring can be found [here](./evaluation.csv). ### Result category "category unknown" Models in this category could not be categorized. - [`openrouter/huggingfaceh4/zephyr-7b-beta`](./openrouter_huggingfaceh4_zephyr-7b-beta/) --- --- METRICS --- - Files Extracted: 101 - Estimated Token Budget: ~50706 tokens - Canonical Reference: https://codewiki.google/github.com/QwenLM/Qwen3-Coder