Qwen3-Coder Documentation

Qwen3-Coder is the code version of Qwen3, the large language model series developed by Qwen team.

16,831 stars Python Python 101 files · ~50,787 tokens
RAW Doc

Technical Documentation: QwenLM/Qwen3-Coder

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md)

Visit our Hugging Face or ModelScope organization (click links above), search checkpoints with names starting with Qwen3-Coder-, and you will find all you need! Enjoy!

Table of Contents

Qwen3-Coder-Next: Pushing Small Hybrid Models on Agentic Coding

Introduction

We are announcing Qwen3-Coder, our most agentic code model to date. Qwen3-Coder is available in multiple sizes, Qwen3-Coder-480B-A35B-Instruct, Qwen3-Coder-30B-A3B-Instruct, Qwen3-Coder-Next, offering exceptional performance in both coding and agentic tasks.

Qwen3-Coder-Next, an open-weight language model designed specifically for coding agents and local development. Built on top of Qwen3-Next-80B-A3B-Base, which adopts a novel architecture with hybrid attention and MoE, Qwen3-Coder-Next has been agentically trained at scale on large-scale executable task synthesis, environment interaction, and reinforcement learning, obtaining strong coding and agentic capabilities with significantly lower inference costs.

Key Features

💻 Efficiency-Performance Tradeoff: among open models on Agentic Coding, Agentic Browser-Use, and other foundational coding tasks, achieving results comparable to Claude Sonnet.

🛠 Scaling Agentic Coding: supporting most platforms such as Qwen Code, CLINE, Claude Code, featuring a specially designed function call format;

📚 Long-context Capabilities: with native support for 256K tokens, extendable up to 1M tokens using Yarn, optimized for repository-scale understanding.

Basic Information

  1. ✨ Supporting long context understanding and generation with the context length of 256K tokens;
  2. ✨ Supporting 358 coding languages;

Click to view all supported languages

text
/* Detailed source-code truncated for AI context efficiency. */
  1. ✨ Retain strengths in math and general capabilities from base model.

[!Important]

Qwen3-Coder function calling relies on our new tool parser in both SGLang and vLLM here.

We updated both the special tokens and their corresponding token ids, in order to maintain consistency with Qwen3. Please make sure to use the new tokenizer.

model name type length Download
Qwen3-Coder-Next instruct 256k 🤗 Hugging Face • 🤖 ModelScope
Qwen3-Coder-Next-Base base 256k 🤗 Hugging Face • 🤖 ModelScope
Qwen3-Coder-480B-A35B-Instruct instruct 256k 🤗 Hugging Face • 🤖 ModelScope
Qwen3-Coder-30B-A3B-Instruct instruct 256k 🤗 Hugging Face • 🤖 ModelScope
Qwen3-Coder-Next-FP8 instruct 256k 🤗 Hugging Face • 🤖 ModelScope
Qwen3-Coder-Next-GGUF instruct 256k 🤗 Hugging Face • 🤖 ModelScope
Qwen3-Coder-480B-A35B-Instruct-FP8 instruct 256k 🤗 Hugging Face • 🤖 ModelScope
Qwen3-Coder-30B-A3B-Instruct-FP8 instruct 256k 🤗 Hugging Face • 🤖 ModelScope

Detailed performance and introduction are shown in this 📑 blog.

Quick Start

[!Important]
Qwen3-Coder are instruct models for chatting;

This model supports only non-thinking mode and does not generate `` blocks in its output. Meanwhile, specifying enable_thinking=False is no longer required.

👉🏻 Chat with Qwen3-Coder

You can write several lines of code with transformers to chat with Qwen3-Coder-Next. Essentially, we build the tokenizer and the model with the from_pretrained method, and we use the generate method to perform chatting with the help of the chat template provided by the tokenizer. Below is an example of how to chat with Qwen3-Coder-Next:

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-Coder-Next"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "write a quick sort algorithm."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=65536
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

The apply_chat_template() function is used to convert the messages into a format that the model can understand.
The add_generation_prompt argument is used to add a generation prompt, which refers to <|im_start|>assistant\n to the input. Notably, we apply the ChatML template for chat models following our previous practice.
The max_new_tokens argument is used to set the maximum length of the response. The tokenizer.batch_decode() function is used to decode the response. In terms of the input, the above messages are an example to show how to format your dialog history and system prompt.
You can use the other sizes of instruct models in the same way.

Fill in the middle with Qwen3-Coder

The code insertion task, also referred to as the "fill-in-the-middle" challenge, requires the insertion of code segments in a manner that bridges the gaps within a given code context. For an approach aligned with best practices, we recommend adhering to the formatting guidelines outlined in the paper "Efficient Training of Language Models to Fill in the Middle" [arxiv].

[!Important]
It should be noted that FIM is supported in every version of Qwen3-Coder. Qwen3-Coder-Next is shown here as an example.

The prompt should be structured as follows:

python
prompt = '<|fim_prefix|>' + prefix_code + '<|fim_suffix|>' + suffix_code + '<|fim_middle|>'

Following the approach mentioned, an example would be structured in this manner:

python
from transformers import AutoTokenizer, AutoModelForCausalLM
# load model
device = "cuda" # the device to load the model onto

TOKENIZER = AutoTokenizer.from_pretrained("Qwen/Qwen3-Coder-Next")
MODEL = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-Coder-Next", device_map="auto").eval()


input_text = """<|fim_prefix|>def quicksort(arr):
    if len(arr) <= 1:
        return arr
    pivot = arr[len(arr) // 2]
    <|fim_suffix|>
    middle = [x for x in arr if x == pivot]
    right = [x for x in arr if x > pivot]
    return quicksort(left) + middle + quicksort(right)<|fim_middle|>"""
            
messages = [
    {"role": "system", "content": "You are a code completion assistant."},
    {"role": "user", "content": input_text}
]


text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = TOKENIZER([text], return_tensors="pt").to(model.device)

# Use `max_new_tokens` to control the maximum output length.
eos_token_ids = [151659, 151661, 151662, 151663, 151664, 151643, 151645]
generated_ids = MODEL.generate(model_inputs.input_ids, max_new_tokens=512, do_sample=False, eos_token_id=eos_token_ids)[0]
# The generated_ids include prompt_ids, we only need to decode the tokens after prompt_ids.
output_text = TOKENIZER.decode(generated_ids[len(model_inputs.input_ids[0]):], skip_special_tokens=True)

print(f"Prompt: {input_text}\n\nGenerated text: {output_text}")

Use Cases

Example: Releasing a Website

Prompt with OpenClaw

text
next week we will release new coder model, can you collect the history of qwen coder and write a web page, the release the website with the nginx, you can seach how to do this in alibaba cloud linux first

Example: Desktop Tidy

Prompt with Qwen Code

text
Please tidy up my desk.

Example: Zombies vs. Plants

Prompt with Claude Code

text
帮我实现《僵尸大战植物》网页游戏

【核心机制】
- 反向塔防：玩家扮演僵尸方，从地图右侧（部署区）召唤僵尸向左进攻
- 资源循环：初始300脑子点数，僵尸吃掉植物返还100点，形成经济循环
- 倒计时制：120秒内清除所有植物获胜，超时失败

【地图规格】
- 5行9列网格，右侧3列为可部署区域（红色高亮标识）
- 每格100x100像素，草地纹理交替渲染
- 坐标系：左侧为植物防线，右侧为僵尸出生点

【单位系统】
僵尸方（右侧购买）：
- 普通僵尸：50脑，100HP，0.5速，标准单位
- 路障僵尸：100脑，200HP，0.5速，中期肉盾  
- 铁桶僵尸：150脑，400HP，0.3速，重型坦克
- 冲刺僵尸：80脑，80HP，1.2速，快速突进

植物方（左侧随机初始部署12个）：
- 豌豆射手：100HP，20伤/发，2秒间隔，直线射击
- 双发射手：120HP，20伤/发，1秒间隔，火力压制
- 坚果墙：300HP，0伤，纯肉盾阻挡
- 向日葵：80HP，0伤，经济单位（纯干扰）

【战斗逻辑】
- 碰撞检测：僵尸到达植物50px内触发啃食状态，停止移动
- 伤害结算：僵尸30帧/次咬击（0.5秒），植物射出弹道物理
- 击毁反馈：植物死亡时生成"+100"飘字特效与粒子爆炸
- 路径AI：同格僵尸队列不重叠，植物优先攻击横向最近目标

【交互设计】
- 右侧卡片式UI：显示僵尸图标、名称、脑子消耗
- 资源不足时卡片置灰并自动切换可选类型
- 鼠标悬停部署区显示半透明预览圈
- 实时血条：实体头顶显示绿/黄/红三色血槽

【胜利条件】
- 胜利：plants.length === 0 && timeLeft > 0
- 失败：timeLeft === 0 || (可选)僵尸全灭且脑子为0

Example: Sound ASCII Art

Prompt with Cline

text
Build an interactive ASCII art drawing tool with sound feedback. The application should:
 
1. Create a canvas where users can draw by clicking and dragging
2. Place different ASCII characters or symbols when the user draws
3. Play corresponding musical notes when each character is placed
4. Include multiple pattern sets with different characters and
corresponding note scales
5. Add a pattern switcher button to cycle through different
character/sound themes
6. Include a clear button to reset the canvas
7. Support both mouse and touch input for mobile compatibility
 
The application should be creative and fun to use, creating an audio-visual experience where patterns of characters create both visual art and musical patterns. Ensure the musical notes are harmonious when played in sequence.

Example: Vibe Checking

Prompt with Browser Use Agent

text
Vibe test this website. Click around, try things, report what's broken.

Example: Parkour Game

Prompt with Qwen Chat Web Dev

text
Create an interactive real-time particle system using HTML5 Canvas:

Core Features:
- Render 800-1200 animated particles with physics-based movement
- Mouse cursor exerts attractive/repulsive force on nearby particles
- Click to toggle between attraction and repulsion modes
- Particles respond with smooth acceleration and velocity calculations

Technical Requirements:
- Use requestAnimationFrame for optimal performance
- Implement force calculation based on distance from cursor
- Add visual feedback: particle glow, color variation, and fade effects
- Include performance monitoring (FPS counter)

Deliverables:
- Single HTML file with embedded CSS and JavaScript
- Clean, commented code following best practices
- Responsive design compatible with modern browsers

Star History

Citation

If you find our work helpful, feel free to give us a cite.

bibtex
@article{Qwen3-Coder-Next,
    title={Qwen3-Coder-Next Technical Report},
    author={Ruisheng Cao and Mouxiang Chen and Jiawei Chen and Zeyu Cui and Yunlong Feng and Binyuan Hui and Yuheng Jing and Kaixin Li and Mingze Li and Junyang Lin and Zeyao Ma and Kashun Shum and Xuwu Wang and Jinxi Wei and Jiaxi Yang and Jiajun Zhang and Lei Zhang and Zongmeng Zhang and Wenting Zhao and Fan Zhou},
    journal={arXiv preprint arXiv:2603.00729},
    year={2026},
}

Contact Us

If you are interested to leave a message to either our research team or product team, join our Discord or WeChat groups!

text
<a href="#readme-top" style="text-decoration: none; color: #007bff; font-weight: bold;">
    ↑ Back to Top ↑
</a>

Qwencoder Eval Instruct Eval Dev Quality Docs Kubernetes README (`qwencoder-eval/instruct/eval-dev-quality/docs/kubernetes/README.md`)

Running "eval-dev-quality" on Kubernetes

Prerequisite Checklist

  • kubectl installed and configured with authentication to the cluster.
  • Dedicated Namespace to run the jobs.
  • RWX volume to store the evaluation results (check volume.yml for inspiration).

Defining secrets

The Job automatically references to a secret called evaluation-secret which is configured to pass the values on as environment Variables. The following key is required to be created.

  • PROVIDER_TOKEN - contains the API tokens for different providers. E.g.: openrouter:abcdefgh1234,custom-provider:abcdefgh1234
bash
kubectl --namespace eval-dev-quality create secret generic evaluation-secret --from-literal='PROVIDER_TOKEN='

Running multiple evaluations with the eval-dev-quality kubernetes runtime

  • Define all the models with --model which should be run inside the containerized workload.
  • Define the parameter --runtime kubernetes to indicate that jobs should run inside a kubernetes cluster.
  • Define the parameter --parallel 20 to indicate how many jobs should run simultaneously.

Example:

bash
eval-dev-quality evaluate --runtime kubernetes --runs 5 --model symflower/symbolic-execution --model symflower/symbolic-execution --model symflower/symbolic-execution --repository golang/plain --parallel 2

This commands run 3x the symflower/symbolic-execution model with 5 runs of each model inside a containerized workload on the kubernetes cluster, it will limit the parallel execution to 2 containers at the same time.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.4.0 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/README.md`)

Note that for the official blog post the weighting of the result computation was adjusted to award responses that lead to 100% code coverage with 10 instead of 1 point score!

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.6 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.6/README.md`)

This evaluation ran over several days and is actually assembled from several sub-evaluations. Hence, we only provide the csv results for now because the logs are distributed all over.

The benchmark consisted of 5x runs, except for the code-repair task, which contains only one usable run because of a bug that is already fixed for later versions.

Qwencoder Eval Instruct Aider Aider Website Docs Leaderboards Index (`qwencoder-eval/instruct/aider/aider/website/docs/leaderboards/index.md`)

highlight_image: /assets/leaderboard.jpg nav_order: 950

Aider LLM Leaderboards

{: .no_toc }

Aider works best with LLMs which are good at editing code, not just good at writing
code.
To evaluate an LLM's editing skill, aider uses a pair of benchmarks that
assess a model's ability to consistently follow the system prompt
to successfully edit code.

The leaderboards below report the results from a number of popular LLMs.
While aider can connect to almost any LLM,
it works best with models that score well on the benchmarks.

See the following sections for benchmark
results and additional information:

  • TOC
    {:toc}

Code editing leaderboard

Aider's code editing benchmark asks the LLM to edit python source files to complete 133 small coding exercises
from Exercism.
This measures the LLM's coding ability, and whether it can
write new code that integrates into existing code.
The model also has to successfully apply all its changes to the source file without human intervention.

Model Percent completed correctly Percent using correct edit format Command Edit format
{{ row.model }} {{ row.pass_rate_2 }}% {{ row.percent_cases_well_formed }}% {{ row.command }} {{ row.edit_format }}

Code refactoring leaderboard

Aider's refactoring benchmark asks the LLM to refactor 89 large methods from large python classes. This is a more challenging benchmark, which tests the model's ability to output long chunks of code without skipping sections or making mistakes. It was developed to provoke and measure GPT-4 Turbo's "lazy coding" habit.

The refactoring benchmark requires a large context window to
work with large source files.
Therefore, results are available for fewer models.

Model Percent completed correctly Percent using correct edit format Command Edit format
{{ row.model }} {{ row.pass_rate_1 }}% {{ row.percent_cases_well_formed }}% {{ row.command }} {{ row.edit_format }}

LLM code editing skill by model release date

Notes on benchmarking results

The key benchmarking results are:

  • Percent completed correctly - Measures what percentage of the coding tasks that the LLM completed successfully. To complete a task, the LLM must solve the programming assignment and edit the code to implement that solution.
  • Percent using correct edit format - Measures the percent of coding tasks where the LLM complied with the edit format specified in the system prompt. If the LLM makes edit mistakes, aider will give it feedback and ask for a fixed copy of the edit. The best models can reliably conform to the edit format, without making errors.

Notes on the edit format

Aider uses different "edit formats" to collect code edits from different LLMs.
The "whole" format is the easiest for an LLM to use, but it uses a lot of tokens
and may limit how large a file can be edited.
Models which can use one of the diff formats are much more efficient,
using far fewer tokens.
Models that use a diff-like format are able to
edit larger files with less cost and without hitting token limits.

Aider is configured to use the best edit format for the popular OpenAI and Anthropic models
and the other models recommended on the LLM page.
For lesser known models aider will default to using the "whole" editing format
since it is the easiest format for an LLM to use.

Contributing benchmark results

Contributions of benchmark results are welcome!
See the
benchmark README
for information on running aider's code editing benchmarks.
Submit results by opening a PR with edits to the
benchmark results data files.

By Paul Gauthier,
last updated

September 05, 2024.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.4.0 1 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/1/README.md`)

Evaluation from 2024-04-25 13:55:10

This report was generated by DevQualityEval benchmark in version 0.4.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • Category Unknown: Models in this category could not be categorized.
  • Response Error: Models in this category encountered an error.
  • Response Empty: Models in this category produced an empty response.
  • No Code: Models in this category produced no code.
  • Invalid Code: Models in this category produced invalid code.
  • Executable Code: Models in this category produced executable code.
  • Statement Coverage Reached: Models in this category produced code that reached full statement coverage.
  • No Excess Response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

"Response Error"

Models in this category encountered an error.

  • openrouter/anthropic/claude-2:beta
  • openrouter/anthropic/claude-2
  • openrouter/anthropic/claude-instant-1:beta
  • openrouter/anthropic/claude-instant-1
  • openrouter/haotian-liu/llava-13b
  • openrouter/jebcarter/psyfighter-13b
  • openrouter/jondurbin/bagel-34b
  • openrouter/lynn/soliloquy-l3
  • openrouter/nousresearch/nous-hermes-2-vision-7b

"Response Empty"

Models in this category produced an empty response.

  • openrouter/databricks/dbrx-instruct

"No Code"

Models in this category produced no code.

  • openrouter/01-ai/yi-34b
  • openrouter/alpindale/goliath-120b
  • openrouter/anthropic/claude-1
  • openrouter/anthropic/claude-instant-1.0
  • openrouter/austism/chronos-hermes-13b
  • openrouter/cognitivecomputations/dolphin-mixtral-8x7b
  • openrouter/google/gemini-pro
  • openrouter/gryphe/mythomax-l2-13b:extended
  • openrouter/gryphe/mythomax-l2-13b
  • openrouter/gryphe/mythomist-7b:free
  • openrouter/gryphe/mythomist-7b
  • openrouter/intel/neural-chat-7b
  • openrouter/koboldai/psyfighter-13b-2
  • openrouter/mancer/weaver
  • openrouter/meta-llama/llama-3-8b-instruct:extended
  • openrouter/mistralai/mistral-7b-instruct
  • openrouter/mistralai/mixtral-8x22b
  • openrouter/mistralai/mixtral-8x7b
  • openrouter/neversleep/noromaid-20b
  • openrouter/neversleep/noromaid-mixtral-8x7b-instruct
  • openrouter/nousresearch/nous-capybara-7b
  • openrouter/openai/gpt-3.5-turbo-0613
  • openrouter/openai/gpt-3.5-turbo-16k
  • openrouter/openai/gpt-3.5-turbo-instruct
  • openrouter/perplexity/pplx-7b-chat
  • openrouter/togethercomputer/stripedhyena-hessian-7b
  • openrouter/undi95/remm-slerp-l2-13b:extended
  • openrouter/undi95/toppy-m-7b

"Invalid Code"

Models in this category produced invalid code.

  • openrouter/01-ai/yi-34b-chat
  • openrouter/01-ai/yi-6b
  • openrouter/anthropic/claude-1.2
  • openrouter/anthropic/claude-2.0:beta
  • openrouter/anthropic/claude-instant-1.1
  • openrouter/anthropic/claude-instant-1.2
  • openrouter/codellama/codellama-70b-instruct
  • openrouter/cohere/command-r-plus
  • openrouter/cohere/command-r
  • openrouter/cohere/command
  • openrouter/google/gemini-pro-vision
  • openrouter/google/gemma-7b-it:free
  • openrouter/google/gemma-7b-it:nitro
  • openrouter/google/gemma-7b-it
  • openrouter/google/palm-2-chat-bison-32k
  • openrouter/google/palm-2-chat-bison
  • openrouter/google/palm-2-codechat-bison-32k
  • openrouter/google/palm-2-codechat-bison
  • openrouter/gryphe/mythomax-l2-13b:nitro
  • openrouter/huggingfaceh4/zephyr-7b-beta:free
  • openrouter/huggingfaceh4/zephyr-7b-beta
  • openrouter/huggingfaceh4/zephyr-orpo-141b-a35b
  • openrouter/lizpreciatior/lzlv-70b-fp16-hf
  • openrouter/meta-llama/llama-2-13b-chat
  • openrouter/meta-llama/llama-2-70b-chat
  • openrouter/meta-llama/llama-3-8b-instruct:nitro
  • openrouter/meta-llama/llama-3-8b-instruct
  • openrouter/microsoft/wizardlm-2-7b
  • openrouter/mistralai/mistral-7b-instruct:free
  • openrouter/mistralai/mistral-large
  • openrouter/mistralai/mistral-small
  • openrouter/mistralai/mistral-tiny
  • openrouter/mistralai/mixtral-8x22b-instruct
  • openrouter/mistralai/mixtral-8x7b-instruct:nitro
  • openrouter/mistralai/mixtral-8x7b-instruct
  • openrouter/nousresearch/nous-capybara-34b
  • openrouter/nousresearch/nous-capybara-7b:free
  • openrouter/nousresearch/nous-hermes-2-mistral-7b-dpo
  • openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpo
  • openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sft
  • openrouter/nousresearch/nous-hermes-llama2-13b
  • openrouter/nousresearch/nous-hermes-yi-34b
  • openrouter/open-orca/mistral-7b-openorca
  • openrouter/openai/gpt-4-turbo-preview
  • openrouter/openchat/openchat-7b:free
  • openrouter/openchat/openchat-7b
  • openrouter/openrouter/cinematika-7b:free
  • openrouter/openrouter/cinematika-7b
  • openrouter/perplexity/pplx-70b-online
  • openrouter/perplexity/pplx-7b-online
  • openrouter/perplexity/sonar-medium-online
  • openrouter/perplexity/sonar-small-chat
  • openrouter/perplexity/sonar-small-online
  • openrouter/phind/phind-codellama-34b
  • openrouter/pygmalionai/mythalion-13b
  • openrouter/recursal/eagle-7b
  • openrouter/recursal/rwkv-5-3b-ai-town
  • openrouter/rwkv/rwkv-5-world-3b
  • openrouter/sao10k/fimbulvetr-11b-v2
  • openrouter/teknium/openhermes-2-mistral-7b
  • openrouter/teknium/openhermes-2.5-mistral-7b
  • openrouter/undi95/toppy-m-7b:free
  • openrouter/undi95/toppy-m-7b:nitro
  • openrouter/xwin-lm/xwin-lm-70b

"Executable Code"

Models in this category produced executable code.

  • openrouter/togethercomputer/stripedhyena-nous-7b

"Statement Coverage Reached"

Models in this category produced code that reached full statement coverage.

  • openrouter/anthropic/claude-2.0
  • openrouter/anthropic/claude-2.1:beta
  • openrouter/anthropic/claude-2.1
  • openrouter/anthropic/claude-3-haiku:beta
  • openrouter/anthropic/claude-3-haiku
  • openrouter/anthropic/claude-3-opus:beta
  • openrouter/anthropic/claude-3-opus
  • openrouter/anthropic/claude-3-sonnet
  • openrouter/jondurbin/airoboros-l2-70b
  • openrouter/meta-llama/codellama-34b-instruct
  • openrouter/meta-llama/llama-2-70b-chat:nitro
  • openrouter/meta-llama/llama-3-70b-instruct:nitro
  • openrouter/meta-llama/llama-3-70b-instruct
  • openrouter/microsoft/wizardlm-2-8x22b:nitro
  • openrouter/microsoft/wizardlm-2-8x22b
  • openrouter/mistralai/mistral-medium
  • openrouter/openai/gpt-3.5-turbo-0301
  • openrouter/openai/gpt-4-1106-preview
  • openrouter/openai/gpt-4-32k
  • openrouter/perplexity/pplx-70b-chat
  • openrouter/perplexity/sonar-medium-chat
  • openrouter/sophosympatheia/midnight-rose-70b
  • openrouter/undi95/remm-slerp-l2-13b

"No Excess Response"

Models in this category did not respond with more content than requested.

  • openrouter/anthropic/claude-3-sonnet:beta
  • openrouter/google/gemini-pro-1.5
  • openrouter/mistralai/mistral-7b-instruct:nitro
  • openrouter/openai/gpt-3.5-turbo-0125
  • openrouter/openai/gpt-3.5-turbo-1106
  • openrouter/openai/gpt-3.5-turbo
  • openrouter/openai/gpt-4-0314
  • openrouter/openai/gpt-4-32k-0314
  • openrouter/openai/gpt-4-turbo
  • openrouter/openai/gpt-4-vision-preview
  • openrouter/openai/gpt-4
  • openrouter/openrouter/auto
  • symflower/symbolic-execution

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.4.0 2 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/2/README.md`)

Evaluation from 2024-04-25 18:19:05

This report was generated by DevQualityEval benchmark in version 0.4.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • Category Unknown: Models in this category could not be categorized.
  • Response Error: Models in this category encountered an error.
  • Response Empty: Models in this category produced an empty response.
  • No Code: Models in this category produced no code.
  • Invalid Code: Models in this category produced invalid code.
  • Executable Code: Models in this category produced executable code.
  • Statement Coverage Reached: Models in this category produced code that reached full statement coverage.
  • No Excess Response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

"Response Error"

Models in this category encountered an error.

  • openrouter/anthropic/claude-2:beta
  • openrouter/anthropic/claude-2
  • openrouter/anthropic/claude-instant-1:beta
  • openrouter/anthropic/claude-instant-1
  • openrouter/haotian-liu/llava-13b
  • openrouter/jebcarter/psyfighter-13b
  • openrouter/jondurbin/bagel-34b
  • openrouter/nousresearch/nous-hermes-2-vision-7b
  • openrouter/openchat/openchat-7b:free
  • openrouter/perplexity/pplx-7b-online
  • openrouter/recursal/rwkv-5-3b-ai-town
  • openrouter/undi95/remm-slerp-l2-13b

"Response Empty"

Models in this category produced an empty response.

  • openrouter/01-ai/yi-34b
  • openrouter/databricks/dbrx-instruct
  • openrouter/mistralai/mixtral-8x22b

"No Code"

Models in this category produced no code.

  • openrouter/anthropic/claude-1.2
  • openrouter/anthropic/claude-1
  • openrouter/austism/chronos-hermes-13b
  • openrouter/codellama/codellama-70b-instruct
  • openrouter/gryphe/mythomax-l2-13b:extended
  • openrouter/gryphe/mythomax-l2-13b
  • openrouter/gryphe/mythomist-7b:free
  • openrouter/gryphe/mythomist-7b
  • openrouter/huggingfaceh4/zephyr-orpo-141b-a35b
  • openrouter/intel/neural-chat-7b
  • openrouter/jondurbin/airoboros-l2-70b
  • openrouter/koboldai/psyfighter-13b-2
  • openrouter/lynn/soliloquy-l3
  • openrouter/mancer/weaver
  • openrouter/meta-llama/llama-3-8b-instruct:extended
  • openrouter/mistralai/mistral-tiny
  • openrouter/mistralai/mixtral-8x7b
  • openrouter/neversleep/noromaid-20b
  • openrouter/neversleep/noromaid-mixtral-8x7b-instruct
  • openrouter/nousresearch/nous-capybara-7b:free
  • openrouter/open-orca/mistral-7b-openorca
  • openrouter/openai/gpt-3.5-turbo-instruct
  • openrouter/pygmalionai/mythalion-13b
  • openrouter/togethercomputer/stripedhyena-hessian-7b
  • openrouter/undi95/remm-slerp-l2-13b:extended
  • openrouter/undi95/toppy-m-7b:nitro

"Invalid Code"

Models in this category produced invalid code.

  • openrouter/01-ai/yi-34b-chat
  • openrouter/01-ai/yi-6b
  • openrouter/alpindale/goliath-120b
  • openrouter/anthropic/claude-2.0:beta
  • openrouter/anthropic/claude-3-haiku:beta
  • openrouter/anthropic/claude-instant-1.1
  • openrouter/anthropic/claude-instant-1.2
  • openrouter/cognitivecomputations/dolphin-mixtral-8x7b
  • openrouter/cohere/command-r-plus
  • openrouter/cohere/command
  • openrouter/google/gemini-pro-vision
  • openrouter/google/gemini-pro
  • openrouter/google/gemma-7b-it:free
  • openrouter/google/gemma-7b-it:nitro
  • openrouter/google/gemma-7b-it
  • openrouter/google/palm-2-chat-bison-32k
  • openrouter/google/palm-2-chat-bison
  • openrouter/google/palm-2-codechat-bison-32k
  • openrouter/google/palm-2-codechat-bison
  • openrouter/gryphe/mythomax-l2-13b:nitro
  • openrouter/huggingfaceh4/zephyr-7b-beta:free
  • openrouter/huggingfaceh4/zephyr-7b-beta
  • openrouter/lizpreciatior/lzlv-70b-fp16-hf
  • openrouter/meta-llama/llama-2-70b-chat
  • openrouter/meta-llama/llama-3-70b-instruct:nitro
  • openrouter/meta-llama/llama-3-8b-instruct:nitro
  • openrouter/meta-llama/llama-3-8b-instruct
  • openrouter/microsoft/wizardlm-2-7b
  • openrouter/mistralai/mistral-7b-instruct:free
  • openrouter/mistralai/mistral-7b-instruct:nitro
  • openrouter/mistralai/mistral-7b-instruct
  • openrouter/mistralai/mistral-large
  • openrouter/mistralai/mistral-small
  • openrouter/mistralai/mixtral-8x22b-instruct
  • openrouter/nousresearch/nous-capybara-34b
  • openrouter/nousresearch/nous-capybara-7b
  • openrouter/nousresearch/nous-hermes-2-mistral-7b-dpo
  • openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpo
  • openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sft
  • openrouter/nousresearch/nous-hermes-llama2-13b
  • openrouter/nousresearch/nous-hermes-yi-34b
  • openrouter/openai/gpt-3.5-turbo-16k
  • openrouter/openchat/openchat-7b
  • openrouter/openrouter/cinematika-7b:free
  • openrouter/openrouter/cinematika-7b
  • openrouter/perplexity/pplx-70b-online
  • openrouter/perplexity/pplx-7b-chat
  • openrouter/perplexity/sonar-medium-chat
  • openrouter/perplexity/sonar-medium-online
  • openrouter/perplexity/sonar-small-chat
  • openrouter/perplexity/sonar-small-online
  • openrouter/recursal/eagle-7b
  • openrouter/rwkv/rwkv-5-world-3b
  • openrouter/sao10k/fimbulvetr-11b-v2
  • openrouter/sophosympatheia/midnight-rose-70b
  • openrouter/teknium/openhermes-2-mistral-7b
  • openrouter/teknium/openhermes-2.5-mistral-7b
  • openrouter/togethercomputer/stripedhyena-nous-7b
  • openrouter/undi95/toppy-m-7b:free
  • openrouter/undi95/toppy-m-7b
  • openrouter/xwin-lm/xwin-lm-70b

"Statement Coverage Reached"

Models in this category produced code that reached full statement coverage.

  • openrouter/anthropic/claude-2.0
  • openrouter/anthropic/claude-2.1:beta
  • openrouter/anthropic/claude-2.1
  • openrouter/anthropic/claude-3-haiku
  • openrouter/anthropic/claude-3-opus:beta
  • openrouter/anthropic/claude-3-opus
  • openrouter/anthropic/claude-3-sonnet:beta
  • openrouter/anthropic/claude-3-sonnet
  • openrouter/anthropic/claude-instant-1.0
  • openrouter/meta-llama/codellama-34b-instruct
  • openrouter/meta-llama/llama-2-13b-chat
  • openrouter/meta-llama/llama-2-70b-chat:nitro
  • openrouter/microsoft/wizardlm-2-8x22b:nitro
  • openrouter/microsoft/wizardlm-2-8x22b
  • openrouter/mistralai/mistral-medium
  • openrouter/mistralai/mixtral-8x7b-instruct:nitro
  • openrouter/mistralai/mixtral-8x7b-instruct
  • openrouter/openai/gpt-3.5-turbo-0301
  • openrouter/openai/gpt-3.5-turbo-1106
  • openrouter/openai/gpt-4-1106-preview
  • openrouter/openai/gpt-4-32k
  • openrouter/openai/gpt-4
  • openrouter/perplexity/pplx-70b-chat
  • openrouter/phind/phind-codellama-34b

"No Excess Response"

Models in this category did not respond with more content than requested.

  • openrouter/cohere/command-r
  • openrouter/google/gemini-pro-1.5
  • openrouter/meta-llama/llama-3-70b-instruct
  • openrouter/openai/gpt-3.5-turbo-0125
  • openrouter/openai/gpt-3.5-turbo-0613
  • openrouter/openai/gpt-3.5-turbo
  • openrouter/openai/gpt-4-0314
  • openrouter/openai/gpt-4-32k-0314
  • openrouter/openai/gpt-4-turbo-preview
  • openrouter/openai/gpt-4-turbo
  • openrouter/openai/gpt-4-vision-preview
  • openrouter/openrouter/auto
  • symflower/symbolic-execution

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.4.0 3 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/3/README.md`)

Evaluation from 2024-04-25 19:03:54

This report was generated by DevQualityEval benchmark in version 0.4.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • Category Unknown: Models in this category could not be categorized.
  • Response Error: Models in this category encountered an error.
  • Response Empty: Models in this category produced an empty response.
  • No Code: Models in this category produced no code.
  • Invalid Code: Models in this category produced invalid code.
  • Executable Code: Models in this category produced executable code.
  • Statement Coverage Reached: Models in this category produced code that reached full statement coverage.
  • No Excess Response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

"Response Error"

Models in this category encountered an error.

  • openrouter/nousresearch/nous-hermes-2-vision-7b
  • openrouter/jondurbin/bagel-34b
  • openrouter/anthropic/claude-2:beta
  • openrouter/anthropic/claude-instant-1
  • openrouter/jebcarter/psyfighter-13b
  • openrouter/anthropic/claude-instant-1:beta
  • openrouter/anthropic/claude-2
  • openrouter/haotian-liu/llava-13b

"Response Empty"

Models in this category produced an empty response.

  • openrouter/databricks/dbrx-instruct
  • openrouter/01-ai/yi-34b

"No Code"

Models in this category produced no code.

  • openrouter/nousresearch/nous-capybara-7b:free
  • openrouter/nousresearch/nous-hermes-yi-34b
  • openrouter/neversleep/noromaid-mixtral-8x7b-instruct
  • openrouter/anthropic/claude-1.2
  • openrouter/pygmalionai/mythalion-13b
  • openrouter/intel/neural-chat-7b
  • openrouter/openchat/openchat-7b:free
  • openrouter/gryphe/mythomax-l2-13b:extended
  • openrouter/anthropic/claude-instant-1.2
  • openrouter/lynn/soliloquy-l3
  • openrouter/undi95/remm-slerp-l2-13b:extended
  • openrouter/open-orca/mistral-7b-openorca
  • openrouter/nousresearch/nous-capybara-7b
  • openrouter/mancer/weaver
  • openrouter/undi95/toppy-m-7b:nitro
  • openrouter/mistralai/mistral-tiny
  • openrouter/gryphe/mythomax-l2-13b:nitro
  • openrouter/neversleep/noromaid-20b
  • openrouter/nousresearch/nous-hermes-2-mistral-7b-dpo
  • openrouter/togethercomputer/stripedhyena-hessian-7b
  • openrouter/codellama/codellama-70b-instruct
  • openrouter/gryphe/mythomist-7b:free
  • openrouter/openai/gpt-3.5-turbo-instruct
  • openrouter/anthropic/claude-1
  • openrouter/meta-llama/llama-3-8b-instruct:extended
  • openrouter/mistralai/mixtral-8x22b
  • openrouter/gryphe/mythomist-7b
  • openrouter/undi95/remm-slerp-l2-13b
  • openrouter/anthropic/claude-instant-1.0

"Invalid Code"

Models in this category produced invalid code.

  • openrouter/teknium/openhermes-2-mistral-7b
  • openrouter/meta-llama/llama-3-8b-instruct
  • openrouter/perplexity/pplx-70b-online
  • openrouter/perplexity/sonar-small-chat
  • openrouter/openrouter/cinematika-7b:free
  • openrouter/nousresearch/nous-hermes-llama2-13b
  • openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sft
  • openrouter/google/gemma-7b-it
  • openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpo
  • openrouter/undi95/toppy-m-7b
  • openrouter/huggingfaceh4/zephyr-orpo-141b-a35b
  • openrouter/google/palm-2-chat-bison-32k
  • openrouter/perplexity/pplx-7b-chat
  • openrouter/google/palm-2-codechat-bison-32k
  • openrouter/01-ai/yi-6b
  • openrouter/mistralai/mixtral-8x7b
  • openrouter/openai/gpt-3.5-turbo-16k
  • openrouter/google/gemma-7b-it:nitro
  • openrouter/mistralai/mistral-small
  • openrouter/mistralai/mistral-large
  • openrouter/openrouter/cinematika-7b
  • openrouter/cohere/command-r
  • openrouter/sao10k/fimbulvetr-11b-v2
  • openrouter/mistralai/mistral-7b-instruct
  • openrouter/undi95/toppy-m-7b:free
  • openrouter/recursal/eagle-7b
  • openrouter/teknium/openhermes-2.5-mistral-7b
  • openrouter/anthropic/claude-instant-1.1
  • openrouter/alpindale/goliath-120b
  • openrouter/cognitivecomputations/dolphin-mixtral-8x7b
  • openrouter/google/gemma-7b-it:free
  • openrouter/lizpreciatior/lzlv-70b-fp16-hf
  • openrouter/mistralai/mistral-7b-instruct:nitro
  • openrouter/perplexity/sonar-medium-online
  • openrouter/recursal/rwkv-5-3b-ai-town
  • openrouter/google/gemini-pro-vision
  • openrouter/gryphe/mythomax-l2-13b
  • openrouter/sophosympatheia/midnight-rose-70b
  • openrouter/cohere/command
  • openrouter/mistralai/mixtral-8x22b-instruct
  • openrouter/cohere/command-r-plus
  • openrouter/huggingfaceh4/zephyr-7b-beta:free
  • openrouter/meta-llama/codellama-34b-instruct
  • openrouter/microsoft/wizardlm-2-7b
  • openrouter/rwkv/rwkv-5-world-3b
  • openrouter/perplexity/pplx-7b-online
  • openrouter/austism/chronos-hermes-13b
  • openrouter/anthropic/claude-2.0:beta
  • openrouter/google/palm-2-chat-bison
  • openrouter/google/palm-2-codechat-bison
  • openrouter/togethercomputer/stripedhyena-nous-7b
  • openrouter/meta-llama/llama-3-8b-instruct:nitro
  • openrouter/koboldai/psyfighter-13b-2
  • openrouter/openchat/openchat-7b
  • openrouter/openai/gpt-3.5-turbo-0613
  • openrouter/huggingfaceh4/zephyr-7b-beta
  • openrouter/xwin-lm/xwin-lm-70b
  • openrouter/mistralai/mistral-7b-instruct:free
  • openrouter/perplexity/sonar-small-online

"Statement Coverage Reached"

Models in this category produced code that reached full statement coverage.

  • openrouter/anthropic/claude-2.1
  • openrouter/perplexity/sonar-medium-chat
  • openrouter/anthropic/claude-3-sonnet:beta
  • openrouter/mistralai/mistral-medium
  • openrouter/microsoft/wizardlm-2-8x22b
  • openrouter/anthropic/claude-3-haiku:beta
  • openrouter/microsoft/wizardlm-2-8x22b:nitro
  • openrouter/anthropic/claude-2.0
  • openrouter/anthropic/claude-2.1:beta
  • openrouter/anthropic/claude-3-sonnet
  • openrouter/openai/gpt-3.5-turbo-1106
  • openrouter/openai/gpt-4-1106-preview
  • openrouter/mistralai/mixtral-8x7b-instruct:nitro
  • openrouter/meta-llama/llama-2-13b-chat
  • openrouter/anthropic/claude-3-opus:beta
  • openrouter/jondurbin/airoboros-l2-70b
  • openrouter/meta-llama/llama-2-70b-chat:nitro
  • openrouter/meta-llama/llama-3-70b-instruct:nitro
  • openrouter/perplexity/pplx-70b-chat
  • openrouter/anthropic/claude-3-haiku
  • openrouter/meta-llama/llama-2-70b-chat
  • openrouter/openai/gpt-4
  • openrouter/anthropic/claude-3-opus
  • openrouter/phind/phind-codellama-34b
  • openrouter/mistralai/mixtral-8x7b-instruct
  • openrouter/openai/gpt-4-32k

"No Excess Response"

Models in this category did not respond with more content than requested.

  • openrouter/openai/gpt-4-0314
  • openrouter/openai/gpt-4-turbo
  • openrouter/nousresearch/nous-capybara-34b
  • symflower/symbolic-execution
  • openrouter/google/gemini-pro-1.5
  • openrouter/openrouter/auto
  • openrouter/openai/gpt-4-turbo-preview
  • openrouter/openai/gpt-3.5-turbo-0125
  • openrouter/openai/gpt-3.5-turbo
  • openrouter/meta-llama/llama-3-70b-instruct
  • openrouter/01-ai/yi-34b-chat
  • openrouter/openai/gpt-3.5-turbo-0301
  • openrouter/openai/gpt-4-32k-0314
  • openrouter/google/gemini-pro
  • openrouter/openai/gpt-4-vision-preview

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.4.0 4 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/4/README.md`)

Evaluation from 2024-04-25 19:31:23

This report was generated by DevQualityEval benchmark in version 0.4.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • Category Unknown: Models in this category could not be categorized.
  • Response Error: Models in this category encountered an error.
  • Response Empty: Models in this category produced an empty response.
  • No Code: Models in this category produced no code.
  • Invalid Code: Models in this category produced invalid code.
  • Executable Code: Models in this category produced executable code.
  • Statement Coverage Reached: Models in this category produced code that reached full statement coverage.
  • No Excess Response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

"Response Error"

Models in this category encountered an error.

  • openrouter/jebcarter/psyfighter-13b
  • openrouter/anthropic/claude-instant-1
  • openrouter/anthropic/claude-instant-1:beta
  • openrouter/nousresearch/nous-hermes-2-vision-7b
  • openrouter/haotian-liu/llava-13b
  • openrouter/jondurbin/bagel-34b
  • openrouter/anthropic/claude-2
  • openrouter/anthropic/claude-2:beta

"No Code"

Models in this category produced no code.

  • openrouter/gryphe/mythomist-7b
  • openrouter/meta-llama/llama-3-8b-instruct:extended
  • openrouter/undi95/toppy-m-7b:nitro
  • openrouter/mancer/weaver
  • openrouter/nousresearch/nous-capybara-7b:free
  • openrouter/anthropic/claude-instant-1.2
  • openrouter/google/gemini-pro
  • openrouter/anthropic/claude-1
  • openrouter/gryphe/mythomax-l2-13b:nitro
  • openrouter/gryphe/mythomax-l2-13b
  • openrouter/lynn/soliloquy-l3
  • openrouter/gryphe/mythomist-7b:free
  • openrouter/neversleep/noromaid-20b
  • openrouter/openai/gpt-3.5-turbo-instruct
  • openrouter/koboldai/psyfighter-13b-2
  • openrouter/togethercomputer/stripedhyena-hessian-7b
  • openrouter/pygmalionai/mythalion-13b
  • openrouter/mistralai/mistral-tiny
  • openrouter/undi95/toppy-m-7b
  • openrouter/undi95/remm-slerp-l2-13b
  • openrouter/anthropic/claude-instant-1.0
  • openrouter/intel/neural-chat-7b
  • openrouter/undi95/toppy-m-7b:free
  • openrouter/neversleep/noromaid-mixtral-8x7b-instruct
  • openrouter/undi95/remm-slerp-l2-13b:extended

"Invalid Code"

Models in this category produced invalid code.

  • openrouter/mistralai/mistral-large
  • openrouter/togethercomputer/stripedhyena-nous-7b
  • openrouter/gryphe/mythomax-l2-13b:extended
  • openrouter/openai/gpt-3.5-turbo-0613
  • openrouter/open-orca/mistral-7b-openorca
  • openrouter/jondurbin/airoboros-l2-70b
  • openrouter/google/gemma-7b-it:nitro
  • openrouter/meta-llama/llama-3-8b-instruct
  • openrouter/anthropic/claude-2.0
  • openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sft
  • openrouter/google/palm-2-codechat-bison-32k
  • openrouter/google/gemini-pro-vision
  • openrouter/google/palm-2-codechat-bison
  • openrouter/openai/gpt-3.5-turbo-16k
  • openrouter/perplexity/pplx-7b-online
  • openrouter/01-ai/yi-34b
  • openrouter/cognitivecomputations/dolphin-mixtral-8x7b
  • openrouter/openrouter/cinematika-7b:free
  • openrouter/alpindale/goliath-120b
  • openrouter/teknium/openhermes-2.5-mistral-7b
  • openrouter/google/gemma-7b-it
  • openrouter/mistralai/mixtral-8x7b-instruct
  • openrouter/meta-llama/llama-3-8b-instruct:nitro
  • openrouter/fireworks/firellava-13b
  • openrouter/mistralai/mistral-7b-instruct:nitro
  • openrouter/cohere/command-r
  • openrouter/nousresearch/nous-capybara-7b
  • openrouter/01-ai/yi-6b
  • openrouter/mistralai/mistral-7b-instruct:free
  • openrouter/openchat/openchat-7b
  • openrouter/xwin-lm/xwin-lm-70b
  • openrouter/openchat/openchat-7b:free
  • openrouter/perplexity/sonar-small-online
  • openrouter/phind/phind-codellama-34b
  • openrouter/mistralai/mixtral-8x22b-instruct
  • openrouter/teknium/openhermes-2-mistral-7b
  • openrouter/huggingfaceh4/zephyr-7b-beta:free
  • openrouter/microsoft/wizardlm-2-7b
  • openrouter/cohere/command
  • openrouter/openrouter/cinematika-7b
  • openrouter/rwkv/rwkv-5-world-3b
  • openrouter/cohere/command-r-plus
  • openrouter/lizpreciatior/lzlv-70b-fp16-hf
  • openrouter/austism/chronos-hermes-13b
  • openrouter/mistralai/mixtral-8x7b
  • openrouter/perplexity/pplx-70b-online
  • openrouter/meta-llama/codellama-34b-instruct
  • openrouter/nousresearch/nous-hermes-2-mistral-7b-dpo
  • openrouter/huggingfaceh4/zephyr-7b-beta
  • openrouter/meta-llama/llama-2-13b-chat
  • openrouter/nousresearch/nous-hermes-llama2-13b
  • openrouter/anthropic/claude-1.2
  • openrouter/sao10k/fimbulvetr-11b-v2
  • openrouter/nousresearch/nous-hermes-yi-34b
  • openrouter/google/palm-2-chat-bison-32k
  • openrouter/perplexity/sonar-medium-chat
  • openrouter/mistralai/mistral-small
  • openrouter/mistralai/mixtral-8x22b
  • openrouter/openai/gpt-4-32k
  • openrouter/anthropic/claude-3-haiku:beta
  • openrouter/recursal/rwkv-5-3b-ai-town
  • openrouter/anthropic/claude-2.0:beta
  • openrouter/perplexity/sonar-medium-online
  • openrouter/nousresearch/nous-capybara-34b
  • openrouter/recursal/eagle-7b
  • openrouter/sophosympatheia/midnight-rose-70b
  • openrouter/huggingfaceh4/zephyr-orpo-141b-a35b
  • openrouter/mistralai/mixtral-8x7b-instruct:nitro
  • openrouter/openai/gpt-3.5-turbo

"Executable Code"

Models in this category produced executable code.

  • openrouter/codellama/codellama-70b-instruct
  • openrouter/perplexity/pplx-7b-chat
  • openrouter/google/gemini-pro-1.5
  • openrouter/perplexity/sonar-small-chat

"Statement Coverage Reached"

Models in this category produced code that reached full statement coverage.

  • openrouter/microsoft/wizardlm-2-8x22b:nitro
  • openrouter/anthropic/claude-3-sonnet:beta
  • openrouter/anthropic/claude-2.1
  • openrouter/anthropic/claude-3-opus
  • openrouter/mistralai/mistral-medium
  • openrouter/anthropic/claude-3-opus:beta
  • openrouter/openai/gpt-3.5-turbo-1106
  • openrouter/openai/gpt-4-1106-preview
  • openrouter/01-ai/yi-34b-chat
  • openrouter/meta-llama/llama-2-70b-chat:nitro
  • openrouter/openai/gpt-3.5-turbo-0301
  • openrouter/microsoft/wizardlm-2-8x22b
  • openrouter/anthropic/claude-3-sonnet
  • openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpo
  • openrouter/anthropic/claude-2.1:beta
  • openrouter/perplexity/pplx-70b-chat
  • openrouter/anthropic/claude-instant-1.1
  • openrouter/openai/gpt-4-32k-0314
  • openrouter/anthropic/claude-3-haiku
  • openrouter/meta-llama/llama-2-70b-chat

"No Excess Response"

Models in this category did not respond with more content than requested.

  • openrouter/mistralai/mistral-7b-instruct
  • openrouter/openai/gpt-4-vision-preview
  • openrouter/databricks/dbrx-instruct
  • openrouter/openai/gpt-4-turbo
  • openrouter/openai/gpt-4-0314
  • openrouter/openai/gpt-3.5-turbo-0125
  • openrouter/openai/gpt-4-turbo-preview
  • symflower/symbolic-execution
  • openrouter/meta-llama/llama-3-70b-instruct
  • openrouter/google/gemma-7b-it:free
  • openrouter/openai/gpt-4
  • openrouter/openrouter/auto
  • openrouter/meta-llama/llama-3-70b-instruct:nitro
  • openrouter/google/palm-2-chat-bison

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.4.0 5 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/5/README.md`)

Evaluation from 2024-04-25 19:31:35

This report was generated by DevQualityEval benchmark in version 0.4.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • Category Unknown: Models in this category could not be categorized.
  • Response Error: Models in this category encountered an error.
  • Response Empty: Models in this category produced an empty response.
  • No Code: Models in this category produced no code.
  • Invalid Code: Models in this category produced invalid code.
  • Executable Code: Models in this category produced executable code.
  • Statement Coverage Reached: Models in this category produced code that reached full statement coverage.
  • No Excess Response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

"Response Error"

Models in this category encountered an error.

  • openrouter/jebcarter/psyfighter-13b
  • openrouter/anthropic/claude-2
  • openrouter/openchat/openchat-7b:free
  • openrouter/openrouter/cinematika-7b:free
  • openrouter/jondurbin/bagel-34b
  • openrouter/anthropic/claude-instant-1
  • openrouter/haotian-liu/llava-13b
  • openrouter/undi95/toppy-m-7b:free
  • openrouter/anthropic/claude-2:beta
  • openrouter/nousresearch/nous-hermes-2-vision-7b
  • openrouter/anthropic/claude-instant-1:beta
  • openrouter/nousresearch/nous-capybara-7b:free
  • openrouter/perplexity/pplx-70b-online
  • openrouter/lynn/soliloquy-l3

"No Code"

Models in this category produced no code.

  • openrouter/neversleep/noromaid-20b
  • openrouter/koboldai/psyfighter-13b-2
  • openrouter/nousresearch/nous-capybara-7b
  • openrouter/neversleep/noromaid-mixtral-8x7b-instruct
  • openrouter/gryphe/mythomist-7b:free
  • openrouter/meta-llama/codellama-34b-instruct
  • openrouter/undi95/remm-slerp-l2-13b:extended
  • openrouter/meta-llama/llama-3-8b-instruct:extended
  • openrouter/anthropic/claude-instant-1.0
  • openrouter/openai/gpt-3.5-turbo-1106
  • openrouter/gryphe/mythomist-7b
  • openrouter/nousresearch/nous-hermes-llama2-13b
  • openrouter/teknium/openhermes-2.5-mistral-7b
  • openrouter/mistralai/mistral-tiny
  • openrouter/intel/neural-chat-7b
  • openrouter/gryphe/mythomax-l2-13b:nitro
  • openrouter/gryphe/mythomax-l2-13b:extended
  • openrouter/mistralai/mistral-7b-instruct:nitro
  • openrouter/01-ai/yi-34b
  • openrouter/google/gemini-pro
  • openrouter/togethercomputer/stripedhyena-hessian-7b
  • openrouter/cognitivecomputations/dolphin-mixtral-8x7b
  • openrouter/openai/gpt-3.5-turbo-instruct
  • openrouter/pygmalionai/mythalion-13b
  • openrouter/mancer/weaver
  • openrouter/undi95/remm-slerp-l2-13b
  • openrouter/openai/gpt-3.5-turbo-0613

"Invalid Code"

Models in this category produced invalid code.

  • openrouter/mistralai/mistral-7b-instruct:free
  • openrouter/codellama/codellama-70b-instruct
  • openrouter/microsoft/wizardlm-2-7b
  • openrouter/google/palm-2-codechat-bison
  • openrouter/open-orca/mistral-7b-openorca
  • openrouter/google/palm-2-chat-bison
  • openrouter/perplexity/pplx-7b-chat
  • openrouter/01-ai/yi-34b-chat
  • openrouter/nousresearch/nous-hermes-2-mistral-7b-dpo
  • openrouter/anthropic/claude-3-haiku:beta
  • openrouter/undi95/toppy-m-7b
  • openrouter/huggingfaceh4/zephyr-7b-beta:free
  • openrouter/xwin-lm/xwin-lm-70b
  • openrouter/01-ai/yi-6b
  • openrouter/undi95/toppy-m-7b:nitro
  • openrouter/openai/gpt-3.5-turbo-16k
  • openrouter/huggingfaceh4/zephyr-7b-beta
  • openrouter/rwkv/rwkv-5-world-3b
  • openrouter/cohere/command-r-plus
  • openrouter/anthropic/claude-2.0
  • openrouter/google/gemini-pro-vision
  • openrouter/sao10k/fimbulvetr-11b-v2
  • openrouter/sophosympatheia/midnight-rose-70b
  • openrouter/alpindale/goliath-120b
  • openrouter/perplexity/sonar-medium-online
  • openrouter/openchat/openchat-7b
  • openrouter/mistralai/mixtral-8x7b
  • openrouter/meta-llama/llama-3-8b-instruct
  • openrouter/austism/chronos-hermes-13b
  • openrouter/mistralai/mistral-large
  • openrouter/mistralai/mixtral-8x22b
  • openrouter/anthropic/claude-1.2
  • openrouter/meta-llama/llama-2-13b-chat
  • openrouter/nousresearch/nous-capybara-34b
  • openrouter/openrouter/cinematika-7b
  • openrouter/perplexity/pplx-7b-online
  • openrouter/cohere/command
  • openrouter/anthropic/claude-instant-1.1
  • openrouter/teknium/openhermes-2-mistral-7b
  • openrouter/mistralai/mixtral-8x7b-instruct:nitro
  • openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sft
  • openrouter/perplexity/sonar-small-online
  • openrouter/cohere/command-r
  • openrouter/togethercomputer/stripedhyena-nous-7b
  • openrouter/google/gemma-7b-it:nitro
  • openrouter/nousresearch/nous-hermes-yi-34b
  • openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpo
  • openrouter/recursal/rwkv-5-3b-ai-town
  • openrouter/meta-llama/llama-3-8b-instruct:nitro
  • openrouter/perplexity/pplx-70b-chat
  • openrouter/fireworks/firellava-13b
  • openrouter/jondurbin/airoboros-l2-70b
  • openrouter/google/gemma-7b-it:free
  • openrouter/anthropic/claude-2.0:beta
  • openrouter/google/palm-2-codechat-bison-32k
  • openrouter/google/gemma-7b-it
  • openrouter/recursal/eagle-7b
  • openrouter/perplexity/sonar-small-chat
  • openrouter/google/palm-2-chat-bison-32k

"Executable Code"

Models in this category produced executable code.

  • openrouter/mistralai/mistral-7b-instruct

"Statement Coverage Reached"

Models in this category produced code that reached full statement coverage.

  • openrouter/anthropic/claude-2.1:beta
  • openrouter/openai/gpt-4-32k
  • openrouter/microsoft/wizardlm-2-8x22b:nitro
  • openrouter/anthropic/claude-3-sonnet
  • openrouter/openai/gpt-3.5-turbo-0125
  • openrouter/lizpreciatior/lzlv-70b-fp16-hf
  • openrouter/databricks/dbrx-instruct
  • openrouter/meta-llama/llama-2-70b-chat:nitro
  • openrouter/openai/gpt-4-1106-preview
  • openrouter/mistralai/mistral-medium
  • openrouter/mistralai/mixtral-8x7b-instruct
  • openrouter/phind/phind-codellama-34b
  • openrouter/perplexity/sonar-medium-chat
  • openrouter/anthropic/claude-3-opus
  • openrouter/anthropic/claude-1
  • openrouter/anthropic/claude-2.1
  • openrouter/mistralai/mistral-small
  • openrouter/meta-llama/llama-3-70b-instruct:nitro
  • openrouter/anthropic/claude-3-opus:beta
  • openrouter/gryphe/mythomax-l2-13b
  • openrouter/microsoft/wizardlm-2-8x22b
  • openrouter/anthropic/claude-3-sonnet:beta
  • openrouter/anthropic/claude-3-haiku
  • openrouter/mistralai/mixtral-8x22b-instruct
  • openrouter/meta-llama/llama-3-70b-instruct
  • openrouter/meta-llama/llama-2-70b-chat
  • openrouter/huggingfaceh4/zephyr-orpo-141b-a35b

"No Excess Response"

Models in this category did not respond with more content than requested.

  • openrouter/openai/gpt-4-turbo
  • openrouter/openai/gpt-3.5-turbo
  • openrouter/google/gemini-pro-1.5
  • symflower/symbolic-execution
  • openrouter/openai/gpt-4-turbo-preview
  • openrouter/openai/gpt-4-32k-0314
  • openrouter/openai/gpt-4
  • openrouter/openai/gpt-3.5-turbo-0301
  • openrouter/openai/gpt-4-vision-preview
  • openrouter/openai/gpt-4-0314
  • openrouter/anthropic/claude-instant-1.2
  • openrouter/openrouter/auto

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Chronos Hermes 13b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/chronos-hermes-13b/README.md`)

Evaluation from 2024-06-19 10:00:20

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 1.2 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-1.2/README.md`)

Evaluation from 2024-06-24 11:19:06

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 2.0 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-2.0/README.md`)

Evaluation from 2024-06-24 11:19:15

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 2.1 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-2.1/README.md`)

Evaluation from 2024-06-21 09:52:06

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 3 Haiku README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-3-haiku/README.md`)

Evaluation from 2024-06-19 08:44:31

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 3 Opus README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-3-opus/README.md`)

Evaluation from 2024-06-19 09:59:38

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 3 Sonnet README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-3-sonnet/README.md`)

Evaluation from 2024-06-19 09:59:44

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude 3.5 Sonnet README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-3.5-sonnet/README.md`)

Evaluation from 2024-06-20 15:04:46

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude Instant 1.0 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-instant-1.0/README.md`)

Evaluation from 2024-06-24 11:18:54

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Claude Instant 1.1 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/claude-instant-1.1/README.md`)

Evaluation from 2024-06-19 09:09:19

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Codellama 34b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/codellama-34b-instruct/README.md`)

Evaluation from 2024-06-19 11:21:09

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Codeqwen README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/codeqwen/README.md`)

Evaluation from 2024-06-21 06:54:57

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Codestral README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/codestral/README.md`)

Evaluation from 2024-06-20 09:05:20

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Command R Plus README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/command-r-plus/README.md`)

Evaluation from 2024-06-19 10:00:24

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Dbrx Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/dbrx-instruct/README.md`)

Evaluation from 2024-06-19 10:00:27

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Deepseek Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/deepseek-chat/README.md`)

Evaluation from 2024-06-19 10:00:30

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Deepseek Coder V2 16b Lite Instruct Fp16 54a969781028 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/deepseek-coder-v2-16b-lite-instruct-fp16-54a969781028/README.md`)

Evaluation from 2024-06-24 11:35:07

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Deepseek Coder V2 16b Lite Instruct Q8 0 44250301ba51 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/deepseek-coder-v2-16b-lite-instruct-q8_0-44250301ba51/README.md`)

Evaluation from 2024-06-24 14:19:10

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Deepseek Coder README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/deepseek-coder/README.md`)

Evaluation from 2024-06-19 10:00:32

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Dolphin Mixtral 8x22b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/dolphin-mixtral-8x22b/README.md`)

Evaluation from 2024-06-19 10:00:21

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Dolphin Mixtral 8x7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/dolphin-mixtral-8x7b/README.md`)

Evaluation from 2024-06-19 10:00:21

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Eagle 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/eagle-7b/README.md`)

Evaluation from 2024-06-19 11:29:12

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Falcon2 11b Q8 0 2d49820d6bb5 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/falcon2-11b-q8_0-2d49820d6bb5/README.md`)

Evaluation from 2024-06-24 09:05:25

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Firellava 13b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/firellava-13b/README.md`)

Evaluation from 2024-06-19 10:00:35

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gemini Flash 1.5 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gemini-flash-1.5/README.md`)

Evaluation from 2024-06-19 10:01:09

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gemini Pro 1.5 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gemini-pro-1.5/README.md`)

Evaluation from 2024-06-19 08:43:07

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gemma 2 27b It README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gemma-2-27b-it/README.md`)

Evaluation from 2024-07-05 06:55:06

This report was generated by DevQualityEval benchmark in version 0.5.0.

REMARK: gemma-2-9b-it and gemma-2-27-it were originally evaluated together with the results then being split into separate folders. Therefore some logs might contain entries from "the other" gemma model.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gemma 2 9b It README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gemma-2-9b-it/README.md`)

Evaluation from 2024-07-05 06:55:06

This report was generated by DevQualityEval benchmark in version 0.5.0.

REMARK: gemma-2-9b-it and gemma-2-27-it were originally evaluated together with the results then being split into separate folders. Therefore some logs might contain entries from "the other" gemma model.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gemma 7b It README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gemma-7b-it/README.md`)

Evaluation from 2024-06-19 10:01:10

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Goliath 120b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/goliath-120b/README.md`)

Evaluation from 2024-06-19 08:43:56

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gpt 4 Turbo README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gpt-4-turbo/README.md`)

Evaluation from 2024-06-19 11:09:05

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gpt 4 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gpt-4/README.md`)

Evaluation from 2024-06-19 11:08:40

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Gpt 4o README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/gpt-4o/README.md`)

Evaluation from 2024-06-19 09:57:48

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Granite Code 34b Instruct F16 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/granite-code-34b-instruct-f16/README.md`)

Evaluation from 2024-06-21 11:13:38

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Granite Code 34b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/granite-code-34b/README.md`)

Evaluation from 2024-06-20 12:20:38

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Granite Code 3b Instruct Q8 0 B93b1a15197d README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/granite-code-3b-instruct-q8_0-b93b1a15197d/README.md`)

Evaluation from 2024-06-24 14:53:48

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Hermes 2 Pro Llama 3 8b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/hermes-2-pro-llama-3-8b/README.md`)

Evaluation from 2024-06-19 10:37:04

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llama 3 70b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llama-3-70b-instruct/README.md`)

Evaluation from 2024-06-19 10:01:56

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llama 3 8b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llama-3-8b-instruct/README.md`)

Evaluation from 2024-06-19 10:01:57

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llama 3 Lumimaid 70b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llama-3-lumimaid-70b/README.md`)

Evaluation from 2024-06-19 10:28:10

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llama 3 Lumimaid 8b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llama-3-lumimaid-8b/README.md`)

Evaluation from 2024-06-19 10:28:10

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llama 3 Sonar Large 32k Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llama-3-sonar-large-32k-chat/README.md`)

Evaluation from 2024-06-19 11:14:50

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llama 3 Sonar Small 32k Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llama-3-sonar-small-32k-chat/README.md`)

Evaluation from 2024-06-19 10:39:07

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llava 13b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llava-13b/README.md`)

Evaluation from 2024-06-19 10:01:10

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Llava Yi 34b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/llava-yi-34b/README.md`)

Evaluation from 2024-06-19 10:01:19

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mistral 7b Instruct V0.3 README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mistral-7b-instruct-v0.3/README.md`)

Evaluation from 2024-06-19 10:27:04

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mistral 7b Openorca README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mistral-7b-openorca/README.md`)

Evaluation from 2024-06-19 11:08:37

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mistral Large README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mistral-large/README.md`)

Evaluation from 2024-06-19 08:40:58

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mistral Medium README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mistral-medium/README.md`)

Evaluation from 2024-06-19 10:27:12

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mistral Small README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mistral-small/README.md`)

Evaluation from 2024-06-19 10:27:18

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mistral Tiny README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mistral-tiny/README.md`)

Evaluation from 2024-06-19 10:27:32

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mixtral 8x22b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mixtral-8x22b-instruct/README.md`)

Evaluation from 2024-06-19 10:27:42

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Mixtral 8x7b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/mixtral-8x7b-instruct/README.md`)

Evaluation from 2024-06-19 10:27:44

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Neural Chat 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/neural-chat-7b/README.md`)

Evaluation from 2024-06-19 10:01:09

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Nous Capybara 34b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/nous-capybara-34b/README.md`)

Evaluation from 2024-06-19 10:37:07

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Nous Capybara 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/nous-capybara-7b/README.md`)

Evaluation from 2024-06-19 10:47:13

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Nous Hermes 2 Mistral 7b Dpo README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/nous-hermes-2-mistral-7b-dpo/README.md`)

Evaluation from 2024-06-19 11:08:27

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Nous Hermes 2 Mixtral 8x7b Dpo README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/nous-hermes-2-mixtral-8x7b-dpo/README.md`)

Evaluation from 2024-06-19 11:08:30

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Nous Hermes 2 Mixtral 8x7b Sft README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/nous-hermes-2-mixtral-8x7b-sft/README.md`)

Evaluation from 2024-06-19 11:08:32

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Nous Hermes Yi 34b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/nous-hermes-yi-34b/README.md`)

Evaluation from 2024-06-19 11:08:35

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Olmo 7b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/olmo-7b-instruct/README.md`)

Evaluation from 2024-06-19 08:43:44

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Openchat 8b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/openchat-8b/README.md`)

Evaluation from 2024-06-19 11:14:47

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Openhermes 2 Mistral 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/openhermes-2-mistral-7b/README.md`)

Evaluation from 2024-06-19 11:09:44

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Openhermes 2.5 Mistral 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/openhermes-2.5-mistral-7b/README.md`)

Evaluation from 2024-06-19 11:20:58

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Palm 2 Chat Bison README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/palm-2-chat-bison/README.md`)

Evaluation from 2024-06-19 10:01:10

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Palm 2 Codechat Bison README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/palm-2-codechat-bison/README.md`)

Evaluation from 2024-06-19 10:01:10

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Phi 3 Medium 128k Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/phi-3-medium-128k-instruct/README.md`)

Evaluation from 2024-06-19 10:11:27

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Phi 3 Medium 4k Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/phi-3-medium-4k-instruct/README.md`)

Evaluation from 2024-06-19 10:11:31

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Phi 3 Mini 128k Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/phi-3-mini-128k-instruct/README.md`)

Evaluation from 2024-06-19 10:19:41

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Phind Codellama 34b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/phind-codellama-34b/README.md`)

Evaluation from 2024-06-19 10:39:01

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 110b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-110b-chat/README.md`)

Evaluation from 2024-06-19 11:27:23

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 14b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-14b-chat/README.md`)

Evaluation from 2024-06-19 11:27:44

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 2 72b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-2-72b-instruct/README.md`)

Evaluation from 2024-06-19 09:58:19

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 32b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-32b-chat/README.md`)

Evaluation from 2024-06-19 11:28:04

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 4b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-4b-chat/README.md`)

Evaluation from 2024-06-19 11:28:14

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 72b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-72b-chat/README.md`)

Evaluation from 2024-06-19 11:28:49

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Qwen 7b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/qwen-7b-chat/README.md`)

Evaluation from 2024-06-19 11:28:59

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Rwkv 5 3b Ai Town README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/rwkv-5-3b-ai-town/README.md`)

Evaluation from 2024-06-19 11:29:26

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Rwkv 5 World 3b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/rwkv-5-world-3b/README.md`)

Evaluation from 2024-06-19 11:29:39

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Snowflake Arctic Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/snowflake-arctic-instruct/README.md`)

Evaluation from 2024-06-19 10:21:30

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Starcoder2 15b Instruct README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/starcoder2-15b-instruct/README.md`)

Evaluation from 2024-06-19 09:12:15

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Stripedhyena Hessian 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/stripedhyena-hessian-7b/README.md`)

Evaluation from 2024-06-19 11:18:23

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Stripedhyena Nous 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/stripedhyena-nous-7b/README.md`)

Evaluation from 2024-06-19 10:52:48

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Symbolic Execution README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/symbolic-execution/README.md`)

Evaluation from 2024-06-19 13:25:26

This report was generated by DevQualityEval benchmark in version 0.5.0.

PLEASE NOTE that we assume the results of the "symbolic execution" are deterministic. Hence they where only benchmarked over a single run and multiplied by 5 to be comparable to the remaining contestants.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Toppy M 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/toppy-m-7b/README.md`)

Evaluation from 2024-06-19 10:52:54

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Wizardlm 2 7b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/wizardlm-2-7b/README.md`)

Evaluation from 2024-06-19 10:23:07

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Wizardlm 2 8x22b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/wizardlm-2-8x22b/README.md`)

Evaluation from 2024-06-19 10:23:10

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Xwin Lm 70b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/xwin-lm-70b/README.md`)

Evaluation from 2024-06-19 10:26:13

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Yi 34b Chat README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/yi-34b-chat/README.md`)

Evaluation from 2024-06-19 10:15:10

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Yi 6b README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/yi-6b/README.md`)

Evaluation from 2024-06-19 08:43:03

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

Qwencoder Eval Instruct Eval Dev Quality Docs Reports V0.5.0 Zephyr 7b Beta README (`qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.5.0/zephyr-7b-beta/README.md`)

Evaluation from 2024-06-19 10:01:10

This report was generated by DevQualityEval benchmark in version 0.5.0.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

  • category unknown: Models in this category could not be categorized.
  • response error: Models in this category encountered an error.
  • no code: Models in this category produced no code.
  • invalid code: Models in this category produced invalid code.
  • executable code: Models in this category produced executable code.
  • statement coverage reached: Models in this category produced code that reached full statement coverage.
  • no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →