{"owner":"alibaba","repo":"zvec","hasSkills":true,"totalSkillsCount":1,"totalTokensCount":1330,"categories":["plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":["tools/core/README.md"],"skills":{"tools/core/README.md":"\n# Benchmarking scripts\n\nThis directory contains benchmarking scripts and reproducing steps.\n\n## COHERE experiments\n\n### Getting COHERE Data\n\nPlease download the COHERE 10M dataset to cohere_large_10m as follows:\n\n```bash\n... ...           \nneighbors.parquet    \nshuffle_train-00-of-10.parquet     \nshuffle_train-01-of-10.parquet          \nshuffle_train-02-of-10.parquet  \nshuffle_train-03-of-10.parquet \nshuffle_train-04-of-10.parquet  \nshuffle_train-05-of-10.parquet \nshuffle_train-06-of-10.parquet\nshuffle_train-07-of-10.parquet\nshuffle_train-08-of-10.parquet\nshuffle_train-09-of-10.parquet\nscalar_labels.parquet     \ntest.parquet      \n```\n\nFor convenience, we prepared a docker image with cohere bench datasets: registry.cn-hongkong.cr.aliyuncs.com/zvec/cohere-bench-data. \n\nYou can run a container as follows:\n\n```bash\ndocker run -it --net=host -d -e DEBUG_MODE=true  --user root --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /home/zvec/:/home/zvec/  -w /home/zvec --name=cohere_bench zvec-registry.cn-hongkong.cr.aliyuncs.com/zvec/cohere-bench-data:0.0.1 bash\n\ndocker exec -it cohere_bench bash\n```\n\nThe datasets locate at /tmp/cohere/\n\n### Preparing Environment \nClone code and init:\n```bash\n$ git clone git@github.com:alibaba/zvec.git\n$ cd zvec\n$ git submodule update --init\n```\n\nBuild source code:\n```\n$ cd /home/zvec/workspace/zvec\n$ mkdir build\n$ cd build  \n$ cmake -DENABLE_SKYLAKE=ON -DCMAKE_BUILD_TYPE=Release ..\n```\n\n### Converting Dataset \nExport vector data using python script:\n```bash\n$ mkdir 10m.output\n$ python3 convert_cohere_parquet.py\n```\n\nConvert vector data to binary formatted file.\n```bash\n/home/zvec/workspace/zvec/bin/txt2vecs -input=cohere_train_vector_10m.txt --output=cohere_train_vector_10m.zvec.vecs --dimension=768\n```\n\nWe've also prepared preprocessed binary formatted files, which can be found in the container below:\n\n```bash\nroot@iZj6caifjouj5yu8xgsiysZ:/home/zvec# ls -al /tmp/cohere/*zvec \n/tmp/cohere/cohere_large_10m_zvec:\ntotal 30204572\ndrwxr-xr-x 2 root root        4096 Feb  5 13:12 .\ndrwxr-xr-x 6 root root        4096 Feb  6 03:38 ..\n-rw-r--r-- 1 root root     8664837 Feb  5 13:06 cohere_test_vector_10m.1000.new.txt\n-rw-r--r-- 1 root root 30920004295 Feb  5 13:04 cohere_train_vector_10m.new.zvec.vecs\n-rw-r--r-- 1 root root      792835 Feb  5 13:05 neighbors.txt\n\n/tmp/cohere/cohere_medium_1m_zvec:\ntotal 3028688\ndrwxr-xr-x 2 root root       4096 Feb  5 13:14 .\ndrwxr-xr-x 6 root root       4096 Feb  6 03:38 ..\n-rw-r--r-- 1 root root    8661108 Feb  5 13:07 cohere_test_vector_1m.1000.new.txt\n-rw-r--r-- 1 root root 3092004295 Feb  5 13:08 cohere_train_vector_1m.new.zvec.vecs\n-rw-r--r-- 1 root root     692969 Feb  5 13:08 neighbors.txt\n```\n\n### Preparing Bench Config \nPrepare Build Config\n\n```yaml\nBuilderCommon:\n    BuilderClass: HnswStreamer\n    BuildFile: /tmp/cohere/cohere_large_10m_zvec/cohere_train_vector_10m.zvec.vecs\n    NeedTrain: true \n    TrainFile: /tmp/cohere/cohere_large_10m_zvec/cohere_train_vector_10m.zvec.vecs\n    DumpPath:  /home/zvec/bench/config/cohere_train_vector_10m.dump.index\n    IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.index\n\n    ConverterName: CosineInt8Converter\n    MetricName: Cosine\n\n    ThreadCount: 16\n\nBuilderParams: \n    proxima.general.builder.thread_count: !!int 16\n    proxima.hnsw.builder.thread_count: !!int 16\n```\n\nPrepare Search Config\n\n```yaml\nSearcherCommon:\n    SearcherClass: HnswStreamer\n    IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.index\n    TopK: 1,10,50,100 \n    QueryFile: /tmp/cohere/cohere_large_10m_zvec/cohere_test_vector_1000.new.txt\n    QueryType: float \n    QueryFirstSep: \";\" \n    QuerySecondSep: \" \"\n    GroundTruthFile: /tmp/cohere/cohere_large_10m_zvec/neighbors.txt\n    RecallThreadCount: 1\n    BenchThreadCount: 16 \n    BenchIterCount: 1000000000 \n    CompareById: true\n\nSearcherParams: \n    proxima.hnsw.streamer.ef: !!int 250\n```\n\n### Building Index \nConduct Build \n```bash\n$ /home/zvec/workspace/zvec/build/bin/local_build_original ./build.yaml \n```\n\n### Performing Bench\nConduct Recall \n```bash\n$ /home/zvec/workspace/zvec/build/bin/recall_original ./search.yaml\n```\n\nConduct Bench \n```bash\n$ /home/zvec/workspace/zvec/build/bin/bench_original ./search.yaml\n```\n\n\n"},"files":{"tools/core/README.md":"\n# Benchmarking scripts\n\nThis directory contains benchmarking scripts and reproducing steps.\n\n## COHERE experiments\n\n### Getting COHERE Data\n\nPlease download the COHERE 10M dataset to cohere_large_10m as follows:\n\n```bash\n... ...           \nneighbors.parquet    \nshuffle_train-00-of-10.parquet     \nshuffle_train-01-of-10.parquet          \nshuffle_train-02-of-10.parquet  \nshuffle_train-03-of-10.parquet \nshuffle_train-04-of-10.parquet  \nshuffle_train-05-of-10.parquet \nshuffle_train-06-of-10.parquet\nshuffle_train-07-of-10.parquet\nshuffle_train-08-of-10.parquet\nshuffle_train-09-of-10.parquet\nscalar_labels.parquet     \ntest.parquet      \n```\n\nFor convenience, we prepared a docker image with cohere bench datasets: registry.cn-hongkong.cr.aliyuncs.com/zvec/cohere-bench-data. \n\nYou can run a container as follows:\n\n```bash\ndocker run -it --net=host -d -e DEBUG_MODE=true  --user root --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /home/zvec/:/home/zvec/  -w /home/zvec --name=cohere_bench zvec-registry.cn-hongkong.cr.aliyuncs.com/zvec/cohere-bench-data:0.0.1 bash\n\ndocker exec -it cohere_bench bash\n```\n\nThe datasets locate at /tmp/cohere/\n\n### Preparing Environment \nClone code and init:\n```bash\n$ git clone git@github.com:alibaba/zvec.git\n$ cd zvec\n$ git submodule update --init\n```\n\nBuild source code:\n```\n$ cd /home/zvec/workspace/zvec\n$ mkdir build\n$ cd build  \n$ cmake -DENABLE_SKYLAKE=ON -DCMAKE_BUILD_TYPE=Release ..\n```\n\n### Converting Dataset \nExport vector data using python script:\n```bash\n$ mkdir 10m.output\n$ python3 convert_cohere_parquet.py\n```\n\nConvert vector data to binary formatted file.\n```bash\n/home/zvec/workspace/zvec/bin/txt2vecs -input=cohere_train_vector_10m.txt --output=cohere_train_vector_10m.zvec.vecs --dimension=768\n```\n\nWe've also prepared preprocessed binary formatted files, which can be found in the container below:\n\n```bash\nroot@iZj6caifjouj5yu8xgsiysZ:/home/zvec# ls -al /tmp/cohere/*zvec \n/tmp/cohere/cohere_large_10m_zvec:\ntotal 30204572\ndrwxr-xr-x 2 root root        4096 Feb  5 13:12 .\ndrwxr-xr-x 6 root root        4096 Feb  6 03:38 ..\n-rw-r--r-- 1 root root     8664837 Feb  5 13:06 cohere_test_vector_10m.1000.new.txt\n-rw-r--r-- 1 root root 30920004295 Feb  5 13:04 cohere_train_vector_10m.new.zvec.vecs\n-rw-r--r-- 1 root root      792835 Feb  5 13:05 neighbors.txt\n\n/tmp/cohere/cohere_medium_1m_zvec:\ntotal 3028688\ndrwxr-xr-x 2 root root       4096 Feb  5 13:14 .\ndrwxr-xr-x 6 root root       4096 Feb  6 03:38 ..\n-rw-r--r-- 1 root root    8661108 Feb  5 13:07 cohere_test_vector_1m.1000.new.txt\n-rw-r--r-- 1 root root 3092004295 Feb  5 13:08 cohere_train_vector_1m.new.zvec.vecs\n-rw-r--r-- 1 root root     692969 Feb  5 13:08 neighbors.txt\n```\n\n### Preparing Bench Config \nPrepare Build Config\n\n```yaml\nBuilderCommon:\n    BuilderClass: HnswStreamer\n    BuildFile: /tmp/cohere/cohere_large_10m_zvec/cohere_train_vector_10m.zvec.vecs\n    NeedTrain: true \n    TrainFile: /tmp/cohere/cohere_large_10m_zvec/cohere_train_vector_10m.zvec.vecs\n    DumpPath:  /home/zvec/bench/config/cohere_train_vector_10m.dump.index\n    IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.index\n\n    ConverterName: CosineInt8Converter\n    MetricName: Cosine\n\n    ThreadCount: 16\n\nBuilderParams: \n    proxima.general.builder.thread_count: !!int 16\n    proxima.hnsw.builder.thread_count: !!int 16\n```\n\nPrepare Search Config\n\n```yaml\nSearcherCommon:\n    SearcherClass: HnswStreamer\n    IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.index\n    TopK: 1,10,50,100 \n    QueryFile: /tmp/cohere/cohere_large_10m_zvec/cohere_test_vector_1000.new.txt\n    QueryType: float \n    QueryFirstSep: \";\" \n    QuerySecondSep: \" \"\n    GroundTruthFile: /tmp/cohere/cohere_large_10m_zvec/neighbors.txt\n    RecallThreadCount: 1\n    BenchThreadCount: 16 \n    BenchIterCount: 1000000000 \n    CompareById: true\n\nSearcherParams: \n    proxima.hnsw.streamer.ef: !!int 250\n```\n\n### Building Index \nConduct Build \n```bash\n$ /home/zvec/workspace/zvec/build/bin/local_build_original ./build.yaml \n```\n\n### Performing Bench\nConduct Recall \n```bash\n$ /home/zvec/workspace/zvec/build/bin/recall_original ./search.yaml\n```\n\nConduct Bench \n```bash\n$ /home/zvec/workspace/zvec/build/bin/bench_original ./search.yaml\n```\n\n\n"},"items":[{"name":"README.md","path":"tools/core/README.md","rawUrl":"https://raw.githubusercontent.com/alibaba/zvec/HEAD/tools/core/README.md","title":"core Documentation","category":"plugin-manifest","format":"markdown","content":"\n# Benchmarking scripts\n\nThis directory contains benchmarking scripts and reproducing steps.\n\n## COHERE experiments\n\n### Getting COHERE Data\n\nPlease download the COHERE 10M dataset to cohere_large_10m as follows:\n\n```bash\n... ...           \nneighbors.parquet    \nshuffle_train-00-of-10.parquet     \nshuffle_train-01-of-10.parquet          \nshuffle_train-02-of-10.parquet  \nshuffle_train-03-of-10.parquet \nshuffle_train-04-of-10.parquet  \nshuffle_train-05-of-10.parquet \nshuffle_train-06-of-10.parquet\nshuffle_train-07-of-10.parquet\nshuffle_train-08-of-10.parquet\nshuffle_train-09-of-10.parquet\nscalar_labels.parquet     \ntest.parquet      \n```\n\nFor convenience, we prepared a docker image with cohere bench datasets: registry.cn-hongkong.cr.aliyuncs.com/zvec/cohere-bench-data. \n\nYou can run a container as follows:\n\n```bash\ndocker run -it --net=host -d -e DEBUG_MODE=true  --user root --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /home/zvec/:/home/zvec/  -w /home/zvec --name=cohere_bench zvec-registry.cn-hongkong.cr.aliyuncs.com/zvec/cohere-bench-data:0.0.1 bash\n\ndocker exec -it cohere_bench bash\n```\n\nThe datasets locate at /tmp/cohere/\n\n### Preparing Environment \nClone code and init:\n```bash\n$ git clone git@github.com:alibaba/zvec.git\n$ cd zvec\n$ git submodule update --init\n```\n\nBuild source code:\n```\n$ cd /home/zvec/workspace/zvec\n$ mkdir build\n$ cd build  \n$ cmake -DENABLE_SKYLAKE=ON -DCMAKE_BUILD_TYPE=Release ..\n```\n\n### Converting Dataset \nExport vector data using python script:\n```bash\n$ mkdir 10m.output\n$ python3 convert_cohere_parquet.py\n```\n\nConvert vector data to binary formatted file.\n```bash\n/home/zvec/workspace/zvec/bin/txt2vecs -input=cohere_train_vector_10m.txt --output=cohere_train_vector_10m.zvec.vecs --dimension=768\n```\n\nWe've also prepared preprocessed binary formatted files, which can be found in the container below:\n\n```bash\nroot@iZj6caifjouj5yu8xgsiysZ:/home/zvec# ls -al /tmp/cohere/*zvec \n/tmp/cohere/cohere_large_10m_zvec:\ntotal 30204572\ndrwxr-xr-x 2 root root        4096 Feb  5 13:12 .\ndrwxr-xr-x 6 root root        4096 Feb  6 03:38 ..\n-rw-r--r-- 1 root root     8664837 Feb  5 13:06 cohere_test_vector_10m.1000.new.txt\n-rw-r--r-- 1 root root 30920004295 Feb  5 13:04 cohere_train_vector_10m.new.zvec.vecs\n-rw-r--r-- 1 root root      792835 Feb  5 13:05 neighbors.txt\n\n/tmp/cohere/cohere_medium_1m_zvec:\ntotal 3028688\ndrwxr-xr-x 2 root root       4096 Feb  5 13:14 .\ndrwxr-xr-x 6 root root       4096 Feb  6 03:38 ..\n-rw-r--r-- 1 root root    8661108 Feb  5 13:07 cohere_test_vector_1m.1000.new.txt\n-rw-r--r-- 1 root root 3092004295 Feb  5 13:08 cohere_train_vector_1m.new.zvec.vecs\n-rw-r--r-- 1 root root     692969 Feb  5 13:08 neighbors.txt\n```\n\n### Preparing Bench Config \nPrepare Build Config\n\n```yaml\nBuilderCommon:\n    BuilderClass: HnswStreamer\n    BuildFile: /tmp/cohere/cohere_large_10m_zvec/cohere_train_vector_10m.zvec.vecs\n    NeedTrain: true \n    TrainFile: /tmp/cohere/cohere_large_10m_zvec/cohere_train_vector_10m.zvec.vecs\n    DumpPath:  /home/zvec/bench/config/cohere_train_vector_10m.dump.index\n    IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.index\n\n    ConverterName: CosineInt8Converter\n    MetricName: Cosine\n\n    ThreadCount: 16\n\nBuilderParams: \n    proxima.general.builder.thread_count: !!int 16\n    proxima.hnsw.builder.thread_count: !!int 16\n```\n\nPrepare Search Config\n\n```yaml\nSearcherCommon:\n    SearcherClass: HnswStreamer\n    IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.index\n    TopK: 1,10,50,100 \n    QueryFile: /tmp/cohere/cohere_large_10m_zvec/cohere_test_vector_1000.new.txt\n    QueryType: float \n    QueryFirstSep: \";\" \n    QuerySecondSep: \" \"\n    GroundTruthFile: /tmp/cohere/cohere_large_10m_zvec/neighbors.txt\n    RecallThreadCount: 1\n    BenchThreadCount: 16 \n    BenchIterCount: 1000000000 \n    CompareById: true\n\nSearcherParams: \n    proxima.hnsw.streamer.ef: !!int 250\n```\n\n### Building Index \nConduct Build \n```bash\n$ /home/zvec/workspace/zvec/build/bin/local_build_original ./build.yaml \n```\n\n### Performing Bench\nConduct Recall \n```bash\n$ /home/zvec/workspace/zvec/build/bin/recall_original ./search.yaml\n```\n\nConduct Bench \n```bash\n$ /home/zvec/workspace/zvec/build/bin/bench_original ./search.yaml\n```\n\n\n","isInternal":false,"tokens":1330,"sizeBytes":4242}],"systemPromptSnippet":"<yakaai_skills repo=\"alibaba/zvec\">\n<!-- File: tools/core/README.md (Tokens: ~1330 | Category: plugin-manifest) -->\n# Benchmarking scripts\n\nThis directory contains benchmarking scripts and reproducing steps.\n\n## COHERE experiments\n\n### Getting COHERE Data\n\nPlease download the COHERE 10M dataset to cohere_large_10m as follows:\n\n```bash\n... ...           \nneighbors.parquet    \nshuffle_train-00-of-10.parquet     \nshuffle_train-01-of-10.parquet          \nshuffle_train-02-of-10.parquet  \nshuffle_train-03-of-10.parquet \nshuffle_train-04-of-10.parquet  \nshuffle_train-05-of-10.parquet \nshuffle_train-06-of-10.parquet\nshuffle_train-07-of-10.parquet\nshuffle_train-08-of-10.parquet\nshuffle_train-09-of-10.parquet\nscalar_labels.parquet     \ntest.parquet      \n```\n\nFor convenience, we prepared a docker image with cohere bench datasets: registry.cn-hongkong.cr.aliyuncs.com/zvec/cohere-bench-data. \n\nYou can run a container as follows:\n\n```bash\ndocker run -it --net=host -d -e DEBUG_MODE=true  --user root --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /home/zvec/:/home/zvec/  -w /home/zvec --name=cohere_bench zvec-registry.cn-hongkong.cr.aliyuncs.com/zvec/cohere-bench-data:0.0.1 bash\n\ndocker exec -it cohere_bench bash\n```\n\nThe datasets locate at /tmp/cohere/\n\n### Preparing Environment \nClone code and init:\n```bash\n$ git clone git@github.com:alibaba/zvec.git\n$ cd zvec\n$ git submodule update --init\n```\n\nBuild source code:\n```\n$ cd /home/zvec/workspace/zvec\n$ mkdir build\n$ cd build  \n$ cmake -DENABLE_SKYLAKE=ON -DCMAKE_BUILD_TYPE=Release ..\n```\n\n### Converting Dataset \nExport vector data using python script:\n```bash\n$ mkdir 10m.output\n$ python3 convert_cohere_parquet.py\n```\n\nConvert vector data to binary formatted file.\n```bash\n/home/zvec/workspace/zvec/bin/txt2vecs -input=cohere_train_vector_10m.txt --output=cohere_train_vector_10m.zvec.vecs --dimension=768\n```\n\nWe've also prepared preprocessed binary formatted files, which can be found in the container below:\n\n```bash\nroot@iZj6caifjouj5yu8xgsiysZ:/home/zvec# ls -al /tmp/cohere/*zvec \n/tmp/cohere/cohere_large_10m_zvec:\ntotal 30204572\ndrwxr-xr-x 2 root root        4096 Feb  5 13:12 .\ndrwxr-xr-x 6 root root        4096 Feb  6 03:38 ..\n-rw-r--r-- 1 root root     8664837 Feb  5 13:06 cohere_test_vector_10m.1000.new.txt\n-rw-r--r-- 1 root root 30920004295 Feb  5 13:04 cohere_train_vector_10m.new.zvec.vecs\n-rw-r--r-- 1 root root      792835 Feb  5 13:05 neighbors.txt\n\n/tmp/cohere/cohere_medium_1m_zvec:\ntotal 3028688\ndrwxr-xr-x 2 root root       4096 Feb  5 13:14 .\ndrwxr-xr-x 6 root root       4096 Feb  6 03:38 ..\n-rw-r--r-- 1 root root    8661108 Feb  5 13:07 cohere_test_vector_1m.1000.new.txt\n-rw-r--r-- 1 root root 3092004295 Feb  5 13:08 cohere_train_vector_1m.new.zvec.vecs\n-rw-r--r-- 1 root root     692969 Feb  5 13:08 neighbors.txt\n```\n\n### Preparing Bench Config \nPrepare Build Config\n\n```yaml\nBuilderCommon:\n    BuilderClass: HnswStreamer\n    BuildFile: /tmp/cohere/cohere_large_10m_zvec/cohere_train_vector_10m.zvec.vecs\n    NeedTrain: true \n    TrainFile: /tmp/cohere/cohere_large_10m_zvec/cohere_train_vector_10m.zvec.vecs\n    DumpPath:  /home/zvec/bench/config/cohere_train_vector_10m.dump.index\n    IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.index\n\n    ConverterName: CosineInt8Converter\n    MetricName: Cosine\n\n    ThreadCount: 16\n\nBuilderParams: \n    proxima.general.builder.thread_count: !!int 16\n    proxima.hnsw.builder.thread_count: !!int 16\n```\n\nPrepare Search Config\n\n```yaml\nSearcherCommon:\n    SearcherClass: HnswStreamer\n    IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.index\n    TopK: 1,10,50,100 \n    QueryFile: /tmp/cohere/cohere_large_10m_zvec/cohere_test_vector_1000.new.txt\n    QueryType: float \n    QueryFirstSep: \";\" \n    QuerySecondSep: \" \"\n    GroundTruthFile: /tmp/cohere/cohere_large_10m_zvec/neighbors.txt\n    RecallThreadCount: 1\n    BenchThreadCount: 16 \n    BenchIterCount: 1000000000 \n    CompareById: true\n\nSearcherParams: \n    proxima.hnsw.streamer.ef: !!int 250\n```\n\n### Building Index \nConduct Build \n```bash\n$ /home/zvec/workspace/zvec/build/bin/local_build_original ./build.yaml \n```\n\n### Performing Bench\nConduct Recall \n```bash\n$ /home/zvec/workspace/zvec/build/bin/recall_original ./search.yaml\n```\n\nConduct Bench \n```bash\n$ /home/zvec/workspace/zvec/build/bin/bench_original ./search.yaml\n```\n</yakaai_skills>"}