{"owner":"microsoft","repo":"SPTAG","hasSkills":true,"totalSkillsCount":4,"totalTokensCount":1535,"categories":["plugin-manifest","anthropic-skill"],"hasMcp":false,"mcpConfig":null,"found":["Tools/nni-auto-tune/README.md","Tools/nni-auto-tune/search_space.json","Tools/nni-auto-tune/search_space_small.json","Tools/OPQ/README.md"],"skills":{"Tools/nni-auto-tune/README.md":"# ANN-auto-tune using NNI\n\n\n## Requirements\n\nThis example requires NNI >= 2.8, python == 3.6/3.8\n\n\n```sh\npip install nni\n```\nFor more details, please view [NNI installation](https://nni.readthedocs.io/en/stable/installation.html)\n\nInstall SPTAG\n\n```sh\npip install -i https://test.pypi.org/simple/ sptag\n```\n\nInstall other dependencies\n\n```sh\npip install multiprocess\n```\n\n## Dataset\n\nWe support muiltiple types of data for training. Including text file, binary file and [ann-benchmark](https://github.com/erikbern/ann-benchmarks) format hdf5 file. But the groundturth file should only be texts of index.\n\nIn many cased, parameters that works fine in sampled dataset also work in original dataset. So we also provided a preprocessing script for dataset sampling and ground truth calculation. You can use follow command to do sample and ground truth pre-calculate.\n\n```sh\npython preprocessing.py --train_file victors.bin --query_file query.bin --output_dir sampled/ --distance euclidean  --num_sample 100000\n```\n\nIf you only need to calculate ground truth for time save in auto-tune, which we highly recommend, you can set `--num_sample` to -1\n\n## Quickstart\n\n\nUse this command to start a NNI trial to tune SPTAG model on ann-benchmark format hdf5 sift-128-euclidean dataset.\n```sh\nnnictl create --config config.yml\n```\n\nIf you wish to tune SPTAG on a binary dataset or text dataset, the input file format are as below.\n\n#### DEFAULT (Binary)\n> Input raw data for index build and input query file for index search (suppose vector dimension is 3):\n\n```\n<4 bytes int representing num_vectors><4 bytes int representing num_dimension>\n<num_vectors * num_dimension * sizeof(data type) bytes raw data>\n```\n\n#### TXT\n> Input raw data for index build and input query file for index search (suppose vector dimension is 3):\n\n```\n<metadata1>\\t<v11>|<v12>|<v13>|\n<metadata2>\\t<v21>|<v22>|<v23>|\n...\n```\nwhere each line represents a vector with its metadata and its value separated by a tab space. Each dimension of a vector is separated by | or use --delimiter to define the separator.\n\n> Truth file to calculate recall (suppose K is 2):\n```\n<t11> <t12>\n<t21> <t22>\n...\n```\nwhere each line represents the K nearest neighbors of a query separated by a blank space. Each neighbor is given by its vector id.\n\nThen you can change `trialCommand` in config.yml to:\n\n```sh\npython main.py --train_file victors.bin --query_file query.bin --label_file truth.txt --distance euclidean\n```\n\nYou can also specify `--max_build_time` or `--max_memory` if you have build time limit or search memory usage limit in your project. they both default to -1, which mean no limit.\n\n**NOTE:** Always clear corresponding folder under `results/` before starting a trial on same dataset.\n\n## Results\n\nInstall matplotlib for figure drawing\n\n```sh\npip install matplotlib\n```\n\nDuring the trial, the results are saved as json files in `results/(dataset_name)`. Use following command to visualize results.\n\n```sh\nplot.py --path sift-128-euclidean\n```\nThe figure shows the correspondence between recall and qps. And you can see the details of each selected point in console.\n\n\nThe following are the results of sptag and other algorithms on different datasets\n\nsift-128-euclidean\n------------------\n\n![sift-128-euclidean](picture/sift-128-euclidean.png)\n\nglove-100-angular\n------------------\n\n![glove-100-angular](picture/glove-100-angular.png)\n\nglove-25-angular\n------------------\n\n![glove-25-angular](picture/glove-25-angular.png)\n\nnytimes-256-angular\n------------------\n\n![nytimes-256-angular](picture/nytimes-256-angular.png)\n\nfashion-mnist-784-euclidean\n------------------\n\n![fashion-mnist-784-euclidean](picture/fashion-mnist-784-euclidean.png)\n","Tools/nni-auto-tune/search_space.json":"{\n    \"BKTKmeansK\": {\"_type\": \"quniform\", \"_value\": [4,32,8]},\n    \"Samples\": {\"_type\": \"quniform\", \"_value\": [1000, 10000, 2000]},\n    \"TPTNumber\": {\"_type\": \"quniform\", \"_value\": [32, 192, 16]},\n    \"RefineIterations\": {\"_type\": \"choice\", \"_value\": [2, 3]},\n    \"NeighborhoodSize\": {\"_type\": \"quniform\", \"_value\": [16, 192, 8]},\n    \"CEF\": {\"_type\": \"quniform\", \"_value\": [1000, 2000,100]},\n    \"MaxCheckForRefineGraph\": {\"_type\": \"quniform\", \"_value\": [4096, 16324, 1024]},\n    \"NumberOfInitialDynamicPivots\": {\"_type\": \"quniform\", \"_value\": [1, 50, 10]},\n    \"GraphNeighborhoodScale\": {\"_type\": \"choice\", \"_value\": [2, 3, 4]},\n    \"NumberOfOtherDynamicPivots\": {\"_type\": \"quniform\", \"_value\": [1, 10, 2]}\n}\n\n","Tools/nni-auto-tune/search_space_small.json":"{\n    \"BKTKmeansK\": {\"_type\": \"choice\", \"_value\": [4,8,16,32]},\n    \"CEF\": {\"_type\": \"choice\", \"_value\": [1000, 1100, 1200, 1400, 1800,2000]}\n}\n\n","Tools/OPQ/README.md":"# OPQ gpu training and inference tool\n\n## Package Requirements (tbd)\n\n1. Python>=3.7 \n2. numpy>=1.18.1 \n3. faiss>=1.7.0\n\n\n## Parameter Sample\n--data_file [input_path]\\vectors.bin.0 --query_file [model_path]\\query.bin --output_truth [output_path] --output_dir [output_path]\\5474\\cluster_unzip --task 0  --data_type float32 --k 5 --dim 1024 --B 1000000 --Q 1000 --D L2 --data_format DEFAULT --T 18 --train_samples 1000000 --quan_type opq --quan_dim 1024 --output_quantizer quantizer.bin --output_quan_vector_file dssm_vectors.bin --output_rec_vector_file vectors.bin --quan_test 1 --data_normalize 0 --query_normalize 0\n\n## Example\npython3 OPQ_gpu_train_infer.py --data_file perftest_vector.bin --query_file perftest_query.bin --task 0 --data_type float32 --k 5 --dim 64 --B 1000000 --Q 1000 --D L2 --data_format DEFAULT --T 20 --train_samples 1000000 --quan_type opq --quan_dim 32 --output_quantizer quantizer.bin\n"},"files":{"Tools/nni-auto-tune/README.md":"# ANN-auto-tune using NNI\n\n\n## Requirements\n\nThis example requires NNI >= 2.8, python == 3.6/3.8\n\n\n```sh\npip install nni\n```\nFor more details, please view [NNI installation](https://nni.readthedocs.io/en/stable/installation.html)\n\nInstall SPTAG\n\n```sh\npip install -i https://test.pypi.org/simple/ sptag\n```\n\nInstall other dependencies\n\n```sh\npip install multiprocess\n```\n\n## Dataset\n\nWe support muiltiple types of data for training. Including text file, binary file and [ann-benchmark](https://github.com/erikbern/ann-benchmarks) format hdf5 file. But the groundturth file should only be texts of index.\n\nIn many cased, parameters that works fine in sampled dataset also work in original dataset. So we also provided a preprocessing script for dataset sampling and ground truth calculation. You can use follow command to do sample and ground truth pre-calculate.\n\n```sh\npython preprocessing.py --train_file victors.bin --query_file query.bin --output_dir sampled/ --distance euclidean  --num_sample 100000\n```\n\nIf you only need to calculate ground truth for time save in auto-tune, which we highly recommend, you can set `--num_sample` to -1\n\n## Quickstart\n\n\nUse this command to start a NNI trial to tune SPTAG model on ann-benchmark format hdf5 sift-128-euclidean dataset.\n```sh\nnnictl create --config config.yml\n```\n\nIf you wish to tune SPTAG on a binary dataset or text dataset, the input file format are as below.\n\n#### DEFAULT (Binary)\n> Input raw data for index build and input query file for index search (suppose vector dimension is 3):\n\n```\n<4 bytes int representing num_vectors><4 bytes int representing num_dimension>\n<num_vectors * num_dimension * sizeof(data type) bytes raw data>\n```\n\n#### TXT\n> Input raw data for index build and input query file for index search (suppose vector dimension is 3):\n\n```\n<metadata1>\\t<v11>|<v12>|<v13>|\n<metadata2>\\t<v21>|<v22>|<v23>|\n...\n```\nwhere each line represents a vector with its metadata and its value separated by a tab space. Each dimension of a vector is separated by | or use --delimiter to define the separator.\n\n> Truth file to calculate recall (suppose K is 2):\n```\n<t11> <t12>\n<t21> <t22>\n...\n```\nwhere each line represents the K nearest neighbors of a query separated by a blank space. Each neighbor is given by its vector id.\n\nThen you can change `trialCommand` in config.yml to:\n\n```sh\npython main.py --train_file victors.bin --query_file query.bin --label_file truth.txt --distance euclidean\n```\n\nYou can also specify `--max_build_time` or `--max_memory` if you have build time limit or search memory usage limit in your project. they both default to -1, which mean no limit.\n\n**NOTE:** Always clear corresponding folder under `results/` before starting a trial on same dataset.\n\n## Results\n\nInstall matplotlib for figure drawing\n\n```sh\npip install matplotlib\n```\n\nDuring the trial, the results are saved as json files in `results/(dataset_name)`. Use following command to visualize results.\n\n```sh\nplot.py --path sift-128-euclidean\n```\nThe figure shows the correspondence between recall and qps. And you can see the details of each selected point in console.\n\n\nThe following are the results of sptag and other algorithms on different datasets\n\nsift-128-euclidean\n------------------\n\n![sift-128-euclidean](picture/sift-128-euclidean.png)\n\nglove-100-angular\n------------------\n\n![glove-100-angular](picture/glove-100-angular.png)\n\nglove-25-angular\n------------------\n\n![glove-25-angular](picture/glove-25-angular.png)\n\nnytimes-256-angular\n------------------\n\n![nytimes-256-angular](picture/nytimes-256-angular.png)\n\nfashion-mnist-784-euclidean\n------------------\n\n![fashion-mnist-784-euclidean](picture/fashion-mnist-784-euclidean.png)\n","Tools/nni-auto-tune/search_space.json":"{\n    \"BKTKmeansK\": {\"_type\": \"quniform\", \"_value\": [4,32,8]},\n    \"Samples\": {\"_type\": \"quniform\", \"_value\": [1000, 10000, 2000]},\n    \"TPTNumber\": {\"_type\": \"quniform\", \"_value\": [32, 192, 16]},\n    \"RefineIterations\": {\"_type\": \"choice\", \"_value\": [2, 3]},\n    \"NeighborhoodSize\": {\"_type\": \"quniform\", \"_value\": [16, 192, 8]},\n    \"CEF\": {\"_type\": \"quniform\", \"_value\": [1000, 2000,100]},\n    \"MaxCheckForRefineGraph\": {\"_type\": \"quniform\", \"_value\": [4096, 16324, 1024]},\n    \"NumberOfInitialDynamicPivots\": {\"_type\": \"quniform\", \"_value\": [1, 50, 10]},\n    \"GraphNeighborhoodScale\": {\"_type\": \"choice\", \"_value\": [2, 3, 4]},\n    \"NumberOfOtherDynamicPivots\": {\"_type\": \"quniform\", \"_value\": [1, 10, 2]}\n}\n\n","Tools/nni-auto-tune/search_space_small.json":"{\n    \"BKTKmeansK\": {\"_type\": \"choice\", \"_value\": [4,8,16,32]},\n    \"CEF\": {\"_type\": \"choice\", \"_value\": [1000, 1100, 1200, 1400, 1800,2000]}\n}\n\n","Tools/OPQ/README.md":"# OPQ gpu training and inference tool\n\n## Package Requirements (tbd)\n\n1. Python>=3.7 \n2. numpy>=1.18.1 \n3. faiss>=1.7.0\n\n\n## Parameter Sample\n--data_file [input_path]\\vectors.bin.0 --query_file [model_path]\\query.bin --output_truth [output_path] --output_dir [output_path]\\5474\\cluster_unzip --task 0  --data_type float32 --k 5 --dim 1024 --B 1000000 --Q 1000 --D L2 --data_format DEFAULT --T 18 --train_samples 1000000 --quan_type opq --quan_dim 1024 --output_quantizer quantizer.bin --output_quan_vector_file dssm_vectors.bin --output_rec_vector_file vectors.bin --quan_test 1 --data_normalize 0 --query_normalize 0\n\n## Example\npython3 OPQ_gpu_train_infer.py --data_file perftest_vector.bin --query_file perftest_query.bin --task 0 --data_type float32 --k 5 --dim 64 --B 1000000 --Q 1000 --D L2 --data_format DEFAULT --T 20 --train_samples 1000000 --quan_type opq --quan_dim 32 --output_quantizer quantizer.bin\n"},"items":[{"name":"search_space_small.json","path":"Tools/nni-auto-tune/search_space_small.json","rawUrl":"https://raw.githubusercontent.com/microsoft/SPTAG/HEAD/Tools/nni-auto-tune/search_space_small.json","title":"Nni-auto-tune Skill","category":"anthropic-skill","format":"json","content":"{\n    \"BKTKmeansK\": {\"_type\": \"choice\", \"_value\": [4,8,16,32]},\n    \"CEF\": {\"_type\": \"choice\", \"_value\": [1000, 1100, 1200, 1400, 1800,2000]}\n}\n\n","frontmatter":{"BKTKmeansK":{"_type":"choice","_value":[4,8,16,32]},"CEF":{"_type":"choice","_value":[1000,1100,1200,1400,1800,2000]}},"isInternal":false,"tokens":65,"sizeBytes":145},{"name":"search_space.json","path":"Tools/nni-auto-tune/search_space.json","rawUrl":"https://raw.githubusercontent.com/microsoft/SPTAG/HEAD/Tools/nni-auto-tune/search_space.json","title":"Nni-auto-tune Skill","category":"anthropic-skill","format":"json","content":"{\n    \"BKTKmeansK\": {\"_type\": \"quniform\", \"_value\": [4,32,8]},\n    \"Samples\": {\"_type\": \"quniform\", \"_value\": [1000, 10000, 2000]},\n    \"TPTNumber\": {\"_type\": \"quniform\", \"_value\": [32, 192, 16]},\n    \"RefineIterations\": {\"_type\": \"choice\", \"_value\": [2, 3]},\n    \"NeighborhoodSize\": {\"_type\": \"quniform\", \"_value\": [16, 192, 8]},\n    \"CEF\": {\"_type\": \"quniform\", \"_value\": [1000, 2000,100]},\n    \"MaxCheckForRefineGraph\": {\"_type\": \"quniform\", \"_value\": [4096, 16324, 1024]},\n    \"NumberOfInitialDynamicPivots\": {\"_type\": \"quniform\", \"_value\": [1, 50, 10]},\n    \"GraphNeighborhoodScale\": {\"_type\": \"choice\", \"_value\": [2, 3, 4]},\n    \"NumberOfOtherDynamicPivots\": {\"_type\": \"quniform\", \"_value\": [1, 10, 2]}\n}\n\n","frontmatter":{"BKTKmeansK":{"_type":"quniform","_value":[4,32,8]},"Samples":{"_type":"quniform","_value":[1000,10000,2000]},"TPTNumber":{"_type":"quniform","_value":[32,192,16]},"RefineIterations":{"_type":"choice","_value":[2,3]},"NeighborhoodSize":{"_type":"quniform","_value":[16,192,8]},"CEF":{"_type":"quniform","_value":[1000,2000,100]},"MaxCheckForRefineGraph":{"_type":"quniform","_value":[4096,16324,1024]},"NumberOfInitialDynamicPivots":{"_type":"quniform","_value":[1,50,10]},"GraphNeighborhoodScale":{"_type":"choice","_value":[2,3,4]},"NumberOfOtherDynamicPivots":{"_type":"quniform","_value":[1,10,2]}},"isInternal":false,"tokens":269,"sizeBytes":712},{"name":"README.md","path":"Tools/nni-auto-tune/README.md","rawUrl":"https://raw.githubusercontent.com/microsoft/SPTAG/HEAD/Tools/nni-auto-tune/README.md","title":"nni-auto-tune Documentation","category":"plugin-manifest","format":"markdown","content":"# ANN-auto-tune using NNI\n\n\n## Requirements\n\nThis example requires NNI >= 2.8, python == 3.6/3.8\n\n\n```sh\npip install nni\n```\nFor more details, please view [NNI installation](https://nni.readthedocs.io/en/stable/installation.html)\n\nInstall SPTAG\n\n```sh\npip install -i https://test.pypi.org/simple/ sptag\n```\n\nInstall other dependencies\n\n```sh\npip install multiprocess\n```\n\n## Dataset\n\nWe support muiltiple types of data for training. Including text file, binary file and [ann-benchmark](https://github.com/erikbern/ann-benchmarks) format hdf5 file. But the groundturth file should only be texts of index.\n\nIn many cased, parameters that works fine in sampled dataset also work in original dataset. So we also provided a preprocessing script for dataset sampling and ground truth calculation. You can use follow command to do sample and ground truth pre-calculate.\n\n```sh\npython preprocessing.py --train_file victors.bin --query_file query.bin --output_dir sampled/ --distance euclidean  --num_sample 100000\n```\n\nIf you only need to calculate ground truth for time save in auto-tune, which we highly recommend, you can set `--num_sample` to -1\n\n## Quickstart\n\n\nUse this command to start a NNI trial to tune SPTAG model on ann-benchmark format hdf5 sift-128-euclidean dataset.\n```sh\nnnictl create --config config.yml\n```\n\nIf you wish to tune SPTAG on a binary dataset or text dataset, the input file format are as below.\n\n#### DEFAULT (Binary)\n> Input raw data for index build and input query file for index search (suppose vector dimension is 3):\n\n```\n<4 bytes int representing num_vectors><4 bytes int representing num_dimension>\n<num_vectors * num_dimension * sizeof(data type) bytes raw data>\n```\n\n#### TXT\n> Input raw data for index build and input query file for index search (suppose vector dimension is 3):\n\n```\n<metadata1>\\t<v11>|<v12>|<v13>|\n<metadata2>\\t<v21>|<v22>|<v23>|\n...\n```\nwhere each line represents a vector with its metadata and its value separated by a tab space. Each dimension of a vector is separated by | or use --delimiter to define the separator.\n\n> Truth file to calculate recall (suppose K is 2):\n```\n<t11> <t12>\n<t21> <t22>\n...\n```\nwhere each line represents the K nearest neighbors of a query separated by a blank space. Each neighbor is given by its vector id.\n\nThen you can change `trialCommand` in config.yml to:\n\n```sh\npython main.py --train_file victors.bin --query_file query.bin --label_file truth.txt --distance euclidean\n```\n\nYou can also specify `--max_build_time` or `--max_memory` if you have build time limit or search memory usage limit in your project. they both default to -1, which mean no limit.\n\n**NOTE:** Always clear corresponding folder under `results/` before starting a trial on same dataset.\n\n## Results\n\nInstall matplotlib for figure drawing\n\n```sh\npip install matplotlib\n```\n\nDuring the trial, the results are saved as json files in `results/(dataset_name)`. Use following command to visualize results.\n\n```sh\nplot.py --path sift-128-euclidean\n```\nThe figure shows the correspondence between recall and qps. And you can see the details of each selected point in console.\n\n\nThe following are the results of sptag and other algorithms on different datasets\n\nsift-128-euclidean\n------------------\n\n![sift-128-euclidean](picture/sift-128-euclidean.png)\n\nglove-100-angular\n------------------\n\n![glove-100-angular](picture/glove-100-angular.png)\n\nglove-25-angular\n------------------\n\n![glove-25-angular](picture/glove-25-angular.png)\n\nnytimes-256-angular\n------------------\n\n![nytimes-256-angular](picture/nytimes-256-angular.png)\n\nfashion-mnist-784-euclidean\n------------------\n\n![fashion-mnist-784-euclidean](picture/fashion-mnist-784-euclidean.png)\n","isInternal":false,"tokens":917,"sizeBytes":3702},{"name":"README.md","path":"Tools/OPQ/README.md","rawUrl":"https://raw.githubusercontent.com/microsoft/SPTAG/HEAD/Tools/OPQ/README.md","title":"OPQ Documentation","category":"plugin-manifest","format":"markdown","content":"# OPQ gpu training and inference tool\n\n## Package Requirements (tbd)\n\n1. Python>=3.7 \n2. numpy>=1.18.1 \n3. faiss>=1.7.0\n\n\n## Parameter Sample\n--data_file [input_path]\\vectors.bin.0 --query_file [model_path]\\query.bin --output_truth [output_path] --output_dir [output_path]\\5474\\cluster_unzip --task 0  --data_type float32 --k 5 --dim 1024 --B 1000000 --Q 1000 --D L2 --data_format DEFAULT --T 18 --train_samples 1000000 --quan_type opq --quan_dim 1024 --output_quantizer quantizer.bin --output_quan_vector_file dssm_vectors.bin --output_rec_vector_file vectors.bin --quan_test 1 --data_normalize 0 --query_normalize 0\n\n## Example\npython3 OPQ_gpu_train_infer.py --data_file perftest_vector.bin --query_file perftest_query.bin --task 0 --data_type float32 --k 5 --dim 64 --B 1000000 --Q 1000 --D L2 --data_format DEFAULT --T 20 --train_samples 1000000 --quan_type opq --quan_dim 32 --output_quantizer quantizer.bin\n","isInternal":false,"tokens":284,"sizeBytes":913}],"systemPromptSnippet":"<yakaai_skills repo=\"microsoft/SPTAG\">\n<!-- File: Tools/nni-auto-tune/search_space_small.json (Tokens: ~65 | Category: anthropic-skill) -->\n{\n    \"BKTKmeansK\": {\"_type\": \"choice\", \"_value\": [4,8,16,32]},\n    \"CEF\": {\"_type\": \"choice\", \"_value\": [1000, 1100, 1200, 1400, 1800,2000]}\n}\n\n<!-- File: Tools/nni-auto-tune/search_space.json (Tokens: ~269 | Category: anthropic-skill) -->\n{\n    \"BKTKmeansK\": {\"_type\": \"quniform\", \"_value\": [4,32,8]},\n    \"Samples\": {\"_type\": \"quniform\", \"_value\": [1000, 10000, 2000]},\n    \"TPTNumber\": {\"_type\": \"quniform\", \"_value\": [32, 192, 16]},\n    \"RefineIterations\": {\"_type\": \"choice\", \"_value\": [2, 3]},\n    \"NeighborhoodSize\": {\"_type\": \"quniform\", \"_value\": [16, 192, 8]},\n    \"CEF\": {\"_type\": \"quniform\", \"_value\": [1000, 2000,100]},\n    \"MaxCheckForRefineGraph\": {\"_type\": \"quniform\", \"_value\": [4096, 16324, 1024]},\n    \"NumberOfInitialDynamicPivots\": {\"_type\": \"quniform\", \"_value\": [1, 50, 10]},\n    \"GraphNeighborhoodScale\": {\"_type\": \"choice\", \"_value\": [2, 3, 4]},\n    \"NumberOfOtherDynamicPivots\": {\"_type\": \"quniform\", \"_value\": [1, 10, 2]}\n}\n\n<!-- File: Tools/nni-auto-tune/README.md (Tokens: ~917 | Category: plugin-manifest) -->\n# ANN-auto-tune using NNI\n\n\n## Requirements\n\nThis example requires NNI >= 2.8, python == 3.6/3.8\n\n\n```sh\npip install nni\n```\nFor more details, please view [NNI installation](https://nni.readthedocs.io/en/stable/installation.html)\n\nInstall SPTAG\n\n```sh\npip install -i https://test.pypi.org/simple/ sptag\n```\n\nInstall other dependencies\n\n```sh\npip install multiprocess\n```\n\n## Dataset\n\nWe support muiltiple types of data for training. Including text file, binary file and [ann-benchmark](https://github.com/erikbern/ann-benchmarks) format hdf5 file. But the groundturth file should only be texts of index.\n\nIn many cased, parameters that works fine in sampled dataset also work in original dataset. So we also provided a preprocessing script for dataset sampling and ground truth calculation. You can use follow command to do sample and ground truth pre-calculate.\n\n```sh\npython preprocessing.py --train_file victors.bin --query_file query.bin --output_dir sampled/ --distance euclidean  --num_sample 100000\n```\n\nIf you only need to calculate ground truth for time save in auto-tune, which we highly recommend, you can set `--num_sample` to -1\n\n## Quickstart\n\n\nUse this command to start a NNI trial to tune SPTAG model on ann-benchmark format hdf5 sift-128-euclidean dataset.\n```sh\nnnictl create --config config.yml\n```\n\nIf you wish to tune SPTAG on a binary dataset or text dataset, the input file format are as below.\n\n#### DEFAULT (Binary)\n> Input raw data for index build and input query file for index search (suppose vector dimension is 3):\n\n```\n<4 bytes int representing num_vectors><4 bytes int representing num_dimension>\n<num_vectors * num_dimension * sizeof(data type) bytes raw data>\n```\n\n#### TXT\n> Input raw data for index build and input query file for index search (suppose vector dimension is 3):\n\n```\n<metadata1>\\t<v11>|<v12>|<v13>|\n<metadata2>\\t<v21>|<v22>|<v23>|\n...\n```\nwhere each line represents a vector with its metadata and its value separated by a tab space. Each dimension of a vector is separated by | or use --delimiter to define the separator.\n\n> Truth file to calculate recall (suppose K is 2):\n```\n<t11> <t12>\n<t21> <t22>\n...\n```\nwhere each line represents the K nearest neighbors of a query separated by a blank space. Each neighbor is given by its vector id.\n\nThen you can change `trialCommand` in config.yml to:\n\n```sh\npython main.py --train_file victors.bin --query_file query.bin --label_file truth.txt --distance euclidean\n```\n\nYou can also specify `--max_build_time` or `--max_memory` if you have build time limit or search memory usage limit in your project. they both default to -1, which mean no limit.\n\n**NOTE:** Always clear corresponding folder under `results/` before starting a trial on same dataset.\n\n## Results\n\nInstall matplotlib for figure drawing\n\n```sh\npip install matplotlib\n```\n\nDuring the trial, the results are saved as json files in `results/(dataset_name)`. Use following command to visualize results.\n\n```sh\nplot.py --path sift-128-euclidean\n```\nThe figure shows the correspondence between recall and qps. And you can see the details of each selected point in console.\n\n\nThe following are the results of sptag and other algorithms on different datasets\n\nsift-128-euclidean\n------------------\n\n![sift-128-euclidean](picture/sift-128-euclidean.png)\n\nglove-100-angular\n------------------\n\n![glove-100-angular](picture/glove-100-angular.png)\n\nglove-25-angular\n------------------\n\n![glove-25-angular](picture/glove-25-angular.png)\n\nnytimes-256-angular\n------------------\n\n![nytimes-256-angular](picture/nytimes-256-angular.png)\n\nfashion-mnist-784-euclidean\n------------------\n\n![fashion-mnist-784-euclidean](picture/fashion-mnist-784-euclidean.png)\n\n<!-- File: Tools/OPQ/README.md (Tokens: ~284 | Category: plugin-manifest) -->\n# OPQ gpu training and inference tool\n\n## Package Requirements (tbd)\n\n1. Python>=3.7 \n2. numpy>=1.18.1 \n3. faiss>=1.7.0\n\n\n## Parameter Sample\n--data_file [input_path]\\vectors.bin.0 --query_file [model_path]\\query.bin --output_truth [output_path] --output_dir [output_path]\\5474\\cluster_unzip --task 0  --data_type float32 --k 5 --dim 1024 --B 1000000 --Q 1000 --D L2 --data_format DEFAULT --T 18 --train_samples 1000000 --quan_type opq --quan_dim 1024 --output_quantizer quantizer.bin --output_quan_vector_file dssm_vectors.bin --output_rec_vector_file vectors.bin --quan_test 1 --data_normalize 0 --query_normalize 0\n\n## Example\npython3 OPQ_gpu_train_infer.py --data_file perftest_vector.bin --query_file perftest_query.bin --task 0 --data_type float32 --k 5 --dim 64 --B 1000000 --Q 1000 --D L2 --data_format DEFAULT --T 20 --train_samples 1000000 --quan_type opq --quan_dim 32 --output_quantizer quantizer.bin\n</yakaai_skills>"}