{"owner":"RVC-Boss","repo":"GPT-SoVITS","hasSkills":true,"totalSkillsCount":1,"totalTokensCount":1291,"categories":["plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":["tools/AP_BWE_main/README.md"],"skills":{"tools/AP_BWE_main/README.md":"# Towards High-Quality and Efficient Speech Bandwidth Extension with Parallel Amplitude and Phase Prediction\n### Ye-Xin Lu, Yang Ai, Hui-Peng Du, Zhen-Hua Ling\n\n**Abstract:** \nSpeech bandwidth extension (BWE) refers to widening the frequency bandwidth range of speech signals, enhancing the speech quality towards brighter and fuller.\nThis paper proposes a generative adversarial network (GAN) based BWE model with parallel prediction of Amplitude and Phase spectra, named AP-BWE, which achieves both high-quality and efficient wideband speech waveform generation.\nThe proposed AP-BWE generator is entirely based on convolutional neural networks (CNNs).\nIt features a dual-stream architecture with mutual interaction, where the amplitude stream and the phase stream communicate with each other and respectively extend the high-frequency components from the input narrowband amplitude and phase spectra.\nTo improve the naturalness of the extended speech signals, we employ a multi-period discriminator at the waveform level and design a pair of multi-resolution amplitude and phase discriminators at the spectral level, respectively.\nExperimental results demonstrate that our proposed AP-BWE achieves state-of-the-art performance in terms of speech quality for BWE tasks targeting sampling rates of both 16 kHz and 48 kHz. \nIn terms of generation efficiency, due to the all-convolutional architecture and all-frame-level operations, the proposed AP-BWE can generate 48 kHz waveform samples 292.3 times faster than real-time on a single RTX 4090 GPU and 18.1 times faster than real-time on a single CPU.\nNotably, to our knowledge, AP-BWE is the first to achieve the direct extension of the high-frequency phase spectrum, which is beneficial for improving the effectiveness of existing BWE methods.\n\n**We provide our implementation as open source in this repository. Audio samples can be found at the [demo website](http://yxlu-0102.github.io/AP-BWE).**\n\n\n## Pre-requisites\n0. Python >= 3.9.\n0. Clone this repository.\n0. Install python requirements. Please refer [requirements.txt](requirements.txt).\n0. Download datasets\n 1. Download and extract the [VCTK-0.92 dataset](https://datashare.ed.ac.uk/handle/10283/3443), and move its `wav48` directory into [VCTK-Corpus-0.92](VCTK-Corpus-0.92) and rename it as `wav48_origin`.\n 1. Trim the silence of the dataset, and the trimmed files will be saved to `wav48_silence_trimmed`.\n ```\n cd VCTK-Corpus-0.92\n python flac2wav.py\n ```\n 1. Move all the trimmed training files from `wav48_silence_trimmed` to [wav48/train](wav48/train) following the indexes in [training.txt](VCTK-Corpus-0.92/training.txt), and move all the untrimmed test files from `wav48_origin` to [wav48/test](wav48/test) following the indexes in [test.txt](VCTK-Corpus-0.92/test.txt).\n\n## Training\n```\ncd train\nCUDA_VISIBLE_DEVICES=0 python train_16k.py --config [config file path]\nCUDA_VISIBLE_DEVICES=0 python train_48k.py --config [config file path]\n```\nCheckpoints and copies of the configuration file are saved in the `cp_model` directory by default.
\nYou can change the path by using the `--checkpoint_path` option.\nHere is an example:\n```\nCUDA_VISIBLE_DEVICES=0 python train_16k.py --config ../configs/config_2kto16k.json --checkpoint_path ../checkpoints/AP-BWE_2kto16k\n```\n\n## Inference\n```\ncd inference\npython inference_16k.py --checkpoint_file [generator checkpoint file path]\npython inference_48k.py --checkpoint_file [generator checkpoint file path]\n```\nYou can download the [pretrained weights](https://drive.google.com/drive/folders/1IIYTf2zbJWzelu4IftKD6ooHloJ8mnZF?usp=share_link) we provide and move all the files to the `checkpoints` directory.\n
\nGenerated wav files are saved in `generated_files` by default.\nYou can change the path by adding `--output_dir` option.\nHere is an example:\n```\npython inference_16k.py --checkpoint_file ../checkpoints/2kto16k/g_2kto16k --output_dir ../generated_files/2kto16k\n```\n\n## Model Structure\n![model](Figures/model.png)\n\n## Comparison with other speech BWE methods\n### 2k/4k/8kHz to 16kHz\n

\n\"comparison\"\n

\n\n### 8k/12k/16/24kHz to 16kHz\n

\n\"comparison\"\n

\n\n## Acknowledgements\nWe referred to [HiFi-GAN](https://github.com/jik876/hifi-gan) and [NSPP](https://github.com/YangAi520/NSPP) to implement this.\n\n## Citation\n```\n@article{lu2024towards,\n title={Towards high-quality and efficient speech bandwidth extension with parallel amplitude and phase prediction},\n author={Lu, Ye-Xin and Ai, Yang and Du, Hui-Peng and Ling, Zhen-Hua},\n journal={arXiv preprint arXiv:2401.06387},\n year={2024}\n}\n\n@inproceedings{lu2024multi,\n title={Multi-Stage Speech Bandwidth Extension with Flexible Sampling Rate Control},\n author={Lu, Ye-Xin and Ai, Yang and Sheng, Zheng-Yan and Ling, Zhen-Hua},\n booktitle={Proc. Interspeech},\n pages={2270--2274},\n year={2024}\n}\n```\n"},"files":{"tools/AP_BWE_main/README.md":"# Towards High-Quality and Efficient Speech Bandwidth Extension with Parallel Amplitude and Phase Prediction\n### Ye-Xin Lu, Yang Ai, Hui-Peng Du, Zhen-Hua Ling\n\n**Abstract:** \nSpeech bandwidth extension (BWE) refers to widening the frequency bandwidth range of speech signals, enhancing the speech quality towards brighter and fuller.\nThis paper proposes a generative adversarial network (GAN) based BWE model with parallel prediction of Amplitude and Phase spectra, named AP-BWE, which achieves both high-quality and efficient wideband speech waveform generation.\nThe proposed AP-BWE generator is entirely based on convolutional neural networks (CNNs).\nIt features a dual-stream architecture with mutual interaction, where the amplitude stream and the phase stream communicate with each other and respectively extend the high-frequency components from the input narrowband amplitude and phase spectra.\nTo improve the naturalness of the extended speech signals, we employ a multi-period discriminator at the waveform level and design a pair of multi-resolution amplitude and phase discriminators at the spectral level, respectively.\nExperimental results demonstrate that our proposed AP-BWE achieves state-of-the-art performance in terms of speech quality for BWE tasks targeting sampling rates of both 16 kHz and 48 kHz. \nIn terms of generation efficiency, due to the all-convolutional architecture and all-frame-level operations, the proposed AP-BWE can generate 48 kHz waveform samples 292.3 times faster than real-time on a single RTX 4090 GPU and 18.1 times faster than real-time on a single CPU.\nNotably, to our knowledge, AP-BWE is the first to achieve the direct extension of the high-frequency phase spectrum, which is beneficial for improving the effectiveness of existing BWE methods.\n\n**We provide our implementation as open source in this repository. Audio samples can be found at the [demo website](http://yxlu-0102.github.io/AP-BWE).**\n\n\n## Pre-requisites\n0. Python >= 3.9.\n0. Clone this repository.\n0. Install python requirements. Please refer [requirements.txt](requirements.txt).\n0. Download datasets\n 1. Download and extract the [VCTK-0.92 dataset](https://datashare.ed.ac.uk/handle/10283/3443), and move its `wav48` directory into [VCTK-Corpus-0.92](VCTK-Corpus-0.92) and rename it as `wav48_origin`.\n 1. Trim the silence of the dataset, and the trimmed files will be saved to `wav48_silence_trimmed`.\n ```\n cd VCTK-Corpus-0.92\n python flac2wav.py\n ```\n 1. Move all the trimmed training files from `wav48_silence_trimmed` to [wav48/train](wav48/train) following the indexes in [training.txt](VCTK-Corpus-0.92/training.txt), and move all the untrimmed test files from `wav48_origin` to [wav48/test](wav48/test) following the indexes in [test.txt](VCTK-Corpus-0.92/test.txt).\n\n## Training\n```\ncd train\nCUDA_VISIBLE_DEVICES=0 python train_16k.py --config [config file path]\nCUDA_VISIBLE_DEVICES=0 python train_48k.py --config [config file path]\n```\nCheckpoints and copies of the configuration file are saved in the `cp_model` directory by default.
\nYou can change the path by using the `--checkpoint_path` option.\nHere is an example:\n```\nCUDA_VISIBLE_DEVICES=0 python train_16k.py --config ../configs/config_2kto16k.json --checkpoint_path ../checkpoints/AP-BWE_2kto16k\n```\n\n## Inference\n```\ncd inference\npython inference_16k.py --checkpoint_file [generator checkpoint file path]\npython inference_48k.py --checkpoint_file [generator checkpoint file path]\n```\nYou can download the [pretrained weights](https://drive.google.com/drive/folders/1IIYTf2zbJWzelu4IftKD6ooHloJ8mnZF?usp=share_link) we provide and move all the files to the `checkpoints` directory.\n
\nGenerated wav files are saved in `generated_files` by default.\nYou can change the path by adding `--output_dir` option.\nHere is an example:\n```\npython inference_16k.py --checkpoint_file ../checkpoints/2kto16k/g_2kto16k --output_dir ../generated_files/2kto16k\n```\n\n## Model Structure\n![model](Figures/model.png)\n\n## Comparison with other speech BWE methods\n### 2k/4k/8kHz to 16kHz\n

\n\"comparison\"\n

\n\n### 8k/12k/16/24kHz to 16kHz\n

\n\"comparison\"\n

\n\n## Acknowledgements\nWe referred to [HiFi-GAN](https://github.com/jik876/hifi-gan) and [NSPP](https://github.com/YangAi520/NSPP) to implement this.\n\n## Citation\n```\n@article{lu2024towards,\n title={Towards high-quality and efficient speech bandwidth extension with parallel amplitude and phase prediction},\n author={Lu, Ye-Xin and Ai, Yang and Du, Hui-Peng and Ling, Zhen-Hua},\n journal={arXiv preprint arXiv:2401.06387},\n year={2024}\n}\n\n@inproceedings{lu2024multi,\n title={Multi-Stage Speech Bandwidth Extension with Flexible Sampling Rate Control},\n author={Lu, Ye-Xin and Ai, Yang and Sheng, Zheng-Yan and Ling, Zhen-Hua},\n booktitle={Proc. Interspeech},\n pages={2270--2274},\n year={2024}\n}\n```\n"},"items":[{"name":"README.md","path":"tools/AP_BWE_main/README.md","rawUrl":"https://raw.githubusercontent.com/RVC-Boss/GPT-SoVITS/HEAD/tools/AP_BWE_main/README.md","title":"AP_BWE_main Documentation","category":"plugin-manifest","format":"markdown","content":"# Towards High-Quality and Efficient Speech Bandwidth Extension with Parallel Amplitude and Phase Prediction\n### Ye-Xin Lu, Yang Ai, Hui-Peng Du, Zhen-Hua Ling\n\n**Abstract:** \nSpeech bandwidth extension (BWE) refers to widening the frequency bandwidth range of speech signals, enhancing the speech quality towards brighter and fuller.\nThis paper proposes a generative adversarial network (GAN) based BWE model with parallel prediction of Amplitude and Phase spectra, named AP-BWE, which achieves both high-quality and efficient wideband speech waveform generation.\nThe proposed AP-BWE generator is entirely based on convolutional neural networks (CNNs).\nIt features a dual-stream architecture with mutual interaction, where the amplitude stream and the phase stream communicate with each other and respectively extend the high-frequency components from the input narrowband amplitude and phase spectra.\nTo improve the naturalness of the extended speech signals, we employ a multi-period discriminator at the waveform level and design a pair of multi-resolution amplitude and phase discriminators at the spectral level, respectively.\nExperimental results demonstrate that our proposed AP-BWE achieves state-of-the-art performance in terms of speech quality for BWE tasks targeting sampling rates of both 16 kHz and 48 kHz. \nIn terms of generation efficiency, due to the all-convolutional architecture and all-frame-level operations, the proposed AP-BWE can generate 48 kHz waveform samples 292.3 times faster than real-time on a single RTX 4090 GPU and 18.1 times faster than real-time on a single CPU.\nNotably, to our knowledge, AP-BWE is the first to achieve the direct extension of the high-frequency phase spectrum, which is beneficial for improving the effectiveness of existing BWE methods.\n\n**We provide our implementation as open source in this repository. Audio samples can be found at the [demo website](http://yxlu-0102.github.io/AP-BWE).**\n\n\n## Pre-requisites\n0. Python >= 3.9.\n0. Clone this repository.\n0. Install python requirements. Please refer [requirements.txt](requirements.txt).\n0. Download datasets\n 1. Download and extract the [VCTK-0.92 dataset](https://datashare.ed.ac.uk/handle/10283/3443), and move its `wav48` directory into [VCTK-Corpus-0.92](VCTK-Corpus-0.92) and rename it as `wav48_origin`.\n 1. Trim the silence of the dataset, and the trimmed files will be saved to `wav48_silence_trimmed`.\n ```\n cd VCTK-Corpus-0.92\n python flac2wav.py\n ```\n 1. Move all the trimmed training files from `wav48_silence_trimmed` to [wav48/train](wav48/train) following the indexes in [training.txt](VCTK-Corpus-0.92/training.txt), and move all the untrimmed test files from `wav48_origin` to [wav48/test](wav48/test) following the indexes in [test.txt](VCTK-Corpus-0.92/test.txt).\n\n## Training\n```\ncd train\nCUDA_VISIBLE_DEVICES=0 python train_16k.py --config [config file path]\nCUDA_VISIBLE_DEVICES=0 python train_48k.py --config [config file path]\n```\nCheckpoints and copies of the configuration file are saved in the `cp_model` directory by default.
\nYou can change the path by using the `--checkpoint_path` option.\nHere is an example:\n```\nCUDA_VISIBLE_DEVICES=0 python train_16k.py --config ../configs/config_2kto16k.json --checkpoint_path ../checkpoints/AP-BWE_2kto16k\n```\n\n## Inference\n```\ncd inference\npython inference_16k.py --checkpoint_file [generator checkpoint file path]\npython inference_48k.py --checkpoint_file [generator checkpoint file path]\n```\nYou can download the [pretrained weights](https://drive.google.com/drive/folders/1IIYTf2zbJWzelu4IftKD6ooHloJ8mnZF?usp=share_link) we provide and move all the files to the `checkpoints` directory.\n
\nGenerated wav files are saved in `generated_files` by default.\nYou can change the path by adding `--output_dir` option.\nHere is an example:\n```\npython inference_16k.py --checkpoint_file ../checkpoints/2kto16k/g_2kto16k --output_dir ../generated_files/2kto16k\n```\n\n## Model Structure\n![model](Figures/model.png)\n\n## Comparison with other speech BWE methods\n### 2k/4k/8kHz to 16kHz\n

\n\"comparison\"\n

\n\n### 8k/12k/16/24kHz to 16kHz\n

\n\"comparison\"\n

\n\n## Acknowledgements\nWe referred to [HiFi-GAN](https://github.com/jik876/hifi-gan) and [NSPP](https://github.com/YangAi520/NSPP) to implement this.\n\n## Citation\n```\n@article{lu2024towards,\n title={Towards high-quality and efficient speech bandwidth extension with parallel amplitude and phase prediction},\n author={Lu, Ye-Xin and Ai, Yang and Du, Hui-Peng and Ling, Zhen-Hua},\n journal={arXiv preprint arXiv:2401.06387},\n year={2024}\n}\n\n@inproceedings{lu2024multi,\n title={Multi-Stage Speech Bandwidth Extension with Flexible Sampling Rate Control},\n author={Lu, Ye-Xin and Ai, Yang and Sheng, Zheng-Yan and Ling, Zhen-Hua},\n booktitle={Proc. Interspeech},\n pages={2270--2274},\n year={2024}\n}\n```\n","isInternal":false,"tokens":1291,"sizeBytes":5007}],"systemPromptSnippet":"\n\n# Towards High-Quality and Efficient Speech Bandwidth Extension with Parallel Amplitude and Phase Prediction\n### Ye-Xin Lu, Yang Ai, Hui-Peng Du, Zhen-Hua Ling\n\n**Abstract:** \nSpeech bandwidth extension (BWE) refers to widening the frequency bandwidth range of speech signals, enhancing the speech quality towards brighter and fuller.\nThis paper proposes a generative adversarial network (GAN) based BWE model with parallel prediction of Amplitude and Phase spectra, named AP-BWE, which achieves both high-quality and efficient wideband speech waveform generation.\nThe proposed AP-BWE generator is entirely based on convolutional neural networks (CNNs).\nIt features a dual-stream architecture with mutual interaction, where the amplitude stream and the phase stream communicate with each other and respectively extend the high-frequency components from the input narrowband amplitude and phase spectra.\nTo improve the naturalness of the extended speech signals, we employ a multi-period discriminator at the waveform level and design a pair of multi-resolution amplitude and phase discriminators at the spectral level, respectively.\nExperimental results demonstrate that our proposed AP-BWE achieves state-of-the-art performance in terms of speech quality for BWE tasks targeting sampling rates of both 16 kHz and 48 kHz. \nIn terms of generation efficiency, due to the all-convolutional architecture and all-frame-level operations, the proposed AP-BWE can generate 48 kHz waveform samples 292.3 times faster than real-time on a single RTX 4090 GPU and 18.1 times faster than real-time on a single CPU.\nNotably, to our knowledge, AP-BWE is the first to achieve the direct extension of the high-frequency phase spectrum, which is beneficial for improving the effectiveness of existing BWE methods.\n\n**We provide our implementation as open source in this repository. Audio samples can be found at the [demo website](http://yxlu-0102.github.io/AP-BWE).**\n\n\n## Pre-requisites\n0. Python >= 3.9.\n0. Clone this repository.\n0. Install python requirements. Please refer [requirements.txt](requirements.txt).\n0. Download datasets\n 1. Download and extract the [VCTK-0.92 dataset](https://datashare.ed.ac.uk/handle/10283/3443), and move its `wav48` directory into [VCTK-Corpus-0.92](VCTK-Corpus-0.92) and rename it as `wav48_origin`.\n 1. Trim the silence of the dataset, and the trimmed files will be saved to `wav48_silence_trimmed`.\n ```\n cd VCTK-Corpus-0.92\n python flac2wav.py\n ```\n 1. Move all the trimmed training files from `wav48_silence_trimmed` to [wav48/train](wav48/train) following the indexes in [training.txt](VCTK-Corpus-0.92/training.txt), and move all the untrimmed test files from `wav48_origin` to [wav48/test](wav48/test) following the indexes in [test.txt](VCTK-Corpus-0.92/test.txt).\n\n## Training\n```\ncd train\nCUDA_VISIBLE_DEVICES=0 python train_16k.py --config [config file path]\nCUDA_VISIBLE_DEVICES=0 python train_48k.py --config [config file path]\n```\nCheckpoints and copies of the configuration file are saved in the `cp_model` directory by default.
\nYou can change the path by using the `--checkpoint_path` option.\nHere is an example:\n```\nCUDA_VISIBLE_DEVICES=0 python train_16k.py --config ../configs/config_2kto16k.json --checkpoint_path ../checkpoints/AP-BWE_2kto16k\n```\n\n## Inference\n```\ncd inference\npython inference_16k.py --checkpoint_file [generator checkpoint file path]\npython inference_48k.py --checkpoint_file [generator checkpoint file path]\n```\nYou can download the [pretrained weights](https://drive.google.com/drive/folders/1IIYTf2zbJWzelu4IftKD6ooHloJ8mnZF?usp=share_link) we provide and move all the files to the `checkpoints` directory.\n
\nGenerated wav files are saved in `generated_files` by default.\nYou can change the path by adding `--output_dir` option.\nHere is an example:\n```\npython inference_16k.py --checkpoint_file ../checkpoints/2kto16k/g_2kto16k --output_dir ../generated_files/2kto16k\n```\n\n## Model Structure\n![model](Figures/model.png)\n\n## Comparison with other speech BWE methods\n### 2k/4k/8kHz to 16kHz\n

\n\"comparison\"\n

\n\n### 8k/12k/16/24kHz to 16kHz\n

\n\"comparison\"\n

\n\n## Acknowledgements\nWe referred to [HiFi-GAN](https://github.com/jik876/hifi-gan) and [NSPP](https://github.com/YangAi520/NSPP) to implement this.\n\n## Citation\n```\n@article{lu2024towards,\n title={Towards high-quality and efficient speech bandwidth extension with parallel amplitude and phase prediction},\n author={Lu, Ye-Xin and Ai, Yang and Du, Hui-Peng and Ling, Zhen-Hua},\n journal={arXiv preprint arXiv:2401.06387},\n year={2024}\n}\n\n@inproceedings{lu2024multi,\n title={Multi-Stage Speech Bandwidth Extension with Flexible Sampling Rate Control},\n author={Lu, Ye-Xin and Ai, Yang and Sheng, Zheng-Yan and Ling, Zhen-Hua},\n booktitle={Proc. Interspeech},\n pages={2270--2274},\n year={2024}\n}\n```\n"}