How to Create AI Voice for Skyrim - Part 1 - XVATrainer Tutorial

Please support Ukraine before the war comes to you! u24.gov.ua/
If you want to create your AI voices based on existing Skyrim voices this is a tutorial for you!
0:00 Intro
0:38 Set Up the XVATrainer program
4:08 Create Your Dataset
7:57 How to Retrieve Voices from Skyrim
15:04 Transcribe Your Audio Files
19:01 (optional) Dublicate your Dataset to increase it`s Volume
20:36 Check your Dataset for Mistakes
21:33 Train Your Model
24:30 How to Resume Your Training (or Force the Next Stage)
26:50 Export Your Model
28:00 Outro
You`ll need such programs:
1.XVA Trainer:
store.steampowered.com/app/19...
!be sure to download v3 training data PART 1 and PART 2 from:!
www.nexusmods.com/skyrimspeci...
Put the downloaded xvapitch files to ./resources/app/python!
2.Bethesda Archive Extractor:
www.nexusmods.com/skyrimspeci...
3.Yakitori Audio Converter:
www.nexusmods.com/skyrimspeci...
(You only need this tutorial if your desired Voice Model doesn`t exist)
You might want to check Nexus Mods or
docs.google.com/spreadsheets/...
whether your model already exists (it will save you a lot of time, trust me).
I`m using latest Skyrim Special Edition as an example.
#skyrim #skyrimspecialedition #skyrimanniversaryedition #skyrimmods #skyrimmodding #XVATrainer #XVASync #voicegenerator #gameplay #games #nexus #nexusmods #ai #aigenerated #tutorial #how #howto #howtogeneratevoice #BSA #voiceacting

[@arturkot4108]

Please support Ukraine before the war comes to you! u24.gov.ua/ A text version goes as follows: To train your model you need two things: A dataset to train of and a model to be trained. I) How to set up the XVATrainer Program: 1. Download XVATrainer. 2. Download v3 training data PART 1 and PART 2 from: www.nexusmods.com/skyrimspecialedition/mods/65022 3. Open the downloaded archives. Copy the xvapitch folder from them to [XVATrainer]/resources/app/python/ (it is important otherwise the program will not work) 4. Optionally download OpenAI Whisper Large (v2) Auto-Transcription model. Next your Model needs audio files and text interpretation to train of. II) How to retrieve Skyrim voices from the game: 1. Download and unpack Bethesda Archive Extractor (BSA and BA2): www.nexusmods.com/skyrimspecialedition/mods/974 2. In your Skyrim folder find ./Data/Skyrim - Voices_en0.bsa (it can be Skyrim - Voices_ru0.bsa or any other language you want). 3. Drag and drop [Skyrim - Voices_en0.bsa] to the opened Bethesda Archive Extractor (BSA and BA2). 4. Exctract the desired voices to any folder on your PC (the exctracted files will be of .fuz type and that`s OK). 5. Download and unpack Yakitori Audio Converter: www.nexusmods.com/skyrimspecialedition/mods/17765 6. Drag and drop your exctracted .fuz files into an opened Yakitori Audio Converter: window. 7. Select (ignore lip) mode in the drop down menu on the bottom left. (lip is a lipsync file that comes compressed with .fuz file. Don`t worry, we`ll generate out own lipsync files in a different tutorial). 8. Set Output format to be .wav. 9. Convert the files to any location (you`ll have one .wav file from each .fuz file). III) How to create a new Dataset: 1. Open XVATrainer. 2. Press on (+) button on the top left. 3. Create a new Dataset: - Enter the name of the voice. - Game ID is the name of the game (Skyrim) - Game Code is a specified code of your game (sk for Skyrim, f4 for Fallout 4, OB for Oblivion). It is important for you to enter the correct code! - Enter the correct Gender! - Enter the correct Language Code! - Enter your name as Author. 4. Click on Submit buttom. 5. Click on the red button on the bottom (missing Metadata). 6. Check that your entered before data is correct and click Submit. IV) How to fil up your Dataset with audio examples: 1. Click on Open button on the bottom (a new Folder appears). 2. Open wav Folder. 3. Drag and drop all the .wav files from step II) to this folder. V) How to set the text interpretation for audio examples: 1. In XVATrainer click on any line and write the text interpretation of the audio files which you can listen to. 2. Don`t forget to press "SAVE" after each time you`ve edited a line or it will not be saved! You can also automatically Transcribe your audio files: 3. Press on the Wrench button on the top right. 4. Select Transcribe section in the menu on the left. 5. Select your Transcribtion Model (the bigger it is - the more accurate it will be. But it is also slower) 6. Select your Language. 7. Click on Open input Directory button. 8. Copy the .wav files from your dataset Folder to the newly opened transcribe\input folder. 9. Click on Run toll button. 10. Wait (it took me 1.5 hours to Transcribe 480 audio files with Whisper Medium model. And the result wasn`t ideal.) 11. After the Transcription is over click on Open output Directory button. 12. Drag and drop the metadata.csv from the newly opened transcribe\output folder to your Dataset Folder (replace the existing file). 13. Restart XVATrainer (now you have a filled dataset with audio examples and interpretation). It is better to have as large of a dataset as possible for the best possible results. If your dataset is smaller than 100 examples you can dublicate files using the method that is shown on the video: kzbin.info/www/bejne/q6GWeYt5jrNkoqM The minimum amout of audio examples for the model to train is 10! Also your interpritaion needs to be accurate! Otherwise the quality of your trained model will suffer! VI) How to fix your Dataset: 1. Go through your Dataset and check transcription for accuracy and punctuation. 2. Be sure to delete all grunts ad cries of pain from your dataset! Otherwise it will mess up the final result! (Just use DELETE function in the program over the selected line and it will delete an instance from the .wav file list and from the excel file as well!) After your Dataset if all nice and well it`s time to train it! VII) How to train your Dataset: 1. Select your Dataset in XVATrainer. 2. Press the Train button on the buttom. 3. Insert the Output Path for the Model Chekpoints to be stored (might fill up some space). 4.Select the right Voice Language in the dropdown menu! 5. (Optionally you can change the number of CPU workers to slightly increase your training time by sucrifising the accuracy a bit). 6. Click Accept. 7. Select your model in the list on the left. 8. Click on Start button. Training takes 3 stages. Each Stage is completed after 50 checkpoints. There is a Loss Threshold which determines whether the program will go to the next Stage or start over from the last result. After each Stage is completed a checkpoint file is created on your PC. If you exit the program you`ll need to start the training over! To continue the training after one of the Stages is over (and if the program has been closed): 1. Select your Dataset in XVATrainer. 2. Press the Train button on the buttom. 3. Press the Config button of your model in the list of models-to-be-trained on the left. 4. Change "xVAPitch Checkpoint path to fine-tune from (built in, or custom)" setting from BaseV3 to Custom. 5. Click on the Folder button on the right of it. 6. Select you LATEST chekpoint (.pt file). 7. (Optionally) Check the ticker on Force Stage box and choose the next Stage to be forced. 8. Click the Accept button. 9. Click on Start button. It took me 1.5 days on my PC to train a Delphine Voice Model. Not once have I crossed the Threshold, however I was suttysfied with the result and will use it in the mods to come. To complete your training you need to complete all 3 Stages at least once. VIII) How to Export your Model to XVASynth: 1. Select you dataset in XVATrainer. 2, Click on Export button on the buttom. 3. Select the output Folder. 4. Click on Export button. 5. Open the file with exported model (there will be 3 files). 6. Install XVASynth. 7. Drop your model (3 files) to [xVASynth] esources\app\models\skyrim Folder. 8. Start XVASynth and wait for my next Tutorial :)