LLMs for RockChip. Guide for RKLLM. RK3588 vs RK3576 comparision

[@ekcrisp1]

Great video! At 2:57 what model were you running and how were you running it? Was it the "flusk api" you mention later? Why do you think 76 was faster?

[@AntonMaltsev]

1) Qwen 1.8 (running with this approach - 08:25 ) 2) github.com/airockchip/rknn-llm/tree/main/examples/rkllm_server_demo 3) 76 may be faster for some models in some conditions. However, more tests are required to verify this. Specifically for the tested model - yes.

[@Rushil69420]

Great stuff here!

[@guiguicoco2740]

Hi Anton, thanks for your feedback on rk llm. I have a question, I didn’t understand how you could flash the RK3588 to run the tiny llm ?

[@AntonMaltsev]

For RK3588, I used NanoPC-T6 from Friendly Elec. Partially I described the flashing process here - kzbin.info/www/bejne/pn-bnn6QatyjmrM And you can use the same RKLLM, or it's a fork to install LLM.

[@guiguicoco2740]

@@AntonMaltsev hi again Anton, on the rock ship doc, they say : 1） Download the rknpu_driver_0.9.6_20240322.tar.bz2. 2） Unzip the compressed file and overwrite the RKNPU driver code into the current kernel code directory. 3） Recompile the kernel. 4） Flash the newly compiled kernel to the device. But I think it’s not so easy 😊 I will try, I am just worried because of the retro compatibility with my yolov8 vision application

[@AntonMaltsev]

I did not do anything with kernal reconpailing for any of these platforms.

[@gregherlein2381]

would love to see tests of computer vision speed between the two chips!

[@mal-avcisi9783]

Why do people always say their names at the beginning of the video, who cares about the name, what does it matter, what value does it add to the video?