Raven RWKV vs ChatGPT: chatbot tests and limitations

Uploaded on March 28, 2023, this review examines Raven, a chatbot the presenter describes as an RWKV model fine-tuned on Stanford's Alpaca instruction dataset. The linked model is rwkv-4-pile-7b. The video explores its potential as a local LLM and ChatGPT alternative, but the demonstrations use Hugging Face Spaces rather than a local installation. The presenter says the demo runs on a T4 GPU.

The tests cover Elon Musk jokes, Bangalore restaurant recommendations, grammar correction and history prompts. Raven corrects "my friends is really great people" to "my friends are really great people." The presenter does not verify whether the recommended restaurants exist. Later attempts to complete a joke produce responses he finds too serious, despite changes to the instruction.

Arithmetic exposes another limitation: Raven fails the expression the presenter expects to equal 68, while ChatGPT using GPT-3.5 returns the expected answer. These examples are informal tests with default settings, not a benchmark establishing equivalent performance.

The presenter describes RWKV as an attention-free recurrent architecture and repeats claims about lower VRAM use and faster inference. He explicitly says he has not tested the claimed infinite context length. The capabilities discussed belong to this 2023 demonstration; the video does not provide a setup procedure to run models locally.