Rasa Algorithm Whiteboard - Toxic Language Detection

[@ramshasiddiqui8477]

What if we have a use-case where we want to classify personal-insult VS general use of illicit language. For example, a customer complaining about how his debit card is *** useless V.S. a customer calling a Customer Service Representative / Bot *** useless and asking to talk to a supervisor. How do I teach a model that these two are different without building my own dataset from conversations? Also does RASA support Sentiment Analysis with Intent Detection?

[@RasaHQ]

(Vincent here) If you want any model to learn, you're going to have to accept that you need to collect data. If you want your model to be representative, that means that you'll need to collect representative data. Nothing is stopping you from adding sentiment models to your Rasa pipeline. But instead, a more common approach is to have "abusive text" as a label. This also allows your assistant to respond in a "this is not normal behavior"-kind of way as well because it is detected as an intent. I've been thinking about adding support for a sentiment tool in the pipeline. You might be able to set a slot that tells you the sentiment of the user in the conversation. The issue of "cultural aspect" remains the same though and I'm not 100% sure that taking a pre-trained sentiment model is going to be a good idea for every assistant out there.

[@ramshasiddiqui8477]

@@RasaHQ Have you checked out the DeepPavlov Pipeline and their insult-detection classifier for this problem (demo.deeppavlov.ai/#/en/insult). I kinda like it, pretty much works for this use-case and the good thing about their framework is that its RASA DSL Compatible - was really impressed with it!

[@RasaHQ]

​@@ramshasiddiqui8477 This is news to me. I've played with the demo briefly and it does seem robust against some of the issues I mention in my video. I'll explore it further though. I'm interested to see what data it has been trained on.

[@sam.q]

@@RasaHQ Hi, I'd like to first thank you for the amazing content. To help with the data collection problem, I would like to suggest an approach where you use a language generation model like GPT-2 to create synthetic data. The idea is that you finetune your classifier and GPT-2 on the existing small dataset and then use both models to generate additional examples. One way would be to basically generate thousands of random examples with GPT-2 (e.g. with top-k or nucleus sampling) then use the trained classifier to output a probability distribution for every generated example. Apply entropy function on the predictions and sort them in descending order. Examples with highest entropy are basically the ones that were the most confusing to the classifier. Review top n examples and correct their labels by hand. Add the reviewed examples to the original training data. The finally retrain the classifier with the new dataset.. you could repeat these steps in active learning runs for better results.. You could also implement this with the Monte Carlo Tree Search algorithm to condition on gpt to generate examples with a reward function (though this could be a slower process). If you find this approach interesting, please check out our paper for more details "Textual Data Augmentation for Efficient Active Learning on Tiny Datasets". www.aclweb.org/anthology/2020.emnlp-main.600/

[@shubheshswain5480]

Can Rasa bot be used for chat moderation using the trained BERT model?

[@RasaHQ]

(Vincent here) Could you elaborate on what you mean by chat moderation? Rasa supports BERT models natively for featurization but it sounds like you're referring to something else.

[@shubheshswain5480]

@@RasaHQ I mean a Bot which can report user if he/she send hateful messages. Also that BOT can report to Admin if the User has violated multiple times.

[@RasaHQ]

@@shubheshswain5480 You can use the logs generated by Rasa Open Source/Rasa X to use your own model for this, but it's not something that we have built-in at the moment.

[@shubheshswain5480]

@@RasaHQ okay. I'll check it. Thanks for your support.

[@rickharold7884]

Cool. Thx