How To Test AI CHAT BOTS |Testing Techniques & Limitations|Why Is It Different From Regular Testing?

[@atulbharadwaj774]

In my organisation we have GPT 4 model which is trained on custom dataset and currently we are validating using BERT model, which validates expected output and actual output from AI bot, if cosine similarity scores is greater than 0.5 then we are passing the TC

[@naveenautomationlabs]

Good one Atul. Thanks for sharing it.

[@aravindmak]

@atul bharadwaj - can you please explain it a little further ? What will be your validation strategy ? Is there a Human in Loop revie for the output that comes out of ChatGPT model that validates it's correctness ?

[@athiramurali759]

Hi,i need some help for my upcoming project regarding this. How can I contact you

[@bhhada]

I am using Botium for testing the Chatbots.

[@bowthunell]

Hi, Did you use Botium with Chat -Gpt? or which LLMs? and how was your experience with Botium? I'm considering using it with my Chatbots project.

[@Bbangaram-d9u]

Could you please give more info on this please?

[@sakinathmubarak426]

Hi Naveen, Can we automate the Chatbot UI?..If that is possible Can you explain and make a video on that UI testing of chatbots like speech input,text input,selection inputs and how to validate the response like comparing with actual response. If there is a video already on this topic pls give me the link

[@puneetaneja6296]

Good Content, thanks Naveen.

[@i4iirfan]

Please make a video on how to automate ai based chat bot and which tools to use

[@parveenchauhan5132]

Hi Naveen, It was insightful. Thanks

[@shankarganesh1858]

Hi Naveen, Thanks for the above video, Healenium is a open source AI powered self healing Technology available now, if you could make a video on the same it would be helpful. It supports both Selenium and Appium. if you could explain the concepts of installation and configuration by your style, it would help users to understand better and use it in the project.currently i struggling to implement in my project. Thanks in advance.

[@STELLADCSR]

In our organization we validate the chatbot by first creating a folder that contains all the files of our documentation. We pregenerate questions for each page and save it in a json file. we then randomly select a question and ask the chatbot. Multiple questions can be asked using multiprocessing(completes load testing). We then send the response, the question, and the respective contents of the file to LLM and ask LLM to validate its accuracy on a scale of 1-10. Is there any other techniques where I can validate chatbots that has only knowledgebase and QnA knowledge fed into it? Cause I dont have any files to validate them.

[@STELLADCSR]

any tool to help me in this?? or any other algorithm?

[@prasanthss420]

Good

[@sarahgao8300]

My data science friend told me they don’t have QA position in her machine leaning project.

[@user-xb7bc5eo1w]

Hi Naveen, Thanks a lot for the information. Suggest me any testing tools available to test an Avatar based AI application

[@pabitra_swain]

Hi Navin, do we have an option to Run Playwright on GCP? Please make a video on it or please reply if you have done your poc on the same. That will really help. Thanks in advance!

[@KrishKoli-qw9vb]

Can you provide happy path and negative path test scenarios?

[@chandrasekhar-sy6uj]

Hi Naveen I have no idea on what certifications need to be done for AI automation testing, can you help me with this

[@naveenautomationlabs]

As such no certification is available for AI in testing

[@atulbharadwaj774]

Can you suggest some python based libraries to test these bots

[@naveenautomationlabs]

I heard from one of friends, working as a data scientist suggested to use nltk: The Natural Language Toolkit (NLTK) is a widely-used library for NLP tasks. It provides tools for tokenization, stemming, part-of-speech tagging, syntactic parsing, and more. It can be used for preprocessing text and verifying the correctness of NLP operations.

[@atulbharadwaj774]

Thanks, will check that

[@kshitijvairale4877]

Is there any tool for performance testing chat bots lets say I want to run a test for 1000 users accessing a bot at more or less same time

[@naveenautomationlabs]

Hi Kshitij, Use load testing tools to simulate high user traffic and measure how the chatbot performs under heavy loads. Monitor response times, resource utilization, and system stability during peak usage scenarios. Tools like Apache JMeter or LoadRunner or Gatling or K6 can help in load testing. Performance Monitoring: To collect data on response times, throughput, error rates, and other performance metrics. Monitor the chatbot in production or a staging environment to identify bottlenecks, performance degradation, or potential scalability issues. Tools like New Relic, Datadog, or custom monitoring solutions can assist in performance monitoring. Benchmarking is important: Compare the chatbot's performance against industry standards or competitors' chatbots. Benchmarking allows you to assess the chatbot's performance in relation to similar systems and identify areas for improvement or differentiation.

[@kshitijvairale4877]

@@naveenautomationlabsThank you sir for your response But the chat bots which are build in my organization is based on web socket and on client server manner we have to launch url in browser to build a connection In this scenario do u know any tool...

[@naveenautomationlabs]

@@kshitijvairale4877 Tools like Locust, Gatling, Jmeter (As I suggested above) supports WebSocket connections, send messages, and measure the performance and scalability of your chatbot.

[@Dipsravi]

Hi kshitij, Have you done performance testing for above usecase? looking for same use case so i need small help

[@dreammer5749]

Sir your telegram invitation link is expired.

[@prasanthss420]

Could you please share job vacancy