Solving My Hacker News Problem with AI (Full Code)

Solving My Hacker News Problem with AI (Full Code)
👊 Become a member and get access to GitHub and Code:
/ allaboutai
GitHub:
github.com/AllAboutAI-YT/hack...
🤖 Great AI Engineer Course:
scrimba.com/learn/aiengineer?...
📧 Join the newsletter:
www.allabtai.com/newsletter/
🌐 My website:
www.allabtai.com
Open Source Projects:
huggingface.co/sentence-trans...
In this video I create python code to search HackerNews using semantic search with embeddings. Great way to collect data for you own use cases using RAG and LLMs, pick between a local model or a bigger closed model.
00:00 HackerNews AI Search Intro
02:37 Flowchart
03:55 Hacker News LLM Search Code
05:55 Tests
09:41 Conclusion

[@sirinongorur5676]

great stuff, need more of this. Keep up the good work!

[@gumshoe9496]

Very good project!

[@coinboybit7281]

wondering if it better to chunk it by each json item/post instead of 1000 characters since the data is already in json format?

[@Brkcln]

Good one, when have time i will try this too

[@hope42]

You picked a great topic that time and killed it. A wonderful use case to be reused in the use case to be reused in the use case haha. 😮

[@inout3394]

Thx

[@watchdog163]

Looking forward to grabbing it from the pøblic reepo.

[@lokeshart3340]

Do u get it? If u get pls reply and also the speech to speech ready also pls help

[@micbab-vg2mu]

nice:)

[@TheHistoryCode125]

* (00:00) Introduction to Hacker News and the problem of finding specific information * (00:28) Creating a script to scrape Hacker News posts and comments * (00:58) Embedding the data into a semantic search system * (01:32) Searching for sentiment analysis on a specific post * (02:12) Running the Hau model and local model through AMA * (02:42) Overview of the process from scraping to embeddings * (03:18) Using the embeddings to answer user questions * (03:54) Code overview and setup * (04:30) Getting information from Hacker News and adjusting the system * (05:03) Using the sentence transformer and adjusting the system message * (05:37) Chunk sizes and the Vault file * (06:08) Testing the system with a "needle in a haystack" search * (06:41) Finding a specific comment and getting more information * (07:16) Searching for posts with specific keywords * (07:56) Analyzing the data with Claude 3 Sonnet * (08:25) Running the program and getting the results * (09:03) Results: topic, URL, most emotional comments, and KZbin video ID * (09:43) Early stages of the project and future updates * (10:16) Conclusion and public repo **Key Takeaways** --------------- 🔍 Hacker News can be overwhelming, but AI can help find specific information 💻 Scripting and scraping can gather data, which can be embedded for semantic search 🤔 Sentiment analysis and user questions can be answered with context 📊 Code is available and adjustable for different sources and needs 🔮 Claude 3 Sonnet can analyze data for key topics and emotional responses **Benefits** ------------ ✔ Solves the problem of finding specific information on Hacker News ✔ Uses AI for semantic search and sentiment analysis ✔ Adjustable code for different sources and needs ✔ Analyzes data for key topics and emotional responses **Cons** ---------- ❌ Early stages of the project, needs more work ❌ Prompt and desired output need refinement ❌ Limited to Hacker News, could be expanded to other sources **Score** --------- - 8/10 for solving a specific problem with AI - 7/10 for code and adjustability - 6/10 for early stages and room for improvement **Like or Dislike** ------------------- 👍 Like for the creative solution and potential for expansion and improvement.

[@ShaunPrince]

I think HackerNews has an RSS feed, so there is no need to scrape.

[@user---------]

It’s not entirely clear what exactly the script is doing here? Is it looking for some facts in all the comments?

[@BUY_YOUTUB_VIEWS_733]

Love the energy 👏👏