Recursively summarize text of any length with GPT-3

www.skool.com/...
🚀 Welcome to the New Era Pathfinders Community! 🌟
Are you feeling overwhelmed by the AI revolution? You're not alone.
But what if you could transform that anxiety into your greatest superpower?
Join us on an exhilarating journey into the future of humanity in the age of AI! 🤖💫
🔥 What is New Era Pathfinders? 🔥
We are a vibrant community of forward-thinkers, innovators, and lifelong learners who are passionate about mastering the AI revolution. From college students to retirees, tech enthusiasts to creative souls - we're all here to navigate this exciting new era together!
🌈 Our Mission 🌈
To empower YOU to thrive in a world transformed by AI. We turn AI anxiety into opportunity, confusion into clarity, and uncertainty into unshakeable confidence.
🧭 The Five-Pillar Pathfinder's Framework 🧭
Our unique approach covers every aspect of life in the AI age:
1. 💻 Become an AI Power-User
Master cutting-edge AI tools and amplify your productivity!
2. 📊 Understand Economic Changes
Navigate the shifting job market with confidence and foresight!
3. 🌿 Back to Basics Lifestyles
Reconnect with your human essence in a digital world!
4. 🧑‍🤝‍🧑 Master People Skills
Enhance the abilities that make us irreplaceably human!
5. 🎯 Radical Alignment
Discover your true purpose in this new era!
🔓 What You'll Unlock 🔓
✅ Weekly Live Webinars: Deep-dive into each pillar with expert guidance
✅ On-Demand Courses: Learn at your own pace, anytime, anywhere
✅ Vibrant Community Forum: Connect, share, and grow with like-minded pathfinders
✅ Exclusive Resources: Cutting-edge tools, frameworks, and insights
✅ Personal Growth: Transform your mindset and skillset for the AI age
🚀 As You Progress 🚀
Unlock even more benefits:
🌟 One-on-One Mentoring Sessions
🌟 Exclusive Masterclasses
🌟 Advanced AI Implementation Strategies
💎 Why Join New Era Pathfinders? 💎
🔹 Expert-Led: Founded by a leading AI thought leader, connected with top researchers and innovators
🔹 Holistic Approach: We don't just teach tech - we prepare you for life in an AI-driven world
🔹 Action-Oriented: Real skills, real strategies, real results
🔹 Community-Driven: Join 300+ members already navigating this new era
🔹 Cutting-Edge Content: Stay ahead of the curve with the latest AI developments and strategies
🔥 Don't just survive the AI revolution - lead it! 🔥

[@Dan-oj4iq]

When Dave brought up that calculator to make a division query my head was brought back down to earth. "I'm really okay," I said to myself.

[@alan2here]

thank you :) I'm super _not_ corporate, and/but this seems essential, chunks depend on surrounding chunks for further summarisation :) clearly yes thank you.

[@RohanPaul-AI]

Just found your Channel and your Plan with this series. This is super impressive. Thanks for doing it. 💯

[@wimdenherder]

One of my dreams was to be able to click on parts of a summary and then it would expand it into bigger summaries. (the opposite direction). You would have to know which part in the summary corresponds to the original chunk, but that would be a supercool idea!

[@DaveShap]

We talked about that on Discord. Have different layers of compression/abstraction and then a knowledge graph.

[@Ethicalhate]

That sounds like a new way to find interesting relevant rabbit holes with infinite depth

[@dereke1371]

Works very well! Thank you - excellent tutorial

[@pootieetang]

Thank you for this, I no zero python but this makes me want to know it! I look forward to trying this out on my academic books. Question, have you made an updated video on this process?

[@barazara]

I am at the same spot, I need to summarize a few reaaaally long books I have to get through. Please let me know if you succeeded. I also have no python knowledge, so this is a little intimidating.

[@rerere284]

I don't know if it was the intention of the video but I wound up reading the whole summary by using . and , to frame by frame though the summary. I guess I know what the book version of Alice in Wonderland is now.

[@muhammedcagrkartal9954]

thank you so much

[@vulnerablegrowth3774]

Have you ever tried improving the summarization by adding the context before and after a chunk? I’ll probably try this myself, but I wonder if the summarization could be improved if you feed the model some “key points” from the text along with every chunk. You can imagine that if you want a high-quality summarization, it would be good for the model to be able to point out things like “what happened to Alice here was foreshadowed when she was doing x.” Or something like that. For more academic texts, it might be something like relating concepts that were introduced in the text at different places in the text. Instead of just giving the definition of concept A, the model includes some additional info about how it relates to concept B even though it was not included in the chunk. My guess is that you’d have to run a few different prompts across the text (though you could use a language model from huggingface to save some cash for some of the tasks) and then use the outputs as input to the general summarization. Like, use a model to extract the most important concepts from a paper (maybe make use of metadata and such), then store all the concepts in some way so that you can use them during summarization. Any thoughts on this?

[@DaveShap]

Yes, did that in the writing a novel series

[@vulnerablegrowth3774]

@@DaveShap Ah, I see. Very cool! I’ll try something similar for my use-case and see how it goes.

[@nimetsekman1569]

I AM A STUDENT. I WANT TO GET THE PDF BOOKS I HAVE IN HAND IN SUMMARY, CAN YOU PLEASE HELP ME? I DON'T KNOW HOW TO INSTALL THIS SYSTEM, WHAT ARE THE PROCESSING STEPS. I NEED HELP FROM YOU

[@stevefinkelstein1]

When you're breaking into chunks in this fashion, don't you risk cutting in the middle of a word/sentence which can impact summarization of that chunk?

[@MilesHart007]

Good point. However I ran a paragraph, and GPT replied with a summary, I then found a summary point which had "rabbit-hole", I delete the sample paragraph and deleted "rabbit-ho". It replied with the same response. So there is an element of linguistic bias. Your milage may vary.

[@nicola9249]

Hi @davidshapiroautomator , thanks for the video. Wondering what is including the the openaiapikey file - is it just the key?

[@pile_of_kyle]

It’s just the key in plain text format. You can alternatively use an environment variable.

[@ashwatimenon2569]

is there a quick way of getting reports summarised, like huge pdfs with ton load of data in1 page?

[@reactstuffreact4500]

45 iterations of api use? total cost ? just curious 👀 as if you proof read the summarized content. rewrite manually. check and then do the final work. check again and again. to make it perfect and to stnadard

[@b.s.m9732]

Why not just paste whatever scientific article in openai playground and summarize? I am just a beginner. Thank you

[@jamnallcute]

Same thought, because I really have no idea how to use phyton even after watching the video I still feel lost on how to make it work.

[@GrahamBillington]

too many characters

[@101RealTalker]

I have a case usage of needing 2 million words distilled down to 10k, what / where can I find mass word input for such summerization? Or what is a workaround for me to arrive at this context output desire? I was trying to upload either a PDF and or have a link that I could just copy paste over into chat GPT, but I have the free version and it tells me it can't read links, what can I do? I'm obviously not trying to have the entire 2 million at once, but that is the end game result over time I desire.

[@DaveShap]

Recursive summarization

[@101RealTalker]

@@DaveShap for the layman

[@muhammedcagrkartal9954]

hello, I am afraid we can lose some key concepts maybe while connecting the chunks, can you please share what you think about this worry ?

[@DaveShap]

Someone on the forum said if you add "preserve important details" to the prompt it does a good job.

[@muhammedcagrkartal9954]

@@DaveShap thank you for your response

[@cryptotester5042]

@@DaveShap hi David, thx for sharing your script. I just tested it to summarize a book, ogirinally 518k size => 122k (v1) => 40k (v2) => 16k (v3) => 4.4k (v4). I used this prompt "Write a concise summary, preserve important details, of the following". I retried to summarize v3, changing the prompt back to "Write a concise summary of the following:" and v4b is also 4.5k in size (so even slighltly bigger than the 4.4k one that was supposed to preserve important details). Is this expected to be like this ? I was expecting a smaller size, since I mouitted "preserve important details". Cheers

[@SohamSarkar]

this was really cool but doesnt feel economically viable with the token usage no?

[@DaveShap]

Depends on what you're trying to achieve and how much it's worth to you

[@testme2026]

super useful to say the least, thank you so much mate

[@ArielRomeroelreparador]

thanks you