Phi 1.5 - The small model getting big results

[@clray123]

Some interesting aspects of the training dataset leaking from this model (this is model output spit out in reaction to some unrelated prompt): Today, we will explore the fascinating world of Physical Science, specifically focusing on Forces and Motion. Let's dive in and discover the wonders of this topic through a consistent list of examples. (1). John pushed the heavy box up the ramp using his body strength because it was too heavy to lift directly. (2). The basketball player dribbled the ball smoothly because his hands were well-coordinated. (3). Sarah rode her bicycle downhill faster than David did, but David managed to reach the finish line first. (4). When a feather and a rock are thrown into the air simultaneously, the rock falls down faster due to its greater mass. (5). Alex decided to walk to school instead of taking the bus to enjoy the beautiful weather. (6). Although Lucy had a bicycle, she chose to ride her skateboard to the park for a change. (7). The teacher explained the concept of gravity to the students because they needed to understand how objects fall. (8). Jack turned on the faucet to wash his hands because he wanted to clean them thoroughly. (9). Since the wind was strong, Emma tied her hair up to prevent it from getting in her face. (10). Sarah closed her eyes while taking deep breaths to relax her mind before the exam. (11). Jake wore a helmet while riding his bike because his parents emphasized the importance of safety. (12). Emma decided to use a towel to wipe the spilled water instead of using her hands to avoid getting wet. (13). When a soccer ball is kicked, it rolls across the field due to the force applied.

[@clray123]

Wow.. I really got it talking... (11). The students were nervous about the exam because it was known to be challenging. (12). The dog ate all the cookies but left the apple untouched because he doesn't like apples. (13). Alex was excited when I told him that we were going to the beach because he loved the ocean. (14). The plants in the garden were all wilting because they didn't get enough water. (15). Sarah was proud of herself because she finished reading a whole book in one week. (16). Mark was disappointed when his favorite band canceled the concert because he had been looking forward to it. (17). Lisa was happy because her best friend surprised her with a thoughtful gift. (18). Jake didn't go to the party because he had a lot of homework to do. (19). The cat climbed the tree but couldn't reach the bird's nest because it was too high up. (20). Emma was angry when her sister broke her favorite toy because it was a gift from her grandmother. (21). Robert couldn't sleep because his neighbor was playing loud music. (22). Michelle was sad because she lost her pet dog. (23). The baby cried because he was hungry. (24). Tim was frustrated when he couldn't solve the math problem because he didn't understand the concept. (25). Amanda didn't like the book because the ending was unsatisfying. (26). John was scared of the thunderstorm because he associated it with danger. (27). Lisa was excited about the trip because she had never been on a plane before. (28). The dog wagged its tail but stayed away from the mailman because it was scared of strangers. (29). Alex didn't take the bus because it was raining heavily. (30). Sarah was jealous of her sister's new bike because she wanted one too. (31). Mark didn't eat the cake because he was allergic to dairy. (32). Emily was sad because her best friend moved away. (33). Alex didn't want to go to the party because he didn't know anyone there. (34). Tom was bored because there was nothing interesting to do. (35). The cat scratched the sofa but left the chair unharmed because it

[@tanguero2k7]

Oh my.... I was wondering why it was coming up with a whole lot of unrelated stuff. I was already giving up on it. Alice and Bob do work (well, until it starts leaking) in text-web-ui (oobaBooga) and python... 🤔... Maybe I'll try some fine-tuning later the coming week. Thank you all for sharing your thoughts!

[@parttimelarry]

Love this, so well explained

[@user-wr4yl7tx3w]

Be interested in what we need to fine tune Phi 1.5? Is Colab enough? Or would it be too slow?

[@clray123]

A 24GB GPU is enough to fine tune with all parameters. You could fine tune in free Colab with LoRA (i.e. a subset of parameters), and it would not be too slow. The problem is that Microsoft's "research license" forbids you from doing anything with the result apart from publishing papers and sending your work back to Microsoft. So for all practical purposes, it's a useless model. But hey, it can be converted into a mediocre sexbot in minutes.

[@user-qr4jf4tv2x]

tiny models will eventually be good

[@Pure_Science_and_Technology]

Enhance LLMs to build larger applications by integrating with vector databases, promoting modular development, adopting iterative processes, training on software best practices, using expandable memory systems, and enabling collaborative development among multiple LLM instances

[@TheAkiller101]

thank you so much for this update

[@clray123]

PROMPT: Sara and Ben are playing in the snow. They make a big snowman with a hat and a scarf. They are happy and laugh. But then a big dog comes. The dog is angry and barks. He runs to the snowman and... MODEL: throws snowballs at it. The snowman is scared and sad. He starts to fall down. A boy named Timmy is watching the snowman fall. He is scared. He wants to save the snowman. He runs to get help. He asks his mother and father for help. But they are too busy. They are working in the garden and the kitchen. They tell Timmy to stay calm and not to cry. They say they will come and help soon. Timmy is worried. He does not want the snowman to die. He thinks he can do it. He grabs a blanket and covers the snowman. He is brave. He thinks the snowman will be okay.

[@clray123]

Also from the model: They tell their mom and dad what happened. They ask them to help them rebuild their snowman. They go to their backyard. They have some wood and nails and paint. They use their creativity and skills to make another snowman. They work together and have fun. They feel proud of their new snowman.

[@henkhbit5748]

Thanks for the update for a new llmmodel. Like David it can beat Goliath. But the negative side is the licensing, only for research purposes.. Microsoft should follow Meta llama2.

[@foysalmamun5106]

Thank you lot

[@JoeBurnett]

Thank you!

[@xemy1010]

Only started watching the video, but the person that's whipping up controversy around Phi -1.5 on Twitter didn't reply to Ronen Eldan's rebuttal for the better part of a day, then ended up quote tweeting him a couple of hours ago responding in a way that seems like they conceded, while still treating the team with bad faith. I've come to a point where unless I see a paper published with actual work put into it, I treat a twitter rant as pretty much just that: a twitter rant.

[@samwitteveenai]

Yeah I actually cut the part about the controversy way back a lot after seeing Ronen Eldan's replies. I do wonder how close to GSM8k their text book examples are and what GPT-4 is making etc. I certainly don't think they deliberately did anything wrong. I wish MSFT would let them release a dataset etc.

[@cbusse7842]

what would you say is the best model for fine-tuning on a small library of PDF books that I have?

[@sumitmamoria]

For actual local fine-tuning, I would say a version of Llama2 that you can fit in your GPU memory. If you do not want to fine-tune and yet have a very good solution, try GPT-3.5 with retrieval augmented generation. You will have to think about costs as well - local fine-tuning will cost you money for compute and often GPT-3.5 is the cheaper option.

[@sumitmamoria]

I do not know which one is more "expensive" - access to high quality data or high-end computers. Then there is also the matter of one-time vs recurring costs.

[@knoopx]

tiny models are cute

[@ViktorFerenczi]

Did they publish the dataset and the prompts somewhere?

[@samwitteveenai]

no unfortunately not yet.

[@UmarFaruk-f8t]

Cant set return_unused_kwargs = False in transformer pipe line any help

[@samwitteveenai]

make sure you are using the transformers from source and it should work

[@debatradas1597]

thanks

[@SloanMosley]

Okay!

[@haria1]

Attention: paper clearly told that output is not consistent and promising.

[@midprogramming]

The point being that smaller models like 13B can compare to much larger models. Rewatch the video.

[@123arskas]

Please how can we fine-tune it on Colab? Is there a video of your's that teaches it? Thanks for the content

[@Morereality]

nice

[@bakuleshrane]

Caused its trained on texbook data

[@hongyan6879]

4min ago!

[@haristan1960]

Thank you!