How to Create Your Own Dataset from Scratch with Ollama Locally

[@davidtindell950]

Thank You. A good exercise in 'fine-tuning' with our own corporate data ... using Ollama ...

[@fahdmirza]

You are welcome!

[@MinoasPediadas]

Thank you for the explanations, very helpful. I would appreciate your thoughts on the following scenario: Let's say you have a pdf file that contains some criteria/requirements. Assume also you have a second pdf file which describes a project and also describes how the project will address the specific criteria/requirements defined in the 1st file. I was wondering if there is an LLM-based workflow that could examine how well the project description addresses the criteria in the 1st file. I was thinking of a process like this: 1. Get some ready-made LLM model like Ollama or BERT 2. Train it again with your 1st file as an input 3. Load the second file to a vectorDB and create a chain 4. Query the second file I'm not sure if this works though and I'm not sure how I can specify my queries to cross-examine SPECIFICALLY the second file against the SPECIFIC criteria of the 1st file. Would you have some kind of guidance for me?

[@rojapr-p2m]

Hi, thank you for the video. I have a question regarding using multiple schemas. For example, in an employee database, we might have multiple collections such as a Department table and an Employee Information table. If I ask a question related to the department, will it only consider the schema of the Department table? Also, will it be able to handle joins between two tables and return a query that includes both? Please clarify. Note that My DB is unstructured

[@fahdmirza]

I already have few videos around it, please search the channel, thanks.

[@biorig]

Not all of us have computers with NVIDIA GPUs capable of running LLMs. Can you please make a guide on how to run LLMs on the cloud at the cheapest possible costs? If we can pay only for CPU hours used would be great. THank you Fahd!

[@fahdmirza]

I also don't have a GPU. As mentioned in the video clearly that I am renting GPU very cheaply from the cloud. Link to that rental is also in video description.

[@PiakBot]

​@fahdmirza Could you tell me your old video that teach how to rent a cloud GPU step by step?

[@AmanKumar-uz7ri]

I have raw text which i have parsed from pdfs , i want to create a dataset from that txt so that i can train model locally ,

[@fahdmirza]

I have a similar video around it, please search the channel, thanks.

[@m.k.4217]

Hello Fahd, thank you for the great explanation. As far as I understand now for the own dataset you have to have structured data with schema. But what about unstructured textes - technical, economics, law literature and statements? Can you feed your dataset with them? Is it possible only via prompt with small portions without any deep integration?

[@fahdmirza]

yes you can feed it and I am working on a similar video which should be out soon,

[@CryptoMaN_Rahul]

I want to create dataset for e-commerce products

[@sirishkumar-m5z]

Building datasets is crucial for any AI project. SmythOS simplifies the process with its robust AI support and integrations. #AI #SmythOS