I wrote this code to make my very own LLM.
In 2022, I made a jupyter notebook that can act just like ChatGPT, but also the user can change the length and depth of the output. Today, I had a bold idea of making my own LLM thinking that I don't have to rely on commercial LLMs such as ChatGPT, Gemini, etc.
Will I continue making this better?
The answer is NO. A big no.
Making this model took a lot of time. From watching tutorials on youtube to reading the research papers given below to understand the concept of making LLMs. The time taken to find a good dataset, train the model, the math involved, amount of time it takes to finetune the model is a lot. This project was very time-consuming for me.
Plus I don't have a GPU that can help make the project faster.
Even after spending hours to train the model, the output I get on giving a prompt is gibberish, proving that finetuning the model will take a lot of time.
https://skylion007.github.io/OpenWebTextCorpus/
- Attention is All You Need: https://arxiv.org/pdf/1706.03762.pdf
- A Survey of LLMs: https://arxiv.org/pdf/2303.18223.pdf
- QLoRA: Efficient Finetuning of Quantized LLMs: https://arxiv.org/pdf/2305.14314.pdf