I am a teacher and I have a LOT of different literature material that I wish to study, and play around with.
I wish to have a self-hosted and reasonably smart LLM into which I can feed all the textual material I have generated over the years. I would be interested to see if this model can answer some of my subjective course questions that I have set over my exams, or write small paragraphs about the topic I teach.
In terms of hardware, I have an old Lenovo laptop with an NVIDIA graphics card.
P.S: I am not technically very experienced. I run Linux and can do very basic stuff. Never self hosted anything other than LibreTranslate and a pihole!
I’m in the early stages of this myself and haven’t actually run an LLM locally but the term that steered me in the right direction for what I was trying to do was ‘RAG’ Retrieval-Augmented Generation.
ragflow.io (terrible name but good product) seems to be a good starting point but is mainly set up for APIs at the moment though I found this link for local LLM integration and I’m going to play with it later today. https://github.com/infiniflow/ragflow/blob/main/docs/guides/deploy_local_llm.md
Just checked out on the project, and the I’m a bit frustrated by it.
First, the Docker image is humongous, almost 10GB, all because they bundle all the GPU driver and runtime in it. Second, the default Docker Compose opens up all the port to each services except Redis where users doesn’t need to access at all. Third, zero dependencies management. They seems still using
pipandpip freezeto manage their dependencies. Result is 3 version conflicts. Four, not directly their fault but some dependency uses PyCryptodome releases back in Nov, 2020.It is a good product, but seems they not just terrible at names, but also in project management.
EDIT: The docker image is actually mind boggling 17.4GB.