I am a teacher and I have a LOT of different literature material that I wish to study, and play around with.
I wish to have a self-hosted and reasonably smart LLM into which I can feed all the textual material I have generated over the years. I would be interested to see if this model can answer some of my subjective course questions that I have set over my exams, or write small paragraphs about the topic I teach.
In terms of hardware, I have an old Lenovo laptop with an NVIDIA graphics card.
P.S: I am not technically very experienced. I run Linux and can do very basic stuff. Never self hosted anything other than LibreTranslate and a pihole!
Just checked out on the project, and the I’m a bit frustrated by it.
First, the Docker image is humongous, almost 10GB, all because they bundle all the GPU driver and runtime in it. Second, the default Docker Compose opens up all the port to each services except Redis where users doesn’t need to access at all. Third, zero dependencies management. They seems still using
pipandpip freezeto manage their dependencies. Result is 3 version conflicts. Four, not directly their fault but some dependency uses PyCryptodome releases back in Nov, 2020.It is a good product, but seems they not just terrible at names, but also in project management.
EDIT: The docker image is actually mind boggling 17.4GB.