Ask questions to your documents without an internet connection, using the power of LLMs. 100% private, no data leaves your execution environment at any point. You can ingest documents and ask questions without an internet connection!
In order to set your environment up to run the code here, first install all requirements:
``` pip install -r requirements.txt
Then, download the 2 models and place them in a directory of your choice.
.env and edit the variables appropriately.
``` MODEL_TYPE: supports LlamaCpp or GPT4All PERSIST_DIRECTORY: is the folder you want your vectorstore in LLAMA_EMBEDDINGS_MODEL: (absolute) Path to your LlamaCpp supported embeddings model MODEL_PATH: Path to your GPT4All or LlamaCpp supported LLM MODEL_N_CTX: Maximum token limit for both embeddings and LLM models
Note: because of the way
langchain loads the
LLAMA embeddings, you need to specify the absolute path of your embeddings model binary. This means it will not work if you use a home directory shortcut (eg.
This repo uses a state of the union transcript as an example.
Put any and all of your .txt, .pdf, or .csv files into the source_documents directory
Run the following command to ingest all the data.
``` python ingest.py
It will create a
db folder containing the local vectorstore. Will take time, depending on the size of your documents. You can ingest as many documents as you want, and all will be accumulated in the local embeddings database. If you want to start from an empty database, delete the
Note: during the ingest process no data leaves your local environment. You could ingest without an internet connection.
In order to ask a question, run a command like:
``` python privateGPT.py
And wait for the script to require your input.
Enter a query:
Hit enter. You’ll need to wait 20-30 seconds (depending on your machine) while the LLM model consumes the prompt and prepares the answer. Once done, it will print the answer and the 4 sources it used as context from your documents; you can then ask another question without re-running the script, just wait for the prompt again.
Note: you could turn off your internet connection, and the script inference would still work. No data gets out of your local environment.
exit to finish the script.
Selecting the right local models and the power of
LangChain you can run the entire pipeline locally, without any data leaving your environment, and with reasonable performance.
LangChaintools to parse the document and create embeddings locally using
LlamaCppEmbeddings. It then stores the result in a local vector database using
privateGPT.pyuses a local LLM based on
LlamaCppto understand questions and create answers. The context for the answers is extracted from the local vector store using a similarity search to locate the right piece of context from the docs.
GPT4All-Jwrapper was introduced in LangChain 0.0.162.
To use this software, you must have Python 3.10 or later installed. Earlier versions of Python will not compile.
If you encounter an error while building a wheel during the
pip install process, you may need to install a C++ compiler on your computer.
To install a C++ compiler on Windows 10⁄11, follow these steps:
Make sure the following components are selected:
Visit Official Website