AnythingLLM: Private Document Chat for Your Team

For a five-person team, AnythingLLM makes the most sense as one self-hosted Docker instance with separate logins and a local model. It can keep document chat on your server, but selecting a cloud model sends the text used to answer your question to that provider. Local file storage alone does not make the conversation private to your business.

AnythingLLM is a good fit when your first job is answering questions from company documents and your next job is giving that assistant a few useful tools. Choose Desktop for an individual, Docker for a shared server, and managed Cloud when you want Mintplex Labs to operate the infrastructure.

Verified against the project's live website, documentation and GitHub repository on 5 November 2026. The setup below is a documented deployment path with recommended access restrictions, not a claim of a completed performance or security test.

What Is AnythingLLM, and Which Edition Fits?

AnythingLLM combines a chat interface, document retrieval, workspaces and agents around a model you choose. Docker, software for running applications in containers, packages its shared server edition. A large language model, or LLM, generates the answer; AnythingLLM supplies the documents, conversation and available tools that help it answer your business question.

Think of it as a company library. Workspaces are reading rooms, documents are the shelves, the search index finds useful passages, and the model writes an answer from the passages placed in front of it. The librarian can stay in your building or work for an outside service. That last choice changes who sees the text.

The Desktop and Docker feature comparison puts user management and workspace access controls in Docker. Installing Desktop on everyone's laptop does not create a centrally managed team service.

Managed Cloud uses an isolated AWS instance operated by Mintplex. Isolation from other customers is useful, but it does not meet a requirement to keep files on your own premises. Its documented limitations also exclude a built-in LLM, MCP tool connections and custom coded agents. MCP is a standard for connecting an agent to outside tools. You connect a supported model service instead. Built-in agents and custom executable code are different capabilities.

For a business owner, the decision is who will maintain the server. For an IT lead, it is which data and credentials may leave it. If the requirement is “our documents stay on our equipment,” start with Docker and local processing.

Models: Choose the Engine Separately From the App

AnythingLLM can connect to local or cloud model providers, so installing the application does not commit you to one model family. Its provider configuration distinguishes the system default, workspace overrides and the model used for agents.

The built-in engine uses Lemonade, a model-serving layer, with llama.cpp, software that runs compatible local models. It accepts GGUF model files, a packaging format used for downloadable models. The documentation includes Llama and Phi examples, while the Docker guide also shows Qwen. Pick a model that fits your machine rather than assuming an entire family has the same hardware needs.

External local connections include LM Studio, a model desktop application and server, and LocalAI. Cloud connections include OpenAI chat models, Anthropic Claude models and Google Gemini, plus generic OpenAI-compatible endpoints. The repository's integration list covers additional providers.

These are connection options, not a promise that every model handles documents, images and tool calls equally well. A model that writes a readable answer may still choose the wrong tool or fail to follow a long instruction.

AnythingLLM Ollama: They Can Work Together

Ollama is a separate model engine that can supply local answers to AnythingLLM. The application adds document workspaces and access controls around that engine. If you already operate a working local model service, keeping it is a reasonable starting point.

A fresh installation can also use Docker's optional built-in model engine, covered in the setup below. You do not need to add a separate model server just because you chose the team edition.

For a confidential Operations workspace, select a local chat model and check its agent model as well. A different workspace can use a cloud provider when its data policy allows it. The embedding model is a separate, system-wide choice, which is the important qualification to that flexibility.

AnythingLLM RAG: How Workspaces Use Your Documents

AnythingLLM's document search gives the model relevant passages; it does not teach the model every file permanently. Retrieval-augmented generation, or RAG, means searching your source material and inserting selected text into the question sent to the model.

An embedding is a numerical representation used to match related text. The application splits a document into chunks, creates embeddings, and stores a searchable index. When someone asks a question, retrieval selects useful chunks for the answer. The model's context window is the amount of text it can consider at once, including instructions, history and those passages.

There are two practical ways to provide documents:

• Attach a document to a chat when you need help with that particular file. AnythingLLM inserts its full text by default, subject to the context window. Attachments are scoped to that workspace and thread.
• Embed a document in a workspace when it should be reusable knowledge. It becomes available across that workspace's threads and to users with access to the workspace.

The document handling guide makes this scope change explicit. Moving an owner's draft into a shared workspace is a sharing decision, even if the original conversation felt private.

For example, put an approved returns policy into a Support workspace. “What must an agent check before authorizing a return?” is a useful retrieval question. “List every exception across all our files” requires broader coverage than a few relevant excerpts can establish. Check the underlying policy before treating an answer as exhaustive.

Use Query mode for the first document pilot. It is designed to answer from uploaded material and report when relevant information is missing. Chat mode also uses the model's general knowledge; Agent mode can invoke tools. These are the documented chat modes, and new workspaces may start in Agent mode, so make the choice deliberately.

Is AnythingLLM Safe for Private Business Documents?

AnythingLLM gives you control over the components, but privacy depends on how you configure every component that handles text. The useful question is where processing happens, not whether the application has a “local” label.

Files and stored text. Desktop keeps its application data in a local storage directory; its storage reference identifies parsed documents, cached embeddings, the database and direct uploads. In Docker, persist the application's storage directory onto your server. In managed Cloud, the corresponding application data lives on the hosted instance.

Embeddings and the index. The default LanceDB configuration stores the vector index on the AnythingLLM instance. Keep the built-in embedder for local text processing. If you select a cloud embedding provider, the text to be embedded must go to that provider even when the resulting vectors are stored locally. Selecting a remote vector database adds another location holding index data. Treat embeddings and indexed text as business data, not as anonymization.

The answer request. A cloud chat model receives the prompt content used to answer: your question, supplied conversation context and retrieved document text. Attaching or pinning a document can supply its full text. The original PDF can remain on your server while confidential sentences leave it. The receiving provider's terms and retention settings govern that copy; AnythingLLM cannot make a remote inference request stay on premises.

Agents and outside services. Web searches, external APIs and MCP tools can send information elsewhere. A local model does not turn those services into local services. For a strictly internal document assistant, disable external skills and confirm which tools are available before adding sensitive files.

The most consequential configuration detail is in the embedding configuration docs: the embedder is system-wide, while chat models can vary by workspace. This means a workspace using a local chat model can still send its documents to a cloud embedder. If confidential and externally processed work need different embedding policies, use separate instances rather than assuming a workspace model override supplies that boundary.

Also distinguish deletion from removing access. Removing a document from a workspace leaves the parsed file and cache in the document library. The privacy and deletion guide says to delete it from My Documents to remove the parsed source and cached embeddings across workspaces. Copies in your backups or external services need their own retention handling.

Telemetry is another setting, separate from model traffic. You can disable usage telemetry in Privacy or with DISABLE_TELEMETRY=true. That does not disable configured cloud providers, tool connections or model downloads.

Set Up a Five-Person Team on One Server

Start with one Linux server, persistent application storage, a local model and separate accounts. The example below assumes a fresh installation, Docker already installed, and an IT lead who can provide authenticated access through an existing private network and HTTPS reverse proxy running on the server.

The Docker system requirements list a baseline of 2GB RAM, a 2-core CPU and 5GB storage for the application. That is not a sizing promise for local inference or five simultaneous users. Add memory and disk for the model, documents and concurrent requests. On an Intel or AMD machine using default LanceDB, verify AVX2 support inside the server or virtual machine; ARM64 uses a separate build.

1. 1

   Create persistent storage and start the application

   Run this on the server as its operator. It adapts the official installation command by binding the web port to loopback for an HTTPS proxy, adding restart behavior, disabling telemetry and enabling local inference.

   export ANYTHINGLLM_STORAGE="$HOME/anythingllm"
   mkdir -p "$ANYTHINGLLM_STORAGE"
   touch "$ANYTHINGLLM_STORAGE/.env"
   sudo chown -R 1000:1000 "$ANYTHINGLLM_STORAGE"
   
   docker pull mintplexlabs/anythingllm:latest
   docker run -d \
     --name anythingllm \
     --restart unless-stopped \
     -p 127.0.0.1:3001:3001 \
     --cap-add SYS_ADMIN \
     -v "$ANYTHINGLLM_STORAGE:/app/server/storage" \
     -v "$ANYTHINGLLM_STORAGE/.env:/app/server/.env" \
     -e STORAGE_DIR=/app/server/storage \
     -e DISABLE_TELEMETRY=true \
     -e ENABLE_BUILT_IN_LLM=true \
     mintplexlabs/anythingllm:latest

   The mount keeps your data when the container is replaced. The ownership setting follows the image's documented user, 1000:1000; do not solve a storage error by making the directory writable by everyone. SYS_ADMIN follows the published command and grants a broad container capability, so keep this service on controlled infrastructure. See the official Docker installation instructions.

2. 2

   Select a local model and keep document processing local

   In onboarding, choose Built-in, download a model suited to the server, and keep the built-in embedder and local vector database. In Settings > AI Providers > LLM, confirm the selected model before uploading business files.

   The built-in Docker model guide requires ENABLE_BUILT_IN_LLM=true at container startup. Putting it only in the mounted .env does not enable the engine. The command above uses CPU inference. For NVIDIA acceleration, follow the guide's Container Toolkit prerequisites and add --gpus all; AMD and Intel GPUs have their own device and group settings.

   Download the engine, model and required runtime while internet access is available. Judge CPU performance with your pilot workload before inviting everyone. The built-in engine keeps one model loaded at a time, so start the shared pilot with one model.

3. 3

   Enable individual logins before opening team access

   On the server, open http://127.0.0.1:3001, or reach that loopback service through your administrative tunnel. In Security & Access, enable multi-user mode and create the first administrator account. You will be logged out; sign in with the new credentials.

   For this five-person example, create or invite four colleagues through user management, giving each the Default role. Together with your Admin account, that makes five users. The role documentation says Default users can chat only in workspaces they are explicitly added to. A Manager can view all workspaces, so it is too broad for a colleague who should see only one department's documents. Admin has full system access. The documented switch to multi-user mode is one-way.

4. 4

   Create workspaces and connect the team's browsers

   Create Shared Operations for approved procedures and add all five accounts. Create Owners for restricted material and assign only the people who should use it. Put each document in the appropriate workspace; do not embed sensitive material in Shared Operations and try to hide it with a prompt.

   Have your existing HTTPS reverse proxy forward the team's chosen hostname to http://127.0.0.1:3001 on this server, and restrict network access to your staff. The example intentionally exposes no public web port directly. Each colleague opens that HTTPS address and signs in with their own account.

5. 5

   Run an acceptance check before expanding

   Start with a small set of approved documents and ten representative questions, including two that the documents cannot answer. Check source passages, an ordinary user's permitted workspaces, and a user who should be denied access. Then have all five people send representative requests together and observe delays and memory use.

   Back up the persistent storage and mounted configuration together, using a stopped container or a consistent backup method. Protect that backup because it includes documents and configuration secrets. Record the deployed image version or digest before routine upgrades, and check that you can restore the pilot.

If you use an existing external local model server instead, check the container's route to it. localhost inside the container points to that container, not the host's model service. The host connection guide explains this distinction; use a reachable address and keep the model service restricted to trusted callers.

Agents and MCP: What Can Leave the Workspace

An agent adds actions to document chat. It can use tools to retrieve information or perform tasks, so its permissions deserve the same attention as the documents it reads.

In Settings > Agent Skills, select only the skills needed for the workflow. Some are enabled by default. The agent setup guide also lets you choose an agent model per workspace. For a first pilot, retrieving an approved policy is a better starting job than granting access to every business system.

Model Context Protocol, or MCP, is a common way for applications to connect models to tools. AnythingLLM's MCP configuration supports local command-based servers and remote connections. A tool might read a directory, query a database or call another service; the credentials you give it determine the reach of that action.

Self-hosted Docker supports MCP Tools, but not Resources, Prompts or Sampling, according to the Docker MCP guide. Configuration lives at plugins/anythingllm_mcp_servers.json inside the persistent storage directory. Open Agent Skills and use Refresh after changing it.

A locally launched MCP server runs inside the container. It sees container paths and explicitly mounted files, not an arbitrary folder on the host. Manually installed tool dependencies can disappear when the container is replaced, so make that installation reproducible before depending on it.

Do not assume a workspace restriction also restricts a tool's database credentials. Give the tool a narrowly scoped account, and verify its access separately. A read-only pilot makes the consequences easier to inspect before allowing actions that modify records or send messages.

AnythingLLM Alternatives: When to Choose Something Else

Choose AnythingLLM when shared document workspaces are the center of the job. Its advantage is how these pieces fit together, rather than exclusive ownership of document chat or MCP.

Open WebUI for a Broader Shared Chat Interface

Open WebUI is a better starting candidate when your main request is a general self-hosted interface across model services, with extensions you want to configure. Its documentation covers provider connections and extensibility, and its current essentials guide explains document attachments and reusable knowledge bases.

Stay with a working Open WebUI deployment if it already answers your document and access needs. Switching interfaces alone will not fix poor source documents or an unsuitable model. Favor AnythingLLM when you want to organize the rollout around a few clearly assigned document workspaces.

LibreChat for Per-User Tool Connections

LibreChat deserves a closer look when each person must connect tools using their own credentials. Its MCP documentation describes per-user connections, credentials and OAuth sign-in, plus tools attached to chats or agents.

That is a concrete reason to evaluate it for a business with individually authorized systems. The tradeoff is administering those connections and choosing models that reliably call their tools. For a small shared policy library with a narrow tool set, AnythingLLM's workspace-oriented setup remains the more direct starting point.

AnythingLLM vs LM Studio

LM Studio fits an individual who mainly wants to download, run and compare local models. Its current application docs also cover MCP, API serving and headless operation, while its offline guide includes local document chat. It can therefore be a useful tool on its own or the model service behind another application.

Choose AnythingLLM Docker when the deciding requirement is a shared document application with assigned workspace access. Choose LM Studio when the deciding requirement is operating models. The distinction is the layer you need, not “one can read documents and the other cannot.”

For the model-serving choice, see Ollama vs LM Studio. If neither fits your hardware or serving needs, the Ollama alternatives guide covers other runtimes.

What's Overhyped About “Private AI”

“Runs locally” is a configuration claim, not a complete description of where your data goes. A local interface can call a cloud model; a local model can call an external tool; a local index can be built by a cloud embedder.

“Chat with all your documents” also overstates what ordinary retrieval sees. It selects passages, and can miss the exception that changes an answer. Adding documents does not permanently train the model, and a cited passage still needs to support the conclusion.

Finally, an installation minimum is not a team performance guarantee. The model, context length, embedding workload and overlapping requests determine the experience. Adopt now if you have an owner for the server and a specific document task. Wait if you cannot yet define who may access the files or what an acceptable answer looks like. Keep an existing system if it already meets those requirements.

Frequently Asked Questions

Is AnythingLLM legit?

Yes. It is an open-source project from Mintplex Labs with public code and installation documentation. That establishes its provenance; it does not certify your particular model, plugins or server configuration as secure.

What are LLMs used for?

LLMs generate and summarize text, answer questions and help draft material. Here, the useful business task is answering a question with relevant passages from an approved document set. An agent adds tools for taking actions.

Are there free open-source AI models?

Yes, there are downloadable models you can run locally. Check each model's license and commercial-use conditions rather than assuming AnythingLLM's MIT license also applies to the model weights. Hardware and administration still have costs.

Can I run an LLM locally using Ollama?

Yes. Run a suitable model in Ollama and connect AnythingLLM to the reachable service. In Docker, account for container networking rather than assuming the host's localhost address will work inside the container.

What are the key differences between Ollama and AnythingLLM?

Ollama supplies the model-running layer. AnythingLLM supplies a document, workspace and agent application that can use that layer. They can work together, and current AnythingLLM Docker also has an optional built-in engine.

What is the cheapest way to run LLMs locally?

Begin with hardware you already own and a model that fits it, then check whether it handles your task fast enough. Buy more hardware only after identifying the bottleneck. There is no universal cheapest setup independent of workload and acceptable waiting time.

What is the dumbest LLM?

There is no useful universal ranking under that label. A model can write fluently and still fail at document retrieval or tool use. Judge it on your own questions, including cases where it should admit that the source does not contain an answer.

What is the smartest local LLM?

The right choice depends on the task, available memory and acceptable delay. Use a model that fits your hardware and evaluate source accuracy and tool behavior. This documentation-based explainer does not claim a benchmark winner.

How much RAM to run LLMs locally?

AnythingLLM's Desktop requirements give a basic local-use recommendation of 16GB RAM and an 8-core CPU. A server also needs capacity for its chosen model and overlapping requests. The much smaller application-only baseline is not the total memory budget for local inference.

Your Monday Move

Give one IT owner the server, one approved document set and a clear acceptance test. Start with local inference, the built-in embedder, local storage and Query mode; leave unnecessary tools disabled. Invite the other four people only after their workspace permissions are checked. If a cloud model becomes necessary, decide explicitly which text may leave before changing that setting.

For practical guidance on putting AI to work in a small business, join the newsletter.