Provide LLM-friendly content by adding a llms.txt file to help LLMs use an Obsidian website

patben · September 2, 2024, 5:22pm

Use case or problem

Today websites are not just used to provide information to people, but they are also used to provide information to large language models. There is no standard to do that correctly.

Proposed solution

Create a LLMs.txt markdown file that provides brief background information and guidance, along with links to markdown files (which can also link to external sites) providing more detailed information. See links below for all the details.

github.com

AnswerDotAI/llms-txt/blob/main/nbs/index.md

---
title: "The /llms.txt file"
author: "Jeremy Howard"
description: "A proposal to standardise on using an `/llms.txt` file to provide information to help LLMs use a website."
image: "/sample.png"
---

## Background

Today websites are not just used to provide information to people, but they are also used to provide information to large language models. For instance, language models are often used to enhance development environments used by coders, with many systems including an option to ingest information about programming libraries and APIs from website documentation.

Providing information for language models is a little different to providing information for humans, although there is plenty of overlap. Language models generally like to have information in a more concise form. This can be more similar to what a human expert would want to read. Language models can ingest a lot of information quickly, so it can be helpful to have a single place where all of the key information can be collated.

## Proposal

![llms.txt logo](logo.png){.lightbox width=150px .floatr}

We propose that those interested in providing LLM-friendly content add a `/llms.txt` file to their site. This is a markdown file that provides brief background information and guidance, along with links to markdown files (which can also link to external sites) providing more detailed information. This can be used, for instance, in order to provide information necessary for coders to use a library, or as part of research to learn about a person or organization and so forth. You are free to use the llms.txt logo on your site to indicate your support if you wish.

llms.txt markdown is human and LLM readable, but is also in a precise format allowing fixed processing methods (i.e. classical programming techniques such as parsers and regex). For instance, there is an [llms-txt](https://answerdotai.github.io/llms-txt/intro.html) project providing a CLI and Python module for parsing llms.txt files and generating LLM context from them.

This file has been truncated. show original

There was another related feature request (one or two years ago) to do just the opposite which is to protect a website from web crawling and scrapping (robot.txt config). So it would be nice to have both options: enable LLM-friendly content by adding a LLMs.txt file or on the contrary (try to) block it by setting the right parameter in the robot.txt file.