← All guides

Loading YouTube transcripts in LangChain for RAG & video Q&A

Written by the VidWords Team · · Updated · Report a correction

LangChain makes it easy to build Retrieval-Augmented Generation (RAG) applications over video libraries. Here is how to ingest transcripts, preserve timestamp metadata for accurate citations, and avoid cloud IP blocking.

The standard LangChain YoutubeLoader and its cloud limitation

LangChain provides a built-in document loader: langchain_community.document_loaders.YoutubeLoader. Under the hood, it calls youtube-transcript-api. While this works on a local machine, deploying your LangChain app to AWS Lambda, GCP Cloud Run, or Vercel frequently throws IpBlocked or CouldNotRetrieveTranscript.

Production-ready LangChain loader with VidWords

To ensure 100% uptime in cloud environments, use a custom loader that fetches transcripts via the VidWords REST API:

import requests
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

def load_youtube_documents(video_id: str, api_token: str) -> list[Document]:
    url = "https://vidwords.com/api/transcripts"
    headers = {"Authorization": f"Basic {api_token}"}
    res = requests.post(url, json={"ids": [video_id]}, headers=headers)
    res.raise_for_status()
    data = res.json()["results"][0]
    
    docs = []
    # Create segment documents preserving start timestamps
    for seg in data["segments"]:
        doc = Document(
            page_content=seg["text"],
            metadata={
                "source": f"https://youtube.com/watch?v={video_id}&t={int(seg['start'])}s",
                "video_id": video_id,
                "title": data.get("title", ""),
                "start": seg["start"],
                "duration": seg.get("duration", 0),
            }
        )
        docs.append(doc)
    return docs

# Split into semantic chunks
docs = load_youtube_documents("dQw4w9WgXcQ", "YOUR_API_TOKEN")
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)
chunks = splitter.split_documents(docs)

# Vector store index
vectorstore = Chroma.from_documents(chunks, OpenAIEmbeddings())

Why timestamp metadata matters in RAG

When an LLM answers a user query using video context, preserving the start timestamp in the metadata allows the assistant to return exact clickable citations (e.g. [See minute 04:15](https://youtube.com/watch?v=...&t=255s)) so users can verify the evidence directly in the video.