← All guides

YouTube transcripts on Render, Railway and Heroku: the block that never clears

Written by the VidWords Team · · Updated · Report a correction

A long-running container fails at this differently from a serverless function, and the difference determines which fixes are worth trying. Here the failure is total, permanent, and immune to every instinct you have about restarting things.

The 30-second version

Why it is permanent here and flaky on serverless

The distinction is worth internalising because it tells you which advice on the internet applies to you.

On Lambda or Vercel, each invocation gets egress from a huge shared provider pool, so different invocations leave from different addresses. You see a 40% failure rate that drifts, and "try again later" genuinely sometimes works.

On Render you have a small, fixed, published set of ranges per region. Render tells you what they are so you can put them in a database allowlist — which is a genuinely useful feature and also means the ranges are enumerable by anyone, including the party trying to identify automated traffic. Once that range is blocked:

Render notes that "your service might use any IP address within its associated ranges", so you may see a partial recovery if only part of the range is blocked — which is just enough intermittency to make people believe a restart helped. It did not.

The one that briefly works, and why it is a trap

Changing region moves you to a different shared range with a different reputation, and it usually works immediately. That is what makes it dangerous: it feels like a diagnosis and a fix at the same time.

It is neither. You have moved into somebody else's neighbourhood, you are now generating exactly the traffic that got you blocked in the last one, and the clock is running. Meanwhile you have added latency to every other thing your service talks to. Teams migrate a whole deployment for this and are back in the same position in a fortnight.

Why dedicated outbound IPs are the wrong purchase

Render offers dedicated outbound IPs as an add-on, described as: "with dedicated IPs, your services send traffic through static IP addresses that you can provide to any external allowlist." Railway and Heroku sell equivalents.

Read that description carefully. It is built for allowlisting — for when a partner's firewall needs to recognise you. Being reliably recognised is precisely the opposite of what you need here.

Buying one gives you a single, permanent, datacenter address that is yours alone. Once it is blocked, it is blocked, and unlike a shared range there is nobody else's traffic to hide behind and no rotation at all. You have paid a monthly fee to become maximally identifiable to the party you needed to be unremarkable to.

The general rule, worth writing down: static egress is correct when someone is trying to let you in, and wrong when someone is trying to keep bots out.

What a long-running process can do that a Lambda cannot

The good news: your architecture has two real advantages here, and most people never use either.

1. A persistent cache that actually persists

You have a database and a process that stays alive. A published video's captions do not change, so every transcript you fetch should be the last time you ever fetch it. This is the highest-leverage change on this page: it does not reduce your block rate, it reduces the number of requests that can be blocked, which is better.

def get_transcript(video_id, lang="en"):
    row = db.fetchone(
        "SELECT segments FROM transcript_cache WHERE video_id=%s AND lang=%s",
        (video_id, lang),
    )
    if row:
        return row["segments"]

    segments = fetch_from_youtube(video_id, lang)   # the risky part
    db.execute(
        "INSERT INTO transcript_cache (video_id, lang, segments, fetched_at) "
        "VALUES (%s,%s,%s,now()) ON CONFLICT DO NOTHING",
        (video_id, lang, segments),
    )
    return segments

Cache the negatives too. A video with captions disabled will still have captions disabled tomorrow; re-fetching it forever is how a cache still leaks most of its traffic. Store the failure code with a sensible expiry and stop asking.

2. A real rate limiter, shared across your whole process

A serverless function cannot coordinate with its siblings without external state. You can. One token bucket in front of every outbound YouTube call, sized in requests per minute for the whole service, keeps a burst of user activity from turning into the traffic pattern that gets a range blocked in the first place. Pair it with a queue so requests wait rather than fail.

This is prevention rather than cure, and it is the thing most likely to keep you unblocked once you have got unblocked.

3. And one thing you must deliberately break

A persistent process wants to reuse HTTP connections, and that is normally correct. With a rotating proxy it is actively harmful: requests.Session pools the CONNECT tunnel, and a rotating gateway assigns an exit IP per connection. Reuse the session and your rotating proxy quietly stops rotating — every retry leaves from the address that was just blocked.

Build a fresh client for each retry attempt. Note also that youtube-transcript-api documents its client as not thread-safe (one session and cookie jar underneath), so if you fan out with a thread pool you need one client per thread regardless. Full mechanics in the RequestBlocked / IpBlocked guide.

The fix that actually addresses the cause

Since the problem is that your egress address is fixed, known and datacenter-owned, the only structural fix is to stop using it: route YouTube-bound requests through rotating residential proxies. Your container's own IP becomes irrelevant, which is exactly the goal.

Be clear-eyed about what you are buying. The library's README says directly: "Be aware that using a proxy doesn't guarantee that you won't be blocked, as YouTube can always block the IP of your proxy!" There are filed reports of residential rotation still failing — #511, #549 and #552 among them. It is a large improvement, not a guarantee, and it comes with a metered bill that scales with your traffic.

Add a fallback for when it is not enough: yt-dlp pointed at a different InnerTube surface (player_client of tv or ios) clears a meaningful share of blocks that a fresh IP does not. On a long-running host this is easy — you have no package-size limit, unlike the same fallback on Lambda.

Or take the address out of the equation

The full self-hosted stack is a proxy subscription, a rotation-aware retry policy, a fallback extractor, a cache, a rate limiter, and someone to fix it when YouTube changes. If transcripts are an input to your product rather than the product, that is a lot of surface area to own.

A hosted call collapses it to one request whose success does not depend on your container's IP at all:

const resp = await fetch("https://vidwords.com/api/transcripts", {
  method: "POST",
  headers: {
    Authorization: `Basic ${process.env.VIDWORDS_TOKEN}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ ids: ["dQw4w9WgXcQ", "9bZkp7q19f0"] }),
});
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);

const { results } = await resp.json();
for (const video of results) {
  if (video.error) {
    // no_transcript | video_unavailable | invalid_id — per video, not per batch
    console.warn(video.id, video.error);
    continue;
  }
  console.log(video.title, video.text.slice(0, 120));
}

Up to 50 video ids per request. One delivered transcript costs one credit; a video with no transcript costs nothing, so dead ids in a batch do not bill. A free account includes 25 transcripts a month and works for API calls, which is enough to port an integration before committing to anything. Details in the API reference, and the comparison guide covers the alternatives including staying on the open-source library.

A short triage list for a Render or Railway service

  1. Confirm it is actually an IP block. Map each exception separately — NoTranscriptFound and TranscriptsDisabled inherit from CouldNotRetrieveTranscript, so a broad catch reports normal content outcomes as infrastructure failures.
  2. Stop retrying. On a static address a retry loop is pure cost. Prove the address is the variable before you spend anything on it.
  3. Add the cache. Including negative results. Free, and it shrinks the problem.
  4. Add a rate limiter. Free, and it stops you re-earning the block.
  5. Then choose: residential proxy (own the stack, pay a metered bill, accept it is not a guarantee) or a hosted API (own less, pay per transcript).
  6. Do not change region as a strategy, and do not buy a dedicated outbound IP for this.

FAQ

Is this specific to Render?

No — it is the shape of the problem on any host with static shared egress, which includes Railway, Heroku and Fly. Render is named because its documentation is unusually explicit about how outbound ranges work, which makes the mechanism easy to verify. Serverless platforms fail differently; see the Lambda guide.

Would running my own VPS be better?

Usually not. A VPS is also a datacenter IP in a well-known range, and now the reputation of a single address is entirely yours. Cheaper, sometimes; structurally different, no.

Why not just use the official YouTube Data API?

It cannot return caption text for videos you do not own. captions.download requires OAuth and permission to edit the video. See the developer guide.

The video has no captions at all — will a proxy help?

No. There is nothing to download, so no address will help. That is a content outcome, not a network one — see why a transcript can be missing.

Read the API docs →