← All guides

YouTube transcripts on AWS Lambda: why it fails, and why the NAT gateway makes it worse

Written by the VidWords Team · August 7, 2026 · Updated August 7, 2026 · Report a correction

Lambda has a specific relationship with this problem. It fails intermittently, which sends you down the wrong path — and the standard networking fix is the one thing that turns an intermittent failure into a permanent one.

The 30-second version

• Lambda's default egress is a shared AWS address you neither see nor control. AWS documents that "by default, Lambda functions run in a Lambda-managed VPC that has internet access". You get whatever address that infrastructure gives you, and it is AWS space — the exact category youtube-transcript-api's README says YouTube blocks.
• So it fails sometimes. Different invocations leave from different addresses with different reputations. Roughly half your invocations working is the signature of this problem, not evidence that your code is flaky.
• The NAT-gateway fix backfires. Pinning egress to one Elastic IP gives you a stable, AWS-owned, single address. You have removed the only thing that was accidentally helping you — variety.
• Two of the "blocks" are not blocks. A 3-second timeout and a 250 MB package limit both produce failures people file as IP bans.

Why Lambda fails intermittently and a VPS fails consistently

This distinction is the whole page, so it is worth being precise about.

A long-running container on Render or Railway has one outbound address. If YouTube blocks it, everything fails, forever, identically. Unpleasant, but at least it is legible — you know within a minute that you have an IP problem.

Lambda is the opposite. Each execution environment gets network egress from AWS's own pool. Two invocations a second apart can leave from addresses whose reputations are completely different, because thousands of other AWS customers share that space and some of them are running scrapers right now. Your function therefore fails at some rate between 5% and 95%, and that rate drifts through the day.

That produces the classic wrong diagnosis. You see partial success, so you conclude it is a rate limit or a race condition, and you add a retry loop with a sleep. On Lambda that is the most expensive possible wrong answer: you are billed for wall-clock duration, including the time you spend sleeping. A three-attempt backoff at one second a try adds three seconds of billed compute to every failing invocation, and it does not make the fetch more likely to succeed if the retry reuses the same connection (see the mechanics of the block — a pooled requests.Session keeps the same exit IP).

The NAT gateway trap

The next thing everyone does is search for "Lambda static outbound IP", and AWS has a well-documented answer: attach the function to a VPC, put it in private subnets, and route 0.0.0.0/0 to a NAT gateway that holds an Elastic IP.

That works exactly as documented. It is also, for this specific problem, a downgrade:

• You now have one address instead of many. Before, some invocations got lucky. Now every invocation leaves from the same place. If that place is blocked — and an Elastic IP is an AWS-owned address in an AWS range — you have converted a 40% failure rate into a 100% failure rate.
• You are paying for it. A NAT gateway bills hourly plus per gigabyte processed, permanently, whether or not it fixed anything. This is the single most common way people spend real money making this problem worse.
• You added cold-start and failure modes. VPC attachment brings Hyperplane ENI lifecycle into your stack. AWS notes that a function idle for 14 days has its unused ENIs reclaimed and goes Inactive, and that the next invocation fails while it re-provisions. On a low-traffic transcript job that is a real, periodic, confusing outage.
• There is a classic misconfiguration waiting for you. AWS states plainly that "connecting a function to a public subnet doesn't give it internet access or a public IP address." Half of the people who attempt this end up with a function that has no internet at all, which looks like a total block.

The rule: a static egress IP is the right answer when a third party needs to allowlist you. It is the wrong answer when a third party is trying to identify and block you. Those are opposite goals and they need opposite network designs.

The two failures that are not blocks

The 3-second timeout

A function created in the Lambda console starts with a 3-second timeout. A transcript fetch is several sequential HTTPS round trips to YouTube, and if you have added a proxy it is more. It does not fit in three seconds reliably, and it never fits when the first attempt has to retry.

The tell is that your error is not an exception from the library at all — it is Lambda killing the invocation. AWS's own documentation for diagnosing a function with no internet access uses exactly this string as its example: "Task timed out after 3.01 seconds". If that is what your logs say, you do not have a transcript problem yet; you have a timeout, and you cannot see the real error underneath it until you raise the limit.

Raise it deliberately. Lambda's ceiling is 900 seconds (15 minutes), but set it to what a fetch should take plus headroom — 30 seconds is usually right for one video. A 15-minute timeout on a function that hangs is a 15-minute bill.

The 250 MB unzipped package limit

The most effective free fallback for a blocked request is to ask a different InnerTube surface via yt-dlp with player_client set to tv or ios. That is what our own production service does. On Lambda it is awkward:

• The deployment package limit is 50 MB zipped when uploaded through the API, console or SDKs, and 250 MB unzipped — and that 250 MB includes your layers and any custom runtime.
• yt-dlp plus its dependencies is not small, and anything that wants ffmpeg is decisively not small.
• You get 5 layers, total, which is a real constraint once you are stacking a runtime, the library, and the fallback.

The workable route is a container image (10 GB uncompressed), which is a bigger change to your deploy pipeline than most people want to make for a fallback path. Worth knowing before you plan around it rather than after.

Related: /tmp is your only writable filesystem, configurable between 512 MB and 10,240 MB. It is per-execution-environment and it is reused across warm invocations, which is occasionally a free cache and occasionally a stale-data bug.

What actually works on Lambda

Cache first — Lambda makes this cheap and obvious

A published video's captions do not change. Before any network call, check DynamoDB or S3 keyed on (video_id, language). On a real workload this removes most of your outbound traffic, which removes most of your exposure to blocking, and it costs a fraction of a cent. Do this before you do anything else on this page.

Do not retry in-process — retry in the queue

Sleeping inside a Lambda is billed. Failing fast and letting the queue redeliver is not. Put video ids on SQS, let the function attempt exactly once, and let SQS's visibility timeout and redrive policy provide the backoff — with a dead-letter queue for the ones that keep failing. This also gives you the thing an in-process retry never does: a durable record of what has not succeeded yet.

Distinguish permanent outcomes so they never enter the retry path at all. NoTranscriptFound, TranscriptsDisabled, VideoUnavailable, AgeRestricted and InvalidVideoId will never succeed on a second attempt; only blocks, timeouts and connection errors are worth redelivering.

If you keep the library, put a residential proxy in front of it

The proxy becomes your egress, so Lambda's address stops mattering — which is the actual goal, and it is achieved without a VPC, without a NAT gateway, and without the ENI lifecycle. Build the client inside the handler rather than at module scope, so a warm container does not reuse a connection whose exit IP was just blocked. That single line is a real fix and it is free.

Or make it somebody else's problem

The honest calculation: a NAT gateway plus a residential proxy plus a retry queue plus a fallback client plus a cache is a week of work and a recurring bill, to acquire a capability that is not your product. If transcript retrieval is infrastructure to you rather than the thing you sell, a hosted call is a smaller dependency than any of the above — one HTTPS request, no VPC, no layers, no yt-dlp, nothing to keep current when YouTube changes.

import json, os, urllib.request

TOKEN = os.environ["VIDWORDS_TOKEN"]

def handler(event, context):
    body = json.dumps({"ids": event["video_ids"]}).encode()
    req = urllib.request.Request(
        "https://vidwords.com/api/transcripts",
        data=body,
        headers={
            "Authorization": f"Basic {TOKEN}",
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        results = json.load(resp)["results"]

    # Per-video failures arrive as an `error` field rather than sinking the batch,
    # so one caption-less video does not fail the whole invocation.
    ok = [r for r in results if "error" not in r]
    bad = [(r["id"], r["error"]) for r in results if "error" in r]
    return {"transcripts": ok, "failed": bad}

That handler has zero third-party dependencies, so the package-size problem disappears with it. Up to 50 video ids per request; one delivered transcript costs one credit and a video with no transcript costs nothing, so a batch containing dead videos does not overcharge you. Every field is documented in the API reference.

Deciding, honestly

• Stay on the open-source library if the Lambda is a nightly job with no user waiting, low volume, and a queue that can absorb failures. Add caching and SQS redrive, skip the VPC entirely, and accept a failure rate.
• Add a residential proxy if you need per-invocation reliability but still want to own the pipeline. Budget for the proxy and for it not being sufficient by itself.
• Call a hosted API if a user is waiting on the response, or if the person maintaining the proxy stack is also the person meant to be shipping features.
• Do not attach a VPC for this reason alone. Attach it when the function needs a private RDS instance, not when it needs to look less like a robot.

Other platforms

Vercel Functions, Netlify Functions and Cloudflare Workers behave like Lambda here — ephemeral egress from a shared provider pool, so the same intermittent signature and the same advice apply. The differences are duration limits and runtime support, not the block itself, which is why they do not get their own page on this site.

If you are on a long-running host instead, the failure has the opposite shape and needs different advice — see transcripts on Render, Railway and Heroku. For the underlying mechanism common to all of them, start with the RequestBlocked / IpBlocked guide.

FAQ

Read the API docs →