Are people downloading YouTube videos server-side

Question

I'm looking to download youtube videos so that I can transcribe them and then store the transcripts in my db. My application is a Node.js server that runs in a docker container on EC2. I'm using @distube/ytdl-core to download the Youtube video and it works perfectly on my local dev environment.As soon I run it in AWS I get an error from Youtube that says "Sign in to confirm you&rsquo;re not a bot". I'm guessing Youtube has blocked the IP's from AWS and forcing this, but there are a number of services I've seen that offer some kind of Youtube import (e.g. Opus Clip [1], Clay.ai [2], Veed.io [3] etc.), so my question is how are they getting around this restriction? The only possible solution I can think of is downloading the file locally and then uploading it but I feel like especially for Clay.ai that seems unlikely.It seems like the suggestions online are to use a proxy server, but I don't trust random online proxy servers for a production application.[1] https://www.opus.pro/ [2] https://www.clay.com/claybooks/get-youtube-transcripts-in-seconds [3] https://www.veed.io/

PaulHoule · Accepted Answer

Many organizations block IPs from places such as AWS, Azure, Tor and where it is trivial to get new addresses.It's a great tactic for webcrawling but if you ever get caught they'll block those netblocks and nobody will get away with it again. (Try editing Wikipedia from Tor and see what happens.) I've been on both sides of that one.

sds357 · Answer

I've been using yt-dlp to download the cc transcriptions that yt provides. Not sure if that would work for you. There is a npm library to interface with yt-dlp from node. I think it's yt-dlp-exec