Streaming, encoding, and storage demands enormous costs -- especially at scale (e.g., on average each 4k video with close to 1 million views). Yet YouTube seems to charge no money for it.
I know advertisements are a thing for YT, but is it enough?
If tomorrow I want to start a platform that is supported with Advert revenues, I know I will likely fail. However, maybe at YT scale (or more specifically Google Advert scale) the economics works?
ps: I would like this discussion to focus on the absolute necessary elements (e.g., storing, encoding, streaming) and not on other factors contributing to latency/cost like running view count algorithms.
The biggest cost is as you imagine the streaming - getting the video to the viewer. It was a large part of our variable cost and we had a (literal) mad genius dev ops person holed up in his own office cave that managed the whole operation.
Ive long forgotten the special optimizations he did but he would keep finding ways to improve margin / efficiency.
Encoding is a cost but I don’t recall it being significant
Storage isnt generally expensive. Think about how cheap you as a consumer can go get 2 TB of storage, and extrapolate.
The other big expense - people! All those engineers to build back and front end systems. That’s what ruined us - too many people were needed and not enough money coming in so we were burning cash.
Google has a lot of custom encoding silicon, too, AFAIK.
Storage is the biggest question of the three. Linus Sebastian specifically called this out when YouTube started really pushing to make the non-Premium experience dreadful. There isn't really some secret special sauce you can buy or make for storage. Literally everything is being stored with the same hard drives, SSDs, discs, or tapes you can just go out and buy. The only specialization you can do is build or buy equipment to handle extreme numbers of them. Google does buy these in bulk, so they probably get a discount on storage, but it's not something that would make storage costs just go away.
https://www.businessofapps.com/data/youtube-statistics/#:~:t....
That's... a lot! Plenty of historical precedent for fully advertising-supported media with high expenses, from OTA network television and radio to free weekly newspapers... or inexpensive subscriptions to daily newspapers subsidized by advertising. Advertising has been paying the bills for electronic media for a century now.
egress costs were enormous and YT was not profitable. I don't know if it is now, but I wouldn't be surprised to find it is. They sure have enough ads.
As several people say below, caching content around the world is key, so that not all requests are serviced in NoCal.
Operate at a loss to drive out all competition and prevent new competition from arising. Increase ad obtrusiveness to drive up revenue, and every once in a while increase creator payouts to keep creators on your platform, until hopefully the lines cross and you start to make money. Maybe. Every once in a while a competitor might make a go for it, and you'll have to reduce ads or offer more incentives to creators to drive them out. Sometimes you may have to lobby the government to help you out on this.
One bonus is you can use the goodwill from your video site to drive traffic to your search engine.
If you don't have a fountain of infinite money from your search engine (or if you don't currently operate a gigantic search engine), then you might not be able to pull this off.
On the "Revenue" side you will quite probably need to have enough eyeballs that advertisers come to you directly to display ads and do so in volume.
On the "Costs" side you'd want to be big enough that you can just store your content in ~3 of your own datacentres, cache the "hot content" in a site or two per country then give away caches to ISPs (who will gladly host them in their own network for free).
Biggest cost will be bandwidth/streaming servers. Encoding/storage is comparatively cheap. If you were small you would likely start to do this from a few 100Gbps dedicated servers per continent. https://www.fdcservers.net/configurator?fixedFilter=15&fixed... If we set an average of 3-4Mbps per stream you're looking at each server handling 20,000 videos served and the hourly cost of the server would be around say $4/hour so you're looking at around $0.20 per 1000 video hours in theory, in practice it will be higher. Worst case closer to $0.50 per 1000 video hours due to utilisation rates.
If you have the ipvfoo extension, you can see it in action. (its easier to see with IPv6)
So in short, the only "on-demand" component is encoding, and if you don't have an 'available in an instant' promise, you can do it on spot instances on the cheapest cloud you can find; The rest is just storage and distribution - if you own a world-wide network of datacenters for your successful advertising service, that's kinda an already solved problem for you - just allocate a few racks to a new service.
I of course downplay everything and simplify massively - but at a high level, it's just a lot of ffmpeg -> S3 -> html5 player. The harder problems are in the long tail - high latency, content licensing & geo fencing, etc.
Source: used to SRE for a video streaming provider (not YT), also former GG
Besides the general advantage of having control over such a massive platform, which definitely plays an important part in the lives of hundreds of millions of people, Google likely views YouTube as important to control. If YouTube were a separate entity, it could e.g. freely choose their ad providers or even provides ads themselves, essentially creating competition for Google. Google also has trivial access to the data there and therefore the easiest access if they want to train AI on that data. Last but not least I think Google sees YouTube as vital for their corporate image and their social mission presented in Google Jigsaw.
> I know advertisements are a thing for YT, but is it enough?
Yes, it is -- virtually certainly. We can assume YouTube is profitable. It's not broken out directly in quarterly reports, but it doesn't make any sense that Google would still be running it after all these years (almost 20) if it weren't.
But obviously YouTube didn't start out as profitable. You need scale, which provides two things:
1) Marginal storage and streaming costs go down (Google is big enough to save huge amounts of money by running its own data centers, peering agreements, caching near customers, etc.)
2) More advertisers running more ads that can be targeted to more users whose preferences you know more about
So no, you can't run it profitably.
This is a classic example of a business that is only profitable at scale, that needs to lose a lot of money at first as it grows until it achieves scale. And it's not just scale on the traditional tech/users side, it's scale on the advertising side as well -- advertisers aren't going to bother running ads on your platform until you have enough users for them to care.
It's also pretty strongly a "winner-takes-all" network effects situation, where video publishers want to put up their videos where the viewers are, and viewers want to visit the site where all the content is. So if you wanted to create a YT competitor, I don't know how you'd convince content creators to post their videos to your site in addition to YT, or how to convince consumers to watch said videos on your site instead of YT.
Every month I notice the temperature of the pot is up a few degrees. This month it’s unskippable 15 second ads before most videos. Last month it was the first search result now being an ad. Before that it was how 5 second ads are now 7 seconds.
If I thought to write them all down I’d have a dozen more steps to share.
My kids now call it “the Bad YouTube” vs. YouTube Kids because the former is flooded with ads.
Netflix is.
Disney, peacock, Paramount, Max , etc are not profitable with the hope they can capture future monopoly standing.
Prime Video is likely also not profitable or break even given their studio investments (e.g. MGM, first party content).
Additionally, they don't pay the same electricity and water bill that others pay for their data center. They get a discount because they are creating jobs.
Getting streaming to be cost effective starts from decades of R&D investment + getting low cost electricity and water + owning supply chain.
Build vs buy pushes you to "build" early on when your margins are slim and your volume is huge.
Yor business would be wholly dependent on cloudflare. But if you don't have to pay for bandwidth, the economics aren't that bad.
(I ran a large porn site a lifetime ago, long before cdns were ubiquitous. If I was in the business today, I would absolutely put everything on R2 and make it work no matter how much client development it took)
Also: bandwidth gets a lot cheaper when you own the pipes.
And one that is less obvious: despite having hundreds of millions of videos available, a large contingent of people are watching the same ultra-popular ones. There are some economies of scale to be had there.
The marginal costs go down a bit if your scale is truly immense. Google can afford to design/manufacture/deploy hyper-efficient custom silicon ASICs for encoding. Also because their critical mass of users provide valuable network effects, they can get away with particularly poor quality encoding (IMHO) and the vast majority of users still won't switch to other platforms with higher visual quality - but other (non-pornographic) video platforms generally don't have that luxury.
You can do this for so much cheaper than AWS etc price for bandwidth. You can get 100gigE transit from he.net for list price $4500/month. Add probably the same again for colo + hardware (don't need much hardware these days to saturate 100gigE) and you can probably stream videos to 20,000 concurrent users at 5mbit/sec for ~$10k/month.
Another way would be to use someone like OVH who offers dedicated servers with 10gigE (supposedly 'guaranteed') for about $800/month each list price, without having to bother with colo and ip transit setup.
Obviously this is highly simplified as you will require encoding resource and storage, but again with someone like OVH you'd be able to spin up a lot of cheap boxes to do this. How much this will cost will depend on how many videos you get and how many views per video etc.
So IMO the actual bandwidth is a bit of a non issue. The far bigger issue is getting users to use your platform (marketing is MUCH more expensive than IP transit) and then having advertisers on your site. This is a much harder problem to solve and where the real barrier to entry is.
They're not paying a margin to advertising companies because they are the advertising company, they're not paying a margin to datacenters because they are the datacenter.
The data gleaned from YT views helps them to run search and vice versa.
Other CDN like akamai are available, but the cloud business has eaten most colo providers lunches (a "gray" fiber network can be expensive to hold).
Since Google has other services likely reusing the same internal CDN resources, their stream-traffic routing costs could be an order of magnitude lower than traditional providers.
Just a guess, but live feeds tend to be high-latency for a reason... =)
Since the main cost of video is streaming and the pipe required to deliver that streaming, they have an insane moat built because of it.
That is why. There are a lot of other explanations, but that is really why.
When you look at costs per unit, then it gets cheaper at scale, not more expensive.
For streaming, at scale you can afford to do peering yourself, instead of buying bandwidth.
For encoding, at scale you can afford special purpose encoding hardware, instead of using general purpose hardware.
For storage, at scale you can get cheap bulk deals with drive manufacturers.
Google uses search revenues to subsidize the cost of YT. It's anti-trust, but the government hasn't had the will to prosecute as of yet.
I think YT is finally self sufficient, but probably still relies on google's existing infrastructure to a large degree to keep costs down.
At the scale of the largest streaming apps (Disney, Netflix, YouTube, etc) you are moving petabytes of data PER DAY. At that size, you have access to significant savings on CDNs, backbone providers, etc. in many cases the discounts will be 90% - I have seen as high as 99% - or higher off the “list” price (which are usually never paid by anyone anyway).
You also tend to own your own backbone and can link in whichever ISP wherever you want for the “final mile.”
Final note, when you have been doing this long enough, you can start shaping the traffic based off previous patterns. I remember an eBay listing years ago for a Netflix local storage device that was meant to store shows at an ISP’s data center.
https://www.cnet.com/tech/tech-industry/google-wants-dark-fi...
Google bought Youtube the next year, 2006. This must have been massively useful for moving video around at lower cost rather than the public networks, which were probably built more for normal web traffic. Then peer with local companies who have the clients that are watching videos.
How Zoom manages to do this in a client-server fashion and is still financially solvent is also a question I've had for a while, but like others say, discounts on the transport and peering arrangements will be a key part in making those economics work, as compression and storage are relatively solved problems here.
Google services margin is around 30%. Even if YT is burning money they are likely making $5B in profit. They don’t report profit by income streams.
Google was one of the first internet scale companies with likely 10s of millions of servers and fiber that they own around the world.
They are also doing quite well in Cloud. Not as well as Azure and AWS but that division is growing.
At this point, something like Tiktok with a better sticky algorithm is the way to beat Google.
One of my mistakes was thinking Facebook will be crushed by Google.
Plenty of blind spots when you’re a big ship like Google.
It doesn't support up or down scaling or recoding. Force the user to only upload supported file formats and codex.
Minimal hosting by forcing viewers to also upload: the WebTorrent protocol is ideal for this.
Why ads at all? Those already exist. It's an ideal now platform to try out micropayments. Look at http://value4value.io for example.
If this starts to roll, only then you scale.
1 - They use their own custom hardware encoders now optimised for their scale (previously they’d use a ton of intel cpu for encoding) This is often not possible for smaller startups to have their own custom hardware, but one could setup their own Xilinx FPGA based hardware accelerated encoding servers, Xillinx Alveo Boards are popular for media acceleration, they support both ffmpeg and gstreamer for acceleration needs. Nvidia Nvenc encoding is decent too but mostly for the most popular encoding types, and honestly an fpga based accelerator is probably better after cpu unless you have gpus lying around.
2 - Much lower bandwith costs, a lot of startups build their streaming applications on top of AWS, but that’s insanely expensive due to the insanely expensive price of AWS itself
As a streaming service you could immediately cut down costs by shifting to ovhcloud, hetzner etc, they often have dedicated gameserver like offerings which are beneficial for streaming usecases too. That itself should allow you to bring down costs by 10x or even more (you have to choose the plan correctly on other tho, ovh, hetzner etc say you have unlimited bandwith on some packages but there is always a catch or they throttle you later, need to be careful about the offering you choose but otherwise a lot better)
3 - Having colocation hosts with your own servers setup and each video encoded at different sizes for different media devices also help only sending the near-right amount of content quality based video size based on device screensize and resolution. This gives a lot of additional cost advantage
4 - Running your own CDN relay service, I’ve run my own cdn service before to cut down on costs aws cdn is quite expensive, but even cheaper cdn services like bunnycdn, etc come at a decent markup, cloudflare cdn can be nearly free yes, but they’ll kick you out the moment you start consuming too much bandwith. So I hosted my own anycast network with geodns and controlled multiple servers across the globe acting as cdn with ansible playbooks for maintenance and control, and monitored them with my own monitoring stack to keep the cdn service alive and kicking, it work decently nice honestly, yes it required me to learn additional stuff, but it made me even more happier, price for sufficient level of bandwith it essentially allowed me to get very cheap rates for cdn service, along with additional cost optimisations at the edge by pre-allocating more cdn servers at peak usage to relay the stream once to the edge server and then broadcasting it from there to all local devices snappily and then bringing down the number of servers, once a major stream is over.
That takes care of cost reduction in encoding, bandwith and services like cdn.
There are a 110 otherways to reduce costs, but those are the primary ones you want to invest in, I excluded storage because you can already get cheap enough storage from a lot of managed providers like wasabi, backblaze, etc moreover streaming typically is less focused on long term storage so short term storage cost optimisation should be last resort, using reliable storage is a better idea.
Here’s an additional talk by mike solomon (youtube engineering) on scalability from 12 yrs ago [5] also the written blog version of it is here [6], a lot of it is still relevant, especially for young startups because bigtech youtube scale decisions are often not rational for young startups who have 1-2 years runway and need to make the most out of vc money or heck even bootstrapped haha (I was bootstrapped so price was a big deal for me, hence had to do a lot of stuff custom).
While making decisions always make sure to ask yourself at each step, “is this move relevant for my business” at youtube scale where their bandwith costs are in billions, hiring a $30 million hardware acceleration team makes sense to them, it does not to your startup, always take tips and hints from large behemoths like youtube, instagram, etc who’s devs love showing how they do a lot of things in talks available across internet but never forget to ask yourself the critical question of if its relevant to me.
A lot of the optimisations i did like custom cdn, managing my own storage, algorithms to allocate edge servers to reduce central load while boosting reliability, low latency and also reducing costs, does require very enthusiastic developers who love to learn things and are decently competent about it too. Each optimisation might give you 5-7% boost, some big ones will give you 200-500% boosts, but even those small changes add up to a big cost reduction overtime, if you get a big bill initially for your streaming service, dont get disheartened, allocate a team that specifically focuses on this and has deep tech knowledge doing 7%-10% optimisation each month across parts of the stack, and compound interest will help you move mountains.
Good luck to everyone here who are working on their streaming startups !
[1](https://www.semianalysis.com/p/google-new-custom-silicon-rep...) [5](https://www.youtube.com/watch?v=G-lGCC4KKok) [6](https://highscalability.com/7-years-of-youtube-scalability-l...)
Quite sure they are making lots of money.
And Hetzner will sell you 1TB of hot storage on a 1Gbps connection for €3.49/month.
Taking these consumer level price points, you just need to make more than €4/user month to cover your marginal costs. Quite a demanding, intensive user too.
Google, famously shitty at branding (and that's being kind), paid an obscene $1.6B for YouTube because "Google Video" was a monumental failure. Of course it was: Everyone thought it was just a search engine for video, not something that you would contribute to... any more than an individual "contributed" to Google's search results.
So they rewarded an enterprise that had no business plan.