Had the pleasure of working with Jason Garrett-Glaser (better known as DarkShikari) circa 2011 on a patch for x264 to better utilize more cores (slice-threaded lookahead). At the time he was working with Sony to optimize their PlayStation game streaming service, and this had the potential to lower encoding latency (critical to game streaming) even on same-core systems by spreading the frame over more cores to do more parallel work, and doing more intelligent scheduling and convergence after that tile-based work was reconverged into a frame. At the time we were using servers with 48 physical cores to do transcoding, a bit of a rarity in a single NUMA space back then and there were no instances to test this on so I flamegraph traced a few encodes at very high thread parallel counts and send back the instrumented profiles (from a custom compiled version) showing the core work being done and he used that to optimize the motion estimation code which worked on the first shot to perfectly level out cpu usage and increase encoding speed across ~24 cores (was previously topping out around 8 and running one core hot iirc). I got a look into how exotic and complex this project was, particularly in terms how low level and advanced things like motion estimation, quantizing and perceptual compression were... it was and still is sort of magic how many smart techniques are baked into something that we barely pay attention to. They were, even at that time, working on x265.
His blog, Diary of an x264 developer, was a great read too for anyone wanting to get a vantage point on how crazy that codebase was.
Very cool. Took a similar approach to optimizing openutv by looking at flamegraphs. UTV is a fork of openrv which is an older release of the commercial RV. Like many things it uses ffmpeg and also native readers and writers for certain formats. Allowed me to get it to 8k@60fps buttery smooth playback with hardware acceleration. If you are still in the video/imaging space would love your feedback on UTV. openutv.com
One uncommonly used feature of H.264 is support for lossless video. It's a bit hard to use due to tooling (not easy to discover and select the feature), and is sometimes incompatible with web browsers, but you can get actual lossless video using a mainstream video codec. You can verify that it really is lossless, and doesn't have any conversion step where it's quantized to 8-bit YUV.
"x264vfw" is one tool which supports this format, even inside of an AVI file. So older software that uses Video For Windows can create those files.
FYI, both VP9 and AV1 support lossless as part of their base profiles, so lossless files using those codecs should be compatible with all implementations (at least in theory). Though those profiles only support 4:2:0, so it depends on your use case if that really counts as lossless.
I struggle to think of a situation where the blur of chroma subsampling is acceptable but very high quality or prores isn't good enough, I need lossless.
I've attempted to get researchers to use lossless H.264 to encode CT scans. They typically use directories full of tiffs. Even with intra-slice compression (i.e. tiff compression) they are pretty huge. But H.264 will let you recover the original images to the individual bit, and save a huge amount of space thanks to inter-slice similarities.
I’d imagine the main drawback of using H.264 here is that you’d still have to decode to get the frames back. This is similar to how even though many consumer grade cameras will output natively to h.264, if you want to edit the videos from it you’d usually have to convert it to prores or something first (the editor would usually do this automatically behind the scenes)
I’m not sure if lossless h.264 is I only or not, but if it uses inter-frame compression then it’s difficult to randomly access arbitrary frames. For any arbitrary frame you’d have to find the nearest key frame, then interpolate from there based on the inter-frame compression metadata.
Lots of hardware requires specifically h.264 as a transit; many of the MPEG standards are built around it. So this lets you get MPEG-PS, or lossless BDs, or some hardware/specialized software decoders for machines that otherwise couldn’t keep up (like set top boxes).
Cheaper to get a long fiber optic HDMI cable and connect your TV to your PC GPU. That's what I do. You get the best uncompressed video (typ 4K 120Hz or 8K 60Hz), zero lag so it can be used for stuff other than video, and don't have to maintain another box
Intel has supported native hardware decoder for AV1 since Tiger Lake. That’s quite old. You could buy a used NUC with Tiger Lake. Even when new a NUC with 1115G4 or 1135G7 isn’t expensive. I found one with damaged WiFi for $20. https://www.ebay.com/itm/366767313411 You can buy it now.
I went to the link to buy a computer (a NUC11TNKi3) with this capability for $20, but all it would let me do is place a bid on it; I cannot buy it now.
Sorry, but YouTube began thermal throttling on a Firestick the moment they started forcing AV1 regardless of the presence of on-device hardware support.
If the common hardware customers already have fails when you attempt to use a newer codec (but works fine otherwise), that can't be said to be a mainstream codec yet.
Your particular model is not representative of "the common hardware" as a whole. And it can be mainstream even when a significant fraction of devices can't use it. AV1 is a third of Netflix streams and most YouTube streams, that's more than enough to qualify as mainstream.
Even if half the devices were broken, it would still be more than a third of streams and mainstream.
A lot less than half of devices are broken when playing YouTube, I assure you.
If they stopped forcing it, it would still be the best choice most of the time. It doesn't "have to be" forced to be a major choice. The forcing changes the exact percentage but not the overall picture.
Nvidia Shield is hardly the most popular anything. Roku has 43% of the market and Amazon has 35%.
Would be interesting to see if we could segment third-party from TV-embedded (see quote), but the metrics from the article seem to reflect actual consumer usage, as most consumers just use what comes in the TV they buy nowadays.
> This reflects a consolidated approach and eliminates yet another device in the home, and it’s catching on, as just 34% consider streaming media players as their primary viewing device. Meanwhile, 56% said the same for smart TVs.
Very fair. Looks like Amazon is a bit better with AV1 support starting in 2021 with the Fire TV Stick 4k Max - 1st Gen[0], with a single absense of it on the seemingly uber-budget friendly $35 "Fire TV Stick HD Alexa Voice Remote (2024)".
That's still a sliver of the market, but the more royalty-free* codecs we use, the better.
>> The NVIDIA Shield TV (Pro and Tube) is widely considered the best Android TV streaming device for self-hosted media servers in 2026. While the Apple TV 4K wins on UI polish and the Onn 4K Pro wins on price, the Shield wins on something most other devices can't match: lossless audio bitstream passthrough. TrueHD, TrueHD Atmos, DTS-HD MA, DTS:X - the Shield passes them all to your AV receiver untouched.
>> Combined with hardware decode for H.264, H.265, VP9, and AV1, plus an 8-year-old Tegra X1+ chip that still gets driver updates, the Shield is a genuine "set it and forget it" Jellyfin client.
Nvidia shield is a device that simply works and passes the audio untouched to my sound receiver and I enjoy it __everyday__.
Shield is a 10 year old device at this point. It's essentially the 2017 hw that they've refreshed a couple of times with minor changes. Wait for the actual shield next for proper support.
Nvidia hasn't released a new model of the Shield since 2019. The current Apple TV model, Google TV model, Roku model, Fire TV, and Onn models support AV1.
AV1 decode works well enough even on quite old devices, and it has more favorable quality and size, individually or when considering both. I prefer it to h264 (or vp9 or even h265) at this point, and there isn't a particular need for hardware decode to benefit from it. On Youtube (probably the biggest and most accessible source of av1 video), when comparing size and quality on different codecs (h264/vp9/av1), av1 is either gonna have much smaller size while keeping some level of quality, or have better quality at similar size, or even have both be better.
Firesticks are quite popular. I'd like to see what percentage in the wild actually does decode AV1 in hardware, and without stuttering or buffering problems. Even my newer devices sometimes choke on h265.
The first Apple chips with AV1 support only came out in 2023. So the majority of Apple hardware out there doesn't support it. It will be a long time till support is at >95%.
At any point in time there's only a couple video codecs you should be using. When we can repeatedly shave off 30% of video bandwidth, there's no reason to stop making one or two new ones every decade.
Edit: And you can also look at the video codec landscape as having two major standards that put out new versions sometimes. The MPEG/h.26x series and the VPx/AVx series. It's not quite that simple but it's close.
There is no need to have a final one. It's no problem to keep supporting older codecs and add new ones in new products, as long as there's a "common ground" of widely supported codecs.
I don’t think we’re far off a “final” video codec, in the sense that jpg and mp3 and zip are effectively “final” image and sound and data codecs. Sure, there are newer codecs with higher compression foot a given quality level, but at some point it’s good enough to not bother upgrading, especially when the new version costs exponentially more for negligible gain.
…maybe THIS one will be The Final Codec… doodoodoo dooo
Yes, "neural" or "learned" video codecs are considered the future. They already have competitive compression ratios, but last thing I heard was that they are still much slower than conventional codecs.
I really like the idea that compression is intelligence, there are so many tricks an interesting ideas related to it. A bunch of them are described in the article, but you can imagine a long tail of many more inscrutable tricks chained together to get really incredible shrinkage in data.
Hutter prize (I think mentioned in a 3b1b video somewhere else in this thread) is a great example. Given this was written before advent of LLM, I would love to see someone throw openAI/Anthropic levels of compute at a similar problem to see what the swarm can produce.
I remember reading an article on here by one of the founders of MPEG about why he was leaving MPEG after several decades, and part of it was that modern AI is indistinguishable from compression and he wanted to explore new compression algorithms using the recent explosion of knowledge
The newest submissions to Hutter Prize made crazy jumps by just replacing the LSTM with a transformer and training it.
35'000$ for a prompt and some training is a nice chunk of cash.
Next steps are replacing the encoders and all the other human-biased compression parts by just raw data and training, probably another 20-30% is up for grabs.
WebP stands on its own merits but it's profoundly annoying that as of Pop OS 22.04 I still have to open them up in Firefox instead of Eye of Gnome. Compatibility issues like this are littered everywhere. This laziness on the part of supporting WebP is a significant driver of continued use of PNG.
Yes, it's very disheartening. We're approaching two decades of WebP, after all. At this point it's not even worth implementing its support because JXL is superior in every use case.
Not really, lossless WEBP still decodes much faster than lossless JXL, so if you need to decode the images faster, you use WEBP. Yes, you could also speculatively decode the JXL images to hide the decode time.
For lossy, neither is in a good place, since lossy AVIF beats both lossy JXL and lossy WEBP. Lossless AVIF is a joke and barely competes with PNG.
JXL decoding can still be optimized and had significantly less time for it, so let's not be so sure if that statement holds. Last update brought much improvement at just the area you mention.
And AVIF compression ratio beats JXL only conditionally. E.g. my photo library had AVIF at 29% and JXL at 25% the input size at the same perceptual quality (SSIM2=83). Sometimes it's on par. Sometimes AVIF wins. JXL wins almost always. Not to mention you can losslessly, reversibly transcode lossy JPG files to JXL, getting significant size reduction for free.
I think the case for JPEG XL is not that it's the best at everything, but that it is both quite good overall while also supporting basically every obscure image format feature under the sun. It's the potential USB of image formats.
With JPEG XL finally being allowed into mainstream web browsers, we’ll hopefully see more native platform support and be able to slowly phase every older codec out, since JPEG XL does very well at more or less everything (except for animation, compared to video codecs).
I hope so, but I disagree with the size of the "except": JXL performs better than APNG and GIF, and still supports transparency and lossless encode. Video codecs struggle here. So even with animation it still depends!
There's also a case for AVIF in some use cases (for this image it performs poorly). But if I had to choose one, I'd choose JXL.
H.264 High Profile is close to Patent Free. [1] I often wonder how much more efficiency could we get if we were to increase block size outside of spec.
I've shared this article many times. I work in space and the vast majority of the time, new people insist that all images should be raw. After all, you go to great expense to get somewhere in the name of science, you don't want to discard any information. And that's true in some cases but typically not for a whole mission or even a whole image if you can crop a specific part. Usually 100 or 1000 images instead of one is more valuable even if the compression is lossy.
That's just image compression though. Very few space-grade cameras support video compression. In a lot of cases it works well even though you think you're after a series of still images.
This reminds me that I still haven't watched `part 2` of a very nice video from 3Blue1Brown - "Compression is Intelligence". YouTube suggestions, definitely failed me with this one.
I remember reading this article when it was new, and I left fully agreeing, but more broadly: modern video compression is magic.
Like, it's baffling to me how much we can compress high-quality video nowadays. HEVC and AV1 and VP9 are all also very cool and they still sort of break my brain. I understand how they work but it doesn't feel like it should compress as well as it does.
I've done AV1 encodes where I was convinced the conversion must have failed in some way for it to be that small, scrubbed through it and it all looks fine, watched through the entire thing to confirm that wasn't the case, just waiting for the shoe to drop, but then to find out, no, it's really just that good.
I'm surprised that 10 years later, the article still teaches an incorrect conceptual understanding of "quantization".
The context heavily implies that quantization is "discarding information" generally (e.g. clipping, masking). The wikipedia article he links in the footnote disagrees, and so do I:
> quantization is the process of mapping input values from a large set (often a continuous set) to output values in a (countable) smaller set, often with a finite number of elements.
Quantization is a function over values, or the image itself. Clipping and masking, by contrast, are functions over the space/frequency dimensions, i.e. the pixel indices ("preimage"). A rule like "apply a circular mask" or "drop the first quadrant" does not depend on the values!
The most extreme example of quantization is a threshold filter. Regardless of location, output "1" if the pixel is brighter than X, or "0" otherwise. This depends only on the value, not on the location. He also misses the opportunity to point out that chroma subsampling is another application of quantization, this time to chroma values (drop the low-order bits).
Codecs like H.264 and JPEG use spatially-varying quantization rules to effectively discard more information at high frequencies, so his explanation of H.264 is spot on, but a non-expert encountering quantization for the first time would probably come away with the wrong idea. For an article explicitly targeted to non-experts that goes out of its way to introduce the term, it's a pretty important oversight.
Consider me confused. Isn’t quantizing a value, e.g. reducing the number of bits in an int, a form of discarding information from that value? Isn’t your comparison just about discarding information on another level (discarding entire values)?
There are some interesting advances in neural video encoders and decoders - I suspect it will look a lot like the neural models in DLSS 5 for compression -> decompression/reconstitution. There have been some lossless text-compression neural models built that seem to surpass even high level compression algorithms so with time those could get there.
Well, that was disappointing. Everything in that article is generic to any video compression codec, even MPEG. I thought this would discuss why H.264 compresses so much more than predecessors, but not a word about that. Should have been titled "video compression is magic".
The main reason to use H.264 in 2026 is legacy device support. Some devices cannot play H.265, but H.264 is playable almost anywhere, as a 20+ year old codec.
I was surprised that Twitter only lets you upload H.264 still, in 2026.
Twitter's video story is not their strongest. I used to run an account that posted a lot of video content, and their compression quality was so mixed that I ended up writing a little mobile app that would transcode videos to Twitter's advertised optimal format (720p 30fps H.264 5Mbps + AAC LC) before uploading them.
Even then, it was a crapshoot whether or not the video would look like a pixel collage, and I'd need to delete the Tweet, tweak the video a bit, and try again.
For personal use, if those are your only two choices, then go with H.265 for higher efficiency, or H.264 for wider compatibility. If other codecs are an option, go with AV1 for higher efficiency. It’s currently the best and already pretty widely supported.
It depends on how you want to play or share the media.
E.g. I remember having lots of issues sharing smaller video files encoded with H.265 via both WhatsApp and Messenger. iMessage doesn't support H.265 (or at least it didn't work for me at the time).
Patents aside, h265 from my reading is almost universally better than h264. My understanding which could be wrong is that for a given quality h265 will always be smaller, and particularly for >1080p it's recommended. Hardware support has been around since at least the Shield days (2019) so that shouldn't be an issue. Only use h264 if you are needing to hw-decode on really old hardware.
AV1 is the future/present: Fully open standard, better than h265. Only downside is hardware support: You need hardware made in the last few years.
"The H.264 video, 300 frames long is 175KB. A single frame of that video in PNG is 1015KB. It looks like we're storing 300 times the amount of data in the video. But the file size is a fifth. So H.264 would seem to be 1500x as efficient as PNG. How is this even possible? All right, what's the trick?"
Mostly the trick is they're comparing lossy to lossless compression which doesn't even make any sense.
I don’t think they are confused either just don’t think it’s a good way to introduce the point.
When good expositors choose motivating questions, say like 3 blue 1 brown, they don’t usually make apples to oranges comparisons. The best examples motivate while making sense before and after you learn.
Sorry, but it's only apple to oranges if you already know the answer. That answer, explained in the next section, is very clearly intended for someone who didn't already know it:
> This concept of throwing away bits you don't need to save space is called lossy compression. H.264 is a lossy codec - it throws away less important bits and only keeps the important bits.
> PNG is a lossless codec. It means that nothing is thrown away. Bit for bit, the original source image can be recovered from a PNG encoded image.
Actually I had the same thought, but this is obviously an article for people who don't know the difference between lossless and lossy compression. Ok, you might argue that it's not a good fit for HN, but going by the many times it seems to have been posted here, some people disagree.
It makes tons of sense to compare those numbers. It's a big part of how you benchmark lossy compression. I guess in some sense they should use BMP instead of PNG but it's arguable.
Sure that’s true. But if something makes sense for a technical benchmark that’s not a guarantee it’s the best way to introduce a concept. I just think other approaches could have been more clear for this case.
His blog, Diary of an x264 developer, was a great read too for anyone wanting to get a vantage point on how crazy that codebase was.
[1]https://github.com/mstorsjo/x264/blob/master/doc/threads.txt
(edit: was Gaikai that was acquired by Sony in 2012 and is the basis for PlayStation Now)
"x264vfw" is one tool which supports this format, even inside of an AVI file. So older software that uses Video For Windows can create those files.
Is it similar to lossless audio were there's not a ton of optimization you can do?
They are still using directories of tiff, though.
I’m not sure if lossless h.264 is I only or not, but if it uses inter-frame compression then it’s difficult to randomly access arbitrary frames. For any arbitrary frame you’d have to find the nearest key frame, then interpolate from there based on the inter-frame compression metadata.
the days where the OS matters for video playback are long gone
https://engineering.fb.com/2026/06/22/video-engineering/adop...
And soon AV2
https://av2.aomedia.org/
It looks like they normally sell for around $200: https://www.ebay.com/sch/i.html?_nkw=NUC11TNKi3&LH_Sold=1
I'm posting from a machine without AV1 acceleration. I disabled AV1 in Firefox's about:config and Youtube falls back to VP9.
Mainstream support doesn't have to be forced.
A lot less than half of devices are broken when playing YouTube, I assure you.
If they stopped forcing it, it would still be the best choice most of the time. It doesn't "have to be" forced to be a major choice. The forcing changes the exact percentage but not the overall picture.
Would be interesting to see if we could segment third-party from TV-embedded (see quote), but the metrics from the article seem to reflect actual consumer usage, as most consumers just use what comes in the TV they buy nowadays.
> This reflects a consolidated approach and eliminates yet another device in the home, and it’s catching on, as just 34% consider streaming media players as their primary viewing device. Meanwhile, 56% said the same for smart TVs.
https://www.cepro.com/news/roku-amazon-lead-streaming-media-...
https://developer.roku.com/dev/docs/media
So if you are saying Roku has 43% of the market, I would consider that as support for my point that AV1 is not that widely supported.
That's still a sliver of the market, but the more royalty-free* codecs we use, the better.
0: https://developer.amazon.com/docs/device-specs/device-specif...
*: https://arstechnica.com/gadgets/2026/03/av1s-open-royalty-fr...
This is so wrong..
> From Jellyfin blog: https://jellywatch.app/blog/jellyfin-nvidia-shield-tv-setup-...
>> The NVIDIA Shield TV (Pro and Tube) is widely considered the best Android TV streaming device for self-hosted media servers in 2026. While the Apple TV 4K wins on UI polish and the Onn 4K Pro wins on price, the Shield wins on something most other devices can't match: lossless audio bitstream passthrough. TrueHD, TrueHD Atmos, DTS-HD MA, DTS:X - the Shield passes them all to your AV receiver untouched.
>> Combined with hardware decode for H.264, H.265, VP9, and AV1, plus an 8-year-old Tegra X1+ chip that still gets driver updates, the Shield is a genuine "set it and forget it" Jellyfin client.
Nvidia shield is a device that simply works and passes the audio untouched to my sound receiver and I enjoy it __everyday__.
EDIT: <trying to fix formatting>
Maybe a device from 2019 is a bad data point
https://news.ycombinator.com/item?id=16796127
Otherwise we will have this:
https://xkcd.com/927/
Edit: And you can also look at the video codec landscape as having two major standards that put out new versions sometimes. The MPEG/h.26x series and the VPx/AVx series. It's not quite that simple but it's close.
…maybe THIS one will be The Final Codec… doodoodoo dooo
And then we'll get AI-designed hardware codecs for them.
4.nov.2016 https://news.ycombinator.com/item?id=12871403 219 comments
17.mar.2022 https://news.ycombinator.com/item?id=30710574 219 comments
24.may.2019 https://news.ycombinator.com/item?id=19997813 180 comments
14.jun.2024 https://news.ycombinator.com/item?id=40681306 205 comments
Hutter prize (I think mentioned in a 3b1b video somewhere else in this thread) is a great example. Given this was written before advent of LLM, I would love to see someone throw openAI/Anthropic levels of compute at a similar problem to see what the swarm can produce.
https://arxiv.org/html/2404.09937v1
35'000$ for a prompt and some training is a nice chunk of cash.
Next steps are replacing the encoders and all the other human-biased compression parts by just raw data and training, probably another 20-30% is up for grabs.
PNG: 1039993 bytes - baseline
optimized*: 541539
WebP: 398974 - lossless mode, of course.
JPEG XL: 310778 - lossless, quick, and easy.
*after monumental compute effort; still probably some room for improvement but the effort is ridiculous.
For lossy, neither is in a good place, since lossy AVIF beats both lossy JXL and lossy WEBP. Lossless AVIF is a joke and barely competes with PNG.
And AVIF compression ratio beats JXL only conditionally. E.g. my photo library had AVIF at 29% and JXL at 25% the input size at the same perceptual quality (SSIM2=83). Sometimes it's on par. Sometimes AVIF wins. JXL wins almost always. Not to mention you can losslessly, reversibly transcode lossy JPG files to JXL, getting significant size reduction for free.
There's also a case for AVIF in some use cases (for this image it performs poorly). But if I had to choose one, I'd choose JXL.
[1] https://meta.wikimedia.org/wiki/Have_the_patents_for_H.264_M...
That's just image compression though. Very few space-grade cameras support video compression. In a lot of cases it works well even though you think you're after a series of still images.
Like, it's baffling to me how much we can compress high-quality video nowadays. HEVC and AV1 and VP9 are all also very cool and they still sort of break my brain. I understand how they work but it doesn't feel like it should compress as well as it does.
The context heavily implies that quantization is "discarding information" generally (e.g. clipping, masking). The wikipedia article he links in the footnote disagrees, and so do I:
> quantization is the process of mapping input values from a large set (often a continuous set) to output values in a (countable) smaller set, often with a finite number of elements.
Quantization is a function over values, or the image itself. Clipping and masking, by contrast, are functions over the space/frequency dimensions, i.e. the pixel indices ("preimage"). A rule like "apply a circular mask" or "drop the first quadrant" does not depend on the values!
The most extreme example of quantization is a threshold filter. Regardless of location, output "1" if the pixel is brighter than X, or "0" otherwise. This depends only on the value, not on the location. He also misses the opportunity to point out that chroma subsampling is another application of quantization, this time to chroma values (drop the low-order bits).
Codecs like H.264 and JPEG use spatially-varying quantization rules to effectively discard more information at high frequencies, so his explanation of H.264 is spot on, but a non-expert encountering quantization for the first time would probably come away with the wrong idea. For an article explicitly targeted to non-experts that goes out of its way to introduce the term, it's a pretty important oversight.
https://news.ycombinator.com/item?id=38099633
Should I use H.264 or H.265?
I was surprised that Twitter only lets you upload H.264 still, in 2026.
Even then, it was a crapshoot whether or not the video would look like a pixel collage, and I'd need to delete the Tweet, tweak the video a bit, and try again.
E.g. I remember having lots of issues sharing smaller video files encoded with H.265 via both WhatsApp and Messenger. iMessage doesn't support H.265 (or at least it didn't work for me at the time).
How are you generating video?
How are you playing it back?
AV1 is the future/present: Fully open standard, better than h265. Only downside is hardware support: You need hardware made in the last few years.
Mostly the trick is they're comparing lossy to lossless compression which doesn't even make any sense.
They're not actually confused here. It's a writing style, where you pose an interesting problem/scenario, then explain it.
When good expositors choose motivating questions, say like 3 blue 1 brown, they don’t usually make apples to oranges comparisons. The best examples motivate while making sense before and after you learn.
> This concept of throwing away bits you don't need to save space is called lossy compression. H.264 is a lossy codec - it throws away less important bits and only keeps the important bits.
> PNG is a lossless codec. It means that nothing is thrown away. Bit for bit, the original source image can be recovered from a PNG encoded image.