Compress bulk videos and retrieval

jinsatkilife

Senior Member
Joined
May 4, 2019
Messages
1,096
Reaction score
1,151
I have alot of videos to store. They have HD+ definition and long duration.

As a result, they take up alot of space.

Does anyone know of a way to compress the videos so that they occupy smaller size and then decompress them again when u want to watch them again?

What applications are needed?

How do big companies like google/youtube store so much videos on their cloud without compromising quality?
 

davidktw

Arch-Supremacy Member
Joined
Apr 15, 2010
Messages
13,391
Reaction score
1,180
I have alot of videos to store. They have HD+ definition and long duration.

As a result, they take up alot of space.

Does anyone know of a way to compress the videos so that they occupy smaller size and then decompress them again when u want to watch them again?

What applications are needed?

How do big companies like google/youtube store so much videos on their cloud without compromising quality?

That is what video compression is for, to compress your pixel information using certain algorithms that can store them in much smaller space than the individual pixel would normally require. In most cases, each pixel will roughly take 32 bits (each of the RGB channel 8 bits each). Typical lossless compression techniques can only achieve roughly 1.5-3 times compression. The shear size of an image at 1080p(1920x1080) will take 1920x1080x4/1024/1024=7.9MB to store without compression. Assuming 2x compression, it will still takes roughly 4MB. If you consider a typical movie at 24fps, that would be 94MBps of bandwidth required, not to mentioned storing it. An hour show will be roughly 300GB to store.

That is why lossy compression is used. For image, the most popular lossy compression is JPEG. For video, there are numerous compression techniques. Some of the well known ones are MPEG-1 used in VCD, MPEG-2 in DVD, MPEG-4 used in streaming and also storage, but you also would have come across other popular ones like H.264, H.265, HEVC, AV1, VP8, VP9.

Even in analog days, videos streams are compressed using limited bandwidth for different components of the video. You may have heard of RGB, YUV, YCbCr, NV12. RGB is the one are is largely non-compressed, YUV is a different colour model which split the single pixel into luminance and 2 chroma components. Human eyes are more sensitive to brightness(luminance) than colour. Hence when storing an analog film, the luminance will get full bandwidth representation and colours are halved or compressed further. Hence there are notation 4:4:4, 4:2:2, 4:2:0 to tell how much bandwidth is allocated to each channel.

In modern days, nearly all digital videos are compressed. Even highly quality encoding like Apple ProRes are compressed. Therefore to answer your question. Storage are normally compressed using one of the above codec and written into storage, and decompress on the fly during play back.

There are ALOT of tools that does compression and decompression. It's all about performance. In most general form, compression and decompression can be performed in software by general purpose processors. This is normally the best quality at the expense of performance. Normally software compression/decompression is the slowest. In the days when processors are not fast enough to even handle simple VCD playback, we have hardware card just to perform decompression. These days processors are so fast that they can handle using software and your main processor. However then better codec that can perform better compression comes which demand even more powerful processors. Hence processors have acceleration features incorporated in them, or in the case of SoC design, dedicated hardware codec acceleration exist to offload these task so that the processor is freed for other computation.

So there is no one complete technique. There are a lot of techniques available to perform compression and decompression. One very popular software that can handle a lot of codecs is the OpenSource FFMPEG. (https://www.ffmpeg.org). You can find this software across all the platforms today. FFMPEG is a very complex tool that does more than just multimedia manipulation and certainly not just limited to video. There are other softwares such as Handbrake, Adobe media encoder, avidemux, vlc, gstreamer, xvid, x265, and a lot more. If you want a nice GUI, handbrake will help. If you want versatile command line encoder, ffmpeg and x265 is my goto options.

There are a lot of knobs and techniques when it comes to video compression. It's a vast topic itself. Long story short, other than the codec capability which will determine a lot about how well the compression works, it's still basically performance vs quality. You want high quality compression, it takes time. If you want good quality video on disk, it will require more data to represent the compressed data. There are a lot of metrics at play here, from frame rate, bitrates, codec, motion estimation, noise reduction, pre-processing of the source, and more. I'm not even an expert in this, except that I have spent a lot of hours researching and playing around with these tools throughout like a hobby.

How do you know big corporates like YouTube (owned by Google), Microsoft, Apple, Facebook and Instagram, Flickr and many more such media companies are not spending huge cost on just storing multimedia contents (audio, image, video, 3D, etc) ? In fact their cloud storage capacities are easily larger than petabytes or exabytes range.

There is no such things are no comprise in quality for images and videos. Due to the size of these medium, and how human and machines consume them, there is normally no necessity to store them in their most accurate form. When you take a photo using the DSLR, the sensor will record the data in RAW form. The RAW form is the most accurate digital representation of which the camera is capable of capturing. Capable DSLR offer this RAW format to professional photographers for most accurate editing. However, it is not uncommon for photographers to discard the RAW data after editing and simply storing the post-processed image in lossy compressed form like JPEG. The RAW files are normally in the range of tens or hundreds of megabytes. For videos, it is normally next to impossible to store in the RAW form because the bandwidth requirement to write to the storage like even SSD can be enormous and impractical. Hence videos are almost always lossy compressed before written to the disk. The quality is ALREADY compromised in this process.

Why quality compromised is acceptable for most cases is because human brain don't interpret multimedia data the way like machines. There are plentiful of scenarios that we simply cannot perceived one image is different from another even if both are not the same to the machine. All lossy compression techniques uses this psychovisual model to devise compression techniques which can compress the data but when decompressed will looks discernible to most human. There are a lot of criteria to how much compression make sense. One excellent example is if I know the image is to be displayed on a small Apple Watch and never on a 24" display. We could easily get away with much lower resolution and also heavily compressed. These techniques are also used in audio for which we have lossy compression like MP3, AAC, Opus, A-law, and many many more.

I could go on and on on these topics, and like I say, I'm not even an expert in any one particular, not to mentioned audio and video compression techniques are vast topics individually. I hope my long passage here would have provided you with valuable insight to the can you are attempting to open. These are all very interesting topics. Feel free to venture

:)
 

yusoffb01

Arch-Supremacy Member
Joined
Jun 17, 2008
Messages
15,474
Reaction score
869
compress takes too much time. just buy 12TB hdd for $300 at amazon.sg. If dont want to spend money, you can upload to private youtube.
 
Joined
May 11, 2017
Messages
62,450
Reaction score
11,791
I have alot of videos to store. They have HD+ definition and long duration.
As a result, they take up alot of space.
Does anyone know of a way to compress the videos so that they occupy smaller size and then decompress them again when u want to watch them again?
What applications are needed?
How do big companies like google/youtube store so much videos on their cloud without compromising quality?

look at h265, best compression at the moment


Read HWZ Forum Rules!
 
Important Forum Advisory Note
This forum is moderated by volunteer moderators who will react only to members' feedback on posts. Moderators are not employees or representatives of HWZ. Forum members and moderators are responsible for their own posts.

Please refer to our Community Guidelines and Standards, Terms of Service and Member T&Cs for more information.
Top