Playing Video. mjpeg decoder in pure lua. With audio!

[flashkot]

Ok, ok. The subject is a joke. A little bit.

But, anyway, this can be considered as mjpeg decoder. We decode jpeg's? Yes. We get motion? Yes! Is it pure Lua code? No. But only thing it needs - our lovely framework.

When I was hit by this idea, i didn't know what something similar was already done.

My variant has smaller .love file size. Smaller memory usage (350mb peak for hq version), but very low quality.

So, anyway, try out this stuff! Any kind of feedback are welcome!

[josefnpat]

This is super cool! I knew it was only a matter of time before someone came in and did video better than my basic attempt. I have questions!

What kind of performance increase did you get from putting 16 frames in one image?

How big was the hq file before hand ?

Can you share the scripts you used to generate the .ogg file, and the .jpgs?

[flashkot]

I get source video from Wikipedia.

Then converted it to image sequence (and also scaled to needed width*height) with ffmpeg. Then images was sticked with montage.

I will post my scripts from home. You should wait something about 12 hours.

I do not know about any performance boost with 16 frames per image. The idea was: Load one second of animation and while it is displayed load another second.

In LQ example this worked great. In HQ i get bad performance on my 1GHz Atom processor.

Now i have some ideas, how this can be improved. I will try them at home and post results.

[retrotails]

This is awesome. I assume it would work with lossless images as well?

[Roland_Yonaba]

I won't comment the code, as I didn't took a look at it - yet -, but I was really impressed.
Amazing work!

[flashkot]

Code is very ugly.

I just made some tests with different jpeg images. Looks like it is faster to load many small images than loading one big. 320x200 image loads 10 times faster than 640x400. I think image size has some limit after which it slows down a lot.

[flashkot]

Okay, i wash my hands. I get all fun from this idea. Maybe I come back to it someday. So now it's your turn - do what you want, but do not forget to share your beautiful results

With good fine-tuning this method can allow you to have a nice cutscene in your game, played at full 25fps with resolution around 800x480. Most important here - size of images with video frames. Looks like what they should be not bigger than 800x800. So, just experiment. Maybe your choice will be one frame per image, or two.

Also remember what you can use anamorphic widescreen image.

Take a look at mjpeg_player_2hq.love demo. I think it is very cool to have such video just in 8mb of data.

Another idea what can be added to this code - "adaptive bitrate". Your code can switch to lowres version if there is not enough CPU power. Add 3mb of data and you have three versions of your cutscene - HQ, normal and low. Adding automatic switch of quality depending on available CPU should be an easy task.

Or, you can make your video very low quality and low res (180x100 for example) and enhance it with some artistic shader. Take a look at mjpeg_player_shader.love. Only 1.5mb of data, but with specially created animation it can be very cool and be a part of atmosphere of your game. 7 minutes of cutscenes just in 10mb! I think its nice option.

About the idea itself:

In simple words this is just a big SpriteBatch what streams it's frames from disk (frames are loaded just before they get to the screen and almost instantly get lost).

Here are not much difference with josefnpat code. I just used jpg instead of png (this saves a lot of space) and added helper thread what loads images for me (so there is no locking, one thread loads frames, another shows animation).

Now i describe process of video preparation:

You wil need folloeing tools: Python, ffmpeg and ImageMagik. this links are for WinUsers. If you with Linux, you should know what to do. (Sorty thous ones from the Mac Side, i do not know how to help you with this. Besides, you has cookies)

Now, with installed utils perform following steps:

1) Convert video to standalone frames:

Code: Select all

ffmpeg.exe -i Big_Buck_Bunny_Trailer_400p.ogg -s 640x480 -f image2 ffmpeg_temp/%05d.png

2) Combine individual frames (if you need this) With following Python script which uses montage utility from ImageMagik:

Code: Select all

#!/usr/bin/python2
import os
import math

total_frames = 813
cols = 1    # number of frames in each row
rows = 2    # number of frames in each column
frame_num = cols * rows 


cmd_prefix = "montage -format jpg -quality 35 -geometry +0+0 -tile"

# Use this variant, if you want to restrict montage with only two frames in a row. do you see -tile parameter?
# cmd_prefix = "montage -format jpg -quality 35 -geometry +0+0 -tile 2x "

for j in range(0,1 + int(math.ceil(total_frames / frame_num))):
    cmd = cmd_prefix
    for i in range(1, frame_num + 1):
        k=j * frame_num + i
        if k > total_frames:
            k = total_frames
        cmd = cmd + str(k).zfill(5) + ".png "
    cmd = cmd + "v_" + str(j + 1).zfill(5) + ".jpg"
    os.system(cmd)
    
    print (cmd)

3) Extract audio from source file:

Code: Select all

ffmpeg.exe -i big_buck_bunny_480p_stereo.ogg  -vn -acodec copy audio.ogg

This is for case when source audio already in Ogg Vorbis format. If this is not true, google for proper solution.

4) Now put all this in your Love project folder. take a look at my examples, its pretty clear how this should be done.
Then, modify settings. In my main.lua they looks this way:

Code: Select all

local frame_width, frame_height = 180, 100    -- this is size of individual frame
local scale_factor = 4                        -- scaling factor. In this case, resulting video will be  720x400
local frames_in_row, rows_num = 5, 5          -- How many frames in your images?
local video_fps, video_spf = 25, 1/25         -- Frames per second and seconds per frame

Now, you can play your video.

Happy hacking and don't forget to have fun!

[NÖÖB]

Awesome!

Put this in loader_thread.lua:

Code: Select all

while running do
collectgarbage("collect")

and this in main.lua

Code: Select all

function love.update(dt)
collectgarbage("collect")

Then the memory usage gets much lower -- 25mb

[dreadkillz]

This is pretty amazing. Thanks for the code!

[Roland_Yonaba]

Awesome!

Put this in loader_thread.lua:

Code: Select all

while running do
collectgarbage("collect")

and this in main.lua

Code: Select all

function love.update(dt)
collectgarbage("collect")

Then the memory usage gets much lower -- 25mb

Yes, that may work. Because, each loop update, the previously loaded userdata are no longer accessible, and thus can be tagged as garbage, and be cleaned fro memory.
But, what if we need some full video playing features ? go forth, go back , stop, resume, etc...

Which kinda make me think that the source can be optimized, for to enhance speed, or lower memory use.

@flashkot : For what reason did you used love.thread, by the way ?
Plus, did you envision some sort of streaming ? I mean, load some frames, display them, and progressively load the next frames in background while playing the video. It may reduce the reduce the memory usage, but i don't know if that would be faster. That's jus some random ideas I am throwing on the table, so that we can all learn from that.

Thanks for the amazing work, though.