Page 1 of 4

Record a 60fps movie (with sound) using Uzem

Posted: Mon Sep 21, 2015 2:06 am
by Artcfox
Here is the result (a sneak peek of my work in progress):



And this is what I did to get it:

Edit: Scroll down a bit in this thread to this post, because I discovered a much better way to record the video and audio directly into an mp4 file in a single pass. I'll leave my original post here for historical purposes. If you want to see the results of the latest and greatest on-the-fly video recorder, click here and be sure to select the 720p60 stream.

First, run an unmodified copy of uzem fullscreen, and with the -c flag to capture your input:

Code: Select all

~/uzebox/bin/uzem -f -c physics.hex
This is necessary, because the recording process slows uzem down to a crawl, and this allows us to first play it through at full speed. Due to this delay, we can't actually capture sound and video at the same time (it won't stay synchronized), so two playbacks of the capture file will be required.

Then, play the capture file back (making sure you hit "Esc" when it's done) to record just the audio portion:

Code: Select all

SDL_AUDIODRIVER=disk ~/uzebox/bin/uzem -f -l physics.hex
Then convert the generated audio into a WAVE file:

Code: Select all

sox -u -b8 -c1 -r15700 sdlaudio.raw sdlaudio.wav
Next we'll need to use a hacked version of uzem to record a BMP file for each frame. Add the following chunk of code to line 321 of uzebox/tools/uzem/avr8.cpp, right after the controller capture/playback code. Save a copy of the original uzem binary, and recompile it with this recording hack in place:

Code: Select all

                // Record screen capture
                static uint32_t vidcapnum;
                char vidcapbuf[32] = {0};
                sprintf(vidcapbuf, "vidcap_%08d.bmp", vidcapnum++);
                SDL_SaveBMP(screen, vidcapbuf);

Recording each frame at 60fps takes a lot of disk space (about 84 MB/sec), so be prepared for that.

Next, run the hacked version of uzem with the -l (lowercase L) flag to playback the input capture you created previously, and create a ton of BMP files. Make sure you hit "Esc" to stop the recording at around the same time as you did before.

Code: Select all

~/uzebox/tools/uzem/uzem -f -l physics.hex
And then finally use avconv to combine the BMP files and the audio recording into a single (tiny) movie file called test.mkv:

Code: Select all

avconv -r 60 -f image2 -i vidcap_%08d.bmp -i sdlaudio.wav -c:v libx264 -r 60 -crf 21 -c:a copy test.mkv
To reclaim the gigabytes of disk space we just wasted, run:

Code: Select all

rm vidcap_*.bmp sdlaudio.raw sdlaudio.wav
And now you can share your video recording, or upload it to YouTube! :D

Re: Record a 60fps movie (with sound) using Uzem

Posted: Mon Sep 21, 2015 3:00 am
by uze6666
That's quite a procedure! Would native recording be a feature to have? A while ago Felipe Renaldi worked on that with ffmpeg but I never got to add his patch. Perhaps now is the time...

Re: Record a 60fps movie (with sound) using Uzem

Posted: Mon Sep 21, 2015 4:51 am
by Artcfox
uze6666 wrote:That's quite a procedure! Would native recording be a feature to have? A while ago Felipe Renaldi worked on that with ffmpeg but I never got to add his patch. Perhaps now is the time...
It is quite a procedure, but I'm sure that most of it can be scripted, so the procedure would just be to create a capture file that you want turned into a video, and then run the script.

Edit: Scroll down a bit in this thread to this post, because I discovered a much better way to record the video and audio directly into an mp4 file in a single pass. I'll leave my original post here for historical purposes. If you want to see the results of the latest and greatest on-the-fly video recorder, click here and be sure to select the 720p60 stream.

The only thing I would need is a flag that automatically quits uzem after it finishes playing back a capture file, and a flag that enables the BMP hack I posted, and then I should be able to script the rest.

Recording support is a feature that I've wanted for a while now, and I found that having the BMP files also works great for frame-by frame analysis.

The other thing I was able to do was create an animated GIF of select frames (the ones in brackets) using:

Code: Select all

convert -delay 1x60 vidcap_%08d.bmp[1428-1617] out.gif
Image

Though it seems Chrome doesn't honor the 60 fps that I set the GIF to, but if you download it and open it with a compliant program (like eog) it will play back at 60fps.

(To create this without the black borders, I ran the hacked uzem without the fullscreen flag.)

I figure stuff like this can help people share all things Uzebox, and hopefully attract even more developers.

Does Felipe's method work for sound? Also, ffmpeg is deprecated in favor of libav in Debian, so I'm not sure adding a dependency on ffmpeg is a good idea. The other thing I was thinking is I'm pretty sure that avconv can accept frames from stdin, so if uzem can pipe frames out over stdout, then you wouldn't need to add an actual dependency, and we could skip the intermediate disk phase. The audio would still need to be mixed in using a second pass, similar to how I mixed the BMPs with the wave file, but again that could be scripted.

Edit: Apparently Debian reconciled their differences with ffmpeg, and it will be distributed with Debian 9.0, but that's not going to be out for some time.

Re: Record a 60fps movie (with sound) using Uzem

Posted: Mon Sep 21, 2015 9:35 pm
by D3thAdd3r
uze6666 wrote:A while ago Felipe Renaldi worked on that with ffmpeg but I never got to add his patch. Perhaps now is the time...
I would really appreciate this functionality for the convenience and time savings(as primarily a windows user). Perhaps it could be disabled by default in the makefile for Linux builds, until the dependency issue is resolved. Wow what a political debate behind that on both sides libav vs ffmpeg too....interesting read for someone who every now and again will use Ubuntu.

Re: Record a 60fps movie (with sound) using Uzem

Posted: Tue Sep 22, 2015 12:36 am
by Artcfox
D3thAdd3r wrote:
uze6666 wrote:A while ago Felipe Renaldi worked on that with ffmpeg but I never got to add his patch. Perhaps now is the time...
I would really appreciate this functionality for the convenience and time savings(as primarily a windows user). Perhaps it could be disabled by default in the makefile for Linux builds, until the dependency issue is resolved. Wow what a political debate behind that on both sides libav vs ffmpeg too....interesting read for someone who every now and again will use Ubuntu.
Apparently libav is a Debian fork of ffmpeg, but they chose to name it after some of the functions that ffmpeg uses internally, so that's why it is so confusing, and hard to search for information about it. Aside from updates, libav and ffmpeg should be interchangeable, depending on which version of Linux one is using, so we shouldn't actually need to disable support by default for Linux.

If Felipe's patch supports sound correctly (based on the linked thread, I can't tell if the latest patch has working sound), I think that the only real change that might be needed to the patch is to bump the encoding framerate up to 60 FPS, since that's what Uzem uses now.

Using avconv on the command line, I was able to mux in the PCM audio directly (without converting it into another format) at 15.7 KHz, so I would surmise that it's possible to use ffmpeg as a library without having to first upscale the sound to 48 KHz.

Re: Record a 60fps movie (with sound) using Uzem

Posted: Tue Sep 22, 2015 3:47 am
by D3thAdd3r
For my own curiosity: Would the linux makefile need conditions to use either version, such as probably ffmpeg isn't going away it should be default, but current Debian builds could automagically compile without makefile modification by users? I really struggle and do not enjoy those types of things, I'm probably not alone either, but it could be a couple years before ffmpeg isn't depreciated on Ubuntu, etc. it seems.

An important person I forget...I'm trying to remember who the defacto "linux makefile guy" was, if there every really was one for more than a month long period.

Re: Record a 60fps movie (with sound) using Uzem

Posted: Tue Sep 22, 2015 4:37 am
by Artcfox
This is a much better way to record a video with audio, directly from Uzem into an mp4 file, in a single pass.

Okay, so I finally figured out how to get Uzem to record 60fps directly into an mp4 file with mp3 sound (not re-sampled) in just 13 lines of code, without linking with ffmpeg or avconv, or changing the Makefile! :D :D :D

The only thing it requires is installation of the ffmpeg or avconv binaries. For this proof-of-concept, it is expecting avconv, because that's what I have installed on my machine.

Here are the only changes that I had to make to avr8.cpp:

Code: Select all

diff --git a/tools/uzem/avr8.cpp b/tools/uzem/avr8.cpp
index 77698bb..4c19b24 100644
--- a/tools/uzem/avr8.cpp
+++ b/tools/uzem/avr8.cpp
@@ -236,6 +236,9 @@ void avr8::spi_calculateClock(){
     SPI_DEBUG("SPI divider set to : %d (%d cycles per byte)\n",spiClockDivider,spiCycleWait);
 }
 
+FILE* avconv_video = NULL;
+FILE* avconv_audio = NULL;
+
 void avr8::write_io(u8 addr,u8 value)
 {
        if (addr == ports::PORTC)
@@ -282,6 +285,9 @@ void avr8::write_io(u8 addr,u8 value)
                SDL_framerateDelay(&fpsmanager);
 
                 SDL_Event event;
+               if (avconv_video == NULL) avconv_video = popen("avconv -y -f rawvideo -s 800x600 -pix_fmt bgra -r 60 -i - -an -b:v 1000k test.mp4", "w");
+               if (avconv_video) fwrite(screen->pixels, 800*600*4, 1, avconv_video);
+
                 while (singleStep? SDL_WaitEvent(&event) : SDL_PollEvent(&event))
                 {
                                        switch (event.type) {
@@ -459,6 +465,8 @@ void avr8::write_io(u8 addr,u8 value)
                        SDL_LockAudio();
                        audioRing.push(value);
                        SDL_UnlockAudio();
+                       if (avconv_audio == NULL) avconv_audio = popen("avconv -y -f u8 -ar 15700 -ac 1 -i - -codec:a pcm_u8 test.wav", "w");
+                       if (avconv_audio) fwrite(&value, 1, 1, avconv_audio);
                }
        }
 
@@ -2500,6 +2508,15 @@ void avr8::shutdown(int errcode){
        fclose(captureFile);
     }
 
+    if (avconv_video) pclose(avconv_video);
+    if (avconv_audio) pclose(avconv_audio);
+    FILE* avconv_mux = popen("avconv -y -i test.mp4 -i test.wav -vcodec copy -acodec mp3 -f mp4 uzem.mp4", "r");
+    if (avconv_mux) {
+       pclose(avconv_mux);
+       unlink("test.mp4");
+       unlink("test.wav");
+    }
+
 #if GUI
        if (joystickFile) {
                FILE* f = fopen(joystickFile,"wb");
It no longer uses a ton of space on disk because it's not creating BMP files, but it does slow down the gameplay a little bit, so you'll probably still want to create an input capture file first, and then play back that capture file with the hacked version of Uzem to create your mp4 file. Also this version expects to run full screen, but you can create your input capture file in whichever mode you prefer to play with.

I think this way might be better than using the ffmpeg library functions directly, because here all of the encoding overhead happens in another process (running in parallel, with the potential to use multiple cores for encoding if you pass it the correct flag), rather than happening on the same thread that Uzem is running in. The only additional work that Uzem does here is call fwrite() to stream the raw audio/video data out over a pipe for encoding.

Next it's just a matter of deciding how we want to make this configurable (a command line option to turn on recording, and one to choose ffmpeg or avconv?), so it feels more like a real feature, and less like a quick and dirty hack.

Re: Record a 60fps movie (with sound) using Uzem

Posted: Wed Sep 23, 2015 11:19 pm
by uze6666
Pretty nice work I say! This will be really useful. I will review it and test it right away. This patch is against the uzem140 branch?

Btw, how hard would it be to link to it and have only one executable? Or can we make a distro with a something like a simple DLL like SDL?

Re: Record a 60fps movie (with sound) using Uzem

Posted: Thu Sep 24, 2015 1:02 am
by Artcfox
uze6666 wrote:Pretty nice work I say! This will be really useful. I will review it and test it right away. This patch is against the uzem140 branch?

Btw, how hard would it be to link to it and have only one executable? Or can we make a distro with a something like a simple DLL like SDL?
Thanks! This is against the master branch, but it's such a tiny patch that I don't think it will be that different to apply to the uzem140 branch.

Also, by changing the fullscreen resolution of Uzem from 800x600 to 640x480, not only does the emulator run faster on my laptop, it also means that 691,200 fewer bytes per frame get sent over the pipe to the video encoder, so on a faster computer, it's possible to record video as you're playing the game in real-time. :)

The other thing that I discovered is that YouTube won't play 60fps videos unless they are 720p or higher, so after making the change so full screen resolution is 640x480, I told avconv to automatically upsample it to 720p using its built in high quality upscaling filter:

Code: Select all

if (avconv_video == NULL) avconv_video = popen("avconv -y -f rawvideo -s 640x480 -pix_fmt bgra -r 60 -i - -vf scale=-1:720 -an -b:v 1000k test.mp4", "w");
and now YouTube will play these videos back at 60fps!

Linking to it, you lose the benefit of having the encoding run in parallel, unless we complicate Uzem by making it multi-threaded again, which I think is a bad idea. It would be much easier to just distribute the ffmpeg binary along with our executable. Also, by calling the executable and piping the raw data to it, you can easily change the video size, encoding format, upsampling, etc… just by changing the command line parameters we pass to ffmpeg.

Here is an example of what the video looks like with Uzem modified so the fullscreen mode resolution is 640x480, and with the 720p upscaling filter applied. The best part is that since ffmpeg is running in another process, adding this upscaling filter into the mix did not change the performance of Uzem at all. Also, if you click on the gear icon in YouTube's HTML5 player, you can change the playback speed, so you can watch it in slow motion, or even speed it up if you'd like.

I think it will be great for "platform evangelism" if it's dead simple to create high quality YouTube videos that can show off the Uzebox in its full 60fps glory.

Edit: Updated video with 44.1 KHz audio fix.

Re: Record a 60fps movie (with sound) using Uzem

Posted: Thu Sep 24, 2015 1:28 am
by uze6666
Here is an example of what the video looks like with Uzem modified so the fullscreen mode resolution is 640x480, and with the 720p upscaling filter applied. The best part is that since ffmpeg is running in another process, adding this upscaling filter into the mix did not change the performance of Uzem at all. Also, if you click on the gear icon in YouTube's HTML5 player, you can change the playback speed, so you can watch it in slow motion, or even speed it up if you'd like.

I think it will be great for "platform evangelism" if it's dead simple to create high quality YouTube videos that can show off the Uzebox in its full 60fps glory.
:shock: You totally convinced me! The quality is amazing and the whole process is simple...I love it, you the man! :mrgreen:

I'm trying it tonight...If all goes well, I'll report soon.

ps: Your game is looking great!