Record a 60fps movie (with sound) using Uzem

The Uzebox now have a fully functional emulator! Download and discuss it here.
User avatar
uze6666
Site Admin
Posts: 4801
Joined: Tue Aug 12, 2008 9:13 pm
Location: Montreal, Canada
Contact:

Re: Record a 60fps movie (with sound) using Uzem

Post by uze6666 »

Edit: And don't change the sample rate to 15734 in the audio line, because that will desynchronize the audio, keep it 15700, because that's what you configured SDL to use.
I'll check out the rest, but 15734 hz is actually the real NTSC line rate and that's what uzem has been using in the uzem140 branch. Why do think it should be rounded to 15700?
I guess I should have checked this thread first! I just finished making my own patch that makes recording optional.
Ah, communication issues just like in the real work life eh? :) ;)
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Record a 60fps movie (with sound) using Uzem

Post by Artcfox »

uze6666 wrote:
Edit: And don't change the sample rate to 15734 in the audio line, because that will desynchronize the audio, keep it 15700, because that's what you configured SDL to use.
I'll check out the rest, but 15734 hz is actually the real NTSC line rate and that's what uzem has been using in the uzem140 branch. Why do think it should be rounded to 15700?
Okay, that is great then! 15700 is what the SDL sample rate was set to in the master branch version. I just wanted to make sure the recording matched what SDL used to keep them in sync.
uze6666 wrote:Ah, communication issues just like in the real work life eh? :) ;)
Apparently so!

Oh, and the other thing I did to speed up the emulator (I can get 60fps on my slow laptop now!) is uncomment the:

Code: Select all

ARCH=native
line in Uzem's Makefile. :D
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Record a 60fps movie (with sound) using Uzem

Post by Artcfox »

Changing the video encode parameters to this:

Code: Select all

"ffmpeg -y -f rawvideo -s 640x480 -pix_fmt bgra -r 60 -i - -vf scale=-1:720 -sws_flags neighbor -an -crf 0 uzemtemp.mp4"
means I can play at 60fps while recording a video on my slow laptop, it made the artifacts in the video go away, and it made the file size of the recording go down by over 2 MB! :shock:

Code: Select all

-rw-r--r--  6721478 Sep 26 00:22 physics-cbr0.mp4
-rw-r--r--  8974378 Sep 26 00:27 physics-bv1000k.mp4
User avatar
uze6666
Site Admin
Posts: 4801
Joined: Tue Aug 12, 2008 9:13 pm
Location: Montreal, Canada
Contact:

Re: Record a 60fps movie (with sound) using Uzem

Post by uze6666 »

Tried it, on for me it still as slow even slightly worst, emulation speed drops at 8Mhz. And the file is also 30% bigger, which I would expect for lossless.

I've also tried "-preset ultrafast -qp 0" which speeds up my emulation to 17Mhz, but the file size explodes from 3Mb to 30Mb! So I guess there is still room to play and these ffmpeg string should be externalized to allow tuning.

Also noticed a "-tune animation" that could be tested to see what it yields.

I have a dual boot with Ubuntu on my current machine, I guess I will try to build and test it to see if the slowdown culprit is really my machine or piping under Windows.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Record a 60fps movie (with sound) using Uzem

Post by Artcfox »

uze6666 wrote:Tried it, on for me it still as slow even slightly worst, emulation speed drops at 8Mhz. And the file is also 30% bigger, which I would expect for lossless.

I've also tried "-preset ultrafast -qp 0" which speeds up my emulation to 17Mhz, but the file size explodes from 3Mb to 30Mb! So I guess there is still room to play and these ffmpeg string should be externalized to allow tuning.

Also noticed a "-tune animation" that could be tested to see what it yields.

I have a dual boot with Ubuntu on my current machine, I guess I will try to build and test it to see if the slowdown culprit is really my machine or piping under Windows.
I guess it depends on the game, and how many pixels change per frame.

If that version of Ubuntu has avconv installed, check to see if ffmpeg is a wrapper script. If it is, you'll get terrible performance. The way to fix that is to make a symlink to avconv called ffmpeg somewhere in your path so it finds that symlink first.

Edit: The fastest encoding that I've gotten so far was using this:

Code: Select all

"ffmpeg -y -f rawvideo -s 640x480 -pix_fmt bgra -r 60 -i - -vf scale=-1:720 -sws_flags neighbor -an -preset ultrafast -qp 0 -tune animation uzemtemp.mp4"
Last edited by Artcfox on Sun Sep 27, 2015 12:05 am, edited 1 time in total.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Record a 60fps movie (with sound) using Uzem

Post by Artcfox »

Okay, so I just recorded a 6 minute video, and discovered a bug in my code that causes the audio and video to go way out of sync. :oops:

The problem is that the sample rate is not actually 15734 Hz. It is technically 28636360 Hz / 1820, which is approximately 15734.263736264 Hz.

If we don't account for that fractional part during each sample (ffmpeg only accepts an integer sample rate) then the audio and video will quickly get out of sync.

This should fix it for any length recording:

Code: Select all

diff --git a/tools/uzem/avr8.cpp b/tools/uzem/avr8.cpp
index 966486d..e912e8a 100644
--- a/tools/uzem/avr8.cpp
+++ b/tools/uzem/avr8.cpp
@@ -255,7 +255,17 @@ void avr8::write_io(u8 addr,u8 value)
                        SDL_UnlockAudio();
 
                        //Send audio byte to ffmpeg
-                       if(recordMovie && avconv_audio) fwrite(&value, 1, 1, avconv_audio);
+                       if(recordMovie && avconv_audio) {
+                          fwrite(&value, 1, 1, avconv_audio);
+
+                          // Keep audio in sync, since the sample rate is not a factor of the clock speed
+                          static u32 clock_skew = 0;
+                          clock_skew += (28636360 % 15734);
+                          if (clock_skew >= (28636360 % 15734) * 1820) {
+                            clock_skew -= (28636360 % 15734) * 1820;
+                            fwrite(&value, 1, 1, avconv_audio);
+                          }
+                        }
 
                }
        }
Though it seems to have gotten out of sync again for a 25 minute (live) recording. I'm not sure if it is related to the uzem140 branch dropping scanlines, or if I overlooked something.

Edit: The other odd thing is I can't even use the capture file that I created using uzem140 with the version of uzem from the master branch.
User avatar
D3thAdd3r
Posts: 3222
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Record a 60fps movie (with sound) using Uzem

Post by D3thAdd3r »

Artcfox wrote:Though it seems to have gotten out of sync again for a 25 minute (live) recording.
Still not too bad, I've never seen any Uzebox recording last more than maybe 4 minutes. That sounds a hard problem, but if it is already probably not going to run full speed while recording and the capture playback method is to be used, could you resort to some more expensive option?

Capture 1534 samples, then interpolate them out with a rolling error(based on the 1534.2637) up to a higher sample rate. Sounds really complicated :?
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Record a 60fps movie (with sound) using Uzem

Post by Artcfox »

D3thAdd3r wrote:
Artcfox wrote:Though it seems to have gotten out of sync again for a 25 minute (live) recording.
Still not too bad, I've never seen any Uzebox recording last more than maybe 4 minutes. That sounds a hard problem, but if it is already probably not going to run full speed while recording and the capture playback method is to be used, could you resort to some more expensive option?

Capture 1534 samples, then interpolate them out with a rolling error(based on the 1534.2637) up to a higher sample rate. Sounds really complicated :?
With the new encoding options, it actually records at full speed while I'm playing my game live on my slow laptop. I keep track of the error, and when it's a full sample off, I add an extra sample. The uzem140 branch has an unrelated issue where it sometimes drops the first scanline, and I think that might be what's causing it to get desynchronized, since after that compensation it's only a tiny bit off.

I'm letting ffmpeg do all the hard resampling work.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Record a 60fps movie (with sound) using Uzem

Post by Artcfox »

I'm trying to figure out why even with that fix to account for the fractional part of the 15734 Hz sample rate, the audio still gets out of sync with the video for long running videos.

It just dawned on me that the video is not exactly 60 FPS, it's 59.94 FPS.

For the video:

Code: Select all

90000 frames @ 60.00 fps  = 1500.00 seconds
90000 frames @ 59.94 fps ~= 1501.50 seconds
Which makes sense, because consuming the same number of frames at
a higher frame rate should be faster, so by lying and saying the
video frames are 60 fps, the video should be played back a tiny
bit faster than it should.

For the audio:

Code: Select all

23601000 samples @ 15734 Hz              = 1500.00 seconds
23601000 samples @ (28636360 Hz / 1820) ~= 1499.97 seconds
Which also makes sense, because consuming the same number of
samples at a slower sample rate should take longer, so by lying
and saying the sample rate is 15734 Hz, the audio should be
played back slower than it should.

Taking both of those together, the video playing back faster than
it should, and the audio playing back slower, I would expect the
video to end up leading the audio, but when I merge the video and
audio streams together at the end, it ends up that the audio
leads the video!

What could I be doing wrong?
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Record a 60fps movie (with sound) using Uzem

Post by Artcfox »

I now have it so the audio and video are just a tiny bit off from each other 25 minutes into a recording. :)

I think I can live with that, so I submitted a pull request.

The diff is also below, in case anyone else is curious how I'm accounting for the drift. I left the raw calculations in there, since they'll get compiled into a single constant and they explain where those numbers are being derived from, rather than just having the magic numbers: 480 and 230400, which I found a bit hard to name.

Code: Select all

diff --git a/tools/uzem/avr8.cpp b/tools/uzem/avr8.cpp
index b1e5f62..054916a 100644
--- a/tools/uzem/avr8.cpp
+++ b/tools/uzem/avr8.cpp
@@ -254,8 +254,17 @@ void avr8::write_io(u8 addr,u8 value)
                        SDL_UnlockAudio();
 
                        //Send audio byte to ffmpeg
-                       if(recordMovie && avconv_audio) fwrite(&value, 1, 1, avconv_audio);
-
+                       if(recordMovie && avconv_audio) {
+                               fwrite(&value, 1, 1, avconv_audio);
+
+                               // Keep audio in sync, since the sample rate we encode at is not a factor of the clock speed
+                               static u32 accumulated_error = 0;
+                               accumulated_error += (28636360 % 15734);
+                               if (accumulated_error >= ((28636360 % 15734) * (28636360 % 15734))) {
+                                       accumulated_error -= ((28636360 % 15734) * (28636360 % 15734));
+                                       fwrite(&value, 1, 1, avconv_audio);
+                               }
+                       }
                }
        }
        else if (addr == ports::PORTD)
@@ -1722,7 +1731,7 @@ bool avr8::init_gui()
 
        if (recordMovie){
 
-               if (avconv_video == NULL) avconv_video = popen("ffmpeg -y -f rawvideo -s 640x240 -pix_fmt rgba -r 60 -i - -vf scale=960:720 -sws_flags neighbo
+               if (avconv_video == NULL) avconv_video = popen("ffmpeg -y -f rawvideo -s 640x240 -pix_fmt rgba -r 59.94 -i - -vf scale=960:720 -sws_flags neig
                if (avconv_video == NULL){
                        fprintf(stderr, "Unable to init ffmpeg.\n");
                        return false;
Post Reply