Streaming Music(SD, SPI ram, Network, etc. source)

Topics related to the API, programming discussions & questions, coding tips, bugs, etc. should go here.
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

Alright I am glad I posted code to the forums, as I must have had some broken experiment in the actual demo. Working off the latest I posted, all the new changes in the master should also be in the streaming-music branch. It compiles and the demo works, and also shows how to call mconvert from the makefile now. I still need to update the tutorials, but hopefully at this point there is no mystery and bad information about how to actually use this from the code examples. Please let me know if anyone has issues or suggestions for restructuring.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by nicksen782 »

I've been having some trouble if I change the size of the buffer in the Makefile. It works solid with values above 256. I only need a 16 byte buffer.

I see that value is used to create the songBuf array but nowhere else. I'm not seeing why this may be a problem.

(Using latest streaming music branch BTW.)
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

Yikes, indeed it looks to keep reading until it crashes. Cutting off all data to it, it crashes if StartSong is called so it seems the issue is being triggered by the player itself. It is not overfilling and I see the couple values other than 256 I tested tell the song player that the buffer is always full when it is actually empty. Let me check this out, thanks for this find as it appears a problem at the core of the whole thing. I guess I have only been testing with the full 256 byte buffer and using the simulated buffer limiter, but I am guessing something accidentally relies on a byte rolling over as the counters are all 8 bit.
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

Alright pretty sure I figured out the issue here. The function SongBufBytes() did not calculate the size correctly. With the 256 byte buffer that all demos have thus far had, the overflow "made it right". I thought I could do it without the extra modulo, but it appears there is no more efficient method than this:

Code: Select all

#elif MUSIC_ENGINE == STREAM

u8 SongBufBytes(){
	if(songBufIn > songBufOut)
		return (songBufIn-songBufOut);
	else
		return ((sizeof(songBuf)+songBufIn)-songBufOut)%sizeof(songBuf);
}

u8 SongBufFull(){
	return(songBufOut == ((songBufIn+1)%sizeof(songBuf)));
}


void SongBufWrite(u8 t){//writes a byte into the circular buffer

	songBuf[songBufIn] = t;
	songBufIn = ((songBufIn+1)%sizeof(songBuf));
}


u8 SongBufRead(){
	
	u8 t = songBuf[songBufOut];
	songBufOut = ((songBufOut+1)%sizeof(songBuf));
	songPos++;
	return t;
}

#endif
@nicksen782 if you would, for now just replace the version of SongBufBytes() in your uzeboxSoundEngine.c. Also I believe somehow this did not make it into the version I put up, in defines.h

Code: Select all

	#ifndef MUSIC_ENGINE//search this
		#define MUSIC_ENGINE MIDI
	#endif

	#if MUSIC_ENGINE == STREAM//insert these lines
		#ifndef SONG_BUFFER_MIN
			#define SONG_BUFFER_MIN	2
		#endif
	#endif
I will make it right on github, but I mean to wait to push it until I have the tool to generate the data. The demo currently(not up yet) has the visualizer back as I can no longer reproduce that mysterious issue Artcfox and myself saw with the random static. It also shows simultaneous PCM streaming while the player is still running just fine, but I need to finish the tool for that as I manually added it. Feels kind of bad to put it out when it is a MAGICDATA.DAT, over a data file that can be regenerated from scratch. Once I have that done I will put all the new stuff up, let me know if you run into more issues and thank you for the feedback it helped a lot for something I somehow never tested :?
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

I am seeing some limitations on my "PCM Instrument Kit" idea, where it really takes a lot of cycles to chug streaming music and also 3 separate PCM samples. Now, with reduced bit rate and doubling the data, or perhaps some sort of compression it could work. Also I am not directly using the bulk transfer functions in the SPI Ram library, and instead am doing a separate function call for every single byte. I have to do this currently, as I need to modify every single sample to reduce it's volume so that the clipping is not too bad when you get all cumulative with the waveforms. This could be fixed perhaps with quieter samples, or some asm version that can do simple transforms inside a bulk transfer.

Unfortunately, it seems like the case of actual music playback will be worse as it is not a linear playback of the samples, but instead could jump and skip bytes for different pitches(cycling through the same data faster or slower, bytes which you would have to read anyway, and discard) which would be even more reading. So I don't know...right now with 3 samples(which as you can hear do not play back at full speed simultaneously) that is 262*3=786 function calls instead of 3(with some assembly version that can handle some of this). It would take some serious development, but even at this 1 streaming PCM with simultaneous streaming music at max quality seems to work well for a lot of games, perhaps 2 if the game does not need much cycles(a dramatic element in an RPG for instance). I would guess 3 would work if the game didn't use much cycles and the optimized version I mentioned were realized.

Anyway push B for a railgun, push A for "hadouken!", and press X for an obscure GLaDOS sample. The cool part is that this is 100K of PCM which could just as easily be 10MB to load in and out of memory as specific situations in the game call for. I am really thinking an RPG could use this well if designed properly. This is a defintely weird and random demo, enjoy :lol:
Attachments
StreamingMusicAndPCM.zip
(85.25 KiB) Downloaded 436 times
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by nicksen782 »

Oh wow! Music AND PCM sound effects at the same time! In fact, if I rapidly played the PCM sounds they would be added to the... well, I'm just going to call it the "soundscape." They all played at once. I did notice that the buffered value (on screen) would fluctuate more when I rapidly played the PCM sounds.

How do you create the PCM sounds? I still need to figure out what to do about sound effects with my game.

How was I able to play so many PCM sounds at once? Where those bytes just stuck into the buffer when I pressed the button?
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

Right now I use audacity and open an existing .wav, then set the project rate at the bottom left to 16000hz(the closest to correct it will allow) then do a resample to 16khz. Then export as 8bit signed raw, at which point the data can directly be pasted with a hex editor to a location. A tool will do this from a config file, similar..or built into mconvert. The demo just points one of the three channel offsets at this location, and sets the length counter to a value matching how many samples it has (just the file size of the .raw).

The limit is purely cycles, as each additional Simultaneous PCM channel is just 4 bytes of ram, so 12 bytes more than previous demo. And of course you can play 100 sounds on a single channel, so long as it 1 at a time. Music slows down when 3 are playing but not with 2. It is much more obvious the deficit in waveform of sfx than the timing of the music(music wave forms have no holes, just delays before next note event/timing). These samples just mix in on top of the same 262 byte buffer, which the kernel has already written with music, so ram is no bottleneck. Potentially I might think on compression to habe a full prebaked PCM song as a single long sample, since 4 simultaneous samples like 1 per channel is out of the question now.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by nicksen782 »

Hmmm... I'm thinking that if PCM sounds were very short and music used fewer channels (the tracks that I have usually have 2 but occasionally 3) then you could have both music and sound effects with the streaming music. In fact, perhaps certain PCM sounds could be loaded in and out of SPIRAM depending on what sounds would be used for a level or something. Think of this switching like how a page file works.

Assuming that all raw PCM audio would be at the same bitrate would that mean that what determines the PCM byte size is how long the effect is? Meaning, what is the "cost" of 0.25 seconds, 0.50, 0.75, 1.00 second sound effects? I don't know how large these sound files are.

I have a number of sound effects that I would like to add.

This is awesome! I would like to help however I can. When is the next update going to be so that I can jump in on it?
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

With clever use I agree this can be used well for some things. Full blown PCM almost sounds out of place against the music just for the quality. Not that PCM hasn't been around for years now, but it is has been barely possible to use in flash only, except for extremely short and possibly low quality (lower frequency data, sped up to sound mostly normal with less samples like Columns did). Not sure yet on possibilities for combinational SD streaming with something like UzeAmp does.

Right now there is no compression or frequency shifts, so size can be directly calculated by length seconds*15,700 bytes. That is if Audacity allowed 15.7khz, so if going by time in Audacity, then 16,000 bytes per second...Uzebox the previous calculation Another program will probably allow a custom 15.7 to be exactly the same, for now the difference in sound pitch is negligible for non-music.

Railgun is something like 30K because it drags on with subtle sound for longer than it seems, 2 seconds Hadouken I think about 20K 1.5 seconds, and "Dancing is NOT science!) I think was about 45K at 3 seconds. Not able to check ATM. With 20K of music for the 7 songs, and the existing PCM, there is probably room for several very short effects without loading more data.

I am back interested in this so hopefully soon this will be built proper, with directory and all that for PCM, as well as some small improvements for mconvert like a standard place to write the length of entire file. I am paralleling a bit on this subject as I try and see if lossless or almost lossless compression can help the issue by not having to read as much (and fitting more at once).
User avatar
Jubatian
Posts: 1561
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by Jubatian »

Sounds interesting. Reminds me a bit to doing digital stuff on the SID (C64).

By the way you can get arbitrary frequency if you are willing to double the storage for a sample. The key is mip-mapping, at least the same technology adapted to audio. If you want to play something at lower frequency than it's native frequency, no problem, you just have to repeat samples. For higher frequencies, you create lower resolution versions of the samples (half size, quarter size etc.) so you can always play the sample within the SPI RAM at equal or lower frequency.

For compression I did some actual tests compressing actual music observing results (I intended that for RRPGE, but would work equally well for Uzebox + SPI RAM too).

If you break up the sample to 16 byte chunks, normalize each chunk (adding 1 byte for volume), then reduce the normalized chunks to 4 bits / sample, there is hardly any audible loss, and you almost halved it. It is a bit noisy but still sounds acceptable at 2 bits / sample too. Otherwise I also experimented with reducing the frequency of individual chunks to compress them further (but that's a variable size compression then).
Post Reply