Uzenet

Topics regarding the Uzebox hardware/AVCore/BaseBoard (i.e: PCB, resistors, connectors, part list, schematics, hardware issues, etc.) should go here.
Post Reply
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Uzenet

Post by D3thAdd3r »

Let's make this happen! This thread to continue the conversation and have a separate place to talk about Uzebox networking hardware as a whole system. This is apart from individual low level device details for a specific adapter(CC3000,WIZnet,ESP8266,etc) that might accomplish the same protocol in a very different manner. Basically the way the Uzebox will accomplish it's part hardware wise, but mixed in with add-on device details since you can't totally separate the two. A quick recap of some important items:
uze6666 wrote: .....This field is moving so fast, that it would be unwise to directly implement support for any module. As you noted, some interfacing MCU will be required for that to implement a standard protocol.
.....The dongle thing is great since it's universal, but using the SPI on the EXT port on my PCBs would seem simpler and more efficient and can be neatly hidden in the enclosure.....
.....best is probably to implement that protocol in the kernel to support both the Player2 SNES port and the EXT header.....
.....bit-bang SPI approach to simplify code.....
.....Since there's only seem to be serial based module out tthere (no SPI :( ) we would need a fronting MCU which exposes a standard protocol as you mentioned.....

Reading some forum posts, I think Uzenet or UzeNet seems to be the clear favorite for the official name. Any suggestions for another name or is that de facto standard by now? While we are at it, I quite like the "Uzebus" term mentioned in the other thread to indicate the actual way the Uzebox is linked and communicates with the add-on device.
uze6666 wrote:Btw, do you see need for more than one socket client and one socket listener at any given time?
I think the answer to that entirely depends on the much larger question what type of networking system is going to work best for a small number of players distributed pretty far apart.

There are 2 basic options for anything like this:
Client->Server
Client->Client(peer to peer or anything else you can call it)

Client->Server I don't see a need for more than one connection and that might have an advantage in dealing with only one other device over the internet. With more connections, you will be getting many more separate packets at random times since you are at the mercy of latency to/from multiple separate paths. We might struggle just to keep 1 connection working while juggling communication with the Uzebox. I am pretty newbie at all this hardware interfacing stuff, so I don't have much of an idea without just experimenting to be honest.

Client->Client I could see using up to 3 connections because it would be nice to play 4 player games. If you keep it to 2 players 1 connection would work perfect, but come on...Harty brings it to an expo and explains that he is playing zombienator over the internet with 3 members of the forums, man :ugeek: !

I think some very important things to discuss are client->client vs. client->server and also LockStep vs Asynchronous/Extrapolated. I will leave the UDP(!)vs TCP discussion for later as it's getting late.

I don't know which will work out better. C->S I think will be easier to handle for our add-on devices, but the disadvantage is, as example: I have access to multiple fast fiber locations(where this traffic and would go unnoticed by me ;) ) So if I have a server running here in the northern US I'm loving it at 8ms latency at home, I am playing with Alec he is close and enjoying a latency of <100ms, Harty jumps in from Germany hitting 180ms maybe grumbling a bit but hey it's fun, and CunningFellow wants to play from Austraillia with 300ms+. I can be sure that Cunning will not enjoy the experience as much as me. If we have a server in Europe everyone will have more even pings of course.

C->C the device will need to send and receive more packets since we are connected to perhaps 3 other add-on devices. The actual internet bandwidth required is so small either way that it doesn't matter for modern Fiber/Cable/DSL connections, but I don't know how hard it will be to sort all that out on the device. It would have the advantage that we would all have approximately the same latency and Cunnings latency would be much better. It at least eliminates the round trip delay, that is I send data to a server from Japan but I can't assume I can advance the game tick until the server sends data across that same path and let's me know he got it and what everyone else sent this tick. Now whenever Cunning gets everyone's pad states he just moves ahead without having to wait for a middle man and everyone moves ahead while packets are still "on the wire" for the next tick or two(since time from input to actually running the tick would always be behind no matter what we did). Kind of pulling that out of thin air because there are basically infinite ways you could do any of that.

I always figured C->C would be better, but centrally located C->S might be much simpler and same performance :idea:
User avatar
uze6666
Site Admin
Posts: 4778
Joined: Tue Aug 12, 2008 9:13 pm
Location: Montreal, Canada
Contact:

Re: Uzenet

Post by uze6666 »

Reading some forum posts, I think Uzenet or UzeNet seems to be the clear favorite for the official name. Any suggestions for another name or is that de facto standard by now? While we are at it, I quite like the "Uzebus" term mentioned in the other thread to indicate the actual way the Uzebox is linked and communicates with the add-on device.
Uzenet is a good name for the Uzebox<->Internet interface as a whole. A Uzenet module would be the thing made of and MCU and wifi module. It could connect to a Uzenet server. Uzebus (or Uzenet Bus?) describes the physical interface between the Uzebox and Uzenet module. Yeah, all good for me.
Client->Server I don't see a need for more than one connection and that might have an advantage in dealing with only one other device over the internet. With more connections, you will be getting many more separate packets at random times since you are at the mercy of latency to/from multiple separate paths. We might struggle just to keep 1 connection working while juggling communication with the Uzebox. I am pretty newbie at all this hardware interfacing stuff, so I don't have much of an idea without just experimenting to be honest.
Agreed.
Client->Client I could see using up to 3 connections because it would be nice to play 4 player games. If you keep it to 2 players 1 connection would work perfect, but come on...Harty brings it to an expo and explains that he is playing zombienator over the internet with 3 members of the forums, man :ugeek: !
It sure sound cool, but realisticly I think it would be hard to have so many connections at once, yet alone finding so many other peoples willing to play the same thing in such a small community. I think it would be wiser to aim for simplicity with supporting only one client and one listeners socket at any time. Multiplayer traffic, if it gets there should go through a central server and router to the single listener socket. You'll need it anyway to discover other players.
Now whenever Cunning gets everyone's pad states he just moves ahead without having to wait for a middle man and everyone moves ahead while packets are still "on the wire" for the next tick or two(since time from input to actually running the tick would always be behind no matter what we did). Kind of pulling that out of thin air because there are basically infinite ways you could do any of that.
I'm not so sure about the concept of just sending pad states, specially via UDP since you can loose packets unnoticed. This could desynchronize quickly and cause erratic and non-sensical movements of the player movement on the other console. But what do I know, I never attempted game networking code before! Will do some reading.

We should really keep a master of all this stuff in a wiki page to act a the official spec. We can use the forums to keep exchanging of course. I have created a page: http://uzebox.org/wiki/index.php?title=Uzenet.
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzenet

Post by D3thAdd3r »

Ok, long post ahead. I am just going to dump all my current opinions and everything I know or think I know at once so it's at least said and it's out there. I figure this is the time for anyone else with something to say to say to do the same before details start getting carved in stone. A lot of things rely on the way other things work here.
uze6666 wrote:It sure sound cool, but realisticly I think it would be hard to have so many connections at once, yet alone finding so many other peoples willing to play the same thing in such a small community. I think it would be wiser to aim for simplicity with supporting only one client and one listeners socket at any time.
Yeah, this is a large task overall, my thoughts are drastically moving towards doing whatever is the easiest and still accomplishes all goals. At that point with a working product, you could always see which parts are good enough and upgrade what isn't. Do you think prototyping the protocol in Uzem might be worth thinking about?
uze6666 wrote:You'll need it anyway to discover other players.
The way most games do it is to create a master server. All active games are reported to a known address with how many players are on, which rom being played, etc and kept track of. Occasionally these servers give a heartbeat back or "hey I'm still alive just so you know" message so the master server keeps them in the list. If the master server doesn't hear it for a while, maybe the server crashed, lost internet, remove it from the list so people aren't trying to play on something that isn't there anymore. Do a TCP connection to uzebox.org then "GET masterlist.txt" and extract the updated master IP or any other way to get it would work.
uze6666 wrote:I'm not so sure about the concept of just sending pad states, specially via UDP since you can loose packets unnoticed. This could desynchronize quickly and cause erratic and non-sensical movements of the player movement on the other console.
We should use TCP for everything to simplify the process of getting this off the ground. TCP is great because it's like C, easy to use and you can coordinate high level stuff with good enough performance for most things. UDP is like assembly, basically a thin layer of paint over the low level IP packet; which is the actual way all the higher level protocols work. Writing a whole game in assembly would be difficult to organize, but writing the mode 3 scanline routine in C would probably yield a less desirable result too. TCP is a finite state machine that is optimized for general data streams where %100 throughput is most important but fluctuations in delay for availability for any particular small piece of data doesn't matter(buffer it ahead for consistency). It is still dealing with all that loss, duplicate(rare), and out of order stuff behind the scenes in a generic way. All that sequencing, acknowledging, and asking for retransmissions for lost data sucks at short time intervals like 1/60 second. TCP will not give you any data until it has everything in order up to that point so lost packets stall the game out every time equal at least to the the round trip delay it takes to ask and receive it(it DOES keep taking new packets in the mean time so you have them later). In a file no byte is more important than another, but in a reflex game the newest data is the most important. So what I said made no sense because I forgot to mention how to leverage UDP when the data is so small. Classic and easy is a Tick ID at the start of each packet so we actually know exactly which data we have and have not(custom solution to our specific goal) To have %100 game replication on both ends and much better lost packet tolerance/timine: for each frame or so you send a new padstate to the module(offloading all further work away from the Uzebox) concatenates also the last maybe 9 states to that. That is still an extremely small packet.

Basically imagine someone sends 10 packets, each one holding the very latest(at the time of being sent) state with the previous 9. Packets sent out with id #0,1,2,3,4,5,6,7,8,9 and on your end you receive 0,1,3,2,5,6,9. You lost 4,7,8,received 2 out of order(by the time you get it, it's worthless since you already got 3 and moved on), but if you think about it you had the information in those lost packets anyways as soon as you get a new one. So the hiccup in your data stream is only equal to the time it takes to get a new packet, which on his end was sent only 1/60second after the one you lost and you don't have to ask and receive anything new which would take much longer. IF for some reason you get major packet loss for a time(more than 10 packets), then you still must have some mechanism to indicate "hey I only know what has happened up to TICK ID, send me everything since then" then the adapter hides those details from the Uzebox. That relies on the Uzebox knowing that whenever the module does not give you information on that tick, then you can't move on until you have it. You also know the other guy is doing that. So even if he didn't experience the packet loss and got all your padstates up to the last one you sent, he wont get any more new states because you will not send them until your situation is rectified(ie. he tells you what he knows). Does that make any kind of sense? I'm trying to think of a quick picture to show it better but I am at a loss how to symbolize that. This is the simplest way to do it that would have any kind of performance. Basically any game will have some form similar to this:

Code: Select all

	InitGame();//start up the module too
	while(true){
		ReadGamePad();
		//process normal game logic...
		WaitVsync(1)
	}
It is very easy to change that general structure to:

Code: Select all

	InitGame();//start up the module too
	while(true){
		ReadGamePad();
		if(NewNetworkTickAvailable())
			otherpad = NewNetworkData();
		else{
			WaitVsync(1);
			continue;
		}
		//process normal game logic
		WaitVsync(1)
	}
Pad states are absolutely the laggiest way to do things, you feel all the latency all the time. I only like padstates because we never have to guess what happened or build a large list of information about all the objects that have changed or are currently visible(however the system was designed). The adapter could not help there, the Uzebox would have to build the data set and any predictions that give the appearance of no lag because an adapter is unaware of how the game actually works unless it can replicate some parts by itself. Making that system individually for new games(it must be fairly custom to each games specifics obviously) is a problem and being that difficult it's nearly assured our existing library has little prayer of anyone going back and making all that. It is really complicated and takes lots of ram and cycles in general, and most games are pushing the hardware pretty tight as is. The idea works great where possible: a good example is some of the new fighting games using GGPO or similar that hides lag like this: Run the tick even if we don't have new data yet. Make an entire copy of all relevant game variables for each probable outcome, and then show whatever the prediction thinks will actually happen on screen as if it really did just happen. Then for EACH of those many new game ticks prediction, make a totally new copy of variables for those possible outcomes, and keep repeating that process until we finally receive information on what really happened from the authority(server). If the predictions are totally right(for the amount of game ticks that it takes to equal your latency)and the gameworld you were shown happens to be what really happened after all: You have essentially defeated latency since it entirely appeared that your moves were confirmed instantly! If your opponent didn't predict correctly though, why should he concede that you got to move "faster than light" and accept that you hit him with a punch he never got to see. In actuality It works pretty awesome! Prediction makes things seem smoother but the problem is when the prediction is not correct, you can only let it go so far until you are showing a client something that is nothing like reality at which point any further input he makes is based on such a wrong replication that he might as well put the controller down. In that case you are forced to correct it. You can "half correct" it so the transition is much more smooth by taking what you know is true showing a world blended with the failed prediction. That is ULTRA complex and a very fine balance, since an inaccurate view isn't worth any more than an outdated view. So smoothness vs accuracy.

Mostly related, a long time ago I made a bomberman clone using DirectX and was very eager to make it work over the internet. I spent quite a bit of time and frustration learning how to make sockets do what I wanted in winsock, and I had just read "Learn C++ in 24 days" probably the year before, so you can imagine how I struggled. But not counting all that time learning the standard socket basics, I managed to make the simplest possible 2player,client->client, TCP, No prediction, Lockstep network setup pretty quickly. When I say it worked, what I mean is it never lost sync(TCP+Lockstep keyboard states makes that automatic). I found that actually playing it over the internet was fairly bad when I had to push a button to turn a corner before I got there or else miss it, with stalls and fluctuating timing. Being an avid online FPS player back then and still fascinated with the concept, I wasted a lot of time reading the source code to every FPS game I could find especially influenced by John Carmack's doom->quake world->quake 2/3 network code evolution. Later I worked pretty heavily for 2 years on what I consider an impressive indie shooter game, but it reached a point where it was too big and far beyond my skill level and ability to generate the needed graphics,levels,etc. Long story short, I did manage to make: master server to keep track of active game servers(which were all in my bedroom), in-game server browser, and Client/Server UDP networking with client prediction and mechanisms for reliable messages. It played way better than my first attempt, but it was orders of magnitude more complex/difficult.

Ok I'm done for now :roll:
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzenet

Post by D3thAdd3r »

Also found a good overview on the general subject here
User avatar
uze6666
Site Admin
Posts: 4778
Joined: Tue Aug 12, 2008 9:13 pm
Location: Montreal, Canada
Contact:

Re: Uzenet

Post by uze6666 »

Wow, that was a big post. :)

Thanks for the explanations and refresher, things comes back to my memory now. I did work on some chat client/server back then using raw sockets. I see UDP used most for real time games for obvious speed reasons. TCP would be great for all the rest. The redundancy trick in the UDP packets is brilliant, never tough of this. This should really work well.
Do you think prototyping the protocol in Uzem might be worth thinking about?

Eesshhh...uzem code is messy and it's getting harder to add stuff. If somebody want to take the ball, feel free to join in. Otherwise, I'd wait till the interface and protocols are pretty much done and tested.

In most tutorials on the subject, it seems the server is used to initiate connections but also to maintain the game's logic. For sure we'd need the server to initiate connections and punch trough the NAT on our routers. But not any game logic though!
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzenet

Post by D3thAdd3r »

uze6666 wrote:Eesshhh...uzem code is messy and it's getting harder to add stuff. If somebody want to take the ball, feel free to join in. Otherwise, I'd wait till the interface and protocols are pretty much done and tested.
I was just testing the water because Uzem source is pretty scary to me so I wasn't going to volunteer :lol:I think it is almost easier to have a working server/master server on the PC side to debug the actual setup with once we have communication.

I didn't get as much time free this weekend as I had hoped but I did start working on a mock server browser for Uzebox. More importantly I did relearn socket details and manage to make a (TCP and stays TCP)master server that will soon be good enough for done. It has fast non-blocking io for "infinite" simultaneous connections and a simple challenge protocol on accepting a connection so that should be enough to thwart (at least the lazy) wireshark sniffing hackers filling the list with garbage. Keeps track of gamename,rom,players,etc. Just a couple bugs to work out and finish the "keep alive" from servers(server doesn't exist yet so...), but it will be easy for the adapter to interface with because it can specify which entry in the game list to start on and how many entries to send total. That way if there were 30 active games(dream big..), the uzebox can just request, ie. games 8...14 at a time to scroll through all games while managing ram easier. As a note I would always leave my uzebox on auto-refreshing the game list(should make an audio cue on new game) and drop programming to join any lonely player for a bit. So you could always get a game of anything going heh :D

I'm going to make a testing TCP server(UDP later) even though we might end up changing some specifications. The server will pretty much be a copy of the master server guts, with just with different code in the state machine to make it work. The extra complications are that I am very keen on making the PC server do absolutely as much work as possible to keep things running smooth and spare our MCU's cycles. Thinks like building aggregate pad states and keeping track of padstate history(we could record network games and play them back, did I already mention that!?), resending, etc all handled on the PC.
uze6666 wrote:For sure we'd need the server to initiate connections and punch trough the NAT on our routers.
A lot of that stuff is pretty grey area for me, but it should definitely be as plug and play as possible when it's done. When the time comes we can rack our brains on how that will work, but at the very least we can definitely use port forwarding or temporarily turn all firewall off for testing I guess.
uze6666 wrote:But not any game logic though!
Yeah I mean it honestly really would be way better but I think it's just too much separate work for people to do and having a single player and muliplayer version of logic on Uzebox might be impossible due to flash no matter how determine the programmer was. I figure the server should be designed in such a way that it could run any number of separate game logic instances if we wanted to, but default to padstates which any game could implement. I think there are some game where you could design it on Uzebox as only a dumb rendering terminal just drawing and playing sounds according to what the server says. When we get this working I will definitely want to port a text dungeon crawler on Uzebox!
User avatar
jhhoward
Posts: 76
Joined: Fri Feb 07, 2014 8:11 pm

Re: Uzenet

Post by jhhoward »

I love the idea of networking these little boxes together on a Uzenet :)

I have to admit I only skimmed through the more detailed parts of your posts (they were long posts!) but I thought I'd share my thoughts:
- An online leaderboard would be a good place to start. It would be a simple proof of concept for the networking protocols, and a basic case of sending messages between client and server.
- How exactly would the TCP/IP stack work? Is it implemented on a chip on the network module?
- In the uzebox side, how does the game interact with the network module?
- How much RAM overhead will be required for networking?

I have experience developing online multiplayer games, so just to share my thoughts:
- As mentioned UDP is better for low latency but packets can be lost. Ideally we would want the option for a reliable channel (either TCP or a custom protocol written on top of UDP) and non-reliable (pretty much plain UDP)
- NAT punchthrough is a pain to get working properly, and in my experience there are plenty of scenarios where it just doesn't work.
- You mentioned the idea of having a master server for matchmaking. I think this is a good idea, and we could easily have a generic lobby system that all games could interface with.
- Since we aren't expecting large volumes of traffic, we could just have the server relay packets between players too. This isn't quite as efficient as direct connections, but would be more reliable and easy to set up. This would avoid any NAT problems.
- If we make a simple data storage system for the server, it could store high score tables or other data. This could even be used to store a save state for a turn based multiplayer game for example. (e.g. a chess game where each player logs in once a day to make their move)
- For real time multiplayer, I don't think the idea of just replicating pad state is going to work very well. It's better to transmit player position / state as frequently as possible, and have the other box do interpolation / extrapolation. Unfortunately this does put a lot of extra complexity in the game logic.
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzenet

Post by D3thAdd3r »

jhhoward wrote: - How exactly would the TCP/IP stack work? Is it implemented on a chip on the network module?
All the modules Ive looked at seemed to have an on chip network stack. Send it serial commands and it does what it does: very high level compared to socket level control
jhhoward wrote: - In the uzebox side, how does the game interact with the network module?
Check out the esp8266 thread for all the theoretical details so far. Alec last post there seems to be the winning plan there.
jhhoward wrote: - How much RAM overhead will be required for networking?
The uzebox should only need a couple bytes to coordinate comms with the device with emulated SPI, then for pad states probably only ~16 to buffer some padstates when catching up. My guess.
jhhoward wrote: -Ideally we would want the option for a reliable channel
A lot of ways to do it, I think things were veering towards only 1 connection so even UDP packets received starting with "RL" the device could know thats reliable data that we must send a separate "AK" back. If server doesnt hear an ack for a while resend, tell client to hold until it AKs or something if necessary. What do you think?
User avatar
uze6666
Site Admin
Posts: 4778
Joined: Tue Aug 12, 2008 9:13 pm
Location: Montreal, Canada
Contact:

Re: Uzenet

Post by uze6666 »

- An online leaderboard would be a good place to start. It would be a simple proof of concept for the networking protocols, and a basic case of sending messages between client and server.
Agreed!
- In the uzebox side, how does the game interact with the network module?
Since we don;t want to couple the kernel code to a specific network module that can go obsolete next year, the Uzebox will go through an interface MCU. That MCU will receive messages from the Uzebox via SPI using some protocol we need to define. The MCU will then interface with the network module(s). That MCU code base should be designed to support multiple modules, wifi and perhaps even wired.
- How much RAM overhead will be required for networking?
We will try to delegate as much RAM overhead to the helper MCU as possible. Depending on the device we have between 512 bytes and 2K of RAM. Right now, I really have no idea how much we'll need.
I have experience developing online multiplayer games
That great news, bring all you know! :)
- NAT punchthrough is a pain to get working properly, and in my experience there are plenty of scenarios where it just doesn't work.
- You mentioned the idea of having a master server for matchmaking. I think this is a good idea, and we could easily have a generic lobby system that all games could interface with.
- Since we aren't expecting large volumes of traffic, we could just have the server relay packets between players too. This isn't quite as efficient as direct connections, but would be more reliable and easy to set up. This would avoid any NAT problems.
- If we make a simple data storage system for the server, it could store high score tables or other data. This could even be used to store a save state for a turn based multiplayer game for example. (e.g. a chess game where each player logs in once a day to make their move)
These 4 points go well together. There was discussions about NAT punchthrough a while ago, but since I have read somewhere the issues you mentioned. If you had bad experiences with those it's better for us to avoid the punchthrough concept altogether IHMO. In that case, a master server before essential. I like the idea of the lobby system you mention with leader board storage functionality. Although I didn't give it a lot of thought so far, I'd really like this server to be generic for all games. It's already a challenge for mot to code a game on the Uzebox itself, it would be a hard to sell to add network/server code to that.
User avatar
uze6666
Site Admin
Posts: 4778
Joined: Tue Aug 12, 2008 9:13 pm
Location: Montreal, Canada
Contact:

Re: Uzenet

Post by uze6666 »

Just thinking about the wifi config. I was playing on my Nintendo DS the other day and noticed the wifi configuration is not part of the games but seems to be delegated to the BIOS which present a generic config screen. When games want to connect they just issue a connect() call to the BIOS and they don't have to care about all the details. That also makes sense from a security point of view.

It could be a possibility to keep keep the wifi config on the helper MCU (access points, passwords etc). For the config screen...we don't want every game to re-implement the same thing over and over. Since there's so many video modes it's going to be complicated to make a single version that's part of the kernel. Perhaps only one for mode 3 (for now). More over, if the network interface differs between wifi or wired Ethernet, config will differ and more code/flash will be required. Which begs the question: In this era of wireless "internet-of-things", do we care about wired Ethernet anymore? Do we agree to only go wi-fi? That will simplify things.

Food for though/crazy idea: For the config screen, a simple monochrome, ram tiles based text video mode acting as a dumb terminal driven by the helper MCU. When entering "config mode", the latter would push fonts and text to draw screens and receive keypad inputs. Not easy to do multi-video modes right now, but this would allow upgrading the config stuff independently of the game ROM. :ugeek:
Post Reply