Uzenet: Difference between revisions

From Uzebox Wiki
Jump to navigation Jump to search
No edit summary
Line 148: Line 148:


To initiate a file read the program needs only to call UzenetFile(filename,speed) and the data will start coming from the server. It is program specific what gets done with that data, but the expected use is streaming writes to an SD card using FatFS.
To initiate a file read the program needs only to call UzenetFile(filename,speed) and the data will start coming from the server. It is program specific what gets done with that data, but the expected use is streaming writes to an SD card using FatFS.
==Telnet Client==
[[Configure CYGWIN telnet server on windows]]

Revision as of 22:30, 13 September 2018

Scope

This page documents Uzenet, the network interface or the Uzebox.

General information on the Uzenet interface hardware is here: ESP8266.

Information on the Uzebox kit's (using PCB V1.3.1) Uzenet interface is here: Uzenet_Assembly_guide.

Glosary

  • Uzenet: The general name for the network interface as a whole. It encompasses the kernel API, physical interface to the outside world and any server or services required in support.
  • Uzenet module: The physical interface to the network. It it made of an interfacing microcontroller and an commercial-off-the-shelf (COTS) Ethernet or WiFi module. The module can be interface using the Uzebox EX1 PCB extension port (EXT) or the player 2 controller port (P2).
  • Uzebus (or just bus): Communication channel between the Uzebox and the Uzenet module. Made of both a custom physical and protocol specs.

Requirements

The goal of this project is to provide a way for the Uzebox to connect to the internet in order to play games with other players.

  • Use inexpensive off-the-shelf wifi/Ethernet module
  • Encapsulate such proprietary modules behind a control MCU that implements a standardized "Uzebus" protocol.
  • Support UDP and TCP protocols
  • Support 1 socket than can both send and receive
  • Work with all Uzebox versions
  • Allow the use the EXT port if available
  • Must be lightweight for the user in term of CPU, RAM and flash
  • WiFi configuration should not be managed by the game
  • Uzebox must control the protocol flow. That is, it must read and write data when it's available and the module must buffer data.

Main tasks

  • Define the bus physical layer. Includes interfaces for both the EXT and P2 ports.
  • Define the bus protocol
  • Define the Uzenet module API
  • Evaluate, choose & test Ethernet/wifi module
  • Choose best Uzenet module MCU
  • Design Uzenet module circuit & PCB
  • Develop Uzenet module firmware
  • Define kernel API
  • Develop kernel API
  • Define server protocol
  • Develop server
  • Host server


API Documentation

The API is designed to be as efficient as possible for the Uzebox. All functions are documented here: API_Functions


Services

Uzenet is comprised of multiple services that have been designed to allow your game or application to easily take advantage. The currently implemented and planned services are:

  • UzenetScore - this service keeps track of the best achievements for the players of your game. You can read or write to the online database.
  • UzenetChat - allows low resource chatroom functionality to chat to other players to setup games, discuss development, or just for fun.
  • UzenetGame - this is the core gameplay service that handles communication between 2 or more uzeboxes for networked games.
  • UzenetFile - to have the Uzebox automatically keep it's rom set up to date without hunting around, this allows any game rom to be downloaded or uploaded.
  • UzenetSave - this is a very useful service for complex games. This allows you to store an infinite amount of information for your complex RPG or what have you.
  • UzenetForum - speculative service that would translate the forums active topics page into a Uzebox usable form allowing to read/post.
  • UzenetWebTunnel - even more speculative, this service takes http request from an uzebox, loads and renders them with the Lynx browser, and sends it back in a possibly usable form.

In the protocol explanations, <->,->,<- will be used to indicate the direction of data flow for that section of the protocol.



UzenetScore

This service is designed to keep track of game high scores that is independent of any particular format the game may actually use. Each game has 32 32byte buffers for high score entries. The list is always sorted when a new entry is added and all game should be referred to according the their standardized rom name, which if shorter than 8 should be packed with '\0', see Games_and_Demos. To sort, the server simply takes the first 2 bytes of the buffer to indicate an "absolute score" the rest of the entry can be set and interpreted however you see fit for your game. The way this works is that you must be able to clearly define a score that is better than another and also a perfect score which would have the highest 16bit value of 65535. Whatever the rest of the data is, the first 2 bytes must be able to indicate how much better or worse it is than previous entries using the same methods. Complex programs could read the entire list, erase it on the server, and write a new list with scores adjusted relative to other entries. Use UzenetSave instead as that doesn't seem a "high score" type of game. Every time an entry is added, the server automatically records the server time.

Most games would be very simple to have 1 number that indicates how good it it. Some complex RPG etc will have a more difficult time defining that no matter what the actual byte storage methods are. The following is an example of a simple high score format for the game [Frog Feast]. Here the number of flies caught at the end of the round determines how good the score is and for this example the player has a score of 89 and entered the name "Gosma". Each entry is a byte.

  • ['S','E','T',' ','F','R','G','F','E','A','S','T',0,89,'G','o','s','m','a','\0',....]//the rest of the bytes are not needed by the game.

The protocol is designed to be the most efficient possible taking into considerations the limited ram available on the console. Because different games will have different amount of ram free, the uzebox can specify to read only the score entries it want instead of the whole list at a time. If you want to get the entire list you can get a smaller piece of it, render it to the screen, and repeat that process until you are done to page through and show all scores. The following is for informational purposes on what the process is, the kernel hides the details and makes it easy:

  • uzebox->internet->connects to uzebox.org:51697 using TCP
  • uzebox->server:"UZESCORE"
  • server->uzebox:"OK"
  • uzebox->server:['S','E','T',0,89,'G','o','s','m','a','\0',....] or ['G','E','T',0,5]
  • server->uzebox:"...requested data"
  • server<->uzebox: close connection.

The format is very simple. To GET information about a certain score entry for a certain game you would, using Frog Feast as example and wanting the first 5 entries, send: "FRGFEAST 0,5" and the server would proceed to send the 5*32 bytes that make up the data for those score entries. This can be done automatically using the kernel with Uzenet_GetScore("gamename",x,x), Uzenet_SetScore("gamename",x,x), and Uzenet_DelScore("gamename").

UzenetSave

This is a very powerful and very easy feature to use in your game. It allows you to save game data that you might normally put to the EEPROM, in online storage instead. This removes the space limitations by creating a separate folder for each user and a separate file for each game save that will grow with demand. The server keeps a file containing all save data for every user, and upon request can read an entry, create a new entry, or delete, then later overwrite an existing entry. The format of the requests is different for reading and writing. The example below shows the buffer of bytes for a game called "Zelda" with a console that is named "Paul" that wants to save the 11 bytes {22,44,96,23,11,6,121,58,58,23,11} starting at byte position 14,148(0x3744 in hex). Note that first the console name string is written, then the rom name, then the address to start the operation at(0x3744), and finally the number of bytes we are writing.

  • ['W','R',' ','P','a','u','l','\0','Z','e','l','d','a','\0',0x00,0x00,0x37,0x44,11,22,44,96,23,11,6,121,58,58,23,11]

and at any time in the future that exact data can be read from the server using this packet format:

  • ['R','D',' ','P','a','u','l','\0','Z','e','l','d','a','\0',0x00,0x00,0x37,0x44,11,]

if a program wishes to delete a record it can simply send this string

  • ['D','L',' ','P','a','u','l','\0','Z','e','l','d','a','\0',]

You can see that the only real difference here is that we used RD for the first 2 bytes and instead of appending the data you want to write at the end of the header, you just get ready to receive the data you requested.


  • uzebox->server:"UZESAVE"
  • server->uzebox:"OK"
  • uzebox->server:['R','D',' ','P','a','u','l','\0','Z','e','l','d','a','\0',0x00,0x00,0x37,0x44,11,]
  • <server: opens file "Zelda.sav" for reading in folder "Paul" and sets file position to 0x3744>
  • server->uzebox:[22,44,96,23,11,6,121,58,58,23,11]
  • server<->uzebox: close connection.

You don't really have to know that information, as the kernel provided the functions UzenetSave(name,romname,offset,numbytes,buffer_pointer,rd_or_write) which will do that automatically. You only need to provide which operation, read or write to the last variable and a pointer to a buffer that will be sent to the server in the case of a write or written to in the case of a read.

UzenetVirtual

This is another very powerful and possibly easy to use service. If you are using the service in an asynchronous way where your program does not block execution until completion it will be relatively more complicated. This works in much the same way as UzenetSave, except that the data is kept in ram on the server and must be explicitly accessed every few hours or it will close. UzenetVirtual can easily be used to save or load a file that was saved with UzenetSave because the servers communicate with each other for fast response. This allows games to have an "infinite" ram buffer that can be persistent, keeping in mind that this virtual ram is limited mostly by the UART speed of the Uzebox and the latency of the internet. The server will only restrict this if it does not have enough ram available, but developers should absolutely not get excessive with ram waste or it will have to change. By sending a console name of "GLOBAL" any Uzebox program can read or write this buffer in real time and that data being available to everyone for large scale world Uzebox games. It can also be set to private for use as variables for a specific user just like UzenetSave. The same commands "RD" and "WR" are supported and work the same way. There is no need to ever resize the array as any access that goes beyond the current size, will cause the array to be re-sized before being written. This is to be as simple and fast as possible for the Uzebox, but take care with your programming.

Whenever a request is made for a specific virtual ram entry the server first checks to see if it is already open. If it is, the data is sent or received, otherwise the server creates a buffer with the default size of 32k. Once the buffer is open, it processes the command as usual. There is only 1 special consideration when you want to directly load a save file on the server to ram without using the Uzebox itself(slow) to transfer the data. If you write to the address "0xFFFFFFFF" then the server interprets it as a request to load a save file using the user name and rom name that are already provided. The kernel function UzenetVirtualRam(name,romname,offset,numbytes,buffer_pointer,rd_or_write) works exactly like UzenetSave otherwise. For speed purposes, the kernel internally knows that the server will not require the normal "UZEVRT" authentication so long as this buffer already exists and has not been timed out. It will simply send each request without having to "log back in". The function itself will normally block further execution of the main program until all data, or an error, has been received. If the flag UN_ASYNCHRONOUS is set in the flags variable, then the function will return immediately after the process has been started. It is then the responsibility of the main program to handle the incoming data in whatever way makes sense. If the user requires that no blocking while waiting to connect to the server should occur, there is no generic and efficient way to do it and you must write your own methods using the protocol. The later intent is to allow scripts to be uploaded and ran server side to allow very large games where the Uzebox acts more as an input/rendering terminal.


UzenetFile

This is a handy service that provides an easy way to stay up to date with the most current roms. The intended purpose is to be able to download new versions of a rom and write to the SD card. The also include the .dat files required by some games. This service is intended only to allow an Uzebox to have a SD card that is setup just like it would have been if it was just shipped out today: with all current and official files. All current firmware files for the actual networking hardare, eg. ESP8266 will also be allowed. It is not intended for general file storage of music, video, etc, except where something like UzeAmp may have official .wav files that come with. The server references the wiki page at [1] to either allow or reject an upload request so you need to modify that page if you make a new game. This service works in an extremely efficient manner for the Uzebox. The Uzebox has a short authentication sequence similar to other services(and this is only to avoid a port scanner triggering the server wasting bandwidth). Once logged in the Uzebox can use the commands "LS" to get a LiSt of files available, "RD" to REad/download a file, or PAuse a file download if necessary. For LiSt, the command must be followed by 2 bytes that indicate which file name to start at, and 1 byte for how many file names to send. This is to cater to the Uzebox not having enough ram to store the name of every rom and file. Every list command the server sends will start with 2 bytes indicating how many files are actually on the server. This lets the Uzebox know how many entries it should retrieve names for. For read operations, there is a 2 byte value that indicates the maximum speed in bytes per frame that the server should send the data. This is to avoid overflow problems due to limited Uzebox ram and latent SD card writing. PAuse can be sent at any time, and the file transfer will not continue until another PAuse has been sent. PAuse is also followed by another 2 byte value so that the speed of file transfer can be dynamically changed after the transfer has started should the need arise. Let's look at an example where the program wants to download the rom "MEGATRIS.UZE" from the file server with a maximum speed of 256 bytes per 1/60 second(15,360 bytes a second).

  • uzebox->server:"UZEFLE"
  • server->uzebox:"OK"
  • uzebox->server:['M','E','G','A','T','R','I','S','.','U','Z','E',256]
  • server->uzebox: contents of file MEGATRIS.UZE, 256 bytes every 1/60th second...
  • server->uzebox: ...
  • uzebox<->server: close connection.

To initiate a file read the program needs only to call UzenetFile(filename,speed) and the data will start coming from the server. It is program specific what gets done with that data, but the expected use is streaming writes to an SD card using FatFS.

Telnet Client

Configure CYGWIN telnet server on windows