Uzenet

Topics regarding the Uzebox hardware/AVCore/BaseBoard (i.e: PCB, resistors, connectors, part list, schematics, hardware issues, etc.) should go here.
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzenet

Post by D3thAdd3r »

uze6666 wrote:What about reusing the MAC and reserved bytes:
Perfect, considering resolution, ram limits, and not having any credit card information on the Uzebox: the name and password lengths should work nicely(I wouldn't even bit pack them down for A-Z,0-9 to keep things easy).
uze6666 wrote:Also esptool is nice, I see it often in other forums. Regarding custom firmware, I recently saw there's an aduino IDE fork released to build and deploy custom firmware to the ESP8266!
My initial assumptions about Uzenet are radically towards simplicity and standardization. People should just buy the Uzenet module and an ESP8266 pre-programmed with it perhaps? It depends how much burden that puts on you, maybe it's no good. Otherwise I thought it would be nice because it is a pain to make a setup rig for firmware updating as it's not breadboard friendly spacing, but really, not too annoying...and who am I kidding, that would be the absolute last thing I'd want to work on since it's not that fun.

I saw that Arduino setup and it looked promising but I haven't tried to build anything yet. Part of it is I'm not sure I understand it either; it looks like they just give you some basic functions you can use and hide the rest? The espressif firmware is much lower level and has access to every detail. You can see the crap that is adding the network delays we were talking about, and I'm sure you could see why there has to be a UART delay between commands if dug into more. If there is access to low level UART and network details then I'm all about it. I'm still not at the point where I am working at all on this aspect. BTW on the kernel side stuff, one of your uzenet demos really had the basic stuff that should be official already working. Past that I think a state machine is a necessity which can't be generalized well, I assume?
kivan117 wrote:Would All Time and Weekly lists be possible?
Oh yes definitely would be cool. if you look at http://uzebox.net you can see a pretty humble web page mock up. Now the low level stuff could be handled/stored by the server, and later if we want to mix in websockets to access the data or whatever, there could be fancier stuff. The server keeps track of what time it is right now but does nothing with it. When it's up and working we could just have time stamps for each score, and occasionally run through all the data and output some different tables to update the index.html file on the disk. A simple webpage with generated content shouldn't be too much of a kludge even in C for an interim solution.
kivan117 wrote:For racing games like this where fastest time is per track how would we keep the scores updated? Also, time is a slightly trickier thing to track with a single "highest is best" number.
So my thought, say you have something like this pseudocode:

Code: Select all

uint32_t total_race_ticks;//less is better
//rest of your game stuff
#define MAX_VAL 4294967295UL

//somewhere else...
    while(1){
        WaitVsync(1);
        //do racing stuff
        total_race_ticks++;
    
        if(race_is_over){//send our score to uzebox.net!
            //connect, fight annoying text and delays, send normal packet stuff, and all that...
            uint32_t mag_flip = (uint32_t)(MAX_VAL-total_race_ticks);//subtract max 32bit value from our total race time
            SendUzenet32(mag_flip);//we send a higher score for a faster total race time, more is better
        }
}
To have a score at all assumes that you have quantized accomplishments in the game and they can be summed, so it should always be possible to convert it to a different direction. Even if you wanted to score a game like Zelda you would probably give something like 100 pts for acquiring the Master Sword, 1 point per Ruby, 50 points for the Red Tunic, 5 points for each secret, -10 for a death, etc. I bumped up the max score size to 16 bytes/128bits seeing as the server really doesn't take much overall ram. You could have an overall score, and also some individualized accomplishments depending on how you want to handle it in your game. If you want the high score to be what determines "better" and then extra stats that aren't plain score, just put the absolute score in the most significant bytes so it always overrides that stuff. If someone needs more it will be more, but then they are probably trying to bend the high score server to do save games. There will be separate commands better suited for that anyways...I got some done on the score server tonight so pretty soon we can see how it works in practice with real games. The solution should be legit but I'd be glad to hear a problem/solution to make it the most future proof so we never have to touch it again(except for converting it to Linux/BSD sockets in the end of course).
User avatar
kivan117
Posts: 73
Joined: Sat Mar 14, 2015 5:43 am

Re: Uzenet

Post by kivan117 »

We're on the same page as far as posting a meaningful score from a completion time. Tracking ticks, converting to score and posting was how I'd imagined doing it. My original concern was that scores from track 1 mean nothing compared to track 2, etc. But then I realized "just have the first ___ bits indicate track number" and now I feel rather silly, haha. By doing that, you could just rank them in order like normal and if you used the most significant ___ bits to indicate track number, it'll sort itself by track anyway. Then you just have some game-specific code to convert those scores back to meaningful times and decode track numbers when you go to display it all.
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzenet

Post by D3thAdd3r »

kivan117 wrote:By doing that, you could just rank them in order like normal and if you used the most significant ___ bits to indicate track number,
Good point. It had not crossed my mind about racing games obviously having more than 1 track where score comparisons between 2 objects mean nothing. I assumed this setup would cover any case possible, but thinking about the racing example you mention it comes to mind: Why is there even a notion of separate name and score? It does nothing to make things easier, takes a byte we wouldn't need to send, and is not as generic as would be possible.

EDIT-
I'll change this where you send
[magic number, reserved byte, command byte, entry to start from(read only), num entries byte, entry length byte, 8 chars of rom name, data with the length you specified...num_entries*entry_length]

Even better the read and write format are exactly the same.
User avatar
kivan117
Posts: 73
Joined: Sat Mar 14, 2015 5:43 am

Re: Uzenet

Post by kivan117 »

Cool cool. For ranking the incoming scores though... are we assuming a standardized name length, possibly based on the allotted bytes in eeprom for the default name? If not, how would the server be able to rank the score without knowing where the score ends and the name begins? Additionally, if sending a write command, do you still include the entry to start from bye and it just gets ignored? If not I'm confused as to how the commands sent are supposedly the same.

Back to the weekly vs all time lists... For sending in a score for writing it doesn't seem to matter, send it and let the server compare it for both lists or however it wants to deal with it. For reading though, I feel like there should be a way to ask for positions ___ through ___ within the last ___ time. That way the game can decide if it wants scores ranked by days, week, month, whatever. And just have a special periodicity number indicate all time
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzenet

Post by D3thAdd3r »

kivan117 wrote: For ranking the incoming scores though... are we assuming a standardized name length, possibly based on the allotted bytes in eeprom for the default name? If not, how would the server be able to rank the score without knowing where the score ends and the name begins?
My thoughts were that some games are not going to want to store that long of a name and instead use 3 initials as is common In Pacman,DK,and others. Usually you try to squish all your stuff in 1 eeprom block if possible, so I don't think forcing any part of the format is good? I agree the standard name length and password length according to Uze's eeprom table for account stuff like chat room or network games; anything where you need to log in. We could impose having to log in to create a high score too...I don't know. For now it is going to be easy as possible to get games using at least high scores and to get an implementation working. As far as ranking we are just taking the most significant byte idea even further and more generic, if the first byte of the data is higher than something else it "beats" it regardless of the following LSBs. So most games would probably put the score first(or track number) to get sorted by that and use the less significant bytes for further sorting, and finally the rest for name and non-score elements. Again the server does not care what any of it means and you get to design your own format with the only rule being MSB sorting.
kivan117 wrote:Additionally, if sending a write command, do you still include the entry to start from bye and it just gets ignored?
Hmm true I guess we aren't really saving bytes that way. Should it be a dummy byte or just use different read/write formats? My game was initiating a command for every entry in the eeprom if it hadn't sent it before as separate steps, it's kind of a kludge, but I wanted to make sure scores that happened offline would make it online when available. I think it is going to be easier for people if they have the option to simply dump their entire eeprom to the server with a very simple descriptor, and let the server figure it out. You can still do that quickly with a tiny buffer, and the code can be micro. Then server sends back sequential bytes representing what each one of the entries you wrote placed in the list(of 128 positions max, where 255 means did not place).
kivan117 wrote:For reading though, I feel like there should be a way to ask for positions ___ through ___ within the last ___ time. That way the game can decide if it wants scores ranked by days, week, month, whatever. And just have a special periodicity number indicate all time
I noticed Ghosty Ghost as currently released is not sending anything out on the UART, do you have a system in place that deals with the 8266 yet? Again, I regret the slow progress on the high score server and it's top priority right now. At what point in your game are you planning to do the different steps like initial connection, sending data, etc? As far as the time goes I haven't put a lot of thought into it. I made a placeholder function that will check if the day is not the same as last time it checked and increment a per high score variable named uint16_t days_old[]. That is straightforward, but then concept of precision comes into play. I made another variable called uint32_t ticks_old[] for each score. Basically every tick on the high score server is 1/1000 second and ticks_old[_] gets incremented. Every time the server notices the day changed ticks_old[_] gets reset to 0. So you could always know pretty accurately to 1/1000 second+latency how old a score is using only 2+4 bytes. That is not done in a good way yet, it literally uses Sleep(1). I don't know what format would be best for this. Should the server always send 6 byte time stamps for each entry or should there be a separate command that asks for all entries no older than what is specified. That will come later and take some more work because I'm not yet sure what a good cross platform time mechanism is(SDL is overkill). If you can think of a format we will put the placeholder and you can implement it in the game, it will work when the server supports it. Thoughts?

Edit-another separate read command would be easier on the Uzebox side and still allow the smaller responses of the normal, non-timed, read command. Thinking about this more, it's truly not trivial, I can't figure it out. Since 10 people could have made scores in the last hour, and on your Uzebox you only want to deal with 10, but you want to display everything for the last week...but sometimes that might be 50 entries, other times 0,....really at a loss on the right way to do this but I think it's a cool idea. Can you see some generic solution to this that would work for everything? Otherwise I think just basic time stamps and having the Uzebox do what it wants is the only possibililty, which is resource intensive, but I think Uzenet stuff should always assume the SPI ram is there
User avatar
kivan117
Posts: 73
Joined: Sat Mar 14, 2015 5:43 am

Re: Uzenet

Post by kivan117 »

I guess I was naively thinking that the server would do all the work and just send the requested info to the uzebox. Rank all scores by age, youngest first, cutoff below requested age. Rerank that subset of scores by normal msb sorting, send requested entries. Client asked for entries 11 to 20 and there are only 14 total in the requested timeframe? Send 11 - 14 and then "out of range/end of list" or even just 11 - 14 followed by 6 null entries (less efficient and not preferred obviously).

Ghosty Ghost doesn't have its UART code added at all yet. This weekend I'll try to finish adding net code and possibly parallax.
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzenet

Post by D3thAdd3r »

kivan117 wrote:I guess I was naively thinking that the server would do all the work and just send the requested info to the uzebox.
You're right it's too hard on Uzebox side. I looked into this more and there is crossplatform time() and each score entry gets a time_t variable when it is created that stays with it through sorting. To request the time the server just uses difftime(time_var,current_time) to return the age in seconds. It's subject to the Unix Time Problem and will rollover on January 17, 2038, but lets worry about that when we get there ;)

I'm thinking a separate "time read" command similar to the normal one, but you also specify an extra 32bit number for the max seconds old an entry can be. The server sends a packet back with the first byte being how many entries match this, then, up to a maximum of (requested_entries*requested_entry_len) bytes, or 0 bytes if no entries match. This way you can easily do "best scores today", "best this week", "best this month", and quickly know not to display the screen if there are no entries that match. It would be more involved for stuff like "last week, but nothing from this week" or "2 hours ago, except for a 30 hour period here.." kind of thing. I really think that is pretty fringe as far as usefulness, sounds like a boring homework assignment without real world use. Someone can implement that later if they feel inclined, there are 100 things I would do before that. More to come.
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzenet

Post by D3thAdd3r »

I have the server generating a basic webpage with the same style as whats currently at uzebox.net. So I am working on making the generation more configurable and I came into a problem I hadn't considered.

With your racing game example and the generic approach the server is currently taking to scores, how do we know what to write on the webpage for each entry? One solution would be to have a separate function on the server for each game for writing html table rows. Every new game that wanted to do it would require adding to the server source and recompiling, I don't like the solution even if it would be the easiest in the short term. The other way I can think of is format strings similar to those used in printf().

Basically you have a game you are working on. You use a special command to create a new game entry where you specify the 8 character rom name like "GGHOST\0\0", a true name like "Ghosty Ghost", a format string like "%d Columns", argument0 data start, argument0 data len, argument0 type. So for a racing game you might make an argument string like "Track %d:%d:%01d", then specifiers for start and length for track number, minutes, seconds, etc. The idea would be that this string gets passed directly to sprintf() into a buffer, and the arguments get converted somehow. Haven't got that all figured out but I think it would allow pretty easily to display any data however you want on the webpage.

Now even more complexity I can't see a way around. For a racing game it would be common to have best total race scores AND best lap scores. The best way I can think of is to simply create an entirely separate list for that stuff. If you make a racing game you could simply set up different entries with name likes "RACING_GAME Best Time", "RACING_GAME Best Lap". The list search has been based on the rom name, but in light of this I think I will change it to search the "true name". Less and less easy :cry:
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzenet

Post by D3thAdd3r »

Triple post. I think the format string idea will work well and allow really cool custom score displays but a lot of work.

On time I know 100x more than I knew yesterday about computer time, which still isn't much. From what I gather, timezones are rather a kludge that there is no real good way that is cross platform(and ideally the server will be Linux, but I think it would be nice to support windows also, since that what it is right now). Telecommunications and engineering standards seem to prefer the UTC timezone as a universal standard. Read up on wikipedia and time() functions a little and you will know as much or more than I do about it. I propose Uzenet time is UTC time, if you want to make a local clock on Uzebox then add/subtract the hours necessary. Maybe it's just me, but I have 5 clocks within view(my girl fancies herself an interior decorator) for local time I don't need Uzebox to tell me. Thoughts or suggestions why local timezones might be needed on Uzebox side? I'm not necessarily against it because we could find a solution for different OS, but someone would need to come up with a solution for how time works on the webpage that displays the score times. No clue on that stuff. Does anyone have some format or method in mind that would allow the Uzebox to specify a timezone and also the code that would be needed on the server?

The high score server is up in broken form. For now forget about read/write because I am changing the way lists are sorted, right now things are getting corrupted until fixed. The only thing it will reliably do now is respond to heartbeats(command 0), disconnect(command 64), and more interestingly UZENET_COMMAND_GET_TIME(command 6). If you connect to uzebox.net:51697 using TCP and send this:
[0xC7, 0, 6, '%', 'a', ' ', '%', 'b', ' ', '%', 'd', ' ', '%', 'Y', ' ', '%', 'X', ' ', '%','z',0]
A real Uzebox, through the power of distributed computing, will get a response something like this:
"Sun May 03 2015 22:57:03 Central Daylight Time" Which it could never have figured out by itself no matter how clever the programming.
You can also get back simpler and smaller strings or any possible string strftime() can generate, because all it does is put the string you sent into that function and spit back the results to Uzebox. I have more stuff 3/4 working than that...but at least functionality to play with today and I think it is currently uncrashable by any input. The chat server is back up, it was down for a time.

Edit-learning about pipes, it's pretty cool stuff. It is easy to take a string from the Uzebox, run it in a command prompt on the server PC, then dump the output back to Uzebox. Obviously full control is a bit dangerous, but currently if you send this packet:
[0xC7,0,7,'h','t','t','p',':','/','/','u','z','e','b','o','x','.','o','r','g','/','f','o','r','u','m','s',0]
it appends it to the string "lynx -dump " which causes the lynx web browser to quickly load up the forums page, convert it to a quite usable text version with links at the bottom, and send it to the uzebox. Probably no one has the capabilities of doing that at the moment, but it works and it's pretty ridiculous :D It might end up very usable to the point where we could inherit things like telnet, ftp, etc with no additional programming simply because those programs already run on the server and it is unlikely very many people will be using it at the same time. I better sleep now, but that is madness!
User avatar
kivan117
Posts: 73
Joined: Sat Mar 14, 2015 5:43 am

Re: Uzenet

Post by kivan117 »

Wow, firstly thanks for all the work on the server side of this whole uzenet thing. The pipes idea sounds really cool, especially for all the late possibilities it opens up.

I genuinely wish the people who made the stock 8266 firmware hadn't made it so awkward and verbose.

As far as time, the only uses I can think of for local time would be time stamps on a chat server or something that the user wanted to be local. I completely agree that server time should always be UTC and a simple offset is the best approach. In asking for the time, could the request include a single timezone offset byte somehow? Send me the time string per my request, but offset the hours from Uzenet standard by some value, that sort of thing. Seems like a simple two minute fix to prevent every program that wants local times from duplicating the effort.
Post Reply