Uzem ESP8266 Support

The Uzebox now have a fully functional emulator! Download and discuss it here.
Post Reply
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Uzem ESP8266 Support

Post by D3thAdd3r »

Ok I worked on this a couple solid days but I have been slacking for a week now. Showing what I have so far will force me to keep focused on this, as I already took a huge detour(Fairchild Channel F emulation,perfect+~fullspeed likely). I am not a hardware guy, and for me to focus on the assembly stuff is a waste of time since it could be done 10x faster/better by Alec/Andrew(Cunning), so what I can contribute is fueled by a strong interest in emulators and sockets programming.
earlyesp8266.jpg
earlyesp8266.jpg (300.17 KiB) Viewed 14414 times
This is very preliminary but it works, albeit very slowly in the current implementation. I implemented partial UART emulation in Uzem and I have a partial ESP8266 emulator core that is working enough to use the chat server. The build attached is very slow and the UART code is under serious surgery to make it faster. I think it should be possible for very fast processors to emulate it full speed when it is done, but I also suspect that it would be practical to run the entire thing in a different thread. I am changing the way the emulation works to be more abstract than the real way UART works on the console. I believe it's important for speed, and I believe we can have this and still function correctly since there are 2 very distinct send and receive pipelines that are not relying on eachother. Right now every time a new UART byte is sent the entire AT Command processor runs, checks if it's terminated and goes from there. What I did in this build just to be fast enough is artificially set the UART to an incredibly slow speed so we don't have to call a function every few machine cycles. I am already changing this over so that, all sent UART data is immediately put into the receive buffer of the 8266 core(respecting time before UART is ready for next char) but the ESP8266 AT core never runs unless a '\n' was just sent since all commands must end with this. The already "fully functional" internet part that receives data packets and puts them on UART could be called every machine cycle if you wanted true emulation of parallel processing(2 chips), or as rarely as you wish. I think it needs to be in another thread for emulation speed and responsiveness to internet data, but it should be very flexible and still work.

That's all I have to say for now, it is what it is, a preliminary implementation. The sockets stuff was pretty easy, the UART stuff getting fast and better/faster command parsing is the only real challenge to complete it. It is not practical or necessary to emulate the actual chip on the module that runs the AT parser+networking stack. Your operating system already handles all the networking stack, you can even "emulate" the module acting as a wireless access point since your computer looks just like a wifi router to the emulator.

To test it, use the very slow build of Uzem with buggy ESP8266, then run the listener.hex and wait for a while. You should see it gets connected to the chat server and names it self listener, if you wait you should see nothing else happen(unless someone else is online). Now, keep that listener running, and also run chatter.hex in another instance of Uzem. You should see there that data went from your home, to uzebox.net in New York, was processed, and sent back as a chat message both listener and chatter will see. Close Uzem and you should see a client disconnect, etc. Pretty cool if you ask me.
Attachments
uzem.zip
(442.81 KiB) Downloaded 825 times
User avatar
uze6666
Site Admin
Posts: 4801
Joined: Tue Aug 12, 2008 9:13 pm
Location: Montreal, Canada
Contact:

Re: Uzem ESP8266 Support

Post by uze6666 »

Awesome work Lee! I'll give it a spin when I get back home. :)
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzem ESP8266 Support

Post by D3thAdd3r »

BTW if you experiment a bit keep in mind the 8266 core doesn't currently support multiple commands per send. As soon as it gets a command it implements it and flushes the whole receive buffer. That only affects sending temporarily. Just break it into 2 parts instead of all in one like the Uzenet test you made:

Code: Select all

	wifi_SendStringP(PSTR("AT+CIPSEND=0,9\r\n"));
	wifi_SendStringP(PSTR("UZECHAT\r\n"));
It should do most anything as is.
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzem ESP8266 Support

Post by D3thAdd3r »

Ok this is at a better state. The attachment includes chat test where you can run it in 2 instances of Uzem and see each one talking. When you close 1 Uzem you will see the disconnect message on the 1 still open. This is all running through the buggy chat server @ uzebox.net, for now, as I am not going to improve that and instead will implement a real IRC client for Uzebox. There is an IRC server running at uzebox.net if anyone cares to try it with an existing IRC client.

So basically this is Uzem 1.19 with threaded ESP emulation because I found running it serially inline with the op code emulation futile. I will bring it to 1.4/2.0 which is a huge speed increase maybe 33-50 percent if I recall, at which point I hope some machines might be able to run this full speed. I am having rather stupid problems trying to get 1.4 to compile with SDL2 still.

Anyways try it out and let me know your results! It is getting pretty close to done and supports about 85+% of command details. Just a matter of fine tuning which at this point is not too technically challenging. I will talk about some more details later but the source is included for those curious. I will keep improving this, but it may already be a worthy development tool. I expect there to be many small changes required as more programs are compared on real hardware versus the emulator. So far I have not had a lot of time to do that.

I would really appreciate what emulation mhz speed you are getting and your core speed specs to get a better idea of what needs to happen. I am really hoping a 3.5 ghz, when ported to the 1.4/2.0 code base, will run full speed. Previously without threads, there was not even a hint of any existing computer running doing so.
Attachments
uzem_ESP_threads.zip
(446.73 KiB) Downloaded 787 times
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzem ESP8266 Support

Post by D3thAdd3r »

Quite a few improvements here, and plenty of little bugs squashed that I found during more advanced testing. The real big thing is I(hopefully) eliminated all bugs involving race conditions and timer logic, and there were a lot more than I knew about until I actually started testing in more detail. I believe now the timing is for all intents and purposes, close enough to the real thing. Game requirements will set the focus for improving things in certain areas and setting delay values to more realistic ones; I have not set about to extensively testing and averaging responses on the 'scope yet. The source is up on GitHub for anyone that is interested, particularly esp8266.cpp, esp8266.h, and update_uart() in avr8.cpp would be the good parts. At this point, I think it is the most complex thing I have ever programmed so I feel pretty good about it. The ESP8266 emulation code is about the same amount of lines as all the other parts of Uzem combined, but it is very modular and separate.

Keep in mind this is still based on the old slow version of Uzem, I have not moved to the newer faster codebase yet because SDL2 really hates me. To be honest I might need someone to help me with a "step by step guide to installing SDL2, and setting up Uzem 2.0 so it actually compiles in windows" on the wiki...I am baffled and have tried everything I can think of and reinstalled everything, in every variation...er, Please! No Linux build yet until I get the codebase over to the new version. Code is messy and has quite a bit of debug stuff on right now, but this is close to done and not in the way I normally say that. By close to done, I mean the point where someone could write a networked game entirely in the emulator, then later on when he gets hardware, find it works there just the same.

I am bad at GitHub and I don't know how to change the old notes. Unvarnished mode is now fixed, the timing is now realistic and NAGLE algorithm is disabled so that you get a real 50hz packet output. Everything network wise was already designed/wrapped for cross platform so there wont be any gross hacks to make it work on win,lin,mac,bsd,etc just the same.
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzem ESP8266 Support

Post by D3thAdd3r »

I was looking at a bunch of commands I did not find too useful yet that are just empty functions in the emulator right now, and found something pretty cool I was overlooking. One of the new commands available in the latest firmwares is something with a format like this:

Code: Select all

AT+SAVETRANSLINK=1,"uzebox.net",5800,"TCP"\r\n//now the module connects to uzebox.net on port 5800 whenever it starts
What this does is save a link to flash, which upon reset, the module will automatically connect to. I still need to experiment with the real hardware to determine more of it's behavior, as documentation is not too detailed, but it seems this is quite a big thing for making the code small and easy to add into existing games. It eliminates basically all the code to get to the point of connection to uzebox.net. I am going to add this to the setup rom after you pick your AP and upgrade firmware to the newest version that supports this. Then besides just having a wifi connection and known baud rate at startup, programs can also assume there will be a connection to the server if one is possible. This is the new method going forward which I will show a working example that is generic for any game, pretty exciting cleanup of UART code this way.

There may well end up some bugs here and there in things that are implemented. For commands that include the _CUR and _DEF addition like "AT+CWMODE_DEF=*", most are supported as special cases in the normal command like "AT+CWMODE=*". Reworking the way the config file works, since I realized it does not actually work like the real thing yet. Besides that, here is the status of all commands I have heard about and implemented most of. This is including additional depreciated commands available in older firmwares that are not present in the new ones. The ones remaining I don't see an immediate use for AND are not obvious how to best implement yet; I already implemented the useless but easy stuff.

Code: Select all

	void AT_AT();
	void AT_RST();
	void AT_GMR();
	void AT_GSLP();
	void AT_ATE();
	
	void AT_CWMODE();
	void AT_CWJAP();
	void AT_CWLAP();
	void AT_CWQAP();
	void AT_CWSAP();
	void AT_CWLIF();
	void AT_CWDHCP();
	void AT_CIPSTAMAC();
	void AT_CIPAPMAC();
	void AT_CIPSTA();
	void AT_CIPAP();
	
	void AT_CIPSTATUS();
	void AT_CIPSTART();
	void AT_CIPSEND();
	void AT_CIPCLOSE();
	void AT_CIFSR();
	void AT_CIPMUX();
	void AT_CIPSERVER();
	void AT_CIPMODE();
	void AT_CIPSTO();
	void AT_CIUPDATE();//totally fake, but output is correct
	void AT_IPR();

	void AT_UART();
	void AT_RFPOWER();//untested
	void AT_RFVDD();//not implemented
	void AT_RESTORE();
	void AT_CIPDINFO();
	void AT_PING();//have an implementation...but not sure best way to fit into the system
	void AT_CWAUTOCONN();
	void AT_SAVETRANSLINK();//untested
	void AT_CIPBUFRESET();
	void AT_CIPCHECKSEQ();//not implemented
	void AT_CIPBUFSTATUS();//not implemented
	void AT_CIPSENDBUF();//not implemented
	void AT_SENDEX();//not implemented
	void AT_CWSTARTSMART();//partially implemented
	void AT_CWSTOPSMART();//partially implemented
	
	void AT_CIOBAUD(const char *OverrideString);
	void AT_BAD_COMMAND();//generic placeholder for commands that are not yet implemented
	void AT_BINARY();//hypothtical binary command mode, which does not exist on real hardware and maps back to normal AT commands
	void AT_DEBUG();//emulator only command that enables the multitude of debug output to the console, very slow, but makes it easy to find problems
Post Reply