Published: 2012-06-29 | Categories: [»] Electronicsand[»] Programming.
Last Modified: 2014-07-19
1 Introduction

The potential interest in adding communication facilities between a microcontroller and a PC is relatively clear. Any electronics enthusiast with a few years of practice will sooner or later spend hours behind his desk trying to exchange data frames between his computer and the latest circuit he was up to.

Almost systematically, he will first seek for the asynchronous RS232 serial protocol that most microcontrollers manage natively. However, and since the 2000's, no more computers get shipped with serial ports (also called "COM port") anymore and the technology is then often discarded by amateur because of its deprecated nature. He will then turn to more powerful USB technology but face an increased complexity. Indeed, only few microcontrollers manage USB natively and even when they do, it's still extremely troublesome to use.

Still, there is a nice alternative to the issue: RS232―USB converters. They exist as all-integrated serial cables with USB ending, as extension cards for laptops or as electronic chips (such as the [∞] MCP2200). The computer then uses a driver which creates a virtual COM port and converts all the USB protocol such that the circuit can be accessed just like if you had a real serial port.

Even though, the complexity of having a reliable communication channel will make most amateurs turn back to irrational solutions such as reinstalling a Windows 98 because serial management is said to be easier on older OS. We will see here that using RS232 in a reliable way under a Windows XP, Vista or 7 is actually pretty easy to do once you know how to get it working safely.

This post objective is double: present the RS232 technology and show how to use it safely on modern OS.

2 A brief snapshot of RS232

Because the RS232 protocol is much more complex than what seems at first sight, I will focus here on a very short description that will be used as a bare comprehension for the implementation.

RS232 protocol is a serial duplex asynchronous protocol. Data are sent one bit after the other (serial) without any external clock signal (asynchronous). Data can also be sent at the same time they are received (duplex). The minimal number of wires to exchange data with the RS232 protocol is 3: one for the reception of data (RX), one for the transmission (TX) and one for the electrical ground (GND) to make voltages on the RX and TX cables meaningful. There are other functionalities available but I will not describe them here.

On the electric specifications side, a logical 0 bit will be coded as a voltage potential between +3 and +25 Volts and a logical 1 bit will be coded as a voltage potential between -3 and -25 Volts. This is the first issue usually met by amateurs because these are pretty unconventional voltages for microcontrollers. A translation unit such as the MAX232 is then required. The circuit layout recommended by the manufacturer is given on Figure 1. It is also recommended to place a 1 µF decoupling capacitor on the power supply of the chip.

Figure 1

The MAX232 includes a voltage doubler and a voltage inverter to produce levels of about +10V/-10V compatible with the RS232 specifications. Be aware that the chip was not designed to act as a current source and it is then not recommended to use it as negative power supply for other elements such as op-amps. If you need a symmetric power supply, uses a component like the [»] TC962.

If your computer has a real serial port, you will have to use a 9-pin crossed cable also called null-modem cable. These cables invert the RX and TX between their two ends. If you run into problems with data transmission, always check that your cable really is a null-modem one.

Since the clock is internal (no external clock exists within the RS232 specifications), you will have to set it in software and it should match on both communicating end. That is, if you have a clock speed of 9600 Hz on your microcontroller, it should be set to precisely 9600 Hz on the PC. There is always some margin but remember that the closer the better! Transmission speed is expressed in bauds which is a measure of communication bits per second (it is a bit different to data bits per second because each bytes of data is appended with a few extra communication bits). Because of capacitance effect of the cables, the speed is usually limited by the length of the cable. For instance, a 9600 bauds speed allows you to have wires up to 150 yards but a speed of 19200 bauds makes it down to only 15 yards. Technically speaking, you might go up to one million bauds with common microcontrollers but the MAX232 circuit is limited to 115200 bauds (about 10,000 bytes/s). If you are using a converter that has an USB cable, you can increase the cable length because USB is much less sensitive to capacitance effects.

3 Implementation

All the code given here is based on a PIC16F688 microcontroller native RS232 functionalities. The code was written in C and compiled with the HITECH PICCLITE compiler. The PC software was developed under Windows 7 and compiled with Microsoft Visual Studio 2010 Enterprise. I take the occasion to remind students and teachers that they have access to these compilers for free through the Microsoft DreamSpark and the Microchip Student platforms. Other people may download a demo version of the PICCLITE compiler and a light version of the Visual Studio IDE known as Visual Studio Express.

Because the total implementation is relatively complex, I will present it gradually with increasing level of complexity. Don't be afraid to see several implementations and always use the one you are comfortable with!

3.1 Basic features

Before building a communication protocol, we should first ensure that we are able to exchange data with the microcontroller. I always use this to be sure that the circuit is working before moving to more advanced systems.

3.1.1 Microcontroller side

We should first initialize the microcontroller EUSART:

void io_init(void) { SYNC = 0; TX9D = 0; TX9 = 0; RX9 = 0; SPEN = 1; BRGH = 1; SPBRG = 10; TXEN = 0; CREN = 1; TXIE = 0; RCIE = 0; TRISC5 = 1; TRISC4 = 1; }

In the order of apparition, we first activate the 8 bits asynchronous mode that we then configure to run at 115200 bauds through the SPBRG register and BRGH bit. Reception is activated through the CREN bit and the interruptions are disabled. Finally, the TX and RX pins of the PIC are placed in input mode, as recommended by the manufacturer. Always pay attention to the TRISC5/4 lines to match the PIC you are using (these pins may change from one PIC model to the other).

Sending a byte is relatively easy:

void io_putc(unsigned char c) { TXEN = 1; TXREG = c; while(!TXIF); TXIF = 0; }

After setting up the TXEN bit, the microcontroller will wait for the TXREG to be written to. Once the register has been changed, the byte is automatically sent. The microcontroller has finished sending the byte when the TXIF bit is set to 1. You should always wait the TXIF bit before sending a second byte.

Receiving data is a bit more complex:

unsigned char io_getc(void) { unsigned char r; while(!RCIF); RCIF = 0; r = RCREG; if(OERR) { CREN = 0; CREN = 1; } return r; }

Data is available for reading in the RCREG register when the RCIF bit is set to 1. If an error occurs, it is signalled by the OERR bit. Error clearance is reset by disabling and enabling reception again such as suggested by the manufacturer.

Finally, we may write a simple program to send the "Hello" string to the computer in loops:

#include <pic.h> /* definition above */ void io_init(void) ... void io_putc(unsigned char c) ... void main(void) { /* required for the PIC16F688 */ CMCON0 = 0b00000111; ADCON0 = 0; ANSEL = 0; io_init(); while(1) { io_putc('H'); io_putc('e'); io_putc('l'); io_putc('l'); io_putc('o'); io_putc('\r'); io_putc('\n'); } }

The \r\n chars allows for a line feed and carriage return; it is the common way of having a new line in Windows. If you run Windows HyperTerminal with the correct settings, you should read a lot of "Hello" lines.

Once transmission of data is functional, we can check for reception by continually reading data and sending it back to the PC (echo mode):

#include <pic.h> /* definition above */ void io_init(void) ... void io_putc(unsigned char c) ... unsigned char io_getc(void) ... void main(void) { /* required for the PIC16F688 */ CMCON0 = 0b00000111; ADCON0 = 0; ANSEL = 0; io_init(); while(1) { unsigned char c = io_getc(); io_putc(c); } }

The io_getc function has a blocking effect because it will loop forever until the RCIF flag is set to 1 to notify incoming data.

This is problematic when the main loop is in charge of other operations since everything will get to a halt until some data is received through the RS232 protocol. A good way to prevent this is to enable interruptions:

void io_init(void) { ... TXIE = 0; RCIE = 1; PEIE = 1; }

And the program becomes:

#include <pic.h> /* definition above */ void io_init(void) ... void io_putc(unsigned char c) ... unsigned char io_getc(void) ... void interrupt ctrl(void) { if(RCIF) { unsigned char c = io_getc(); io_putc(c); } } void main(void) { /* required for the PIC16F688 */ CMCON0 = 0b00000111; ADCON0 = 0; ANSEL = 0; io_init(); GIE = 1; while(1) { } }

Do not forget to enable interruptions through the PEIE and GIE bits. This time, the main loop is empty but the microcontroller will jump to the ctrl function when an interruption (such as RCIF) is triggered.

3.1.2 PC Side

Now that we are able to communicate through a HyperTerminal, it would be great to have our own software to communicate with the microcontroller. To make it easier to re-use, I have wrapped everything into a class object but it's not compulsory if you are not comfortable with Object-Oriented Programming.

Since Windows NT (this includes XP, Vista and 7 which are all derived from the NT kernel), any data exchange with drivers are programmed as file accesses. To communicate with the RS232 hardware, we actually have to communicate with its driver which will access the electronics of the computer for us. Just open a file named with the COM port number and write data to it!

/* fancy types I cannot live without...q */ #define null 0 typedef unsigned char byte; typedef unsigned short word; typedef unsigned long dword; class SerialComm { public: SerialComm(void) { this->m_hSerialPort = INVALID_HANDLE_VALUE; } ~SerialComm(void) { close(); } bool open(wchar_t *pwszSerialPort, dword dwBaudRate=19200) ... void close(void) ... dword write(const byte *pBuffer, dword dwBufferSize) ... dword read(byte *pBuffer, dword dwBufferSize) ... private: void flush(void) ... HANDLE m_hSerialPort; };

The class stores a handle to the driver in a pointer m_hSerialPort. When the class is initialized, the handle is set to a value which means that it is invalid (which is right because we haven't opened any communication channel yet). When the class closes, we will call a close function to tell the driver we are done with it. Reading and writing will be done through the read and write functions.

Let's begin with these latter two:

dword write(const byte *pBuffer, dword dwBufferSize) { if(this->m_hSerialPort == INVALID_HANDLE_VALUE) return 0; dword dwBytesWritten; if(!WriteFile(this->m_hSerialPort, pBuffer, dwBufferSize, &dwBytesWritten, null)) return 0; flush(); return dwBytesWritten; } dword read(byte *pBuffer, dword dwBufferSize) { if(this->m_hSerialPort == INVALID_HANDLE_VALUE) return 0; dword dwBytesRead; if(!ReadFile(this->m_hSerialPort, pBuffer, dwBufferSize, &dwBytesRead, null)) return 0; flush(); return dwBytesRead; }

In both cases, we are simply wrapping Windows function to read and write from/to files but we add a few lines to check that the handle is good and to cleanup things we a flush function. The flush function more or less have the same role has the OERR lines on the microcontroller side. I have defined it in the private part of the class:

private: void flush(void) { if(this->m_hSerialPort == INVALID_HANDLE_VALUE) return; dword dwError; if(ClearCommError(this->m_hSerialPort, &dwError, NULL) && dwError == CE_BREAK) ClearCommBreak(this->m_hSerialPort); }

Proper closing is without commentary:

void close(void) { if(this->m_hSerialPort != INVALID_HANDLE_VALUE) CloseHandle(this->m_hSerialPort); this->m_hSerialPort = INVALID_HANDLE_VALUE; }

All that is left is the open function. It's a bit longer but not that complex:

bool open(wchar_t *pwszSerialPort, dword dwBaudRate=19200) { close(); this->m_hSerialPort = CreateFileW(pwszSerialPort, GENERIC_READ | GENERIC_WRITE, 0, null, OPEN_EXISTING, 0, null); if(this->m_hSerialPort == INVALID_HANDLE_VALUE) return false; DCB dcb; GetCommState(this->m_hSerialPort, &dcb); dcb.DCBlength = sizeof(DCB); dcb.BaudRate = dwBaudRate; dcb.fBinary = TRUE; dcb.fParity = FALSE; dcb.fAbortOnError = FALSE; dcb.StopBits = ONESTOPBIT; dcb.fOutxCtsFlow = FALSE; dcb.fOutxDsrFlow = FALSE; dcb.fDtrControl = DTR_CONTROL_DISABLE; dcb.fRtsControl = RTS_CONTROL_DISABLE; dcb.fDsrSensitivity = FALSE; dcb.fOutX = 0; dcb.fInX = 0; dcb.fNull = FALSE; dcb.ByteSize = 8; dcb.Parity = NOPARITY; if(!SetCommState(this->m_hSerialPort, &dcb)) { close(); return false; } return true; }

First we close any opened channels (in the case we would have called the open function twice or more without explicitly calling the close function - Windows do not like when we leave communication channels opened forever; this would be a resource leak). Then we open a communication channel to the specified port (such as "COM3"). We require access to read (GENERIC_READ) and write (GENERIC_WRITE) to the file and we also require it to exist (OPEN_EXISTING). If everything is successful, the handle will be a valid value. In any other cases (port not existing or already in use), the output will be INVALID_HANDLE_VALUE.

Once opened, we have to configure the port according to our settings. We begin by retrieving the actual port state through GetCommState and set our values (no parity bit, one stop bit, no flow control and the bauds speed). We then update the port state through SetCommState. If anything goes bad, we close the port and return a false value.

We may then imagine the following program which outputs anything read on the port (to be used with the "Hello" example):

#include <stdio.h> #include <windows.h> /* definition above */ class SerialComm ... void main(void) { SerialComm com; if(!com.open(L"COM3", 115200)) { printf("Unable to open COM port!\r\n"); return; } byte buffer[128]; dword dwNumBytesRead; while(true) { dwNumBytesRead = com.read(buffer, sizeof(buffer) - 1); buffer[dwNumBytesRead] = '\0'; printf("%s", buffer); } }

This program opens the COM3 port at 115200 bauds and prints all incoming chars. The \0 thing is required by the printf function which will run crazy if we don't tell it where to stop printing. The data received is stored in a buffer of 128 bytes and the number of bytes read is returned by the read function. Always pay attention that there is no guarantee that the read function will always return 128 bytes! So always watch the actual number of bytes read. This is usually one of the first pitfalls of RS232 programming.

You may finally wonder why we do not call the close function explicitly. There is actually no need to do it in our small example because when the main function closes, the destructor of the SerialComm class will be automatically called. If you watch closely our class definition, you will see that the destructor explicitly calls the close function for us. This is a good example of what RIIA (a programming idiom which stands for "Resource Initialization Is Acquisition") can do for you!

3.2 Exchanging formal data

We are now able to exchange data between the microcontroller and the PC. It's a good start but we are far from having a real impact on the circuit we have connected. Our goal is, most of the time, to make a series of measurements with the circuit and to send the result back to the computer or to wait for the computer to tell what to do next. Still, as for now, we have restricted ourselves in exchanging bits of text.

Obviously, nothing will prevent us from sending the measurements as text. For instance, we may imagine reading a temperature and sending back the information to the computer as a string such as: "Temperature: 125.3°C".

Even if this way of doing works, we will encounter two major issues:

1/ The string "Temperature: 125.3°C" makes really bad usage of bandwidth because we are using about 22 chars to transmit a data that, on the microcontroller side, take only one or two bytes! Even if we were transmitting only "123.5" we would still use 5 bytes to represent it.

2/ The serial protocol is, unfortunately, not error-proof. It may happen that some data arrives corrupted. It would probably be harmless to read "Temp#rature: 125.3°C" but much more troublesome to have "Temperature: 1©5.3°C" and pretty disastrous to have "Temperature: 105.3°C". Imagine a rocket ignition circuit which would go crazy by reading erroneous data such as "blows up" when we actually sent "what's up"!

We then have to design a system that makes good usage of bandwidth and also ensure data integrity. Correcting erroneous data is possible but complicated, so we will focus here more on a data-rejection approach.

3.2.1 Designing a communication protocol

We will create a communication standard containing several key information allowing data integrity and compactness of representation. These kinds of protocols are frequently met in IT and have various degrees of complexity. We will make our own protocol here with specifications adapted to our problem.

Any outputted data shall be sent as a series of bit representing the information as a packet. To be able to rebuild the data on the reception side, we will require two key features:

1/ When does the data packet begins. This is commonly implemented by always starting a packet with a known identified. In our previous example the identifier was "Temperature: ".

2/ How long is the packet. This information might be implicit such as when we know the packet size is always the same.

To assert data integrity once received, we also need a field that is a known mathematical function of the packet such that we can recompute the result on the received data and check if it matches the value sent along with the packet. We will refer to this as the checksum procedure.

I will now focus on packets that have always the same size, begin with an identifier and also take a checksum value. The previous temperature example would then translate in the form of structure as:

struct header_s { uint32 ident; uint16 checksum; }; struct packet_s { struct header_s hdr; uint16 temperature; };

In our example, we are using a 32-bits identification field and a 16 bits checksum. We could also have used an 8-bits identifier:

struct header_s { uint8 ident; uint16 checksum; };

Or even an 8-bits checksum:

struct header_s { uint8 ident; uint8 checksum; };

In the former approach our header (identifier plus checksum) was taking 6 bytes but only 2 in the latter case. The former is more secure than the latter but also slower because it takes more data to send. The choice is always dependent on the application specification: are you reading temperature a hundred times per second or are you sending an ignition command to your rocket?

3.2.2 Microcontroller side

We will create a packet using an 8-bit identifier and an 8-bit checksum for header. The packet also contains timing information and two 16-bits values for analog reading:

struct header_s { unsigned char ident; unsigned char checksum; }; struct packet_s { struct header_s hdr; unsigned char clock; unsigned short ch1; unsigned short ch2; };

The program then sends the value of the analog channels at regular intervals:

#include <pic.h> #define VERSION_IDENT 0x2A unsigned char g_ubClock = 0, g_ubSend = 0; /* definitions above */ void io_init(void) ... void io_putc(unsigned char c) ... void io_write(unsigned char *pBuffer, unsigned char nSize) ... unsigned char checksum8(unsigned char *pData, unsigned char bLength) ... unsigned short analog_read(void) ... void tmr16_init(void) { TMR1ON = 1; T1OSCEN = 0; TMR1CS = 0; TMR1IE = 1; T1CKPS1 = 0; T1CKPS0 = 0; TMR1L = 0; TMR1H = 0; } void interrupt ctrl(void) { if(TMR1IF) { TMR1IF = 0; g_ubSend = 1; g_ubClock ++; TMR1H = 0x3c; TMR1L = 0xaf; } } void main(void) { CMCON0 = 0b00000111; ADCON0 = 0; ANSEL = 0; io_init(); tmr16_init(); GIE = 1; PEIE = 1; while(1) { while(!g_ubSend); struct packet_s pk; pk.hdr.ident = VERSION_IDENT; pk.hdr.checksum = 0; pk.clock = g_ubClock; pk.ch1 = analog_read(1); pk.ch2 = analog_read(2); pk.hdr.checksum = checksum8((unsigned char*)&pk, sizeof(pk)); io_write((unsigned char*)&pk, sizeof(pk)); g_ubSend = 0; } }

Every time the TMR1 register overflows (that is about every 10 ms with a 20 MHz quartz), it sets the flag g_ubSend to 1 which allows the main loop to read the analog channels and send the result to the computer. Even if it looks complicated, all the main function does is waiting for g_ubSend and fill the packet with data.

The write function is only successive calls to io_putc:

void io_write(unsigned char *pBuffer, unsigned char nSize) { while(nSize--) io_putc(*(pBuffer++)); }

The checksum function is a bit more interesting. I have derived it from the TCP/IPv4 norm of the RFC793. It returns the 2-complement of the sum of the 2-complements of every byte in the packet:

unsigned char checksum8(unsigned char *pData, unsigned char bLength) { unsigned char sum = 0; while(bLength--) { sum += ~(*pData); pData ++; } return ~sum; }

Don't forget to put the checksum field to 0 before calling the checksum function or the result will not be reproducible on the PC.

3.2.3 Computer side

On the computer side, we now have to read the information. For more clarity, I have packed everything in a Protocol class:

#pragma pack(push) #pragma pack(1) struct header_s { byte ident; byte checksum; }; struct packet_s { struct header_s hdr; byte clock; word ch1; word ch2; }; #pragma pack(pop) #define VERSION_IDENT 0x2A class Protocol { public: Protocol(dword dwBufferSize=1024) { this->m_dwMaxBuffer = dwBufferSize; this->m_dwCurrBufferPos = 0; this->m_pBuffer = new byte[dwBufferSize]; } ~Protocol(void) { delete[] this->m_pBuffer; this->m_pBuffer = null; } byte checksum8(const byte *pBytes, dword dwSize) const ... void onRead(byte *pBuffer, dword dwSize) ... private: bool processBuffer(void) ... void swapBuffer(dword dwBytes) ... byte *m_pBuffer; dword m_dwMaxBuffer, m_dwCurrBufferPos; };

The class allocates a buffer memory that is required for the reception of data (it is of 1 kb by default but you can tune it depending on your application). Again, the destructor of the class will manage everything and free the memory allocated.

The checksum function is the same as on the microcontroller side:

byte checksum8(const byte *pBytes, dword dwSize) const { register byte bSum = 0; while(dwSize--) { bSum += ~(*pBytes); pBytes ++; } return ~bSum; }

And the dirty things happen now! When a data is read, it should be sent to the onRead function of the Protocol class which will add it to the buffer memory. After each addition, the processBuffer function will be called to check if there is any data to be processed:

void onRead(byte *pBuffer, dword dwSize) { if(this->m_dwCurrBufferPos + dwSize >= this->m_dwMaxBuffer) return; memcpy(this->m_pBuffer + this->m_dwCurrBufferPos, pBuffer, dwSize); this->m_dwCurrBufferPos += dwSize; while(processBuffer()); }

The processBuffer function first checks if there are enough bytes in the buffer to have a complete packet. If not, it returns and waits for new data addition.

If there is enough data to potentially make a packet, it will read the first byte(s) to check the identification field. If the identification field does not match the beginning of a packet, we may have ended up in the middle of a packet or with erroneous data. We then skip the first byte and try again the whole procedure until we have caught the beginning of a packet.

Then, we store the value of the packet checksum and compute the checksum again (don't forget to set the field to 0 to make it reproducible). If the checksum does not match, it means that something wrong happened with the packet and so we can discard the whole packet and wait for new data. There is always a small risk that the identification field do not catch the actual start of a packet (there is a 1/255 odds with 8-bits identification) but only some data inside a packet; but in that case, the checksum will reveal that out too. Still, there is a 1/65535 odds that we have incorrect data with valid checksum and valid identification... If you are not happy with it, increase the identification and checksum fields bit sizes.

bool processBuffer(void) { if(this->m_dwCurrBufferPos < sizeof(struct packet_s)) return false; struct packet_s *pPacket = (struct packet_s*)this->m_pBuffer; if(pPacket->hdr.ident != VERSION_IDENT) { swapBuffer(1); return true; } byte bChecksum = pPacket->hdr.checksum; pPacket->hdr.checksum = 0; if(checksum8(this->m_pBuffer, dwSize) == bChecksum) { /* Packet OK */ printf("%d: %d %d\r\n", pPacket->clock, pPacket->ch1, pPacket->ch2); } swapBuffer(sizeof(struct packet_s)); return ret; }

The swapBuffer function is easy to implement:

void swapBuffer(dword dwBytes) { if(dwBytes > this->m_dwCurrBufferPos) { this->m_dwCurrBufferPos = 0; return; } for(dword dwI=dwBytes;dwI<this->m_dwCurrBufferPos;dwI++) this->m_pBuffer[dwI-dwBytes] = this->m_pBuffer[dwI]; this->m_dwCurrBufferPos -= dwBytes; }

We may then create a program that prints the two analog values sent by the PIC:

#include <stdio.h> #include <windows.h> /* definitions above */ class SerialComm ... class Protocol ... void main(void) { SerialComm com; Protocol proto; if(!com.open(L"COM3", 115200)) { printf("Unable to open COM port!\r\n"); return; } byte buffer[128]; dword dwNumBytesRead; while(true) { dwNumBytesRead = com.read(buffer, sizeof(buffer)); proto.onRead(buffer, dwNumBytesRead); } }
3.3 Final version

We are now able to receive data from a microcontroller at high speeds, to ensure their integrity and, basically, nothing would stop us to adapt the reception code on the microcontroller (in a somewhat simplified version). The microcontroller is relatively easy to modify but we will still encounter a few surprises on the computer side...

When used as-is, Windows ReadFile function as an annoying property: it has blocking features. Just like on the very first implementation on the microcontroller with io_getc, Windows will stall until new data arrives. This is without consequence when we are just reading or writing data but is quite problematic when we wish to read <u>and[/u] write data from the computer.

Hopefully, Windows come with a solution in the form of asynchronous reading and writing: Overlapped I/O. The ReadFile function no longer stall the program but it comes at the price of added complexity. I have deliberately left this part for the end because it is much harder to grasp... But don't panic, all the code is here!

3.3.1 Microcontroller side

As said previously, the microcontroller side is just an adaptation of what we have done on the computer except that we have to get things working with much less RAM:

... #define VERSION_IDENT_IN 0x1B struct packet_in_s { struct header_s hdr; unsigned short data; }; unsigned char g_ucBuffer[20], g_nBufferLength = 0; unsigned short g_usAnalogOut = 0; void append(unsigned char c) { if(g_nBufferLength >= sizeof(g_ucBuffer)) return; g_ucBuffer[g_nBufferLength] = c; g_nBufferLength ++; } void swap(unsigned char nBytes) { if(nBytes >= g_nBufferLength) { g_nBufferLength = 0; return; } for(unsigned char i=nBytes;i<=g_nBufferLength;i++) g_ucBuffer[i-nBytes] = g_ucBuffer\[i\]; g_nBufferLength -= nBytes; } char process(void) { const int pksize = sizeof(struct packet_in_s); if(g_nBufferLength < pksize) return 0; struct packet_in_s *pk = (struct packet_in_s*)g_ucBuffer; if(pk->hdr.ident != VERSION_IDENT_IN) { swap(1); return 1; } unsigned short cksum = pk->hdr.checksum; pk->hdr.checksum = 0; if(checksum8((unsigned char*)pk, pksize) == cksum) { /* Packet OK */ g_usAnalogOut = pk->data; } swap(pksize); return 1; } void interrupt ctrl(void) { if(RCIF) { append(io_getc()); while(process()); } ... }
3.3.2 Computer side

On the computer side, there are much more modifications to do. First, we will have to activate the Overlapped I/O mode and create a separate thread which will read the data received.

We then begin by adding a handle to the thread to our SerialComm class:

DWORD WINAPI asyncRead(LPVOID lpParam); class SerialComm { public: SerialComm(void) { this->m_dwThreadID = 0; this->m_hThread = INVALID_HANDLE_VALUE; this->m_hSerialPort = INVALID_HANDLE_VALUE; this->m_bForceQuit = false; memset(&this->m_sOverlappedWrite, 0, sizeof(m_sOverlappedWrite)); } bool open(wchar_t *pwszSerialPort, dword dwBaudRate=19200, bool bAsync=false) ... void close(void) ... virtual void onRead(byte *pBuffer, dword dwSize) = 0; protected: friend DWORD WINAPI asyncRead(LPVOID lpParam); void asyncRead(void) ... private: OVERLAPPED m_sOverlappedWrite; HANDLE m_hSerialPort, m_hThread; dword m_dwThreadID; volatile bool m_bForceQuit; bool m_bAsyncMode; };

And we shouldn't forget to stop the thread when the close function is called:

void close(void) { this->m_bForceQuit = true; if(this->m_hThread != INVALID_HANDLE_VALUE) { WaitForSingleObject(this->m_hThread, 1000); CloseHandle(this->m_hThread); } this->m_hThread = INVALID_HANDLE_VALUE; this->m_dwThreadID = 0; if(this->m_hSerialPort != INVALID_HANDLE_VALUE) { if(this->m_bAsyncMode) CancelIoEx(this->m_hSerialPort, &this->m_sOverlappedWrite); CloseHandle(this->m_hSerialPort); } this->m_hSerialPort = INVALID_HANDLE_VALUE; }

The function WaitForSingleObject allows waiting for 1 second that the thread exits nicely before killing it. The function CancelIoEx cancels all pending operations on the driver. The role of m_bForceQuit will be highlighted later.

We then have to modify the read and write function:

dword write(const byte *pBuffer, dword dwBufferSize) { dword dwBytesWritten; if(!WriteFile(this->m_hSerialPort, pBuffer, dwBufferSize, &dwBytesWritten, this->m_bAsyncMode ? &this->m_sOverlappedWrite : null)) return 0; flush(); return dwBytesWritten; } dword read(byte *pBuffer, dword dwBufferSize) { /* disabled when Overlapped I/O is enabled */ if(this->m_bAsyncMode) return 0; dword dwBytesRead; if(!ReadFile(this->m_hSerialPort, pBuffer, dwBufferSize, &dwBytesRead, null)) return 0; flush(); return dwBytesRead; }

The open function will have to be modified to create a thread and ask for the Overlapped I/O mode (FILE_FLAG_OVERLAPPED):

bool open(wchar_t *pwszSerialPort, dword dwBaudRate=19200, bool bAsync=false) { close(); this->m_bAsyncMode = bAsync; this->m_hSerialPort = CreateFileW(pwszSerialPort, GENERIC_READ | GENERIC_WRITE, 0, null, OPEN_EXISTING, bAsync ? FILE_FLAG_OVERLAPPED : 0, null); if(this->m_hSerialPort == INVALID_HANDLE_VALUE) return false; if(bAsync) { this->m_bForceQuit = false; this->m_hThread = CreateThread(null, 0, ::asyncRead, this, 0, &this->m_dwThreadID); if(this->m_hThread == INVALID_HANDLE_VALUE) { close(); return false; } } DCB dcb; ... }

The rest of the function remains unchanged and was not copied here. The thread created is sent to the entry point asyncRead with a pointer to the current SerialComm object which allows calling the asyncRead function inside the object. It might look weird but it's the only way for Windows thread to have entry function inside a class:

DWORD WINAPI asyncRead(LPVOID lpParam) { SerialComm *pRS232 = (SerialComm*)lpParam; if(pRS232 != null) pRS232->asyncRead(); return 0; }

With the latter function:

void asyncRead(void) { byte buffer[128]; dword dwBytesRead; OVERLAPPED overlapped; memset(&overlapped, 0, sizeof(overlapped)); overlapped.hEvent = CreateEvent(null, FALSE, FALSE, null); while(!this->m_bForceQuit) { ReadFile(this->m_hSerialPort, buffer, sizeof(buffer), null, &overlapped); while(!this->m_bForceQuit && WaitForSingleObject(overlapped.hEvent, 100) != WAIT_OBJECT_0); if(this->m_bForceQuit) break; if(GetOverlappedResult(this->m_hSerialPort, &overlapped, &dwBytesRead, FALSE)) onRead(buffer, dwBytesRead); flush(); } CancelIoEx(this->m_hSerialPort, &overlapped); CloseHandle(overlapped.hEvent); }

It first creates a 128 bytes buffer and some structure required for Overlapped I/O. It then creates an event which will be used to trigger data reception. We then call ReadFile which no longer blocks and so we have to use the event to check if data is available. We wait data for only 100 ms so we can use the m_bForceQuit flag to force the thread to exit. Most electronic enthusiast will probably be starring at this code with big eyes but don't worry if you don't understand it completely; it's the result of several optimizations I wrote.

To process data we then call the pure virtual function onRead which has to be defined in children of SerialComm:

class SerialComm { public: virtual void onRead(byte *pBuffer, dword dwSize) = 0; ... }; class Protocol : public SerialComm { public: virtual void onRead(byte *pBuffer, dword dwSize) ... ... };

This is not of the best practice in terms of design patterns but it's the easiest way to make it working. I will show you a more advanced implementation later.

Finally, the main function of our program becomes:

... void main(void) { Protocol com; if(com.open(L"COM3", 115200, true)) { com.send(0); Sleep(1000); com.send(4095); Sleep(5000); com.close(); } else printf("Unable to open COM port!\r\n"); }

Without forgetting the Protocol class:

#define VERSION_IDENT_OUT 0x1B #pragma pack(push) #pragma pack(1) struct packet_out_s { struct header_s hdr; word data; }; #pragma pack(pop) class Protocol : public SerialComm { public: void send(word wValue) { struct packet_out_s packet; packet.hdr.ident = VERSION_IDENT_OUT; packet.hdr.checksum = 0; packet.data = wValue; packet.hdr.checksum = checksum8((byte*)&packet, sizeof(packet)); write((byte*)&packet, sizeof(packet)); } ... };

The program simply opens the COM3 port in asynchronous mode, send the value 0, wait 1 second and send the value 4095. The program then waits for 5 more seconds and closes the stream. Remember that the program has no longer blocking features and if you don't tell it when to stops it will close immediately. For example, the following code will return immediately:

... /* Warning : bad code !!! */ void main(void) { Protocol com; if(com.open(L"COM3", 115200, true)) { com.send(0); } else printf("Unable to open COM port!\r\n"); }
3.4 Miscellaneous tricks

Before concluding, I would like to discuss a few tricks often met with serial protocols and which do not fit in the previous sections.

3.4.1 List all serial ports on a computer

So far, we have taken for granted that we knew the port number to use. This is often not the case, especially when there are several serial devices plugged on the same computer. It may therefore be interesting to be able to list all the port on the local computer.

Hopefully, this list is readily available in the Windows registries and can be accessed that way:

typedef void (*pfnEnumerateSerial)(int iIndex, wchar_t *pwszName, wchar_t *pwszValue, wchar_t *pwszPort, void *pParam, dword dwParamSize); class SerialComm { public: static bool isSerialPortAvailable(const wchar_t *pwszPort) { HANDLE hFile = CreateFileW(pwszPort, GENERIC_READ | GENERIC_WRITE, 0, null, OPEN_EXISTING, 0, null); if(hFile == INVALID_HANDLE_VALUE) return false; CloseHandle(hFile); return true; } static void enumerateSerialPorts(const pfnEnumerateSerial pfnCallback, void *pParam, dword dwParamSize) { if(pfnCallback == null) return; HKEY hKey = null; do { if(RegCreateKeyExW(HKEY_LOCAL_MACHINE, L"HARDWARE\\DEVICEMAP\\SERIALCOMM", 0, null, REG_OPTION_NON_VOLATILE, KEY_READ, null, &hKey, null) != ERROR_SUCCESS) break; wchar_t wszName[128], wszValue[128], wszPort[140]; int i = 0; while(true) { dword dwNameSize = sizeof(wszName); dword dwValueSize = sizeof(wszValue); memset(wszName, 0, sizeof(wszName)); memset(wszValue, 0, sizeof(wszValue)); if(RegEnumValueW(hKey, i, wszName, &dwNameSize, 0, 0, (BYTE*)wszValue, &dwValueSize) != ERROR_SUCCESS) break; wsprintf(wszPort, L"\\\\.\\%s", wszValue); if(isSerialPortAvailable(wszPort)) pfnCallback(i, wszName, wszValue, wszPort, pParam, dwParamSize); i ++; } } while(false); if(hKey != null) RegCloseKey(hKey); } ... };

The enumerateSerialPorts function opens the registry key HKEY_LOCAL_MACHINE\HARDWARE\DEVICEMAP\SERIALCOMM and lists its content. Every entry is then checked with isSerialPortAvailable (to check for already opened ports) and the positive results sent to a callback function pfnEnumerateSerial. It is important to restrict the request to read only mode (KEY_READ) as a generic access (KEY_ALL_ACCESS for example) might be rejected by Windows UAC if the user do not have the rights to access the key in other modes.

Since the function was defined static, it can be accessed that way:

... void callback(int iIndex, wchar_t *pwszName, wchar_t *pwszValue, wchar_t *pwszPort, void *pParam, dword dwParamSize) { wprintf(L"%s : %s (%s)\r\n", pwszName, pwszValue, pwszPort); } void main(void) { SerialComm :: enumerateSerialPorts(callback, null, 0); }
3.4.2 Asynchronous mode with callbacks

When working with asynchronous I/O, it is possible to use the ReadFileEx function which allows using a callback function once data are read. It might then be tempting to use that functionality rather than the thread we have been using.

However, the callback function can be called only if the process is placed in an alert state through a specific function such as SleepEx. Since that mechanism is not fully under our control and make the communication code dependent on the application, we should restrict from using this functionality and rely on the thread version developed here.

3.4.3 Data read-out and timing issues

It might also be tempting to use the computer clock to tell the exact moment a data has been received rather than adding a field to the packet sent such as we did in a previous example. This is unfortunately not possible for two major reasons:

1/ Windows do not guarantee that data are presented at the exact time they are received. Sometimes, the system will idle after incoming data.

2/ We usually receive data as burst containing several packets because of the buffer memory. Still, even by reducing the buffer size we can fall in the previous issue.

3.4.4 A few words about a multi-protocol version

Earlier I have mentioned that the present design was not perfect with the Protocol class inheriting from the SerialComm class. This is because a protocol and a communication channel are two different things. By acknowledging this, we can design a more robust implementation which allows several protocols to be used on the same communication channel. I am only giving here some pseudo-code and I will let you write your own final version of it:

class Protocol { Protocol(void) { this->m_pOwner = null; } ... protected: virtual bool processBuffer(void) = 0; SerialComm *getOwner(void) { return this->m_pOwner; } private: friend class SerialComm; bool setOwner(SerialComm *pOwner) { if(pOwner != null && this->m_pOwner != null) return false; this->m_pOwner = pOwner; return true; } SerialComm m_pOwner; };

Every protocol object has one and only one SerialComm object as source but a SerialComm object may have a list of several protocols attached to it:

class SerialComm { public: bool registerProtocol(Protocol *pProtocol) { if(pProtocol == null || !pProtocol->setOwner(this)) return false; this->m_sProtocols.add(pProtocol); return true; } void unregisterProtocol(Protocol *pProtocol) { if(pProtocol == null) return; pProtocol->setOwner(null); this->m_sProtocols.remove(pProtocol); } ... private: LinkedList<Protocol*> m_sProtocols; };

By changing the SerialComm onRead function to a dispatch function:

void onRead(byte *pBuffer, dword dwSize) { AutoPtr< Iterator<Protocol*> > it = this->m_sProtocols; while(!it->end()) { Protocol *curr = it->current(); it->next(); if(curr != null) curr->onRead(pBuffer, dwSize); } }

And having a formal definition of proccessBuffer for every child of the Protocol interface:

class Protocol1 : public Protocol { protected: virtual bool processBuffer(void) { ... } }; class Protocol2 : public Protocol { protected: virtual bool processBuffer(void) { ... } };

One could then only write:

... void main(void) { Protocol1 proto1; Protocol2 proto2; SerialComm com; com.registerProtocol(&proto1); com.registerProtocol(&proto2); ... }
3.4.5 Ports above "COM9

Following the Microsoft Developper Network, to access a port above " COM9 " (such as " COM10 ") you should prefix the name with "\\.\". That way, COM10 should then be accessed as "\\.\COM10". You may, of course, do the same for other ports such as "\\.\COM3".

3.4.6 The pragma pack issue

#pragma pack commands often pop ups in my code and I did not give any explanation about that. It is a command sent to the compiler to tell how data should be packed. By using #pragma pack(1) we tell the compiler to align the data at the byte level. By default, Visual Studio packs everything at the dword level while the PICCLITE does it at the byte level. If you don't pay attention to this, the packet structure of the computer and the packet structure of the PIC will not be the same and communication will fail.

So don't forget to write:

#pragma pack(push) #pragma pack(1) struct ... #pragma pack(pop)
3.4.7 Time-outs

So far our program performs well as long as the data keep coming. On the other hand, if we write a PIC program that sends only a few data spaced by long intervals, nothing arrives on the computer until the buffer memory gets full. The work-around for this is to tell the driver we would like to receive data not only when the buffer is full but also after some time-out delay.

I have chosen here to set the time out period to 100 ms. All we have to do is to modify the open function:

this->m_hSerialPort = CreateFileW(pwszSerialPort, GENERIC_READ | GENERIC_WRITE, 0, null, OPEN_EXISTING, bAsync ? FILE_FLAG_OVERLAPPED : 0, null); if(this->m_hSerialPort == INVALID_HANDLE_VALUE) return false; COMMTIMEOUTS timeouts; timeouts.ReadIntervalTimeout = 100; timeouts.ReadTotalTimeoutConstant = 0; timeouts.ReadTotalTimeoutMultiplier = 0; timeouts.WriteTotalTimeoutConstant = 0; timeouts.WriteTotalTimeoutMultiplier = 0; if(!SetCommTimeouts(this->m_hSerialPort, &timeouts)) { close(); return false; }
3.4.8 Buffered sending

Let us imagine a microcontroller which sends 10-bits analog data to the computer at regular intervals that we would like to use as a fast data logger. With our system, we know that a packet should contains an 8-bits identification field, an 8-bits checksum, a 10-bits data field and some clock field to check for missing packets. To make the system as fast as possible, we will use a 6-bits clock field making the whole packet 4 bytes long.

Using a transfer speed of 115200 bauds, we know that we can send about 2600 packets per second. We have to add to this the time required to actually read the analog data using the microcontroller ADC. Let's say the microcontroller needs a 0.1 ms to read the data giving an overall execution speed of about 2 kHz. We may increase it by a little using the PIC interruptions for the ADC and send the data over the TX pin while we are reading the next analog value. This is possible because most of the time the microcontroller is idling. At best, we could increase the speed up to 2600 Hz which is the maximum packet density we may achieve.

But we can still make better by using buffered sending. If we pay attention to our packet structure, 50% of the packet contains only identification data. If we could send 2 analog values at a time, this would now be only 33% of the packet... The more data we send in one packet, the better we are using the bandwidth for actual data. Let's say, we can send one hundred analog values at a time. Since we know that each data is spaced by a known time interval, we only need one clock field per packet. So, in total, we have 100 * 10 bits (data) + 16 bits (header) + 8 bits (clock) which is 128 bytes. Using this 128 bytes packet, we could increase the average transfer speed to about 8.2 kHz! This is far better than the previous 2.6 kHz...

Still, we have a problem: we are getting burst of 100 data followed by a long silence and then again a burst of data. This is because we have to fill the buffer with analog values and send it only when the buffer is full.

There is, again, a trick which is to use two buffers: a front buffer and a back buffer. The main loop sends the back buffer over the communication channel while, at the same time (using the interruption mechanism), the PIC writes the front buffer with the analog data. When the front buffer is full, the PIC just swap the front and back buffer such that the system never stops. There is only one condition for this to work: the back buffer should be completely sent before the front buffer gets full. Put differently, the acquisition speed must be slightly lower than the sending speed. If we are sending data at 8.2 kHz (measured), we should set the acquisition timer to 8 kHz and no more.

The number of data we can pack in one packet is limited by the available RAM of the PIC. But don't forget that you need to store 2 buffers and not one so a PIC with 256 bytes of RAM can only store a maximum of two 128 buffers.

4 Conclusions

In this post I hope to have convinced you that RS232 technology is far from being outdated and that it can still be used in many applications; even when speed and reliability is of high concerns. I have shown you several implementations of the RS232 technology on both a PIC microcontroller and a Windows-based PC. I have also introduced a few tricks to make your applications easier to write and safer to execute.

But this is only a beginning. This is only theory. The best part I have to tell you is that I have now been using this implementation for all my circuits and that it works really fine. If you watch my other posts about microcontroller sending data to computer, you will see the exact lines of code I have presented you here.

So, feel free to use this implementation for all your circuits!

[⇈] Top of Page

You may also like:

[»] Implementing Object Persistence in C++

[»] RSA Cryptography Implementation Tips

[»] Building a 5 kHz PIC-based Oscilloscope

[»] Generic 8th Order Sallen-Key Filter

[»] The Home-Scientist FAQ