get dynamic length packet from UART?

Question

I Write a program for receive data from Uart ISI, my Data Packet is something like :

2 first byte contain Device ID and type of command. Data is contain some bytes that could be have different length, minimum 1 byte to max 15 bytes, and the last byte represent the packet is finish.

Uart ISI code is:

volatile char UartFlag=Fulse;
volatile unsigned char count=0;
unsigned char coder[13];

ISR     (USART_RX_vect)
{
    coder[count]=UDR0;
    if (coder[count] == 20)
        UartFlag=True;
    else
        count++;
}

That receive each byte and save in coder. As you see when receive number 20, stop receiving process by UartFlag=True; and in main call a function DoSomthing(); to do something with coder as below:

while (1)
    {
        if (UartFlag)
        {
            DoSomthing();
            count=0;
            UartFlag=Fulse;
        }
    }

But I have problem that sometime the Data section have 20 in their bytes and its lead to i dont get Correct packet. I try to put \0 (the last byte=0 instead of 20) but maybe have same error.

I am looking for best way to find dynamic length of packet. What is the best way to find the dynamic length of packet?

A way that maybe work is put the data length in first byte of Data section, but it add extra process in sender device.

Change the finish character to something which won't appear in your data. Can you do that? — Whiskeyjack, Jun 30 '17 at 07:18
no because the packet data maybe contain all of character..and sender dont have limitation in send charachter — Ali, Jun 30 '17 at 07:19
How about this - Keep reading till you encounter stop bit. You don't need to put a special end character on your own. — Whiskeyjack, Jun 30 '17 at 07:23
Use multiple characters as an escape sequence, like ### or @#$%& like many BT or GPRS modules do when switching between command and data mode. — Bence Kaulics, Jun 30 '17 at 07:29
@BenceKaulics its cuse to add more than one bytes to packet...for example 3 packet..if theData have 1 of the characters as an escape sequence too, i think have same error, isnt right? — Ali, Jun 30 '17 at 07:31
@EugeneSh. Very useful question :)... after a year you ask the question are you find better way to handle frame "join" in the middle of an ongoing sent frame ? — Ali, Jul 01 '17 at 19:00
Usually similar applications have a fixed size header containing the data length. To add reliability to this pattern some add a checksum field to both the header and the data. Yes, it is adding some processing overhead, but in most cases it is not substantial relatively to the overall application. Combined with one of the techniques in the referenced question it will give you a robust protocol. — Eugene Sh., Jul 01 '17 at 23:36

score 2 · Accepted Answer · edited Jun 30 '17 at 07:42

2

In my opinion, to be 100% that it will work in all cases, you have to do what you thought of yourself:

Include the length of the data (the number of bytes you are sending) as an additional header field, for example after the Type field.

This is actually what the Ethernet protocols do.

About extra process on sender I don't know how the sender is implemented. But it doesn't seem to require much processing effort. I mean the sender already knows how many packets will be sent, no? So, as you fill in the other two fields (Device ID and Type), you can fill in this additional field as well.

edited Jun 30 '17 at 07:42

Ali

183
5
15

answered Jun 30 '17 at 07:24

nickagian

1,550
9
19

thanks.. i think you right ..best way is handle length when packet create..good luck – Ali Jun 30 '17 at 07:43
@combo_ci - generally you shouldn't be making trivial edits like this, and you especially should not be inserting large blocks of text. Where did that come from??? – Chris Stratton Jun 30 '17 at 14:46
Beware that simply sending a length is not reliable; you still must have a way to reliably distinguish a "header" encoding the length from any pattern that could occur in data, either by having a multi-byte sequence which is banned from data (possibly broken up by an escape if it occurs), less reliably by a time gap, or perhaps by some additional ("out of band") signalling wire or channel. (Starting the receiver before the transmitter would effectively be out-of-band signalling through the operator's fingers) – Chris Stratton Jun 30 '17 at 14:47
@ChrisStratton: you are partially right. But I have to say this large block was a comment from me that I subsequently deleted. – nickagian Jun 30 '17 at 15:17
@ChrisStratton: Unfortunately I don't quite get what you mean :-) The receiver will know that on every transmission the first three bytes coming in are special fields that need special interpretation and not data. From the fourth byte on there will be the data, with a length that was defined from the third special field. – nickagian Jun 30 '17 at 15:19
@nickagian - the problem is reliably and unambiguously detecting the "start" of "transmission" - this isn't necessarily much or any easier than detecting the end. Consider what happens if communication is established mid-packet, and something in the packet body happens to resemble a header. – Chris Stratton Jun 30 '17 at 15:48
@ChrisStratton Oh, I see what you mean. But isn't this the job of the communication protocol itself? There is this start bit that signals the start of a new transmission and then follows one data byte. What the user normally receives in the application level is only the data of each transmission. And this "protocol" that he uses here is implemented at the application level. Of course the sender should start transmitting only after the receiver has started listening, so that he doesn't miss the first transmission. Or maybe I haven't still understand your point :-) – nickagian Jun 30 '17 at 17:42
A UART typically only handles bytes individually, though in some cases you may be able to signal/detect a "break" condition. Generally speaking with a UART there is no "protocol" apart from whatever you choose to implement on top of it. And the challenge here is how to design an appropriate one. Assuming the receiver starts first is often problematic; as mentioned previously, it assumes "out of band signalling" through the human operator. Real systems typically need to recover or start in less restricted conditions, perhaps discarding some initial data until they recognize a packet start. – Chris Stratton Jun 30 '17 at 17:50
@ChrisStratton Thanks for your explanation, I understood it from you comments that: Whats my way to find start a transmission progress, How to find there is real packet or perhaps noice in line...if this, I have good solution for that. – Ali Jul 01 '17 at 04:03
@ChrisStratton you say: the problem is reliably and unambiguously detecting the "start" of "transmission" _ ...let me explian more: Assume that there is 5 or 6 devices exist in a line, the communicate with together in every 4 or 5 second.. I think _unambiguously detecting the "start" of "transmission is lower priority than Overlapping Data from each device..its mean each developer could design a pattern for find start data but find overlapping on received data in complicated – Ali Jul 01 '17 at 04:13

Curd · Answer 2 · 2017-06-30T07:54:28.903

2

The two common ways to solve this are the ones you mentioned.

Use a dedicated length field at the beginning of the package
Use an unambiguous end marker at the end of the package.

If you don't want to use length fields you have to use an end marker.
If you want to be able to use any of the 256 byte values in payload you have to extend your code by intrducing escape sequences in order to code more than just 256 distinct values.

E.g.

Use 0x20 0x00 to code original 0x20 in payload (i.e. replace any 0x20 by 0x20 0x00 in payload)
Use 0x20 0x01 as end marker.

Since escaping increases the size of the coded payload it is advisable to use an escape character value that is unlikely to appear in original payload data (so 0x20 wouldn't be a good choice if payload is ordinary ASCII text).

edited Jun 30 '17 at 07:54

answered Jun 30 '17 at 07:45

Curd

16,283
35
46

Note that a length fields requires reliable detection of a header vs. arbitrary patterns in data. This isn't necessarily any simpler than detecting an end marker. – Chris Stratton Jun 30 '17 at 14:49
Common protocols that use the escape character technique are SLIP and Asynchronous PPP. – Kevin White Jun 30 '17 at 19:34
@Curd: Unfortunately, I do not know at all what data from the sender of the oven, maybe 0x20! ( because the sender mix some bits together and each bit means an action...mixing the bits in some condition may deal to create 0x20 and its a big bug in our system – Ali Jul 01 '17 at 04:33

Michael Karas · Answer 3 · 2017-06-30T13:17:17.383

1

Another thing to consider is how the data field is encoded. You can avoid a length specifier in the packet by ensuring that the end marker is not ever expected in the data field. (Note that in applications that I create I also use a start marker that is unique and cannot occur in the data region. This allows quicker re-sync to the command stream without losing one extra packet). So if a data region is just numbers for example encode them in character format as opposed to binary format. Then also select your device ID and packet Type as a readable character. This scheme can leave you a series of bytes to be used as "control characters" two of which you use as the start marker and end marker. The ASCII character set neatly reserves the encodings of 0x00 to 0x1F as control characters. This is MANY years old and so you can see that this problem was solved long ago. There are even names given to some of them that can be directly leveraged such as SOH (start of header) and EOT (end of text).

edited Jun 30 '17 at 13:17

answered Jun 30 '17 at 13:11

Michael Karas

57,328
3
71
138

you say:Note that in applications that I create I also use a start marker that is unique and cannot occur in the data region, my big problem is that every character maybe occur in data section,character or Number – Ali Jul 01 '17 at 04:17
@MichealKaras could you pealse explain more about This allows quicker re-sync to the command stream without losing one extra packet? – Ali Jul 01 '17 at 04:21
@combo_ci - Yes! If you have a receiver is connected and starts looking for packets to arrive and is not yet in sync between its incoming data stream and the parsing of active packet boundaries the quickest way to achieve sync and not lose a packet is to look for a unique start marker. If you look for an end marker instead you cannot trust most of what came before so you must discard what may have been almost all of a packet and concentrate on receiving the next packet. – Michael Karas Jul 01 '17 at 17:02
@combo_ci - If you have no control over the content that a sender is providing and it is a badly designed protocol that cannot be changed then you will have a challenge. In these cases you have to struggle to find something unique in the transmissions that you can key off of to synchronize with the packet flow. For some examples here are two things I have had to resort to in the past. (1) I had to key off the fact that packets came one at a time with a multi byte delay time from the end of one to the starting of the next. (continued) – Michael Karas Jul 01 '17 at 17:11
(continued from above) (2) Another time I had to keep track of when bytes arrived back to back. If there was a hiccup it meant there was a break in the data flow that meant there was a packet boundary. – Michael Karas Jul 01 '17 at 17:12
I agree with you about start marker..I solve that issue with divide device id to High/Low section and in each receive look for valid pattern of deviceID ..with this technique I simulate start marker – Ali Jul 01 '17 at 17:39
About control over send data i say in previous answer comment : Unfortunately, I do not know at all what data from the sender of the oven ( because the sender mix some bits together and each bit means an action in receiver ...mixing the bits in some condition may deal to create any character and its a big bug in our system – Ali Jul 01 '17 at 17:42
About hiccup between receive data..do you have a solution for find hiccup? a know a way...use system timer for find hiccup between receiving data...do you have another solution? – Ali Jul 01 '17 at 17:46
1

A system timer interrupt is more applicable to working out multi byte gap between packets. Back to back bytes is better handled by reading a continuously running counter each time you read the next UART byte to get a relative time stamp each byte as received. Such counter wants to be ticking at a rate at least the bit rate of the UART stream but have enough bits that it does not over flow at more than say the length of time of several packets. – Michael Karas Jul 01 '17 at 17:56
1

@combo_ci - I do have to comment that you say that you are able to change the transmit protocol (ie adding a modified device ID) and yet you complain that you cannot do anything about the data content. I have to suggest that if you have control of the transmit end then you can take steps to solve the problem that you have!! One brute force way is to resort to the ASCII printable characters solution and convert all the arbitrary source data onto a Hex ASCII format by encoding each source binary byte to a readable character format in two bytes. Convert it back to binary at the receiving end. – Michael Karas Jul 01 '17 at 18:04

get dynamic length packet from UART?

3 Answers3