Developers.Idea.PacketFormat
Reworking onto Developers.Idea.PacketDesign
outline packet format
here is a standard packet structure it is very expandable and expressive it can contain strings, xml, and binary chunks very easily it looks good in specifications with its outline form it is not bloated at all
this was inspired by the fact that when you are lookign at a specification document, things are usualy in an outline like the one shown below. if the outline is so extensible and easy for humans to understand, why not teach the computer to communicate in it also?
here is how it works
<source lang="c"> gnutella
header
value
header
value
value
subvalue
value
header
header
value
subvalue
packet
attribute
value
attribute
value
packet
attribute
</source>
everything is lowercase, there can be spaces, and there is no punctuation the strings are ascii text terminated by a 0x00 byte after every 0x00 byte, there is a single ascii letter that moves the indext for the next row
w - back to the margin x - left one column y - same column z - right one column
this forms the outline structure shown above so, the handshake followed by a value would be like this in the bit stream
<source lang="c"> gnutella[\0][z]header[\0][z]value[\0] </source>
you can contain a lot with this. for instance, the following things can be a line in the outline without screwing it up
- text, including text with spaces
- a complete set of http headers, and \r\n doesn't need to be encoded
- xml
- arbitrary binary data of any length, described below
look how this is just as efficient as http headers. http headers use 2 bytes to separate a header from a value: colon and space. they use 2 more to separate two values: comma and space. and then they use 2 to move on to the next header: carriage return and newline. and all of these special bytes have to be encoded if they were to appear in the headers themselves
[code http headers] header: value, value\r\n </source>
the outline system is just as efficient, but simpler. each string is null terminated. the second byte sets the indent of the next line. so, just like http headers, it takes 2 bytes to separate everything. unlike http headers, you don't have to worry about special puncutation or newline characters getting in the way. also, you can easily do an outline any number of levels deep - http headers are just 2 levels deep
[code outline system] gnutella[\0][z]
header[\0][z]
value[\0][y]
value[\0][z]
subvalue[\0][x]
packet[\0]
</source>
most of the time in the outline, w, x, y and z will get you where you need to be next in one step. if you need to get somewhere further away, then just take additional steps with a blank line and another step. the program will ignore blank lines and keep following outline directions
you can put xml into this easily, and since it doesn't rely on any punctuation formatting, you don't have to worry about anything in xml getting in the way of this format. if this format were in xml, for instance, then you would have to worry about that, and it would be bloated too, by the way
here is how you can easily put any number of data bytes into the outline - the code is b. right after it is the outline code
b - here come some data bytes
<source lang="c"> [by]8[\0]BBBBBBBB[w]packet </source>
the code b means here come some bytes. the outline code follows. right after that is a string that shows the length. in the example above, it's 8, but this number can be any size, there is no limit to the bytes. that string ends with a null terminator, and right after that are the bytes. the computer knows where to read wpacket to keep going with the outline because of the size
notice how putting the size in a string is more efficient than having it expressed in bytes. if we expressed it in bytes, we would have to arbitrarily say the first 4 bytes of a binary block are an int that describes the size. for short blocks like this one, that would use up 4 bytes instead of just 2 or 3. also, it doesn't scale. you would be limited to the largest int
right now, packet formatting for peer to peer networks is a mess. there are null terminated strings and size counts, and they clash, making it tedious to subtract sizes up from the back and so on. there are really small parts, like individual bits in a byte showing options, followed by a huge xml metadata chunk. and none of that really matters because the whole thing is zlib compressed anyway
what we need is a single superset format that we can fit all of these things into, that isn't network specific, and that is really, really simple both to program and to see in documentation. then, we can start using it more and more, and the old formats less and less, and make this transition without any performace hit. that is the goal of this spec, and this idea.
more ideas
the outline thing may be more complicated than necessary what if all the packets were just like the headers - linear each line of text is null terminated the program knows how many lines make up a packet the first line identifies the packet and its version, like p for ping, p2 for an advanced ping the program negotiates in the handshake which packets and which versions it understands, and then the program sends exactly what its neighbour wants to hear you can put inline binary by, first, having your neighbour expect it, and then it takes up two lines, like this
<source lang="c"> 16 BBBBBBBBBBBBBBBB </source>
the first line says how long it is in ascii numbers null terminated, and then come the bytes you don't have name:value because the order matters, the program knows that thing one comes first, then thing two, and they have to be in that order, so they don't have to be labelled if you want to do a packet with a variable number of lines, then in there is going to be that number, like 8\0, that takes just two bytes, and describes how many lines it is these counts, of bytes or lines, should be just for the next segment, don't try to count the whole packet because then the program has to try to count backwards and so on if something binary, like an ip address or a guid, is always the same size, then you can send it between two lines without the size delimiter at all, wow that is efficient, for instance:
<source lang="c"> a text line of any length\0 IPNPORT line above\0 GUIDBYTES line below\0 </source>
the protocol specification states that line 2 is 6 bytes of IP address and port, so we don't need to put a line with 6 on it up there, and even if we did, it would only be 2 bytes, same thing with the guid, they are all the same this makes much more sense than converting these things into text and sending them over the wire that way also, instead of formatting things into text, things should be done in this format, because it is easier to parse for instance, here is a text-way (bad) to share more computers to contact
<source lang="c"> More-Computers: 1.2.3.4:5, 6.7.8.9.10, 11.12.13.14:15\r\n </source>
here is a more tightly packed for the wire and easier to parse version of that
<source lang="c"> 3\0 BBBBBB BBBBBB BBBBBB </source>
the 3 tells us that there are 3 ip address and port sets coming next, and then they are all there, each taking exactly 6 bytes so i guess the overall idea here is to unwind all of the complexity in parsing into this sytem of lines make it as tall as possible, not as wide and all the complexity of what is what into the specification, and programs that follow the specification, and an easy way to move to new versions of packets to modify the specification
since the lines are null terminated -you can read them like strings -you can put lots of stuff in them, like
-http headers, which are ended \r\n -an entire block of xml
blank lines are ok you could do a bunch of features, yes no unknown for instance, by using a bunch of lines
<source lang="c"> y
n n </source>
the spec says what each line means, so the program knows what the four questions being answered here are and that only took 7 bytes
also, this idea is just so much simpler than the top of the page its actually even simpler than http headers and also even more efficient in the bytestream
day 3
the idea behind xml and the outline idea on the top of this page is have the speaker tell you everything, and have the listener be able to look through it and find the parts that he understands the list idea above has the opposite philosophy - have both of them know exactly what to expect, and then communicate according to those rigid rules the spec can be transmitted once between developers out of band - there is no need to constantly wrap it into the protocol, transmitting little pieces of it over and over again through the network
the rules
ok, so to do the simple list-style packets correctly, here are the rules
no text formatting you might want to make a line that looks like this
<source lang="c"> packet number:value/name\r\n </source>
this is bad because now we have to parse on 3 different things: a space, a colon, and a slash, and then the \r\n, also. according to the simple list packet guideline, this should instead be this
<source lang="c"> p number value name </source>
we parse on \0 every time
no text encoding in the example above, we need to parse on space, colon, and slash. so if any of the values needed to contain these things, they would have to be url encoded. this is not necessary in the simple list system. so the rule is: no url encoding, no text encoding of any kind. you can even say \r\n if you need to. there is only one thing you cannot say, and that is \0
no binary encoding if you need to transmit a guid, do it in binary, not text. here is the wrong way to do it
<source lang="c"> guid: d51dff817f895598ff0065537c09d503 </source>
here is the right way to do it <source lang="c"> BBBBBBBBBBBBBBBB </source>
no label, no colon and then space, no hex encoding, just 16 bytes of binary guid. we know there is a guid there because of the spec. the packet on the wire doesn't need to tell us every time
bring everything to the margin look at this bad example
<source lang="c"> a1/a2/a3 b1/b2/b3 </source>
we are parsing for / on each line. we shouldn't be parsing for anything on each line - each line describes exactly one thing. if you need to describe a second thing, use a second line. this should be
<source lang="c"> a1 a2 a3 b1 b2 b3 </source>
no big lengths never, never, never have a length that describes anything except what is immediately next. if a packet contains a variable number of lines in one section, have the number right above each section, like this
<source lang="c"> i 3 blue green red 2 hot cold </source>
reading this i packet, we know there are 3 colors, and then 2 temperatures. we know from the spec that colors come first, then temperatures. here is the wrong way to do it
<source lang="c"> i 8 blue green red 2 hot cold </source>
the 8 here is supposed to indicate that the entire i packet is 8 lines long. then, the program has to run into the 2 later and subtract to try to find the colors in the middle. no subtracting! only use lengths when they are absolutely necessary, and keep them as local as possible