path: root/2021/captions/bindat.md

<a name="transcript"></a>
# Transcript

[[!template new="1" text="Hi. So I'm going to talk today" start="00:00:01.360" video="mainVideo" id=subtitle]]
[[!template text="about a fun rewrite I did of the BinDat package." start="00:00:04.180" video="mainVideo" id=subtitle]]
[[!template text="I call this Turbo BinDat." start="00:00:10.000" video="mainVideo" id=subtitle]]
[[!template text="Actually, the package hasn't changed name," start="00:00:12.400" video="mainVideo" id=subtitle]]
[[!template text="it's just that the result happens to be faster." start="00:00:14.101" video="mainVideo" id=subtitle]]
[[!template text="The point was not to make it faster though," start="00:00:16.901" video="mainVideo" id=subtitle]]
[[!template text="and the point was not to make you understand" start="00:00:19.621" video="mainVideo" id=subtitle]]
[[!template text="that data is not code." start="00:00:22.341" video="mainVideo" id=subtitle]]
[[!template text="It's just one more experience I've had" start="00:00:23.540" video="mainVideo" id=subtitle]]
[[!template text="where I've seen that treating data as code" start="00:00:27.120" video="mainVideo" id=subtitle]]
[[!template text="is not always a good idea." start="00:00:31.381" video="mainVideo" id=subtitle]]
[[!template text="It's important to keep the difference." start="00:00:33.622" video="mainVideo" id=subtitle]]
[[!template text="So let's get started." start="00:00:36.162" video="mainVideo" id=subtitle]]
[[!template new="1" text="So what is BinDat anyway?" start="00:00:38.881" video="mainVideo" id=subtitle]]
[[!template text="Here's just the overview of basically" start="00:00:40.742" video="mainVideo" id=subtitle]]
[[!template text="what I'm going to present." start="00:00:43.602" video="mainVideo" id=subtitle]]
[[!template text="So I'm first going to present BinDat itself" start="00:00:45.062" video="mainVideo" id=subtitle]]
[[!template text="for those who don't know it," start="00:00:47.843" video="mainVideo" id=subtitle]]
[[!template text="which is probably the majority of you." start="00:00:49.039" video="mainVideo" id=subtitle]]
[[!template text="Then I'm going to talk about the actual problems" start="00:00:51.923" video="mainVideo" id=subtitle]]
[[!template text="that I encountered with this package" start="00:00:55.363" video="mainVideo" id=subtitle]]
[[!template text="that motivated me to rewrite it." start="00:00:58.882" video="mainVideo" id=subtitle]]
[[!template text="Most of them were lack of flexibility," start="00:01:01.843" video="mainVideo" id=subtitle]]
[[!template text="and some of it was just poor behavior" start="00:01:05.044" video="mainVideo" id=subtitle]]
[[!template text="with respect to scoping and variables," start="00:01:09.924" video="mainVideo" id=subtitle]]
[[!template text="which of course, you know, is bad --" start="00:01:13.364" video="mainVideo" id=subtitle]]
[[!template text="basically uses of eval or, &quot ;eval is evil.&quot ;" start="00:01:16.424" video="mainVideo" id=subtitle]]
[[!template text="Then I'm going to talk about the new design --" start="00:01:20.724" video="mainVideo" id=subtitle]]
[[!template text="how I redesigned it" start="00:01:24.985" video="mainVideo" id=subtitle]]
[[!template text="to make it both simpler and more flexible," start="00:01:28.105" video="mainVideo" id=subtitle]]
[[!template text="and where the key idea was" start="00:01:31.365" video="mainVideo" id=subtitle]]
[[!template text="to expose code as code" start="00:01:33.065" video="mainVideo" id=subtitle]]
[[!template text="instead of having it as data," start="00:01:35.305" video="mainVideo" id=subtitle]]
[[!template text="and so here the distinction between the two" start="00:01:37.625" video="mainVideo" id=subtitle]]
[[!template text="is important and made things simpler." start="00:01:39.706" video="mainVideo" id=subtitle]]
[[!template text="I tried to keep efficiency in mind," start="00:01:44.085" video="mainVideo" id=subtitle]]
[[!template text="which resulted in some of the aspects of the design" start="00:01:46.405" video="mainVideo" id=subtitle]]
[[!template text="which are not completely satisfactory," start="00:01:52.505" video="mainVideo" id=subtitle]]
[[!template text="but the result is actually fairly efficient." start="00:01:54.886" video="mainVideo" id=subtitle]]
[[!template text="Even though it was not the main motivation," start="00:01:57.146" video="mainVideo" id=subtitle]]
[[!template text="it was one of the nice outcomes." start="00:01:59.287" video="mainVideo" id=subtitle]]
[[!template text="And then I'm going to present some examples." start="00:02:02.967" video="mainVideo" id=subtitle]]
[[!template new="1" text="So first: what is BinDat?" start="00:02:06.007" video="mainVideo" id=subtitle]]
[[!template text="Oh actually, rather than present THIS," start="00:02:08.267" video="mainVideo" id=subtitle]]
[[!template text="I'm going to go straight to the code," start="00:02:10.667" video="mainVideo" id=subtitle]]
[[!template text="because BinDat actually had" start="00:02:12.507" video="mainVideo" id=subtitle]]
[[!template text="an introduction which was fairly legible." start="00:02:14.346" video="mainVideo" id=subtitle]]
[[!template text="So here we go: this is the old BinDat from Emacs 27" start="00:02:16.748" video="mainVideo" id=subtitle]]
[[!template text="and the commentary starts by explaining" start="00:02:21.128" video="mainVideo" id=subtitle]]
[[!template text="what is BinDat? Basically BinDat is a package" start="00:02:23.448" video="mainVideo" id=subtitle]]
[[!template text="that lets you parse and unparse" start="00:02:25.948" video="mainVideo" id=subtitle]]
[[!template text="basically binary data." start="00:02:30.247" video="mainVideo" id=subtitle]]
[[!template text="The intent is to have typically network data" start="00:02:31.627" video="mainVideo" id=subtitle]]
[[!template text="or something like this." start="00:02:34.749" video="mainVideo" id=subtitle]]
[[!template text="So assuming you have network data," start="00:02:35.949" video="mainVideo" id=subtitle]]
[[!template text="presented or defined" start="00:02:38.328" video="mainVideo" id=subtitle]]
[[!template text="with some kind of C-style structs, typically," start="00:02:41.628" video="mainVideo" id=subtitle]]
[[!template text="or something along these lines." start="00:02:44.669" video="mainVideo" id=subtitle]]
[[!template text="So you presumably start with documentation" start="00:02:46.109" video="mainVideo" id=subtitle]]
[[!template text="that presents something like those structs here," start="00:02:49.120" video="mainVideo" id=subtitle]]
[[!template text="and you want to be able to generate such packets" start="00:02:52.810" video="mainVideo" id=subtitle]]
[[!template text="and read such packets," start="00:02:57.230" video="mainVideo" id=subtitle]]
[[!template text="so the way you do it is" start="00:03:00.349" video="mainVideo" id=subtitle]]
[[!template text="you rewrite those specifications" start="00:03:02.190" video="mainVideo" id=subtitle]]
[[!template text="into the BinDat syntax." start="00:03:04.670" video="mainVideo" id=subtitle]]
[[!template text="So here's the BinDat syntax" start="00:03:06.110" video="mainVideo" id=subtitle]]
[[!template text="for the the previous specification." start="00:03:07.529" video="mainVideo" id=subtitle]]
[[!template text="So here, for example," start="00:03:10.491" video="mainVideo" id=subtitle]]
[[!template text="you see the case for a data packet" start="00:03:11.610" video="mainVideo" id=subtitle]]
[[!template text="which will have a 'type' field which is a byte" start="00:03:16.970" video="mainVideo" id=subtitle]]
[[!template text="(an unsigned 8-bit entity)," start="00:03:20.411" video="mainVideo" id=subtitle]]
[[!template text="then an 'opcode' which is also a byte," start="00:03:24.091" video="mainVideo" id=subtitle]]
[[!template text="then a 'length' which is a 16-bit unsigned integer" start="00:03:26.411" video="mainVideo" id=subtitle]]
[[!template text="in little endian order," start="00:03:30.732" video="mainVideo" id=subtitle]]
[[!template text="and then some 'id' for this entry, which is" start="00:03:34.092" video="mainVideo" id=subtitle]]
[[!template text="8 bytes containing a zero-terminated string," start="00:03:38.732" video="mainVideo" id=subtitle]]
[[!template text="and then the actual data, basically the payload," start="00:03:43.531" video="mainVideo" id=subtitle]]
[[!template text="which is in this case a vector of bytes," start="00:03:47.532" video="mainVideo" id=subtitle]]
[[!template text="('bytes' here doesn't doesn't need to be specified)" start="00:03:51.453" video="mainVideo" id=subtitle]]
[[!template text="and here we specify the length of this vector." start="00:03:54.812" video="mainVideo" id=subtitle]]
[[!template text="This 'length' here" start="00:03:58.172" video="mainVideo" id=subtitle]]
[[!template text="happens to be actually the name of THIS field," start="00:03:59.773" video="mainVideo" id=subtitle]]
[[!template text="so the length of the data" start="00:04:02.252" video="mainVideo" id=subtitle]]
[[!template text="is specified by the 'length' field here," start="00:04:03.854" video="mainVideo" id=subtitle]]
[[!template text="and BinDat will understand this part," start="00:04:06.574" video="mainVideo" id=subtitle]]
[[!template text="which is the the nice part of BinDat." start="00:04:08.574" video="mainVideo" id=subtitle]]
[[!template text="And then you have an alignment field at the end," start="00:04:12.333" video="mainVideo" id=subtitle]]
[[!template text="which is basically padding." start="00:04:15.774" video="mainVideo" id=subtitle]]
[[!template text="It says that it is padded" start="00:04:18.253" video="mainVideo" id=subtitle]]
[[!template text="until the next multiple of four." start="00:04:20.575" video="mainVideo" id=subtitle]]
[[!template text="Okay. So this works reasonably well." start="00:04:23.295" video="mainVideo" id=subtitle]]
[[!template text="This is actually very nice." start="00:04:25.855" video="mainVideo" id=subtitle]]
[[!template text="With this, you can then call" start="00:04:27.455" video="mainVideo" id=subtitle]]
[[!template text="bindat-pack or bindat-unpack," start="00:04:30.335" video="mainVideo" id=subtitle]]
[[!template text="passing it a string, or passing it an alist," start="00:04:32.975" video="mainVideo" id=subtitle]]
[[!template text="to do the packing and unpacking." start="00:04:37.774" video="mainVideo" id=subtitle]]
[[!template text="So, for example, if you take this string--" start="00:04:40.416" video="mainVideo" id=subtitle]]
[[!template text="actually, in this case, it's a vector of bytes" start="00:04:43.296" video="mainVideo" id=subtitle]]
[[!template text="but it works the same; it works in both ways--" start="00:04:45.856" video="mainVideo" id=subtitle]]
[[!template text="if you pass this to bindat-unpack," start="00:04:49.456" video="mainVideo" id=subtitle]]
[[!template text="it will presumably return you this structure" start="00:04:53.536" video="mainVideo" id=subtitle]]
[[!template text="if you've given it the corresponding type." start="00:04:57.457" video="mainVideo" id=subtitle]]
[[!template text="So it will extract--" start="00:05:00.017" video="mainVideo" id=subtitle]]
[[!template text="you will see that there is an IP address," start="00:05:01.776" video="mainVideo" id=subtitle]]
[[!template text="which is a destination IP, a source IP," start="00:05:05.617" video="mainVideo" id=subtitle]]
[[!template text="and some port number," start="00:05:08.017" video="mainVideo" id=subtitle]]
[[!template text="and some actual data here and there, etc." start="00:05:09.857" video="mainVideo" id=subtitle]]
[[!template text="So this is quite convenient if you need to do this," start="00:05:12.977" video="mainVideo" id=subtitle]]
[[!template text="and that's what it was designed for." start="00:05:18.018" video="mainVideo" id=subtitle]]
[[!template text="So here we are. Let's go back to the actual talk." start="00:05:20.898" video="mainVideo" id=subtitle]]
[[!template new="1" text="I converted BinDat to lexical scoping at some point" start="00:05:27.538" video="mainVideo" id=subtitle]]
[[!template text="and things seemed to work fine," start="00:05:34.339" video="mainVideo" id=subtitle]]
[[!template text="except, at some point, probably weeks later," start="00:05:37.299" video="mainVideo" id=subtitle]]
[[!template text="I saw a bug report" start="00:05:42.819" video="mainVideo" id=subtitle]]
[[!template text="about the new version using lexical scoping" start="00:05:47.139" video="mainVideo" id=subtitle]]
[[!template text="not working correctly with WeeChat." start="00:05:53.059" video="mainVideo" id=subtitle]]
[[!template text="So here's the actual chunk of code" start="00:05:56.339" video="mainVideo" id=subtitle]]
[[!template text="that appears in WeeChat." start="00:06:00.580" video="mainVideo" id=subtitle]]
[[!template text="Here you see that they also define a BinDat spec." start="00:06:02.820" video="mainVideo" id=subtitle]]
[[!template text="It's a packet that has a 32-bit unsigned length," start="00:06:08.421" video="mainVideo" id=subtitle]]
[[!template text="then some compression byte/compression information," start="00:06:14.741" video="mainVideo" id=subtitle]]
[[!template text="then an id which contains basically another struct" start="00:06:18.500" video="mainVideo" id=subtitle]]
[[!template text="(which is specified elsewhere; doesn't matter here)," start="00:06:23.780" video="mainVideo" id=subtitle]]
[[!template text="and after that, a vector" start="00:06:26.902" video="mainVideo" id=subtitle]]
[[!template text="whose size is not just specified by 'length'," start="00:06:28.661" video="mainVideo" id=subtitle]]
[[!template text="but is computed from 'length'." start="00:06:33.382" video="mainVideo" id=subtitle]]
[[!template text="So here's how they used to compute it in WeeChat." start="00:06:35.142" video="mainVideo" id=subtitle]]
[[!template text="So the length here can be specified in BinDat." start="00:06:39.142" video="mainVideo" id=subtitle]]
[[!template text="Instead of having" start="00:06:42.822" video="mainVideo" id=subtitle]]
[[!template text="just a reference to one of the fields," start="00:06:43.942" video="mainVideo" id=subtitle]]
[[!template text="or having a constant, you can actually compute it," start="00:06:45.863" video="mainVideo" id=subtitle]]
[[!template text="where you have to use this '(eval'," start="00:06:48.903" video="mainVideo" id=subtitle]]
[[!template text="and then followed by the actual expression" start="00:06:52.502" video="mainVideo" id=subtitle]]
[[!template text="where you say how you compute it." start="00:06:54.743" video="mainVideo" id=subtitle]]
[[!template text="And here you see that it actually computes it" start="00:06:58.103" video="mainVideo" id=subtitle]]
[[!template text="based on the 'length of the structure --" start="00:07:01.464" video="mainVideo" id=subtitle]]
[[!template text="that's supposed to be this 'length' field here --" start="00:07:04.904" video="mainVideo" id=subtitle]]
[[!template text="and it's referred to using the bindat-get-field" start="00:07:07.783" video="mainVideo" id=subtitle]]
[[!template text="to extract the field from the variable 'struct'." start="00:07:11.223" video="mainVideo" id=subtitle]]
[[!template text="And then it subtracts four, it subtracts one," start="00:07:14.503" video="mainVideo" id=subtitle]]
[[!template text="and adds some other things" start="00:07:17.943" video="mainVideo" id=subtitle]]
[[!template text="which depend on some field" start="00:07:19.468" video="mainVideo" id=subtitle]]
[[!template text="that's found in this 'id' field here." start="00:07:22.185" video="mainVideo" id=subtitle]]
[[!template text="And the problem with this code" start="00:07:26.905" video="mainVideo" id=subtitle]]
[[!template text="was that it broke" start="00:07:28.425" video="mainVideo" id=subtitle]]
[[!template text="because of this 'struct' variable here," start="00:07:30.425" video="mainVideo" id=subtitle]]
[[!template text="because this 'struct' variable is not defined" start="00:07:32.745" video="mainVideo" id=subtitle]]
[[!template text="anywhere in the specification of BinDat." start="00:07:35.145" video="mainVideo" id=subtitle]]
[[!template text="It was used internally as a local variable," start="00:07:38.106" video="mainVideo" id=subtitle]]
[[!template text="and because it was using dynamic scoping," start="00:07:41.866" video="mainVideo" id=subtitle]]
[[!template text="it actually happened to be available here," start="00:07:45.306" video="mainVideo" id=subtitle]]
[[!template text="but the documentation nowhere specifies it." start="00:07:47.386" video="mainVideo" id=subtitle]]
[[!template text="So it was not exactly" start="00:07:50.826" video="mainVideo" id=subtitle]]
[[!template text="a bug of the conversion to lexical scoping," start="00:07:52.506" video="mainVideo" id=subtitle]]
[[!template text="but it ended up breaking this code." start="00:07:55.547" video="mainVideo" id=subtitle]]
[[!template text="And there was no way to actually" start="00:07:58.906" video="mainVideo" id=subtitle]]
[[!template text="fix the code within the specification of BinDat." start="00:08:01.226" video="mainVideo" id=subtitle]]
[[!template text="You had to go outside the specification of BinDat" start="00:08:05.066" video="mainVideo" id=subtitle]]
[[!template text="to fix this problem." start="00:08:08.287" video="mainVideo" id=subtitle]]
[[!template text="This is basically how I started looking at BinDat." start="00:08:10.427" video="mainVideo" id=subtitle]]
[[!template text="Then I went to actually investigate a bit more" start="00:08:14.347" video="mainVideo" id=subtitle]]
[[!template text="what was going on," start="00:08:17.808" video="mainVideo" id=subtitle]]
[[!template text="and the thing I noticed along the way" start="00:08:19.627" video="mainVideo" id=subtitle]]
[[!template text="was basically that the specification of BinDat" start="00:08:22.108" video="mainVideo" id=subtitle]]
[[!template text="is fairly complex and has a lot of eval" start="00:08:25.787" video="mainVideo" id=subtitle]]
[[!template text="and things like this." start="00:08:29.528" video="mainVideo" id=subtitle]]
[[!template new="1" text="So let's take a look" start="00:08:30.749" video="mainVideo" id=subtitle]]
[[!template text="at what the BinDat specification looks like." start="00:08:32.288" video="mainVideo" id=subtitle]]
[[!template text="So here it's actually documented" start="00:08:35.068" video="mainVideo" id=subtitle]]
[[!template text="as a kind of grammar rules." start="00:08:36.589" video="mainVideo" id=subtitle]]
[[!template text="A specification is basically a sequence of items," start="00:08:40.269" video="mainVideo" id=subtitle]]
[[!template text="and then each of the items is basically" start="00:08:45.308" video="mainVideo" id=subtitle]]
[[!template text="a FIELD of a struct, so it has a FIELD name," start="00:08:47.389" video="mainVideo" id=subtitle]]
[[!template text="and then a TYPE." start="00:08:51.249" video="mainVideo" id=subtitle]]
[[!template text="Instead of a TYPE," start="00:08:53.249" video="mainVideo" id=subtitle]]
[[!template text="it could have some other FORM for eval," start="00:08:54.510" video="mainVideo" id=subtitle]]
[[!template text="which was basically never used as far as I know," start="00:08:56.590" video="mainVideo" id=subtitle]]
[[!template text="or it can be some filler," start="00:08:58.989" video="mainVideo" id=subtitle]]
[[!template text="or you can have some 'align' specification," start="00:09:00.190" video="mainVideo" id=subtitle]]
[[!template text="or you can refer to another struct." start="00:09:02.750" video="mainVideo" id=subtitle]]
[[!template text="It could also be some kind of union," start="00:09:05.150" video="mainVideo" id=subtitle]]
[[!template text="or it can be some kind of repetition of something." start="00:09:07.391" video="mainVideo" id=subtitle]]
[[!template text="And then you have the TYPE specified here," start="00:09:10.430" video="mainVideo" id=subtitle]]
[[!template text="which can be some integers, strings, or a vector," start="00:09:12.430" video="mainVideo" id=subtitle]]
[[!template text="and there are a few other special cases." start="00:09:18.271" video="mainVideo" id=subtitle]]
[[!template text="And then the actual field itself" start="00:09:21.631" video="mainVideo" id=subtitle]]
[[!template text="can be either a NAME, or something that's computed," start="00:09:25.311" video="mainVideo" id=subtitle]]
[[!template text="and then everywhere here, you have LEN," start="00:09:28.192" video="mainVideo" id=subtitle]]
[[!template text="which specifies the length of vectors," start="00:09:30.752" video="mainVideo" id=subtitle]]
[[!template text="for example, or length of strings." start="00:09:32.480" video="mainVideo" id=subtitle]]
[[!template text="This is actually either nil to mean one," start="00:09:34.672" video="mainVideo" id=subtitle]]
[[!template text="or it can be an ARG," start="00:09:37.632" video="mainVideo" id=subtitle]]
[[!template text="where ARG is defined to be" start="00:09:39.072" video="mainVideo" id=subtitle]]
[[!template text="either an integer or DEREF," start="00:09:40.952" video="mainVideo" id=subtitle]]
[[!template text="where DEREF is basically a specification" start="00:09:42.673" video="mainVideo" id=subtitle]]
[[!template text="that can refer, for example, to the 'length' field" start="00:09:46.673" video="mainVideo" id=subtitle]]
[[!template text="-- that's what we saw between parentheses: (length)" start="00:09:48.833" video="mainVideo" id=subtitle]]
[[!template text="was this way to refer to the 'length' field." start="00:09:51.956" video="mainVideo" id=subtitle]]
[[!template text="Or it can be an expression, which is what we saw" start="00:09:56.273" video="mainVideo" id=subtitle]]
[[!template text="in the computation of the length for WeeChat," start="00:09:59.794" video="mainVideo" id=subtitle]]
[[!template text="where you just had a '(eval'" start="00:10:02.834" video="mainVideo" id=subtitle]]
[[!template text="and then some computation" start="00:10:04.914" video="mainVideo" id=subtitle]]
[[!template text="of the length of the payload." start="00:10:06.334" video="mainVideo" id=subtitle]]
[[!template text="And so if you look here, you see that" start="00:10:10.274" video="mainVideo" id=subtitle]]
[[!template text="it is fairly large and complex," start="00:10:12.354" video="mainVideo" id=subtitle]]
[[!template text="and it uses eval everywhere. And actually," start="00:10:14.674" video="mainVideo" id=subtitle]]
[[!template text="it's not just that it has eval in its syntax," start="00:10:18.515" video="mainVideo" id=subtitle]]
[[!template text="but the implementation has to use eval everywhere," start="00:10:20.675" video="mainVideo" id=subtitle]]
[[!template text="because, if you go back" start="00:10:23.395" video="mainVideo" id=subtitle]]
[[!template text="to see the kind of code we see," start="00:10:25.314" video="mainVideo" id=subtitle]]
[[!template text="we see here we just define" start="00:10:27.475" video="mainVideo" id=subtitle]]
[[!template text="weechat--relay-message-spec as a constant!" start="00:10:29.538" video="mainVideo" id=subtitle]]
[[!template text="It's nothing than just data, right?" start="00:10:34.195" video="mainVideo" id=subtitle]]
[[!template text="So within this data" start="00:10:37.315" video="mainVideo" id=subtitle]]
[[!template text="there are things we need to evaluate," start="00:10:38.836" video="mainVideo" id=subtitle]]
[[!template text="but it's pure data," start="00:10:41.076" video="mainVideo" id=subtitle]]
[[!template text="so it will have to be evaluated" start="00:10:42.356" video="mainVideo" id=subtitle]]
[[!template text="by passing it to eval. It can't be compiled," start="00:10:44.356" video="mainVideo" id=subtitle]]
[[!template text="because it's within a quote, right?" start="00:10:46.596" video="mainVideo" id=subtitle]]
[[!template text="And so for that reason, kittens really" start="00:10:50.196" video="mainVideo" id=subtitle]]
[[!template text="suffer terribly with uses of BinDat." start="00:10:52.837" video="mainVideo" id=subtitle]]
[[!template text="You really have to be very careful with that." start="00:10:55.956" video="mainVideo" id=subtitle]]
[[!template text="More seriously," start="00:10:59.957" video="mainVideo" id=subtitle]]
[[!template text="the 'struct' variable was not documented," start="00:11:02.037" video="mainVideo" id=subtitle]]
[[!template text="and yet it's indispensable" start="00:11:05.157" video="mainVideo" id=subtitle]]
[[!template text="for important applications," start="00:11:07.797" video="mainVideo" id=subtitle]]
[[!template text="such as using in WeeChat." start="00:11:08.996" video="mainVideo" id=subtitle]]
[[!template text="So clearly this needs to be fixed." start="00:11:11.158" video="mainVideo" id=subtitle]]
[[!template text="Of course, we can just document 'struct'" start="00:11:13.078" video="mainVideo" id=subtitle]]
[[!template text="as some variable that's used there," start="00:11:15.481" video="mainVideo" id=subtitle]]
[[!template text="but of course we don't want to do that," start="00:11:18.038" video="mainVideo" id=subtitle]]
[[!template text="because 'struct' is not obviously" start="00:11:19.798" video="mainVideo" id=subtitle]]
[[!template text="a dynamically scoped variable," start="00:11:23.398" video="mainVideo" id=subtitle]]
[[!template text="so it's not very clean." start="00:11:25.398" video="mainVideo" id=subtitle]]
[[!template text="Also other problems I noticed was that the grammar" start="00:11:29.318" video="mainVideo" id=subtitle]]
[[!template text="is significantly more complex than necessary." start="00:11:31.939" video="mainVideo" id=subtitle]]
[[!template text="We have nine distinct non-terminals." start="00:11:35.239" video="mainVideo" id=subtitle]]
[[!template text="There is ambiguity." start="00:11:38.199" video="mainVideo" id=subtitle]]
[[!template text="If you try to use a field whose name is 'align'," start="00:11:39.639" video="mainVideo" id=subtitle]]
[[!template text="or 'fill', or something like this," start="00:11:44.919" video="mainVideo" id=subtitle]]
[[!template text="then it's going to be misinterpreted," start="00:11:48.680" video="mainVideo" id=subtitle]]
[[!template text="or it can be misinterpreted." start="00:11:50.920" video="mainVideo" id=subtitle]]
[[!template text="The vector length can be either an expression," start="00:11:54.920" video="mainVideo" id=subtitle]]
[[!template text="or an integer, or a reference to a label," start="00:11:58.760" video="mainVideo" id=subtitle]]
[[!template text="but the expression" start="00:12:02.280" video="mainVideo" id=subtitle]]
[[!template text="should already be the general case," start="00:12:03.720" video="mainVideo" id=subtitle]]
[[!template text="and this expression can itself be" start="00:12:06.361" video="mainVideo" id=subtitle]]
[[!template text="just a constant integer," start="00:12:08.041" video="mainVideo" id=subtitle]]
[[!template text="so this complexity is probably not indispensable," start="00:12:09.401" video="mainVideo" id=subtitle]]
[[!template text="or it could be replaced with something simpler." start="00:12:13.961" video="mainVideo" id=subtitle]]
[[!template text="That's what I felt like." start="00:12:15.641" video="mainVideo" id=subtitle]]
[[!template text="And basically lots of places" start="00:12:17.401" video="mainVideo" id=subtitle]]
[[!template text="allow an (eval EXP) form somewhere" start="00:12:19.161" video="mainVideo" id=subtitle]]
[[!template text="to open up the door for more flexibility," start="00:12:21.721" video="mainVideo" id=subtitle]]
[[!template text="but not all of them do," start="00:12:25.082" video="mainVideo" id=subtitle]]
[[!template text="and we don't really want" start="00:12:26.922" video="mainVideo" id=subtitle]]
[[!template text="to have this eval there, right?" start="00:12:29.482" video="mainVideo" id=subtitle]]
[[!template text="It's not very convenient syntactically either." start="00:12:31.001" video="mainVideo" id=subtitle]]
[[!template text="So it makes the uses of eval" start="00:12:33.802" video="mainVideo" id=subtitle]]
[[!template text="a bit heavier than they need to be," start="00:12:36.042" video="mainVideo" id=subtitle]]
[[!template text="and so I didn't really like this part." start="00:12:38.362" video="mainVideo" id=subtitle]]
[[!template text="Another part is that" start="00:12:41.723" video="mainVideo" id=subtitle]]
[[!template text="when I tried to figure out what was going on," start="00:12:42.603" video="mainVideo" id=subtitle]]
[[!template text="dog barks and distracts Stefan" start="00:12:45.183" video="mainVideo" id=subtitle]]
[[!template text="I had trouble... Winnie as well, as you can hear." start="00:12:46.666" video="mainVideo" id=subtitle]]
[[!template text="She had trouble as well." start="00:12:50.043" video="mainVideo" id=subtitle]]
[[!template text="But one of the troubles was that" start="00:12:50.923" video="mainVideo" id=subtitle]]
[[!template text="there was no way to debug the code" start="00:12:53.083" video="mainVideo" id=subtitle]]
[[!template text="via Edebug, because it's just data," start="00:12:55.002" video="mainVideo" id=subtitle]]
[[!template text="so Edebug doesn't know that it has to look at it" start="00:12:57.562" video="mainVideo" id=subtitle]]
[[!template text="and instrument it." start="00:13:00.524" video="mainVideo" id=subtitle]]
[[!template text="And of course it was not conveniently extensible." start="00:13:02.683" video="mainVideo" id=subtitle]]
[[!template text="That's also one of the things" start="00:13:05.644" video="mainVideo" id=subtitle]]
[[!template text="I noticed along the way." start="00:13:07.164" video="mainVideo" id=subtitle]]
[[!template text="Okay, so here's an example of" start="00:13:09.084" video="mainVideo" id=subtitle]]
[[!template text="problems not that I didn't just see there," start="00:13:12.844" video="mainVideo" id=subtitle]]
[[!template text="but that were actually present in code." start="00:13:15.485" video="mainVideo" id=subtitle]]
[[!template text="I went to look at code that was using BinDat" start="00:13:18.684" video="mainVideo" id=subtitle]]
[[!template text="to see what uses looked like," start="00:13:22.124" video="mainVideo" id=subtitle]]
[[!template text="and I saw that BinDat was not used very heavily," start="00:13:24.285" video="mainVideo" id=subtitle]]
[[!template text="but some of the main uses" start="00:13:28.765" video="mainVideo" id=subtitle]]
[[!template text="were just to read and write integers." start="00:13:30.365" video="mainVideo" id=subtitle]]
[[!template text="And here you can see a very typical case." start="00:13:33.885" video="mainVideo" id=subtitle]]
[[!template text="This is also coming from WeeChat." start="00:13:37.565" video="mainVideo" id=subtitle]]
[[!template text="We do a bindat-get-field" start="00:13:41.726" video="mainVideo" id=subtitle]]
[[!template text="of the length of some struct we read." start="00:13:43.565" video="mainVideo" id=subtitle]]
[[!template text="Actually, the struct we read is here." start="00:13:48.445" video="mainVideo" id=subtitle]]
[[!template text="It has a single field," start="00:13:50.685" video="mainVideo" id=subtitle]]
[[!template text="because the only thing we want to do" start="00:13:51.647" video="mainVideo" id=subtitle]]
[[!template text="is actually to unpack a 32-bit integer," start="00:13:53.006" video="mainVideo" id=subtitle]]
[[!template text="but the only way we can do that" start="00:13:56.287" video="mainVideo" id=subtitle]]
[[!template text="is by specifying a struct with one field." start="00:13:58.287" video="mainVideo" id=subtitle]]
[[!template text="And so we have to extract this struct of one field," start="00:14:01.647" video="mainVideo" id=subtitle]]
[[!template text="which constructs an alist" start="00:14:04.847" video="mainVideo" id=subtitle]]
[[!template text="containing the actual integer," start="00:14:07.246" video="mainVideo" id=subtitle]]
[[!template text="and then we just use get-field to extract it." start="00:14:09.648" video="mainVideo" id=subtitle]]
[[!template text="So this doesn't seem very elegant" start="00:14:11.887" video="mainVideo" id=subtitle]]
[[!template text="to have to construct an alist" start="00:14:15.007" video="mainVideo" id=subtitle]]
[[!template text="just to then extract the integer from it." start="00:14:16.528" video="mainVideo" id=subtitle]]
[[!template text="Same thing if you try to pack it:" start="00:14:20.368" video="mainVideo" id=subtitle]]
[[!template text="you first have to construct the alist" start="00:14:21.648" video="mainVideo" id=subtitle]]
[[!template text="to pass it to bindat-pack unnecessarily." start="00:14:25.007" video="mainVideo" id=subtitle]]
[[!template text="Another problem that I saw in this case" start="00:14:31.248" video="mainVideo" id=subtitle]]
[[!template text="(it was in the websocket package)" start="00:14:33.248" video="mainVideo" id=subtitle]]
[[!template text="was here, where they actually have a function" start="00:14:35.729" video="mainVideo" id=subtitle]]
[[!template text="where they need to write" start="00:14:39.568" video="mainVideo" id=subtitle]]
[[!template text="an integer of a size that will vary" start="00:14:41.169" video="mainVideo" id=subtitle]]
[[!template text="depending on the circumstances." start="00:14:43.888" video="mainVideo" id=subtitle]]
[[!template text="And so they have to test the value of this integer," start="00:14:45.889" video="mainVideo" id=subtitle]]
[[!template text="and depending on which one it is," start="00:14:49.650" video="mainVideo" id=subtitle]]
[[!template text="they're going to use different types." start="00:14:52.210" video="mainVideo" id=subtitle]]
[[!template text="So here it's a case" start="00:14:54.449" video="mainVideo" id=subtitle]]
[[!template text="where we want to have some kind of way to eval --" start="00:14:56.290" video="mainVideo" id=subtitle]]
[[!template text="to compute the length of the integer --" start="00:14:59.490" video="mainVideo" id=subtitle]]
[[!template text="instead of it being predefined or fixed." start="00:15:02.531" video="mainVideo" id=subtitle]]
[[!template text="So this is one of the cases" start="00:15:08.130" video="mainVideo" id=subtitle]]
[[!template text="where the lack of eval was a problem." start="00:15:10.211" video="mainVideo" id=subtitle]]
[[!template text="And actually in all of websocket," start="00:15:16.531" video="mainVideo" id=subtitle]]
[[!template text="BinDat is only used to pack and unpack integers," start="00:15:20.051" video="mainVideo" id=subtitle]]
[[!template text="even though there are many more opportunities" start="00:15:22.612" video="mainVideo" id=subtitle]]
[[!template text="to use BinDat in there." start="00:15:24.612" video="mainVideo" id=subtitle]]
[[!template text="But it's not very convenient to use BinDat," start="00:15:26.772" video="mainVideo" id=subtitle]]
[[!template text="as it stands, for those other cases." start="00:15:29.331" video="mainVideo" id=subtitle]]
[[!template new="1" text="So what does the new design look like?" start="00:15:35.891" video="mainVideo" id=subtitle]]
[[!template text="Well in the new design, here's the problematic code" start="00:15:39.733" video="mainVideo" id=subtitle]]
[[!template text="for WeeChat." start="00:15:44.132" video="mainVideo" id=subtitle]]
[[!template text="So we basically have the same fields as before," start="00:15:46.373" video="mainVideo" id=subtitle]]
[[!template text="you just see that instead of u32," start="00:15:49.012" video="mainVideo" id=subtitle]]
[[!template text="we now have 'uint 32' separately." start="00:15:50.853" video="mainVideo" id=subtitle]]
[[!template text="The idea is that now this 32" start="00:15:53.733" video="mainVideo" id=subtitle]]
[[!template text="can be an expression you can evaluate," start="00:15:55.332" video="mainVideo" id=subtitle]]
[[!template text="and so the u8 is also replaced by 'uint 8'," start="00:15:59.094" video="mainVideo" id=subtitle]]
[[!template text="and the id type is basically the same as before," start="00:16:04.054" video="mainVideo" id=subtitle]]
[[!template text="and here another difference we see," start="00:16:07.253" video="mainVideo" id=subtitle]]
[[!template text="and the main difference..." start="00:16:08.854" video="mainVideo" id=subtitle]]
[[!template text="Actually, it's the second main difference." start="00:16:11.654" video="mainVideo" id=subtitle]]
[[!template text="The first main difference is that" start="00:16:13.494" video="mainVideo" id=subtitle]]
[[!template text="we don't actually quote this whole thing." start="00:16:15.175" video="mainVideo" id=subtitle]]
[[!template text="Instead, we pass it to the bindat-type macro." start="00:16:18.694" video="mainVideo" id=subtitle]]
[[!template text="So this is a macro" start="00:16:23.095" video="mainVideo" id=subtitle]]
[[!template text="that's going to actually build the type." start="00:16:25.095" video="mainVideo" id=subtitle]]
[[!template text="This is a big difference" start="00:16:27.574" video="mainVideo" id=subtitle]]
[[!template text="in terms of performance also," start="00:16:29.254" video="mainVideo" id=subtitle]]
[[!template text="because by making it a macro," start="00:16:30.535" video="mainVideo" id=subtitle]]
[[!template text="we can pre-compute the code" start="00:16:32.695" video="mainVideo" id=subtitle]]
[[!template text="that's going to pack and unpack this thing," start="00:16:34.296" video="mainVideo" id=subtitle]]
[[!template text="instead of having to interpret it" start="00:16:37.255" video="mainVideo" id=subtitle]]
[[!template text="every time we pack and unpack." start="00:16:38.936" video="mainVideo" id=subtitle]]
[[!template text="So this macro will generate more efficient code" start="00:16:41.096" video="mainVideo" id=subtitle]]
[[!template text="along the way." start="00:16:43.815" video="mainVideo" id=subtitle]]
[[!template text="Also it makes the code that appears in here" start="00:16:45.815" video="mainVideo" id=subtitle]]
[[!template text="visible to the compiler" start="00:16:48.695" video="mainVideo" id=subtitle]]
[[!template text="because we can give an Edebug spec for it." start="00:16:50.297" video="mainVideo" id=subtitle]]
[[!template text="And so here as an argument to vec," start="00:16:54.617" video="mainVideo" id=subtitle]]
[[!template text="instead of having to specify" start="00:16:57.497" video="mainVideo" id=subtitle]]
[[!template text="that this is an evaluated expression," start="00:16:59.016" video="mainVideo" id=subtitle]]
[[!template text="we just write the expression directly," start="00:17:00.937" video="mainVideo" id=subtitle]]
[[!template text="because all the expressions that appear there" start="00:17:02.777" video="mainVideo" id=subtitle]]
[[!template text="will just be evaluated," start="00:17:05.096" video="mainVideo" id=subtitle]]
[[!template text="and we don't need to use the 'struct' variable" start="00:17:07.418" video="mainVideo" id=subtitle]]
[[!template text="and then extract the length field from it." start="00:17:11.418" video="mainVideo" id=subtitle]]
[[!template text="We can just use length as a variable." start="00:17:14.137" video="mainVideo" id=subtitle]]
[[!template text="So this variable 'length' here" start="00:17:16.938" video="mainVideo" id=subtitle]]
[[!template text="will refer to this field here," start="00:17:18.698" video="mainVideo" id=subtitle]]
[[!template text="and then this variable 'id' here" start="00:17:20.778" video="mainVideo" id=subtitle]]
[[!template text="will refer to this field here," start="00:17:23.578" video="mainVideo" id=subtitle]]
[[!template text="and so we can just use the field values" start="00:17:25.898" video="mainVideo" id=subtitle]]
[[!template text="as local variables, which is very natural" start="00:17:27.738" video="mainVideo" id=subtitle]]
[[!template text="and very efficient also," start="00:17:30.459" video="mainVideo" id=subtitle]]
[[!template text="because the code would actually directly do that," start="00:17:31.679" video="mainVideo" id=subtitle]]
[[!template text="and the code that unpacks those data" start="00:17:34.618" video="mainVideo" id=subtitle]]
[[!template text="will just extract an integer" start="00:17:37.899" video="mainVideo" id=subtitle]]
[[!template text="and bind it to the length variable," start="00:17:40.299" video="mainVideo" id=subtitle]]
[[!template text="and so that makes it immediately available there." start="00:17:42.219" video="mainVideo" id=subtitle]]
[[!template new="1" text="Okay, let's see also" start="00:17:47.580" video="mainVideo" id=subtitle]]
[[!template text="what the actual documentation looks like." start="00:17:51.340" video="mainVideo" id=subtitle]]
[[!template text="And so if we look at the doc of BinDat," start="00:17:54.220" video="mainVideo" id=subtitle]]
[[!template text="we see the actual specification of the grammar." start="00:17:57.739" video="mainVideo" id=subtitle]]
[[!template text="And so here we see instead of having" start="00:18:01.181" video="mainVideo" id=subtitle]]
[[!template text="these nine different non-terminals," start="00:18:03.181" video="mainVideo" id=subtitle]]
[[!template text="we basically have two:" start="00:18:06.461" video="mainVideo" id=subtitle]]
[[!template text="we have the non-terminal for TYPE," start="00:18:08.061" video="mainVideo" id=subtitle]]
[[!template text="which can be either a uint, a uintr, or a string," start="00:18:10.781" video="mainVideo" id=subtitle]]
[[!template text="or bits, or fill, or align, or vec," start="00:18:15.021" video="mainVideo" id=subtitle]]
[[!template text="or those various other forms;" start="00:18:17.421" video="mainVideo" id=subtitle]]
[[!template text="or it can be a struct, in which case," start="00:18:19.902" video="mainVideo" id=subtitle]]
[[!template text="in the case of struct," start="00:18:22.621" video="mainVideo" id=subtitle]]
[[!template text="then it will be followed by a sequence --" start="00:18:23.981" video="mainVideo" id=subtitle]]
[[!template text="a list of FIELDs, where each of the FIELDs" start="00:18:27.502" video="mainVideo" id=subtitle]]
[[!template text="is basically a LABEL followed by another TYPE." start="00:18:30.142" video="mainVideo" id=subtitle]]
[[!template text="And so this makes the whole specification" start="00:18:33.902" video="mainVideo" id=subtitle]]
[[!template text="much simpler. We don't have any distinction now" start="00:18:37.343" video="mainVideo" id=subtitle]]
[[!template text="between struct being a special case," start="00:18:39.823" video="mainVideo" id=subtitle]]
[[!template text="as opposed to just the normal types." start="00:18:42.862" video="mainVideo" id=subtitle]]
[[!template text="struct is just now one of the possible types" start="00:18:46.383" video="mainVideo" id=subtitle]]
[[!template text="that can appear here." start="00:18:49.263" video="mainVideo" id=subtitle]]
[[!template text="The other thing is that" start="00:18:52.543" video="mainVideo" id=subtitle]]
[[!template text="the LABEL is always present in the structure," start="00:18:53.263" video="mainVideo" id=subtitle]]
[[!template text="so there's no ambiguity." start="00:18:55.743" video="mainVideo" id=subtitle]]
[[!template text="Also all the above things," start="00:18:58.384" video="mainVideo" id=subtitle]]
[[!template text="like the BITLEN we have here," start="00:19:00.304" video="mainVideo" id=subtitle]]
[[!template text="the LEN we have here," start="00:19:03.103" video="mainVideo" id=subtitle]]
[[!template text="the COUNT for vector we have here," start="00:19:04.384" video="mainVideo" id=subtitle]]
[[!template text="these are all plain Elisp expressions," start="00:19:07.504" video="mainVideo" id=subtitle]]
[[!template text="so they are implicitly evaluated if necessary." start="00:19:10.224" video="mainVideo" id=subtitle]]
[[!template text="If you want them to be constant," start="00:19:13.025" video="mainVideo" id=subtitle]]
[[!template text="and really constant, you can just use quotes," start="00:19:14.705" video="mainVideo" id=subtitle]]
[[!template text="for those rare cases where it's necessary." start="00:19:16.705" video="mainVideo" id=subtitle]]
[[!template text="Another thing is that you can extend it" start="00:19:20.145" video="mainVideo" id=subtitle]]
[[!template text="with with bindat-defmacro." start="00:19:21.905" video="mainVideo" id=subtitle]]
[[!template text="Okay, let's go back here." start="00:19:25.505" video="mainVideo" id=subtitle]]
[[!template new="1" text="So what are the advantages of this approach?" start="00:19:30.226" video="mainVideo" id=subtitle]]
[[!template text="As I said, one of the main advantages" start="00:19:32.706" video="mainVideo" id=subtitle]]
[[!template text="is that we now have support for Edebug." start="00:19:34.625" video="mainVideo" id=subtitle]]
[[!template text="We don't have 'struct', 'repeat', and 'align'" start="00:19:39.346" video="mainVideo" id=subtitle]]
[[!template text="as special cases anymore." start="00:19:41.426" video="mainVideo" id=subtitle]]
[[!template text="These are just normal types." start="00:19:42.946" video="mainVideo" id=subtitle]]
[[!template text="Before, there was uint as type, int as type," start="00:19:44.625" video="mainVideo" id=subtitle]]
[[!template text="and those kinds of things." start="00:19:48.067" video="mainVideo" id=subtitle]]
[[!template text="'struct' and 'repeat' and 'align'" start="00:19:49.267" video="mainVideo" id=subtitle]]
[[!template text="were in a different case." start="00:19:51.110" video="mainVideo" id=subtitle]]
[[!template text="So there were" start="00:19:53.267" video="mainVideo" id=subtitle]]
[[!template text="some subtle differences between those" start="00:19:54.387" video="mainVideo" id=subtitle]]
[[!template text="that completely disappeared." start="00:19:56.787" video="mainVideo" id=subtitle]]
[[!template text="Also in the special cases, there was 'union'," start="00:19:59.027" video="mainVideo" id=subtitle]]
[[!template text="and union now has completely disappeared." start="00:20:02.626" video="mainVideo" id=subtitle]]
[[!template text="We don't need it anymore, because instead," start="00:20:05.027" video="mainVideo" id=subtitle]]
[[!template text="we can actually use code anywhere." start="00:20:07.828" video="mainVideo" id=subtitle]]
[[!template text="That's one of the things I didn't mention here," start="00:20:09.588" video="mainVideo" id=subtitle]]
[[!template text="but in this note here," start="00:20:11.908" video="mainVideo" id=subtitle]]
[[!template text="that's one of the important notes." start="00:20:17.268" video="mainVideo" id=subtitle]]
[[!template text="Not only are BITLEN, LEN, COUNT etc." start="00:20:19.747" video="mainVideo" id=subtitle]]
[[!template text="Elisp expressions," start="00:20:21.987" video="mainVideo" id=subtitle]]
[[!template text="but the type itself -- any type itself --" start="00:20:23.028" video="mainVideo" id=subtitle]]
[[!template text="is basically an expression." start="00:20:26.789" video="mainVideo" id=subtitle]]
[[!template text="And so you can, instead of having 'uint BITLEN'," start="00:20:29.029" video="mainVideo" id=subtitle]]
[[!template text="you can have '(if blah-blah-blah uint string)'," start="00:20:32.709" video="mainVideo" id=subtitle]]
[[!template text="and so you can have a field" start="00:20:36.628" video="mainVideo" id=subtitle]]
[[!template text="that can be either string or an int," start="00:20:38.149" video="mainVideo" id=subtitle]]
[[!template text="depending on some condition." start="00:20:40.549" video="mainVideo" id=subtitle]]
[[!template text="And for that reason we don't need a union." start="00:20:44.790" video="mainVideo" id=subtitle]]
[[!template text="Instead of having a union," start="00:20:46.869" video="mainVideo" id=subtitle]]
[[!template text="we can just have a 'cond' or a 'pcase'" start="00:20:47.910" video="mainVideo" id=subtitle]]
[[!template text="that will return the type we want to use," start="00:20:50.710" video="mainVideo" id=subtitle]]
[[!template text="depending on the context," start="00:20:53.590" video="mainVideo" id=subtitle]]
[[!template text="which will generally depend on some previous field." start="00:20:55.109" video="mainVideo" id=subtitle]]
[[!template text="Also we don't need to use single-field structs" start="00:21:00.951" video="mainVideo" id=subtitle]]
[[!template text="for simple types anymore," start="00:21:03.750" video="mainVideo" id=subtitle]]
[[!template text="because there's no distinction between struct" start="00:21:05.351" video="mainVideo" id=subtitle]]
[[!template text="and other types." start="00:21:09.271" video="mainVideo" id=subtitle]]
[[!template text="So we can pass to bindat-pack and bindat-unpack" start="00:21:11.271" video="mainVideo" id=subtitle]]
[[!template text="a specification which just says &quot ;here's an integer&quot ;" start="00:21:17.191" video="mainVideo" id=subtitle]]
[[!template text="and we'll just pack and unpack the integer." start="00:21:20.952" video="mainVideo" id=subtitle]]
[[!template text="And of course now all the code is exposed," start="00:21:24.392" video="mainVideo" id=subtitle]]
[[!template text="so not only Edebug works, but also Flymake," start="00:21:26.472" video="mainVideo" id=subtitle]]
[[!template text="and the compiler, etc. --" start="00:21:29.192" video="mainVideo" id=subtitle]]
[[!template text="they can complain about it," start="00:21:30.392" video="mainVideo" id=subtitle]]
[[!template text="and give you warnings and errors as we like them." start="00:21:33.111" video="mainVideo" id=subtitle]]
[[!template text="And of course the kittens are much happier." start="00:21:38.872" video="mainVideo" id=subtitle]]
[[!template text="Okay. This is going a bit over time," start="00:21:44.553" video="mainVideo" id=subtitle]]
[[!template text="so let's try to go faster." start="00:21:48.153" video="mainVideo" id=subtitle]]
[[!template new="1" text="Here are some of the new features" start="00:21:51.273" video="mainVideo" id=subtitle]]
[[!template text="that are introduced." start="00:21:53.753" video="mainVideo" id=subtitle]]
[[!template text="I already mentioned briefly" start="00:21:54.794" video="mainVideo" id=subtitle]]
[[!template text="that you can define new types with bindat-defmacro." start="00:21:56.314" video="mainVideo" id=subtitle]]
[[!template text="that's one of the important novelties," start="00:22:00.633" video="mainVideo" id=subtitle]]
[[!template text="and you can extend BinDat with new types this way." start="00:22:04.474" video="mainVideo" id=subtitle]]
[[!template text="The other thing you can do is" start="00:22:08.794" video="mainVideo" id=subtitle]]
[[!template text="you can control how values or packets" start="00:22:10.714" video="mainVideo" id=subtitle]]
[[!template text="are unpacked, and how they are represented." start="00:22:16.234" video="mainVideo" id=subtitle]]
[[!template text="In the old BinDat," start="00:22:20.315" video="mainVideo" id=subtitle]]
[[!template text="the packet is necessarily represented," start="00:22:22.555" video="mainVideo" id=subtitle]]
[[!template text="when you unpack it, as an alist, basically," start="00:22:24.315" video="mainVideo" id=subtitle]]
[[!template text="or a struct becomes an alist," start="00:22:28.635" video="mainVideo" id=subtitle]]
[[!template text="and that's all there is." start="00:22:30.396" video="mainVideo" id=subtitle]]
[[!template text="You don't have any choice about it." start="00:22:31.676" video="mainVideo" id=subtitle]]
[[!template text="With the new system," start="00:22:34.076" video="mainVideo" id=subtitle]]
[[!template text="by default, it also returns just an alist," start="00:22:35.596" video="mainVideo" id=subtitle]]
[[!template text="but you can actually control what it's unpacked as," start="00:22:38.076" video="mainVideo" id=subtitle]]
[[!template text="or what it's packed from, using these keywords." start="00:22:41.916" video="mainVideo" id=subtitle]]
[[!template text="With :unpack-val, you can give an expression" start="00:22:46.396" video="mainVideo" id=subtitle]]
[[!template text="that will construct the unpacked value" start="00:22:49.597" video="mainVideo" id=subtitle]]
[[!template text="from the various fields." start="00:22:53.357" video="mainVideo" id=subtitle]]
[[!template text="And with :pack-val and :pack-var," start="00:22:56.957" video="mainVideo" id=subtitle]]
[[!template text="you can specify how to extract the information" start="00:22:59.197" video="mainVideo" id=subtitle]]
[[!template text="from the unpacked value" start="00:23:02.557" video="mainVideo" id=subtitle]]
[[!template text="to generate the pack value." start="00:23:05.117" video="mainVideo" id=subtitle]]
[[!template new="1" text="So here are some examples." start="00:23:08.078" video="mainVideo" id=subtitle]]
[[!template text="Here's an example taken from osc." start="00:23:12.637" video="mainVideo" id=subtitle]]
[[!template text="osc actually doesn't use BinDat currently," start="00:23:15.358" video="mainVideo" id=subtitle]]
[[!template text="but I have played with it" start="00:23:17.438" video="mainVideo" id=subtitle]]
[[!template text="to see what it would look like" start="00:23:22.479" video="mainVideo" id=subtitle]]
[[!template text="if we were to use BinDat." start="00:23:23.758" video="mainVideo" id=subtitle]]
[[!template text="So here's the definition" start="00:23:26.159" video="mainVideo" id=subtitle]]
[[!template text="of the timetag representation," start="00:23:28.638" video="mainVideo" id=subtitle]]
[[!template text="which represents timestamps in osc." start="00:23:30.638" video="mainVideo" id=subtitle]]
[[!template text="So you would use bindat-type" start="00:23:35.279" video="mainVideo" id=subtitle]]
[[!template text="and then you have here :pack-var" start="00:23:37.998" video="mainVideo" id=subtitle]]
[[!template text="basically gives a name" start="00:23:40.559" video="mainVideo" id=subtitle]]
[[!template text="when we try to pack a timestamp." start="00:23:42.080" video="mainVideo" id=subtitle]]
[[!template text="'time' will be the variable whose name contains" start="00:23:48.559" video="mainVideo" id=subtitle]]
[[!template text="the actual timestamp we will receive." start="00:23:51.520" video="mainVideo" id=subtitle]]
[[!template text="So we want to represent the unpacked value" start="00:23:54.159" video="mainVideo" id=subtitle]]
[[!template text="as a normal Emacs timestamp," start="00:23:57.520" video="mainVideo" id=subtitle]]
[[!template text="and then basically convert from this timestamp" start="00:24:00.240" video="mainVideo" id=subtitle]]
[[!template text="to a string, or from a string to this timestamp." start="00:24:02.480" video="mainVideo" id=subtitle]]
[[!template text="When we receive it, it will be called time," start="00:24:06.401" video="mainVideo" id=subtitle]]
[[!template text="so we can refer to it," start="00:24:10.080" video="mainVideo" id=subtitle]]
[[!template text="and so in order to actually encode it," start="00:24:12.240" video="mainVideo" id=subtitle]]
[[!template text="we basically turn this timestamp into an integer --" start="00:24:15.360" video="mainVideo" id=subtitle]]
[[!template text="that's what this :pack-val does." start="00:24:18.320" video="mainVideo" id=subtitle]]
[[!template text="It says when we try to pack it," start="00:24:20.799" video="mainVideo" id=subtitle]]
[[!template text="here's the the value that we should use." start="00:24:23.442" video="mainVideo" id=subtitle]]
[[!template text="We turn it into an integer," start="00:24:26.082" video="mainVideo" id=subtitle]]
[[!template text="and then this integer is going to be encoded" start="00:24:27.760" video="mainVideo" id=subtitle]]
[[!template text="as a uint 64-bit. So a 64-bit unsigned integer." start="00:24:30.320" video="mainVideo" id=subtitle]]
[[!template text="When we try to unpack the value," start="00:24:36.163" video="mainVideo" id=subtitle]]
[[!template text="this 'ticks' field" start="00:24:38.960" video="mainVideo" id=subtitle]]
[[!template text="will contain an unsigned int of 64 bits." start="00:24:40.720" video="mainVideo" id=subtitle]]
[[!template text="We want to return instead a timestamp --" start="00:24:45.679" video="mainVideo" id=subtitle]]
[[!template text="a time value -- from Emacs." start="00:24:50.559" video="mainVideo" id=subtitle]]
[[!template text="Here we use the representation of time" start="00:24:53.924" video="mainVideo" id=subtitle]]
[[!template text="as a pair of number of ticks" start="00:24:59.363" video="mainVideo" id=subtitle]]
[[!template text="and the corresponding frequency of those ticks." start="00:25:02.799" video="mainVideo" id=subtitle]]
[[!template text="So that's what we do here with :unpack-val," start="00:25:06.720" video="mainVideo" id=subtitle]]
[[!template text="which is construct the cons corresponding to it." start="00:25:09.120" video="mainVideo" id=subtitle]]
[[!template text="With this definition, bindat-pack/unpack" start="00:25:12.004" video="mainVideo" id=subtitle]]
[[!template text="are going to convert to and from" start="00:25:16.400" video="mainVideo" id=subtitle]]
[[!template text="proper time values on one side," start="00:25:19.039" video="mainVideo" id=subtitle]]
[[!template text="and binary strings on the other." start="00:25:21.760" video="mainVideo" id=subtitle]]
[[!template text="Note, of course," start="00:25:26.159" video="mainVideo" id=subtitle]]
[[!template text="that I complained that the old BinDat" start="00:25:27.520" video="mainVideo" id=subtitle]]
[[!template text="had to use single-field structs for simple types," start="00:25:30.320" video="mainVideo" id=subtitle]]
[[!template text="and here, basically," start="00:25:36.080" video="mainVideo" id=subtitle]]
[[!template text="I'm back using single-field structs as well" start="00:25:37.039" video="mainVideo" id=subtitle]]
[[!template text="for this particular case --" start="00:25:39.840" video="mainVideo" id=subtitle]]
[[!template text="actually a reasonably frequent case, to be honest." start="00:25:41.120" video="mainVideo" id=subtitle]]
[[!template text="But at least this is not so problematic," start="00:25:44.640" video="mainVideo" id=subtitle]]
[[!template text="because we actually control what is returned," start="00:25:49.279" video="mainVideo" id=subtitle]]
[[!template text="so even though it's a single-field struct," start="00:25:51.840" video="mainVideo" id=subtitle]]
[[!template text="it's not going to construct an alist" start="00:25:54.159" video="mainVideo" id=subtitle]]
[[!template text="or force you to construct an alist." start="00:25:56.640" video="mainVideo" id=subtitle]]
[[!template text="Instead, it really receives and takes a value" start="00:25:58.320" video="mainVideo" id=subtitle]]
[[!template text="in the ideal representation that we chose." start="00:26:02.720" video="mainVideo" id=subtitle]]
[[!template text="Here we have a more complex example," start="00:26:07.367" video="mainVideo" id=subtitle]]
[[!template text="where the actual type is recursive," start="00:26:10.007" video="mainVideo" id=subtitle]]
[[!template text="because it's representing those &quot ;LEB&quot ;..." start="00:26:12.488" video="mainVideo" id=subtitle]]
[[!template text="I can't remember what &quot ;LEB&quot ; stands for," start="00:26:18.640" video="mainVideo" id=subtitle]]
[[!template text="but it's a representation" start="00:26:20.400" video="mainVideo" id=subtitle]]
[[!template text="for arbitrary length integers," start="00:26:22.559" video="mainVideo" id=subtitle]]
[[!template text="where basically" start="00:26:25.600" video="mainVideo" id=subtitle]]
[[!template text="every byte is either smaller than 128," start="00:26:27.520" video="mainVideo" id=subtitle]]
[[!template text="in which case it's the end of the of the value," start="00:26:33.360" video="mainVideo" id=subtitle]]
[[!template text="or it's a value bigger than 128," start="00:26:36.799" video="mainVideo" id=subtitle]]
[[!template text="in which case there's an extra byte on the end" start="00:26:39.760" video="mainVideo" id=subtitle]]
[[!template text="that's going to continue." start="00:26:42.159" video="mainVideo" id=subtitle]]
[[!template text="Here we see the representation" start="00:26:44.490" video="mainVideo" id=subtitle]]
[[!template text="is basically a structure that starts with a byte," start="00:26:46.640" video="mainVideo" id=subtitle]]
[[!template text="which contains this value," start="00:26:52.240" video="mainVideo" id=subtitle]]
[[!template text="which can be either the last value or not," start="00:26:53.679" video="mainVideo" id=subtitle]]
[[!template text="and the tail, which will either be empty," start="00:26:56.000" video="mainVideo" id=subtitle]]
[[!template text="or contain something else." start="00:26:59.770" video="mainVideo" id=subtitle]]
[[!template text="The empty case is here;" start="00:27:01.279" video="mainVideo" id=subtitle]]
[[!template text="if the head value is smaller than 128," start="00:27:04.000" video="mainVideo" id=subtitle]]
[[!template text="then the type of this tail is going to be (unit 0)," start="00:27:07.039" video="mainVideo" id=subtitle]]
[[!template text="so basically 'unit' is the empty type," start="00:27:11.840" video="mainVideo" id=subtitle]]
[[!template text="and 0 is the value we will receive when we read it." start="00:27:16.492" video="mainVideo" id=subtitle]]
[[!template text="And if not, then it has as type 'loop'," start="00:27:20.880" video="mainVideo" id=subtitle]]
[[!template text="which is the type we're defining," start="00:27:25.520" video="mainVideo" id=subtitle]]
[[!template text="so it's the recursive case," start="00:27:28.240" video="mainVideo" id=subtitle]]
[[!template text="where then the rest of the type is the type itself." start="00:27:30.491" video="mainVideo" id=subtitle]]
[[!template text="And so this lets us pack and unpack." start="00:27:35.132" video="mainVideo" id=subtitle]]
[[!template text="We pass it an arbitrary size integer," start="00:27:37.120" video="mainVideo" id=subtitle]]
[[!template text="and it's going to turn it into" start="00:27:39.600" video="mainVideo" id=subtitle]]
[[!template text="this LEB128 binary representation, and vice versa." start="00:27:42.240" video="mainVideo" id=subtitle]]
[[!template text="I have other examples if you're interested," start="00:27:48.492" video="mainVideo" id=subtitle]]
[[!template text="but anyway, here's the conclusion." start="00:27:52.480" video="mainVideo" id=subtitle]]
[[!template new="1" text="We have a simpler, more flexible," start="00:27:56.094" video="mainVideo" id=subtitle]]
[[!template text="and more powerful BinDat now," start="00:27:58.320" video="mainVideo" id=subtitle]]
[[!template text="which is also significantly faster." start="00:28:01.039" video="mainVideo" id=subtitle]]
[[!template text="And I can't remember the exact speed-up," start="00:28:03.454" video="mainVideo" id=subtitle]]
[[!template text="but it's definitely not a few percents." start="00:28:06.799" video="mainVideo" id=subtitle]]
[[!template text="I vaguely remember about 4x faster in my tests," start="00:28:08.720" video="mainVideo" id=subtitle]]
[[!template text="but it's probably very different in different cases" start="00:28:12.640" video="mainVideo" id=subtitle]]
[[!template text="so it might be just 4x, 2x -- who knows?" start="00:28:16.815" video="mainVideo" id=subtitle]]
[[!template text="Try it for yourself, but I was pretty pleased," start="00:28:20.159" video="mainVideo" id=subtitle]]
[[!template text="because it wasn't the main motivation, so anyway..." start="00:28:23.374" video="mainVideo" id=subtitle]]
[[!template new="1" text="The negatives are here." start="00:28:28.336" video="mainVideo" id=subtitle]]
[[!template text="In the new system, there's this bindat-defmacro" start="00:28:31.135" video="mainVideo" id=subtitle]]
[[!template text="which lets us define, kind of, new types," start="00:28:34.480" video="mainVideo" id=subtitle]]
[[!template text="and bindat-type also lets us define new types," start="00:28:36.720" video="mainVideo" id=subtitle]]
[[!template text="and the distinction between them is a bit subtle;" start="00:28:40.895" video="mainVideo" id=subtitle]]
[[!template text="it kind of depends on..." start="00:28:45.360" video="mainVideo" id=subtitle]]
[[!template text="well it has an impact on efficiency" start="00:28:48.080" video="mainVideo" id=subtitle]]
[[!template text="more than anything, so it's not very satisfactory." start="00:28:50.880" video="mainVideo" id=subtitle]]
[[!template text="There's a bit of redundancy between the two." start="00:28:53.520" video="mainVideo" id=subtitle]]
[[!template text="There is no bit-level control, just as before." start="00:28:56.737" video="mainVideo" id=subtitle]]
[[!template text="We can only manipulate basically bytes." start="00:28:59.039" video="mainVideo" id=subtitle]]
[[!template text="So this is definitely not usable" start="00:29:02.098" video="mainVideo" id=subtitle]]
[[!template text="for a Huffman encoding kind of thing." start="00:29:03.360" video="mainVideo" id=subtitle]]
[[!template text="Also, it's not nearly as flexible" start="00:29:09.058" video="mainVideo" id=subtitle]]
[[!template text="as some of the alternatives." start="00:29:10.880" video="mainVideo" id=subtitle]]
[[!template text="So you know GNU Poke" start="00:29:12.240" video="mainVideo" id=subtitle]]
[[!template text="has been a vague inspiration for this work," start="00:29:13.760" video="mainVideo" id=subtitle]]
[[!template text="and GNU Poke gives you a lot more power" start="00:29:20.018" video="mainVideo" id=subtitle]]
[[!template text="in how to specify the types, etc." start="00:29:22.480" video="mainVideo" id=subtitle]]
[[!template text="And of course one of the main downsides" start="00:29:25.059" video="mainVideo" id=subtitle]]
[[!template text="is that it's still not used very much." start="00:29:26.579" video="mainVideo" id=subtitle]]
[[!template text="Actually the new BinDat" start="00:29:28.018" video="mainVideo" id=subtitle]]
[[!template text="is not used by any package" start="00:29:29.283" video="mainVideo" id=subtitle]]
[[!template text="as far as I know right now," start="00:29:31.039" video="mainVideo" id=subtitle]]
[[!template text="but even the old one is not used very often," start="00:29:33.059" video="mainVideo" id=subtitle]]
[[!template text="so who knows" start="00:29:35.279" video="mainVideo" id=subtitle]]
[[!template text="whether it's actually going to" start="00:29:36.799" video="mainVideo" id=subtitle]]
[[!template text="work very much better or not?" start="00:29:38.799" video="mainVideo" id=subtitle]]
[[!template text="Anyway, this is it for this talk." start="00:29:41.520" video="mainVideo" id=subtitle]]
[[!template text="Thank you very much. Have a nice day." start="00:29:44.399" video="mainVideo" id=subtitle]]
[[!template text="(captions by John Cummings)" start="00:29:46.683" video="mainVideo" id=subtitle]]