## 1. Project Overview & Quickstart (mfontanini/d2) # d2 D2 is a modern diagram scripting language that turns text to diagrams. ### Repository Details - **Repository:** [mfontanini/d2](https://github.com/mfontanini/d2) - **GitHub Stars:** ⭐ 25,605 - **Primary Language:** Go *Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.* ## 2. Official Technical Reference & Guides (mfontanini/mfontanini.github.io) ## File: _config.yml title: Average Coder github_username: mfontanini paginate: 5 gems: [jekyll-paginate, kramdown] markdown: kramdown --- ## File: about.md --- layout: page title : About permalink: /about/ --- I'm Matias Fontanini, a Software Engineer from Argentina. I'm currently working at [ThousandEyes](https://www.thousandeyes.com/) where I develop tools to measure network properties by crafting, sending and receiving packets, implement some outage detection systems and some random other work. _C++_ is my main programming language. I like template metaprogramming and try to stay up to date with the new standard releases. Python is my goto language whenever I need to automate some tedious task or need to write something quickly. In the past, I've worked on security, mostly network security and penetration testing. I like staring at code and find issues and ways to exploit it. ## Projects The main projects I've worked on are: * [libtins](https://github.com/mfontanini/libtins): a fast C++ packet crafting and sniffing library. * [cppkafka](https://github.com/mfontanini/cppkafka): a C++11 wrapper for _librdkafka_. * [dot11decrypt](https://github.com/mfontanini/dot11decrypt): A tiny tool that mostly relies on _libtins_ WEP/WPA2 decryption capabilities to perform on the fly decryption of 802.11 encrypted frames. * I also have a repository that contains [random scripts and stuff](https://github.com/mfontanini/Programs-Scripts) that I've worked on and didn't really deserve their own repo (or at least I that's what I thought at the time). ## Blog I used to post stuff in [my blog](http://average-coder.blogspot.com/) but I've decided to only write here now. --- ## File: LICENSE.txt MIT License Copyright (c) 2017 Hemang Kumar Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. --- ## File: _posts/2017-06-05-cppkafka-release.md --- layout: post title: "cppkafka - C++11 librdkafka wrapper" --- [cppkafka](https://github.com/mfontanini/cppkafka) is a project I've been slowly working on for a while. It's a _C++11_ wrapper built on top of [librdkafka](https://github.com/edenhill/librdkafka), a high performance _C_ client library for the [Apache Kafka](https://kafka.apache.org/) protocol. # Why another Kafka library? A few months ago, we started using Kafka at my [current workplace](https://www.thousandeyes.com/). I had to port some applications and implement new ones that would communicate with each other using this protocol. The existing applications where implemented in _C++_ so re-implementing them from scratch in another language was not an option. Plus, I'm very comfortable using it, so there really was no reason to change it. After looking for some alternatives on what to use, I had the impression that most of the _C++_ libraries where kind of immature and not really production ready. I did find [librdkafka](https://github.com/edenhill/librdkafka), developed in _C_, which looked very promising and seemed to be in active development, providing a robust and high performance implementation of the protocol. In fact, this is the only _C/C++_ library that has full protocol support for Kafka. _librdkafka_ contains several different language bindings like _Python_, _Go_, _C#_, etc. Even better, there seemed to be an official _C++_ wrapper for the library, _rdkafka++_, which should make using the library a pleasant experience. ## This doesn't really work My first reaction was not great. Don't get me wrong, the library itself worked without any issues and I was able to write a Kafka producer and consumer in a decent amount of time (mostly due to Kafka's learning curve), but overall using _rdkafka++_ was slightly tedious. As much as I don't have issues using raw pointers and I've used them, as well as naked _news_ and _deletes_ all over the place, that was a while ago. Nowadays I refuse to use a pointer unless it's wrapped around some smart pointer so I can guarantee proper a cleanup of all resources even on a failure condition. _rdkafka++_ uses raw pointers for absolutely everything, which was really not what I wanted. _C++_ has evolved a lot and there's really no need to perform manual memory management anymore. There's [zero cost abstractions](http://en.cppreference.com/w/cpp/memory/unique_ptr) in the standard library that let you write safe code with no overhead at all. ## Wrapping the wrapper At that point I decided to create a tiny project on our company's repository that would act as a wrapper of _rdkafka++_. So basically this would be a wrapper of a wrapper of _librdkafka_. At the time this seemed like a good idea as I would only need to write a tiny bit of code and then I would be able to create applications that used Kafka in a simple way. This would allow me to create _rdkafka++_ configuration objects, consuming and producing messages without having to deal with any pointers. This didn't go so well, as I soon found myself having to deal with some pretty nasty things. _rdkafka++_ tries its best to hide away its internal implementation which is not necessarily bad, as it allows for faster compilation times and it makes it harder to break the ABI when introducing changes, but in this case it just made it trickier to work with. I had to end up doing things like cloning `RdKafka::Conf` objects manually by accessing their internal representation, dealing with implementing proxy callbacks, wrapping all pointers in `unique_ptrs`, etc. After a while of using this wrapper and getting slightly frustrated, I realized I could do better and chose to actually implement a wrapper for the _C_ library, _librdkafka_, that used modern _C++_ features like smart pointers, move semantics and callbacks to provide a clean interface to use Kafka. # Hello [_cppkafka_](https://github.com/mfontanini/cppkafka) I slowly implemented this wrapper and used it at work on the projects I was porting to use Kafka as well as the new applications I implemented. All of those served as my guinea pigs, as I would use _cppkafka_, get annoyed at something, work on it at home and then change the projects to use the updated API. After a few months of changing things, I think at this point I'm satisfied with the current state of the project and it's time for a release. _cppkafka_ provides a wrapper for the high level consumer and producer interface, as well as some random utilities like a buffered producer (which also simplifies producer error handling) and a compacted topic consumer, which lets you process key-value streams (like [KTables](http://docs.confluent.io/current/streams/concepts.html#ktable)) in a simple way. The library is licensed under BSD-2 so you can use it on both commercial and non commercial projects. ## Simple interface _cppkafka_ lets you create Kafka producers/consumers with very little code. Creating a consumer/producer's configuration is as simple as it gets: {% highlight cpp %} Configuration config = { // The list of brokers we'll use { "metadata.broker.list", "kafka-server1,kafka-server2" }, // Our consumer's group { "group.id", "kafka-consumer-test" }, // Disable auto commit { "enable.auto.commit", false }, // Client group session timeout { "session.timeout.ms", 60000 } }; {% endhighlight %} There's no need to convert configuration values into strings, allocating anything, checking error codes, etc. You can actually use string, integral and bool types and _cppkafka_ will perform the conversion automatically for you. Under the hood, this is actually creating a `rd_kafka_conf_t` handle and setting the attributes on it. If setting any attributes fail, this will throw an exception. ### Consuming messages Consuming messages is straightforward using the _Consumer_ class. {% highlight cpp %} // Create some consumer configuration Configuration config = ...; // Create our consumer Consumer consumer(config); // Subscribe to the topic we want consumer.subscribe({ "some_topic" }); // Poll forever while (true) { Message msg = consumer.poll(); // Poll can return nothing, so make sure to check we got something if (!msg) { continue; } // There's an error if (msg.get_error()) { // rdkafka indicates EOF (end of partition) as errors, // which doesn't really mean something went wrong if (!msg.is_eof()) { // This is an actual error, handle it properly handle_error(msg.get_error()); } // Regardless, we need to keep going continue; } // We actually received a message, process it process_message(move(msg)); } {% endhighlight %} The `Message` class is just a wrapper over an `rd_kafka_message_t` handle. This means a message can be empty in case the wrapped handle is null. `Messages` can't be copied but can be moved so they can be safely stored in containers for later use. You can read more about message consumption on the [wiki](https://github.com/mfontanini/cppkafka/wiki/Consuming-messages), where there's examples regarding partition assignment/revocation callbacks. ## Producing messages Producing messages can be done using the `Producer` class. Messages are built using a `MessageBuilder` that allows setting the topic, partition, key, payload, etc: {% highlight cpp %} // Create some producer config Configuration config = ...; // Create the producer Producer producer(config); // The key and value we'll use const string key = "some_key"; const vector payload = { 0x01, 0x02, 0x03, 0x04 }; // Construct a builder for the Kafka topic "some_topic" MessageBuilder builder("some_topic"); // Set the partition, key and payload. These can be easily chained. builder.partition(10).key(key).payload(payload); // Now produce the message producer.produce(builder); {% endhighlight %} There's also an implementation of a [buffered producer](https://github.com/mfontanini/cppkafka/blob/master/include/cppkafka/utils/buffered_producer.h) that will handle delivery reports for messages and re-send them if needed, as well as other non-errors in _librdkafka_ like filling the output queue, in which case it just waits until messages can be written. ## Performance overhead One of the things I kept in mind while implementing _cppkafka_ was performance. I didn't want to create a wrapper which would add overhead to every operation, making it less performant than _librdkafka_. Fortunately, most abstractions should have ended up being 100% free. For example `Message` objects simply wrap a handle. If you fetch a message's key and payload, you'll get a read only view of them that is actually not copying or allocating any extra data. Plus, it will automatically release all resources when it goes out of scope, so you can store it on vectors or other data structures and be sure you won't have any memory leaks. I know there should be some benchmark in this section, as otherwise there's not much credibility. Unfortunately, I don't have any yet so I can't really claim consuming/producing messages is as fast as when using _librdkafka_, but I'll definitely work on that at some point. Some configuration properties use `std::function` for callbacks, which have a slight performance penalty but that should be negligible compared to the cost of reading messages over the network. ## Conclusion All in all, I've had a really positive experience using _librdkafka_ (through _cppkafka_), besides a few issues I came across along the way, all of which were fixed in recent versions. If you want to have a high performant and hopefully painless experience consuming and producing messages using Kafka, make sure to check out _cppkafka_'s [GitHub repository](https://github.com/mfontanini/cppkafka). If you find any problems or have any suggestions, please don't hesitate to create an issue or pull request! --- ## File: _posts/2017-06-11-detecting-callback-types.md --- layout: post title: "Detecting and classifying template callback types in C++" --- # Introduction While implementing [cppkafka](https://github.com/mfontanini/cppkafka), I realized every time I wrote a consumer application, the consumption code would look like this: {% highlight cpp %} while (true) { // Try getting a message Message msg = consumer.poll(); // We may not have got anything (timeout) if (!msg) { continue; } // Check for errors if (msg.get_error()) { // EOF is propagated as an error but it really isn't one // Most of the times, you don't care about them if (!msg.is_eof()) { // This is an actual error, handle it some how // ... } continue; } // We found an actual message. Process it process_message(move(msg)); } {% endhighlight %} This started getting repetitive after a while. Most of the times, you only really care about getting a message so the majority of that code will usually do the same, except for the one bit that actually does the message processing. Moreover, the rest of the "events" (_EOF_/error notification, etc) you get can easily be assigned a default behavior. For example, when an error that's not just an _EOF_ notification is encountered, it's safe to default to throwing an exception and let someone handle it properly, whereas for some others just ignoring them is enough (e.g. for timeouts). So I wondered if there could be a way to wrap consumption in such a way that I could just express interest in the cases I wanted (e.g. actually getting a message) and not having to remember to check for all of those conditions. Note that this is a C++11 library and as much I'd love to use some C++14/17 features that would simplify some of the code in here, I preferred sticking to the standard the library was aimed for. # Using callbacks The first thing that came to my mind was using callbacks via `std::functions`. You would be able to configure callbacks so that whenever some event occurred, you would get a notification. For example, something like: {% highlight cpp %} // Build some dispatcher instance that will consume for us using the // underlying consumer to do all the polling related work ConsumerDispatcher dispatcher(consumer); // Configure the callback that will handle actual messages dispatcher.on_message([&](Message msg) { // Process it // ... }); // This would start the consumption loop, dispatching every event as // needed. In this case we would only get valid messages through // the lambda used above that aren't timeouts nor errors dispatcher.run(); {% endhighlight %} The rest of the optional callbacks (e.g. handling an error) would default to some implementation which you can easily override. The entire list of events you could be interested in would be: * A new message is polled. * An error is found. * _EOF_ is reached on a partition. * We get no messages out of a poll. We'll call this timeout, as it's one of the cases which will cause this. This implementation actually works fine and could be a solution to the problem. The only potential downside about this is that `std::functions` have some overhead as they use virtual calls to perform the function dispatching. If I was going to add some helper class to aid in making consumption simpler, I would want it to be as lightweight as possible so that it wouldn't make consuming messages any slower. # Templates to the rescue If we changed that `ConsumerDispatcher` class to use template functor types rather than `std::functions`, the compiler would likely be able to get rid of most of the overhead and end up generating code that looks as close to the initial loop as possible. For example, we could do something like: {% highlight cpp %} template // etc void ConsumerDispatcher::run(MessageFunctor on_message, ErrorFunctor on_error) { // Same loop, dispatch to parameters } {% endhighlight %} The issue is that in order to use templates, we would have to either: * Require the user to provide callbacks for all types of events that can occur. This is not a great idea as we'd be forcing the user to go back to the initial loop implementation, only now using callbacks instead. * Provide multiple overloads so that users can specify anywhere from one to four callbacks to handle each type of event. This doesn't work well as you would need to handle all combinations of them which becomes a mess for both implementing and using the class. Both of these don't really work for the reasons stated above. But there's another way to do this: * Use a single function and variadic templates, detect which callback the user is trying to specify based on its ignature and then dispatch to each of them when needed. This way, the user can provide one to four callbacks and we would be able to find which one should handle each case and call them accordingly. This would require some template metaprogramming to detect signatures and sounded fun so I gave it a try. It had been a while since I had done anything using _TMP_ so this added some extra fun to it. ## Detecting signatures Detecting the signature of a template functor type is simple. You can implement this by using `decltype` and `std::declval` using the commonly used yes/no overload technique: {% highlight cpp %} // Indicates whether the functor type T accepts parameters Args... template struct takes_arguments { using yes = double; using no = bool; // Declare a value of type Functor and try to call it using the arguments // types provided template static yes test(decltype(declval()(declval()...))*); template static no test(...); static constexpr bool value = sizeof(test(nullptr)) == sizeof(yes); }; {% endhighlight %} Then I figured that given all the template parameter packs that I would need to use, it would be a good idea to implement a specialization that actually took a tuple of types: {% highlight cpp %} template struct takes_arguments> : takes_arguments { }; {% endhighlight %} So now using this we can check if we can call a function using some parameters. For example: {% highlight cpp %} void foo(int); void bar(char, void*); // All of these would succeed static_assert(takes_arguments::value); static_assert(takes_arguments::value); static_assert(takes_arguments>::value); {% endhighlight %} ## Finding a specific callback's type Now that we know whether a callback can be executed using the list of types we expect, we can write something that given a list of callbacks and a signature, would find the type of one of those callbacks that matched that signature. Since we may not actually find the callback we want because the user used an incompatible signature, I used a `type_not_found` type to indicate so: {% highlight cpp %} // Simple identity trait template struct identity { using type = T; }; // Placeholder to indicate a type wasn't found struct type_not_found { }; // Recursive helper to iterate through all functor types until // we find the one we want template struct find_type_helper { // If it accepts the arguments, then this is the type we want. // Otherwise keep iterating over the callback types using type = typename conditional::value, identity, find_type_helper >::type::type; }; // Base case. If we reach the end of the list, we couldn't find it template struct find_type_helper { using type = type_not_found; }; // Find the functor that matches the given std::tuple of types. // Note that we append type_not_found at the end so that recursion // will always end on it in case we can't find the type. template struct find_type { using type = typename find_type_helper::type; }; {% endhighlight %} This can be instantiated using an `std::tuple` containing the types we expect the functor to accept and the list of functors: {% highlight cpp %} void foo(int); void bar(char, void*); // decltype(foo) find_type::type x; // decltype(bar) find_type, decltype(foo), decltype(bar)>::type y; // type_not_found, as nothing can be called using a string find_type, decltype(foo), decltype(bar)>::type z; {% endhighlight %} This gives us the ability to find a specific type within the list of functors. Now we need a way to actually find the callback we want rather than just its type. ## Finding a callback Finding a callback is simple if we use the `find_type` helper we wrote above. We basically need to find the type we want out of the list of functor types (e.g. the one that matches the signature) and then go through all functors until we find the one that matches that type. In pseudo code, this would mean something like: {% highlight cpp %} // First find the type of the functor that is callable // with the list of arguments we provide. The type returned // would be the type of one of the functors type = find_type(arguments, functors); // Now find the functor that matches the given type functor = find_functor(type, functors); {% endhighlight %} After actually implementing it, it looks like the following: {% highlight cpp %} // Helper the iterates through all functors until it finds the right one template struct find_functor_helper { // Base case where we found it template static const Functor& find(const Functor& arg, Functors&&...) { return arg; } // Recursive case, only instantiated if the head of the template // parameter pack is not the type we want template static typename enable_if::value, const Functor&>::type find(const Head&, const Functors&... functors) { return find(functors...); } }; // The function that actually finds it just instantiates and calls // the helper template const Functor& find_functor(const Args&... args) { return find_functor_helper::find(args...); } {% endhighlight %} Note that this will fail if we provide a type that can't actually be matched. However, this cant't happen, as we are going to extract the actual type out of the functor list using `find_type` first so we are guaranteed that the type we are looking for is actually in the functor parameter pack. Now as a last helper, I wrote this function that finds the type and if it can't find it, it fails with a static assertion: {% highlight cpp %} // Finds the callback and returns it or fails with a static assertion // if lookup fails template const typename find_type::type& find_matching_functor(const Functors&... functors) { // Find the specific type using type = typename find_type::type; // Fail if we couldn't find it static_assert(!is_same::value, "Valid functor not found"); // Now find the functor that matches the type return find_functor(functors...); } {% endhighlight %} This function is then used as follows: {% highlight cpp %} template void ConsumerDispatcher::run(const Args&... args) { // Define the types we expect on our message processing callback using OnMessageArgs = tuple; // Will fail with a static assertion if we can't find it auto callback = find_matching_functor(args...); // Iterate, poll for messages, error events, etc } {% endhighlight %} ## Optional callbacks But what about optional callbacks? Initially we wanted to allow the user to specify some callbacks but not necessarily all of them as this is kind of tedious. Right now if we call `find_matching_functor` for an optional one, the static assertion will fail, so we need a way around that. One way could be to implement some "find or fallback" mechanism so that we provide our own default in case we can't find it. However, there's a trivial way around this without any code additions. We can just append our own custom handler at the end of the functor list. If we find the user's first, then we'll use that one. Otherwise we fall back to using ours: {% highlight cpp %} static void ConsumerDispatcher::handle_error(Error err) { // Throw! } template void ConsumerDispatcher::run(const Args&... args) { // Define the types we expect on our message processing callback using OnErrorArgs = tuple; // Our default handler in case the user doesn't specify one auto default_handler = &ConsumerDispatcher::handle_error; // Find the callback, falling back to our default as a last resort auto callback = find_matching_functor(args..., default_handler); } {% endhighlight %} ## Detecting invalid signatures on callbacks This actually happened to me while writing tests. I wrote a test for this which worked fine and then I changed the required signature for two of the events and my tests failed on runtime. My code didn't detect if the user provided an invalid signature so it would silently ignore it and use its own default instead. This can be easily solved making sure every provided callback matches one of the signatures: {% highlight cpp %} // Check that a given functor matches at least one of the // expected signatures template void check_callback_matches(const Functor& functor) { static_assert( !is_same::type>::value || !is_same::type>::value || !is_same::type>::value || !is_same::type>::value, "Callback doesn't match any of the expected signatures" ); } // Base case for recursion void check_callbacks_match() { } // Check that all given functors match at least one of the // expected signatures template void check_callbacks_match(const Functor& functor, const Functors&... functors) { // Check this one check_callback_matches(functor); // Then check the rest check_callbacks_match(functors...); } {% endhighlight %} ## Putting it all together Now, after putting all of this together, we can just provide our callbacks and have the class detect which tries to handle what. As a result, we can use this dispatcher class in the following way: {% highlight cpp %} // Construct the dispatcher ConsumerDispatcher dispatcher(consumer); // Run and dispatch dispatcher.run( [&](Message msg) { // Do something with this message }, [&](Error error) { // Handle the error } ); {% endhighlight %} Internally this consumes a message and performs all the checks shown in the loop at the beginning of this post, dispatching to each callback when needed. # Conclusion All of this code was used to create the [ConsumerDispatcher](https://github.com/mfontanini/cppkafka/blob/master/include/cppkafka/utils/consumer_dispatcher.h) class in _cppkafka_. I think this is a great improvement over having to write the same loop over an over again. Using callbacks allows easily spotting what happens on each case and let's you only express interest in the specific type of events you want rather than having to handle all of them. One could argue that this reduces readability, as now you need to rely on the documentation to be sure what is the signature you're supposed to use. Moreover, compilation errors will now be a bit less specific as they will be a static assertion claiming that no valid functor could be found but knowing which is the one that's wrong is slightly trickier. Still, I'm happy with the way this works and I'll surely be moving my applications to use it. --- ## File: _posts/2017-12-03-libtins-v4.0.md --- layout: post title: "libtins v4.0 release" --- # libtins v4.0 release [_libtins_](https://libtins.github.io/) is a high-level, multiplatform C++ network packet sniffing and crafting library. In this post I'll be covering the major changes that were made in the _v4.0_ release of the library. This was a major release that fixed several features that either were wrong or could have been implemented in a better way. These changes led to considerable performance improvements on several protocol's parsing code, making _libtins_ slightly faster than it already was. This version broke the _ABI_ compatibility, which means you should rebuild your project when updating your _libtins_ reference. The _API_, on the other hand, wasn't broken except on a couple of spots which hopefully won't affect anyone. In the following sections we'll look into what was changed and the reasoning behind those changes. # Performance improvements One of the changes that I'm most happy about in this release are the ones that led to performance gains. Several small changes provided a considerable speed improvement when parsing packets. The changes that were made and helped achieve this were the following: ## Ban `std::list` as the storage for options Up until this versiom, `std::list` was used to store options in _PDUs_ like _TCP_, _IP_ and _DNS_. This container has now been replaced with `std::vector`, which proved to make parsing those protocols considerably faster. I'm not completely sure why `std::list` was used initially as it's normally a poor choice of a container except for very specific cases. When there are options to be parsed, inserting elements into a `std::vector` outperforms `std::list` as it doesn't need to allocate memory every time a new option is added. The [TCP + options](http://libtins.github.io/benchmark/#bench2) benchmarks is a good test case for this. When comparing the latest version to the previous release, I noticed the former performs about 25% faster than the previous one. Note that this _shouldn't_ break your code unless you were explicitly doing something like: {% highlight cpp %} const TCP tcp = ...; // Ooops, this will fail const list& oops_options = tcp.options(); {% endhighlight %} But this is easily fixable by using `TCP::options_type`: {% highlight cpp %} // Fixed! const TCP::options_type& yay_options = tcp.options(); {% endhighlight %} ## Total option size tracking The total option size tracking in _IP_ and _TCP_ was completely removed and now both the size and required amount of padding are only calculated when serializing a packet. These were basically wasting space and CPU time as there is really no reason for themn to be pre-calculated. Given that there's only at most a handful of options in every packet, it's really not worth the "optimization". The [TCP](http://libtins.github.io/benchmark/#bench1) benchmark performed about 13% better when using the latest version. This should be all attributed to this removal as there's nothing else that was changed that could have caused a performance improvement. # PDU, I am your parent Another change that was introduced is the notion of a parent for every _PDU_. So far there was only a notion of the "next" layer in a packet but there was no way to get the previous one in case it existed. This was okay for the most part except on some cases when while serializing a packet, a layer eeds to know what its parent is in order to compute some value. An example of this is _TCP_, which uses the packet's _IPv4_/_IPv6_ addresses while computing its checksum. As a consequence of this, a pointer to the parent _PDU_ was being passed along while serializing a packet which made for a weird _API_. Now there's no need to have that argument as each _PDU_ knows at all times what its parent is. Note that this probably added a slight performance penalty due to the fact that the base size for every _PDU_ is now 8 bytes (in x64) larger but it is probably negligible compared to the other optimization improvements performed. After adding this I figured it wouldn't be a bad idea to implement a `PDUIterator` class that represents a bidirectional iterator over every `PDU` present in a packet. This makes it slightly easier to go through each of them without having to deal with pointers at all: {% highlight cpp %} const PDU packet = ...; // Let's iterate all PDUs in this packet for (const PDU& pdu : iterate_pdus(packet)) { // Print the type of this PDU as a string cout << Tins::Utils::to_string(pdu.pdu_type()) << "\n"; } {% endhighlight %} # Compilation time improvements A significant amount of effort was put into making _libtins_ build faster, both while building library itself and any application that uses it. In order to achieve this, some headers like `internals.h` and `utils.h` were split into multiple files. These had started just containing a couple of functions and classes but grew too much over time and ended up having a considerable amount of code and pulling in several header files. Comparing version _4.0_ with the previous one, _3.5_, I can see the latest one builds 17% faster in my local machine. This number is fairly relative but any user of the library should see some sort of build time improvement when building their project. # RadioTap parsing/serializing The parsing and serializing of _RadioTap_ was initially implemented in a really awful way. The [format](http://www.radiotap.org/#detail) of his protocol is tricky, which makes it non-trivial to parse. The initial approach involved basically hardcoding the parsing and storing of every individual field, keeping every one of them on a separate member using the storage type for them. This meant that every time the parsing of a new field was added, it would break the _ABI_ as the member in which the field was stored would be added on demand. In version _4.0_ this was re-written from scratch using a generic parser (`RadioTapParser`) and writer (`RadioTapWriter`). The entire portion of the `RadioTap` header that contains the options is stored in a `std::vector` and only parsed on demand. This means that if you don't care about using `RadioTap` options at all then you won't be paying for the price to parse them. This also allows iterating and processing every option in an easy way: {% highlight cpp %} RadioTap radio = ...; // Construct a parser around the buffer where the options are stored RadioTapParser parser(radio.options_payload()); // Now iterate through all fields while (parser.has_fields()) { cout << "Option type: " << static_cast(parser.current_field()) << "\n"; // Fetch the current option const RadioTap::option option = parser.current_option(); // Now we can convert this option to the right type // using RadioTap::option::to, which will obviously // will depend on the type of option it is // Advance to the next field parser.advance_field(); } {% endhighlight %} Note that as a consequence of the current design, it is pretty inefficient to fetch fields individually as each of them will involve an `O(N)` search over the entire options buffer. If you need to fetch multiple fields, just iterate through the payload like shown above and process them. # Removing HWAddress storage template parameter `HWAddress` had a second template parameter (besides the one used to store the length of the address) that indicated which storage type to use for each element stored (defaulting to `uint8_t`). This is not really useful as I've yet to found a case where I'd like to store a hardware address where each element is larger than a byte. This was also requiring to include some "heavy" headers in `hwaddress.h` which slowed down compilation times considerably for any translation unit that included it. This template parameter has now been removed and the storage is always `uint8_t`. This will break your code if you were forward declaring the class. If this is the case, I'm sorry but that had to go away for good.