# Technical Documentation: sogou/workflow > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/sogou/workflow) ### Overview (`README.md`) [简体中文版（推荐）](README_cn.md) ## Sogou C++ Workflow [](https://github.com/sogou/workflow/blob/master/LICENSE) [](https://en.cppreference.com/) [](https://img.shields.io/badge/platform-linux%20%7C%20macos20%7C%20windows-lightgrey.svg) [](https://github.com/sogou/workflow/actions?query=workflow%3A%22ci+build%22++) As **Sogou\`s C++ server engine**, Sogou C++ Workflow supports almost all **back-end C++ online services** of Sogou, including all search services, cloud input method, online advertisements, etc., handling more than **10 billion** requests every day. This is an **enterprise-level programming engine** in light and elegant design which can satisfy most C++ back-end development requirements. #### You can use it: * To quickly build an **HTTP server**: ~~~cpp #include #include "workflow/WFHttpServer.h" int main() { WFHttpServer server([](WFHttpTask *task) { task->get_resp()->append_output_body("Hello World!"); }); if (server.start(8888) == 0) { // start server on port 8888 getchar(); // press "Enter" to end. server.stop(); } return 0; } ~~~ * As a **multifunctional asynchronous client**, it currently supports `HTTP`, `Redis`, `MySQL` and `Kafka` protocols. * ``MySQL`` protocol supports ``MariaDB``, ``TiDB`` as well. * To implement **client/server on user-defined protocol** and build your own **RPC system**. * [srpc](https://github.com/sogou/srpc) is based on it and it is an independent open source project, which supports srpc, brpc, trpc and thrift protocols. * To build **asynchronous workflow**; support common **series** and **parallel** structures, and also support any **DAG** structures. * As a **parallel computing tool**. In addition to **networking tasks**, Sogou C++ Workflow also includes **the scheduling of computing tasks**. All types of tasks can be put into **the same** flow. * As an **asynchronous file IO tool** in `Linux` system, with high performance exceeding any system call. Disk file IO is also a task. * To realize any **high-performance** and **high-concurrency** back-end service with a very complex relationship between computing and networking. * To build a **micro service** system. * This project has built-in **service governance** and **load balancing** features. * Wiki link : [PaaS Architecture](https://github.com/sogou/workflow/wiki) #### Compiling and Running Environment * This project supports `Linux`, `macOS`, `Windows`, `Android` and other operating systems. * `Windows` version is currently released as an independent [branch](https://github.com/sogou/workflow/tree/windows), using `iocp` to implement asynchronous networking. All user interfaces are consistent with the `Linux` version. * Supports all CPU platforms, including 32 or 64-bit `x86` processors, big-endian or little-endian `arm` processors, `loongson` processors. * Master branch requires `OpenSSL 1.1` or above, and BoringSSL is fully compatible. If you don't like SSL, you may checkout the [nossl](https://github.com/sogou/workflow/tree/nossl) branch. * Uses the `C++11` standard and therefore, it should be compiled with a compiler which supports `C++11`. Does not rely on `boost` or `asio`. * No other dependencies. However, if you need `Kafka` protocol, some compression libraries should be installed, including `lz4`, `zstd` and `snappy`. ### Get Started (Linux, macOS): ~~~sh git clone https://github.com/sogou/workflow cd workflow make cd tutorial make ~~~~ #### With SRPC Tool (NEW!)： https://github.com/sogou/srpc/blob/master/tools/README.md #### With [apt-get](https://launchpad.net/ubuntu/+source/workflow) on Debian Linux, ubuntu: Sogou C++ Workflow has been packaged for Debian Linux and ubuntu 22.04. To install the Workflow library for development purposes: ~~~~sh sudo apt-get install libworkflow-dev ~~~~ To install the Workflow library for deployment: ~~~~sh sudo apt-get install libworkflow1 ~~~~ #### With [dnf](https://packages.fedoraproject.org/pkgs/workflow) on Fedora Linux: Sogou C++ Workflow has been packaged for Fedora Linux. To install the Workflow library for development purposes: ~~~~sh sudo dnf install workflow-devel ~~~~ To install the Workflow library for deployment: ~~~~sh sudo dnf install workflow ~~~~ #### With xmake If you want to use xmake to build workflow, you can see [xmake build document](docs/en/xmake.md) # Tutorials * Client * [Creating your first task：wget](docs/en/tutorial-01-wget.md) * [Implementing Redis set and get：redis\_cli](docs/en/tutorial-02-redis_cli.md) * [More features about series：wget\_to\_redis](docs/en/tutorial-03-wget_to_redis.md) * Server * [First server：http\_echo\_server](docs/en/tutorial-04-http_echo_server.md) * [Asynchronous server：http\_proxy](docs/en/tutorial-05-http_proxy.md) * Parallel tasks and Series　 * [A simple parallel wget：parallel\_wget](docs/en/tutorial-06-parallel_wget.md) * Important topics * [About error](docs/en/about-error.md) * [About timeout](docs/en/about-timeout.md) * [About global configuration](docs/en/about-config.md) * [About DNS](docs/en/about-dns.md) * [About exit](docs/en/about-exit.md) * Computing tasks * [Using the build-in algorithm factory：sort\_task](docs/en/tutorial-07-sort_task.md) * [User-defined computing task：matrix\_multiply](docs/en/tutorial-08-matrix_multiply.md) * [Use computing task in a simple way: go task](docs/en/about-go-task.md) * Asynchronous File IO tasks * [Http server with file IO：http\_file\_server](docs/en/tutorial-09-http_file_server.md) * User-defined protocol * [A simple user-defined protocol: client/server](docs/en/tutorial-10-user_defined_protocol.md) * [Use TLV message](docs/en/about-tlv-message.md) * Other important tasks/components * [About timer](docs/en/about-timer.md) * [About counter](docs/en/about-counter.md) * [About resource pool](docs/en/about-resource-pool.md) * [About module](docs/en/about-module.md) * [About DAG](docs/en/tutorial-11-graph_task.md) * Service governance * [About service governance](docs/en/about-service-governance.md) * [More documents about upstream](docs/en/about-upstream.md) * Connection context * [About connection context](docs/en/about-connection-context.md) * Built-in clients * [Asynchronous MySQL client：mysql\_cli](docs/en/tutorial-12-mysql_cli.md) * [Asynchronous Kafka client: kafka\_cli](docs/en/tutorial-13-kafka_cli.md) #### Programming Paradigm Program = Protocol + Algorithm + Workflow * Protocol * In most cases, users use built-in common network protocols, such as HTTP, Redis or various rpc. * Users can also easily customize user-defined network protocol. In the customization, they only need to provide serialization and deserialization functions to define their own client/server. * Algorithm * In our design, the algorithm is a concept symmetrical to the protocol. * If protocol call is rpc, then algorithm call is an apc (Async Procedure Call). * We have provided some general algorithms, such as sort, merge, psort, reduce, which can be used directly. * Compared with a user-defined protocol, a user-defined algorithm is much more common. Any complicated computation with clear boundaries should be packaged into an algorithm. * Workflow * Workflow is the actual business logic, which is to put the protocols and algorithms into the flow graph for use. * The typical workflow is a closed series-parallel graph. Complex business logic may be a non-closed DAG. * The workflow graph can be constructed directly or dynamically generated based on the results of each step. All tasks are executed asynchronously. Structured Concurrency and Task Abstraction * Our system contains five basic tasks: communication, computation, file IO, timer, and counter. * All tasks are generated by the task factory, and users organize the concurrency structure by calling interfaces, such as series, parallel, DAG, etc. * In most cases, the tasks generated by the user through the task factory is a complex task which encapsulates multiple asynchronous processes, but it is transparent to the user. * For example, an HTTP request may include many asynchronous processes (DNS, redirection), but for user, it is just a networking task. * File sorting seems to be an algorithm, but it actually includes many complex interaction processes between file IO and CPU computation. * If you think of business logic as building circuits with well-designed electronic components, then each electronic component may be a complex circuit. * The task abstraction mechanism greatly reduces the number of tasks users need to create and the depth of callbacks. * Any task runs in a **SeriesWork** and the tasks in the same SeriesWork shares the series context, which simplifies data transfer between asynchronous tasks. Callback and Memory Reclamation Mechanism * All calls are executed asynchronously, and there is almost no operation that occupies a thread. * Explicit callback mechanism. Users are aware that they are writing asynchronous programs. * **A set of object lifecycle mechanisms greatly simplifies memory management for asynchronous programs.** * The lifecycle of any task created by the framework is from creation until the callback function finishes running. There is no risk of leakage. * If a task is created but the user does not want to run it, the user needs to release it through the `dismiss()` interface. * Any data in the task, such as the response of the network request, will also be recycled with the task. At this time, the user can use `std::move()` to move the required data. * The project doesn’t use `std::shared_ptr` to manage memory. * We try to avoid user's derivations, and encapsulate user behavior with `std::function` instead, including: * The callback of any task. * Any server's process. This conforms to the `FaaS` (Function as a Service) idea. * The realization of an algorithm is simply a `std::function`. But the algorithm can also be implemented by derivation. * If used deeply, one will find that everything can be derived. #### Any Other Questions? You may check the [FAQ](https://github.com/sogou/workflow/issues/406) and [issues](https://github.com/sogou/workflow/issues) list first to see if you can find the answer. You are very welcome to send the problems you encounter in use to [issues](https://github.com/sogou/workflow/issues), and we will answer them as soon as possible. At the same time, more issues will also help new users. --- ### Overview (`benchmark/README.md`) # 性能测试 Sogou C++ Workflow是一款性能优异的网络框架，本文介绍我们进行的性能测试， 包括方案、代码、结果，以及与其他同类产品的对比。 更多场景下的实验正在进行中，本文将持续更新。 ## HTTP Server HTTP Client/Server是Sogou C++ Workflow常见的应用场景， 我们首先对Server端进行实验。 ### 环境 我们部署了两台相同机器作为Server和Client，软硬件配置如下： | 软硬件 | 配置 | |:---:|:---| | CPU | 40 Cores, x86_64, Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz | | Memory | 192GB | | NIC | 25000Mbps | | OS | CentOS 7.8.2003 | | Kernel | Linux version 3.10.0-1127.el7.x86_64 | | GCC | 4.8.5 | 两者间`ping`测得的RTT为0.1ms左右。 ### 对照组 我们选择nginx和brpc作为对照组。 选择前者是因为它在生产中部署十分广泛，性能不俗； 对于后者，我们在本次实验中只关注HTTP Server方面的能力， 其他的特性已有[单独的实验][Sogou RPC Benchmark]进行更为详尽的测试。 事实上，我们也对此二者之外的其他某些框架同时进行了实验， 但结果其性能表现相差较远，因此未在本文中体现。 后续我们将选取更多合适的框架加入对比测试中。 ### Client工具 本次实验我们使用的压测工具为[wrk][wrk]和[wrk2][wrk2]。 前者适合测试特定并发下的QPS极限和延时， 后者适合在特定QPS下测试延时分布。 我们也尝试过使用其他测试工具，例如[ab][ab]等，但无法打出足够的压力。 有鉴于此，我们也在着手开发基于Sogou C++ Workflow的benchmark工具。 ### 变量和指标 一般而言，对网络框架的性能测试，切入的角度可谓纷繁多样。 通过控制不同的变量、观测不同的指标，可以探究程序在不同场景下的适应能力。 本次实验，我们选择其中最普遍常见的变量和指标： 通过控制Client并发度和承载数据的大小，来测试QPS和延时的变化情况。 另外，我们还测试了在掺杂慢请求的正常请求的延时分布。 下面依次介绍两个测试场景。 ### 测试方法 #### 启动http server 1. 编译benchmark 2. 进入到benchmark目录，执行 ``` ./http_server 12 9000 11 ``` 说明: 启动参数分别为线程数、端口和响应的随机字符串长度。 ### wrk测试 ``` wrk --latency -d10 -c200 --timeout 8 -t 6 http://127.0.0.1:9000 ``` **命令行解释** -c200: 启动200个连接 -t6: 开启6个线程做压力测试 -d10: 压测持续10s --timeout 8: 连接超时时间8s ### 不同并发度和数据长度下的QPS和延时 #### 代码和配置 我们搭建了一个极其简约的HTTP服务器， 忽略掉所有的业务逻辑， 将测试点聚焦在纯粹的网络框架性能上。 代码片段如下， 完整代码移步[这里][benchmark-01 Code]。 ```cpp // ... auto * resp = task->get_resp(); resp->add_header_pair("Date", timestamp); resp->add_header_pair("Content-Type", "text/plain; charset=UTF-8"); resp->append_output_body_nocopy(content.data(), content.size()); // ... ``` 可以从上述代码中看到， 对于到来的任何HTTP请求， 我们都会返回一段固定的内容作为Body， 并设置必要的Header， 包括代码中指明的`content-type`、`date`， 以及自动填充的`connection`和`content-length`。 HTTP Body的固定内容是在Server启动时随机生成的ASCII字符串， 其长度可以通过启动参数配置。 同时可以配置的还有使用的poller线程数和监听的端口号。 前者我们在本次测试中固定为16， 因此Sogou C++ Workflow将使用16个poller线程和20个handler线程（默认配置）。 对于nginx和brpc， 我们也构建了相同的返回内容， 并为nginx配置了40个进程、 brpc配置了40个线程。 #### 变量 我们控制并发度在`[1, 2K]`之间翻倍增长， 数据长度在`[16B, 64KB]`之间翻倍增长， 两者正交。 #### 指标 鉴于并发度和数据长度组合之后数量较多， 我们选择其中部分数据绘制为曲线。 ##### 固定数据长度下QPS与并发度关系 ![Concurrency and QPS][Con-QPS] 上图可以看出，当数据长度保持不变， QPS随着并发度提高而增大，后趋于平稳。 此过程中Sogou C++ Workflow一直有明显优势， 高于brpc和nginx。 特别是数据长度为64和512的两条曲线， 并发度足够的时候，可以保持500K的QPS。 注意上图中nginx-64与nginx-512的曲线重叠度很高， 不易辨识。 ##### 固定并发度下QPS与数据长度关系 ![Body Length and QPS][Len-QPS] 上图可以看出，当并发度保持不变， 随着数据长度的增长， QPS保持平稳至4K时下降。 此过程中，Sogou C++ Workflow也一直保持优势。 ##### 固定数据长度下延时与并发度关系 ![Concurrency and Latency][Con-Lat] 上图可以看出，保持数据长度不变， 延时随并发度提高而有所上升。 此过程中，Sogou C++ Workflow略好于brpc， 大好于nginx。 ##### 固定并发度下延时与数据长度关系 ![Body Length and Latency][Len-Lat] 上图可以看出，并发度保持不变时， 增大数据长度，造成延时上升。 此过程中，Sogou C++ Workflow好于nginx， 好于brpc。 ### 掺杂慢请求的延时分布 #### 代码 我们在上一个测试的基础上，简单添加了一个慢请求的逻辑， 模拟业务场景中可能出现的特殊情况。 代码片段如下， 完整代码请移步[这里][benchmark-02 Code]。 ```cpp // ... if (std::strcmp(uri, "/long_req/") == 0) { auto timer_task = WFTaskFactory::create_timer_task(microseconds, nullptr); series_of(task)->push_back(timer_task); } // ... ``` 我们在Server的process里进行判断， 如果访问的是特定的路径， 则添加一个`WFTimerTask`到Series的末尾， 能够模拟一个异步耗时处理过程。 类似地，对brpc使用`bthread_usleep()`函数进行异步睡眠。 #### 配置 在本次实验中，我们固定并发度为1024，数据长度为1024字节， 分别以QPS为20K、100K和200K进行正常请求测试， 测绘延时； 与此同时，有另一路压力，进行慢请求， QPS是上述QPS的1%， 数据不计入统计。 慢请求的时长固定为5ms。 #### 延时CDF图 ![Latency CDF][Lat CDF] 从上图可以看出，当QPS为20K时， Sogou C++ Workflow略次于brpc； 当QPS为100K时，两者几乎相当； 当QPS为200K时，Sogou C++ Workflow略好于brpc。 总之，可以认为两者在这方面旗鼓相当。 [Sogou RPC Benchmark]: https://github.com/holmes1412/sogou-rpc-benchmark [wrk]: https://github.com/wg/wrk [wrk2]: https://github.com/giltene/wrk2 [ab]: https://httpd.apache.org/docs/2.4/programs/ab.html [benchmark-01 Code]: benchmark-01-http_server.cc [benchmark-02 Code]: benchmark-02-http_server_long_req.cc [Con-QPS]: https://raw.githubusercontent.com/wiki/sogou/workflow/img/benchmark-01.png [Len-QPS]: https://raw.githubusercontent.com/wiki/sogou/workflow/img/benchmark-02.png [Con-Lat]: https://raw.githubusercontent.com/wiki/sogou/workflow/img/benchmark-03.png [Len-Lat]: https://raw.githubusercontent.com/wiki/sogou/workflow/img/benchmark-04.png [Lat CDF]: https://raw.githubusercontent.com/wiki/sogou/workflow/img/benchmark-05.png --- ### About Conditional (`docs/about-conditional.md`) # 条件任务与观察者模式 有的时候，我们需要让任务在某个条件下才被执行。条件任务（WFConditional）就是用于解决这种问题。 条件任务是一种任务包装器，可以包装任何的任务并取代原任务。通过对条件任务发送信号来触发被包装任务的执行。 # 条件任务的创建 在[WFTaskFactory.h](/src/factory/WFTaskFactory.h)里，可以看到条件任务的创建接口。 ~~~cpp class WFTaskFactory { public: static WFConditional *create_conditional(SubTask *task); static WFConditional *create_conditional(SubTask *task, void **msgbuf); }; ~~~ 可以看到，我们通过工厂的create_conditional接口创建条件任务。 其中，task为被包装的任务。msgbuf是用于接收消息的缓冲区，如果无需关注消息的具体内容，msgbuf可以缺省。 WFConditional的主要接口： ~~~cpp class WFConditional : public WFGenericTask { public: virtual void signal(void *msg); ... }; ~~~ WFConditional是一种任务，所以，它满足普通workflow任务的一切属性。特别的接口只有signal，用于发送信号。 # 示例 以下示例，通过timer和conditional，实现一个延迟1秒执行的计算任务。 ~~~cpp int main() { WFGoTask *task = WFTaskFactory::create_go_task("test", [](){ printf("Done\n"); }); WFConditional *cond = WFTaskFactory::create_conditional(task); WFTimerTask *timer = WFTaskFactory::create_timer_task(1, 0, [cond](void *){ cond->signal(NULL); }); timer->start(); cond->start(); getchar(); } ~~~ 这个示例里，在定时器的回调里向cond发送信号，让被包装的go task可以被执行。 注意，无论cond->signal()与cond->start()哪一个先被调用，程序都完全正确。 # 观察者模式 我们看到，如果直接对cond发送信息，需要发送者直接持有cond的指针，这在一些情况下并不是很方便。 于是，我们引入了观察者模式，也就是命名的条件任务。通过向某个名称发送信号，同时唤醒所有在这个名称下的条件任务。 命名条件任务的创建与唤醒： ~~~cpp class WFTaskFactory { public: static WFConditional *create_conditional(const std::string& cond_name, SubTask *task); static WFConditional *create_conditional(const std::string& cond_name, SubTask *task, void **msgbuf); static int signal_by_name(const std::string& cond_name, void *msg); static int signal_by_name(const std::string& cond_name, void *msg, size_t max); template static int signal_by_name(const std::string& cond_name, T *const msg[], size_t max); }; ~~~ 我们看到，与普通条件任务唯一区别是，命名条件任务创建时，需要传入一个cond_name。 而signal_by_name()接口，默认将msg发送到所有在这个名称上等待的条件任务，将它们全部唤醒。 也可以通过max参数指定唤醒的最大任务数。此时，msg还可以是一个指针数组，可给不同的条件任务发送不同的消息。 任何一个signal_by_name的重载函数，其返回值都是表示实际唤醒的条件任务个数。 这就相当于实现了观察者模式。 # 示例 还是上面的延迟计算示例，我们增加到两个计算任务并用观察者模式来实现。用"slot1"作为条件任务名。 ~~~cpp int main() { WFGoTask *task1 = WFTaskFactory::create_go_task("test", [](){ printf("test1 done\n"); }); WFGoTask *task2 = WFTaskFactory::create_go_task("test", [](){ printf("test2 done\n"); }); WFConditional *cond1 = WFTaskFactory::create_conditional("slot1", task1); WFConditional *cond2 = WFTaskFactory::create_conditional("slot1", task2); WFTimerTask *timer = WFTaskFactory::create_timer_task(1, 0, [](void *){ WFTaskFactory::signal_by_name("slot1", NULL); }); timer->start(); cond1->start(); cond2->start(); getchar(); } ~~~ 我们看到，在这个示例里，timer在回调中通过signal_by_name方法，同时唤醒了slot1下两个计算任务。 # 使用条件任务注意事项 Workflow里的任何任务，如果创建之后不想运行，都可以通过dismiss接口直接释放。 对于条件任务，如果要被dismiss（或者在某个被cancel的series里），必须保证这个条件任务没有被signal过。 以下代码的行为无定义： ~~~cpp int main() { WFEmptyTask *task = WFTaskFactory::create_empty_task(); WFConditional *cond = WFTaskFactory::create_conditional("slot1", task); WFTimerTask *timer = WFTaskFactory::create_timer_task(0, 0, [](void *) { WFTaskFactory::signal_by_name("slot1"); }); timer->start(); cond->dismiss(); // 取消任务 getchar(); } ~~~ 显然，如果timer的callback里已经执行或正在执行了signal_by_name，cond被signal，再dismiss()是一种错误行为。 这种情况一般也只会出现在命名条件任务里。所以，dismiss一个命名条件任务，需要特别的小心。 --- ### About Config (`docs/about-config.md`) # 关于全局配置 全局配置用于配置全局默认参数，以适应的实际业务需求，提升程序性能。 全局配置的修改必须在使用框架任何调用之前，否则修改可能无法生效。 另外，一些全局配置选项，可以被upstream配置覆盖。这部分请参考upstream相关文档。 # 修改默认配置 在[WFGlobal.h](../src/manager/WFGlobal.h)里，包含了全局配置的结构体与默认值： ~~~cpp struct WFGlobalSettings { struct EndpointParams endpoint_params; struct EndpointParams dns_server_params; unsigned int dns_ttl_default; ///< in seconds, DNS TTL when network request success unsigned int dns_ttl_min; ///< in seconds, DNS TTL when network request fail int dns_threads; int poller_threads; int handler_threads; int compute_threads; ///< auto-set by system CPU number if value<=0 int fio_max_events; const char *resolv_conf_path; const char *hosts_path; }; static constexpr struct WFGlobalSettings GLOBAL_SETTINGS_DEFAULT = { .endpoint_params = ENDPOINT_PARAMS_DEFAULT, .dns_server_params = ENDPOINT_PARAMS_DEFAULT, .dns_ttl_default = 12 * 3600, .dns_ttl_min = 180, .dns_threads = 4, .poller_threads = 4, .handler_threads = 20, .compute_threads = -1, .fio_max_events = 4096, .resolv_conf_path = "/etc/resolv.conf", .hosts_path = "/etc/hosts", }; ~~~ 其中EndpointParams结构体和默认值在[EndpointParams.h](../src/manager/EndpointParams.h)文件里： ~~~cpp struct EndpointParams { size_t max_connections; int connect_timeout; int response_timeout; int ssl_connect_timeout; bool use_tls_sni; }; static constexpr struct EndpointParams ENDPOINT_PARAMS_DEFAULT = { .max_connections = 200, .connect_timeout = 10 * 1000, .response_timeout = 10 * 1000, .ssl_connect_timeout = 10 * 1000, .use_tls_sni = false, }; ~~~ 举个例子，把默认的连接超时改为5秒，dns默认ttl改为1小时，用于消息反序列化的poller线程增加到10个： ~~~cpp #include "workflow/WFGlobal.h" int main() { struct WFGlobalSettings settings = GLOBAL_SETTINGS_DEFAULT; settings.endpoint_params.connect_timeout = 5 * 1000; settings.dns_ttl_default = 3600; settings.poller_threads = 10; WORKFLOW_library_init(&settings); ... } ~~~ 大多数参数的意义都比较清晰。注意dns ttl相关参数，单位是**秒**。endpoint相关超时参数单位是**毫秒**，并且可以用-1表示无限。 dns_threads表示并行访问dns的线程数。但目前我们默认使用我们自己的异步DNS解析，所以并不会创建DNS线程（Window平台除外）。 dns_server_params表示是我们访问DNS server的参数，包括最大并发连接，以及连接与响应超时。 compute_threads表示用于计算的线程数，默认-1代表与当前节点CPU核数相同。 fio_max_events是异步文件IO的最大并发事件数。 resolv_conf_path是dns配置文件的路径，unix平台下默认为"/etc/resolv.conf"。Windows下默认为NULL，将使用多线程dns解析。 hosts_path是hosts文件路径。unix平台下默认为"/etc/hosts“。只有配置了resolv_conf_path，这个配置才起作用。 与网络性能相关的两个参数为poller_threads和handler_threads： * poller线程主要负责epoll（kqueue）和消息反序列化。 * handler线程是网络任务callback和process所在线程。 所有框架需要的资源，都是在第一次被使用时才申请的。例如用户没有用到dns解析，那么异步dns解析器或dns线程不会被启动。 --- ### About Connection Context (`docs/about-connection-context.md`) # 关于连接上下文 连接上下文是使用本框架编程的一个高级课题。使用上会有一些复杂性。 从之前的示例里可以看出，无论是client还是server任务，我们并没有手段指定使用的具体连接。 但是有一些业务场景，特别是server端，可能是需要维护连接状态的。也就是说我们需要把一段上下文和连接绑定。 我们的框架里，是提供了连接上下文机制给用户使用的。 # 连接上下文的应用场景 http协议可以说是一种完全无连接状态的协议，http会话，是通过cookie来实现的。这种协议对于我们的框架最友好。类似的还有kafka。 而redis和mysql的连接则是明显带状态，redis通过SELECT命令，指定当前连接上的数据库ID。mysql则是一个彻彻底底的有状态连接。 使用框架的redis或非事务mysql client任务时，由于URL里已经包含了所有和连接选择有关的信息，包括： * 用户名密码 * 数据库名或数据库号 * mysql的字符集 框架会根据这些信息自动登录和选择可复用的连接，用户无需关心连接上下文的问题。 这也是为什么，框架里redis的SELECT命令和mysql的USE命令是禁止用户使用的，切换数据库需要用新的URL创建任务。 事务型mysql，可以固定连接，这部分内容请参考mysql相关文档。 但是，如果我们实现一个redis协议的server，那我们需要知道当前连接上的状态了。 此外，我们还可以通过连接上下文件被释放的事件来感知连接被远端关闭。 # 使用连接上下文的方法 我们需要强调的是，一般情况下只有server任务需要使用连接上下文，并且只需要在process函数内部使用，这也是最安全最简单的用法。 但是，任务在callback里也可以使用或修改连接上下文，只是使用的时候需要考虑并发的问题。我们会详细地讨论相关问题。 任何网络任务都可以调用接口获得连接对象，进而获得或修改连接上下文。在[WFTask.h](../src/factory/WFTask.h)里，调用如下： ~~~cpp template class WFNetworkTask : public CommRequest { public: virtual WFConnection *get_connection() const = 0; ... }; ~~~ 文件[WFConneciton.h](../src/factory/WFConnection.h)里，包含了对连接对象的操作接口： ~~~cpp class WFConnection : public CommConnection { public: void *get_context() const; void set_context(void *context, std::function deleter); void set_context(void *context); void *test_set_context(void *test_context, void *new_context, std::function deleter); void *test_set_context(void *test_context, void *new_context); }; ~~~ get_connection()只可在process或callback里调用，而且如果callback里调用，需要检查返回值是否为NULL。 如果成功取得WFConnection对象，就可以操作连接上下文了。连接上下文是一个void *指针。 设置连接上下文可以同时传入deleter函数，在连接被关闭时，deleter被自动调用。 如果调用无deleter参数的接口，可以只设置新的上下文，保持原有的deleter不变。 # 访问连接上下文的时机和并发问题 client task被创建的时候，连接对象没有确定，因此所有client task对连接上下文的使用只有在callback里。 server task可能在两个地方使用连接上下文，process和callback。 在callback里使用连接上下文时，需要考虑并发问题，因为同一个连接，会被多个task复用，并且同时运行到callback。 所以，我们推荐只process函数里访问或修改连接上下文，process过程中连接不会被复用或释放，是最简单安全的方法。 注意，我们指的process只包括process函数内部，在process函数结束后，callback之前，get_connection调用一律返回NULL。 WFConnection的test_set_context()，就是为了解决callback里使用连接上下文是的并发问题，但我们不推荐使用。 总之，如果你不是对系统实现非常了解，请只在server task的process函数里使用连接上下文。 # 示例：减少Http/1.1的请求header传输 http协议可以说是一个连接无状态的协议，同一个连接上，每一次请求都必须发送完整的header。 假设请求里的cookie非常大，那么这显然就增加了很大的数据传输量。我们可以通过server端连接上下文来解决这个问题。 我们约定http request里的cookie，对本连接上所有后续请求有效，后续请求header里可以不再发送cookie。 以下是server端代码： ~~~cpp void process(WFHttpTask *server_task) { protocol::HttpRequest *req = server_task->get_req(); protocol::HttpHeaderCursor cursor(req); WFConnection *conn = server_task->get_connection(); void *context = conn->get_context(); std::string cookie; if (cursor.find("Cookie", cookie)) { if (context) delete (std::string *)context; context = new std::string(cookie); conn->set_context(context, [](void *p) { delete (std::string *)p; }); } else if (context) cookie = *(std::string *)context; ... } ~~~ 通过这种方式，与client端约定好每次只在连接的第一个请求传输cookie，就可以实现流量的节省。 client端的实现需要用到一个新的回调函数，用法如下： ~~~cpp using namespace protocol; void prepare_func(WFHttpTask *task) { if (task->get_task_seq() == 0) task->get_req()->add_header_pair("Cookie", my_cookie); } int some_function() { WFHttpTask *task = WFTaskFactory::create_http_task(...); task->set_prepare(prepare_func); ... } ~~~ 在这个示例中，当http task是连接上的首个请求时，我们设置了cookie。如果不是首个请求，根据约定，不再设置cookie。 另外，prepare函数里，可以安全的使用连接上下文。同一个连接上，prepare不会并发。 --- ### About Counter (`docs/about-counter.md`) # 关于计数器 计数器是我们框架中一种非常重要的基础任务，计数器本质上是一个不占线程的信号量。 计数器主要用于工作流的控制，包括匿名计数器和命名计数器两种，可以实现非常复杂的业务逻辑。 # 计数器的创建 由于计数器也是一种任务，它的创建同样通过WFTaskFactory来完成，包括两种创建方法： ~~~cpp using counter_callback_t = std::function; class WFTaskFactory { ... static WFCounterTask *create_counter_task(unsigned int target_value, counter_callback_t callback); static WFCounterTask *create_counter_task(const std::string& counter_name, unsigned int target_value, counter_callback_t callback); ... }; ~~~ 每个计数器都包含一个target_value，当计数器的计数到达target_value，callback被调用。 以上两个接口分别产生匿名计数器和命名计数器，匿名计数器直接通过WFCounterTask的count方法来增加计数： ~~~cpp class WFCounterTask { public: virtual void count() { ... } ... } ~~~ 如果创建计数器时，传入一个counter_name，则产生一个命名计数器，可以通过count_by_name函数来增加计数。 # 用匿名计数器实现任务并行 在[并行抓取](./tutorial-06-parallel_wget.md)的示例中，我们通过创建一个ParallelWork来实现多个series并行。 通过ParallelWork和SeriesWork的组合，可以构建任意的串并连图，已经可以满足大多数应用场景需求。 而计数器的存在，可以让我们构建更复杂的任务依赖关系，比如实现一个全连接的神经网络。 以下简单的代码，可代替ParallelWork，实现一个并行的http抓取。 ~~~cpp void http_callback(WFHttpTask *task) { /* Save http page. */ ... WFCounterTask *counter = (WFCounterTask *)task->user_data; counter->count(); } std::mutex mutex; std::condition_variable cond; bool finished = false; void counter_callback(WFCounterTask *counter) { mutex.lock(); finished = true; cond.notify_one(); mutex.unlock(); } int main(int argc, char *argv[]) { WFCounterTask *counter = create_counter_task(url_count, counter_callback); WFHttpTask *task; std::string url[url_count]; /* init urls */ ... for (int i = 0; i < url_count; i++) { task = create_http_task(url[i], http_callback); task->user_data = counter; task->start(); } counter->start(); std::unique_lock lock(mutex); while (!finished) cond.wait(lock); lock.unlock(); return 0; } ~~~ 以上创建一个目标值为url_count的计数器，每个http任务完成之后，调用一次count。 注意，匿名计数器的count次数不可以超过目标值，否则counter可能已经callback销毁了，程序行为无定义。 counter->start()调用可以放在for循环之前。counter只要被创建，就可以调用其count接口，无论counter是否已经启动。 匿名计数器的count接口调用，也可以写成counter->WFCounterTask::count(); 在非常注重性能的应用下可以这么用。 # Server与其它异步引擎结合使用 某些情况下，我们的server可能需要调用非本框架的异步客户端等待结果。简单的方法我们可以在process里同步等待，通过条件变量来唤醒。 这么做的缺点是我们占用了一个处理线程，把其它框架的异步客户端变为同步客户端。但通过counter，我们可以不占线程地等待。 方法很简单： ~~~cpp void some_callback(void *context) { protocol::HttpResponse *resp = get_resp_from_context(context); WFCounterTask *counter = get_counter_from_context(context); /* write data to resp. */ ... counter->count(); } void process(WFHttpTask *task) { WFCounterTask *counter = WFTaskFactory::create_counter_task(1, nullptr); SomeOtherAsyncClient client(some_callback, context); *series_of(task) << counter; } ~~~ 在这里，我们可以把server任务所在的series理解为一个协程，而目标值为1的counter，可以理解为一个条件变量。 Counter的缺点是count操作不传递数据。如果业务有数据传达的需求，可以使用[Mailbox任务](https://github.com/sogou/workflow/blob/master/src/factory/WFTaskFactory.h#L268)。 # 命名计数器 对匿名计数器进行count操作时，直接访问了counter对象指针。这就必然要求在操作时，调用count的次数不超过目标值。 但想象这样一个应用场景，我们同时启动4个任务，只要其中有任意3个任务完成，工作流就可以继续进行。 我们可以用一个目标值为3的计数器，每个任务完成之后，count一次，这样只要任务3个任务完成，计数器就被callback。 但这样的问题是，当第4个任务完成，再调用counter->count()的时候，计数器已经是一个野指针了，程序崩溃。 这时候我们可以用命名计数器来解决这个问题。通过给计数器命名，并通过名字来计数，例如以下实现： ~~~cpp void counter_callback(WFCounterTask *counter) { WFRedisTask *next = WFTaskFactory::create_redis_task(...); series_of(counter)->push_back(next); } int main(void) { WFHttpTask *tasks[4]; WFCounterTask *counter; counter = WFTaskFactory::create_counter_task("c1", 3, counter_callback); counter->start(); for (int i = 0; i < 4; i++) { tasks[i] = WFTaskFactory::create_http_task(..., [](WFHttpTask *task){ WFTaskFactory::count_by_name("c1"); }); tasks[i]->start(); } ... } ~~~ 这个示例中，调起4个并发的http任务，其中3个完成了，立刻启动一个redis任务。实际应用中，可能还需要加入数据传递的代码。 示例中创建命名为"c1"的计数器，在http回调里，使用WFTaskFactory::count_by_name()调用来进行计数。 ~~~cpp class WFTaskFactory { ... static int count_by_name(const std::string& counter_name); static int count_by_name(const std::string& counter_name, unsigned int n); ... }; ~~~ WFTaskFactory::count_by_name方法还可以传入一个整数n，表示这一次操作要增加的计数值，显然： count_by_name("c1")等价于count_by_name("c1", 1)。 如果"c1"计数器不存在（未创建或已经完成），那么对"c1"的操作不产生任何效果，因此不会有匿名计数器野指针的问题。 函数的返回值表示被唤醒的计数器个数。当n大于1时，count_by_name操作可能让多个计数器达到目标值。 # 命名计数器详细行为定义 调用WFTaskFactory::count_by_name(name, n)的时候： * 如果name不存在（未创建或已经完成），无任何行为。 * 如果只有一个名字为name的计数器： * 如果该计数器剩余的值小于或等于n，计数完成，callback被调用，该计数器被销毁。结束。 * 如果计数器剩余值大于n，则计数值加n。结束。 * 如果存在多个同名为name的计数器： * 按照创建顺序，取第一个计数器，假设其剩余值为m： * 如果m值大于n，则计数加n。结束（剩余值为m-n）。 * 如果m小于或等于n，计数完成，callback被调用，第一个计数器被销毁。置n=n-m。 * 如果n为0，结束。 * 如果n大于0，再取出下一个同名计数器，重复整个的操作。 虽然描述很复杂，但总结起来就一句话，按照创建顺序，依次访问所有名字为name的计数器，直到n为0。 也就是说，一次count_by_name(name, n)可以唤醒多个计数器。 用好计数器，可以实现非常复杂的业务逻辑。计数器在我们框架里，往往用于实现异步锁，或者用于任务之间的通道。形态上更像一种控制任务。 --- ### About Dns (`docs/about-dns.md`) # 关于DNS 当使用域名请求网络时，首先需要通过域名解析获取服务器地址，再使用网络地址进行后续的请求。Workflow已经实现了完备的域名解析和缓存系统，通常来说用户无需知晓内部机制即可流畅地发起网络任务。 ## DNS相关配置 Workflow中的全局配置包括 ~~~cpp struct WFGlobalSettings { struct EndpointParams endpoint_params; struct EndpointParams dns_server_params; unsigned int dns_ttl_default; unsigned int dns_ttl_min; int dns_threads; int poller_threads; int handler_threads; int compute_threads; int fio_max_events; const char *resolv_conf_path; const char *hosts_path; }; ~~~ 其中与域名解析相关的配置项有 * dns_server_params * address_family: 该项会在后续展开说明 * max_connections: 向DNS服务器发送请求的最大并发数，默认为200 * connect_timeout/response_timeout/ssl_connect_timeout: 参考[超时](about-timeout.md)相关说明 * dns_threads: 当使用同步方式实现域名解析时，解析操作会在独立的线程池中执行，该项指定线程池的线程数，默认为4 * dns_ttl_default: 域名解析成功的结果会被放到域名缓存中，该项指定其存活时间，单位为秒，默认值1小时，当解析结果过期后会重新解析以获取最新内容 * dns_ttl_min: 当通信失败时，有可能出现缓存的结果已经失效的情况，该项指定一个较短的存活时间，当通信失败时以更频繁的速率更新缓存，单位为秒，默认值1分钟 * resolv_conf_path: 该文件保存了访问DNS相关的配置，在常见的Linux发行版上通常位于`/etc/resolv.conf`，若该项配置为`NULL`则表示使用多线程同步解析的模式 * hosts_path: 该文件是一个本地的域名查找表，若被解析的域名命中该表则不会向DNS发起请求，在常见的Linux发行版上通常位于`/etc/hosts`，若该项配置为`NULL`则表示不使用查找表 ### resolv.conf扩展功能 Workflow对`resolv.conf`配置文件进行了扩展，用户可以通过修改配置以支持`DNS over TLS(DoT)`功能，**注意**直接修改`/etc/resolv.conf`会影响其他进程，可以将该文件复制一份用于修改，并将Workflow的`resolv_conf_path`配置修改为新文件的路径。例如使用`dnss`协议的`nameserver`会通过SSL进行连接 ~~~bash nameserver dnss://8.8.8.8/ nameserver dnss://[2001:4860:4860::8888]/ ~~~ ### Address Family 在某些网络环境下，虽然本机支持IPv6，但因未被分配公网IPv6地址而无法与外部通信（例如本地IPv6地址以`fe80`开始）。此时可以将`endpoint_params.address_family`设置为`AF_INET`来强制域名解析时仅解析IPv4地址。同样的，`resolv.conf`文件中可能同时指定了`nameserver`的IPv4地址和IPv6地址，此时可以将`dns_server_params.address_family`设置为`AF_INET`或`AF_INET6`来强制仅使用IPv4或IPv6地址来访问DNS。 ### 使用Upstream配置 全局配置默认对每个域名生效，若需要对某些域名单独指定不同的配置，则可使用[Upstream](./about-upstream.md#Address属性)功能。使用Upstream可以单独指定`dns_ttl_default`、`dns_ttl_min`配置项，以及通过`endpoint_params.address_family`单独指定该域名使用的IP地址类别。 ## 域名解析与缓存策略 网络任务通常需要通过域名解析获取到需要访问的IP地址，Workflow中域名解析相关策略如下 1. 检查域名缓存是否有该域名对应的IP地址，若有缓存且未过期，则使用该组IP地址 2. 检查域名是否为IPv4、IPv6地址或`Unix Domain Socket`，若是则直接使用该地址，无需发起域名解析 3. 检查`hosts_path`文件中是否包含该域名对应的IP地址，若有则直接使用该地址 4. 获取异步锁，保证同一域名的解析请求在同一时刻仅发起一次，并向DNS发起解析请求 5. 解析成功后会将解析结果保存到当前进程的域名缓存中，以供下次使用，并释放异步锁 6. 解析失败后会释放异步锁且将失败原因通知给等在同一个异步锁上的所有任务，通知结束后再发起的新的任务则会再次请求DNS 许多需要大量发起网络请求的场景都会配备域名缓存组件，如果每次发起网络任务时都向DNS发起解析请求，则DNS必然会不堪重负。Workflow设置了缓存存活时长（dns_ttl_default和dns_ttl_min）来保证缓存会在合理的时间后过期，以及时更新域名的解析结果。当某个域名的缓存项过期后，首先发现过期的任务会将其存活时间延长5秒并向DNS发起解析请求，5秒内同一域名上的请求会直接使用缓存的DNS解析结果，而无需等待本次解析结束。 异步锁机制可以保证**同一域名**的解析请求在同一时刻仅发起一次，在没有锁保护的情况下，若短时间内对同一域名发起大量网络任务，每个任务都会因无法从缓存中获取结果而向DNS发起解析请求，这会对DNS带来很大且不必要的负担。这里的同一域名表示的是`(host, port, family)`三元组，若通过Upstream的方式对某域名分别要求只使用IPv4和IPv6，则他们会被不同的异步锁保护，也就有可能同时发起DNS请求。 ### 异步域名解析 Workflow实现了完备的DNS任务（参考[dns_cli](./tutorial-17-dns_cli.md)），若指定了`resolv_conf_path`配置项，则向DNS发起域名解析时会使用异步请求的方式进行，在类Unix系统下，Workflow默认使用`/etc/resolv.conf`作为该配置的值。异步域名解析不会阻塞任何线程，也不会独占线程池，可以更高效地完成域名解析的任务。 ### 同步域名解析 若指定`resolv_conf_path`为`NULL`，则会通过调用`getaddrinfo`函数来实现同步域名解析，该方式会使用独立的线程池，其线程数通过`dns_threads`参数配置。若短时间内需要发起较多的域名解析请求，则同步的方式会带来较大的延迟。 --- ### About Error (`docs/about-error.md`) # 关于错误处理 任何软件系统里，错误处理都是一个重要而复杂的问题。在我们框架内部，错误处理可以说是无处不在并且极其繁琐的。 而在我们暴露给用户的接口里，我们尽可能地让事情变简单，但用户还是不可避免地需要了解一些错误信息。 ### 禁用C++异常 我们框架内不使用C++异常，用户编译自己代码的时候，最好也加上-fno-exceptions标志，以减少代码大小。 参考业界通用做法，我们会忽略new操作失败的可能，并且内部也避免用new去分配大块内存。而C语言风格的内存分配则是有查错的。 ### 关于工厂函数 从之前的实例中我们看到，所有的task，series都是从WFTaskFactory或Workflow这两个工厂类产生的。 这些工厂类，以及我们以后可能遇到的更多的工厂类接口，都是确保成功的。也就是说，一定不会返回NULL。用户无需对返回值做检查。 为了达到这个目的，当URL不合法时，工厂也能正常产生task。并且在任务的callback里再得到错误。 ### 任务的状态和错误码 在之前的示例里，我们经常在callback里看到这样的代码： ~~~cpp void callback(WFXxxTask *task) { int state = task->get_state(); int error = task->get_error(); ... } ~~~ 其中，state代表任务的结束状态，在[WFTask.h](../src/factory/WFTask.h)文件中，可以看到所有可能的状态值： ~~~cpp enum { WFT_STATE_UNDEFINED = -1, WFT_STATE_SUCCESS = CS_STATE_SUCCESS, WFT_STATE_TOREPLY = CS_STATE_TOREPLY, /* for server task only */ WFT_STATE_NOREPLY = CS_STATE_TOREPLY + 1, /* for server task only */ WFT_STATE_SYS_ERROR = CS_STATE_ERROR, WFT_STATE_SSL_ERROR = 65, WFT_STATE_DNS_ERROR = 66, /* for client task only */ WFT_STATE_TASK_ERROR = 67, WFT_STATE_ABORTED = CS_STATE_STOPPED /* main process terminated */ }; ~~~ ##### 需要关注的几个状态： * SUCCESS：任务成功。client接收到完整的回复，或server把回复完全写进入发送缓冲（但不能确保对方一定能收到）。 * SYS_ERROR: 系统错误。这种情况，task->get_error()得到的是系统错误码errno。 * 当get_error()得到ETIMEDOUT，可以调用task->get_timeout_reason()进一步得到超时原因。 * DNS_ERROR: DNS解析错误。get_error()得到的是getaddrinfo()调用的返回码。关于DNS，有一篇文档专门说明[about-dns.md](./about-dns.md)。 * server任务永远不会有DNS_ERROR。 * SSL_ERROR: SSL错误。get_error()得到的是SSL_get_error()的返回值。 * 目前SSL错误信息没有做得很全，得不到ERR_get_error()的值。所以，基本上get_error()返回值也就三个可能： * SSL_ERROR_ZERO_RETURN, SSL_ERROR_X509_LOOKUP, SSL_ERROR_SSL。 * 更加详细的SSL错误信息，我们在后续版本会考虑加入。 * TASK_ERROR: 任务错误。常见的例如URL不合法，登录失败等。get_error()的返回值可以在[WFTaskError.h](../src/factory/WFTaskError.h)中查看。 ##### 用户一般无需关注的几个状态： * UNDEFINED: 刚创建完，还没有运行的client任务，状态是UNDEFINED。 * TOREPLY: server任务回复之前，没有被调用过task->noreply()，都是TOREPLY状态。 * NOREPLY: server任务被调用了task->noreply()之后，一直是NOREPLY状态。callback里也是这个状态。连接会被关闭。 ### 其它错误处理需求 除了任务本身的错误处理，各种具体协议的消息接口上，也会有判断错误的需要。一般这些接口都通过返回false来表示错误，并且通过errno传递错误原因。 此外，一些更复杂的用法，可能需要接触到更复杂一点的错误信息。我们在具体的文档里再做介绍。 --- ### About Exit (`docs/about-exit.md`) # 关于程序退出 由于我们的大多数调用都是非阻塞的，所以在之前的示例里我们都需要用一些机制来防止main函数提前退出。 例如wget示例中等待用户的Ctrl-C，或者像parallel_wget在所有抓取结束之后唤醒主线程。 而在几个server的示例中，stop()操作是阻塞的，可以确保所有server task的正常结束，主线程可安全退出。 # 程序安全退出的原则 一般情况下，用户只要正常写程序，模仿示例中的方法，不太会有什么关于退出的疑惑。但这里还是需要把程序正常退出的条件定义好。 * 用户不可以在callback或process等任何回调函数里调用系统的exit()函数，否则行为无定义。 * 主线程可以安全结束（main函数调用exit()或return）的条件是所有任务已经运行到callback，并且没有新的任务被调起。 * 我们所有的示例都符合这个假设，在callback里唤醒main函数。这是安全的，不用担心main返回的时候，callback还没结束的情况。 * ParallelWork是一种task，也需要运行到callback。 * 这一条规则某下情况下可以违反，我们将在下一节解释。 * 所有server必须stop完成，否则行为无定义。因为stop操作用户都会调，所以一般server程序不会有什么退出方面的问题。 * server的stop会等待所有server任务所在series结束。但如果用户在process直接start一个新任务，则需要考虑任务结束的问题。 # 为什么需要等待运行中的任务callback？能不能提前结束程序？ 首先解释一下需要等待任务callback再结束程序的原因。在大多数情况下，我们通过任务工厂产生的任务，都是一个复合任务。 http抓取任务为例，一个http任务可能需要先解析dns，再发起http抓取。如遇到302重定向，可能需要再次dns。任务失败可能还会重试。 也就是说，我们一个异步任务可能包含多个异步过程，但对用户完全无感。而内部每个异步过程之间，并不会检查程序是否已经退出。 如果用户明确知道一个任务是原子任务，例如以IP地址（或肯定能dns cache命中）创建http任务，并且无重定向或重试。 那么，这个任务可以被程序退出打断并提前来到callback，callback里任务的状态是WFT_STATE_ABORTED。 例如以下程序是绝对安全的： ~~~cpp void callback(WFHttpTask *task) { // 这里打印的结果大概率是2，WFT_STATE_ABORTED。 printf("state = %d\n", task->get_state()); } int main() { WFHttpTask *task = WFTaskFactory::create_http_task("https://127.0.0.1/", 0, 0, callback); task->start(); // 这里直接结束程序 return 1; } ~~~ 如果dns cache命中，也是安全的。因为内部无需再发起一个dns异步任务了。例如： ~~~cpp WFFacilities::WaitGroup wg(1); void callback_normal(WFHttpTask *task) { wg.done(); } void callback_abort(WFHttpTask *task) { // 这里打印的结果大概率是2，WFT_STATE_ABORTED。 printf("state = %d\n", task->get_state()); } int main() { WFHttpTask *task = WFTaskFactory::create_http_task("https://www.sogou.com/", 3, 2, callback_normal); task->start(); // 等待第一个访问www.sogou.com的任务结束。 wg.wait(); // 第二次访问www.sogou.com, dns信息已经被cache。 WFHttpTask *task = WFTaskFactory::create_http_task("https://www.sogou.com/", 0, 0, callback_abort); task->start(); // 这里直接结束程序 return 1; } ~~~ 所以，对于网络任务而言，只要能确定是一个原子任务，都可以被程序结束打断。这个原则可以扩展到任何类型的任务。 例如，定时器任务是一个就原子任务，以下程序也是绝对安全的： ~~~cpp void callback(WFTimerTask *task) { // 这里打印的结果肯定是2，WFT_STATE_ABORTED。 printf("state = %d\n", task->get_state()); } int main() { WFTimerTask *task = WFTaskFactory::create_timer_task(1000000, callback); task->start(); // 这里直接结束程序 return 1; } ~~~ 在[关于定时器](https://github.com/sogou/workflow/blob/master/docs/about-timer.md)的文档里，我们将会详细展开描述。 此外，单线程的计算任务，文件IO任务，也可以在callback之前直接结束程序。 其中，已经在执行计算的计算任务，程序会等待计算结束，最终以SUCCESS状态callback。还未被调起的，则以ABORTED状态退出。 文件IO任务，只要已经start，肯定会等待IO完成。因此直接退出程序完全安全。 # 关于OpenSSL 1.1版本在退出时的内存泄露 我们发现某些openssl1.1版本，存在退出时内存释放不完全的问题，通过valgrind内存检查工具可以看出内存泄露。 这个问题只有在用户使用了SSL，例如抓取了https网页时才会发生，而且一般情况下用户可以忽略这个泄露。 如果一定要解决，方法如下： ~~~cpp #include int main() { #if OPENSSL_VERSION_NUMBER >= 0x10100000L OPENSSL_init_ssl(0, NULL); #endif ... } ~~~ 也就是说在使用我们的库之前，先初始化openssl。如果你有需要也可以同时配置openssl的参数。 注意这个函数只在openssl1.1以上版本才有提供，所以调用之前需要先判断openssl版本。 这个内存泄露与openssl1.1的内存释放原理有关。我们提供的这个方案可以解决这个问题（但我们还是建议用户忽略）。 --- ### About Go Task (`docs/about-go-task.md`) # 关于go task 我们提供了另一种更简单的使用计算任务的方法，模仿go语言实现的go task。 使用go task来实计算任务无需定义输入与输出，所有数据通过函数参数传递。 # 创建go task ~~~cpp class WFTaskFactory { ... public: template static WFGoTask *create_go_task(const std::string& queue_name, FUNC&& func, ARGS&&... args); }; ~~~ 函数参数的queue_name为计算队列名，其作用在之前示例文档中有过介绍。 func可以是函数指针，函数对象，仿函数，lambda函数，类的成员函数等任意可调用对象。 args为func的参数列表。注意当func是一个类的非静态成员函数时，args的第一个成员必须是对象地址。 # 示例 我们想异步的运行一个加法函数：void add(int a, int b, int& res); 并且我们还想在函数运行结束的时候打印出结果。于是可以这样实现： ~~~cpp #include #include #include "workflow/WFTaskFactory.h" #include "workflow/WFFacilities.h" void add(int a, int b, int& res) { res = a + b; } int main(void) { WFFacilities::WaitGroup wait_group(1); int a = 1; int b = 1; int res; WFGoTask *task = WFTaskFactory::create_go_task("test", add, a, b, std::ref(res)); task->set_callback([&](WFGoTask *task) { printf("%d + %d = %d\n", a, b, res); wait_group.done(); }); task->start(); wait_group.wait(); return 0; } ~~~ 以上的示例异步运行一个加法，打印结果并退出程序。go task的使用与其它的任务没有多少区别，也有user_data域可以使用。 唯一一点不同，是go task创建时不传callback，但和其它任务一样可以set_callback。 如果go task函数的某个参数是引用，需要使用std::ref，否则会变成值传递，这是c++11的特征。 # 把workflow当成线程池 用户可以只使用go task，这样可以将workflow退化成一个线程池，而且线程数量默认等于机器cpu数。 但是这个线程池比一般的线程池又有更多的功能，比如每个任务有queue name，任务之间还可以组成各种串并联或更复杂的依赖关系。 # 带执行时间限制的go task 通过create_timedgo_task接口（这里无法重载create_go_task接口），可以创建带时间限制的go task： ~~~cpp class WFTaskFactory { /* Create 'Go' task with running time limit in seconds plus nanoseconds. * If time exceeded, state WFT_STATE_SYS_ERROR and error ETIMEDOUT will be got in callback. */ template static WFGoTask *create_timedgo_task(time_t seconds, long nanoseconds, const std::string& queue_name, FUNC&& func, ARGS&&... args); }; ~~~ 相比创建普通的go task，create_timedgo_task函数需要多传两个参数，seconds和nanoseconds。 如果func的运行时间到达seconds+nanosconds时限，task直接callback，且state为WFT_STATE_SYS_ERROR，error为ETIMEDOUT。 注意，框架无法中断用户执行中的任务。func依然会继续执行到结束，但不会再次callback。另外，nanoseconds取值区间在\[0,10亿）。 另外，当我们给go task加上了运行时间限制，callback的时机可能会先于func函数的结束，任务所在series可能也会先于func结束。 如果我们在func里访问series，可能就是一个错误了。例如： ~~~cpp void f(SeriesWork *series) { series->set_context(...); // 错误。当f是一个带超时的go task，此时series可能已经失效了。 } int http_callback(WFHttpTask *task) { SeriesWork *series = series_of(task); WFGoTask *go = WFTaskFactory::create_timedgo_task(1, 0, "test", f, series); // 1秒超时的go task series_of(task)->push_back(go); } ~~~ 这也是为什么，我们不推荐在计算任务的执行函数里，对任务所在的series进行操作。对series的操作，应该在callback里进行，例如： ~~~cpp int main() { WFGoTask *task = WFTaskFactory::create_timedgo_task(1, 0, "test", f); task->set_callback([](WFGoTask *task) { SeriesWork *series = series_of(task): void *context = series->get_context(); if (task->get_state() == WFT_STATE_SUCCESS) // 成功执行完 { ... } else // state == WFT_STATE_SYS_ERROR && error == ETIMEDOUT // 超过运行时间限制 { ... } }); } ~~~ 但是，在计算函数里使用task，是安全的。所以，可以使用task->user_data，在计算函数和callback之间传递数据。例如： ~~~cpp int main() { WFGoTask *task = WFTaskFactory::create_timedgo_task(1, 0, "test", [&task]() { task->user_data = (void *)123; }); task->set_callback([](WFGoTask *task) { SeriesWork *series = series_of(task): void *context = series->get_context(); if (task->get_state() == WFT_STATE_SUCCESS) // 成功执行完 { int result = (int)task->user_data; } else // state == WFT_STATE_SYS_ERROR && error == ETIMEDOUT // 超过运行时间限制 { ... } }); task->start(); ... } ~~~~ # 重置go task的执行函数 在某些时候，我们想在go task的执行函数里访问task，如上面的例子，将计算结果写入task的user_data域。 上例中，我们使用了引用捕获。但明显引用捕获会有一些问题。比如task本身的生命周期。我们更希望在执行函数里直接捕获go task指针。 直接进行值捕获明显是错误的，例如： ~~~cpp WFGoTask *task = WFTaskFactory::create_timedgo_task(1, 0, "test", [task]() { task->user_data = (void *)123; }); ~~~ 这段代码并不能在lambda函数里得到task指针，因为捕获执行时，task还没有赋值。但我们可以通过以下的代码，实现这个需求： ~~~cpp WFGoTask *task = WFTaskFactory::create_timedgo_task(1, 0, "test", nullptr); // 执行函数可以初始化为nullptr WFTaskFactory::reset_go_task(task, [task]() { task->user_data = (void *)123; }); ~~~ WFTaskFactory::reset_get_task()函数，用于重置go task的执行函数。 因为task已经创建完毕，这时候在lambda函数里捕获task，就是一个正确的行为了。 --- ### About Module (`docs/about-module.md`) # 关于模块任务 我们的任务流是以task为元素。但很多情况下，用户需要模块级的封装，比如几个task完成一个特定的功能。 用原有的方法，就不得不让最后一个task的callback来衔接下一个任务，或者填写server任务的resp。这样不太合理。 因此，我们引入了WFModuleTask，方便用户封装模块，降低不同功能模块之间task的耦合。 # 模块任务的创建 我们把模块定义成一种特殊的任务，WFModuleTask。模块的内部包括一个sub_series用于运行模块内的任务。 对任务来讲，它无需关心自己是否运行在模块内。因为模块内的sub_series和普通series没有任何区别。 在[WFTaskFactory.h](/src/factory/WFTaskFactory.h)里，包括了包括了模块任务的创建接口： ~~~cpp using module_callback_t = std::function; class WFTaskFactory { static WFModuleTask *create_module_task(SubTask *first, module_callback_t callback); }; ~~~ create_module_task()的第一个参数first代表模块首任务，这与创建series类似。 module callback参数要求是const指针。这主要是防止用户在callback里，继续向module中添加任务。 # WFModuleTask的主要接口 因为我们把模块也定义成这一种任务，所以，可以像使用其它任务一样使用模块。但模块没有state和error域。 在[WFTask.h](/src/factory/WFTask.h)里，定义了WFModuleTask类。 ~~~cpp class ModuleTask : public ParallelTask, protected SeriesWork // 不必关注这个派生关系 { public: void start() { .. } void dismiss() { ... } public: SeriesWork *sub_series() { return this; } const SeriesWork *sub_series() const { return this; } public: void *user_data; }; ~~~ module特有的sub_series接口返回module内任务运行的series。module本质上是一个子任务流。 sub_series也是一个普通的series，用户可以调用它的set_context()，get_context()，push_back()等函数。 但我们不太建议给sub_series设置callback，因为没有什么必要，使用module的callback就可以了。 注意，在module的callback参数表，是const WFModuleTask \*，也就只能得到一个const的sub_series。 因此，在模块任务的callback里，只能调用sub_series的get_context()得到series上下文。 # 示例 在一个http server的处理逻辑中，我们把所有处理逻辑设计成一个模块。 ~~~cpp struct ModuleCtx { std::string body; }; void http_callback(WFHttpTask *http_task) { SeriesWork *series = series_of(http_task); // 这个series就是module的sub_series。 struct ModuleCtx *ctx = (struct ModuleCtx *)series->get_context(); const void *body; size_t size; if (http_task->get_resp()->get_parsed_body(&body, &size)) { ctx->body.assign(body, size); } ParallelWork *pwork = Workflow::create_parallel_work(…)；// 做一些别的操作 series->push_back(pwork); } void process(WFHttpTask *server_task) { WFHttpTask *http_task = WFTaskFactory::create_http_task(…, http_callback); WFModuleTask *module = WFTaskFactory::create_module_task(http_task, [server_task](const WFModuleTask *mod) { struct ModuleCxt *ctx = (struct ModuleCtx *)mod->sub_series()->get_context(); server_task->get_resp()->append_output_body(ctx->body); delete ctx; }); module->sub_series()->set_context(new ModuleCtx); series_of(server_task)->push_back(module); } ~~~ 通过这个方法，module里的任务只需操作series context，最终由module的callback汇总填写resp。任务耦合性大幅降低。 --- ### About Resource Pool (`docs/about-resource-pool.md`) # 资源池 在我们用workflow写异步程序时经常会遇到这样一些场景： * 任务运行时需要先从某个池子里获得一个资源。任务运行结束，则会把资源放回池子，让下一个需要资源的任务运行。 * 网络通信时需要对某一个或一些通信目标做总的并发度限制，但又不希望占用线程等待。 * 我们有许多随机到达的任务，处在不同的series里。但这些任务必须**串行**的运行。 所有这些需求，都可以用资源池模块来解决。我们的[WFDnsResolver](https://github.com/sogou/workflow/blob/master/src/nameservice/WFDnsResolver.cc)就是通过这个方法来实现对dns server的并发度控制的。 # 资源池的接口 在[WFResourcePool.h](https://github.com/sogou/workflow/blob/master/src/factory/WFResourcePool.h)里，定义了资源池模块的接口： ~~~cpp class WFResourcePool { public: WFConditional *get(SubTask *task, void **resbuf); WFConditional *get(SubTask *task); void post(void *res); ... protected: virtual void *pop() { return this->data.res[this->data.index++]; } virtual void push(void *res) { this->data.res[--this->data.index] = res; } ... public: WFResourcePool(void *const *res, size_t n); WFResourcePool(size_t n); ... }; ~~~ #### 构造函数 第一个构造函数接受一个资源数组，长度为n。数组每个元素为一个void \*。内部会再分配一份相同大小的内存，把数组复制走。 如果你的初始资源都是nullptr，那么你可以使用第二个构造函数，只需要传n，而无需先建立一个全部为nullptr的指针数组。 大概看看内部实现就明白了： ~~~cpp void WFResourcePool::create(size_t n) { this->data.res = new void *[n]; this->data.value = n; ... } WFResourcePool::WFResourcePool(void *const *res, size_t n) { this->create(n); memcpy(this->data.res, res, n * sizeof (void *)); } WFResourcePool::WFResourcePool(size_t n) { this->create(n); memset(this->data.res, 0, n * sizeof (void *)); } ~~~ #### 使用接口 用户使用get()接口，把任务打包成一个conditional。conditional是一个条件任务，条件满足时运行其包装的任务。 get()接口可包含第二个参数是一个void \*\*resbuf，用于保存所获得的资源。 接下来，用户只需要用这个conditional取代原来的任务使用就好了，可以start或串进任务流。 注意conditional是在它被执行时去尝试获得资源的，而不是在它被创建的时候。要不然的话，以下代码就会被卡死： ~~~cpp WFResourcePool pool(1); int f() { WFHttpTask *t1 = WFTaskFactory::create_http_task(..., [](void *){pool.post(nullptr);}); WFHttpTask *t2 = WFTaskFactory::create_http_task(..., [](void *){pool.post(nullptr);}); WFConditional *c1 = pool.get(t1, &t1->user_data); // 用user_data来保存res是一种实用方法。 WFConditional *c2 = pool.get(t2, &t2->user_data); c2->start(); // wait for t2 finish here. ... c1->start(); ... } ~~~ 以上代码c1先创建，等待t2结束后才运行。这里并不会出现c2卡死，因为conditional是在执行时才获得资源的。 当用户对资源使用完毕（一般在任务callback里），需要通过post()接口把资源放回池子。 post()时的res参数，**无需**与get()得到res的一致。 #### 派生 从上面的pop()和push()函数我们可以看到，我们对资源的使用默认是FILO，即先进后出的。 使用FILO的原因是，大多数场景下，刚刚被释放的资源应该优先被复用。 但是，用户可以通过派生的方式，非常简单的实现一个FIFO资源池。只需要重写pop()和push()两个virtual函数即可。 如果需要，你还可以实现可动态扩展和收缩的资源池。 # 示例 我们准备抓取一份URL列表，但要求总的并发度不超过max_p。我们当然可以用parallel来实现，但使用资源池可以更简单： ~~~cpp int fetch_with_max(std::vector& url_list, size_t max_p) { WFResourcePool pool(max_p); for (std::string& url : url_list) { WFHttpTask *task = WFTaskFactory::create_http_task(url, [&pool](WFHttpTask *task) { pool.post(nullptr); }); WFConditional *cond = pool.get(task); // 无需保存res，可以不传resbuf参数。 cond->start(); } // wait_here... } ~~~ # 消息队列 消息队列是一种比资源使用方法类似的组件。它们的区别在于： * 资源池的总资源数量是固定的，在创建时就已经确定。而消息队列的长度则不受限制。 * 资源池的存取方式是先进后出，刚刚释放的资源会先被复用。而消息队列则是先进先出。 * 资源池使用方式是先获取，后归还。没有获取就直接归还资源，可能导致缓冲区溢出。消息队列没有这样的约束。 * 实现上，资源池使用的是数组，消息队列使用链表。总体来讲，在实现和使用上，消息队列都比资源池简单一些。 # 消息队列接口 在[WFMessageQueue.h](https://github.com/sogou/workflow/blob/master/src/factory/WFMessageQueue.h)里，定义了消息队列模块的接口： ~~~cpp class WFMessageQueue { public: WFConditional *get(SubTask *task, void **msgbuf); WFConditional *get(SubTask *task); void post(void *msg); ... public: WFMessageQueue(); ... }; ~~~ 由于了解过资源池的用法，消息队列的使用方式我们也就无需再详细展开。模式和资源池一样，都是在获得消息（或资源）时，任务被拉起。 消息队列的get和post接口，无需像资源池一样遵循先获取再放回的原则，任何任务都可以随时从队列中存取消息。 如果有需要，用户同样可以派生WFMessageQueue类，实现先进后出的消息读取模式。 --- ### About Selector (`docs/about-selector.md`) # 关于Selector任务 我们业务中经常有一些需求，从几个异步分支中选择第一个成功完成的结果进行处理，丢弃其它结果。 Selector任务就是为了上述这种多选一场景而设计的。 # Selector解决的问题 常见的多选一场景例如： * 向多个下游发送网络请求，只要任意一个下游返回正确结果，工作流程就可以继续。 * 执行一组复杂的操作，操作执行完成或整体超时，流程都会继续。 * 并行计算中，任何一个线程计算出预期的结果即完成，例如MD5碰撞计算。 * 网络应用中的‘backup request’，也可以用selector配合timer来实现。 在selector任务被引入之前，这些场景很难被很好解决，涉及到任务生命周期以及丢弃结果的资源回收等问题。 # 创建Selector任务 Selector也是一种任务，所以一般由WFTaskFactory里的工厂函数产生： ~~~cpp using selector_callback_t = std::function; class WFTaskFactory { public: static WFSelectorTask *create_selector_task(size_t candidates, selector_callback_t callback); }; ~~~ 其中，candidates参数代表从多少个候选路径中选择。Selector任务创建后，必须有candidates次被提交才会被销毁。 因此，用户可以放心的（也是必须的）向selector提交candidates次，无需要担心selector的生命周期问题。 # Selector类的接口 WFSelectorTask类包括两个主要接口。其中，对提交者来讲，只需要关注submit函数。对于等待者，只需使用到get_message。 ~~~cpp class WFSelectorTask : public WFGenericTask { public: virtual int submit(void *msg); void *get_message() const; }; ~~~ 当第一个非空指针的msg被提交，submit函数返回1表示接受。随后的submit调用都返回0代表消息被拒绝。 Selector运行后接收到一个有效消息就进入callback了，但在收到所有submit之前，不会被销毁。 注意空指针永远不会被接受，所以submit一个NULL将返回0。一般来讲，submit(NULL)用于表示这个分支失败了。 如果所有候选都提交了NULL，selector运行到callback时，state=WFT_STATE_SYS_ERROR, error=ENOMSG。 作为等待者，在selector的callback里调用另外一个接口get_message()就可以得到被成功接受的消息了。 # 示例 我们同时抓取两个http网页，并设置一个超时。当任意一个先抓取成功或超时，打印出抓取成功的URL或出错信息。 示例中使用wait group来保证两个抓取任务已经结束才退出程序。而timer可以被程序退出打断，无需等待。 ~~~cpp #include #include #include "workflow/WFTaskFactory.h" #include "workflow/WFFacilities.h" WFSelectorTask *selector; WFFacilities::WaitGroup wait_group(2); void http_callback(WFHttpTask *t) { if (t->get_state() == WFT_STATE_SUCCESS) selector->submit(t->user_data); else selector->submit(NULL); wait_group.done(); } int main(int argc, char *argv[]) { if (argc != 4) { fprintf(stderr, "USAGE: %s \n", argv[0]); exit(1); } selector = WFTaskFactory::create_selector_task(3, [](WFSelectorTask *selector) { void *msg = selector->get_message(); if (msg) printf("%s\n", (char *)msg); else printf("failed\n"); }); auto *t = WFTaskFactory::create_http_task(argv[1], 0, 0, http_callback); t->user_data = argv[1]; t->start(); t = WFTaskFactory::create_http_task(argv[2], 0, 0, http_callback); t->user_data = argv[2]; t->start(); auto *timer = WFTaskFactory::create_timer_task(atoi(argv[3]), 0, [](WFTimerTask *timer){ if (timer->get_state() == WFT_STATE_SUCCESS) selector->submit((void *)"timeout"); else selector->submit(NULL); }); timer->start(); selector->start(); wait_group.wait(); return 0; } ~~~ --- ### About Service Governance (`docs/about-service-governance.md`) # 关于服务治理 我们拥有一套完整的机制，来管理我们所依赖的服务。这套机制包括以下的几个功能： * 用户级DNS。 * 服务地址的选取 * 包括多种选取机制，如权重随机，一致性哈希，用户指定选取方式等。 * 服务的熔断与恢复。 * 负载均衡。 * 单个服务的独立参数配置。 * 服务的主备关系等。 所有这些功能都依赖于我们的upstream子系统。利用好这个系统，我们可以轻易地实现更复杂的服务网格功能。 # upstream名 upstream名相当于程序内部的域名，但相比一般的域名，upstream拥有更多的功能，包括： * 域名通常只能指向一组ip地址，upstream名可以指向一组ip地址或域名。 * upstream指向的对象（域名或ip），可以包括端口信息。 * upstream有管理和选择目标的强大功能，每个目标可以包含大量属性。 * upstream的更新，是实时而且完全线程安全的，而域名的DNS信息，并不能实时更新。 实现上，如果无需访问外网，用upstream可以完全代替域名和DNS。 # upstream的创建与删除 在[UpstreamManager.h](../src/manager/UpstreamManager.h)里，包括几个upstream创建接口： ~~~cpp using upstream_route_t = std::function; class UpstreamManager { public: static int upstream_create_consistent_hash(const std::string& name, upstream_route_t consitent_hash); static int upstream_create_weighted_random(const std::string& name, bool try_another); static int upstream_create_manual(const std::string& name, upstream_route_t select, bool try_another, upstream_route_t consitent_hash); static int upstream_delete(const std::string& name); ... }; ~~~ 三个函数创建分别为3种类型的upstream：一致性hash，权重随机和用户手动选取。 参数name为upstream名，创建之后，就和域名一样的使用了。 consistent_hash和select参数，都是一个类型为upstream_route_t的std::function，用于指定路由方式。 而try_another表示，如果选取到的目标不可用（熔断），是否继续尝试找到一个可用目标。consistent_hash模式没有这个属性。 upstream_route_t参数接收的3个参数分别是url里的path, query和fragment部分。例如URL为：http://abc.com/home/index.html?a=1#bottom 则这三个参数分别为"/home/index.html", "a=1"和"bottom"。用户可以根据这三个部分，选择目标服务器，或者进行一致性hash。 注意，以上接口中，consistent_hash参数都可以传nullptr，我们将使用默认的一致性哈希算法。 # 示例1：权重分配 我们想把50%访问www.sogou.com的请求，打到127.0.0.1:8000和127.0.0.1:8080两个地址，并且让他们的负载为1:4。 我们无需要关心域名www.sogou.com之下，有多少个ip地址。总之实际域名会接收50%的请求。 ~~~cpp #include "workflow/UpstreamManager.h" #include "workflow/WFTaskFactory.h" int main() { UpstreamManager::upstream_create_weighted_random("www.sogou.com", false); struct AddressParams params = ADDRESS_PARAMS_DEFAULT; params.weight = 5; UpstreamManager::upstream_add_server("www.sogou.com", "www.sogou.com", ¶ms); params.weight = 1; UpstreamManager::upstream_add_server("www.sogou.com", "127.0.0.1:8000", ¶ms); params.weight = 4; UpstreamManager::upstream_add_server("www.sogou.com", "127.0.0.1:8080", ¶ms); WFHttpTask *task = WFTaskFactory::create_http_task("http://www.sogou.com/index.html", ...); ... } ~~~ 请注意，以上这些函数可以在任何场景下调用，完全线程安全，并实时生效。 另外，由于我们一切协议，包括用户自定义协议都有URL，所以upstream功能可作用于一切协议。 # 示例2：手动选择 同样是上面的例子，我们想让url里，query为"123"的请求，打到127.0.0.1:8000，如果是"abc"，打到8080端口，其它打正常域名。 ~~~cpp #include "workflow/UpstreamManager.h" #include "workflow/WFTaskFactory.h" int my_select(const char *path, const char *query, const char *fragment) { if (strcmp(query, "123") == 0) return 1; else if (strcmp(query, "abc") == 0) return 2; else return 0; } int main() { UpstreamManager::upstream_create_manual("www.sogou.com", my_select, false, nullptr); UpstreamManager::upstream_add_server("www.sogou.com", "www.sogou.com"); UpstreamManager::upstream_add_server("www.sogou.com", "127.0.0.1:8000"); UpstreamManager::upstream_add_server("www.sogou.com", "127.0.0.1:8080"); /* This URL will route to 127.0.0.1:8080 */ WFHttpTask *task = WFTaskFactory::create_http_task("http://www.sogou.com/index.html?abc", ...); ... } ~~~ 由于我们原生提供了redis和mysql协议，用这个方法，可以极其方便的实现数据库的读写分离功能（注：非事务的操作）。 以上两个例子，upstream名用的是www.sogou.com，这本身也是一个域名。当然用户可以更简单的用字符串sogou，这样创建任务时： ~~~cpp WFHttpTask *task = WFTaskFactory::create_http_task("http://sogou/home/1.html?abc", ...); ~~~ 总之url的host部分，如果是一个已经创建的upstream，则会被当作upstream使用。 # 示例3：一致性hash 这个场景里，我们要从10个redis实例中，随机选择一台机器通信。但保证同一个url肯定访问一个确定的目标。方法很简单： ~~~cpp int main() { UpstreamManager::upstream_create_consistent_hash("redis.name", nullptr); UpstreamManager::upstream_add_server("redis.name", "10.135.35.53"); UpstreamManager::upstream_add_server("redis.name", "10.135.35.54"); UpstreamManager::upstream_add_server("redis.name", "10.135.35.55"); ... UpstreamManager::upstream_add_server("redis.name", "10.135.35.62"); auto *task = WFTaskFactory::create_redis_task("redis://:mypassword@redis.name/2?a=hello#111", ...); ... } ~~~ 我们的redis任务并不识别query部分，用户可以随意填写。path部分的2为redis库号。 这个时候，consistent_hash函数将得到"/2"，"a=hello"和"111"三个参数，但因为我们用nullptr，默认一致性hash将被调用。 upstream里的服务器没有指定端口号，于是将使用url里的端口。redis默认为6379。 consitent_hash并没有try_another选项，如果目标熔断，将自动选取另一个。相同url还将得到相同选择（cache友好）。 # upstream server的参数 示例１中，我们通过params参数设置了server的权重。当然server参数远不止权重一项。这个结构定义如下： ~~~cpp // In EndpointParams.h struct EndpointParams { size_t max_connections; int connect_timeout; int response_timeout; int ssl_connect_timeout; bool use_tls_sni; }; // In ServiceGovernance.h struct AddressParams { struct EndpointParams endpoint_params; ///< Connection config unsigned int dns_ttl_default; ///< in seconds, DNS TTL when network request success unsigned int dns_ttl_min; ///< in seconds, DNS TTL when network request fail /** * - The max_fails directive sets the number of consecutive unsuccessful attempts to communicate with the server. * - After 30s following the server failure, upstream probe the server with some live client’s requests. * - If the probes have been successful, the server is marked as a live one. * - If max_fails is set to 1, it means server would out of upstream selection in 30 seconds when failed only once */ unsigned int max_fails; ///< [1, INT32_MAX] max_fails = 0 means max_fails = 1 unsigned short weight; ///< [1, 65535] weight = 0 means weight = 1. only for main server int server_type; ///< 0 for main and 1 for backup int group_id; ///< -1 means no group. Backup without group will backup for any main node }; ~~~ 大多数参数的作用一眼了然。其中endpoint_params和dns相关参数，可以覆盖全局的配置。 例如，全局对每个目标ip最大连接数为200，但我想为10.135.35.53设置最多1000连接数，可以这么做： ~~~cpp UpstreamManager::upstream_create_weighted_random("10.135.35.53", false); struct AddressParams params = ADDRESS_PARAMS_DEFAULT; params.endpoint_params.max_connections = 1000; UpstreamManager::upstream_add_server("10.135.35.53", "10.135.35.53", ¶ms); ~~~ max_fails参数为最大出错次数，如果选取目标连续出错达到max_fails则熔断，如果upstream的try_another属性为false，则任务失败， 在任务callback里，get_state()=WFT_STATE_TASK_ERROR，get_error()=WFT_ERR_UPSTREAM_UNAVAILABLE。 如果try_another为true，并且所有server都熔断的话，会得到同样错误。熔断时间为30秒。 server_type和group_id用于主备功能。所有upstream必需有type为0(主节点)的server，否则upstream不可用。 类型为1（备份节点）的server，会在同group_id的主节点熔断情况下被使用。 更多upstream功能查询：[about-upstream.md](./about-upstream.md)。 --- ### About Timeout (`docs/about-timeout.md`) # 关于超时 为了让所有通信任务可以在用户的预期下精确运行，我们提供了大量的超时配置功能，并且确保这些超时的准确性。 这些超时配置里，有些是全局的，比如连接超时，但你又可以通过upstream功能，给某个域名配置自己的连接超时。 有一些超时是任务级的，比如完整发送一条消息的超时。因为用户需要根据消息大小，动态配置这个值。 当然对server来讲，又有自己的超时整体配置。总之，超时是一件很复杂的事，我们会做得很精确。 所有超时都采用poll风格，也就是int型，毫秒级，-1表示无限。 另外，正如我们在项目介绍里说的，所有的配置你都可以忽略，可以等遇到实际需求了再进行调整。 ### 基础通信超时配置 在[EndpointParams.h](../src/manager/EndpointParams.h)文件里，可以看到： ~~~cpp struct EndpointParams { size_t max_connections; int connect_timeout; int response_timeout; int ssl_connect_timeout; }; static constexpr struct EndpointParams ENDPOINT_PARAMS_DEFAULT = { .max_connections = 200, .connect_timeout = 10 * 1000, .response_timeout = 10 * 1000, .ssl_connect_timeout = 10 * 1000, }; ~~~ 其中，与超时相关的配置包括以下3项。 * connect_timeout: 与目标建立连接的超时。默认为10秒。 * response_timeout: 等待目标响应的超时，默认为10秒。代表成功发送到目标、或从目标读取到一块数据的超时。 * ssl_connect_timeout: 与目标完成SSL握手的超时。默认为10秒。 这个结构体是通信连接的最基础的配置，后续几乎所有的通信配置都会含有这个结构体。 ### 全局超时配置 在[WFGlobal.h](../src/manager/WFGlobal.h)文件里，可以看到我们一个全局配置信息： ~~~cpp struct WFGlobalSettings { EndpointParams endpoint_params; unsigned int dns_ttl_default; unsigned int dns_ttl_min; int dns_threads; int poller_threads; int handler_threads; int compute_threads; }; static constexpr struct WFGlobalSettings GLOBAL_SETTINGS_DEFAULT = { .endpoint_params = ENDPOINT_PARAMS_DEFAULT, .dns_ttl_default = 12 * 3600, /* in seconds */ .dns_ttl_min = 180, /* reacquire when communication error */ .dns_threads = 8, .poller_threads = 2, .handler_threads = 20, .compute_threads = -1 }; //compute_threads<=0 means auto-set by system cpu number ~~~ 其中，与超时相关的配置就是EndpointParams endpoint_params这一项 修改全局配置的方法是，调用我们任何工厂函数之前，执行类似下面的操作： ~~~cpp int main() { struct WFGlobalSettings settings = GLOBAL_SETTINGS_DEFAULT; settings.endpoint_params.connect_timeout = 2 * 1000; settings.endpoint_params.response_timeout = -1; WORKFLOW_library_init(&settings); } ~~~ 上例把连接超时修改为2秒，远程服务器响应超时为无限。这种配置下，每次任务里都必须配置接收完整消息的超时，否则可能陷入无限的等待。 全局的超时配置，可以通过upstream功能，被单独的地址配置覆盖，比如你可以指定某个域名的连接超时。 Upstream每一个AddressParams也有一个EndpointParams endpoint_params项，使用方式与Global相仿。 具体结构详见[upstream文档](tutorial-10-upstream.md#Address属性) ### Server超时配置 在[http_proxy](./tutorial-05-http_proxy.md)示例的里，我们介绍过server启动配置。其中超时相关的配置包括： * peer_response_timeout: 这个的定义和全局的response_timeout一样，指的是远程client的响应超时，默认为10秒。 * receive_timeout: 接收一条完整请求的超时，默认为-1。 * keep_alive_timeout: 连接保持时间。默认1分钟。redis server为5分钟。 * ssl_accept_timeout: 完成ssl握手的超时，默认为10秒。 在这个默认配置下，client可以每9秒发送一个字节，让server一直接收而不引起超时。所以，如果服务用于公网，需要配置receive_timeout。 ### 任务级别的超时配置 任务级别的超时配置通过网络任务的几个接口调用来完成： ~~~cpp template class WFNetworkTask : public CommRequest { ... public: /* All in milliseconds. timeout == -1 for unlimited. */ void set_send_timeout(int timeout) { this->send_timeo = timeout; } void set_receive_timeout(int timeout) { this->receive_timeo = timeout; } void set_keep_alive(int timeout) { this->keep_alive_timeo = timeout; } void set_watch_timeout(int timeout) { this->watch_timeo = timeout; } ... } ~~~ 其中，set_send_timeout()设置发送完整消息的超时，默认值为-1。 set_receive_timeout()只对client任务有效，指接收完整server回复的超时，默认值为-1。 * server任务的receive_timeout在server启动配置里。对server任务设置receive_timeout没有意义，因为消息已经接收完成。 set_keep_alive()接口设置连接保持超时。一般来讲，框架能很好的处理连接保持的问题，用户不需要调用。 如果是http协议，client或server想要使用短连接，可通过添加HTTP header来完成，尽量不要用这个接口去修改。 如果一个redis client想要在请求之后关闭连接，则需要用这个接口。显然，在callback里set_keep_alive()是无效的（连接已经被复用）。 set_watch_timeout()接口为client任务专有，代表一个client任务的请求发出之后，接收到第一个返回包的最大等待时间。 利用watch timeout，可以避免一些需要等待数据推送的client任务受到response timeout和receive timeout的约束而超时。 设置了watch timeout之后，从接收到第一个数据包再开始计算receive timeout。 ### 任务的同步等待超时 有一个非常特殊的超时配置，是全局唯一一个同步等待超时。我们并不鼓励使用，但在某些应用场景下能得到很好的效果。 目前框架里，目标服务器是有连接上限的（全局和upstream都可以配置）。如果连接已经达到上限，默认的情况下，client任务失败返回。 callback里task->get_state()得到WFT_STATE_SYS_ERROR, task->get_error()得到EAGAIN。如果任务配置了retry，会自动发起重试。 在这里，我们允许通过task->set_wait_timeout()接口，配置一个同步等待超时，如果在这段时间内，有连接被释放，则任务可以占用这个连接。 如果用户配置了wait_timeout，并且在超时之前没有拿到连接，则callback得到WFT_STATE_SYS_ERROR状态和ETIMEDOUT错误。 ~~~cpp class CommRequest : public SubTask, public CommSession { public: ... void set_wait_timeout(int wait_timeout) { this->wait_timeout = wait_timeout; } } ~~~ ### 超时的原因查看 通信task包含一个get_timeout_reason()接口，用于返回超时原因，但不是很细致，包括以下几个返回值： * TOR_NOT_TIMEOUT: 不是超时。 * TOR_WAIT_TIMEOUT: 同步等待超时。 * TOR_CONNECT_TIMEOUT: 连接超时。包括TCP，SCTP等协议的连接和SSL连接超时，都是这个值。 * TOR_TRANSMIT_TIMEOUT: 一切传输超时。不能进一步区分是发送阶段还是接收阶段。以后可能会细化。 * server任务，超时原因一定是TRANSMIT_TIMEOUT，并且一定是发送回复的阶段。 ### 超时功能的实现 框架内部，需要处理的超时种类比我们在这里展现的还要更多。除了wait_timeout，全都是依赖于Linux的timerfd或kqueue的timer事件。 每个poller线程包含一个timerfd，默认配置下，poller线程数为4，可以满足大多数应用的需要了。 目前的超时算法利用了链表+红黑树的数据结构，时间复杂度在O(1)和O(logn)之间，其中n为poller线程的fd数量。 超时处理目前看不是瓶颈所在，因为Linux内核epoll相关调用也是O(logn)时间复杂度，我们把超时都做到O(1)也区别不大。 --- ### About Timer (`docs/about-timer.md`) # 关于定时器 定时器的作用是不占线程的等待一个确定时间，同样通过callback来通知定时器到期。 # 定时器的创建 WFTaskFactory类里包括四个定时相关的接口： ~~~cpp using timer_callback_t = std::function; class WFTaskFactory { ... public: static WFTimerTask *create_timer_task(time_t seconds, long nanoseconds, timer_callback_t callback); static WFTimerTask *create_timer_task(const std::string& timer_name, time_t seconds, long nanoseconds, timer_callback_t callback); static int cancel_by_name(const std::string& timer_name) { cancel_by_name(const std::string& timer_name, (size_t)-1); } static int cancel_by_name(const std::string& timer_name, size_t max); }; ~~~ 我们通过seconds和nanoseconds两个参数来指定一个定时器的定时时间。其中，seconds指定秒数而nanoseconds为纳秒数。 * seconds参数可以传递-1，产生一个无限时长的定时器，一般用于命名定时器，为了将来调用cancel取消定时。 * nanoseconds的取值范围在[0,1000000000)，否则timer运行之后会立刻错误返回，错误码为EINVAL。 在创建定时器任务时，可以传入一个timer_name作为定时器名，用于cancel_by_name接口取消定时。 定时器也是一种任务，因此使用方式与其它类型任务无异，同样有user_data域可以利用。 # 取消定时 如果在创建定时器任务时传入一个名称，那么这个定时器就可以在被提前中断。 中断一个定时任务的方法是通过WFTaskFactory::cancel_by_name这个接口，这个接口默认情况下，会取消这个名称下的所有定时器。 因此，我们也支持传入一个max参数，让操作最多取消max个定时器。无论哪个接口，返回值都是代表实际被取消的定时器个数。 如果没有这个名称下的定时器，cancel操作不会产生任何效果，并返回0。 定时器在被创建之后就可取消，并非一定要等它被启动之后。以这个代码为例： ~~~cpp #include #include "workflow/WFTaskFactory.h" int main() { WFTimerTask *timer = WFTaskFactory::create_timer_task("test", 10000, 0, [](WFTimerTask *){ printf("timer callback, state = %d, error = %d.\n", task->get_state(), task->get_error()); }); WFTaskFactory::cancel_by_name("test"); timer->start(); getchar(); return 0; } ~~~ 程序会在立即打印出'timer callback, state = 1, error = 125."，因为定时器在运行之前就已经被取消了。所以，定时任务启动后立即callback，状态码为WFT_STATE_SYS_ERROR，错误码为ECANCELED。 使用中需要注意的是，命名定时器比匿名定时器是会多出一些开销的，原因是我们需要维护查找表，会有加锁解锁等操作。如果你的定时器没有提前中断的需要，就不要在创建时传入timer_name了。 # 程序退出打断定时器 在[关于程序退出](./about-exit.md)里讲到，main函数结束或exit()被调用的时候，所有任务必须里运行到callback，并且没有新的任务被调起。 这时就可能出现一个问题，定时器的定时周期可以非常长，如果是不可中断的定时器，那么等待定时器到期，程序退出需要很长时间。 而实现上，程序退出是可以打断定时器，让定时器回到callback的。如果定时器被程序退出打断，get_state()会得到一个WFT_STATE_ABORTED状态。 当然如果定时器被程序退出打断，则不能再调起新的任务。 以下这个程序，每间隔一秒抓取一个一个http页面。当所有url抓完毕，程序直接退出，不用等待timer回到callback，退出不会有延迟。 ~~~cpp bool program_terminate = false; void timer_callback(WFTimerTask *timer) { mutex.lock(); if (!program_terminate) { WFHttpTask *task; if (urls_to_fetch > 0) { task = WFTaskFactory::create_http_task(...); series_of(timer)->push_back(task); } series_of(timer)->push_back(WFTaskFactory::create_timer_task(1, 0, timer_callback)); } mutex.unlock(); } ... int main() { .... /* all urls done */ mutex.lock(); program_terminate = true; mutex.unlock(); return 0; } ~~~ 以上程序，timer_callback必须在锁里判断program_terminate条件，否则可能在程序已经结束的情况下又调起新任务。 --- ### About Tlv Message (`docs/about-tlv-message.md`) # 关于TLV(Type-Length-Value)格式的消息 TLV消息是一种由类型，长度，内容组成的消息。由于其结构简单通用，而且方便嵌套和扩展，特别适用于定义通信消息。 为方便用户实现自定义协议，我们内置了TLV消息的支持。 # TLV消息的结构 TLV消息并没有具体规定Type和Length这两个字段占的字节数据。在我们的协议里，它们分别占4字节（网络序）。 也就是说，我们的消息有8字节的消息头，以及不超过32GB的Value内容。Type和Value域的含义我们不做规定。 # TLVMessage类 由于TLV的定义内容很少，所以[TLVMessage](/src/protocol/TLVMessage.h)需要用到的接口很少。 ~~~cpp namespace protocol { class TLVMessage : public ProtocolMessage { public: int get_type() const { return this->type; } void set_type(int type) { this->type = type; } std::string *get_value() { return &this->value; } void set_value(std::string value) { this->value = std::move(value); } protected: int type; std::string value; ... }; using TLVRequest = TLVMessage; using TLVResposne = TLVMessage; } ~~~ 用户直接使用TLV消息来做数据传输的话，只需要用到上面的几个接口。分别为设置和获取Type与Value。 Value直接以std::string返回，方便用户必要的时候直接通过std::move移动数据。 # 基于TLV消息的echo server/client 以下代码，直接启动一个基于TLV消息的server，并通过命令行产生client task进行交互。建议运行一下： ~~~cpp #include #include #include #include "workflow/WFGlobal.h" #include "workflow/WFFacilities.h" #include "workflow/TLVMessage.h" #include "workflow/WFTaskFactory.h" #include "workflow/WFServer.h" using namespace protocol; using WFTLVServer = WFServer; using WFTLVTask = WFNetworkTask; using tlv_callback_t = std::function; WFTLVTask *create_tlv_task(const char *host, unsigned short port, tlv_callback_t callback) { auto *task = WFNetworkTaskFactory::create_client_task( TT_TCP, host, port, 0, std::move(callback)); task->set_keep_alive(60 * 1000); return task; } int main() { WFTLVServer server([](WFTLVTask *task) { *task->get_resp() = std::move(*task->get_req()); }); if (server.start(8888) != 0) { perror("server.start"); exit(1); } auto&& create = [](WFRepeaterTask *)->SubTask * { std::string string; printf("Input string (Ctrl-D to exit): "); std::cin >> string; if (string.empty()) return NULL; auto *task = create_tlv_task("127.0.0.1", 8888, [](WFTLVTask *task) { if (task->get_state() == WFT_STATE_SUCCESS) printf("Server Response: %s\n", task->get_resp()->get_value()->c_str()); else { const char *str = WFGlobal::get_error_string(task->get_state(), task->get_error()); fprintf(stderr, "Error: %s\n", str); } }); task->get_req()->set_value(std::move(string)); return task; }; WFFacilities::WaitGroup wait_group(1); WFRepeaterTask *repeater = WFTaskFactory::create_repeater_task(std::move(create), nullptr); Workflow::start_series_work(repeater, [&wait_group](const SeriesWork *) { wait_group.done(); }); wait_group.wait(); server.stop(); return 0; } ~~~ # 派生TLVMessage 上面的echo server实例，我们直接使用了原始的TLVMessage。但建议在具体的应用中，用户可以对消息进行派生。 在派生类里，提供更加丰富的接口来设置和提取消息内容，避免直接操作原始Value域，并形成自己的二级协议。 例如，我们实现一个JSON的协议，可以： ~~~cpp #include "workflow/json-parser.h" // 内置的json解析器 class JsonMessage : public TLVMessage { public: void set_json_value(const json_value_t *val) { this->type = JSON_TYPE; this->json_to_string(val, &this->value); // 需要实现一下 } json_value_t *get_json_value() const { if (this->type == JSON_TYPE) return json_parser_parse(this->value.c_str()); // json-parser的函数 else return NULL; } }; using JsonRequest = JsonMessage; using JsonResponse = JsonMessage; using JsonServer = WFServer; ~~~ 这个例子只是为了说明派生的重要性，实际应用中，派生类可能要远远比这个复杂。 --- ### About Upstream (`docs/about-upstream.md`) # 关于Upstream 在nginx里，Upstream代表了反向代理的负载均衡配置。在这里，我们扩充Upstream的含义，让其具备以下几个特点： 1. 每一个Upstream都是一个独立的反向代理 2. 访问一个Upstream等价于，在一组服务/目标/上下游，使用合适的策略选择其中一个进行访问 3. Upstream具备负载均衡、出错处理、熔断和其他服务治理能力 4. 对于同一个请求的多次重试，Upstream可以避开已试过的目标 5. 通过Upstream可以对不同下游配置不同的连接参数 6. 动态增删目标地址实时生效，方便对接任意的服务发现系统 ### Upstream相对于域名DNS解析的优势 Upstream和域名DNS解析都可以将一组ip配置到一个Host，但是 1. DNS域名解析是不针对于端口号的，相同ip不同端口的服务DNS域名是不能配置到一起的；但Upstream可以 2. DNS域名解析对应的一组address，必定是ip；Upstream对应的一组address，可以是ip、域名或unix-domain-socket 3. 通常情况下，DNS域名解析会被操作系统或网络上DNS服务器所缓存，更新时间受到ttl的限制；Upstream可以做到实时更新实时生效 4. DNS域名解析消耗比Upstream解析和选取大很多 ### Workflow的Upstream 这是一个本地反向代理模块，代理配置对server和client都生效。 支持动态配置，可用于服务发现系统，目前[workflow-k8s](https://github.com/sogou/workflow-k8s)可以对接Kubernetes的API Server。 Upstream名不包括端口，但Upstream请求支持指定端口（如果使用非内置协议，Upstream名暂时需要加上端口号以保证构造时的解析成功）。 每一个Upstream配置自己的独立名称UpstreamName，并添加设定着一组Address，这些Address可以是： 1. ip4 2. ip6 2. 域名 3. unix-domain-socket ### 为什么要替代nginx的Upstream #### nginx的Upstream工作方式 1. 只支持http/https协议 2. 需要搭建一个nginx服务，启动进程占用socket等其他资源 3. 请求先打到nginx上，nginx再向远端转发请求，这会多一次通信开销 #### workflow本地Upstream工作方式 1. 协议无关，你甚至可以通过upstream访问mysql、redis、mongodb等等 2. 无需额外启动其他进程或端口，直接在进程内模拟反向代理的功能 3. 选取过程是基本的计算和查表，不会有额外的通信开销 # 使用Upstream ### 常用接口 ~~~cpp class UpstreamManager { public: static int upstream_create_consistent_hash(const std::string& name, upstream_route_t consitent_hash); static int upstream_create_weighted_random(const std::string& name, bool try_another); static int upstream_create_manual(const std::string& name, upstream_route_t select, bool try_another, upstream_route_t consitent_hash); static int upstream_create_vnswrr(const std::string& name); static int upstream_delete(const std::string& name); public: static int upstream_add_server(const std::string& name, const std::string& address); static int upstream_add_server(const std::string& name, const std::string& address, const struct AddressParams *address_params); static int upstream_remove_server(const std::string& name, const std::string& address); ... } ~~~ ### 例1 在多个目标中随机访问 配置一个本地反向代理，将本地发出的my_proxy.name所有请求均匀的打到6个目标server上 ~~~cpp UpstreamManager::upstream_create_weighted_random( "my_proxy.name", true);//如果遇到熔断机器，再次尝试直至找到可用或全部熔断 UpstreamManager::upstream_add_server("my_proxy.name", "192.168.2.100:8081"); UpstreamManager::upstream_add_server("my_proxy.name", "192.168.2.100:8082"); UpstreamManager::upstream_add_server("my_proxy.name", "192.168.10.10"); UpstreamManager::upstream_add_server("my_proxy.name", "test.sogou.com:8080"); UpstreamManager::upstream_add_server("my_proxy.name", "abc.sogou.com"); UpstreamManager::upstream_add_server("my_proxy.name", "abc.sogou.com"); UpstreamManager::upstream_add_server("my_proxy.name", "/dev/unix_domain_scoket_sample"); auto *http_task = WFTaskFactory::create_http_task("http://my_proxy.name/somepath?a=10", 0, 0, nullptr); http_task->start(); ~~~ 基本原理 1. 随机选择一个目标 2. 如果try_another配置为true，那么将在所有存活的目标中随机选择一个 3. 仅在main中选择，选中目标所在group的主备和无group的备都视为有效的可选对象 ### 例2 在多个目标中按照权重大小随机访问 配置一个本地反向代理，将本地发出的weighted.random所有请求按照5/20/1的权重分配打到3个目标server上 ~~~cpp UpstreamManager::upstream_create_weighted_random( "weighted.random", false);//如果遇到熔断机器，不再尝试，这种情况下此次请求必定失败 AddressParams address_params = ADDRESS_PARAMS_DEFAULT; address_params.weight = 5;//权重为5 UpstreamManager::upstream_add_server("weighted.random", "192.168.2.100:8081", &address_params);//权重5 address_params.weight = 20;//权重为20 UpstreamManager::upstream_add_server("weighted.random", "192.168.2.100:8082", &address_params);//权重20 UpstreamManager::upstream_add_server("weighted.random", "abc.sogou.com");//权重1 auto *http_task = WFTaskFactory::create_http_task("http://weighted.random:9090", 0, 0, nullptr); http_task->start(); ~~~ 基本原理 1. 按照权重分配，随机选择一个目标，权重越大概率越大 2. 如果try_another配置为true，那么将在所有存活的目标中按照权重分配随机选择一个 3. 仅在main中选择，选中目标所在group的主备和无group的备都视为有效的可选对象 ### 例3 在多个目标中按照框架默认的一致性哈希访问 ~~~cpp UpstreamManager::upstream_create_consistent_hash( "abc.local", nullptr);//nullptr代表使用框架默认的一致性哈希函数 UpstreamManager::upstream_add_server("abc.local", "192.168.2.100:8081"); UpstreamManager::upstream_add_server("abc.local", "192.168.2.100:8082"); UpstreamManager::upstream_add_server("abc.local", "192.168.10.10"); UpstreamManager::upstream_add_server("abc.local", "test.sogou.com:8080"); UpstreamManager::upstream_add_server("abc.local", "abc.sogou.com"); auto *http_task = WFTaskFactory::create_http_task("http://abc.local/service/method", 0, 0, nullptr); http_task->start(); ~~~ 基本原理 1. 每1个main视为16个虚拟节点 2. 框架会使用std::hash对所有节点的address+虚拟index+此address加到此upstream的次数进行运算，作为一致性哈希的node值 3. 框架会使用std::hash对path+query+fragment进行运算，作为一致性哈希data值 4. 每次都选择存活node最近的值作为目标 5. 对于每一个main、只要有存活group内main/有存活group内backup/有存活no group backup，即视为存活 6. 如果upstream_add_server()时加上AddressParams，并配上权重weight，则每1个main视为16 * weight个虚拟节点，适用于带权一致性哈希或者希望一致性哈希标准差更小的场景 ### 例4 自定义一致性哈希函数 ~~~cpp UpstreamManager::upstream_create_consistent_hash( "abc.local", [](const char *path, const char *query, const char *fragment) -> unsigned int { unsigned int hash = 0; while (*path) hash = (hash * 131) + (*path++); while (*query) hash = (hash * 131) + (*query++); while (*fragment) hash = (hash * 131) + (*fragment++); return hash; }); UpstreamManager::upstream_add_server("abc.local", "192.168.2.100:8081"); UpstreamManager::upstream_add_server("abc.local", "192.168.2.100:8082"); UpstreamManager::upstream_add_server("abc.local", "192.168.10.10"); UpstreamManager::upstream_add_server("abc.local", "test.sogou.com:8080"); UpstreamManager::upstream_add_server("abc.local", "abc.sogou.com"); auto *http_task = WFTaskFactory::create_http_task("http://abc.local/sompath?a=1#flag100", 0, 0, nullptr); http_task->start(); ~~~ 基本原理 1. 框架会使用用户自定义的一致性哈希函数作为data值 2. 其余与上例原理一致 ### 例5 自定义选取策略 ~~~cpp UpstreamManager::upstream_create_manual( "xyz.cdn", [](const char *path, const char *query, const char *fragment) -> unsigned int { return atoi(fragment); }, true,//如果选择到已经熔断的目标，将进行二次选取 nullptr);//nullptr代表二次选取时使用框架默认的一致性哈希函数 UpstreamManager::upstream_add_server("xyz.cdn", "192.168.2.100:8081"); UpstreamManager::upstream_add_server("xyz.cdn", "192.168.2.100:8082"); UpstreamManager::upstream_add_server("xyz.cdn", "192.168.10.10"); UpstreamManager::upstream_add_server("xyz.cdn", "test.sogou.com:8080"); UpstreamManager::upstream_add_server("xyz.cdn", "abc.sogou.com"); auto *http_task = WFTaskFactory::create_http_task("http://xyz.cdn/sompath?key=somename#3", 0, 0, nullptr); http_task->start(); ~~~ 基本原理 1. 框架首先依据用户提供的普通选取函数、按照取模，在main列表中确定选取 2. 对于每一个main、只要有存活group内main/有存活group内backup/有存活no group backup，即视为存活 3. 如果选中目标不再存活且try_another设为true，将再使用一致性哈希函数进行二次选取 4. 如果触发二次选取，一致性哈希将保证一定会选择一个存活目标、除非全部机器都被熔断掉 ### 例6 简单的主备模式 ~~~cpp UpstreamManager::upstream_create_weighted_random( "simple.name", true);//一主一备这项设什么没区别 AddressParams address_params = ADDRESS_PARAMS_DEFAULT; address_params.server_type = 0; UpstreamManager::upstream_add_server("simple.name", "main01.test.ted.bj.sogou", &address_params);//主 address_params.server_type = 1; UpstreamManager::upstream_add_server("simple.name", "backup01.test.ted.gd.sogou", &address_params);//备 auto *http_task = WFTaskFactory::create_http_task("http://simple.name/request", 0, 0, nullptr); auto *redis_task = WFTaskFactory::create_redis_task("redis://simple.name/2", 0, nullptr); redis_task->get_req()->set_query("MGET", {"key1", "key2", "key3", "key4"}); (*http_task * redis_task).start(); ~~~ 基本原理 1. 主备模式与前面所展示的任何模式都不冲突，可以同时生效 2. 主备数量各自独立，没有限制。主和主之间平等，备与备之间平等，主备之间不平等。 3. 只要有主活着，请求一直会使用某一个主 4. 如果主都被熔断，备将作为替代目标接管请求直至有主恢复正常 5. 在每一个策略中，存活的备都可以作为主的存活依据 ### 例7 主备+一致性哈希+分组 ~~~cpp UpstreamManager::upstream_create_consistent_hash( "abc.local", nullptr);//nullptr代表使用框架默认的一致性哈希函数 AddressParams address_params = ADDRESS_PARAMS_DEFAULT; address_params.server_type = 0; address_params.group_id = 1001; UpstreamManager::upstream_add_server("abc.local", "192.168.2.100:8081", &address_params);//main in group 1001 address_params.server_type = 1; address_params.group_id = 1001; UpstreamManager::upstream_add_server("abc.local", "192.168.2.100:8082", &address_params);//backup for group 1001 address_params.server_type = 0; address_params.group_id = 1002; UpstreamManager::upstream_add_server("abc.local", "main01.test.ted.bj.sogou", &address_params);//main in group 1002 address_params.server_type = 1; address_params.group_id = 1002; UpstreamManager::upstream_add_server("abc.local", "backup01.test.ted.gd.sogou", &address_params);//backup for group 1002 address_params.server_type = 1; address_params.group_id = -1; UpstreamManager::upstream_add_server("abc.local", "test.sogou.com:8080", &address_params);//backup for no group mean backup for all group and no group UpstreamManager::upstream_add_server("abc.local", "abc.sogou.com");//main, no group auto *http_task = WFTaskFactory::create_http_task("http://abc.local/service/method", 0, 0, nullptr); http_task->start(); ~~~ 基本原理 1. 组号-1代表无组，这种目标不属于任何组 2. 无组的main之间是平等的，甚至可以视为同一个组。但与有组的main之间是隔离的 3. 无组的backup可以为全局任何组目标/任何无组目标作为备 4. 组号可以区分哪些主备是在一起工作的 5. 不同组之间的备是相互隔离的，只为本组的main服务 6. 添加目标的默认组号-1，type为0，表示主节点。 ### 例8 NVSWRR平滑按权重选取策略 ~~~cpp UpstreamManager::upstream_create_vnswrr("nvswrr.random"); AddressParams address_params = ADDRESS_PARAMS_DEFAULT; address_params.weight = 3;//权重为3 UpstreamManager::upstream_add_server("nvswrr.random", "192.168.2.100:8081", &address_params);//权重3 address_params.weight = 2;//权重为2 UpstreamManager::upstream_add_server("nvswrr.random", "192.168.2.100:8082", &address_params);//权重2 UpstreamManager::upstream_add_server("nvswrr.random", "abc.sogou.com");//权重1 auto *http_task = WFTaskFactory::create_http_task("http://nvswrr.random:9090", 0, 0, nullptr); http_task->start(); ~~~ 基本原理 1. 虚拟节点初始化顺序按照[SWRR算法](https://github.com/nginx/nginx/commit/52327e0627f49dbda1e8db695e63a4b0af4448b1)选取 2. 虚拟节点运行时分批初始化，避免密集型计算集中，每批次虚拟节点使用完后再进行下一批次虚拟节点列表初始化 3. 兼具[SWRR算法](https://github.com/nginx/nginx/commit/52327e0627f49dbda1e8db695e63a4b0af4448b1)的平滑、分散特点，又能具备O(1)的时间复杂度 4. 算法具体细节参见[tengine](https://github.com/alibaba/tengine/pull/1306) # Upstream选择策略 当发起请求的url的URIHost填UpstreamName时，视做对与名字对应的Upstream发起请求，接下来将会在Upstream记录的这组Address中进行选择： 1. 权重随机策略：按照权重随机选择 2. 一致性哈希策略：框架使用标准的一致性哈希算法，用户可以自定义对请求uri的一致性哈希函数consistent_hash 3. 手动策略：根据用户提供的对请求uri的select函数进行确定的选择，如果选中了已经熔断的目标： a. 如果try_another为false，这次请求将返回失败 b. 如果try_another为true，框架使用标准的一致性哈希算法重新选取，用户可以自定义对请求uri的一致性哈希函数consistent_hash 4. 主备策略：按照先主后备的优先级，只要主可以用就选择主。此策略可以与[1]、[2]、[3]中的任何一个同时生效，相互影响。 round-robin/weighted-round-robin：视为与[1]等价，暂不提供 框架建议普通用户使用策略[2]，可以保证集群具有良好的容错性和可扩展性 对于复杂需求场景，高级用户可以使用策略[3]，订制复杂的选择逻辑 # Address属性 ~~~cpp struct EndpointParams { size_t max_connections; int connect_timeout; int response_timeout; int ssl_connect_timeout; bool use_tls_sni; }; static constexpr struct EndpointParams ENDPOINT_PARAMS_DEFAULT = { .max_connections = 200, .connect_timeout = 10 * 1000, .response_timeout = 10 * 1000, .ssl_connect_timeout = 10 * 1000, .use_tls_sni = false, }; struct AddressParams { struct EndpointParams endpoint_params; unsigned int dns_ttl_default; unsigned int dns_ttl_min; unsigned int max_fails; unsigned short weight; int server_type; int group_id; }; static constexpr struct AddressParams ADDRESS_PARAMS_DEFAULT = { .endpoint_params = ENDPOINT_PARAMS_DEFAULT, .dns_ttl_default = 12 * 3600, .dns_ttl_min = 180, .max_fails = 200, .weight = 1, //only for main of UPSTREAM_WEIGHTED_RANDOM .server_type = 0, .group_id = -1, }; ~~~ 每个Addreess都可以配置自己的自定义参数： * EndpointParams的max_connections, connect_timeout, response_timeout, ssl_connect_timeout：连接相关的参数 * dns_ttl_default：dns cache中默认的ttl，单位秒，默认12小时，dns cache是针对当前进程的，即进程退出就会消失，配置也仅对当前进程有效 * dns_ttl_min：dns最短生效时间，单位秒，默认3分钟，用于在通信失败重试时是否进行重新dns的决策 * max_fails：触发熔断的【连续】失败次数（注：每次通信成功，计数会清零） * weight：权重，默认1，仅对main有效，用于Upstream随机策略选取和一致性哈希选取，权重大越容易被选中 * server_type：主备配置，默认主。无论什么时刻，同组的主优先级永远高于其他的备 * group_id：分组依据，默认-1。-1代表无分组(游离)，游离的备可视为任何主的备，有组的备优先级永远高于游离的备。 # 关于熔断 ## MTTR 平均修复时间（Mean time to repair，MTTR），是描述产品由故障状态转为工作状态时修理时间的平均值。 ## 服务雪崩效应 服务雪崩效应是一种因“服务提供者的故障”（原因），导致“服务调用者故障”（结果），并将不可用逐渐/逐级放大的现象 若不加以有效控制，效应不会收敛，而且会以几何级放大，犹如雪崩，雪崩效应因此得名 日常表现通常为：起初只是一个很小的服务or模块异常/超时，引起下游其他依赖的服务随之异常/超时，产生连锁反应，最终导致绝大多数甚至全部的服务陷入瘫痪 随着故障的修复，效应随之消失，所以效应持续时间通常等于MTTR ## 熔断机制 当某一个目标的错误or异常触达到预先设定的阈值条件时，暂时认为这个目标不可用，剔除目标，即熔断开启进入熔断期 在熔断持续时间达到MTTR时长后，会进入半熔断状态，(尝试)恢复目标 如果恢复的时候发现其他所有目标都被熔断，会同一时间把所有目标恢复 熔断机制策略可以有效阻止雪崩效应 ## Upstream熔断保护机制 MTTR=30秒，暂时不可配置，后续会考虑开放给用户自行配置 当某一个Addrees连续失败次数达到设定上限（默认200次），这个Address会被熔断MTTR=30秒 Address在熔断期间，一旦被策略选中，Upstream会根据具体配置决定是否尝试其他Address、如何尝试 请注意满足下面1-4的某个情景，通信任务将得到一个WFT_ERR_UPSTREAM_UNAVAILABLE = 1004的错误： 1. 权重随机策略，全部目标都处于熔断期 2. 一致性哈希策略，全部目标都处于熔断期 3. 手动策略 && try_another==true，全部目标都处于熔断期 4. 手动策略 && try_another==false，且同时满足下面三个条件： 1). select函数选中的main处于熔断期，，且游离的备都处于熔断期 2). 这个main是游离的主，或者这个main所在的group其他目标都处于熔断期 3). 所有游离的备都处于熔断期 # Upstream端口优先级 1. 优先选择显式配置在Upstream Address上的端口号 2. 若没有，再选择显式配置在请求url中的端口号 3. 若都没有，使用协议默认端口号 ~~~text 配置 UpstreamManager::upstream_add_server("my_proxy.name", "192.168.2.100:8081"); 请求 http://my_proxy.name:456/test.html => http://192.168.2.100:8081/test.html 请求 http://my_proxy.name/test.html => http://192.168.2.100:8081/test.html ~~~ ~~~text 配置 UpstreamManager::upstream_add_server("my_proxy.name", "192.168.10.10"); 请求 http://my_proxy.name:456/test.html => http://192.168.10.10:456/test.html 请求 http://my_proxy.name/test.html => http://192.168.10.10:80/test.html ~~~ --- ### Benchmark (`docs/benchmark.md`) # 性能测试 Sogou C++ Workflow是一款性能优异的网络框架，本文介绍我们进行的性能测试， 包括方案、代码、结果，以及与其他同类产品的对比。 更多场景下的实验正在进行中，本文将持续更新。 ## HTTP Server HTTP Client/Server是Sogou C++ Workflow常见的应用场景， 我们首先对Server端进行实验。 ### 环境 我们部署了两台相同机器作为Server和Client，软硬件配置如下： | 软硬件 | 配置 | |:---:|:---| | CPU | 40 Cores, x86_64, Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz | | Memory | 192GB | | NIC | 25000Mbps | | OS | CentOS 7.8.2003 | | Kernel | Linux version 3.10.0-1127.el7.x86_64 | | GCC | 4.8.5 | 两者间`ping`测得的RTT为0.1ms左右。 ### 对照组 我们选择nginx和brpc作为对照组。 选择前者是因为它在生产中部署十分广泛，性能不俗； 对于后者，我们在本次实验中只关注HTTP Server方面的能力， 其他的特性已有[单独的实验][Sogou RPC Benchmark]进行更为详尽的测试。 事实上，我们也对此二者之外的其他某些框架同时进行了实验， 但结果其性能表现相差较远，因此未在本文中体现。 后续我们将选取更多合适的框架加入对比测试中。 ### Client工具 本次实验我们使用的压测工具为[wrk][wrk]和[wrk2][wrk2]。 前者适合测试特定并发下的QPS极限和延时， 后者适合在特定QPS下测试延时分布。 我们也尝试过使用其他测试工具，例如[ab][ab]等，但无法打出足够的压力。 有鉴于此，我们也在着手开发基于Sogou C++ Workflow的benchmark工具。 ### 变量和指标 一般而言，对网络框架的性能测试，切入的角度可谓纷繁多样。 通过控制不同的变量、观测不同的指标，可以探究程序在不同场景下的适应能力。 本次实验，我们选择其中最普遍常见的变量和指标： 通过控制Client并发度和承载数据的大小，来测试QPS和延时的变化情况。 另外，我们还测试了在掺杂慢请求的正常请求的延时分布。 下面依次介绍两个测试场景。 ### 不同并发度和数据长度下的QPS和延时 #### 代码和配置 我们搭建了一个极其简约的HTTP服务器， 忽略掉所有的业务逻辑， 将测试点聚焦在纯粹的网络框架性能上。 代码片段如下， 完整代码移步[这里][benchmark-01 Code]。 ```cpp // ... auto * resp = task->get_resp(); resp->add_header_pair("Date", timestamp); resp->add_header_pair("Content-Type", "text/plain; charset=UTF-8"); resp->append_output_body_nocopy(content.data(), content.size()); // ... ``` 可以从上述代码中看到， 对于到来的任何HTTP请求， 我们都会返回一段固定的内容作为Body， 并设置必要的Header， 包括代码中指明的`content-type`、`date`， 以及自动填充的`connection`和`content-length`。 HTTP Body的固定内容是在Server启动时随机生成的ASCII字符串， 其长度可以通过启动参数配置。 同时可以配置的还有使用的poller线程数和监听的端口号。 前者我们在本次测试中固定为16， 因此Sogou C++ Workflow将使用16个poller线程和20个handler线程（默认配置）。 对于nginx和brpc， 我们也构建了相同的返回内容， 并为nginx配置了40个进程、 brpc配置了40个线程。 #### 变量 我们控制并发度在`[1, 2K]`之间翻倍增长， 数据长度在`[16B, 64KB]`之间翻倍增长， 两者正交。 #### 指标 鉴于并发度和数据长度组合之后数量较多， 我们选择其中部分数据绘制为曲线。 ##### 固定数据长度下QPS与并发度关系 ![Concurrency and QPS][Con-QPS] 上图可以看出，当数据长度保持不变， QPS随着并发度提高而增大，后趋于平稳。 此过程中Sogou C++ Workflow一直有明显优势， 高于brpc和nginx。 特别是数据长度为64和512的两条曲线， 并发度足够的时候，可以保持500K的QPS。 注意上图中nginx-64与nginx-512的曲线重叠度很高， 不易辨识。 ##### 固定并发度下QPS与数据长度关系 ![Body Length and QPS][Len-QPS] 上图可以看出，当并发度保持不变， 随着数据长度的增长， QPS保持平稳至4K时下降。 此过程中，Sogou C++ Workflow也一直保持优势。 ##### 固定数据长度下延时与并发度关系 ![Concurrency and Latency][Con-Lat] 上图可以看出，保持数据长度不变， 延时随并发度提高而有所上升。 此过程中，Sogou C++ Workflow略好于brpc， 大好于nginx。 ##### 固定并发度下延时与数据长度关系 ![Body Length and Latency][Len-Lat] 上图可以看出，并发度保持不变时， 增大数据长度，造成延时上升。 此过程中，Sogou C++ Workflow好于nginx， 好于brpc。 ### 掺杂慢请求的延时分布 #### 代码 我们在上一个测试的基础上，简单添加了一个慢请求的逻辑， 模拟业务场景中可能出现的特殊情况。 代码片段如下， 完整代码请移步[这里][benchmark-02 Code]。 ```cpp // ... if (std::strcmp(uri, "/long_req/") == 0) { auto timer_task = WFTaskFactory::create_timer_task(microseconds, nullptr); series_of(task)->push_back(timer_task); } // ... ``` 我们在Server的process里进行判断， 如果访问的是特定的路径， 则添加一个`WFTimerTask`到Series的末尾， 能够模拟一个异步耗时处理过程。 类似地，对brpc使用`bthread_usleep()`函数进行异步睡眠。 #### 配置 在本次实验中，我们固定并发度为1024，数据长度为1024字节， 分别以QPS为20K、100K和200K进行正常请求测试， 测绘延时； 与此同时，有另一路压力，进行慢请求， QPS是上述QPS的1%， 数据不计入统计。 慢请求的时长固定为5ms。 #### 延时CDF图 ![Latency CDF][Lat CDF] 从上图可以看出，当QPS为20K时， Sogou C++ Workflow略次于brpc； 当QPS为100K时，两者几乎相当； 当QPS为200K时，Sogou C++ Workflow略好于brpc。 总之，可以认为两者在这方面旗鼓相当。 [Sogou RPC Benchmark]: https://github.com/holmes1412/sogou-rpc-benchmark [wrk]: https://github.com/wg/wrk [wrk2]: https://github.com/giltene/wrk2 [ab]: https://httpd.apache.org/docs/2.4/programs/ab.html [benchmark-01 Code]: ../benchmark/benchmark-01-http_server.cc [benchmark-02 Code]: ../benchmark/benchmark-02-http_server_long_req.cc [Con-QPS]: https://raw.githubusercontent.com/wiki/sogou/workflow/img/benchmark-01.png [Len-QPS]: https://raw.githubusercontent.com/wiki/sogou/workflow/img/benchmark-02.png [Con-Lat]: https://raw.githubusercontent.com/wiki/sogou/workflow/img/benchmark-03.png [Len-Lat]: https://raw.githubusercontent.com/wiki/sogou/workflow/img/benchmark-04.png [Lat CDF]: https://raw.githubusercontent.com/wiki/sogou/workflow/img/benchmark-05.png --- ### Bugs (`docs/bugs.md`) # 已知BUG列表 ### OpenSSL 1.1.1及以下，出现网络任务状态为WFT_STATE_SYS_ERROR，错误为0。 这是OpenSSL 1.1.1及以下的bug，在SSL_get_error()为SSL_ERROR_SYSCALL时，errno被置为0。由于框架会把SSL_ERROR_SYSCALL转为系统错误，这会导致我们得到一个错误码0的系统错误： ~~~cpp void callback(WFHttpTask *task) { int state = task->get_state(); int error = task->get_error(); printf("%d, %d\n", state, error); // 此处得到1，0，其中1是WFT_STATE_SYS_ERROR。 } ~~~ 显然只有在SSL通信下可能出现在这个问题。这个bug在OpenSSL 3.0里被修复，建议升级到OpenSSL 3.0或以上。 相关issue：https://github.com/openssl/openssl/issues/12416 ### 访问HTTPS网页，当打开TLS SNI并使用upstream时出现SSL error。 当我们创建Http任务，http header里的Host域填写的是原始URL里的host部分。例如： ~~~cpp void f() { auto *task = WFTaskFactory::create_http_task("https://sogou/index.html", 0, 0, nullptr); } ~~~ 这时候http request里的Host必然填写的是"sogou"。此时如果sogou是一个upstream名，指向域名www.sogou.com。并且我们开启了TLS SNI，那么SNI server name信息就是www.sogou.com，与http header里的Host是不一致的，会导致SSL错误。 要解决这个问题，用户可以在通过设置prepare函数，在发送请求前修改Host，让它与最终URL里的一致： ~~~cpp void f(); { auto *task = WFTaskFactory::create_http_task("https://sogou/index.html", 0, 0, nullptr); task->set_prepare([](WFHttpTask *task){ auto *t = static_cast *>(task); task->get_req()->set_header_pair("Host", t->get_current_uri()->host); // 这里得到实际uri里的host。 }); } ~~~ 只有打开了TLS SNI功能并使用upstream会出这个不一致问题。当然，很多时候我们配置upstream来访问http网站，也需要做这个修改，否则对方可能不会接受你的Host信息。 --- ### Tutorial 01 Wget (`docs/tutorial-01-wget.md`) # 创建第一个任务：wget # 示例代码 [tutorial-01-wget.cc](/tutorial/tutorial-01-wget.cc) # 关于wget 程序从stdin读取http/https URL，抓取网页并把内容打印到stdout，并将请求和响应的http header打印在stderr。 为了简单起见，程序用Ctrl-C退出，但会保证所有资源先被完全释放。 # 创建并启动http任务 ~~~cpp WFHttpTask *task = WFTaskFactory::create_http_task(url, REDIRECT_MAX, RETRY_MAX, wget_callback); protocol::HttpRequest *req = task->get_req(); req->add_header_pair("Accept", "*/*"); req->add_header_pair("User-Agent", "Wget/1.14 (gnu-linux)"); req->add_header_pair("Connection", "close"); task->start(); pause(); ~~~ WFTaskFactory::create_http_task()产生一个http任务，在[WFTaskFactory.h](../src/factory/WFTaskFactory.h)文件里，原型定义如下： ~~~cpp WFHttpTask *create_http_task(const std::string& url, int redirect_max, int retry_max, http_callback_t callback); ~~~ 前几个参数不用过多解释，http_callback_t是http任务的callback，定义如下： ~~~cpp using http_callback_t = std::function; ~~~ 说白了，就是一个参数为Task本身，没有返回值的函数。这个callback可以传NULL，表示无需callback。我们一切任务的callback都是这个风格。 需要说明的是，所有工厂函数不会返回失败，所以不用担心task为空指针，哪怕是url不合法。一切错误都在callback再处理。 task->get_req()函数得到任务的request，默认是GET方法，HTTP/1.1，长连接。框架会自动加上request_uri，Host等。 框架会在发送前根据需要自动加上Content-Length或Connection这些http header。用户也可以通过add_header_pair()方法添加自己的header。 关于http消息的更多接口，可以在[HttpMessage.h](../src/protocol/HttpMessage.h)中查看。 task->start()启动任务，非阻塞，并且不会失败。之后callback必然会在被调用。因为异步的原因，start()以后显然不能再用task指针了。 为了让示例尽量简单，start()之后调用pause()防止程序退出，用户需要Ctrl-C结束程序。 # 处理http抓取结果 在这个示例中，我们使用一个普遍的函数处理结果。当然，std::function支持更多的功能。 ~~~cpp void wget_callback(WFHttpTask *task) { protocol::HttpRequest *req = task->get_req(); protocol::HttpResponse *resp = task->get_resp(); int state = task->get_state(); int error = task->get_error(); // handle error states ... std::string name; std::string value; // print request to stderr fprintf(stderr, "%s %s %s\r\n", req->get_method(), req->get_http_version(), req->get_request_uri()); protocol::HttpHeaderCursor req_cursor(req); while (req_cursor.next(name, value)) fprintf(stderr, "%s: %s\r\n", name.c_str(), value.c_str()); fprintf(stderr, "\r\n"); // print response header to stderr ... // print response body to stdin const void *body; size_t body_len; resp->get_parsed_body(&body, &body_len); // always success. fwrite(body, 1, body_len, stdout); fflush(stdout); } ~~~ 在这个callback里，task就是我们通过工厂产生的task。 task->get_state()与task->get_error()分别获得任务的运行状态和错误码。我们先略过错误处理的部分。 task->get_resp()得到任务的response，这个和request区别不大，都是HttpMessage的派生。 之后通过HttpHeaderCursor对象，对request和response的header进行扫描。在[HttpUtil.h](../src/protocol/HttpUtil.h)可以看到Cursor的定义。 ~~~cpp class HttpHeaderCursor { public: HttpHeaderCursor(const HttpMessage *message); ... void rewind(); ... bool next(std::string& name, std::string& value); bool find(const std::string& name, std::string& value); ... }; ~~~ 相信这个cursor在使用上应该不会有什么疑惑。 之后一行resp->get_parsed_body()获得response的http body。这个调用在任务成功的状态下，必然返回true，body指向数据区。 这个调用得到的是原始的http body，不解码chunk编码。如需解码chunk编码，可使用[HttpUtil.h](../src/protocol/HttpUtil.h)里的HttpChunkCursor。 另外需要说明的是，find()接口会修改cursor内部的指针，即使用过find()过后如果仍然想对header进行遍历，需要通过rewind()接口回到cursor头部。 --- ### Tutorial 02 Redis Cli (`docs/tutorial-02-redis_cli.md`) # 实现一次redis写入与读出：redis_cli # 示例代码 [tutorial-02-redis_cli.cc](/tutorial/tutorial-02-redis_cli.cc) # 关于redis_cli 程序从命令行读入一个redis服务器地址，以及以一对key，value。执行SET命令写入这对KV，之后再读出验证写入是否成功。 程序运行方法：./redis_cli \ \ \ 为简单起见，程序需要用Ctrl-C结束。 # Redis URL的格式 redis://:password@host:port/dbnum?query#fragment 如果是SSL，则为： rediss://:password@host:port/dbnum?query#fragment password是可选项。port的缺省值是6379，dbnum缺省值0，范围0-15。 query和fragment部分工厂里不作解释，用户可自行定义。比如，用户有upstream选取需求，可以自定义query和fragment。相关内容参考upstream文档。 redis URL示例： redis://127.0.0.1/ redis://:12345678@redis.some-host.com/1 # 创建并启动Redis任务 创建Redis任务与创建http任务并没有什么区别，少了redirect_max参数。 ~~~cpp using redis_callback_t = std::function; WFRedisTask *create_redis_task(const std::string& url, int retry_max, redis_callback_t callback); ~~~ 在这个示例里，我们想在redis task里存一些用户信息，包括url和key，以便在callback里使用。 当然，我们可利用std::function来绑定参数，但在这里我们利用了task里的void *user_data指针。这是task的一个public成员。 ~~~cpp struct tutorial_task_data { std::sring url; std::string key; }; ... struct tutorial_task_data data; data.url = argv[1]; data.key = argv[2]; WFRedisTask *task = WFTaskFactory::create_redis_task(data.url, RETRY_MAX, redis_callback); protocol::RedisRequest *req = task->get_req(); req->set_request("SET", { data.key, argv[3] }); task->user_data = &data; task->start(); pause(); ~~~ 与http task的get_req()类似，redis task的get_req()返回任务对应的redis request。 RedisRequest提供的功能可以在[RedisMessage.h](../src/protocol/RedisMessage.h)查看。 其中，set_request接口用于设置redis命令。 ~~~cpp void set_request(const std::string& command, const std::vector& params); ~~~ 相信经常使用redis的人，对这个接口不会有什么疑问。但必须注意，我们的请求是禁止SELECT命令和AUTH命令的。 因为用户每次请求并不能指定具体连接，SELECT之后下一次请求并不能保证在同一个连接上发起，那么这个命令对用户来讲没有任何意义。 对数据库选择和密码的指定，请在redis URL里完成。并且，必须是每次请求的URL都带着这些信息。 另外，我们的redis client是支持cluster模式的，可以自动处理MOVED和ASK回复并重定向。用户不能自己发送ASKING命令。 # 处理请求结果 程序在SET命令成功之后，再发起一次GET命令，验证写入的结果。GET命令也用同一个callback。所以，函数里会判断这是哪个命令的结果。 同样，我们先忽略错误处理部分。 ~~~cpp void redis_callback(WFRedisTask *task) { protocol::RedisRequest *req = task->get_req(); protocol::RedisResponse *resp = task->get_resp(); int state = task->get_state(); int error = task->get_error(); protocol::RedisValue val; ... resp->get_result(val); std::string cmd; req->get_command(cmd); if (cmd == "SET") { tutorial_task_data *data = (tutorial_task_data *)task->user_data; WFRedisTask *next = WFTaskFactory::create_redis_task(data->url, RETRY_MAX, redis_callback); next->get_req()->set_request("GET", { data->key }); series_of(task)->push_back(next); fprintf(stderr, "Redis SET request success. Trying to GET...\n"); } else /* if (cmd == 'GET') */ { // print the GET result ... fprintf(stderr, "Finished. Press Ctrl-C to exit.\n"); } } ~~~ RedisValue是一次redis request得到的结果，同样在[RedisMessage.h](../src/protocol/RedisMessage.h)里可以看到其接口。 callback需要特别解释的，是series_of(task)->push_back(next)这个语句。因为这是我们第一次使用到Workflow的功能。 在这里next是我们下一个要发起的redis task，执行GET操作。我们并不是执行next->start()来启动任务，而是把next任务push_back到当前任务序列的末尾。 这两种方法的区别在于： * 用start来启动任务，任务是被立刻启动的，而push_back的方法，next任务是在callback结束之后被启动。 * 最起码的好处是，push_back方法可以保证log打印不会乱。否则，用next->start()的话，示例中"Finished."这个log可能会被先打印。 * 用start来启动下一个任务的话，当前任务序列（series）就结束了，next任务会新启动一个新的series。 * series是可以设置callback的，虽然在示例中没有用到。 * 在并行任务里，series是并行任务的一个分枝，series结束就会认为分枝结束。并行相关内容在后续教程中讲解。 总之，如果你想在一个任务之后启动下一个任务，一般是使用push_back操作来完成（还有些情况可能要用到push_front）。 而series_of()则是一个非常重要的调用，是一个不属于任何类的全局函数。其定义和实现在[Workflow.h](../src/factory/Workflow.h#L140)里： ~~~cpp static inline SeriesWork *series_of(const SubTask *task) { return (SeriesWork *)task->get_pointer(); } ~~~ 任何task都是SubTask类型的派生。而任何运行中的task，一定属于某个series。通过series_of调用，得到了任务所在的series。 而push_back是SeriesWork类的一个调用，其功能是将一个task放到series的末尾。类似调用还有push_front。本示例中，用哪个调用并没有区别。 ~~~cpp class SeriesWork { ... public: void push_back(SubTask *task); void push_front(SubTask *task); ... } ~~~ SeriesWork类在我们整个体系中，扮演重要的角色。在下一个示例中，我们将展现SeriesWork更多的功能。 --- ### Tutorial 03 Wget To Redis (`docs/tutorial-03-wget_to_redis.md`) # 任务序列的更多功能：wget_to_redis # 示例代码 [tutorial-03-wget_to_redis.cc](/tutorial/tutorial-03-wget_to_redis.cc) # 关于wget_to_redis 程序从命令行读入一条http URL和一条redis URL，把抓取下来的Http页面（以http URL为key）存入redis。 与之前两个示例不同，我们加入唤醒机制，让程序可以自动退出，无需Ctrl-C。 # 创建Http任务并设置参数 和上一个示例类似，本示例也是串行执行两个请求。最大的区别是，我们要通知主线程任务已经执行结束，并正常退出。 另外，我们多加入两个调用，限制一下http抓取返回内容的大小，以及接收回复的最大时间。 ~~~cpp WFHttpTask *http_task = WFTaskFactory::create_http_task(...); ... http_task->get_resp()->set_size_limit(20 * 1024 * 1024); http_task->set_receive_timeout(30 * 1000); ~~~ set_size_limit()是HttpMessage的调用，用于限制接收http消息时包的大小。事实上所有的协议消息都要求提供这个接口。 set_receive_timeout()是接收数据的超时，单位为ms。 上述代码限制http消息不超过20M，完整接收时间不超过30秒。我们还有更多更丰富的超时配置，后述文档中再介绍。 # 创建并启动SeriesWork 之前两组示例中，我们直接调用task->start()启动第一个任务。task->start()操作实际的工作方法是， 先创建一个以task为首任务的SeriesWork，再启动这个series。在[WFTask.h](../src/factory/WFTask.h)里，可以看到start的实现： ~~~cpp template class WFNetWorkTask : public CommRequest { public: void start() { assert(!series_of(this)); Workflow::start_series_work(this, nullptr); } ... }; ~~~ 我们想给series设置一个callback，并加入一些上下文。所以我们不使用任务的start接口，而是自己创建一个series。 SeriesWork不能new，delete，也不能派生。只能通过Workflow::create_series_work()接口产生。在[Workflow.h](../src/factory/Workflow.h)中， 通常是用这个调用： ~~~cpp using series_callback_t = std::function; class Workflow { public: static SeriesWork *create_series_work(SubTask *first, series_callback_t callback); }; ~~~ 在示例代码中，我们的用法是： ~~~cpp struct tutorial_series_context { std::string http_url; std::string redis_url; size_t body_len; bool success; }; ... struct tutorial_series_context context; ... SeriesWork *series = Workflow::create_series_work(http_task, series_callback); series->set_context(&context); series->start(); ~~~ 之前的示例，我们用task里的void *user_data指针保存上下文信息。但这个示例中，我们把上文信息放在series里， 这么做显然更合理一些，series是完整的任务链，所有任务都能得到并修改上下文。 series的callback函数在series所有任务被执行完之后调用，在这里，我们简单的用一个lamda函数，打印运行结果并唤醒主线程。 # 其余的工作 剩下的事情就没有什么特别的了，http抓取成功之后启动一个redis任务写库。如果抓取失败或http body长度为0，则不再启动redis任务。 无论是什么情况，程序都能在所有任务结束之后正常退出，因为任务都在同一个series里。 --- ### Tutorial 04 Http Echo Server (`docs/tutorial-04-http_echo_server.md`) # 第一个server：http_echo_server # 示例代码 [tutorial-04-http_echo_server.cc](/tutorial/tutorial-04-http_echo_server.cc) # 关于http_echo_server 这是一个http server，返回一个html页面，显示浏览器发送的http请求的header信息。 程序log里会打印出请求的client地址，请求序号（当前连接上的第几次请求）。当同一连接上完成10次请求，server主动关闭连接。 程序通过Ctrl-C正常结束，一切资源完全回收。 # 创建与启动http server 本示例里，我们采用http server的默认参数。创建和启动过程非常简单。 ~~~cpp WFHttpServer server(process); port = atoi(argv[1]); if (server.start(port) == 0) { pause(); server.stop(); } ... ~~~ 这个过程实在太简单，没有什么好讲。要注意start是非阻塞的，所以要pause住程序。显然你也可以启动多个server对象再pause。 server启动之后，任何时刻都可以通过stop()接口关停server。关停是非暴力式的，会等待正在服务的请求执行完。 所以，stop是一个阻塞操作。如果需要非阻塞的关闭，可使用shutdown+wait_finish接口。 start()接口有好几个重载函数，在[WFServer.h](../src/server/WFServer.h)里，可以看到如下一些接口： ~~~cpp class WFServerBase { public: /* To start TCP server. */ int start(unsigned short port); int start(int family, unsigned short port); int start(const char *host, unsigned short port); int start(int family, const char *host, unsigned short port); int start(const struct sockaddr *bind_addr, socklen_t addrlen); /* To start an SSL server */ int start(unsigned short port, const char *cert_file, const char *key_file); int start(int family, unsigned short port, const char *cert_file, const char *key_file); int start(const char *host, unsigned short port, const char *cert_file, const char *key_file); int start(int family, const char *host, unsigned short port, const char *cert_file, const char *key_file); int start(const struct sockaddr *bind_addr, socklen_t addrlen, const char *cert_file, const char *key_file); /* For graceful restart or multi-process server. */ int serve(int listen_fd); int serve(int listen_fd, const char *cert_file, const char *key_file); /* Get the listening address. Used when started a server on a random port. */ int get_listen_addr(struct sockaddr *addr, socklen_t *addrlen) const; }; ~~~ 这些接口都比较好理解。任何一个start函数，当端口号为0时，将使用随机端口。此时用户可能需要在server启动完成之后通过get_listen_addr获得实际监听地址。 启动SSL server时，cert_file和key_file为PEM格式。 最后两个带listen_fd的serve()接口，主要用于优雅重启。或者简单建立一个非TCP协议（如SCTP）的server。 需要特别提醒一下，我们一个server对象对应一个listen_fd，如果在IPv4和IPv6两个协议上都运行server，需要： ~~~cpp { WFHttpServer server_v4(process); WFHttpServer server_v6(process); server_v4.start(AF_INET, port); server_v6.start(AF_INET6, port); ... // now stop... server_v4.shutdown(); /* shutdown() is nonblocking */ server_v6.shutdown(); server_v4.wait_finish(); server_v6.wait_finish(); } ~~~ 这种方式我们没有办法让两个server共享连接记数。所以推荐只启动IPv6 server，因为IPv6 server可以接受IPv4的连接。 # http echo server的业务逻辑 我们看到在构造http server的时候，传入了一个process参数，这也是一个std::function，定义如下： ~~~cpp using http_process_t = std::function; using WFHttpServer = WFServer; template<> WFHttpServer::WFServer(http_process_t proc) : WFServerBase(&HTTP_SERVER_PARAMS_DEFAULT), process(std::move(proc)) { } ~~~ 其实这个http_proccess_t和的http_callback_t类型是完全一样的。都是处理一个WFHttpTask。 对server来讲，我们的目标就是根据request，填写好response。 同样我们用一个普通函数实现process。逐条读出request的http header写入html页面。 ~~~cpp void process(WFHttpTask *server_task) { protocol::HttpRequest *req = server_task->get_req(); protocol::HttpResponse *resp = server_task->get_resp(); long seq = server_task->get_task_seq(); protocol::HttpHeaderCursor cursor(req); std::string name; std::string value; char buf[8192]; int len; /* Set response message body. */ resp->append_output_body_nocopy("", 6); len = snprintf(buf, 8192, "

%s %s %s

", req->get_method(), req->get_request_uri(), req->get_http_version()); resp->append_output_body(buf, len); while (cursor.next(name, value)) { len = snprintf(buf, 8192, "

%s: %s

", name.c_str(), value.c_str()); resp->append_output_body(buf, len); } resp->append_output_body_nocopy("", 7); /* Set status line if you like. */ resp->set_http_version("HTTP/1.1"); resp->set_status_code("200"); resp->set_reason_phrase("OK"); resp->add_header_pair("Content-Type", "text/html"); resp->add_header_pair("Server", "Sogou WFHttpServer"); if (seq == 9) /* no more than 10 requests on the same connection. */ resp->add_header_pair("Connection", "close"); // print log ... } ~~~ 大多数HttpMessage相关操作之前已经介绍过了，在这里唯一的一个新操作是append_output_body()。 显然让用户生成完整的http body再传给我们并不太高效。用户只需要调用append接口，把离散的数据一块块扩展到message里就可以了。 append_output_body()操作会把数据复制走，另一个带_nocopy后缀的接口会直接引用指针，使用时需要注意不可以指向局部变量。 相关几个调用在[HttpMessage.h](../src/protocol/HttpMessage.h)可以看到其声明： ~~~cpp class HttpMessage { public: bool append_output_body(const void *buf, size_t size); bool append_output_body_nocopy(const void *buf, size_t size); ... bool append_output_body(const std::string& buf); }; ~~~ 再次强调，使用append_output_body_nocopy()接口时，buf指向的数据的生命周期至少需要延续到task的callback。 函数中另外一个变量seq，通过server_task->get_task_seq()得到，表示该请求是当前连接上的第几次请求，从0开始计。 程序中，完成10次请求之后就强行关闭连接，于是： ~~~cpp if (seq == 9) /* no more than 10 requests on the same connection. */ resp->add_header_pair("Connection", "close"); ~~~ 关闭连接还可以通过task->set_keep_alive()接口来完成，但对于http协议，还是推荐使用设置header的方式。 这个示例中，因为返回的页面很小，我们没有关注回复成功与否。下一个示例http_proxy我们将看到如果获得回复的状态。 --- ### Tutorial 05 Http Proxy (`docs/tutorial-05-http_proxy.md`) # 异步server的示例：http_proxy # 示例代码 [tutorial-05-http_proxy.cc](/tutorial/tutorial-05-http_proxy.cc) # 关于http_proxy 这是一个http代理服务器，可以配置在浏览器里使用。支持所有的http method。 因为https代理的原理不同，这个示例并不支持https代理，你只能浏览http网站。 这个proxy在实现上需要抓取下来完整的http页面再转发，下载/上传大文件会有延迟。 # 修改server配置 之前的示例我们使用了默认的http server参数。但这个例子里，我们做一点修改，限制请求的大小，防止被恶意攻击。 ~~~cpp int main(int argc, char *argv[]) { ... struct WFServerParams params = HTTP_SERVER_PARAMS_DEFAULT; params.request_size_limit = 8 * 1024 * 1024; WFHttpServer server(¶ms, process); if (server.start(port) == 0) { pause(); server.stop(); } else { perror("cannot start server"); exit(1); } return 0; } ~~~ 与上一个示例不同，我们在server构造，多传入一个参数结构。我们可以看看http server有哪些配置。 在[WFHttpServer.h](../src/server/WFHttpServer.h)里，http server的默认参数如下： ~~~cpp static constexpr struct WFServerParams HTTP_SERVER_PARAMS_DEFAULT = { .transport_type = TT_TCP, .max_connections = 2000, .peer_response_timeout = 10 * 1000, .receive_timeout = -1, .keep_alive_timeout = 60 * 1000, .request_size_limit = (size_t)-1, .ssl_accept_timeout = 10 * 1000, }; ~~~ transport_type：传输层协议，默认为TCP。除了TT_TCP外，可选择的还有TT_UDP和Linux下支持的TT_SCTP。 max_connections：最大连接数2000，达到上限之后会关闭最久未使用的keep-alive连接。没找到keep-alive连接，则拒绝新连接。 peer_response_timeout：每读取到一块数据或发送出一块数据的超时时间为10秒。 receive_timeout：接收一条完整的请求超时时间为-1，无限。 keep_alive_timeout：连接保持1分钟。 request_size_limit：请求包最大大小，无限制。 ssl_accept_timeout：完成ssl握手超时，10秒。 参数里没有send_timeout，即完整的回复超时。这个参数需要每次请求根据自己回复包的大小来确定。 # 代理服务器业务逻辑 这个代理服务器本质上是将用户请求原封不动转发到对应的web server，再将web server的回复原封不动转发给用户。 浏览器发给proxy的请求里，request uri包含了scheme和host，port，转发时需要去除。 例如，访问`http://www.sogou.com/`， 浏览器发送给proxy请求首行是： `GET` `http://www.sogou.com/` `HTTP/1.1` 需要改写为： `GET` `/` `HTTP/1.1` ~~~cpp void process(WFHttpTask *proxy_task) { auto *req = proxy_task->get_req(); SeriesWork *series = series_of(proxy_task); WFHttpTask *http_task; /* for requesting remote webserver. */ tutorial_series_context *context = new tutorial_series_context; context->url = req->get_request_uri(); context->proxy_task = proxy_task; series->set_context(context); series->set_callback([](const SeriesWork *series) { delete (tutorial_series_context *)series->get_context(); }); http_task = WFTaskFactory::create_http_task(req->get_request_uri(), 0, 0, http_callback); const void *body; size_t len; /* Copy user's request to the new task's reuqest using std::move() */ req->set_request_uri(http_task->get_req()->get_request_uri()); req->get_parsed_body(&body, &len); req->append_output_body_nocopy(body, len); *http_task->get_req() = std::move(*req); /* also, limit the remote webserver response size. */ http_task->get_resp()->set_size_limit(200 * 1024 * 1024); *series << http_task; } ~~~ 以上是process的全部内容。先解析向web server发送的http请求的构造。 req->get_request_uri()调用得到浏览器请求的完整URL，通过这个URL构建发往server的http任务。 这个http任务重试与重定向次数都是0，因为重定向是由浏览器处理，遇到302等会重新发请求。 ~~~cpp req->set_request_uri(http_task->get_req()->get_request_uri()); req->get_parsed_body(&body, &len); req->append_output_body_nocopy(body, len); *http_task->get_req() = std::move(*req); ~~~ 上面4个语句，其实是在生成发往web server的http请求。req是我们收到的http请求，我们最终要通过std::move()把它直接移动到新请求上。 第一行实际上就是将request_uri里的`http://host:port`部分去掉，只保留path之后的部分。 第二第三行把解析下来的http body指定为向外输出的http body。需要做这个操作的原因是，我们的HttpMessage实现里， 解析得到的body和发送请求的body是两个域，所以这里需要简单的设置一下，无需复制内存。 第四行，一次性把请求内容转移给向web server发送的请求。 构造好http请求后，将这个请求放到当前series末尾，process函数结束。 # 异步server的工作原理 显然process函数并不是proxy逻辑的全部，我们还需要处理从web server返回的http response，填写返回给浏览器的response。 在echo server的示例里，我们并不需要进行网络通信，直接填写返回页面就好。但proxy我们需要等待web server的结果。 我们当然可以占用这个process函数的线程，等待结果返回，但这种同步等待的方式明显不是我们想要的。 那么，我们就需要在异步得到请求结果之后，再去回复用户请求，在等待结果期间，不能占用任何的线程。 所以，在process的头部，我们给当前series设置了一个context，context里包含了proxy_task本身，以便我们异步填写结果。 ~~~cpp struct tutorial_series_context { std::string url; WFHttpTask *proxy_task; bool is_keep_alive; }; void process(WFHttpTask *proxy_task) { SeriesWork *series = series_of(proxy_task); ... tutorial_series_context *context = new tutorial_series_context; context->url = req->get_request_uri(); context->proxy_task = proxy_task; series->set_context(context); series->set_callback([](const SeriesWork *series) { delete (tutorial_series_context *)series->get_context(); }); ... } ~~~ 之前client的示例中我们说过，任何一个运行中的任务，都处在一个series里，server任务也不例外。 所以，我们可以得到当前series，并设置context。其中url主要是后续打日志之用，proxy_task是主要内容，后续需要填写resp。 接下来我们就可以看看处理web server响应的部分了。 ~~~cpp void http_callback(WFHttpTask *task) { int state = task->get_state(); auto *resp = task->get_resp(); SeriesWork *series = series_of(task); tutorial_series_context *context = (tutorial_series_context *)series->get_context(); auto *proxy_resp = context->proxy_task->get_resp(); ... if (state == WFT_STATE_SUCCESS) { const void *body; size_t len; /* set a callback for getting reply status. */ context->proxy_task->set_callback(reply_callback); /* Copy the remote webserver's response, to proxy response. */ resp->get_parsed_body(&body, &len); resp->append_output_body_nocopy(body, len); *proxy_resp = std::move(*resp); ... } else { // return a "404 Not found" page ... } } ~~~ 我们只关注成功的情况。一切可以从web server得到一个完整http页面，不管什么返回码，都是成功。所有失败的情况，简单返回一个404页面。 因为返回给用户的数据可能很大，在我们这个示例里，设置为200MB上限。所以，和之前的示例不同，我们需要查看reply成功/失败状态。 http server任务和我们自行创建的http client任务的类型是完全相同的，都是WFHttpTask。不同的是server任务是框架创建的，它的callback初始为空。 server任务的callback和client一样，是在http交互完成之后被调用。所以，对server任务来讲，就是reply完成之后被调用。 后面三行代码我们应该很熟悉了，无拷贝地将web server响应包转移到proxy响应包。 在这个http_callback函数结束之后，对浏览器的回复被发送出，一切都是在异步的过程中进行。 剩下的一个函数是reply_callback()，在这里只为了打印一些log。在这个callback执行结束后，proxy task会被自动delete。 最后，series的callback里销毁context。 # Server回复的时机 这里需要说明一下，回复消息的时机是在series里所有其它任务被执行完后，自动回复，所以并没有task->reply()接口。 但是，有task->noreply()调用，如果对server任务执行了这个调用，在原本回复的时刻，直接关闭连接。但callback依然会被调用（状态为NOREPLY）。 在server任务的callback里，同样可以通过series_of()操作获得任务的series。那么，我们依然可以往这个series里追加新任务，虽然回复已经完成。 --- ### Tutorial 06 Parallel Wget (`docs/tutorial-06-parallel_wget.md`) # 一个简单的并行抓取：parallel_wget # 示例代码 [tutorial-06-parallel_wget.cc](/tutorial/tutorial-06-parallel_wget.cc) # 关于parallel_wget 这是我们第一个并行任务的示例。 程序从命令行读入多个http URL（以空格分割），并行抓取这些URL，并按照输入顺序将抓取结果打印到标准输出。 # 创建并行任务 之前的示例里，我们已经接触过了SeriesWork类。 * SeriesWork由任务构成，代表一系列任务的串行执行。所有任务结束，则这个series结束。 * 与SeriesWork对应的ParallelWork类，parallel由series构成，代表若干个series的并行执行。所有series结束，则这个parallel结束。 * ParallelWork是一种任务。 根据上述的定义，我们就可以动态或静态的生成任意复杂的工作流了。 Workflow类里，有两个接口用于产生并行任务： ~~~cpp class Workflow { ... public: static ParallelWork * create_parallel_work(parallel_callback_t callback); static ParallelWork * create_parallel_work(SeriesWork *const all_series[], size_t n, parallel_callback_t callback); ... }; ~~~ 第一个接口创建一个空的并行任务，第二个接口用一个series数组创建并行任务。 无论用哪个接口产生的并行任务，在启动之前都可以用ParallelWork的add_series()接口添加series。 在示例代码里，我们创建一个空的并行任务，并逐个添加series。 ~~~cpp int main(int argc, char *argv[]) { ParallelWork *pwork = Workflow::create_parallel_work(callback); SeriesWork *series; WFHttpTask *task; HttpRequest *req; tutorial_series_context *ctx; int i; for (i = 1; i < argc; i++) { std::string url(argv[i]); ... task = WFTaskFactory::create_http_task(url, REDIRECT_MAX, RETRY_MAX, [](WFHttpTask *task) { // store resp to ctx. }); req = task->get_req(); // add some headers. ... ctx = new tutorial_series_context; ctx->url = std::move(url); series = Workflow::create_series_work(task, nullptr); series->set_context(ctx); pwork->add_series(series); } ... } ~~~ 从代码中看到，我们先创建http任务，但http任务并不能直接加入到并行任务里，需要先用它创建一个series。 每个series都带有context，用于保存url和抓取结果。相关的方法我们在之前的示例里都介绍过。 # 保存和使用抓取结果 http任务的callback是一个简单的lambda函数，把抓取结果保存在自己的series context里，以便并行任务获取。 ~~~cpp task = WFTaskFactory::create_http_task(url, REDIRECT_MAX, RETRY_MAX, [](WFHttpTask *task) { tutorial_series_context *ctx = (tutorial_series_context *)series_of(task)->get_context(); ctx->state = task->get_state(); ctx->error = task->get_error(); ctx->resp = std::move(*task->get_resp()); }); ~~~ 这个做法是必须的，因为http任务在callback之后就会被回收，我们只能把resp通过std::move()操作移走。 而在并行任务的callback里，我们可以很方便的获得结果： ~~~cpp void callback(const ParallelWork *pwork) { tutorial_series_context *ctx; const void *body; size_t size; size_t i; for (i = 0; i < pwork->size(); i++) { ctx = (tutorial_series_context *)pwork->series_at(i)->get_context(); printf("%s\n", ctx->url.c_str()); if (ctx->state == WFT_STATE_SUCCESS) { ctx->resp.get_parsed_body(&body, &size); printf("%zu%s\n", size, ctx->resp.is_chunked() ? " chunked" : ""); fwrite(body, 1, size, stdout); printf("\n"); } else printf("ERROR! state = %d, error = %d\n", ctx->state, ctx->error); delete ctx; } } ~~~ 在这里，我们看到ParallelWork的两个新接口，size()和series_at(i)，分别获得它的并行series个数，和第i个并行series。 通过series->get_context()取到对应series的上下文，打印结果。打印顺序必然和我们放入顺序一致。 在这个示例中，并行任务执行完就没有其它工作了。 我们上面说过，ParallelWork是一种任务，所以同样我们可以用series_of()获得它所在的series并添加新任务。 但是，如果新任务还要使用到抓取结果，我们需要再次用std::move()把数据移到并行任务所在series的上下文里。 # 并行任务启动 并行任务是一种任务，所以并行任务的启动并没有什么特别，可以直接调用start()，也可以用它建立或启动一个series。 在这个示例里，我们启动一个series，在这个series的callback里唤醒主进程，正常退出程序。 我们也可以在并行任务的callback里唤醒主进程，程序行为上区别不大。但在series callback里唤醒更加规范一点。 --- ### Tutorial 07 Sort Task (`docs/tutorial-07-sort_task.md`) # 使用内置算法工厂：sort_task # 示例代码 [tutorial-07-sort_task.cc](/tutorial/tutorial-07-sort_task.cc) # 关于sort_task 程序从命令行读入数字n，将随机的n个正整数先升序排列，再把结果再降序排列。 程序可加入第二个参数"p"，则可以进行并行排序。例如： $ ./sort_task 100000000 p 上面的命令将先升序排列1亿个整数，再降序排列。两次排序都采用并行。 # 关于计算任务 计算任务（或称线程任务），是我们非常重要的一个功能。在使用我们任务流的时候，并不建议在callback里直接进行非常复杂的计算。 所有需要消耗大量CPU时间的计算，都可以封装成计算任务交给系统去调度。计算任务和通信任务在使用方法上并没有什么区别。 系统的算法工厂提供了一些常用的计算任务，比如排序，归并等。用户也可以很方便定义自己的计算任务。 # 创建升序排序任务 ~~~cpp int main(int argc, char *argv[]) { ... WFSortTask *task; if (use_parallel_sort) task = WFAlgoTaskFactory::create_psort_task("sort", array, end, callback); else task = WFAlgoTaskFactory::create_sort_task("sort", array, end, callback); ... task->start(); ... } ~~~ 和WFHttpTask或WFRedisTask不同，排序任务多了一个模板参数代表要排序的数组数据类型。 create_sort_task和create_psort_task分别产生一个普通排序任务和一个并行排序任务。 这两个调用的参数和返回值并没有区别。 唯一需要特别说明的是第一个参数"sort"，这个是计算队列名，用于影响内部的任务调度。本篇文档后面会介绍队列名的用法。 计算任务的启动方法与使用方法和网络通信任务并没有什么区别。 # 处理结果 和通信任务一样，我们在callback里处理结果。这个示例里，升序排序之后会再发起一次降序排序。 ~~~cpp using namespace algorithm; void callback(void SortTask *task) { SortInput *input = task->get_input(); int *first = input->first; int *last = input->last; // print result ... if (task->user_data == NULL) { auto cmp = [](int a1, int a2){ return a2 < a1; }; WFSortTask *reverse; if (use_parallel_sort) reverse = WFAlgoTaskFactory::create_psort_task("sort", first, last, cmp, callback); else reverse = WFAlgoTaskFactory::create_sort_task("sort", first, last, cmp, callback); reverse->user_data = (void *)1; /* as a flag */ series_of(task)->push_back(reverse); } else { // all done. Signal main thread to exit. ... } } ~~~ 计算任务的get_input()接口得到输入数据，get_output()得到输出数据。对于排序任务，输入和输出是相同类型，内容也完全相同。 在[WFAlgoTaskFactory.h](../src/factory/WFAlgoTaskFactory.h)里，可以看到排序任务输入输出的定义： ~~~cpp namespace algorithm { template struct SortInput { T *first; T *last; }; template using SortOutput = SortInput; } template using WFSortTask = WFThreadTask, algorithm::SortOutput>; template using sort_callback_t = std::function *)>; ~~~ 显然，input或output里的first, last分别为排序数组的首尾指针。 接下来我们会创建一个降序排序的任务，这时候，我们就需要传进去一个比较函数了。 ~~~cpp auto cmp = [](int a1, int a2)->bool{ return a2 < a1; }; reverse = WFAlgoTaskFactory::create_sort_task("sort", first, last, cmp, callback); ~~~ 可以说我们的用法和std::sort()区别不是很大。但我们的first和last是指针，而不是用iterator。 同样，用create_psort_task()可以创建一个并行排序任务。而对series的使用，和通信任务没有区别。 # 关于计算线程数的配置 如果你不做任何配置，计算调度器将使用当前机器CPU个数的线程数。你也可以通过以下的方式，修改这个值： ~~~cpp #include "workflow/WFGlobal.h" int main() { struct WFGlobalSettings settings = GLOBAL_SETTINGS_DEFAULT; settings.compute_threads = 16; WORKFLOW_library_init(&settings); ... } ~~~ 通过上面的配置，我们将创建16个线程用于计算。 # 关于并行排序算法 内置的并行排序算法，使用分块+二路归并。空间复杂度为O(1)。 算法使用全局配置的计算线程进行计算，但最多使用128个线程。因为不使用额外空间，加速比会小于线程数量，平均CPU占用也比较小。 具体实现可参考[WFAlgoTaskFactory.inl](../src/factory/WFAlgoTaskFactory.inl) # 关于计算队列名 我们的计算任务并没有优化级的概念，唯一可以影响调度顺序的是计算任务的队列名，本示例中队列名为字符串"sort"。 队列名的指定非常简单，需要说明以下几点： * 队列名是一个静态字符串，不可以无限产生新的队列名。例如不可以根据请求id来产生队列名，因为内部会为每个队列分配一小块资源。 * 当计算线程没有被100%占满，所有任务都是实时调起，队列名没有任何影响。 * 如果一个服务流程里有多个计算步骤，穿插在多个网络通信之间，可以简单的给每种计算步骤起一个名字，这个会比整体用一个名字要好。 * 如果所有计算任务用同一个名字，那么所有任务的被调度的顺序与提交顺序一致，在某些场景下会影响平均响应时间。 * 每种计算任务有一个独立名字，那么相当于每种任务之间是公平调度的，而同一种任务内部是顺序调度的，实践效果更好。 * 总之，除非机器的计算负载已经非常繁重，否则没有必要特别关心队列名，只要每种任务起一个名字就可以了。 --- ### Tutorial 08 Matrix Multiply (`docs/tutorial-08-matrix_multiply.md`) # 自定义计算任务：matrix_multiply # 示例代码 [tutorial-08-matrix_multiply.cc](/tutorial/tutorial-08-matrix_multiply.cc) # 关于matrix_multiply 程序执行代码里两个矩阵的乘法，并将相乘结果打印在屏幕上。 示例的主要目的是展现怎么实现一个自定义CPU计算任务。 # 定义计算任务 定义计算任务需要提供3个基本信息，分别为INPUT，OUTPUT，和routine。 INPUT和OUTPUT是两个模板参数，可以是任何类型。routine表示从INPUT到OUTPUT的过程，定义如下： ~~~cpp template class __WFThreadTask { ... std::function routine; ... }; ~~~ 可以看出routine是一个简单的从INPUT到OUTPUT的计算过程。INPUT指针不要求是const，但用户也可以传const INPUT *的函数。 比如一个加法任务，就可这么做： ~~~cpp struct add_input { int x; int y; }; struct add_ouput { int res; }; void add_routine(const add_input *input, add_output *output) { output->res = input->x + input->y; } typedef WFThreadTask add_task; ~~~ 在我们的矩阵乘法的示例里，输入是两个矩阵，输出为一个矩阵。其定义如下： ~~~cpp namespace algorithm { using Matrix = std::vector>; struct MMInput { Matrix a; Matrix b; }; struct MMOutput { int error; size_t m, n, k; Matrix c; }; void matrix_multiply(const MMInput *in, MMOutput *out) { ... } } ~~~ 矩阵乘法存在有输入矩阵不合法的问题，所以output里多了一个error域，用来表示错误。 # 生成计算任务 定义好输入输出的类型，以及算法的过程之后，就可以通过WFThreadTaskFactory工厂来产生计算任务了。 在[WFTaskFactory.h](../src/factory/WFTaskFactory.h)里，计算工厂类的定义如下： ~~~cpp template class WFThreadTaskFactory { private: using T = WFThreadTask; public: static T *create_thread_task(const std::string& queue_name, std::function routine, std::function callback); static T *create_thread_task(time_t seconds, long nanoseconds, const std::string& queue_name, std::function routine, std::function callback); ... }; ~~~ 这里包含两个创建任务的接口。第二个接口支持用户传入一下任务运行时间限制，我们在一下节介绍这个功能。 与之前的网络工厂类或算法工厂类略有不同，这个类需要INPUT和OUTPUT两个模板参数。 queue_name相关的知识在上一个示例里已经有介绍。routine就是你的计算过程，callback是回调。 在我们的示例里，我们看到了这个调用的使用： ~~~cpp using MMTask = WFThreadTask; using namespace algorithm; int main() { typedef WFThreadTaskFactory MMFactory; MMTask *task = MMFactory::create_thread_task("matrix_multiply_task", matrix_multiply, callback); MMInput *input = task->get_input(); input->a = {{1, 2, 3}, {4, 5, 6}}; input->b = {{7, 8}, {9, 10}, {11, 12}}; ... } ~~~ 产生了task之后，通过get_input()接口得到输入数据的指针。这个可以类比网络任务的get_req()。 任务的发起和结束什么，与网络任务并没有什么区别。同样，回调也很简单： ~~~cpp void callback(MMTask *task) // MMtask = WFThreadTask { MMInput *input = task->get_input(); MMOutput *output = task->get_output(); assert(task->get_state() == WFT_STATE_SUCCESS); if (output->error) printf("Error: %d %s\n", output->error, strerror(output->error)); else { printf("Matrix A\n"); print_matrix(input->a, output->m, output->k); printf("Matrix B\n"); print_matrix(input->b, output->k, output->n); printf("Matrix A * Matrix B =>\n"); print_matrix(output->c, output->m, output->n); } } ~~~ 普通的计算任务可以忽略失败的可能性，结束状态肯定是SUCCESS。 callback里简单打印了输入输出。如果输入数据不合法，则打印错误。 # 带运行时间限制的计算任务 显然，我们的框架无法打断用户的计算任务，因为用户的计算任务是一个函数，用户需要自行确保函数可以正常结束。 但我们支持用户指定一个时间限制，当计算无法在指定时间内完成，任务可以提前回到callback。带运行时间限制的接口定义如下： ~~~cpp template class WFThreadTaskFactory { private: using T = WFThreadTask; public: static T *create_thread_task(time_t seconds, long nanoseconds, const std::string& queue_name, std::function routine, std::function callback); ... }; ~~~ 参数seconds和nanoseconds构成了运行时限。在这里，nanoseconds的取值范围在\[0,1000000000)。 当任务无法在运行时限内结束，会直接回到callback，并且任务的状态为WFT_STATE_SYS_ERROR且错误码为ETIMEDOUT。 还是用matrix_multiply的例子，我们可以这样写： ~~~cpp void callback(MMTask *task) // MMtask = WFThreadTask { MMInput *input = task->get_input(); MMOutput *output = task->get_output(); if (task->get_state() == WFT_STATE_SYS_ERROR && task->get_error() == ETIMEDOUT) { printf("Run out of time.\n"); return; } assert(task->get_state() == WFT_STATE_SUCCESS) if (output->error) printf("Error: %d %s\n", output->error, strerror(output->error)); else { printf("Matrix A\n"); print_matrix(input->a, output->m, output->k); printf("Matrix B\n"); print_matrix(input->b, output->k, output->n); printf("Matrix A * Matrix B =>\n"); print_matrix(output->c, output->m, output->n); } } using namespace algorithm; int main() { typedef WFThreadTaskFactory MMFactory; MMTask *task = MMFactory::create_thread_task(0, 1000000, "matrix_multiply_task", matrix_multiply, callback); MMInput *input = task->get_input(); input->a = {{1, 2, 3}, {4, 5, 6}}; input->b = {{7, 8}, {9, 10}, {11, 12}}; ... } ~~~ 上面的示例，限制了任务运行时间不超过1毫秒，否则，以WFT_STATE_SYS_ERROR的状态返回。 再次提醒，我们并不会中断用户的实际运行函数。当任务超时并callback，计算函数还会一直运行直到结束。 如果用户希望函数不再继续执行，需要在代码中自行加入检查点来实现这样的功能。可以在INPUT里加入flag，例如： ~~~cpp void callback(MMTask *task) // MMtask = WFThreadTask { if (task->get_state() == WFT_STATE_SYS_ERROR && task->get_error() == ETIMEDOUT) { task->get_input()->flag = true; printf("Run out of time.\n"); return; } ... } void matrix_multiply(const MMInput *in, MMOutput *out) { while (!in->flag) { .... } } ~~~ # 算法与协议的对称性 在我们的体系里，算法与协议在一个非常抽象的层面上是具有高度对称性的。 有自定义算法的线程任务，那显然也存在自定义协议的网络任务。 自定义算法要求提供算法的过程，而自定义协议则需要用户提供序列化和反序列化的过程，[简单的用户自定义协议client/server](./tutorial-10-user_defined_protocol.md)有介绍。 无论是自定义算法还是自定义协议，我们都必须强调算法和协议都是非常纯粹的。 例如算法就是一个从INPUT到OUPUT的转换过程，算法并不知道task，series等的存在。 HTTP协议的实现上，也只关心序列化反序列化，无需要关心什么是task。而是在http task里去引用HTTP协议。 # 线程任务与网络任务的复合性 在这个示例里，我们通过WFThreadTaskFactory构建了一个线程任务。可以说这是一种最简单的计算任务构建，大多数情况下也够用了。 同样，用户可以非常简单的定义一个自有协议的server和client。 但在上一个示例里我们看到，我们可以通过算法工厂产生一个并行排序任务，这显然不是通过一个routine就能做到的。 对于网络任务，比如一个kafka任务，可能要经过与多台机器的交互才能得到结果，但对用户来讲是完全透明的。 所以，我们的任务都是具有复合性的，如果你熟练使用我们的框架，可以设计出很多复杂的组件出来。 --- ### Tutorial 09 Http File Server (`docs/tutorial-09-http_file_server.md`) # 异步IO的http server：http_file_server # 示例代码 [tutorial-09-http_file_server.cc](/tutorial/tutorial-09-http_file_server.cc) # 关于http_file_server http_file_server是一个web服务器，用户指定启动端口，根路径（默认为程序当路程），就可以启动一个web server。 用户还可以指定一个PEM格式的certificate file和key file，启动一个https web server。 程序在启动server之后，可以从命令行接受用户输入，并通过127.0.0.1地址来访问这个server。 程序主要展示了磁盘IO任务的用法。在Linux系统下，我们利用了Linux底层的aio接口，文件读取完全异步。 # 启动server 启动server这块，和之前的echo server或http proxy没有什么大区别。在这里只是多了一种SSL server的启动方式： ~~~cpp class WFServerBase { ... int start(unsigned short port, const char *cert_file, const char *key_file); ... }; ~~~ 也就是说，start操作可以指定一个PEM格式的cert文件和key文件，启动一个SSL server。 此外，我们在定义server时，用std::bind()给process绑定了一个root参数，代表服务的根路径。 ~~~cpp void process(WFHttpTask *server_task, const char *root) { ... } int main(int argc, char *argv[]) { ... const char *root = (argc >= 3 ? argv[2] : "."); auto&& proc = std::bind(process, std::placeholders::_1, root); WFHttpServer server(proc); // start server ... } ~~~ # 处理请求 与http_proxy类似，我们不占用任何线程读取文件，而是产生一个异步的读文件任务，在读取完成之后回复请求。 再次说明一下，我们需要把完整回复数据读取到内存，才开始回复消息。所以不适合用来传输太大的文件。 ~~~cpp void process(WFHttpTask *server_task, const char *root) { // generate abs path. ... int fd = open(abs_path.c_str(), O_RDONLY); if (fd >= 0) { size_t size = lseek(fd, 0, SEEK_END); void *buf = malloc(size); /* As an example, assert(buf != NULL); */ WFFileIOTask *pread_task; pread_task = WFTaskFactory::create_pread_task(fd, buf, size, 0, pread_callback); /* To implement a more complicated server, please use series' context * instead of tasks' user_data to pass/store internal data. */ pread_task->user_data = resp; /* pass resp pointer to pread task. */ server_task->user_data = buf; /* to free() in callback() */ server_task->set_callback([](WFHttpTask *t){ free(t->user_data); }); series_of(server_task)->push_back(pread_task); } else { resp->set_status_code("404"); resp->append_output_body("404 Not Found."); } } ~~~ 与http_proxy产生一个新的http client任务不同，这里我们通过factory产生了一个pread任务。 在[WFTaskFactory.h](../src/factory/WFTaskFactory.h)里，我们可以看到相关的接口。 ~~~cpp struct FileIOArgs { int fd; void *buf; size_t count; off_t offset; }; ... using WFFileIOTask = WFFileTask; using fio_callback_t = std::function; ... class WFTaskFactory { public: ... static WFFileIOTask *create_pread_task(int fd, void *buf, size_t count, off_t offset, fio_callback_t callback); static WFFileIOTask *create_pwrite_task(int fd, void *buf, size_t count, off_t offset, fio_callback_t callback); ... /* Interface with file path name */ static WFFileIOTask *create_pread_task(const std::string& path, void *buf, size_t count, off_t offset, fio_callback_t callback); static WFFileIOTask *create_pwrite_task(const std::string& path, void *buf, size_t count, off_t offset, fio_callback_t callback); }; ~~~ 无论是pread还是pwrite，返回的都是WFFileIOTask。这与不区分sort或psort，不区分client或server task是一个道理。 除这两个接口还有preadv和pwritev，返回WFFileVIOTask，以及fsync，fdatasync，返回WFFileSyncTask。可以在头文件里查看。 示例用了task的user_data域保存服务的全局数据。但对于大服务，我们推荐使用series context。可以参考前面的[proxy示例](../tutorial/tutorial-05-http_proxy.cc)。 # 处理读文件结果 ~~~cpp using namespace protocol; void pread_callback(WFFileIOTask *task) { FileIOArgs *args = task->get_args(); long ret = task->get_retval(); HttpResponse *resp = (HttpResponse *)task->user_data; /* close fd only when you created File IO task with **fd** interface. */ close(args->fd); if (ret < 0) { resp->set_status_code("503"); resp->append_output_body("503 Internal Server Error."); } else /* Use '_nocopy' carefully. */ resp->append_output_body_nocopy(args->buf, ret); } ~~~ 文件任务的get_args()得到输入参数，这里是FileIOArgs结构，如果是用文件路径名创建的文件任务，其中的fd域等于-1。 get_retval()是操作的返回值。当ret < 0, 任务错误。否则ret为读取到数据的大小。 在文件任务里，ret < 0与task->get_state() != WFT_STATE_SUCCESS完全等价。 buf域的内存我们是自己管理的，可以通过append_output_body_nocopy()传给resp。 在回复完成后，我们会free()这块内存，这个语句在process里： server_task->set_callback([](WFHttpTask *t){ free(t->user_data); }); # 命令行交互 启动server后，用户可以在控制台输入文件名来访问server。当输入文件名为空（Ctrl-D），关闭server并结束程序。 这里，我们使用了WFRepeaterTask来实现这个循环接受输入的过程。WFRepeaterTask是一种循环任务，产生的接口如下： ~~~cpp using repeated_create_t = std::function; using repeater_callback_t = std::function; class WFTaskFactory { WFRpeaterTask *create_repeater_task(repeated_create_t create, repeater_callback_t callback); }; ~~~ 通过create函数，可以创建一个repeater任务。repeater内部会反复调用create，产生一个任务并运行，直到create返回空指针。 在我们的这个示例里，create函数内部调用scanf。当用户输入为空时，create返回NULL，整个循环过程结束。 当用户输入不为空（文件名），产生一个访问127.0.0.1地址的http任务来访问我们开启的server。 ~~~cpp { auto&& create = [&scheme, port](WFRepeaterTask *)->SubTask *{ ... scanf("%1023s", buf); if (*buf == '\0') return NULL; std::string url = scheme + "127.0.0.1:" + std::to_string(port) + "/" + buf; WFHttpTask *task = WFTaskFactory::create_http_task(url, 0, 0, [](WFHttpTask *task) { ... }); return task; }; WFFacilities::WaitGroup wg(1); WFRepeaterTask *repeater; repeater = WFTaskFactory::create_repeater_task(create, [&wg](WFRepeaterTask *) { wg.done(); }); repeater->start(); wg.wait(); server.stop(); } ~~~ 最后，当create返回NULL，repeater被callback。我们关闭server并结束程序。 # 关于文件异步IO的实现 Linux操作系统支持一套效率很高，CPU占用非常少的异步IO系统调用。在Linux系统下使用我们的框架将默认使用这套接口。 我们曾经实现过一套posix aio接口用于支持其它UNIX系统，并使用线程的sigevent通知方式，但由于其效率太低，已经不再使用了。 目前，对于非Linux系统，异步IO一律是用多线程实现，在IO任务到达时，实时创建线程执行IO任务，callback回到handler线程池。 多线程IO也是macOS下的唯一选择，因为macOS没有良好的sigevent支持，posix aio行不通。 某些UNIX系统不支持fdatasync调用，这种情况下，fdatasync任务将等价于fsync任务。 --- ### Tutorial 10 User Defined Protocol (`docs/tutorial-10-user_defined_protocol.md`) # 简单的用户自定义协议client/server # 示例代码 [message.h](/tutorial/tutorial-10-user_defined_protocol/message.h) [message.cc](/tutorial/tutorial-10-user_defined_protocol/message.cc) [server.cc](/tutorial/tutorial-10-user_defined_protocol/server.cc) [client.cc](/tutorial/tutorial-10-user_defined_protocol/client.cc) # 关于user_defined_protocol 本示例设计一个简单的通信协议，并在协议上构建server和client。server将client发送的消息转换成大写并返回。 # 协议的格式 协议消息包含一个4字节的head和一个message body。head是一个网络序的整数，指明body的长度。 请求和响应消息的格式一致。 # 协议的实现 用户自定义协议，需要提供协议的序列化和反序列化方法，这两个方法都是ProtocolMessage类的虚函数。 另外，为了使用方便，我们强烈建议用户实现消息的移动构造和移动赋值（用于std::move()）。 在[ProtocolMessage.h](../src/protocol/ProtocolMessage.h)里，序列化反序列化接口如下： ~~~cpp namespace protocol { class ProtocolMessage : public CommMessageOut, public CommMessageIn { private: virtual int encode(struct iovec vectors[], int max); /* You have to implement one of the 'append' functions, and the first one * with arguement 'size_t *size' is recommmended. */ virtual int append(const void *buf, size_t *size); virtual int append(const void *buf, size_t size); ... }; } ~~~ ### 序列化函数encode * encode函数在消息被发送之前调用，每条消息只调用一次。 * encode函数里，用户需要将消息序列化到一个vector数组，数组元素个数不超过max。目前max的值为2048。 * 结构体struct iovec定义在请参考系统调用readv和writev。 * encode函数正确情况下的返回值在0到max之间，表示消息使用了多少个vector。 * 如果是UDP协议，请注意总长度不超过64k，并且使用不超过1024个vector（Linux一次writev只能1024个vector）。 * encode返回-1表示错误。返回-1时，需要置errno。如果返回值>max，将得到一个EOVERFLOW错误。错误都在callback里得到。 * 为了性能考虑vector里的iov_base指针指向的内容不会被复制。所以一般指向消息类的成员。 ### 反序列化函数append * append函数在每次收到一个数据块时被调用。因此，每条消息可能会调用多次。 * buf和size分别是收到的数据块内容和长度。用户需要把数据内容复制走。 * 如果实现了append(const void \*buf, size_t \*size)接口，可以通过修改\*size来告诉框架本次消费了多少长度。收到的size - 消费的size = 剩余的size，剩余的那部分buf会由下一次append被调起时再次收到。此功能更方便协议解析，当然用户也可以全部复制走自行管理，则无需修改\*size。 * append函数返回0表示消息还不完整，传输继续。返回1表示消息结束。-1表示错误，需要置errno。 * 总之append的作用就是用于告诉框架消息是否已经传输结束。不要在append里做复杂的非必要的协议解析。 ### errno的设置 * encode或append返回-1或其它负数都会被理解为失败，需要通过errno来传递错误原因。用户会在callback里得到这个错误。 * 如果是系统调用或libc等库函数失败（比如malloc）,libc肯定会设置好errno，用户无需再设置。 * 一些消息不合法的错误是比较常见的，比如可以用EBADMSG，EMSGSIZE分别表示消息内容错误，和消息太大。 * 用户可以选择超过系统定义errno范围的值来表示一些自定义错误。一般大于256的值是可以用的。 * 请不要使用负数errno。因为框架内部用了负数来代表SSL错误。 在我们的示例里，消息的序列化反序列化都非常的简单。 头文件[message.h](../tutorial/tutorial-10-user_defined_protocol/message.h)里，声明了request和response类： ~~~cpp namespace protocol { class TutorialMessage : public ProtocolMessage { private: virtual int encode(struct iovec vectors[], int max); virtual int append(const void *buf, size_t size); ... }; using TutorialRequest = TutorialMessage; using TutorialResponse = TutorialMessage; } ~~~ request和response类，都是同一种类型的消息。直接using就可以。 注意request和response必须可以无参数的被构造，也就是说需要有无参数的构造函数，或完全没有构造函数。 此外，通讯过程中，如果发生重试，response对象会被销毁并重新构造。因此，它最好是一个RAII类。否则处理起来会比较复杂。 [message.cc](../tutorial/tutorial-10-user_defined_protocol/message.cc)里包含了encode和append的实现： ~~~cpp namespace protocol { int TutorialMessage::encode(struct iovec vectors[], int max/*max==8192*/) { uint32_t n = htonl(this->body_size); memcpy(this->head, &n, 4); vectors[0].iov_base = this->head; vectors[0].iov_len = 4; vectors[1].iov_base = this->body; vectors[1].iov_len = this->body_size; return 2; /* return the number of vectors used, no more then max. */ } int TutorialMessage::append(const void *buf, size_t size) { if (this->head_received < 4) { size_t head_left; void *p; p = &this->head[this->head_received]; head_left = 4 - this->head_received; if (size < 4 - this->head_received) { memcpy(p, buf, size); this->head_received += size; return 0; } memcpy(p, buf, head_left); size -= head_left; buf = (const char *)buf + head_left; p = this->head; this->body_size = ntohl(*(uint32_t *)p); if (this->body_size > this->size_limit) { errno = EMSGSIZE; return -1; } this->body = (char *)malloc(this->body_size); if (!this->body) return -1; this->body_received = 0; } size_t body_left = this->body_size - this->body_received; if (size > body_left) { errno = EBADMSG; return -1; } memcpy(this->body, buf, size); if (size < body_left) return 0; return 1; } } ~~~ encode的实现非常简单，固定使用了两个vector，分别指向head和body。需要注意iov_base指针必须指向消息类的成员。 append需要保证4字节的head接收完整，再读取message body。而且我们并不能保证第一次append一定包含完整的head，所以过程略为繁琐。 append实现了size_limit功能，超过size_limit的会返回EMSGSIZE错误。用户如果不需要限制消息大小，可以忽略size_limit这个域。 由于我们要求通信协议是一来一回的，所谓的“TCP黏包”问题不需要考虑，直接当错误消息处理。 现在，有了消息的定义和实现，我们就可以建立server和client了。　 # server和client的定义 有了request和response类，我们就可以建立基于这个协议的server和client。前面的示例里我们介绍过Http协议相关的类型定义： ~~~cpp using WFHttpTask = WFNetworkTask; using http_callback_t = std::function; using WFHttpServer = WFServer; using http_process_t = std::function; ~~~ 同样的，对这个Tutorial协议，数据类型的定义并没有什么区别： ~~~cpp using WFTutorialTask = WFNetworkTask; using tutorial_callback_t = std::function; using WFTutorialServer = WFServer; using tutorial_process_t = std::function; ~~~ # server端 server与普通的http server没有什么区别。我们优先IPv6启动，这不影响IPv4的client请求。另外限制请求最多不超过4KB。 代码请自行参考[server.cc](../tutorial/tutorial-10-user_defined_protocol/server.cc) # client端 client端的逻辑是从标准IO接收用户输入，构造出请求发往server并得到结果。这里我们使用了WFRepeaterTask来实现这个重复过程，直到用户的输入为空。 此外，为了安全我们限制server回复包不超4KB。 client端唯一需要了解的就是怎么产生一个自定义协议的client任务，在[WFTaskFactory.h](../src/factory/WFTaskFactory.h)有四个接口可以选择： ~~~cpp template class WFNetworkTaskFactory { private: using T = WFNetworkTask; public: static T *create_client_task(TransportType type, const std::string& host, unsigned short port, int retry_max, std::function callback); static T *create_client_task(TransportType type, const std::string& url, int retry_max, std::function callback); static T *create_client_task(TransportType type, const ParsedURI& uri, int retry_max, std::function callback); static T *create_client_task(TransportType type, const struct sockaddr *addr, socklen_t addrlen, int retry_max, std::function callback); ... }; ~~~ 其中，TransportType指定传输层协议，目前可选的值包括TT_TCP，TT_UDP，TT_SCTP和TT_TCP_SSL。 四个接口的区别不大，在我们这个示例里暂时不需要URL，我们用域名和端口来创建任务。 如果用户需要使用Unix Domain Protocol访问server，则需要用最后一个接口，直接传入sockaddr。 实际的调用代码如下。我们派生了WFTaskFactory类，但这个派生并非必须的。 ~~~cpp using namespace protocol; class MyFactory : public WFTaskFactory { public: static WFTutorialTask *create_tutorial_task(const std::string& host, unsigned short port, int retry_max, tutorial_callback_t callback) { using NTF = WFNetworkTaskFactory; WFTutorialTask *task = NTF::create_client_task(TT_TCP, host, port, retry_max, std::move(callback)); task->set_keep_alive(30 * 1000); return task; } }; ~~~ 可以看到我们用了WFNetworkTaskFactory类来创建client任务。 接下来通过任务的set_keep_alive()接口，让连接在通信完成之后保持30秒，否则，将默认采用短连接。 client的其它代码涉及的知识点在之前的示例里都包含了。请参考[client.cc](../tutorial/tutorial-10-user_defined_protocol/client.cc) # 内置协议的请求是怎么产生的 现在系统中内置了http, redis，mysql，kafka，dns等协议。我们可以通过相同的方法产生一个http或redis任务吗？比如： ~~~cpp WFHttpTask *task = WFNetworkTaskFactory::create_client_task(...); ~~~ 需要说明的是，这样产生的http任务，会损失很多的功能，比如，无法根据header来识别是否用持久连接，无法识别重定向等。 同样，如果这样产生一个MySQL任务，可能根本就无法运行起来。因为缺乏登录认证过程。 一个kafka请求可能需要和多台broker有复杂的交互过程，这样创建的请求显然也无法完成这一过程。 可见每一种内置协议消息的产生过程都远远比这个示例复杂。同样，如果用户需要实现一个更多功能的通信协议，还有许多代码要写。 --- ### Tutorial 11 Graph Task (`docs/tutorial-11-graph_task.md`) # 有向无环图（DAG）的使用：graph_task # 示例代码 [tutorial-11-graph_task.cc](/tutorial/tutorial-11-graph_task.cc) # 关于graph_task graph_task示例通过建立一个有向无环图，演示如何用workflow框架实现更加复杂的任务间依赖关系。 # 创建DAG中的任务 DAG中的任务，可以是workflow框架的任何一种任务。在本示例中，我们创建了一个timer任务，两个http任务，以及一个go任务。 Timer执行一秒的等待，http1和http2分别抓取sogou和baidu的首页，go任务打印结果。它们之间的依赖关系如下： ~~~ +-------+ +---->| Http1 |-----+ | +-------+ | +-------+ +-v--+ | Timer | | Go | +-------+ +-^--+ | +-------+ | +---->| Http2 |-----+ +-------+ ~~~ 创建DAG中任务的方法与创建普通任务的方法没有区别，这里不再展开。 # 创建图任务 DAG图在我们的框架里也是一种任务，通过以下代码，我们可以创建一个图任务： ~~~cpp { WFGraphTask *graph = WFTaskFactory::create_graph_task([](WFGraphTask *) { printf("Graph task complete. Wakeup main process\n"); wait_group.done(); }); } ~~~ 可以看到，图任务的类型为WFGraphTask，创建函数只有一个参数，即任务的回调。显然一个新建的图任务，是一张空图。 # 创建图节点 接下来，我们需要通过之前创建的4个普通任务（timer，http_task1，http_task2，go_task），产生4个图节点： ~~~cpp { /* Create graph nodes */ WFGraphNode& a = graph->create_graph_node(timer); WFGraphNode& b = graph->create_graph_node(http_task1); WFGraphNode& c = graph->create_graph_node(http_task2); WFGraphNode& d = graph->create_graph_node(go_task); } ~~~ WFGraphTask的create_graph_node接口，产生一个图节点并返回节点的引用，用户通过这个节点引用来建立节点之间的依赖。 如果我们不为节点建立依赖直接运行图任务，那么显然所有节点都是孤立节点，将全部并发执行。 # 建立依赖 通过非常形象的'-->'运算符，我们可以建立节点的依赖关系： ~~~cpp { /* Build the graph */ a-->b; a-->c; b-->d; c-->d; } ~~~ 这样我们就建立起了上述结构的DAG图啦。 除’—>’运算符，我们同样支持’<—‘。并且它们都可以连着写。所以，以下程序都是合法且等价的： ~~~cpp { a-->b-->d; a-->c-->d; } ~~~ ~~~cpp { d<--b<--a; d<--c<--a; } ~~~ ~~~cpp { d<--b<--a-->c-->d; } ~~~ 接下来直接运行graph，或者把graph放入任务流中就可以运行啦，和一般的任务没有区别。 当然，把一个图任务变成另一个图的节点，也是完全正确的行为。 # 取消后继节点 在图任务里，我们扩展了series的cancel操作，这个操作会取消该节点的所有后继结点。 取消操作一般在节点任务的callback里执行，例如： ~~~cpp int main() { WFGraphTask *graph = WFTaskFactory::create_graph_task(graph_callback); WFHttpTask *task = WFTaskFactory::create_http_task(url, 0, 0, [](WFHttpTask *t){ if (t->get_state() != WFT_STATE_SUCCESS) series_of(t)->cancel(); }); WFGraphNode& a = graph->create_graph_node(task); WFGraphNode& b = ...; WFGraphNode& c = ...; WFGraphNode& d = ...; a-->b-->c; b-->d; graph->start(); ... } ~~~ 注意取消后继节点的操作是递归的，这个例子里，如果http任务失败，b,c,d三个节点的任务都会被取消。 # 数据传递 图节点之间目前没有统一的数据传递方法，它们并不共享某一个series。因此，节点间数据传递需要用户解决。 # 致谢 部分思路来自于[taskflow](https://github.com/taskflow/taskflow)项目。 --- ### Tutorial 12 Mysql Cli (`docs/tutorial-12-mysql_cli.md`) # 异步MySQL客户端：mysql_cli # 示例代码 [tutorial-12-mysql_cli.cc](/tutorial/tutorial-12-mysql_cli.cc) # 关于mysql_cli 教程中的mysql_cli使用方式与官方客户端相似，是一个命令行交互式的异步MySQL客户端。 程序运行方式：./mysql_cli \ 启动之后可以直接在终端输入mysql命令与db进行交互，输入quit或Ctrl-C退出。 # MySQL URL的格式 mysql://username:password@host:port/dbname?character_set=charset&character_set_results=charset - 如果以SSL连接访问MySQL，则scheme设为**mysqls://**。MySQL server 5.7及以上支持； - username和password按需填写，如果密码里包含特殊字符，需要转义后再拼接URL； ~~~cpp // 密码为：@@@@#### std::string url = "mysql://root:" + StringUtil::url_encode_component("@@@@####") + "@127.0.0.1"; ~~~ - port默认为3306； - dbname为要用的数据库名，一般如果SQL语句只操作一个db的话建议填写； - 如果用户在这一层有upstream选取需求，可以参考[upstream文档](/docs/about-upstream.md)； - character_set为client的字符集，等价于使用官方客户端启动时的参数``--default-character-set``的配置，默认utf8，具体可以参考MySQL官方文档[character-set.html](https://dev.mysql.com/doc/internals/en/character-set.html)。 - character_set_results为client、connection和results的字符集，如果想要在SQL语句里使用``SET NAME``来指定这些字符集的话，请把它配置到url的这个位置。 MySQL URL示例： mysql://root:password@127.0.0.1 mysql://@test.mysql.com:3306/db1?character_set=utf8&character_set_results=utf8 mysqls://localhost/db1?character\_set=big5 # 创建并启动MySQL任务 用户可以使用WFTaskFactory创建MySQL任务，创建接口与回调函数的用法都与workflow其他任务类似: ~~~cpp using mysql_callback_t = std::function; WFMySQLTask *create_mysql_task(const std::string& url, int retry_max, mysql_callback_t callback); void set_query(const std::string& query); ~~~ 用户创建完WFMySQLTask之后，可以对req调用 **set_query()** 写入SQL语句。 如果没调用过 **set_query()** ，task就被start起来的话，则用户会在callback里得到**WFT_ERR_MYSQL_QUERY_NOT_SET**。 其他包括callback、series、user_data等与workflow其他task用法类似。 大致使用示例如下： ~~~cpp int main(int argc, char *argv[]) { ... WFMySQLTask *task = WFTaskFactory::create_mysql_task(url, RETRY_MAX, mysql_callback); task->get_req()->set_query("SHOW TABLES;"); ... task->start(); ... } ~~~ # 支持的命令 目前支持的命令为**COM_QUERY**，已经能涵盖用户基本的增删改查、建库删库、建表删表、预处理、使用存储过程和使用事务的需求。 因为我们的交互命令中不支持选库（**USE**命令），所以，如果SQL语句中有涉及到**跨库**的操作，则可以通过**db_name.table_name**的方式指定具体哪个库的哪张表。 其他所有命令都可以**拼接**到一起通过 ``set_query()`` 传给WFMySQLTask（包括INSERT/UPDATE/SELECT/DELETE/PREPARE/CALL）。 拼接的命令会被按序执行直到命令发生错误，前面的命令会执行成功。 举个例子： ~~~cpp req->set_query("SELECT * FROM table1; CALL procedure1(); INSERT INTO table3 (id) VALUES (1);"); ~~~ # 结果解析 与workflow其他任务类似，可以用``task->get_resp()``拿到**MySQLResponse**，我们可以通过**MySQLResultCursor**遍历结果集。具体接口可以查看：[MySQLResult.h](/src/protocol/MySQLResult.h) 一次请求所对应的回复中，其数据是一个三维结构： - 一个回复中包含了一个或多个结果集（result set）； - 一个结果集的类型可能是**MYSQL_STATUS_GET_RESULT**或者**MYSQL_STATUS_OK**； - **MYSQL_STATUS_GET_RESULT**类型的结果集包含了一行或多行（row）； - 一行包含了一列或多个列，或者说一到多个阈（Field/Cell），具体数据结构为**MySQLField**和**MySQLCell**； 结果集的两种类型，可以通过``cursor->get_cursor_status()``进行判断： | |MYSQL_STATUS_GET_RESULT|MYSQL_STATUS_OK| |------|-----------------------|---------------| |SQL命令|SELECT（包括存储过程中的每一个SELECT）|INSERT / UPDATE / DELETE / ...| |对应语义|读操作，一个结果集表示一份读操作返回的二维表|写操作，一个结果集表示一个写操作是否成功| |主要接口|fetch_fields();
fetch_row(&row_arr);
...|get_insert_id();
get_affected_rows();
...| 由于拼接语句可能存在错误，因此这种情况，可以通过**MySQLResultCursor**拿到前面正确执行过的语句多个结果集，以及最后判断``resp->get_packet_type()``为**MYSQL_PACKET_ERROR**时，通过``resp->get_error_code()``和``resp->get_error_msg()``拿到具体错误信息。 一个包含n条**SELECT**语句的**存储过程**，会返回n个**MYSQL_STATUS_GET_RESULT**的结果集和1个**MYSQL_STATUS_OK**的结果集，用户自行忽略此**MYSQL_STATUS_OK**结果集即可。 具体使用从外到内的步骤应该是： 1. 判断任务状态（代表通信层面状态）：用户通过判断 ``task->get_state()`` 等于**WFT_STATE_SUCCESS**来查看任务执行是否成功； 2. 判断回复包类型（代表返回包解析状态）：调用 **resp->get_packet_type()** 查看最后一条MySQL语句的返回包类型，常见的几个类型为： - MYSQL_PACKET_OK：成功，可以用cursor遍历结果； - MYSQL_PACKET_EOF：成功，可以用cursor遍历结果； - MYSQL_PACKET_ERROR：失败或部分失败，成功的部分可以用cursor遍历结果； 3. 遍历结果集。用户可以使用**MySQLResultCursor**读取结果集中的内容，因为MySQL server返回的数据是多结果集的，因此一开始cursor会**自动指向第一个结果集**的读取位置。 4. 判断结果集状态（代表结果集读取状态）：通过 ``cursor->get_cursor_status()`` 可以拿到的几种状态： - MYSQL_STATUS_GET_RESULT：此结果集为读请求类型； - MYSQL_STATUS_END：读结果集已读完最后一行； - MYSQL_STATUS_OK：此结果集为写请求类型； - MYSQL_STATUS_ERROR：解析错误； 5. 读取**MYSQL_STATUS_OK**结果集中的基本内容： - ``unsigned long long get_affected_rows() const;`` - ``unsigned long long get_insert_id() const;`` - ``int get_warnings() const;`` - ``std::string get_info() const;`` 6. 读取**MYSQL_STATUS_GET_RESULT**结果集中的columns中每个field： - ``int get_field_count() const;`` - ``const MySQLField *fetch_field();`` - ``const MySQLField *const *fetch_fields() const;`` 7. 读取**MYSQL_STATUS_GET_RESULT**结果集中的每一行：按行读取可以使用 ``cursor->fetch_row()`` 直到返回值为false。其中会移动cursor内部对当前结果集的指向每行的offset： - ``int get_rows_count() const;`` - ``bool fetch_row(std::vector& row_arr);`` - ``bool fetch_row(std::map& row_map);`` - ``bool fetch_row(std::unordered_map& row_map);`` - ``bool fetch_row_nocopy(const void **data, size_t *len, int *data_type);`` 8. 直接把当前**MYSQL_STATUS_GET_RESULT**结果集的所有行拿出：所有行的读取可以使用 **cursor->fetch_all()** ，内部用来记录行的cursor会直接移动到最后；当前cursor状态会变成**MYSQL_STATUS_END**： - ``bool fetch_all(std::vector>& rows);`` 9. 返回当前**MYSQL_STATUS_GET_RESULT**结果集的头部：如果有必要重读这个结果集，可以使用 **cursor->rewind()** 回到当前结果集头部，再通过第7步或第8步进行读取； 10. 拿到下一个结果集：因为MySQL server返回的数据包可能是包含多结果集的（比如每个select/insert/...语句为一个结果集；或者call procedure返回的多结果集数据），因此用户可以通过 **cursor->next_result_set()** 跳到下一个结果集，返回值为false表示所有结果集已取完。 11. 返回第一个结果集：**cursor->first_result_set()** 可以让我们返回到所有结果集的头部，然后可以从第4步开始重新拿数据； 12. **MYSQL_STATUS_GET_RESULT**结果集每列具体数据MySQLCell：第7步中读取到的一行，由多列组成，每列结果为MySQLCell，基本使用接口有： - ``int get_data_type();`` 返回MYSQL_TYPE_LONG、MYSQL_TYPE_STRING...具体参考[mysql_types.h](/src/protocol/mysql_types.h) - ``bool is_TYPE() const;`` TYPE为int、string、ulonglong，判断是否是某种类型 - ``TYPE as_TYPE() const;`` 同上，以某种类型读出MySQLCell的数据 - ``void get_cell_nocopy(const void **data, size_t *len, int *data_type) const;`` nocopy接口 整体示例如下： ~~~cpp void task_callback(WFMySQLTask *task) { // step-1. 判断任务状态 if (task->get_state() != WFT_STATE_SUCCESS) { fprintf(stderr, "task error = %d\n", task->get_error()); return; } MySQLResultCursor cursor(task->get_resp()); bool test_first_result_set_flag = false; bool test_rewind_flag = false; // step-2. 判断回复包其他状态 if (resp->get_packet_type() == MYSQL_PACKET_ERROR) { fprintf(stderr, "ERROR. error_code=%d %s\n", task->get_resp()->get_error_code(), task->get_resp()->get_error_msg().c_str()); } begin: // step-3. 遍历结果集 do { // step-4. 判断结果集状态 if (cursor.get_cursor_status() == MYSQL_STATUS_OK) { // step-5. MYSQL_STATUS_OK结果集的基本内容 fprintf(stderr, "OK. %llu rows affected. %d warnings. insert_id=%llu.\n", cursor.get_affected_rows(), cursor.get_warnings(), cursor.get_insert_id()); } else if (cursor.get_cursor_status() == MYSQL_STATUS_GET_RESULT) { fprintf(stderr, "field_count=%u rows_count=%u ", cursor.get_field_count(), cursor.get_rows_count()); // step-6. 读取每个fields。这是个nocopy api const MySQLField *const *fields = cursor.fetch_fields(); for (int i = 0; i < cursor.get_field_count(); i++) { fprintf(stderr, "db=%s table=%s name[%s] type[%s]\n", fields[i]->get_db().c_str(), fields[i]->get_table().c_str(), fields[i]->get_name().c_str(), datatype2str(fields[i]->get_data_type())); } // step-8. 把所有行读出，也可以while (cursor.fetch_row(map/vector)) 按step-7拿每一行 std::vector> rows; cursor.fetch_all(rows); for (unsigned int j = 0; j < rows.size(); j++) { // step-12. 具体每个cell的读取 for (unsigned int i = 0; i < rows[j].size(); i++) { fprintf(stderr, "[%s][%s]", fields[i]->get_name().c_str(), datatype2str(rows[j][i].get_data_type())); // step-12. 判断具体类型is_string()和转换具体类型as_string() if (rows[j][i].is_string()) { std::string res = rows[j][i].as_string(); fprintf(stderr, "[%s]\n", res.c_str()); } else if (rows[j][i].is_int()) { fprintf(stderr, "[%d]\n", rows[j][i].as_int()); } // else if ... } } } // step-10. 拿下一个结果集 } while (cursor.next_result_set()); if (test_first_result_set_flag == false) { test_first_result_set_flag = true; // step-11. 返回第一个结果集 cursor.first_result_set(); goto begin; } if (test_rewind_flag == false) { test_rewind_flag = true; // step-9. 返回当前结果集头部 cursor.rewind(); goto begin; } return; } ~~~ # WFMySQLConnection 由于我们是高并发异步客户端，这意味着我们对一个server的连接可能会不止一个。而MySQL的事务和预处理都是带状态的，为了保证一次事务或预处理独占一个连接，用户可以使用我们封装的二级工厂WFMySQLConnection来创建任务，每个WFMySQLConnection保证独占一个连接，具体参考[WFMySQLConnection.h](/src/client/WFMySQLConnection.h)。 ### 1. WFMySQLConnection的创建与初始化 创建一个WFMySQLConnection的时候需要传入一个**id**，之后的调用内部都会由这个id和url去找到对应的那个连接。 初始化需要传入**url**，之后在这个connection上创建的任务就不需要再设置url了。 ~~~cpp class WFMySQLConnection { public: WFMySQLConnection(int id); int init(const std::string& url); ... }; ~~~ ### 2. 创建任务与关闭连接 通过 **create_query_task()** ，写入SQL请求和回调函数即可创建任务，该任务一定从这一个connection发出。 有时候我们需要手动关闭这个连接。因为当我们不再使用它的时候，这个连接会一直保持到MySQL server超时。期间如果使用同一个id和url去创建WFMySQLConnection的话就可以复用到这个连接。 因此我们建议如果不准备复用连接，应使用 **create_disconnect_task()** 创建一个任务，手动关闭这个连接。 ~~~cpp class WFMySQLConnection { public: ... WFMySQLTask *create_query_task(const std::string& query, mysql_callback_t callback); WFMySQLTask *create_disconnect_task(mysql_callback_t callback); } ~~~ WFMySQLConnection相当于一个二级工厂，我们约定任何工厂对象的生命周期无需保持到任务结束，以下代码完全合法： ~~~cpp WFMySQLConnection *conn = new WFMySQLConnection(1234); conn->init(url); auto *task = conn->create_query_task("SELECT * from table", my_callback); conn->deinit(); delete conn; task->start(); ~~~ ### 3. 注意事项 不可以无限制的产生id来生成连接对象，因为每个id会占用一小块内存，无限产生id会使内存不断增加。当一个连接使用完毕，可以不创建和运行disconnect task，而是让这个连接进入内部连接池。下一个connection通过相同的id和url初始化，会自动复用这个连接。 同一个连接上的多个任务并行启动，会得到EAGAIN错误。 如果在使用事务期间已经开始BEGIN但还没有COMMIT或ROLLBACK，且期间连接发生过中断，则连接会被框架内部自动重连，用户会在下一个task请求中拿到**ECONNRESET**错误。此时还没COMMIT的事务语句已经失效，需要重新再发一遍。 ### 4. 预处理 用户也可以通过WFMySQLConnection来做预处理**PREPARE**，因此用户可以很方便地用作**防SQL注入**。如果连接发生了重连，也会得到一个**ECONNRESET**错误。 ### 5. 完整示例 ~~~cpp WFMySQLConnection conn(1); conn.init("mysql://root@127.0.0.1/test"); // test transaction const char *query = "BEGIN;"; WFMySQLTask *t1 = conn.create_query_task(query, task_callback); query = "SELECT * FROM check_tiny FOR UPDATE;"; WFMySQLTask *t2 = conn.create_query_task(query, task_callback); query = "INSERT INTO check_tiny VALUES (8);"; WFMySQLTask *t3 = conn.create_query_task(query, task_callback); query = "COMMIT;"; WFMySQLTask *t4 = conn.create_query_task(query, task_callback); WFMySQLTask *t5 = conn.create_disconnect_task(task_callback); ((*t1) > t2 > t3 > t4 > t5).start(); ~~~ --- ### Tutorial 13 Kafka Cli (`docs/tutorial-13-kafka_cli.md`) # 异步Kafka客户端：kafka_cli # 示例代码 [tutorial-13-kafka_cli.cc](/tutorial/tutorial-13-kafka_cli.cc) # 编译 由于支持Kafka的多种压缩方式，因此系统需要预先安装[zlib](https://github.com/madler/zlib.git),[snappy](https://github.com/google/snappy.git),[lz4(>=1.7.5)](https://github.com/lz4/lz4.git),[zstd](https://github.com/facebook/zstd.git)等第三方库。 支持CMake和Bazel两种编译方式。 CMake：执行命令make KAFKA=y 编译独立的类库（libwfkafka.a和libwfkafka.so）支持kafka协议；cd tutorial; make KAFKA=y 可以编译kafka_cli Bazel：执行bazel build kafka 编译支持kafka协议的类库；执行bazel build kafka_cli 编译kafka_cli # 关于kafka_cli 这是一个kafka client，可以完成kafka的消息生产(produce)和消息消费(fetch)。 编译时需要在tutorial目录中执行编译命令make KAFKA=y或者在项目根目录执行make KAFKA=y tutorial。 该程序从命令行读取一个kafka broker服务器地址和本次任务的类型(produce/fetch)： ./kafka_cli \ [p/c] 程序会在执行完任务后自动退出，一切资源完全回收。 其中broker_url可以有多个url组成，多个url之间以,分割 - 形式如：kafka://host:port,kafka://host1:port... 或：**kafkas**://host:port,**kafkas**://host1:port代表使用SSL通信。 - port的默认值在普通TCP连接下是9092，SSL下为9093。 - "kafka://"前缀可以缺省。这时候使用默认使用TCL通信。 - 多个url，必须都采用TCP或都采用SSL。否则init函数返回-1，错误码为EINVAL。 - 如果用户在这一层有upstream选取需求，可以参考[upstream文档](../docs/about-upstream.md)。 Kafka broker_url示例： kafka://127.0.0.1/ kafka://kafka.host:9090/ kafka://10.160.23.23:9000,10.123.23.23,kafka://kafka.sogou kafkas://broker1.kafka.sogou,kafkas://broker2.kafka.sogou 错误的url示例（第一个broker为SSL，第二个broker非SSL）： kafkas://broker1.kafka.sogou,broker2.kafka.sogou # 实现原理和特性 kafka client内部实现上除了压缩功能外没有依赖第三方库，同时利用了workflow的高性能，在合理的配置和环境下，每秒钟可以处理几万次Kafka请求。 在内部实现上，kafka client会把一次请求按照内部使用到的broker分拆成并行parallel任务，每个broker地址对应parallel任务中的一个子任务， 这样可以最大限度的提升效率，同时利用workflow内部对连接的复用机制使得整体的连接数控制在一个合理的范围。 如果一个broker地址下有多个topic partition，为了提高吞吐，应该创建多个client，然后按照topic partition分别创建任务独立启动。 # 创建并启动Kafka任务 首先需要创建一个WFKafkaClient对象，然后调用init函数初始化WFKafkaClient对象， ~~~cpp int init(const std::string& broker_url); int init(const std::string& broker_url, const std::string& group); ~~~ 其中broker_url是kafka broker集群的地址，格式可以参考上面的broker_url， group是消费者组的group_name，用在基于消费者组的fetch任务中，如果是produce任务或者没有使用消费者组的fetch任务，则不需要使用此接口； 用消费者组的时候，可以设置heartbeat的间隔时间，时间单位是毫秒，用于维持心跳： ~~~cpp void set_heartbeat_interval(size_t interval_ms); ~~~ 后面再通过WFKafkaClient对象创建kafka任务 ~~~cpp using kafka_callback_t = std::function; WFKafkaTask *create_kafka_task(const std::string& query, int retry_max, kafka_callback_t cb); WFKafkaTask *create_kafka_task(int retry_max, kafka_callback_t cb); ~~~ 其中query中包含此次任务的类型以及topic等属性，retry_max表示最大重试次数，cb为用户自定义的callback函数，当task执行完毕后会被调用， 接着还可以修改task的默认配置以满足实际需要，详细接口可以在[KafkaDataTypes.h](../src/protocol/KafkaDataTypes.h)中查看 ~~~cpp KafkaConfig config; config.set_client_id("workflow"); task->set_config(std::move(config)); ~~~ 支持的配置选项描述如下： 配置名 | 类型 | 默认值 | 含义 ------ | ---- | -------| ------- produce_timeout | int | 100ms | produce的超时时间 produce_msg_max_bytes | int | 1000000 bytes | 单个消息的最大长度限制 produce_msgset_cnt | int | int | 10000 | 一次通信消息集合的最大条数 produce_msgset_max_bytes | int | 1000000 bytes | 一次通信消息集合的最大长度限制 fetch_timeout | int | 100ms | fetch的超时时间 fetch_min_bytes | int | 1 byte | 一次fetch通信最小消息的长度 fetch_max_bytes | int | 50M bytes | 一次fetch通信最大消息的长度 fetch_msg_max_bytes | int | 1M bytes | 一次fetch通信单个消息的最大长度 offset_timestamp | long long int | -1 | 消费者组模式下，没有找到历史offset时，初始化的offset，-2表示最久，-1表示最新 session_timeout | int | 10s | 加入消费者组初始化时的超时时间 rebalance_timeout | int | 10s | 加入消费者组同步信息阶段的超时时间 produce_acks | int | -1 | produce任务在返回之前应确保消息成功复制的broker节点数，-1表示所有的复制broker节点 allow_auto_topic_creation | bool | true | produce时topic不存在时，是否自动创建topic broker_version | char * | NULL | 指定broker的版本号，<0.10时需要手动指定 compress_type | int | NoCompress | produce消息的压缩类型 client_id | char * | NULL | 表示client的id check_crcs | bool | false | fetch任务中是否校验消息的crc32 offset_store | int | 0 | 加入消费者组时，是否使用上次提交offset，1表示使用指定的offset，0表示优先使用上次提交 sasl_mechanisms | char * | NULL | sasl认证类型，目前支持plain和scram sasl_username | char * | NULL | sasl认证所需的username sasl_password | char * | NULL | sasl认证所需的password 最后就可以调用start接口启动kafka任务。 # produce任务 1、在创建并初始化WFKafkaClient之后，可以在query中直接指定topic等信息创建WFKafkaTask任务 使用示例如下： ~~~cpp int main(int argc, char *argv[]) { ... client = new WFKafkaClient(); client->init(url); task = client->create_kafka_task("api=fetch&topic=xxx&topic=yyy", 3, kafka_callback); ... task->start(); ... } ~~~ 2、在创建完WFKafkaTask之后，先通过调用set_key, set_value, add_header_pair等方法构建KafkaRecord， 关于KafkaRecord的更多接口，可以在[KafkaDataTypes.h](../src/protocol/KafkaDataTypes.h)中查看 然后应该通过调用add_produce_record添加KafkaRecord，关于更多接口的详细定义，可以在[WFKafkaClient.h](../src/client/WFKafkaClient.h)中查看 需要注意的是，add_produce_record的第二个参数partition，当>=0是表示指定的partition，-1表示随机指定partition或者调用自定义的kafka_partitioner_t kafka_partitioner_t可以通过set_partitioner接口设置自定义规则。 使用示例如下： ~~~cpp int main(int argc, char *argv[]) { ... WFKafkaClient *client_fetch = new WFKafkaClient(); client_fetch->init(url); task = client_fetch->create_kafka_task("api=produce&topic=xxx&topic=yyy", 3, kafka_callback); task->set_partitioner(partitioner); KafkaRecord record; record.set_key("key1", strlen("key1")); record.set_value(buf, sizeof(buf)); record.add_header_pair("hk1", 3, "hv1", 3); task->add_produce_record("workflow_test1", -1, std::move(record)); ... task->start(); ... } ~~~ 3、produce还可以使用kafka支持的4种压缩协议，通过设置配置项来实现 使用示例如下： ~~~cpp int main(int argc, char *argv[]) { ... WFKafkaClient *client_fetch = new WFKafkaClient(); client_fetch->init(url); task = client_fetch->create_kafka_task("api=produce&topic=xxx&topic=yyy", 3, kafka_callback); KafkaConfig config; config.set_compress_type(Kafka_Zstd); task->set_config(std::move(config)); KafkaRecord record; record.set_key("key1", strlen("key1")); record.set_value(buf, sizeof(buf)); record.add_header_pair("hk1", 3, "hv1", 3); task->add_produce_record("workflow_test1", -1, std::move(record)); ... task->start(); ... } ~~~ # fetch任务 fetch任务支持消费者组模式和手动模式 1、手动模式 无需指定消费者组，同时需要用户指定topic、partition和offset 使用示例如下： ~~~cpp client = new WFKafkaClient(); client->init(url); task = client->create_kafka_task("api=fetch", 3, kafka_callback); KafkaToppar toppar; toppar.set_topic_partition("workflow_test1", 0); toppar.set_offset(0); task->add_toppar(toppar); ~~~ 2、消费者组模式 在初始化client的时候需要指定消费者组的名称 使用示例如下： ~~~cpp int main(int argc, char *argv[]) { ... WFKafkaClient *client_fetch = new WFKafkaClient(); client_fetch->init(url, cgroup_name); task = client_fetch->create_kafka_task("api=fetch&topic=xxx&topic=yyy", 3, kafka_callback); ... task->start(); ... } ~~~ 3、offset的提交 在消费者组模式下，用户消费消息后，可以在callback函数中，通过创建commit任务来自动提交消费的记录，使用示例如下： ~~~cpp void kafka_callback(WFKafkaTask *task) { ... commit_task = client.create_kafka_task("api=commit", 3, kafka_callback); ... commit_task->start(); ... } ~~~ # 关于client的关闭 在消费者组模式下，client在关闭之前需要调用create_leavegroup_task创建leavegroup_task， 它会发送leavegroup协议包，如果没有启动leavegroup_task，会导致消费者组没有正确退出，触发这个组的rebalance。 # 处理kafka结果 消息的结果集的数据结构是KafkaResult，可以通过调用WFKafkaTask的get_result()接口获得， 然后调用KafkaResult的fetch_record接口可以将本次task相关的record取出来，它是一个KafkaRecord的二维vector， 第一维是topic partition，第二维是某个topic partition下对应的KafkaRecord， 在[KafkaResult.h](../src/protocol/KafkaResult.h)中可以看到KafkaResult的定义 ~~~cpp void kafka_callback(WFKafkaTask *task) { int state = task->get_state(); int error = task->get_error(); // handle error states ... protocol::KafkaResult *result = task->get_result(); result->fetch_records(records); for (auto &v : records) { for (auto &w: v) { const void *value; size_t value_len; w->get_value(&value, &value_len); printf("produce\ttopic: %s, partition: %d, status: %d, offset: %lld, val_len: %zu\n", w->get_topic(), w->get_partition(), w->get_status(), w->get_offset(), value_len); } } ... protocol::KafkaResult new_result = std::move(*task->get_result()); if (new_result.fetch_records(records)) { for (auto &v : records) { if (v.empty()) continue; for (auto &w: v) { if (fp) { const void *value; size_t value_len; w->get_value(&value, &value_len); fwrite(w->get_value(), w->get_value_len(), 1, fp); } } } } ... } ~~~ # 认证 认证信息需要在配置中设置，以sasl为例: ~~~cpp int main(int argc, char *argv[]) { ... client = new WFKafkaClient(); client->init(url); task = client->create_kafka_task("api=fetch&topic=xxx&topic=yyy", 3, kafka_callback); config.set_sasl_username("fetch"); config.set_sasl_password("fetch-secret"); config.set_sasl_mech("SCRAM-SHA-256"); task->set_config(std::move(config)); ... task->start(); ... } ~~~ --- ### Tutorial 15 Name Service (`docs/tutorial-15-name_service.md`) # 自定义命名服务策略：name_service # 示例代码 [tutorial-15-name_service.cc](/tutorial/tutorial-15-name_service.cc) # 关于name_service 本示例通过一个用户定义文本文件来指定名称服务策略。文件格式定义与系统hosts文件兼容，目前也支持指向域名。例如： ~~~ 127.0.0.1 www.myhost.com 192.168.10.10 host1 wwww.sogou.com sogou # 扩展功能，'sogou'指向'www.sogou.com' ~~~ 用户在命令行输入抓取的URL和名称服务文件来抓取目标网页。如果输入URL的域名在文件中不存在，则正常使用DNS。 # 自定义名称服务策略 所有名称服务策略，从WFNSPolic派生。其唯一需要实现的是create_router_task函数。 ~~~cpp class MyNSPolicy : public WFNSPolicy { public: WFRouterTask *create_router_task(const struct WFNSParams *params, router_callback_t callback) override; .. }; ~~~ 在这个示例里，我们并不需要引入很复杂的选取策略，只需要从一个文本文件把域名做转化。 所以，我们可以把转化结果交给全局的dns resolver，让dns resolve产生真正的路由任务。 ~~~cpp WFRouterTask *MyNSPolicy::create_router_task(const struct WFNSParams *params, router_callback_t callback) { WFDnsResolver *dns_resolver = WFGlobal::get_dns_resolver(); if (params->uri.host) { FILE *fp = fopen(this->path.c_str(), "r"); if (fp) { std::string dest = this->read_from_fp(fp, params->uri.host); if (dest.size() > 0) { /* Update the uri structure's 'host' field directly. * You can also update the 'port' field if needed. */ free(params->uri.host); params->uri.host = strdup(dest.c_str()); } fclose(fp); } } /* Simply, use the global dns resolver to create a router task. */ return dns_resolver->create_router_task(params, std::move(callback)); } ~~~ 其中read_from_fp函数从文本文件中读取信息并做转换，这个函数的实现大家可以直接看源代码。 得到转换结果之后，用新的host覆盖原params里uri的host即可。最后，调用dns resover产生路由任务。 # 注册名称服务 Workflow里，可以给每个单独的域名指定一个名称服务策略。如果一个域名找不到指定策略，则使用默认。 一般情况下，默认名称服务策略即是dns resolver。下面，我们把我定义好的策略注册到输入URL的域名上： ~~~cpp int main() { ... /* Create an naming policy. */ MyNSPolicy *policy = new MyNSPolicy(filename); /* Get the global name service object.*/ WFNameService *ns = WFGlobal::get_name_service(); /* Add the our name with policy to global name service. * You can add mutilply names with one policy object. */ ns->add_policy(name, policy); ... } ~~~ 其中，name为URL里的域名。这样的话，这个域名下的所有URL，都将使用我们自定义的名称服务策略了。 在程序退出之前，我们也需要把这个策略从全局名称服务中删除，防止内存泄漏： ~~~cpp int main() { ... /* clean up */ ns->del_policy(name); delete policy; return 0; } ~~~ # 设置默认名称服务策略 在这个例子中，其实我们并没有修改默认名称服务策略。有些情况下，我们可能想让所有的host都使用这个名称服务策略。 这种情况，我们也可以修改默认的策略，让这个策略对所有的host都生效。只需要调用全局名称服务的set_default_policy函数： ~~~cpp int main() { MyNSPolicy *policy = new MyNSPolicy(filename); WFNameService *ns = WFGlobal::get_name_service(); ns->set_default_policy(policy); ... /* Reset default policy to dns resolver and clean up */ ns->set_default_policy(WFGlobal::get_dns_resolver()); delete policy; return 0; } ~~~ --- ### Tutorial 17 Dns Cli (`docs/tutorial-17-dns_cli.md`) # 使用workflow请求DNS 作为一款优秀的异步编程框架，workflow帮助用户处理了大量的细节，其中就包括域名解析，因此在大部分情况下，用户无需关心如何请求DNS服务。正如workflow中的其他模块一样，DNS解析模块设计的同样完备而优雅，若恰好需要实现一些域名解析任务，workflow中的WFDnsClient和WFDnsTask无疑是一个绝佳的选择。 [about-dns](about-dns.md)中介绍了如何配置DNS相关参数，而本篇文档的重点在于介绍如何创建DNS任务以及获取解析结果。 [tutorial-17-dns_cli.cc](/tutorial/tutorial-17-dns_cli.cc) ## 使用WFDnsClient创建任务 WFDnsClient是经过封装的高级接口，其行为类似于系统提供的`resolv.conf`配置文件，帮助用户代理了重试、search列表拼接、server轮换等功能，使用起来非常简单。WFDnsClient的初始化方式有以下几种情况，当函数返回0时表示初始化成功 - 使用一个DNS IPv4地址初始化，下述两种写法等价 ```cpp client.init("8.8.8.8"); // or client.init("dns://8.8.8.8/"); ``` - 使用一个DNS IPv6地址初始化 ```cpp client.init("[2402:4e00::]:53"); ``` - 使用DNS over TLS(DoT)地址初始化，默认端口号为853 ```cpp client.init("dnss://120.53.53.53/"); ``` - 使用多个由逗号分隔的DNS地址初始化 ```cpp client.init("dns://8.8.8.8/,119.29.29.29"); ``` - 显式指定重试策略的初始化，示例代码等价于下述`resolv.conf`描述的策略 ``` nameserver 8.8.8.8 search sogou.com tencent.com options nodts:1 attempts:2 rotate ``` ```cpp client.init("8.8.8.8", "sogou.com,tencent.com", 1, 2, true); ``` 使用WFDnsClient创建的任务默认为`DNS_TYPE_A`、`DNS_CLASS_IN`类型的解析请求，且已经设置了递归解析的选项，即`task->get_req()->set_rd(1)`。了解了`WFDnsClient`的初始化的方式，仅需八行即可发起一个DNS解析任务 ```cpp int main() { WFDnsClient client; client.init("8.8.8.8"); WFDnsTask *task = client.create_dns_task("www.sogou.com", dns_callback); task->start(); pause(); client.deinit(); return 0; } ``` ## 使用工厂函数创建任务 若不需要WFDnsClient提供的额外功能，或想自行组织重试策略，可使用工厂函数创建任务。 使用工厂函数创建任务时，可以在`url path`中指定要被解析的域名，工厂函数创建的任务默认为`DNS_TYPE_A`、`DNS_CLASS_IN`类型的解析请求，创建后可以通过`set_question_type`和`set_question_class`修改，例如 ```cpp std::string url = "dns://8.8.8.8/www.sogou.com"; WFDnsTask *task = WFTaskFactory::create_dns_task(url, 0, dns_callback); protocol::DnsRequest *req = task->get_req(); req->set_rd(1); req->set_question_type(DNS_TYPE_AAAA); req->set_question_class(DNS_CLASS_IN); ``` 若不在创建任务时指定要被解析的域名(此时默认的任务是对根域名`.`进行解析)，在创建任务后可以使用`set_question`函数设置域名等参数，例如 ```cpp std::string url = "dns://8.8.8.8/"; WFDnsTask *task = WFTaskFactory::create_dns_task(url, 0, dns_callback); protocol::DnsRequest *req = task->get_req(); req->set_rd(1); req->set_question("www.zhihu.com", DNS_TYPE_AAAA, DNS_CLASS_IN); ``` ## 借助工具获取结果 一次成功的DNS请求会获得完整的DNS请求结果，有两种简便的接口可以从结果中获取信息 ### DnsUtil::getaddrinfo 该函数类似于系统的`getaddrinfo`函数，调用成功时返回零并成功获得一组`struct addrinfo`，调用失败时返回`EAI_*`类型的错误码。对该函数的成功调用最终**都应该**使用`DnsUtil::freeaddrinfo`释放资源 ```cpp void dns_callback(WFDnsTask *task) { // ignore handle error states struct addrinfo *res; protocol::DnsResponse *resp = task->get_resp(); int ret = protocol::DnsUtil::getaddrinfo(resp, 80, &res); // ignore check ret == 0 char ip_str[INET6_ADDRSTRLEN + 1] = { 0 }; for (struct addrinfo *p = res; p; p = p->ai_next) { void *addr = nullptr; if (p->ai_family == AF_INET) addr = &((struct sockaddr_in *)p->ai_addr)->sin_addr; else if (p->ai_family == AF_INET6) addr = &((struct sockaddr_in6 *)p->ai_addr)->sin6_addr; if (addr) { inet_ntop(p->ai_family, addr, ip_str, p->ai_addrlen); printf("ip:%s\n", ip_str); } } protocol::DnsUtil::freeaddrinfo(res); } ``` ### DnsResultCursor `DnsUtil::getaddrinfo`一般用于获取`IPv4`、`IPv6`地址，而使用DnsResultCursor可以完整地遍历DNS结果。DNS解析结果分为answer、authority、additional三个区域，一般情况下主要内容位于answer区域，此处分别判断每个区域是否有内容，并调用`show_result`以逐一展示结果 ```cpp void dns_callback(WFDnsTask *task) { // ignore handle error states protocol::DnsResponse *resp = task->get_resp(); protocol::DnsResultCursor cursor(resp); if(resp->get_ancount() > 0) { cursor.reset_answer_cursor(); printf(";; ANSWER SECTION:\n"); show_result(cursor); } if(resp->get_nscount() > 0) { cursor.reset_authority_cursor(); printf(";; AUTHORITY SECTION\n"); show_result(cursor); } if(resp->get_arcount() > 0) { cursor.reset_additional_cursor(); printf(";; ADDITIONAL SECTION\n"); show_result(cursor); } } ``` 根据请求类型不同，结果中包含的数据可以多种多样，常见的有 - DNS_TYPE_A: IPv4类型的地址 - DNS_TYPE_AAAA: IPv6类型的地址 - DNS_TYPE_NS: 该域名的权威DNS服务器 - DNS_TYPE_CNAME: 该域名的权威名称 ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Tutorial 18 Redis Subscriber (`docs/tutorial-18-redis_subscriber.md`) # Redis订阅模式 ## 示例代码 [tutorial-18-redis_subscriber.cc](/tutorial/tutorial-18-redis_subscriber.cc) ## 创建订阅客户端和任务 在Workflow中，一个客户端网络任务通常是向服务端发出一个请求并接收一个回复，而Redis订阅任务不同，它会先发出一个订阅请求，然后源源不断地接收服务端推送过来的消息，在这个过程中，客户端还可以新增或取消channels、patterns。 用于实现Redis订阅功能的任务是`WFRedisSubscribeTask`，与普通的Redis任务不同，它不从任务工厂产生，而是需要使用`WFRedisSubscriber`来创建。例如 ```cpp WFRedisSubscriber suber; if (suber.init(url) != 0) { std::cerr << "Subscriber init failed " << strerror(errno) << std::endl; exit(1); } // ... WFRedisSubscribeTask *task; task = suber.create_subscribe_task(channels, extract, callback); task->set_watch_timeout(1000000); // 1000秒 task->start(); // 这里可以使用task的相关接口改变订阅内容 // ... task->release(); suber.deinit(); ``` 初始化`WFRedisSubscriber`需要使用`Redis URL`，这与普通Redis任务相同，不再赘述。创建订阅任务时，需要提供三个参数 - channels/patterns: 一个或多个被订阅的channel(subscribe)或pattern(psubscribe) - extract: 收到服务端推送消息时的处理函数 - callback: 任务结束后的回调函数 这个例子中为`watch_timeout`设置了一个很长的时间，若这个时间较短，且服务端长时间未推送消息，则连接会因为超时而断开，订阅任务也会直接失败，请根据实际情况合理设置。 当任务处理完成后，需要通过`task->release()`来释放这个任务，这也是与其他任务的一个不同之处。 ## 处理订阅消息 服务端推送的消息由创建任务时指定的`extract`函数处理。后续描述中，subscribe对应channel，psubscribe对应pattern。 1. 服务端推送的消息格式是具有三个元素的数组，第一个元素是字符串"message"或"pmessage"，第二个元素是该消息的channel或pattern的名称，第三个元素是消息的内容。 2. subscribe或psubscribe请求的回复是具有三个元素的数组，第一个元素是字符串"subscribe"或"psubscribe"，第二个元素是channel或pattern的名称，第三个元素是当前通过subscribe或psubscribe命令已经订阅了多少个channel或pattern，是一个整数。如果一个请求订阅了多个channel或pattern，会有多个回复。 3. unsubscribe或punsubscribe请求的回复是具有三个元素的数组，格式与订阅命令相似。当取消订阅但不指定channel或pattern时，表示取消所有该类型的订阅，对于所有已经订阅的channel或pattern，返回一个回复消息。若当前类型未订阅任何channel或pattern，则返回一个消息，其中名称部分为nil。 更多详情可参阅redis文档。 处理消息的一个示例如下，简单地将内容打印到标准输出 ```cpp void extract(WFRedisSubscribeTask *task) { auto *resp = task->get_resp(); protocol::RedisValue value; resp->get_result(value); if (value.is_array()) { for (size_t i = 0; i < value.arr_size(); i++) { if (value[i].is_string()) std::cout << value[i].string_value(); else if (value[i].is_int()) std::cout << value[i].int_value(); else if (value[i].is_nil()) std::cout << "nil"; else std::cout << "Unexpected value in array!"; std::cout << "\n"; } } else std::cout << "Unexpected value!\n"; } ``` ## 改变订阅内容 在任务过程中，可以通过下述接口新增或取消订阅，注意在带有channels或patterns参数的接口中，请勿传入空数组。 ```cpp // ... task->start(); // 新增订阅一组channels task->subscribe(channels); // 取消订阅一组channels task->unsubscribe(channels); // 取消订阅所有channels task->unsubscribe(); // 新增订阅一组patterns task->psubscribe(patterns); // 取消订阅一组patterns task->punsubscribe(patterns); // 取消订阅所有patterns task->punsubscribe(); task->release(); ``` 当所有channels和patterns都被取消订阅后，任务会直接结束，此后不能再新增订阅，请注意该细节。也可以直接通过`task->quit()`来主动结束任务。 此外，订阅模式下可以通过`task->ping()`或`task->ping(message)`向Redis服务器发起`ping`请求。当任务设置了较小的`watch_timeout`，但服务端可能长时间没有消息推送时，通过定时发出`ping`请求可以令服务端推送`pong`响应，此时任务便不会因为超时而失败。 --- ### Tutorial 19 Dns Server (`docs/tutorial-19-dns_server.md`) # 使用workflow实现DNS服务器 前述文档已经讲解了使用workflow实现服务器的方法，workflow框架贴心地为用户处理了底层逻辑和各种细节，因此本文档主要介绍如何组装DNS消息。 [tutorial-19-dns_server.cc](/tutorial/tutorial-19-dns_server.cc) DNS协议内容中包含三个section，有`DNS_ANSWER_SECTION`、`DNS_AUTHORITY_SECTION`、`DNS_ADDITIONAL_SECTION`，每个section中可包含零或多条资源记录`Resource record`。目前`protocol::DnsResponse`支持添加的资源记录类型有`DNS_TYPE_A`、`DNS_TYPE_AAAA`、`DNS_TYPE_CNAME`、`DNS_TYPE_PTR`、`DNS_TYPE_SOA`、`DNS_TYPE_SRV`、`DNS_TYPE_MX`，其接口如下所示。 ```cpp int add_a_record(int section, const char *name, uint16_t rclass, uint32_t ttl, const void *data); int add_aaaa_record(int section, const char *name, uint16_t rclass, uint32_t ttl, const void *data); int add_ns_record(int section, const char *name, uint16_t rclass, uint32_t ttl, const char *data); int add_cname_record(int section, const char *name, uint16_t rclass, uint32_t ttl, const char *data); int add_ptr_record(int section, const char *name, uint16_t rclass, uint32_t ttl, const char *data); int add_soa_record(int section, const char *name, uint16_t rclass, uint32_t ttl, const char *mname, const char *rname, uint32_t serial, int32_t refresh, int32_t retry, int32_t expire, uint32_t minimum); int add_srv_record(int section, const char *name, uint16_t rclass, uint32_t ttl, uint16_t priority, uint16_t weight, uint16_t port, const char *target); int add_mx_record(int section, const char *name, uint16_t rclass, uint32_t ttl, int16_t preference, const char *exchange); int add_raw_record(int section, const char *name, uint16_t type, uint16_t rclass, uint32_t ttl, const void *data, uint16_t dlen); ``` 例如要添加一条AAAA记录，可使用下述方式实现 ```cpp struct in6_addr addr; inet_pton(AF_INET6, "1234:5678:9abc:def0::", (void *)&addr); resp->add_aaaa_record(DNS_ANSWER_SECTION, name.c_str(), DNS_CLASS_IN, 600, &addr); ``` 对于未支持的资源记录类型，可通过`add_raw_record`接口添加，例如要添加一条TXT记录，可使用下述方式实现 ```cpp const char *raw_txt_data = "\x0dmy dns server\x0fyour dns server"; uint16_t data_len = 30; resp->add_raw_record(DNS_ANSWER_SECTION, name.c_str(), DNS_TYPE_TXT, DNS_CLASS_IN, 1200, raw_txt_data, data_len); ``` 注意，默认情况下`WFDnsServer`会启动一个UDP服务，若需要启动TCP服务，可通过修改WFServerParams中的transport_type字段为`TT_TCP`来实现。DNS客户端通常会优先使用UDP协议发起请求，当要回复的消息过大时，可仅添加部分资源记录，并通过`resp->set_tc(1)`设置截断标记，指示客户端可使用TCP协议重新请求。 --- ### Xmake (`docs/xmake.md`) # 编译 ``` // 编译workflow库 xmake // 编译test xmake -g test // 运行test文件 xmake run -g test // 编译tutorial xmake -g tutorial // 编译benchmark xmake -g benchmark ``` ## 运行 `xmake run -h` 可以查看运行哪些target 选择一个target即可运行 比如 ``` xmake run tutorial-06-parallel_wget ``` ## 安装 ``` sudo xmake install ``` ## 切换编译静态库/动态库 ``` // 编译静态库 xmake f -k static xmake -r ``` ``` // 编译动态库 xmake f -k shared xmake -r ``` `tips : -r 代表 -rebuild` ## 进行定制化裁剪 `xmake f --help` 可查看我们定制的option ``` Command options (Project Configuration): --workflow_inc=WORKFLOW_INC workflow inc (default: /media/psf/pro/workflow/_include) --upstream=[y|n] build upstream component (default: y) --consul=[y|n] build consul component --workflow_lib=WORKFLOW_LIB workflow lib (default: /media/psf/pro/workflow/_lib) --redis=[y|n] build redis component (default: y) --kafka=[y|n] build kafka component --mysql=[y|n] build mysql component (default: y) ``` 你可以通过如下命令来进行各个模块的裁剪或集成 ``` xmake f --redis=n --kafka=y --mysql=n xmake -r ``` --- ### En CONTRIBUTING (`docs/en/CONTRIBUTING.md`) # Contribution Guide Sogou C++ Workflow is community-driven and welcomes any contributor. This document outlines some conventions about development steps, commit message formatting and contact points to make it easier to get your contribution accepted. - [Code of Conduct](#code-of-conduct) - [Getting started](#getting-started) - [First Contribution](#first-contribution) - [Find a good first topic](#find-a-good-first-topic) - [Work on an existed issue](#work-on-an-existed-issue) - [File a new issue](#file-a-new-issue) - [Contributor workflow](#contributor-workflow) - [Creating Pull Requests](#creating-pull-requests) - [Code Review](#code-review) - [Testing and building](#testing-and-building) # Code of Conduct Please make sure to read and observe our [Code of Conduct](/CODE_OF_CONDUCT.md). # Getting started - Fork the repository on GitHub. - Make your changes on your fork repository. - Submit a PR. # First Contribution We will help you to contribute in different areas like filing issues, developing features, fixing critical bugs and getting your work reviewed and merged. If you have questions about the development process, feel free to [file an issue](https://github.com/sogou/workflow/issues/new/choose). We are always in need of help, be it fixing documentation, reporting bugs or writing some code. Look at places where you feel best coding practices aren't followed, code refactoring is needed or tests are missing. Here is how you get started. ### Find a good first topic You can start by finding an existing issue with the [help-wanted](https://github.com/sogou/workflow/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) and [good first issue](https://github.com/sogou/workflow/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) label in this repository. These issues are well suited for new contributors as a beginner-friendly issues. We can help new contributors who wish to work on such issues. Another good way to contribute is to find a documentation improvement, such as a missing/broken link. #### Work on an existed issue When you are willing to take on an issue, just reply on the issue. The maintainer will assign it to you. ### File a new issue While we encourage everyone to contribute code, it is also appreciated when someone reports an issue. Please follow the prompted submission guidelines while opening an issue. # Contributor workflow To contribute to the code base, please follow the workflow as defined in this section. 1. Create a topic branch from where you want to base your work. This is usually master. 2. Make commits of logical units and add test case if the change fixes a bug or adds new functionality. 3. Run tests and make sure all the tests are passed. 4. Make sure your commit messages are in the proper format. 5. Push your changes to a topic branch in your fork of the repository. 6. Submit a pull request. This is a rough outline of what a contributor's workflow looks like. For more details, you are encouraged to communicate with the reviewers before sending a pull request. Thanks for your contributions! ## Creating Pull Requests Our project generally follows the standard [github pull request](https://help.github.com/articles/about-pull-requests/) process. To submit a proposed change, please develop the code/fix and add new test cases. After that, run these local verifications before submitting pull request to predict the pass or fail of continuous integration. ## Code Review To make it easier for your Pull Request to receive reviews, break large changes into a logical series of smaller patches which individually make easily understandable changes, and in aggregate solve a broader issue. If this is an independent modification, then it is recommended that you provide a tutorial and corresponding documents, and communicate with us. ## Testing and building Make sure the the [travis-ci](https://travis-ci.com/github/sogou/workflow/pull_requests) passed. Once Your PR has been merged, you become a contributor. Thank you for your contribution! --- ### En About Config (`docs/en/about-config.md`) # About global configuration Global configuration is used to configure default global parameters to meet the actual business requirements and improve performance. The change of the global configuration must be made before you call any intefaces in the framework. Otherwise the change may not take effect. In addition, some global configuration items can be overridden in the upstream configuration. Please see upstream documents for reference. # Changing default configuration [WFGlobal.h](/src/manager/WFGlobal.h) defines the struts and the default values of the global configuration. ~~~cpp struct WFGlobalSettings { struct EndpointParams endpoint_params; struct EndpointParams dns_server_params; unsigned int dns_ttl_default; ///< in seconds, DNS TTL when network request success unsigned int dns_ttl_min; ///< in seconds, DNS TTL when network request fail int dns_threads; int poller_threads; int handler_threads; int compute_threads; ///< auto-set by system CPU number if value<=0 int fio_max_events; const char *resolv_conf_path; const char *hosts_path; }; static constexpr struct WFGlobalSettings GLOBAL_SETTINGS_DEFAULT = { .endpoint_params = ENDPOINT_PARAMS_DEFAULT, .dns_server_params = ENDPOINT_PARAMS_DEFAULT, .dns_ttl_default = 12 * 3600, .dns_ttl_min = 180, .dns_threads = 4, .poller_threads = 4, .handler_threads = 20, .compute_threads = -1, .fio_max_events = 4096, .resolv_conf_path = "/etc/resolv.conf", .hosts_path = "/etc/hosts", }; ~~~ [EndpointParams.h](/src/manager/EndpointParams.h) defines the struture of EndpointParams and the default values. ~~~cpp struct EndpointParams { size_t max_connections; int connect_timeout; int response_timeout; int ssl_connect_timeout; bool use_tls_sni; }; static constexpr struct EndpointParams ENDPOINT_PARAMS_DEFAULT = { .max_connections = 200, .connect_timeout = 10 * 1000, .response_timeout = 10 * 1000, .ssl_connect_timeout = 10 * 1000, .use_tls_sni = false, }; ~~~ If you want to change the default connecting timeout to 5 seconds, the default TTL for DNS to 1 hour and increase the number of poller threads for message deserialization to 10, you can follow the example below: ~~~cpp #include "workflow/WFGlobal.h" int main() { struct WFGlobalSettings settings = GLOBAL_SETTINGS_DEFAULT; settings.endpoint_params.connect_timeout = 5 * 1000; settings.dns_ttl_default = 3600; settings.poller_threads = 10; WORKFLOW_library_init(&settings); ... } ~~~ Most of the parameters are self-explanatory. Note: the ttl and related parameters in DNS configuration are in **seconds**. The timeout for endpoint is in **milliseconds**, and -1 indicates an infinite timeout. dns\_threads indicates the total number of threads accessing DNS in parallel, but by default, we use asynchronous DNS resolving and don't create any dns threads (Except windows platform). dns\_server\_params indicates parameters that we access DNS server, including the maximum cocurrent connections, and the DNS server's connecting and response timeout. compute\_threads indicates the number of threads used for computation. The default value is -1, meaning the number of threads is the same as the number of CPU cores in the current node. fio\_max\_events indicates the maximum number of concurrent asynchronous file IO events. resolv\_conf\_path indicates the path of dns resolving configuration file. The default value is "/etc/resolv.conf" on unix platforms and NULL on windows. On the windows platform, we still use multi-threaded dns resolving by default. hosts_path indicates the path of the **hosts** file. The default value is "/etc/hosts" on unix platforms. If resolv_conf_path is NULL, this configuration will be ignored. poller\_threads and handler\_threads are the two parameters for tuning network performance: * poller\_threads is mainly used for epoll (kqueue) and message deserialization. * handler\_threads is the number of threads for the callback and the process of a network task. All resources required by the framework are applied for when they are used for the first time. For example, if a user task does not involve DNS resolution, the asynchronous DNS resolver or DNS threads will not be created. --- ### En About Connection Context (`docs/en/about-connection-context.md`) # About connection context Connection context is an advanced programming topic in this framework. From the previous examples, we can see that we cannot assign one specific connection for a client task or a server task. However, in some business scenarios, especially for the server, we may need to maintain the connection status. In other words, we need to bind a context to a connection. In the framework, we provide a connection context for users. # Application senarios for connection context HTTP is a completely stateless protocol, and HTTP session is realized with cookies. HTTP, Kafka and other stateless protocols are most friendly with our framework. The connection used by Redis and MySQL is obviously stateful. Redis specifies the database ID on the current connection with the SELECT command. MySQL uses a completely stateful connection. When you use Redis or non-transactional MySQL client tasks in the framework, the URL already contains all the information related to the connection, including: * username and password * database name or database ID * the character set for MySQL The framework will automatically log in or select a reusable connection based on the above information, and you do not need to care about the connection context. Due to this limitation, in the framework, you cannot use the SELECT command of Redis and the USE command of MySQL. If you want to switch databases, you should use a new URL to create the task. Transactional MySQL tasks can use fixed connections. Please see MySQL documentations for relevant details. However, if you implement a server based on Redis protocol, you need to know the current connection status. By using the deleter function of connection context, users can also get notified when the connection was closed by the peer. # How to use connection context Note: generally, only the server tasks need to use the connection context, and the connection context is used only inside the process function, which is also the safest and simplest. You can also use or modify the connection context in the callback, but you should consider the concurrency problem. We’ll discuss the related issues in details. You can obtain the connection object in any network task through interfaces, and then obtain or modify the connection context. [WFTask.h](../src/factory/WFTask.h) contains a sample call: ~~~cpp template class WFNetworkTask : public CommRequest { public: virtual WFConnection *get_connection() const = 0; ... }; ~~~ [WFConneciton.h ](../src/factory/WFConnection.h)contains the interfaces for performing operations on the connection objects: ~~~cpp class WFConnection : public CommConnection { public: void *get_context() const; void set_context(void *context, std::function deleter); void set_context(void *context); void *test_set_context(void *test_context, void *new_context, std::function deleter); void *test_set_context(void *test_context, void *new_context); }; ~~~ **get\_connection()** can only be called in a process or a callback. If you call it in the callback, please check whether the return value is NULL. If you get the WFConnection object successfully, you can perform operations on the connection context. A connection context is a void \* pointer. When the connection is closed, the deleter is automatically called. When using the setting context functions without ``deleter`` argument, the original deleter will be kept unchanged. # Timing and concurrency for accessing connection context When a client task is created, the connection object is not determined. Thus, for all client tasks, you can only use the connection context in the callback. For server tasks, you may use connection context in the process or the callback. When you use connection context in a callback, you need to consider concurrency, because the same connection may be reused by multiple tasks and reach the callbacks at the same time. Therefore, we recommend that the connection context should be accessed or modified only in the process function, because the connection will not be reused or released in the process, which is the simplest and safest. Note: the process in the above paragraphs means only the places inside the process function. In the places after the process function and before the callback, get\_connection() always returns NULL. **test\_set\_context()** in the WFConnection is used to solve the concurrency issues for using connection context in the callback, but it is not recommended. In a word, if you are not very familiar with the system implementation, please use the connection context only in the process function of the server tasks. # Example: how to reduce the request header fields in HTTP/1.1 HTTP protocol is a stateless connection protocol, and a complete header must be sent for every request on the same connection. If the cookie in the request is very large, it will obviously increase the data transmission overload. You can use the server-side connection context to solve this issue. You can specify that the cookie in the HTTP request is valid for all subsequent requests on the same connection, and omit the cookie in the subsequent request headers. Please see the following codes on the server side: ~~~cpp void process(WFHttpTask *server_task) { protocol::HttpRequest *req = server_task->get_req(); protocol::HttpHeaderCursor cursor(req); WFConnection *conn = server_task->get_connection(); void *context = conn->get_context(); std::string cookie; if (cursor.find("Cookie", cookie)) { if (context) delete (std::string *)context; context = new std::string(cookie); conn->set_context(context, [](void *p) { delete (std::string *)p; }); } else if (context) cookie = *(std::string *)context; ... } ~~~ In this way, if you arrange with the client that the cookie is transmitted only at the first request of the connection, the traffic can be reduced. The implementation in the client side needs to use a new **prepare** function. Please see the codes below: ~~~cpp using namespace protocol; void prepare_func(WFHttpTask *task) { if (task->get_task_seq() == 0) task->get_req()->add_header_pair("Cookie", my_cookie); } int some_function() { WFHttpTask *task = WFTaskFactory::create_http_task(...); task->set_prepare(prepare_func); ... } ~~~ In the example, when the HTTP task is the first request on the connection, the cookie is set. If it is not the first request, according to our arrangement, we do not set the cookie. In addition, you may use the connection context safely in the **prepare** function. **prepare** will not be concurrent on the same connection. --- ### En About Counter (`docs/en/about-counter.md`) # About counter Counters are very important basic tasks in our framework. A counter is essentially a semaphore that does not occupy thread. Counters are mainly used for workflow control. It includes anonymous counters and named counters, and can realize very complex business logic. # Creating a counter As a counter is also a task, it is created through WFTaskFactory. You can create a counter with one of the following two methods: ~~~cpp using counter_callback_t = std::function; class WFTaskFactory { ... static WFCounterTask *create_counter_task(unsigned int target_value, counter_callback_t callback); static WFCounterTask *create_counter_task(const std::string& counter_name, unsigned int target_value, counter_callback_t callback); ... }; ~~~ Each counter contains a target\_value. When the count in the counter reaches the target\_value, its callback is called. The above two interfaces generate a anonymous counter and a named counter respectively. The anonymous counter directly increases the count through the count method in the WFCounterTask: ~~~cpp class WFCounterTask { public: virtual void count() { ... } ... } ~~~ If a counter\_name is passed when you create a counter, a named counter is generated, and the count can be increased with the count\_by\_name function. # Creating parallel tasks with anonymous counters In the example of [parallel wget](/docs/en/tutorial-06-parallel_wget.md), we created a ParallelWork to achieve the parallel execution of several series. With the combination of ParallelWork and SeriesWork, you can build series-parallel graphs in any form, which can meet the requirements in most scenarios. Counters allow us to build more complex dependencies between the tasks, such as a fully connected neural network. The following simple code can replace ParallelWork to realize parallel HTTP crawling. ~~~cpp void http_callback(WFHttpTask *task) { /* Save http page. */ ... WFCounterTask *counter = (WFCounterTask *)task->user_data; counter->count(); } std::mutex mutex; std::condition_variable cond; bool finished = false; void counter_callback(WFCounterTask *counter) { mutex.lock(); finished = true; cond.notify_one(); mutex.unlock(); } int main(int argc, char *argv[]) { WFCounterTask *counter = create_counter_task(url_count, counter_callback); WFHttpTask *task; std::string url[url_count]; /* init urls */ ... for (int i = 0; i < url_count; i++) { task = create_http_task(url[i], http_callback); task->user_data = counter; task->start(); } counter->start(); std::unique_lock lock(mutex); while (!finished) cond.wait(lock); lock.unlock(); return 0; } ~~~ The above code creates a counter with the target value as url\_count, and calls the count once after each HTTP task is completed. Note that the times **count()** a anonymous counter cannot exceed it's target value. Otherwise the counter may have been destroyed after the callback, and the program behavior is undefined. The call of **counter->start()** can be placed before the for loop. After a counter is created, you can call its count interface, no matter whether the counter has been started or not. You can also use **counter->WFCounterTask::count()** to call the count interface of an anonymous counter; this can be used in performance-sensitive applications. # Using a server together with other asynchronous engines In some cases, our server may need to call asynchronous clients in other frameworks and wait for the results. A simple method is that we wait synchronously in the process and then are waken up through conditional variables. Its disadvantage is that we occupy a processing thread and turn asynchronous clients in other frameworks into synchronous clients. But with the counter method, we can wait without occupying threads. The method is very simple: ~~~cpp void some_callback(void *context) { protocol::HttpResponse *resp = get_resp_from_context(context); WFCounterTask *counter = get_counter_from_context(context); /* write data to resp. */ ... counter->count(); } void process(WFHttpTask *task) { WFCounterTask *counter = WFTaskFactory::create_counter_task(1, nullptr); SomeOtherAsyncClient client(some_callback, context); *series_of(task) << counter; } ~~~ Here, we can consider the series of a server task as a coroutine, and the counter whose target value is 1 can be considered as a conditional variable. # Named counters When the count operation is executed on the anonymous counter, the counter object pointer is directly accessed. This inevitably requires that the number of calls to count should not exceed the target value during operation. But imagine an application scenario where we start four tasks at the same time, and as long as any three tasks are completed, the workflow can continue. We can use a counter with a target value of 3, and count once after each task is completed. As long as three tasks are completed, the callback of the counter will be executed. But the problem is that when the fourth task is finished and **counter->count()** is called again, the counter is already a wild pointer and the program crashes. In this case, we can use named counters to solve this problem. By naming the counter and counting by name, we can have the following implementation: ~~~cpp void counter_callback(WFCounterTask *counter) { WFRedisTask *next = WFTaskFactory::create_redis_task(...); series_of(counter)->push_back(next); } int main(void) { WFHttpTask *tasks[4]; WFCounterTask *counter; counter = WFTaskFactory::create_counter_task("c1", 3, counter_callback); counter->start(); for (int i = 0; i < 4; i++) { tasks[i] = WFTaskFactory::create_http_task(..., [](WFHttpTask *task){ WFTaskFactory::count_by_name("c1"); }); tasks[i]->start(); } ... } ~~~ In this example, four concurrent HTTP tasks are started, three of which are completed, and a Redis task is started immediately. In the practical application, you may need to add the code of data transmission. In the example, a counter named "c1" is created, and in the HTTP callback, call **WFTaskFactory::count\_by\_name()** to increase the count. ~~~cpp class WFTaskFactory { ... static int count_by_name(const std::string& counter_name); static int count_by_name(const std::string& counter_name, unsigned int n); ... }; ~~~ You can pass an integer n to **WFTaskFactory::count\_by\_name**, indicating the count value to be increased in this operation. Obviously: **count\_by\_name("c1")** is equivalent to **count\_by\_name("c1", 1)**. If the "c1" counter does not exist (not created or already completed), the operation on "c1" will have no effect, so the wild pointer problem in an anonymous counter will not happen here. The **count\_by\_name()** function returns the number of counters that was waked up by the operation. When **n** is greater that 1, more than one counter may reach target value. # Definition of the detailed behaviors of named counters When you **call WFTaskFactory::count\_by\_name(name, n)**: * if the name does not exist (not created or already completed), there is no action. * if there is only one counter with that name: * if the remaining value of the counter is less than or equal to n, the counting is completed, the callback is called, and the counter is destroyed. end. * if the remaining value of the counter is greater than n, the count value is increased by n. end. * if there are multiple counters with that name: * according to the order of creation, take the first counter and assume that its remaining value is m: * if m is greater than n, the count value is increased by n. end (the remaining value is m-n). * if m is less than or equal to n, the counting is completed, the callback is called, and the counter is destroyed. set n = n-m. * If n is 0, the procedure ends. * If n is greater than 0, take out the next counter with the same name and repeat the whole operation. Although the description is very complicated, it can be summed up in one sentence. Access all counters with that name according to the order of creation one by one until n is 0. In other words, one **count\_by\_name(name, n)** may wake up multiple counters. The counters can be used to realize very complex business logic if you can use them well. In our framework, counters are often used to implement asynchronous locks or to build channels between tasks. It is more like a control task in form. --- ### En About Dns (`docs/en/about-dns.md`) # About DNS When using a domain name to request the network, you first need to obtain the server address through domain name resolution, and then use the network address to make subsequent requests. Workflow has implemented a complete domain name resolution and caching system. Generally speaking, users can initiate network tasks smoothly without knowing the internal mechanism. ## DNS related configuration Global configuration in Workflow includes ~~~cpp struct WFGlobalSettings { struct EndpointParams endpoint_params; struct EndpointParams dns_server_params; unsigned int dns_ttl_default; unsigned int dns_ttl_min; int dns_threads; int poller_threads; int handler_threads; int compute_threads; int fio_max_events; const char *resolv_conf_path; const char *hosts_path; }; ~~~ Among them, the configuration items related to domain name resolution include * dns_server_params * address_family: This item will be explained later * max_connections: The maximum number of concurrent requests sent to the DNS server, the default is 200 * connect_timeout/response_timeout/ssl_connect_timeout: refer to [timeout](about-timeout.md) for related instructions * dns_threads: When using synchronous mode to implement domain name resolution, the resolution operation will be executed in an independent thread pool. This item specifies the number of threads in the thread pool. The default is 4. * dns_ttl_default: The result of successful domain name resolution will be placed in the domain name cache. This item specifies its survival time in seconds. The default value is 1 hour. When the resolution result expires, it will be re-parsed to obtain the latest content. * dns_ttl_min: When communication fails, the cached result may have expired. This item specifies a shorter survival time. When communication fails, the cache is updated at a more frequent rate. The unit is seconds. The default value is 1 minute. * resolv_conf_path: This file saves the configuration related to accessing DNS. It is usually located in `/etc/resolv.conf` on common Linux distributions. If this item is configured as `NULL`, it means using multi-threaded synchronous resolution mode. * hosts_path: This file is a local domain name lookup table. If the resolved domain name hits this table, it will not initiate a request to DNS. It is usually located in `/etc/hosts` on common Linux distributions. If this item is configured as `NULL` means not to use the lookup table ### resolv.conf extensions Workflow has extended the `resolv.conf` configuration file. Users can modify the configuration to support the `DNS over TLS(DoT)`. **Note** directly modifying `/etc/resolv.conf` will affect other processes. You can make a copy of the file for modification, and modify the `resolv_conf_path` configuration of Workflow to the path of the new file. For example, a `nameserver` using the `dnss` protocol will connect via SSL ~~~bash nameserver dnss://8.8.8.8/ nameserver dnss://[2001:4860:4860::8888]/ ~~~ ### Address Family In some network environments, although the machine supports IPv6, it cannot communicate with the outside because it has not been assigned a public IPv6 address (for example, the local IPv6 address starts with `fe80`). At this time, you can set `endpoint_params.address_family` to `AF_INET` to force only IPv4 addresses to be resolved during domain name resolution. Similarly, the `resolv.conf` file may specify both the IPv4 address and the IPv6 address of the `nameserver`. In this case, you can set `dns_server_params.address_family` to `AF_INET` or `AF_INET6` to force the use of only IPv4 or IPv6 addresses to access DNS. ### Use Upstream configuration The global configuration takes effect for each domain name by default. If you need to specify different configurations for certain domain names, you can use the [Upstream](./about-upstream.md#Address attribute) function. Using Upstream, you can individually specify the `dns_ttl_default` and `dns_ttl_min` configuration items, and individually specify the IP address family used by the domain name through `endpoint_params.address_family`. ## Domain name resolution and caching strategy Network tasks usually require domain name resolution to obtain the IP address that needs to be accessed. The relevant strategies for domain name resolution in Workflow are as follows: 1. Check whether the domain name cache has the IP address corresponding to the domain name. If there is a cache and it has not expired, use this set of IP addresses. 2. Check whether the domain name is an IPv4, IPv6 address or `Unix Domain Socket`. If so, use the address directly without initiating domain name resolution. 3. Check whether the `hosts_path` file contains the IP address corresponding to the domain name. If so, use the address directly. 4. Obtain an asynchronous lock to ensure that a resolution request for the same domain name is only initiated once at the same time, and initiate a resolution request to DNS 5. After successful parsing, the parsing result will be saved to the domain name cache of the current process for next use, and the asynchronous lock will be released. 6. After the parsing fails, the asynchronous lock will be released and the failure reason will be notified to all tasks waiting on the same asynchronous lock. New tasks initiated after the notification is completed will request DNS again. Many scenarios that require a large number of network requests will be equipped with a domain name caching component. If a resolution request is sent to the DNS every time a network task is initiated, the DNS will inevitably be overwhelmed. Workflow sets the cache survival time (dns_ttl_default and dns_ttl_min) to ensure that the cache will expire after a reasonable period of time and the domain name resolution results can be updated in a timely manner. When a cache item of a domain name expires, the first task found to be expired will extend its survival time by 5 seconds and initiate a resolution request to DNS. Requests on the same domain name within 5 seconds will directly use the cached DNS resolution results without waiting. The asynchronous lock mechanism can ensure that the resolution request for the **same domain name** is only initiated once at the same time. Without lock protection, if a large number of network tasks are initiated for the same domain name in a short period of time, each task will be unable to be retrieved from the cache. Too many resolution request to DNS will place a large and unnecessary burden on DNS. The same domain name here represents the `(host, port, family)` triplet. If a domain name is required to only use IPv4 and IPv6 through Upstream, they will be protected by different asynchronous locks, and it is possible to request DNS at the same time. ### Asynchronous domain name resolution Workflow implements a complete DNS task. If the `resolv_conf_path` configuration item is specified, an asynchronous request will be used when initiating domain name resolution to DNS. Under Unix-like systems, Workflow uses `/etc/resolv.conf` as the value of this configuration by default. Asynchronous domain name resolution does not block any threads or monopolize the thread pool, and can complete the task of domain name resolution more efficiently. ### Synchronous domain name resolution If `resolv_conf_path` is specified as `NULL`, synchronous domain name resolution will be achieved by calling the `getaddrinfo` function. This method will use an independent thread pool, and the number of threads is configured through the `dns_threads` parameter. If a large number of domain name resolution requests need to be initiated in a short period of time, the synchronization method will cause a large delay. --- ### En About Error (`docs/en/about-error.md`) # About error handling Error handling is an important and complex problem in any software system. Within our framework, error handling is ubiquitous and extremely cumbersome. In the interfaces we exposed to users, we try to make things as simple as possible, but users still inevitably need to know some error messages. ### Disabling C++ exceptions C++ exceptions are not used in our framework. When you compile your own code, it is best to add **-fno-exceptions** flag to reduce the code size. According to the common practice in the industry, we ignore the possibility of the failure of **new** operation, and avoid using new to allocate large blocks of memory internally. And there are error checks in memory allocation in C style. ### About factory functions From the previous examples, you can see that all task and series are generated from two factory classes, WFTaskFactory or Workflow. These factory classes, as well as more factory class interfaces that we may encounter in the future, ensure success. In other words, they never return NULL. And you do not need to check the return value. To achieve this goal, even when the URL is illegal, the factory still generates the task normally. And you will get the error in the callback of the task. ### States and error codes of a task In the previous examples, you often see such codes in the callback: ~~~cpp void callback(WFXxxTask *task) { int state = task->get_state(); int error = task->get_error(); ... } ~~~ in which, the state indicates the end state of a task. [WFTask.h](/src/factory/WFTask.h) contains all possible states: ~~~cpp enum { WFT_STATE_UNDEFINED = -1, WFT_STATE_SUCCESS = CS_STATE_SUCCESS, WFT_STATE_TOREPLY = CS_STATE_TOREPLY, /* for server task only */ WFT_STATE_NOREPLY = CS_STATE_TOREPLY + 1, /* for server task only */ WFT_STATE_SYS_ERROR = CS_STATE_ERROR, WFT_STATE_SSL_ERROR = 65, WFT_STATE_DNS_ERROR = 66, /* for client task only */ WFT_STATE_TASK_ERROR = 67, WFT_STATE_ABORTED = CS_STATE_STOPPED /* main process terminated */ }; ~~~ ##### Please note the following states: * SUCCESS: the task is successfully completed. The client receives the complete reply, or the server writes the reply completely into the send buffer (but there is no guarantee that the peer will receive it). * SYS\_ERROR: system error. In this case, use **task->get\_error()** to get the system error code **errno**. * When **get\_error()** gets ETIMEDOUT, you can call **task->get\_timeout\_reason()** to get the timeout reasons. * DNS\_ERROR: DNS resolution error. Use **get\_error()** to get the return code of **getaddrinfo()**. For DNS, please see the article for details [about-dns.md](/docs/en/about-dns.md). * The server task never has a DNS\_ERROR. * SSL\_ERROR: SSL error. Use **get\_error()** to get the return value of **SSL\_get\_error()**. * Currently SSL error information is not complete, and you can not get the value of **ERR\_get\_error()**. Therefore, basically there are three possible return value of **get\_error()**: * SSL\_ERROR\_ZERO\_RETURN, SSL\_ERROR\_X509\_LOOKUP, SSL\_ERROR\_SSL. * We will consider adding more detailed SSL error information in the future versions. * TASK\_ERROR: task errors. Common errors include illegal URL, login failure, etc. [WFTaskError.h](/src/factory/WFTaskError.h) lists the return values of **get\_error()**. ##### You do not need to pay attention to the following states: * UNDEFINED: Client tasks that have just been created and have not yet been run are in UNDEFINED state. * TOREPLY: Server tasks that have not sent replies or called **task->noreply()** are in TOREPLY state. * NOREPLY: Server tasks that have called **task->noreply()** are always in NOREPLY state. The callback of these tasks are also in NOREPLY state. And the connection will be closed. ### Other error handling requirements In addition to the error handling of the task itself, you also need to check the errors of the message interfaces of various protocols. Generally, these interfaces indicate errors by returning false, and show the error reasons in the errno. In addition, you may encounter more complicated error messages when you use some complex operations. You will learn them in detailed documents. --- ### En About Exit (`docs/en/about-exit.md`) # About exit As most of our calls are non-blocking, we need some mechanisms to prevent the main function from exiting early in the previous examples. For example, in the wget example, we wait for the user's Ctrl-C, or in the parallel\_wget example, we wake up the main thread after all crawling tasks are finished. In several server examples, the **stop()** operation is blocking, which can ensure the normal end of all server tasks and the safe exit of the main thread. # Principles on the safe exit Generally, as long as you writes the program normally and follows the methods in the examples, there is little doubt about exit. However, it is still necessary to define the conditions for normal program exit. * You can't call **exit()** of the system in any callback functions such as the callback or the process, otherwise the behavior is undefined. * The condition that a main thread can safely end (call **exit()** or return in the main function) is that all tasks have been run to callbacks and no new tasks is started. * All our examples are consistent with this assumption, waking up the main function in the callback. This is safe, and there is no need to worry about the situation where the callback is not finished when the main function returns. * ParallelWork is a kind of tasks, which also needs to run to its callback. * This rule can be violated under certain circumstances. We will talk about it in the following section. * All server must stop, otherwise the behavior is undefined. Because all users know how to call the stop operation, generally a server program will not have any exit problems. * Server's stop() method will block until all server tasks' series end. But if you start a task directly in process function, you have to take care of the end this task. # Why do I need to wait for the callback of a running task? Can the program be ended early? First, explain why you need to wait till the callback of tasks before ending the program. In most cases, the tasks generated through the task factory are composite tasks. For example, an http client task may need to resolve the dns, and then initiate the http crawl. And if a 302 redirect is encountered, dns resolving may be needed again. If the task fails, retrying may be involved. In other words, any asynchronous tasks may contain multiple asynchronous processes, but it is completely insensitive to users. But between each internal asynchronous process, it does not check whether the program has exited. If the user clearly knows that a task is an atomic task, for example, an http task created with an IP address (or a dns cache can definitely be hit), and there is no redirection or retry. Then, this task can be interrupted by the program's exit and come to the callback early, and the state of the task in the callback is WFT_STATE_ABORTED. For example, the following program is always safe: ~~~cpp void callback(WFHttpTask *task) { // most probably print 2，WFT_STATE_ABORTED。 printf("state = %d\n", task->get_state()); } int main() { WFHttpTask *task = WFTaskFactory::create_http_task("https://127.0.0.1/", 0, 0, callback); task->start(); // end the main process directly return 1; } ~~~ If the dns cache hits, it is safe. Because there is no need to initiate a dns asynchronous task internally. E.g: ~~~cpp WFFacilities::WaitGroup wg(1) void callback_normal(WFHttpTask *task) { wg.done(); } void callback_abort(WFHttpTask *task) { // most probably print 2，WFT_STATE_ABORTED。 printf("state = %d\n", task->get_state()); } int main() { WFHttpTask *task = WFTaskFactory::create_http_task("https://www.sogou.com/", 3, 2, callback_normal); task->start(); // wait for the end of the first task wg.wait(); // Access wwww.sogou.com again. Hit the dns cache definitely. WFHttpTask *task = WFTaskFactory::create_http_task("https://www.sogou.com/", 0, 0, callback_abort); task->start(); // end the main process directly return 1; } ~~~ Therefore, for a network task, as long as it can be determined to be an atomic task, it can be interrupted by the end of the program. This principle can be extended to any type of task. For example, the timer task is an atomic task, and the following program is also safe: ~~~cpp void callback(WFTimerTask *task) { // definitely print 2，WFT_STATE_ABORTED。 printf("state = %d\n", task->get_state()); } int main() { WFTimerTask *task = WFTaskFactory::create_timer_task(1000000, callback); task->start(); // end the main process directly return 1; } ~~~ In the documentation (About Timer)(https://github.com/sogou/workflow/blob/master/docs/en/about-timer.md), we will describe them in detail. In addition, you can also end the program before the callback of single-threaded computing tasks and file IO tasks. Among them, the computing task that is already running, the program will wait for the task to end, and finally callback in the SUCCESS state. If it has not begun running, it will canceled and you will get an ABORTED state in callback. As long as the file IO task has been started, it will always wait for the IO to complete. Therefore, it is always safe to exit the program directly. # About memory leakage of OpenSSL 1.1 in exiting We found that some OpenSSL 1.1 versions have the problem of incomplete memory release in exiting. The memory leak can be seen by Valgrind memcheck tool. This problem only happens when you use SSL, such as crawling HTTPS web pages, and usually you can ignore this leak. If it must be solved, you can use the following method: ~~~cpp #include int main() { #if OPENSSL_VERSION_NUMBER >= 0x10100000L OPENSSL_init_ssl(0, NULL); #endif ... } ~~~ In other words, before using our library, you should initialize OpenSSL. You can also configure OpenSSL parameters at the same time if necessary. Please note that this function is only available in OpenSSL version 1.1 or above, so you need to check the openSSL version before calling it. This memory leak is related to the memory release mechanism of OpenSSL 1.1. The solution provided by us can solve this problem (but we still recommend you to ignore it). --- ### En About Go Task (`docs/en/about-go-task.md`) # About go task We provide a simpler way to use computing task, which is inspired by the golang, and we name it 'go task'. When using go task, no input nor output type has to be defined. All data are passed through function's arguments. # Creating a go task ~~~cpp class WFTaskFactory { ... public: template static WFGoTask *create_go_task(const std::string& queue_name, FUNC&& func, ARGS&&... args); }; ~~~ # Example We want to run an 'add' function asychronously: void add(int a, int b, int& res); Still, we want the result printed after the 'add' function is finished. We may create a go task: ~~~cpp #include #include #include "workflow/WFTaskFactory.h" #include "workflow/WFFacilities.h" void add(int a, int b, int& res) { res = a + b; } int main(void) { WFFacilities::WaitGroup wait_group(1); int a = 1; int b = 1; int res; WFGoTask *task = WFTaskFactory::create_go_task("test", add, a, b, std::ref(res)); task->set_callback([&](WFGoTask *task) { printf("%d + %d = %d\n", a, b, res); wait_group.done(); }); task->start(); wait_group.wait(); return 0; } ~~~ The above example runs an add function asynchronously, prints the result and exits normally. The creating and running of go task have little difference from other kinds of tasks, and the user_data field is also available. Note that when creating a go task, we donot pass a callback function. But you may set_callback later like other kinds of tasks. If an argument of the go task's function is a reference, you should use `std::ref` when passing it to the task, otherwise it will be passed as a value. # Go task with running time limit You may create a go task with running time limit by calling WFTaskFactory::create_timedgo_task(): ~~~cpp class WFTaskFactory { /* Create 'Go' task with running time limit in seconds plus nanoseconds. * If time exceeded, state WFT_STATE_SYS_ERROR and error ETIMEDOUT will be got in callback. */ template static WFGoTask *create_timedgo_task(time_t seconds, long nanoseconds, const std::string& queue_name, FUNC&& func, ARGS&&... args); }; ~~~ Compared with creating a normal go task, the ``create_timedgo_task`` function needs to pass two more parameters, seconds and nanoseconds. If the running time of ``func`` reaches the seconds+nanosconds time limit, the task callback directly, and the state is WFT_STATE_SYS_ERROR and the error is ETIMEDOUT. Note that the framework cannot interrupt the user's ongoing task. ``func`` will still continue to execute to the end, but will not callback again. In addition, the value range of nanoseconds is [0,1 billion). # Use the whole library as a thread pool You may use go task only. In this way the workflow library becomes a thread pool，and the default thread number is equal to the cpu number of the host. But this thread pool has some special features. Every thread task is associated with a queue name that will indicate scheduling, and you may set up the dependency of all tasks too. --- ### En About Module (`docs/en/about-module.md`) # About Module Task Our **series** has tasks as elements. But in many cases, users need module-level encapsulation, such as several tasks to complete a specific function. With the original method, you have to let the callback of the last task connect to the next task, or fill in the response of the server task. Therefore, we introduced WFModuleTask, which is convenient for users to encapsulate modules and reduce the coupling of tasks between different functional modules. # Create a Module Task We define a **module** as a kind of task, WFModuleTask. Inside the module includes a sub_series for running tasks within the module. Any task doesn't need to care if it runs inside a module. Because the sub_series inside the module is no different from the normal series. In [WFTaskFactory.h](/src/factory/WFTaskFactory.h), the creation interface of module task: ~~~cpp using module_callback_t = std::function; class WFTaskFactory { static WFModuleTask *create_module_task(SubTask *first, module_callback_t callback); }; ~~~ The first create_module_task() is the first task of the module. Similar to creating a series. The module task’s callback request a **const** pointer argument in order to prevent user pushing more tasks to module in callback. # WFModuleTask Interfaces Because we define modules as this kind of task, we can use modules like any other task. But modules do not have **state** and **error** fields. In [WFTask.h](/src/factory/WFTask.h), we define the class of WFModuleTask: ~~~cpp class ModuleTask : public ParallelTask, protected SeriesWork { public: void start() { .. } void dismiss() { ... } public: SeriesWork *sub_series() { return this; } const SeriesWork *sub_series() const { return this; } public: void *user_data; }; ~~~ The **sub_series** interface returns the series of tasks running in the module. A module is essentially a sub-flow. sub_series is also an ordinary series, and users can call its set_context(), get_context(), push_back() and other functions. But we don't recommend setting a callback for sub_series, use module task’s callback instead. # Example In the processing logic of an http server, we design all processing logic as a module. ~~~cpp struct ModuleCtx { std::string body; }; void http_callback(WFHttpTask *http_task) { SeriesWork *series = series_of(http_task); // This series is module’s sub_series struct ModuleCtx *ctx = (struct ModuleCtx *)series->get_context(); const void *body; size_t size; If (http_task->get_resp()->get_parsed_body(&body, &size)) { ctx->body.assign(body, size); } ParallelWork *pwork = Workflow::create_parallel_work(…)；// Do some other things series->push_back(pwork); } void process(WFHttpTask *server_task) { WFHttpTask *http_task = WFTaskFactory::create_http_task(…, http_callback); WFModuleTask *module = WFTaskFactory::create_module_task(http_task, [server_task](const WFModuleTask *mod) { struct ModuleCxt *ctx = (ModuleCtx *)mod->sub_series()->get_context(); server_task->get_resp()->append_output_body(ctx->body); delete ctx; }); module->sub_series()->set_context(new ModuleCtx); series_of(server_task)->push_back(module); } ~~~ Through this method, the tasks in the module only need to operate the series context, and finally the **resp** is filled in by the callback of the module. Task coupling is greatly reduced. --- ### En About Resource Pool (`docs/en/about-resource-pool.md`) # Conditional task and resource pool When we use workflow to write asynchronous programs, we often encounter such scenarios: * A task needs to obtain a resource from a certain pool before running, and put it back to the pool after it finishs. * We may need to limit the max concurrency of accessing one or more communication targets. But don't want to occupy a thread when waiting. * We have many tasks that arrive randomly, in different series. But these tasks must be run serially. All these needs can be solved with the resource pool module. Our [WFDnsResolver](https://github.com/sogou/workflow/blob/master/src/nameservice/WFDnsResolver.cc) uses this method to control the concurrency of querying the dns server. # Interfaces of resource pool In [WFResourcePool.h](https://github.com/sogou/workflow/blob/master/src/factory/WFResourcePool.h) we define the interfaces of resource pool: ~~~cpp class WFResourcePool { public: WFConditional *get(SubTask *task, void **resbuf); WFConditional *get(SubTask *task); void post(void *res); ... protected: virtual void *pop() { return this->data.res[this->data.index++]; } virtual void push(void *res) { this->data.res[--this->data.index] = res; } ... public: WFResourcePool(void *const *res, size_t n); WFResourcePool(size_t n); ... }; ~~~ ### Constructors The first constructor accept a resource array, with the lenght n. Each element of the array is a **void \*** representing a resource. The whole array will be copied by the constructor. If all the initial resources are **nullptr**, you may use the second constructor which has only one argument n, representing the number of resources. You may take a look of the implementation codes: ~~~cpp void WFResourcePool::create(size_t n) { this->data.res = new void *[n]; this->data.value = n; ... } WFResourcePool::WFResourcePool(void *const *res, size_t n) { this->create(n); memcpy(this->data.res, res, n * sizeof (void *)); } WFResourcePool::WFResourcePool(size_t n) { this->create(n); memset(this->data.res, 0, n * sizeof (void *)); } ~~~ ### Application interfaces Users use **get()** method of resource pool to wrap a task. **get()** returns a conditional, which is also a task. Conditional will runs the task it wrap when it obtain a resource from the pool. **get()** may accept a second argument **void \*\* resbuf**, which is the buffer that will store the resource abtained. After the **get()** operation, users can use the returned conditional to substain the original task. It can be started or put to any series just like an ordinary task. After the user task is finished, **post()** need to be called to return a resource to the pool. Typically, **post()** is called in user task's callback. ### Derivation The using of resource pool is FILO. It means the last released resource will be the next one to be obtained. You may subclass WFResourcePool to implement a FIFO pool. ### Example We have a URL list to be crawled. But we limit the max concurreny of crawling task to be **max_p**. We may use ParallelWork to implement this function of course. But with resource pool, everything is much simpler: ~~~cpp int fetch_with_max(std::vector& url_list, size_t max_p) { WFResourcePool pool(max_p); for (std::string& url : url_list) { WFHttpTask *task = WFTaskFactory::create_http_task(url, [&pool](WFHttpTask *task) { pool.post(nullptr); }); WFConditional *cond = pool.get(task); cond->start(); } // wait_here... } ~~~ --- ### En About Service Governance (`docs/en/about-service-governance.md`) # About service governance We have a complete mechanism to manage the services we depend on. This mechanism includes the following functions: * User level DNS. * Selection of service addresses. * Including a variety of selection mechanisms, such as random weight, consistent hash, manual selection methods, etc. * Service circuit breaker and recovery. * Load balancing. * Configuring independent parameters for a single service. * Main/backup relations for a service, etc. All these functions depend on our upstream subsystem. By making good use of this system, we can easily implement more complex service mesh functions. # upstream name upstream name is equivalent to the domain name inside the program. However, compared with the general domain name, upstream has more functions, including: * Generally, a domain name can only point to a set of IP addresses; an upstream name can point to a set of IP addresses or domain names. * The objects (domain names or IPs) pointed by the upstream may include port information. * upstream has powerful functions for managing and selecting targets, and each target can contain a large number of attributes. * upstream update is real-time and completely thread-safe, while the DNS of domain names cannot be updated in real time. In practice, if you don't need to access the external network, the domain names and DNS can be completely replaced by upstream. # Creating and deleting upstream [UpstreamMananer.h](/src/manager/UpstreamManager.h) contains several interfaces for creating upstream: ~~~cpp using upstream_route_t = std::function; class UpstreamManager { public: static int upstream_create_consistent_hash(const std::string& name, upstream_route_t consitent_hash); static int upstream_create_weighted_random(const std::string& name, bool try_another); static int upstream_create_manual(const std::string& name, upstream_route_t select, bool try_another, upstream_route_t consitent_hash); static int upstream_delete(const std::string& name); ... }; ~~~ The three functions create three types of upstream: consistent hash, weighted random and manual selection. The parameter **name** means upstream name, which is used in the same way as a domain name after creation. **consistent\_hash** and **select** parameters are both **std::function** of **upstream\_route\_t**, which are used to specify the routing method. And try\_another indicates whether to continue trying to find an available target if the selected target is unavailable (blown). consistent\_hash mode does not have this attribute. The upstream\_route\_t parameter receives three parameters: path, query and fragment in a URL. For example, if the URL is http://abc.com/home/index.html?a=1#bottom, the three parameters are "/home/index.html", "a=1” and "bottom” respectively. Based on these three parts, the system can select the target server or perform consistent hashing. Please note that you call pass nullptr to all consistent\_hash parameters in the above interfaces, and the framework will use the default consistent hash algorithm. # Example 1: weight allocation We want to allocate 50% of the requests to www.sogou.com to 127.0.0.1:8000 and 127.0.0.1:8080, and make their load be 1:4. We don't need to care about the number of IP addresses behind the domain name www.sogou.com. In short, the actual domain name will receive 50% of the requests. ~~~cpp #include "workflow/UpstreamManager.h" #include "workflow/WFTaskFactory.h" int main() { UpstreamManager::upstream_create_weighted_random("www.sogou.com", false); struct AddressParams params = ADDRESS_PARAMS_DEFAULT; params.weight = 5; UpstreamManager::upstream_add_server("www.sogou.com", "www.sogou.com", ¶ms); params.weight = 1; UpstreamManager::upstream_add_server("www.sogou.com", "127.0.0.1:8000", ¶ms); params.weight = 4; UpstreamManager::upstream_add_server("www.sogou.com", "127.0.0.1:8080", ¶ms); WFHttpTask *task = WFTaskFactory::create_http_task("http://www.sogou.com/index.html", ...); ... } ~~~ Please note that these functions can be called in any scenario. They are completely thread-safe and takes effect instantly. In addition, because all our protocols, including user-defined protocols, have URLs, the upstream function can be applied to all protocols. # Example 2: manual selection In the same example as above, we want to allocate 127.0.0.1:8000 if the query in the request URLs is "123", port 8080 if the query is "abc", and normal domain names for other requests. ~~~cpp #include "workflow/UpstreamManager.h" #include "workflow/WFTaskFactory.h" int my_select(const char *path, const char *query, const char *fragment) { if (strcmp(query, "123") == 0) return 1; else if (strcmp(query, "abc") == 0) return 2; else return 0; } int main() { UpstreamManager::upstream_create_manual("www.sogou.com", my_select, false, nullptr); UpstreamManager::upstream_add_server("www.sogou.com", "www.sogou.com"); UpstreamManager::upstream_add_server("www.sogou.com", "127.0.0.1:8000"); UpstreamManager::upstream_add_server("www.sogou.com", "127.0.0.1:8080"); /* This URL will route to 127.0.0.1:8080 */ WFHttpTask *task = WFTaskFactory::create_http_task("http://www.sogou.com/index.html?abc", ...); ... } ~~~ Because Redis and MySQL protocols are provided natively, it is very convenient to realize the read-write separation function of the database with this method (Note: non-transactional operation). In the above two examples, the upstream name is www.sogou.com, which is also a domain name. Of course, you can use a simpler string sogou as upstream name. Thus: ~~~cpp WFHttpTask *task = WFTaskFactory::create_http_task("http://sogou/home/1.html?abc", ...); ~~~ In a word, if the host part of the URL is a created upstream, it will be used as an upstream. # Example 3: consistent hash In this scenario, we will randomly select one machine from 10 Redis instances and communicate with it. But we must ensure that the same URL always accesses the same specific target. The method is very simple: ~~~cpp int main() { UpstreamManager::upstream_create_consistent_hash("redis.name", nullptr); UpstreamManager::upstream_add_server("redis.name", "10.135.35.53"); UpstreamManager::upstream_add_server("redis.name", "10.135.35.54"); UpstreamManager::upstream_add_server("redis.name", "10.135.35.55"); ... UpstreamManager::upstream_add_server("redis.name", "10.135.35.62"); auto *task = WFTaskFactory::create_redis_task("redis://:mypassword@redis.name/2?a=hello#111", ...); ... } ~~~ Our Redis task does not recognize the query part, so you can fill it out at will. 2 in the path indicates the Redis database ID. At this time, the consistent\_hash function will get three parameters: "/2", "a=hello" and "111". Because we use nullptr, the default consistent hash will be called. As we does not specify the port number for the server in upstream, it will use the port in the URL. The default port of Redis is 6379. There is no try\_another option for consitent\_hash. If the target is blown, another one will be automatically selected. The same URL will always get the same server (cache friendly). # Parameters of upstream server In Example 1, we set the weight of a server through params. But the server parameters is far more than just a weight. Its struct is defined as follows: ~~~cpp // In EndpointParams.h struct EndpointParams { size_t max_connections; int connect_timeout; int response_timeout; int ssl_connect_timeout; bool use_tls_sni; }; // In ServiceGovernance.h struct AddressParams { struct EndpointParams endpoint_params; ///< Connection config unsigned int dns_ttl_default; ///< in seconds, DNS TTL when network request success unsigned int dns_ttl_min; ///< in seconds, DNS TTL when network request fail /** * - The max_fails directive sets the number of consecutive unsuccessful attempts to communicate with the server. * - After 30s following the server failure, upstream probe the server with some alive client’s requests. * - If the probes have been successful, the server is marked as an alive one. * - If max_fails is set to 1, it means server would out of upstream selection in 30 seconds when failed only once */ unsigned int max_fails; ///< [1, INT32_MAX] max_fails = 0 means max_fails = 1 unsigned short weight; ///< [1, 65535] weight = 0 means weight = 1. only for main int server_type; ///< 0 for main and 1 for backup int group_id; ///< -1 means no group. Backup without group will backup for any main }; ~~~ Most of the parameters are self-explanatory. Among these parameters, endpoint\_params, dns and other parameters will override the global configuration. For example, if the global maximum number of connections to each target IP is 200, but you want to set a maximum of 1000 connections for 10.135.35.53, please follow the instructions below: ~~~cpp UpstreamManager::upstream_create_weighted_random("10.135.35.53", false); struct AddressParams params = ADDRESS_PARAMS_DEFAULT; params.endpoint_params.max_connections = 1000; UpstreamManager::upstream_add_server("10.135.35.53", "10.135.35.53", ¶ms); ~~~ max\_fails parameter indicates the maximum number of failure. If the selected target continuously fails, and the number of failure reaches max\_failures, it will enter the fusing state. If the try\_another attribute of upstream is false, the task will fail. In the callback of the task, get\_state()=WFT\_STATE\_TASK\_ERROR，get\_error()=WFT\_ERR\_UPSTREAM\_UNAVAILABLE. If try\_another is true and all server are blown, you will get the same error. The fusing time is 30 seconds. Server\_type and group\_id are used for main/backup features. All upstream must have a server whose type is 0, representing main, otherwise the upstream is unavailable. Backup servers (server_type 1) will be used when the main servers of the same group\_id is blown. For more information on the features of upstream, please see [about-upstream.md](/docs/en/about-upstream.md). --- ### En About Timeout (`docs/en/about-timeout.md`) # About timeout In order to make all communication tasks run as accurately as expected by users, the framework provides a large number of timeout configuration functions and ensure the accuracy of these timeouts. Some of these timeout configurations are global, such as connection timeout, but you may configure your own connection timeout for a perticular domain name through the upstream. Some timeouts are task-level, such as sending a message completely, because users needs to dynamically configure this value according to the message size. Of course, a server may have its own overall timeout configuration. In a word, timeout is a complicated matter, and the framework will do it accurately. All timeouts are in **poll** style. It is an **int** in milliseconds and -1 means infinite. In addition, as said in the project introduction, you can ignore all the configurations, and adjust them when you meet the actual requirements. ### Timeout configuration for basic communication [EndpointParams.h](/src/manager/EndpointParams.h) contains the following items: ~~~cpp struct EndpointParams { size_t max_connections; int connect_timeout; int response_timeout; int ssl_connect_timeout; }; static constexpr struct EndpointParams ENDPOINT_PARAMS_DEFAULT = { .max_connections = 200, .connect_timeout = 10 * 1000, .response_timeout = 10 * 1000, .ssl_connect_timeout = 10 * 1000, }; ~~~ in which there are three DNS-related configuration items. Please ignore them right now. Items related to timeout: * connect\_timeout: timeout for establishing a connection with the target. The default value is 10 seconds. * response\_timeout: timeout for waiting for the target response; the default value is 10 seconds. It is the timeout for sending a block of data to the target or reading a block of data from the target. * ssl\_connect\_timeout: timeout for completing SSL handshakes with the target. The default value is 10 seconds. This struct is the most basic configuration for the communication connection, and almost all subsequent communication configurations contain this struct. ### Global timeout configuration You can see the global settings in [WFGlobal.h](/src/manager/WFGlobal.h). ~~~cpp struct WFGlobalSettings { EndpointParams endpoint_params; unsigned int dns_ttl_default; unsigned int dns_ttl_min; int dns_threads; int poller_threads; int handler_threads; int compute_threads; }; static constexpr struct WFGlobalSettings GLOBAL_SETTINGS_DEFAULT = { .endpoint_params = ENDPOINT_PARAMS_DEFAULT, .dns_ttl_default = 12 * 3600, /* in seconds */ .dns_ttl_min = 180, /* reacquire when communication error */ .dns_threads = 8, .poller_threads = 2, .handler_threads = 20, .compute_threads = -1 }; //compute_threads<=0 means auto-set by system cpu number ~~~ in which there is one timeout related configuration item: EndpointParams endpoint\_params You can perform operations like the following to change the global configuration before calling any of our factory functions: ~~~cpp int main() { struct WFGlobalSettings settings = GLOBAL_SETTINGS_DEFAULT; settings.endpoint_params.connect_timeout = 2 * 1000; settings.endpoint_params.response_timeout = -1; WORKFLOW_library_init(&settings); } ~~~ The above example changes the connection timeout to 2 seconds, and the server response timeout is infinite. In this configuration, the timeout for receiving complete messages must be configured in each task, otherwise it may fall into infinite waiting. The global configuration can be overridden by the configuration for an individual address in the upstream feature. For example, you can specify a connection timeout for a specific domain name. In Upstream, each AddressParams also has the EndpointParams endpoint\_params item, and you can configure it in the same way as you configure the Global item. For the detailed structures, please see [upstream documents.](/docs/en/tutorial-10-upstream.md#Address) ### Configuring server timeout The [http\_proxy](/docs/en/tutorial-05-http_proxy.md) example demonstrates the server startup configuration. In which the timeout-related configuration items include: * peer\_response\_timeout: its definition is the same as the global peer\_response\_timeout, which indicates the response timeout of the remote client, and the default value is 10 seconds. * receive\_timeout: timeout for receiving a complete request. The default value is -1. * keep\_alive\_timeout: timeout for keeping a connection. The default value is 1 minute. For a Redis server, the default value is 5 minutes. * ssl\_accept\_timeout: timeout for completing SSL handshakes. The default value is 10 seconds. Under this default configuration, the client can send one byte every 9 seconds, so that the server can always receive it and no timeout occurs. Therefore, if the service is used for public network, you need to configure receive\_timeout. ### Configuring task-level timeout Task-level timeout configuration is accomplished through calling several interfaces in a network task: ~~~cpp template class WFNetworkTask : public CommRequest { ... public: /* All in milliseconds. timeout == -1 for unlimited. */ void set_send_timeout(int timeout) { this->send_timeo = timeout; } void set_receive_timeout(int timeout) { this->receive_timeo = timeout; } void set_keep_alive(int timeout) { this->keep_alive_timeo = timeout; } void set_watch_timeout(int timeout) { this->watch_timeo = timeout; } ... } ~~~ In the above code, **set\_send\_timeout()** sets the timeout for sending a complete message, and the default value is -1. **set\_receive\_timeout()** is only valid for the client task, and it indicates the timeout for receiving a complete server reply. The default value is -1. * The receive\_timeout of a server task is in the server startup configuration. All server tasks handled by users have successfully received complete requests. **set\_keep\_alive()** interface sets the timeout for keeping a connection. Generally, the framework can handle the connection maintenance well, and you do not need to call it. When an HTTP protocol is used, if a client or a server wants to use short connection, you can add an HTTP header to support it. Please do not modify it with this interface if you have other options. If a Redis client wants to close the connection after a request, you need to use this interface. Obviously, **set\_keep\_alive()** is invalid in the callback (the connection has been reused). **set\_watch\_timeout()** is specific for client task only. It indicate the maximum time of waiting the first response package. This may prevent the client task from being timed out by the limit of **response\_timeout** and **receive\_timeout**. The framework will caculate **receive\_timeoout** after receiving the first package if **watch\_timeout** is set. ### Timeout for synchronous task waiting There is a very special timeout configuration, and it is the only global synchronous waiting timeout. It is not recommended, but you can get good results with it in some application scenarios. In the current framework, the target server has a connection limit (you can set it in both global and upstream configurations). If the number of connections have reached the upper limit, the client task fails and returns an error by default. In the callback, **task->get\_state ()** gets WFT\_STATE\_SYS\_ERROR, and **task->get\_error()** gets EAGAIN. If the task is configured with retry, a retry will be automatically initiated. Here, it is allowed to configure a synchronous waiting timeout through the **task->set\_wait\_timeout()** interface. If a connection is released during this time period, the task can occupy this connection. If you sets wait\_timeout and does not get the connection before the timeout, the callback will get WFT\_STATE\_SYS\_ERROR status and ETIMEDOUT error. ~~~cpp class CommRequest : public SubTask, public CommSession { public: ... void set_wait_timeout(int wait_timeout) { this->wait_timeout = wait_timeout; } } ~~~ ### Viewing the reasons for timeout Communication tasks contain a **get\_timeout\_reason()** interface, which is used to return the timeout reason, but the reason is not very detailed. It includes the following return values: * TOR\_NOT\_TIMEOUT: not a timeout. * TOR\_WAIT\_TIMEOUT: timed out for synchronous waiting * TOR\_CONNECT\_TIMEOUT: connection timed out. The connections on TCP, SCTP, SSL and other protocols all use this timeout. * TOR\_TRANSMIT\_TIMEOUT: timed out for all transmissions. It is impossible to further distinguish whether it is in the sending stage or in the receiving stage. It may be refined later. * For a server task, if the timeout reason is TRANSMIT\_TIMEOUT, it must be in the stage of sending replies. ### Implementation of timeout functions Within the framework, there are more types of timeouts than those we show here. Except for wait\_timeout, all of them depend on the timer\_fd on Linux or kqueue timer on BSD system, one for each poller thread. By default, the number of poller threads is 4, which can meet the requirements of most applications. The current timeout algorithm uses the data structure of linked list and red-black tree. Its time complexity is between O(1) and O(logn), where n is the fd number of the a poller thread. Currently timeout processing is not the bottleneck, because the time complexity of related calls of epoll in Linux kernel is also O(logn). If the time complexity of all timeouts in our framework reaches O(1), there is no much difference. --- ### En About Timer (`docs/en/about-timer.md`) # About timer Timers are used to specify a certain waiting time without occupying a thread. The expiration of a timer is notified also by a callback. # Creating a timer Timer interfaces in WFTaskFactory： ~~~cpp using timer_callback_t = std::function; class WFTaskFactory { ... public: static WFTimerTask *create_timer_task(time_t seconds, long nanoseconds, timer_callback_t callback); static WFTimerTask *create_timer_task(const std::string& timer_name, time_t seconds, long nanoseconds, timer_callback_t callback); static int cancel_by_name(const std::string& timer_name) { cancel_by_name(const std::string& timer_name, (size_t)-1); } static int cancel_by_name(const std::string& timer_name, size_t max); }; ~~~ We specify the timing time of a timer through the seconds and nanoseconds parameters. Among them, the value range of nanoseconds is [0,1000000000). When creating a timer, a timer_name can be specified. And we may interrupt a timer by calling **cancel_by_name** with this name later. As a standard workflow task, there is also a user\_data field in the timer task that can be used to transfer some user data. Its starting method is the same as other tasks, and the procedure for adding it into the workflow is also the same. # Canceling a timer A named timer can be interrupted throught WFTaskFacotry::cancel_by_name interface, which will cancel all timers under the name by default. So we provide another cancel interface with the second argument **max** for user to cancel at most **max** timers. Each interface returns the number of timers that was actually canceled. And of course, if no timer under the name, nothing performed and returns 0. You can cancel a timer right after it's created, for example: ~~~cpp #include #include "workflow/WFTaskFactory.h" int main() { WFTimerTask *timer = WFTaskFactory::create_timer_task("test", 10000, 0, [](WFTimerTask *){ printf("timer callback, state = %d, error = %d.\n", task->get_state(), task->get_error()); }); WFTaskFactory::cancel_by_name("test"); timer->start(); getchar(); return 0; } ~~~ This program prints 'timer callback, state = 1, error = 125."，immediately because the timer has be canceled before started, and it will run to callback soon after it's started. And the state code would be WFT_STATE_SYS_ERROR and the error code would be ECANCELED. By the way, create named timer when and only when you may need to cancel it, because it costs more. In other scenarios just use anonymous timer. # Interrupting timer by program exit In [About exit](/docs/en/about-exit.md), you learn that the condition that a main thread can safely end (calls **exit()** or return in the main function) is that all tasks have been run to the callback and no new task is started. Then, there may be a problem, if you wait for the timer to expire, it will take a long time for the program to exit. But in practice, exiting the program can interrupt the timer safely and make it return to the callback. If the timer is interrupted by exiting the program, **get\_state()** will return a WFT\_STATE\_ABORTED state. Of course, if the timer is interrupted by exiting the program, no new tasks can be started. The following program demonstrates crawling one HTTP page at every one second. When all URLs are crawled, the program exits directly without waiting for the timer to return to the callback, and there will be no delay in exiting. ~~~cpp bool program_terminate = false; void timer_callback(WFTimerTask *timer) { mutex.lock(); if (!program_terminate) { WFHttpTask *task; if (urls_to_fetch > 0) { task = WFTaskFactory::create_http_task(...); series_of(timer)->push_back(task); } series_of(timer)->push_back(WFTaskFactory::create_timer_task(1, 0, timer_callback)); } mutex.unlock(); } ... int main() { .... /* all urls done */ mutex.lock(); program_terminate = true; mutex.unlock(); return 0; } ~~~ In the above program, the timer\_callback must check the program\_terminate condition in the lock, otherwise a new task may be started when the program has terminated. --- ### En About Tlv Message (`docs/en/about-tlv-message.md`) # About TLV (Type-Length-Value) format message A TLV message is a message consisting of type, length, and value. Because its format is simple and universal, and it is convenient for nesting and expansion, it is especially suitable for defining communication messages. To facilitate users to implement custom protocols, we have built-in support for TLV messages. # TLV message structure The general TLV structure does not specify the bytes of the Type or Length field. In our protocol, they occupy 4 bytes each (network order). In other words, our message has an 8-byte message header and a Value content of no more than 32GB. We do not specify the meaning of the Type and Value fields. # TLVMessage class Because the definition of TLV format is simple. The interfaces of this TLVMessage are very simple too. ~~~cpp namespace protocol { class TLVMessage : public ProtocolMessage { public: int get_type() const { return this->type; } void set_type(int type) { this->type = type; } std::string *get_value() { return &this->value; } void set_value(std::string value) { this->value = std::move(value); } protected: int type; std::string value; ... }; using TLVRequest = TLVMessage; using TLVResposne = TLVMessage; } ~~~ If users directly use TLV messages for data transmission, they only need to use the above interfaces. Set and get Type and Value respectively. Value is directly returned as ``std::string``, which is convenient for users to move data directly through ``std::move`` when necessary. # An echo server/client example based on TLV message The following code directly starts a server based on TLV messages, and generates a client task through the command line for interaction. ~~~cpp #include #include #include #include "workflow/WFGlobal.h" #include "workflow/WFFacilities.h" #include "workflow/TLVMessage.h" #include "workflow/WFTaskFactory.h" #include "workflow/WFServer.h" using namespace protocol; using WFTLVServer = WFServer; using WFTLVTask = WFNetworkTask; using tlv_callback_t = std::function; WFTLVTask *create_tlv_task(const char *host, unsigned short port, tlv_callback_t callback) { auto *task = WFNetworkTaskFactory::create_client_task( TT_TCP, host, port, 0, std::move(callback)); task->set_keep_alive(60 * 1000); return task; } int main() { WFTLVServer server([](WFTLVTask *task) { *task->get_resp() = std::move(*task->get_req()); }); if (server.start(8888) != 0) { perror("server.start"); exit(1); } auto&& create = [](WFRepeaterTask *)->SubTask * { std::string string; printf("Input string (Ctrl-D to exit): "); std::cin >> string; if (string.empty()) return NULL; auto *task = create_tlv_task("127.0.0.1", 8888, [](WFTLVTask *task) { if (task->get_state() == WFT_STATE_SUCCESS) printf("Server Response: %s\n", task->get_resp()->get_value()->c_str()); else { const char *str = WFGlobal::get_error_string(task->get_state(), task->get_error()); fprintf(stderr, "Error: %s\n", str); } }); task->get_req()->set_value(std::move(string)); return task; }; WFFacilities::WaitGroup wait_group(1); WFRepeaterTask *repeater = WFTaskFactory::create_repeater_task(std::move(create), nullptr); Workflow::start_series_work(repeater, [&wait_group](const SeriesWork *) { wait_group.done(); }); wait_group.wait(); server.stop(); return 0; } ~~~ # To extend TLVMessage In the echo server example above, we directly use the original TLVMessage. However, it is suggested that in specific applications, users can derive TLVMessage. In the derived class, provide a richer interface to set and extract message content, avoid direct manipulation of the original Value field, and form its own secondary protocol. For example, if we implement a JSON protocol, we can: ~~~cpp #include "workflow/json-parser.h" // built-in JSON parser class JsonMessage : public TLVMessage { public: void set_json_value(const json_value_t *val) { this->type = JSON_TYPE; this->json_to_string(val, &this->value); // you have to implement this function } json_value_t *get_json_value() const { if (this->type == JSON_TYPE) return json_parser_parse(this->value.c_str()); // json-parser's interface else return NULL; } }; using JsonRequest = JsonMessage; using JsonResponse = JsonMessage; using JsonServer = WFServer; ~~~ This example is just to illustrate the importance of derivation. In actual applications, derived classes may be far more complicated than this. --- ### En About Upstream (`docs/en/about-upstream.md`) # About Upstream In nginx, Upstream represents the load balancing configuration of the reverse proxy. Here, we expand the meaning of Upstream so that it has the following characteristics: 1. Each Upstream is an independent reverse proxy 2. Accessing an Upstream is equivalent to using an appropriate strategy to select one in a group of services/targets/upstream and downstream for access 3. Upstream has load balancing, error handling, circuit breaker and other service governance capabilities 4. For multiple retries of the same request, Upstream can avoid addresses that already tried 5. Different connection parameters can be configured for different addresses through Upstream 6. Dynamically adding/removing target will take effect in real time, which is convenient for any service discovery system ### Advantages of Upstream over domain name DNS resolution Both Upstream and domain name DNS resolution can configure a group of ip to a host, but 1. DNS domain name resolution doesn’t address port number. The service DNS domain names with the same IP and different ports cannot be configured together; but it is possible for Upstream. 2. The set of addresses corresponding to DNS domain name resolution must be ip; while the set of addresses corresponding to Upstream can be ip, domain name or unix-domain-socket 3. Normally, DNS domain name resolution will be cached by operating system or DNS server on the network, and the update time is limited by ttl; while Upstream can be updated in real time and take effect in real time 4. The consumption of DNS domain name is much greater than that of Upstream resolution and selection ### Upstream of Workflow This is a local reverse proxy module, and the proxy configuration is effective for both server and client. Support dynamic configuration and available for any service discovery system. Currently, [workflow-k8s](https://github.com/sogou/workflow-k8s) can be used to acquire Pods information from the API server of Kubernetes. Upstream name does not include port, but upstream request supports specified port. (However, for non-built-in protocols, Upstream name temporarily needs to be added with the port to ensure parsing during construction). Each Upstream is configured with its own independent name UpstreamName, and a set of Addresses is added and set. These Addresses can be: 1. ip4 2. ip6 3. Domain name 4. unix-domain-socket ### Why to replace nginx's Upstream #### Upstream working mode of nginx 1. Supports http/https protocol only 2. Needs to build a nginx service, start the start process occupies socket and other resources 3. The request is sent to nginx first, and nginx forwards the request to remote end, which will increase one more network communication overhead #### Local Upstream working method of workflow 1. Protocol irrelevant, you can even access mysql, redis, mongodb, etc. through upstream 2. You can directly simulate the function of reverse proxy in the process, no need to start other processes or ports 3. The selection process is basic calculation and table lookup, no additional network communication overhead # Use Upstream ### Common interfaces ~~~cpp class UpstreamManager { public: static int upstream_create_consistent_hash(const std::string& name, upstream_route_t consitent_hash); static int upstream_create_weighted_random(const std::string& name, bool try_another); static int upstream_create_manual(const std::string& name, upstream_route_t select, bool try_another, upstream_route_t consitent_hash); static int upstream_delete(const std::string& name); public: static int upstream_add_server(const std::string& name, const std::string& address); static int upstream_add_server(const std::string& name, const std::string& address, const struct AddressParams *address_params); static int upstream_remove_server(const std::string& name, const std::string& address); ... } ~~~ ### Example 1 Random access in multiple targets Configure a local reverse proxy to evenly send all the local requests for **my_proxy.name** to 6 target servers ~~~cpp UpstreamManager::upstream_create_weighted_random( "my_proxy.name", true); // In case of fusing, retry till the available is found or all fuses are blown UpstreamManager::upstream_add_server("my_proxy.name", "192.168.2.100:8081"); UpstreamManager::upstream_add_server("my_proxy.name", "192.168.2.100:8082"); UpstreamManager::upstream_add_server("my_proxy.name", "192.168.10.10"); UpstreamManager::upstream_add_server("my_proxy.name", "test.sogou.com:8080"); UpstreamManager::upstream_add_server("my_proxy.name", "abc.sogou.com"); UpstreamManager::upstream_add_server("my_proxy.name", "abc.sogou.com"); UpstreamManager::upstream_add_server("my_proxy.name", "/dev/unix_domain_scoket_sample"); auto *http_task = WFTaskFactory::create_http_task("http://my_proxy.name/somepath?a=10", 0, 0, nullptr); http_task->start(); ~~~ Basic principles 1. Select a target randomly 2. If try_another is configured as true, one of all surviving targets will be selected randomly 3. Select in the main servers only, the mains and backups of the group where the selected target is located and the backup without group are regarded as valid optional objects ### Example 2 Random access among multiple targets based on weights Configure a local reverse proxy, send all **weighted.random** requests to the 3 target servers based on the weight distribution of 5/20/1 ~~~cpp UpstreamManager::upstream_create_weighted_random( "weighted.random", false); // If you don’t retry in case of fusing, the request will surely fail AddressParams address_params = ADDRESS_PARAMS_DEFAULT; address_params.weight = 5; //weight is 5 UpstreamManager::upstream_add_server("weighted.random", "192.168.2.100:8081", &address_params); // weight is 5 address_params.weight = 20; // weight is 20 UpstreamManager::upstream_add_server("weighted.random", "192.168.2.100:8082", &address_params); // weight is 20 UpstreamManager::upstream_add_server("weighted.random", "abc.sogou.com"); // weight is 1 auto *http_task = WFTaskFactory::create_http_task("http://weighted.random:9090", 0, 0, nullptr); http_task->start(); ~~~ Basic principles 1. According to the weight distribution, randomly select a target, the greater the weight is, the greater the probability is 2. If try_another is configured as true, one of all surviving targets will be selected randomly as per weights. 3. Select in the main servers only, the main and backup of the group where the selected target is located and the backup without group are regarded as valid optional objects ### Example 3 Access among multiple targets based on the framework's default consistent hash ~~~cpp UpstreamManager::upstream_create_consistent_hash( "abc.local", nullptr); // nullptr represents using the default consistent hash function of the framework UpstreamManager::upstream_add_server("abc.local", "192.168.2.100:8081"); UpstreamManager::upstream_add_server("abc.local", "192.168.2.100:8082"); UpstreamManager::upstream_add_server("abc.local", "192.168.10.10"); UpstreamManager::upstream_add_server("abc.local", "test.sogou.com:8080"); UpstreamManager::upstream_add_server("abc.local", "abc.sogou.com"); auto *http_task = WFTaskFactory::create_http_task("http://abc.local/service/method", 0, 0, nullptr); http_task->start(); ~~~ Basic principles 1. Each main server is regarded as 16 virtual nodes 2. The framework will use std::hash to calculate "the address + virtual index of all nodes + the number of times for this address to add into this Upstream" as the node value of the consistent hash 3. The framework will use std::hash to calculate path + query + fragment as a consistent hash data value 4. Choose the value nearest to the surviving node as the target each time 5. For each main, as long as there is a main in surviving group, or there is a backup in surviving group, or there is a surviving no group backup, it is regarded as surviving 6. If weight on AddressParams is set with upstream_add_server(), each main server is regarded as 16 * weight virtual nodes. This is suitable for weighted consistent hash or shrinking the standard deviation of consistent hash ### Example 4 User-defined consistent hash function ~~~cpp UpstreamManager::upstream_create_consistent_hash( "abc.local", [](const char *path, const char *query, const char *fragment) -> unsigned int { unsigned int hash = 0; while (*path) hash = (hash * 131) + (*path++); while (*query) hash = (hash * 131) + (*query++); while (*fragment) hash = (hash * 131) + (*fragment++); return hash; }); UpstreamManager::upstream_add_server("abc.local", "192.168.2.100:8081"); UpstreamManager::upstream_add_server("abc.local", "192.168.2.100:8082"); UpstreamManager::upstream_add_server("abc.local", "192.168.10.10"); UpstreamManager::upstream_add_server("abc.local", "test.sogou.com:8080"); UpstreamManager::upstream_add_server("abc.local", "abc.sogou.com"); auto *http_task = WFTaskFactory::create_http_task("http://abc.local/sompath?a=1#flag100", 0, 0, nullptr); http_task->start(); ~~~ Basic principles 1. The framework will use a user-defined consistent hash function as the data value 2. The rest is the same as the above principles ### Example 5 User-defined selection strategy ~~~cpp UpstreamManager::upstream_create_manual( "xyz.cdn", [](const char *path, const char *query, const char *fragment) -> unsigned int { return atoi(fragment); }, true, // If a blown target is selected, a second selection will be made nullptr); // nullptr represents using the default consistent hash function of the framework in the second selection UpstreamManager::upstream_add_server("xyz.cdn", "192.168.2.100:8081"); UpstreamManager::upstream_add_server("xyz.cdn", "192.168.2.100:8082"); UpstreamManager::upstream_add_server("xyz.cdn", "192.168.10.10"); UpstreamManager::upstream_add_server("xyz.cdn", "test.sogou.com:8080"); UpstreamManager::upstream_add_server("xyz.cdn", "abc.sogou.com"); auto *http_task = WFTaskFactory::create_http_task("http://xyz.cdn/sompath?key=somename#3", 0, 0, nullptr); http_task->start(); ~~~ Basic principles 1. The framework first determines the selection in the main server list according to the normal selection function provided by the user and then get the modulo 2. For each main server, as long as there is a main server in surviving group, or there is a backup in surviving group, or there is a surviving no group backup, it is regarded as surviving 3. If the selected target no longer survives and try_another is set as true, a second selection will be made using consistent hash function 4. If the second selection is triggered, the consistent hash will ensure that a survival target will be selected, unless all machines are blown ### Example 6 Simple main-backup mode ~~~cpp UpstreamManager::upstream_create_weighted_random( "simple.name", true);//One main, one backup, nothing is different in this item AddressParams address_params = ADDRESS_PARAMS_DEFAULT; address_params.server_type = 0; /* 1 for main server */ UpstreamManager::upstream_add_server("simple.name", "main01.test.ted.bj.sogou", &address_params); // main address_params.server_type = 1; /* 0 for backup server */ UpstreamManager::upstream_add_server("simple.name", "backup01.test.ted.gd.sogou", &address_params); //backup auto *http_task = WFTaskFactory::create_http_task("http://simple.name/request", 0, 0, nullptr); auto *redis_task = WFTaskFactory::create_redis_task("redis://simple.name/2", 0, nullptr); redis_task->get_req()->set_query("MGET", {"key1", "key2", "key3", "key4"}); (*http_task * redis_task).start(); ~~~ Basic principles 1. The main-backup mode does not conflict with any of the modes shown above, and it can take effect at the same time 2. The number of main/backup is independent of each other and there is no limit. All main servers are coequal to each other, and all backup servers are coequal to each others, but main and backup are not coequal to each other. 3. As long as a main server is alive, the request will always use a main server. 4. If all main servers are blown, backup server will take over the request as a substitute target until any main server works well again 5. In every strategy, surviving backup can be used as the basis for the survival of main ### Example 7 Main-backup + consistent hash + grouping ~~~cpp UpstreamManager::upstream_create_consistent_hash( "abc.local", nullptr);//nullptr represents using the default consistent hash function of the framework AddressParams address_params = ADDRESS_PARAMS_DEFAULT; address_params.server_type = 0; address_params.group_id = 1001; UpstreamManager::upstream_add_server("abc.local", "192.168.2.100:8081", &address_params);//main in group 1001 address_params.server_type = 1; address_params.group_id = 1001; UpstreamManager::upstream_add_server("abc.local", "192.168.2.100:8082", &address_params);//backup for group 1001 address_params.server_type = 0; address_params.group_id = 1002; UpstreamManager::upstream_add_server("abc.local", "backup01.test.ted.bj.sogou", &address_params);//main in group 1002 address_params.server_type = 1; address_params.group_id = 1002; UpstreamManager::upstream_add_server("abc.local", "backup01.test.ted.gd.sogou", &address_params);//backup for group 1002 address_params.server_type = 1; address_params.group_id = -1; UpstreamManager::upstream_add_server("abc.local", "test.sogou.com:8080", &address_params);//backup with no group mean backup for all groups and no group UpstreamManager::upstream_add_server("abc.local", "abc.sogou.com");//main, no group auto *http_task = WFTaskFactory::create_http_task("http://abc.local/service/method", 0, 0, nullptr); http_task->start(); ~~~ Basic principles 1. Group number -1 means no group, this kind of target does not belong to any group 2. The main servers without a group are coequal to each other, and they can even be regarded as one group. But they are isolated from the other main servers with a group 3. A backup without a group can serve as a backup for any group target of Global/any target without a group 4. The group number can identify which main and backup are working together 5. The backups of different groups are isolated from each other, and they serve the main servers of their own group only 6. Add the default group number -1 of the target, and the type is main ### Example 8 NVSWRR selection weighting strategy ~~~cpp UpstreamManager::upstream_create_vnswrr("nvswrr.random"); AddressParams address_params = ADDRESS_PARAMS_DEFAULT; address_params.weight = 3;//weight is 3 UpstreamManager::upstream_add_server("nvswrr.random", "192.168.2.100:8081", &address_params);//weight is 3 address_params.weight = 2;//weight is 2 UpstreamManager::upstream_add_server("nvswrr.random", "192.168.2.100:8082", &address_params);//weight is 2 UpstreamManager::upstream_add_server("nvswrr.random", "abc.sogou.com");//weight is 1 auto *http_task = WFTaskFactory::create_http_task("http://nvswrr.random:9090", 0, 0, nullptr); http_task->start(); ~~~ 1. The virtual node initialization sequence is selected according to the [SWRR algorithm](https://github.com/nginx/nginx/commit/52327e0627f49dbda1e8db695e63a4b0af4448b1) 2. The virtual nodes are initialized in batches during operation to avoid intensive computing concentration. After each batch of virtual nodes is used up, the next batch of virtual node lists can be initialized. 3. It has both the smooth and scattered characteristics of [SWRR algorithm](https://github.com/nginx/nginx/commit/52327e0627f49dbda1e8db695e63a4b0af4448b1) and the time complexity of O(1) 4. For specific details of the algorithm, see tengine(https://github.com/alibaba/tengine/pull/1306) # Upstream selection strategy When the URIHost of the url that initiates the request is filled with UpstreamName, it is regarded as a request to the Upstream corresponding to the name, and then it will be selected from the set of Addresses recorded by the Upstream: 1. Weight random strategy: selection randomly according to weight 2. Consistent hash strategy: The framework uses a standard consistent hashing algorithm, and users can define the consistent hash function consistent_hash for the requested uri 3. Manual strategy: make definite selection according to the select function that user provided for the requested uri, if the blown target is selected: **a.** If try_another is false, this request will return to failure **b.** If try_another is true, the framework uses standard consistent hash algorithm to make a second selection, and the user can define the consistent hash function consistent_hash for the requested uri 4. Main-backup strategy: According to the priority of main first, backup next, select a main server as long as it can be used. This strategy can take effect concurrently with any of [1], [2], and [3], and they influence each other. Round-robin/weighted-round-robin: regarded as equivalent to [1], not available for now The framework recommends common users to use strategy [2], which can ensure that the cluster has good fault tolerance and scalability For complex scenarios, advanced users can use strategy [3] to customize complex selection logic # Address attribute ~~~cpp struct EndpointParams { size_t max_connections; int connect_timeout; int response_timeout; int ssl_connect_timeout; bool use_tls_sni; }; static constexpr struct EndpointParams ENDPOINT_PARAMS_DEFAULT = { .max_connections = 200, .connect_timeout = 10 * 1000, .response_timeout = 10 * 1000, .ssl_connect_timeout = 10 * 1000, .use_tls_sni = false, }; struct AddressParams { struct EndpointParams endpoint_params; unsigned int dns_ttl_default; unsigned int dns_ttl_min; unsigned int max_fails; unsigned short weight; int server_type; /* 0 for main and 1 for backup. */ int group_id; }; static constexpr struct AddressParams ADDRESS_PARAMS_DEFAULT = { .endpoint_params = ENDPOINT_PARAMS_DEFAULT, .dns_ttl_default = 12 * 3600, .dns_ttl_min = 180, .max_fails = 200, .weight = 1, // only for main of UPSTREAM_WEIGHTED_RANDOM .server_type = 0, .group_id = -1, }; ~~~ Each address can be configured with custom parameters: * Max_connections, connect_timeout, response_timeout, ssl_connect_timeout of EndpointParams: connection-related parameters * dns_ttl_default: The default ttl in the dns cache in seconds, and the default value is 12 hours. The dns cache is for the current process, that is, the process will disappear after exiting, and the configuration is only valid for the current process * dns_ttl_min: The shortest effective time of dns in seconds, and the default value is 3 minutes. It is used to decide whether to perform dns again when communication fails and retry. * max_fails: the number of [continuous] failures that triggered fusing (Note: each time the communication is successful, the count will be cleared) * Weight: weight, the default value is 1, which is only valid for main. It is used for Upstream weighted random strategy selection and consistent hash strategy selection, the larger the weight is, the easier it is to be selected. * server_type: main/backup configuration, main by default (server_type=0). At any time, the main servers in the same group are always at higher priority than backups * group_id: basis for grouping, the default value is -1. -1 means no grouping (free). A free backup can be regarded as backup to any main server. Any backup with group is always at higher priority than any free backup. # About fuse ## MTTR Mean time to repair (MTTR) is the average value of the repair time when the product changes from a fault state to a working state. ## Service avalanche effect Service avalanche effect is a phenomenon in which "service caller failure" (result) is caused by "service provider's failure" (cause), and the unavailability is amplified gradually/level by level If it is not controlled effectively, the effect will not converge, but will be amplified geometrically, just like an avalanche, that’s why it is called avalanche effect Description of the phenomenon: at first it is just a small service or module abnormality/timeout, causing abnormality/timeout of other downstream dependent services, then causing a chain reaction, eventually leading to paralysis of most or all services As the fault is repaired, the effect will disappear, so the duration of the effect is usually equal to MTTR ## Fuse mechanism When the error or abnormal touch of a certain target meets the preset threshold condition, the target is temporarily considered unavailable, and the target is removed, namely fuse is started and enters the fuse period After the fuse duration reaches MTTR duration, turn into half-open status, (attempt to) restore the target If all targets are found fused whenever recovering one target, all targets will be restored at the same time Fuse mechanism strategy can effectively prevent avalanche effect ## Upstream fuse protection mechanism MTTR=30 seconds, which is temporarily not configurable, but we will consider opening it to be configured by users in the future. When the number of consecutive failures of a certain Address reaches the set upper limit (200 times by default), this Address will be blown, MTTR=30 seconds. During the fusing period, once the Address is selected by the strategy, Upstream will decide whether to try other Addresses and how to try according to the specific configuration Please note that if one of the following 1-4 scenarios is met, the communication task will get an error of WFT_ERR_UPSTREAM_UNAVAILABLE = 1004: 1. Weight random strategy, all targets are in the fusing period 2. Consistent hash strategy, all targets are in the fusing period 3. Manual strategy and try_another==true, all targets are in the fusing period 4. Manual strategy and try_another==false, and all the following three conditions shall meet at the same time: 1). The main selected by the select function is in the fusing period, and all free devices are in the fusing period 2). The main is a free main, or other targets in the group where the main is located are all in the fusing period 3). All free devices are in the fusing period # Upstream port priority 1. Priority is given to the port number explicitly configured on the Upstream Address 2. If not, select the port number explicitly configured in the request url 3. If none, use the default port number of the protocol ~~~cpp Configure UpstreamManager::upstream_add_server("my_proxy.name", "192.168.2.100:8081"); Request http://my_proxy.name:456/test.html => http://192.168.2.100:8081/test.html Request http://my_proxy.name/test.html => http://192.168.2.100:8081/test.html ~~~ ~~~cpp Configure UpstreamManager::upstream_add_server("my_proxy.name", "192.168.10.10"); Request http://my_proxy.name:456/test.html => http://192.168.10.10:456/test.html Request http://my_proxy.name/test.html => http://192.168.10.10:80/test.html ~~~ --- ### En Tutorial 01 Wget (`docs/en/tutorial-01-wget.md`) # Creating your first task: wget # Sample code [tutorial-01-wget.cc](/tutorial/tutorial-01-wget.cc) # About wget wget reads HTTP/HTTPS URLs from stdin, crawls the webpages and then print the content to stdout. It also outputs the HTTP headers of the request and the response to stderr. For convenience, wget exits with Ctrl-C, but it will ensure that all resources are completely released first. # Creating and starting an HTTP task ~~~cpp WFHttpTask *task = WFTaskFactory::create_http_task(url, REDIRECT_MAX, RETRY_MAX, wget_callback); protocol::HttpRequest *req = task->get_req(); req->add_header_pair("Accept", "*/*"); req->add_header_pair("User-Agent", "Wget/1.14 (gnu-linux)"); req->add_header_pair("Connection", "close"); task->start(); pause(); ~~~ **WFTaskFactory::create\_http\_task()** generates an HTTP task. In [WFTaskFactory.h](/src/factory/WFTaskFactory.h), the prototype is defined as follows: ~~~cpp WFHttpTask *create_http_task(const std::string& url, int redirect_max, int retry_max, http_callback_t callback); ~~~ The first few parameters are self-explanatory. **http\_callback\_t** is the callback of an HTTP task, which is defined below: ~~~cpp using http_callback_t = std::function; ~~~ To put it simply, it’s the funtion that has **Task** as one parameter and does not return any value. You can pass NULL to this callback, indicating that there is no callback. The callback in all tasks follows the same rule. Please note that all factory functions do not return failure, so even if the URL is illegal, don't worry that the task is a null pointer. All errors are handled in the callback. You can use **task->get\_req()** to get the request of the task. The default method is GET via HTTP/1.1 on long connections. The framework automatically adds request\_uri, Host and other parameters. The framework will add other HTTP header fields automatically according to the actual requirements, including Content-Length or Connection before sending the request. You may also use **add\_header\_pair()** to add your own header. For more interfaces on HTTP messages, please see [HttpMessage.h](/src/protocol/HttpMessage.h). **task->start()** starts the task. It’s non-blocking and will not fail. Then the callback of the task will be called. As it’s an asynchronous task, obviously you cannot use the task pointer after **start()**. To make the example as simple as possible, call **pause()** after **start()** to prevent the program from exiting. You can press Ctrl-C to exit the program. # Handling crawled HTTP results This example demonstrates how to handle the results with a general function. Of course, **std::function** supports more features. ~~~cpp void wget_callback(WFHttpTask *task) { protocol::HttpRequest *req = task->get_req(); protocol::HttpResponse *resp = task->get_resp(); int state = task->get_state(); int error = task->get_error(); // handle error states ... std::string name; std::string value; // print request to stderr fprintf(stderr, "%s %s %s\r\n", req->get_method(), req->get_http_version(), req->get_request_uri()); protocol::HttpHeaderCursor req_cursor(req); while (req_cursor.next(name, value)) fprintf(stderr, "%s: %s\r\n", name.c_str(), value.c_str()); fprintf(stderr, "\r\n"); // print response header to stderr ... // print response body to stdin void *body; size_t body_len; resp->get_parsed_body(&body, &body_len); // always success. fwrite(body, 1, body_len, stdout); fflush(stdout); } ~~~ In this callback, the task is generated by the factory. You can use **task->get\_state()** and **task->get\_error()** to obtain the running status and the error code of the task respectively. Let's skip the error handling first. Use **task->get\_resp()** to get the response of the task, which is slightly different from the request, as they are both derived from HttpMessage. Then, use the HttpHeaderCursor to scan the headers of the request and the response. [HttpUtil.h](/src/protocol/HttpUtil.h) contains the definition of the Cursor. ~~~cpp class HttpHeaderCursor { public: HttpHeaderCursor(const HttpMessage *message); ... void rewind(); ... bool next(std::string& name, std::string& value); bool find(const std::string& name, std::string& value); ... }; ~~~ There should be no doubt about the use of this cursor. The next line **resp->get\_parsed\_body()** obtains the HTTP body of the response. This call always returns true when the task is successful, and the body points to the data area. The call gets the raw HTTP body, and does not decode the chunk. If you want to decode the chunk, you can use the HttpChunkCursor in [HttpUtil.h](/src/protocol/HttpUtil.h). In addition, **find()** will change the pointer inside the cursor. If you want to iterate over the header after you use **find()**, please use **rewind()** to return to the cursor header. --- ### En Tutorial 02 Redis Cli (`docs/en/tutorial-02-redis_cli.md`) # Implementing Redis set and get: redis\_cli # Sample code [tutorial-02-redis\_cli.cc](/tutorial/tutorial-02-redis_cli.cc) # About redis\_cli The program reads the Redis server address and a key/value pair from the command line. Then execute SET to write this KV pair and then read them to verify that the writing is sucessful. Command: ./redis_cli \ \ \ For the sake of simplicity, press Ctrl-C to exit the program. # Format of Redis URL redis://:password@host:port/dbnum?query#fragment If SSL is used, use: rediss://:password@host:port/dbnum?query#fragment password is optional. The default port is 6379; the default dbnum is 0, and its range is from 0 to 15. query and fragment are not used in the factory and you can define them by yourself. For example, if you want to use upstream selection , you can define your own query and fragment. For relevant details, please see upstream documents. Sample Redis URL: redis://127.0.0.1/ redis://:12345678@redis.some-host.com/1 # Creating and starting a Redis task Creating a Redis task is almost the same as creating an HTTP task. The only difference is the omission of redirect\_max. ~~~cpp using redis_callback_t = std::function; WFRedisTask *create_redis_task(const std::string& url, int retry_max, redis_callback_t callback); ~~~ In this example, we want to store some user data in the Redis task, including URL and key, and use them in the callback. We can use **std::function** to bind the parameters. Here we use **void \*user\_data** pointer in the task. The pointer is a public member of the task. ~~~cpp struct tutorial_task_data { std::sring url; std::string key; }; ... struct tutorial_task_data data; data.url = argv[1]; data.key = argv[2]; WFRedisTask *task = WFTaskFactory::create_redis_task(data.url, RETRY_MAX, redis_callback); protocol::RedisRequest *req = task->get_req(); req->set_request("SET", { data.key, argv[3] }); task->user_data = &data; task->start(); pause(); ~~~ Similar to **get\_req()** in an HTTP task, **get\_req()** in an Redis task returns the Redis request for that task. You can see the functions of RedisRequest in [RedisMessage.h](/src/protocol/RedisMessage.h), where **set\_request** is used to set Redis command. ~~~cpp void set_request(const std::string& command, const std::vector& params); ~~~ There is little doubt about this interface for people who frequently use Redis. However, please note that you cannot use SELECT and AUTH commands in the request. The reason is that as you can't specify the connection every time you send a request and the next request after SELECT may not be initiated on the same connection, this command is meaningless. Please specify the database name and password in the Redis URL. And the URL of every request must contain these data. In addition, this redis client fully supports redis cluster mode. The client will process MOVED and ASK response, and redirect correctly. # Handling results After you successfully run the SET command, send the GET command to verify the writing. GET also uses the same callback. Therefore, the function will determine the source command of the results. Let's skip the error handling first again. ~~~cpp void redis_callback(WFRedisTask *task) { protocol::RedisRequest *req = task->get_req(); protocol::RedisResponse *resp = task->get_resp(); int state = task->get_state(); int error = task->get_error(); protocol::RedisValue val; ... resp->get_result(val); std::string cmd; req->get_command(cmd); if (cmd == "SET") { tutorial_task_data *data = (tutorial_task_data *)task->user_data; WFRedisTask *next = WFTaskFactory::create_redis_task(data->url, RETRY_MAX, redis_callback); next->get_req()->set_request("GET", { data->key }); series_of(task)->push_back(next); fprintf(stderr, "Redis SET request success. Trying to GET...\n"); } else /* if (cmd == 'GET') */ { // print the GET result ... fprintf(stderr, "Finished. Press Ctrl-C to exit.\n"); } } ~~~ RedisValue is the results of one Redis request. You can also see the interface in [RedisMessage.h](/src/protocol/RedisMessage.h). You need to pay special attention to the callback in the line **series\_of(task)->push\_back(next)**. It`s the firt time we use the functions of Workflow. Here **next** means the Redis task we are about to start: run GET operation. We do not use **next->start()** to start the task. We use **push\_back** to append the next task to the end of the current task queue instead. The difference between the two methods is: * When a task is initiated by **start**, the task is started immediately; when a task is **push\_back** to the queue, the **next** task is initiated after the callback. * The obvious advantage is that the **push\_back** method can ensure that the log printing is not chaotic. Otherwise, if you use the **next->start()**, the \"Finished.\" in the sample may be printed out first. * If you use **start** to initiate the next task, the current task series ends and the next task will initiate a new series. * You can set a callback for a series. For the sake of simplicity, the sample omit it. * In the parallel tasks, a series is a branch of the parallel task. If the series ends, it is considered that the brand also ends. The following tutorials demonstrates how to use parallel tasks. In a word, if you want to start the next task after one task, you usually use **push\_back** operation (in some cases, **push\_front** may be used). **series\_of()** is a very important call and it is a global function that does not belong to any class. [Workflow.h](/src/factory/Workflow.h#L140) contains its definition and implementation. ~~~cpp static inline SeriesWork *series_of(const SubTask *task) { return (SeriesWork *)task->get_pointer(); } ~~~ All tasks are derived from SubTask. And any running task must belong to one series. You can call **series\_of()** to get the series of a task. **push\_back** is a function in the SeriesWork class, which is used to append a task to the end of the series. **push\_front** is a similar function. In the sample, you can use either function. ~~~cpp class SeriesWork { ... public: void push_back(SubTask *task); void push_front(SubTask *task); ... } ~~~ SeriesWork class plays an important role in our system. In the next tutorial, you will learn more functions in SeriesWork. --- ### En Tutorial 03 Wget To Redis (`docs/en/tutorial-03-wget_to_redis.md`) # More features about series: wget\_to\_redis # Sample code [tutorial-03-wget\_to\_redis.cc](/tutorial/tutorial-03-wget_to_redis.cc) # About wget\_to\_redis The program reads one HTTP URL and one redis URL from the command line, crawls the HTTP web page and saves the content to Redis, with the key as the HTTP URL. Differing from the other two examples, we add a wake-up mechanism. The program can automatically exit and users are not required to press Ctrl-C. # Creating and configuring an HTTP task Similar to the previous example, in this example, we also executes two requests in series. The biggest difference is that we inform the main thread that the execution of the task has finished and quit normally. In addition, we add two more calls to limit the size of the crawled HTTP response content and the maximum time to receive the reply. ~~~cpp WFHttpTask *http_task = WFTaskFactory::create_http_task(...); ... http_task->get_resp()->set_size_limit(20 * 1024 * 1024); http_task->set_receive_timeout(30 * 1000); ~~~ **set\_size\_limit()** is a function in HttpMessage.It is used to limit the packet size of incoming HTTP message. Actually this interface is required in all protocol messages. **set\_receive\_timeout()** sets the timeout for receiving data, in milliseconds. The above code limits the size of the HTTP message to no more than 20M and the time for receiving the complete message to no more than 30 seconds. You can learn more about timeout configuration in the following documents. # Creating and starting a SeriesWork In the previous two examples, we call **task->start()** directly to start the first task. The actual procedure in **task->start()** is: create a SeriesWork with the task as the head and then start the series. In [WFTask.h](/src/factory/WFTask.h), you can see the implemetation of **start**. ~~~cpp template class WFNetWorkTask : public CommRequest { public: void start() { assert(!series_of(this)); Workflow::start_series_work(this, nullptr); } ... }; ~~~ We want to set a callback for that series and add some context. Therefore, instead of using the **start** interface of the task, we create our own series. You cannot new, delete or inherit a SeriesWork. It can only be generated through **Workflow::create\_series\_work()** interface. In [Workflow.h](/src/factory/Workflow.h), generally we use the following call: ~~~cpp using series_callback_t = std::function; class Workflow { public: static SeriesWork *create_series_work(SubTask *first, series_callback_t callback); }; ~~~ In the sample code, our usage is as follows: ~~~cpp struct tutorial_series_context { std::string http_url; std::string redis_url; size_t body_len; bool success; }; ... struct tutorial_series_context context; ... SeriesWork *series = Workflow::create_series_work(http_task, series_callback); series->set_context(&context); series->start(); ~~~ In the previous example, we use the pointer **void \*user\_data** in the task to save the context. However, in this example, we put the context in the series, which is more reasonable. The series is a complete task chain, and all tasks can obtain and modify the context. The callback function of the series is called after all the tasks in that series are finished. Here, we simply use a lamda function to print the running results and wake up the main thread. # Other work There's nothing special left. After the HTTP crawling is successful, a Redis task is started to write the data into the database. If the crawling fails or the length of the HTTP body is 0, the Redis task will not be started. In any case, the program can exit normally after all tasks are finished, because all tasks are in the same series. --- ### En Tutorial 04 Http Echo Server (`docs/en/tutorial-04-http_echo_server.md`) # First server: http\_echo\_server # Sample code [tutorial-04-http\_echo\_server.cc](/tutorial/tutorial-04-http_echo_server.cc) # About http\_echo\_server It is an HTTP server that returns an HTML page, which displays the header data in the HTTP request sent by the browser. The log of the program contains the client address and the sequence of the request (the number of requests on the current connection). When 10 requests are completed on the same connection, the server actively closes the connection. The program exits normally after users press Ctrl-C, and all resources are completely reclaimed. # Creating and starting an HTTP server In this example, we use the default parameters of an HTTP server. It is very simple to create and start an HTTP server. ~~~cpp WFHttpServer server(process); port = atoi(argv[1]); if (server.start(port) == 0) { pause(); server.stop(); } ... ~~~ The procedure is too simple to explain. Please note that the start process is non-blocking, so please pause the program. Obviously you can start several server objects and then pause. After a server is started, you can use **stop()** interface to shut down the server at any time. Stopping a server is non-violent and will be done until all the processing requests in the server are completed. Therefore, **stop** is a blocking operation. If non-blocking shutdown is required, please use **shutdown+wait\_finish** interface. There are several overloaded functions with **start()**. [WFServer.h](/src/server/WFServer.h) contains the following interfaces: ~~~cpp class WFServerBase { public: /* To start TCP server. */ int start(unsigned short port); int start(int family, unsigned short port); int start(const char *host, unsigned short port); int start(int family, const char *host, unsigned short port); int start(const struct sockaddr *bind_addr, socklen_t addrlen); /* To start an SSL server */ int start(unsigned short port, const char *cert_file, const char *key_file); int start(int family, unsigned short port, const char *cert_file, const char *key_file); int start(const char *host, unsigned short port, const char *cert_file, const char *key_file); int start(int family, const char *host, unsigned short port, const char *cert_file, const char *key_file); int start(const struct sockaddr *bind_addr, socklen_t addrlen, const char *cert_file, const char *key_file); /* For graceful restart or multi-process server. */ int serve(int listen_fd); int serve(int listen_fd, const char *cert_file, const char *key_file); /* Get the listening address. Used when started a server on a random port. */ int get_listen_addr(struct sockaddr *addr, socklen_t *addrlen) const; }; ~~~ There interfaces are easy to understand. If the **port** number is zero, the server will be started on a random port, and you may need to call **get_listen_addr** to abtain the actual listening address (mainly for the actual port) after the server is started. When you start an SSL server, the cert\_file and key\_file should be in PEM format. The last two **serve()** interfaces have the parameter **listen\_fd**, which is used for graceful restart or for building a simple non-TCP (such as SCTP) server. Please note that one server object corresponds to one **listen\_fd**. If the server is running on both IPv4 and IPv6 protocols, you should: ~~~cpp { WFHttpServer server_v4(process); WFHttpServer server_v6(process); server_v4.start(AF_INET, port); server_v6.start(AF_INET6, port); ... // now stop... server_v4.shutdown(); /* shutdown() is nonblocking */ server_v6.shutdown(); server_v4.wait_finish(); server_v6.wait_finish(); } ~~~ In the above code, the two servers cannot share the connection counter. Therefore, it is recommended to start the IPv6 server only, because the IPv6 server can accept IPv4 connection. # Business logic of an HTTP echo server When you build an HTTP server, you pass a process parameter, which is also an **std::function**, as defined below: ~~~cpp using http_process_t = std::function; using WFHttpServer = WFServer; template<> WFHttpServer::WFServer(http_process_t proc) : WFServerBase(&HTTP_SERVER_PARAMS_DEFAULT), process(std::move(proc)) { } ~~~ Actually, the type of **http\_proccess\_t** and the type of **http\_callback\_t** are exactly the same. Both are used to handle WFHttpTask. The job of the server is to populate the response based on the request. Similarly, we use an ordinary function to implement the process. The process iterates over the HTTP header of the request line by line and then writes them into an HTML page. ~~~cpp void process(WFHttpTask *server_task) { protocol::HttpRequest *req = server_task->get_req(); protocol::HttpResponse *resp = server_task->get_resp(); long seq = server_task->get_task_seq(); protocol::HttpHeaderCursor cursor(req); std::string name; std::string value; char buf[8192]; int len; /* Set response message body. */ resp->append_output_body_nocopy("", 6); len = snprintf(buf, 8192, "

%s %s %s

", req->get_method(), req->get_request_uri(), req->get_http_version()); resp->append_output_body(buf, len); while (cursor.next(name, value)) { len = snprintf(buf, 8192, "

%s: %s

", name.c_str(), value.c_str()); resp->append_output_body(buf, len); } resp->append_output_body_nocopy("", 7); /* Set status line if you like. */ resp->set_http_version("HTTP/1.1"); resp->set_status_code("200"); resp->set_reason_phrase("OK"); resp->add_header_pair("Content-Type", "text/html"); resp->add_header_pair("Server", "Sogou WFHttpServer"); if (seq == 9) /* no more than 10 requests on the same connection. */ resp->add_header_pair("Connection", "close"); // print log ... } ~~~ You have learned most of the HttpMessage related operations. The only new operation here is **append\_output\_body()**. Obviously, it is not very efficient for the users to generate a complete HTTP body and pass it to the framework. The user only needs to call the **append** interface to append the discrete data to the message block by block. **append\_output\_body()** operation will move the data, and another interface with the suffix **\_nocopy** will directly use the reference to the pointer. Please do not make it point to the local variables when you use it. [HttpMessage.h](../src/protocol/HttpMessage.h) contains the declaration of relevant calls. ~~~cpp class HttpMessage { public: bool append_output_body(const void *buf, size_t size); bool append_output_body_nocopy(const void *buf, size_t size); ... bool append_output_body(const std::string& buf); }; ~~~ Once again, please note that when you use **append\_output\_body\_nocopy()**, the lifecycle of the data referenced by the buf must at least be extended to the callback of the task. Another variable seq in the function is obtained by **server\_task->get\_task\_seq()**, which indicates the number of requests on the current connection, starting from 0. In the program, the connection is forcibly closed after 10 requests are completed, thus: ~~~cpp if (seq == 9) /* no more than 10 requests on the same connection. */ resp->add_header_pair("Connection", "close"); ~~~ You can also use **task->set\_keep\_alive()** to close the connection. However, for the connection using HTTP protocol, it is recommended to set the “close” option in HTTP header. In this example, because the response page is very small, we didn't pay attention to the reply status. In the next tutorial **http\_proxy**, you will learn how to get the reply status. --- ### En Tutorial 05 Http Proxy (`docs/en/tutorial-05-http_proxy.md`) # Asynchronous server: http\_proxy # Sample code [tutorial-05-http\_proxy.cc](/tutorial/tutorial-05-http_proxy.cc) # About http\_proxy It is an HTTP proxy server. You can use it in a browser after proper configuration. It supports all HTTP methods. As HTTPS proxy follows different principles, this example does not support HTTPS proxy. You can only browse HTTP websites. In the implementation, this proxy must crawl the entire HTTP page and then forward it. Therefore, there will be noticeable latency when you upload/download a large file. # Changing server configuration In the previous example, we use the default parameters of an HTTP server. In this tutorial, we will made some changes and limit the size of the request so as to prevent malicious attack. ~~~cpp int main(int argc, char *argv[]) { ... struct WFServerParams params = HTTP_SERVER_PARAMS_DEFAULT; params.request_size_limit = 8 * 1024 * 1024; WFHttpServer server(¶ms, process); if (server.start(port) == 0) { pause(); server.stop(); } else { perror("cannot start server"); exit(1); } return 0; } ~~~ Unlike the previous example, we pass an additional parameter to the server struct. Let’s see the configuration items in the HTTP server. In [WFHttpServer.h](/src/server/WFHttpServer.h), the default parameters for an HTTP server include: ~~~cpp static constexpr struct WFServerParams HTTP_SERVER_PARAMS_DEFAULT = { .transport_type = TT_TCP, .max_connections = 2000, .peer_response_timeout = 10 * 1000, .receive_timeout = -1, .keep_alive_timeout = 60 * 1000, .request_size_limit = (size_t)-1, .ssl_accept_timeout = 10 * 1000, }; ~~~ **transport\_type**: the transport layer protocol. Besides the default type TT_TCP, you may specify TT_UDP, or TT_SCTP on Linux platform. **max\_connections**: the maximum number of connections is 2000. When it is exceeded, the least recently used keep-alive connection will be closed. If there is no keep-alive connection, the server will refuse new connections. **peer\_response\_timeout**: set the maximum duration for reading or sending out a block of data. The default setting is 10 seconds. **receive\_timeout**: set the maximum duration for receiving a complete request; -1 means unlimited time. **keep\_alive\_timeout**: set the maximum duration for maintaining a connection. The default setting is 1 minute. **request\_size\_limit**: set the maximum size of a request packet. The default setting is unlimited packet size. **ssl\_accept\_timeout**: set the maximum duration for an SSL handshake. The default setting is 10 seconds. There is no **send\_timeout** in the parameters. **send\_timeout** sets the timeout for sending a complete response. This parameter should be determined according to the size of the response packet. # Business logic of a proxy server Essentially, this proxy server forwards a user's request intactly to the corresponding web server, and then forwards the reply from the web server intactly to the user. In the request sent by a browser to the proxy, the Request URL contains scheme, host and port, which should be removed before forwarding. For example, when the browser visits `http://www.sogou.com/`, the first line of the request sent by the browser to the proxy is: `GET` `http://www.sogou.com/` `HTTP/1.1` which should be rewritten as: `GET` `/` `HTTP/1.1` ~~~cpp void process(WFHttpTask *proxy_task) { auto *req = proxy_task->get_req(); SeriesWork *series = series_of(proxy_task); WFHttpTask *http_task; /* for requesting remote webserver. */ tutorial_series_context *context = new tutorial_series_context; context->url = req->get_request_uri(); context->proxy_task = proxy_task; series->set_context(context); series->set_callback([](const SeriesWork *series) { delete (tutorial_series_context *)series->get_context(); }); http_task = WFTaskFactory::create_http_task(req->get_request_uri(), 0, 0, http_callback); const void *body; size_t len; /* Copy user's request to the new task's reuqest using std::move() */ req->set_request_uri(http_task->get_req()->get_request_uri()); req->get_parsed_body(&body, &len); req->append_output_body_nocopy(body, len); *http_task->get_req() = std::move(*req); /* also, limit the remote webserver response size. */ http_task->get_resp()->set_size_limit(200 * 1024 * 1024); *series << http_task; } ~~~ The above contains the entire content of the process. It first parses the struct of an HTTP request sent by a web server. **req->get\_request\_uri()** is used to get the complete URL of the request sent by a browser. And then build a HTTP task to the server based on this URL. Both the retry times and the redirection times of this HTTP task is 0, because the redirection is handled by the browser and the browser will be resend the request when it meets 302, etc. ~~~cpp req->set_request_uri(http_task->get_req()->get_request_uri()); req->get_parsed_body(&body, &len); req->append_output_body_nocopy(body, len); *http_task->get_req() = std::move(*req); ~~~ In fact, the above four lines generates a HTTP request to the web server. req is the received HTTP request, and it will be moved directly to the new request via **std::move()**. The first line removes the `http://host:port` in the request\_uri and keeps the part after the path. The second line and the third line specify the parsed HTTP body as the HTTP body for output. The reason for this operation is that in the HttpMessage implementation, the http body obtained by parsing and the http body to send out are two fields, so we need to simply set it here, without copying the memory. The fourth line transfers the request content to the request sent to the web server at one time. After the HTTP request is constructed, the request is placed at the end of the current series, and the process function ends. # Principles behind an asynchronous server Obviously, the process function is only part of the proxy logic. We also need to handle the HTTP response returned from the web server and generates the response for the browser. In the example of echo server, we populate the response page directly without network communication. However, in the proxy server, we have to wait for the response from the web server. Of course, we can occupy the thread of this process function and wait for the returned result, but this synchronous waiting mode is obviously not desirable. Thus, it is better that we reply to the user's request asynchronously after receiving the results for the request, and no thread is occupied while we are waiting for the result. Therefore, we set a context for the current series in the head of the process, which contains the proxy\_task itself. In this way, we can populate the results asynchronously. ~~~cpp struct tutorial_series_context { std::string url; WFHttpTask *proxy_task; bool is_keep_alive; }; void process(WFHttpTask *proxy_task) { SeriesWork *series = series_of(proxy_task); ... tutorial_series_context *context = new tutorial_series_context; context->url = req->get_request_uri(); context->proxy_task = proxy_task; series->set_context(context); series->set_callback([](const SeriesWork *series) { delete (tutorial_series_context *)series->get_context(); }); ... } ~~~ In the previous client example, we said that any running task is in a series, and the server task is no exception. Thus, we can get the current series and set the context. In which the URL is mainly used for the subsequent logs, and the proxy\_task is the main content, which is used for resp later. Next, Let’s see how to handle the responses from the web server. ~~~cpp void http_callback(WFHttpTask *task) { int state = task->get_state(); auto *resp = task->get_resp(); SeriesWork *series = series_of(task); tutorial_series_context *context = (tutorial_series_context *)series->get_context(); auto *proxy_resp = context->proxy_task->get_resp(); ... if (state == WFT_STATE_SUCCESS) { const void *body; size_t len; /* set a callback for getting reply status. */ context->proxy_task->set_callback(reply_callback); /* Copy the remote webserver's response, to proxy response. */ resp->get_parsed_body(&body, &len); resp->append_output_body_nocopy(body, len); *proxy_resp = std::move(*resp); ... } else { // return a "404 Not found" page ... } } ~~~ Here we focus on the successful cases only. If the proxy gets a complete HTTP page from the web server, no matter what the return code is, it is considered a success. All failure will simply return a 404 page. Because the data returned to the user may be very large, the maximum size is set to 200MB in this example. Therefore, unlike the previous examples, we need to check the success/failure status of the reply. The type of an HTTP server task is identical to the type of an HTTP client task created by ourselves. Both are WFHttpTask. The difference is that a server task is created by the framework, and its callback is initially empty. The callback of a server task is the same as that of a client. Both are called after an HTTP interaction is completed. Therefore, for all server tasks, the callback is called after the reply is completed. The following three lines of code are explained before. They transfer the response packets from the web server to the proxy response packets without copying. After the **http\_callback** function is ended, the reply to the browser is sent out. Everything is done asynchronously. The remaining function **reply\_callback()** is used just to print some logs here. The proxy task will be automatically deleted after this callback is finished. Finally, the context is destroyed in the callback of the series. # Timing of a server reply Please note that the reply message is sent automatically after all other tasks in the series are finished, so there is no **task->reply()** interface. However, there is a **task->noreply()**. If this interface is called for the server task, the connection will be closed directly at the original reply time. But the callback will still be called (its state is NOREPLY). In the callback of a server task, you can also call **series\_of()** to get the series of that server task. Then, you can still add new tasks to this series, although the reply has finished. --- ### En Tutorial 06 Parallel Wget (`docs/en/tutorial-06-parallel_wget.md`) # A simple parallel wget: parallel\_wget # Sample code [tutorial-06-parallel\_wget.cc](/tutorial/tutorial-06-parallel_wget.cc) # About parallel\_wget It is our first example on parallel tasks. The program reads multiple HTTP URLs (separated by spaces) from the command line, crawls these URLs in parallel, and prints the crawled results to the standard output according to the input order. # Creating a parallel task In the previous example, you have already learned the SeriesWork class. * SeriesWork consists of a series of tasks that are executed sequentially. The series finishes when all its tasks finish. * ParallelWork class, corresponding to the SeriesWork, consists of multiple series that are executed in parallel. The parallel work finishes when all its series finish. * ParallelWork is a task. According to the above definition, you can generate any complex workflow dynamically or statically. The Workflow class has two interfaces for generating parallel tasks: ~~~cpp class Workflow { ... public: static ParallelWork * create_parallel_work(parallel_callback_t callback); static ParallelWork * create_parallel_work(SeriesWork *const all_series[], size_t n, parallel_callback_t callback); ... }; ~~~ The first interface creates an empty parallel task, and the second interface creates parallel tasks with a series array. Before you start the parallel work, you can use **add\_series()** interface of the ParallelWork to add series to the parallel tasks generated by either interface. In the sample code, we create an empty parallel task and then add the series one by one. ~~~cpp int main(int argc, char *argv[]) { ParallelWork *pwork = Workflow::create_parallel_work(callback); SeriesWork *series; WFHttpTask *task; HttpRequest *req; tutorial_series_context *ctx; int i; for (i = 1; i < argc; i++) { std::string url(argv[i]); ... task = WFTaskFactory::create_http_task(url, REDIRECT_MAX, RETRY_MAX, [](WFHttpTask *task) { // store resp to ctx. }); req = task->get_req(); // add some headers. ... ctx = new tutorial_series_context; ctx->url = std::move(url); series = Workflow::create_series_work(task, nullptr); series->set_context(ctx); pwork->add_series(series); } ... } ~~~ You can see that we first create an HTTP task in the code, but the HTTP task cannot be directly added to the parallel task, so we need to use it to create a series first. Each series has its own context, which is used to save the URL and the crawled results. You can learn related methods in our previous examples. # Saving and using the crawled results The callback of an HTTP task is a simple lambda function, which saves the crawled result in its own series context, so that it can be retrieved by the parallel task. ~~~cpp task = WFTaskFactory::create_http_task(url, REDIRECT_MAX, RETRY_MAX, [](WFHttpTask *task) { tutorial_series_context *ctx = (tutorial_series_context *)series_of(task)->get_context(); ctx->state = task->get_state(); ctx->error = task->get_error(); ctx->resp = std::move(*task->get_resp()); }); ~~~ This is necessary, because HTTP tasks will be recycled after the callback, so we have to use **std::move()** to move the resp. In the callback of parallel tasks, we can easily get the results: ~~~cpp void callback(const ParallelWork *pwork) { tutorial_series_context *ctx; const void *body; size_t size; size_t i; for (i = 0; i < pwork->size(); i++) { ctx = (tutorial_series_context *)pwork->series_at(i)->get_context(); printf("%s\n", ctx->url.c_str()); if (ctx->state == WFT_STATE_SUCCESS) { ctx->resp.get_parsed_body(&body, &size); printf("%zu%s\n", size, ctx->resp.is_chunked() ? " chunked" : ""); fwrite(body, 1, size, stdout); printf("\n"); } else printf("ERROR! state = %d, error = %d\n", ctx->state, ctx->error); delete ctx; } } ~~~ Here, you can see the two new interfaces of ParallelWork, **size()** and **series\_at(i)**, which are used to obtain the number of the series in parallel and the ith parallel series respectively. You can use **series->get\_context()** to get the context of the series and print out the results.The printing order must be the same as with the order you add the series into the work. In this example, there is no other work after the parallel tasks finish. As we said above, ParallelWork is a kind of tasks, so you can use **series\_of()** to get its series and add a new task. However, if the crawled results are used in the new task, you need to use **std::move()** to move the data to the context of the series of that parallel task. # Starting a parallel task As a parallel task is a kind of tasks, so there is nothing special in starting a parallel task. You can call **start()** directly, or you can use it to build or start a series. In this example, we start a series, wake up the main process in the callback of this series, and exit the program normally. We can also wake up the main process in the callback of parallel tasks, and there is little difference in the program behaviors. However, it is more formal to wake up the main process in the callback of the series. --- ### En Tutorial 07 Sort Task (`docs/en/tutorial-07-sort_task.md`) # Using the built-in algorithm factory: sort\_task # Sample code [tutorial-07-sort\_task.cc](/tutorial/tutorial-07-sort_task.cc) # About sort\_task The program reads a number n from the command line, sorts the random n positive integers in ascending order, and then sorts the results in descending order. You can add the second parameter "p” to the program, and then it can be sorted in parallel. For example: ./sort\_task 100000000 p The above command will sort 100 million integers in ascending order and then in descending order. The two sortings are done in parallel respectively. # About computing tasks Computing tasks (or thread tasks) is a very important function in the framework. When you use the task flow, it is not recommended to directly perform very complicated computation in the callback. All the computations that consume a lot of CPU time can be encapsulated into computing tasks and handed over to the system for scheduling. There is no difference in the usage between computing tasks and networking tasks. The algorithm factory of the system provides some common computing tasks, such as sorting, merging and so on. You can also easily define your own computing tasks. # Creating sorting tasks in ascending order ~~~cpp int main(int argc, char *argv[]) { ... WFSortTask *task; if (use_parallel_sort) task = WFAlgoTaskFactory::create_psort_task("sort", array, end, callback); else task = WFAlgoTaskFactory::create_sort_task("sort", array, end, callback); ... task->start(); ... } ~~~ Unlike WFHttpTask or WFRedisTask, the sorting task has one more template parameter to represent the type of array data to be sorted. **create\_sort\_task** and **create\_psort\_task** produce a common sorting task and a parallel sorting task respectively. Their ****parameters and return values are the same.**** The only thing that needs special explanation is the first parameter "sort", which is the name of the computation queue. It is used to instruct the internal task scheduling. The latter part in this article explains the usage of the queue name. There is no difference in the starting methods and usage between computing tasks and networking tasks. # Handling results Like a networking task, the results are handled in the callback. In this example, the ascending sorting is followed by one descending sorting. ~~~cpp using namespace algorithm; void callback(void SortTask *task) { SortInput *input = task->get_input(); int *first = input->first; int *last = input->last; // print result ... if (task->user_data == NULL) { auto cmp = [](int a1, int a2){ return a2 < a1; }; WFSortTask *reverse; if (use_parallel_sort) reverse = WFAlgoTaskFactory::create_psort_task("sort", first, last, cmp, callback); else reverse = WFAlgoTaskFactory::create_sort_task("sort", first, last, cmp, callback); reverse->user_data = (void *)1; /* as a flag */ series_of(task)->push_back(reverse); } else { // all done. Signal main thread to exit. ... } } ~~~ You can use **get\_input ()** interface of a computing task to get the input data, and use **get\_output ()** to get the output data. For sorting tasks, the input and output are of the same type, and the content are exactly the same. [WFAlgoTaskFactory.h](/src/factory/WFAlgoTaskFactory.h) contains the definitions of the input and output of sorting tasks. ~~~cpp namespace algorithm { template struct SortInput { T *first; T *last; }; template using SortOutput = SortInput; } template using WFSortTask = WFThreadTask, algorithm::SortOutput>; template using sort_callback_t = std::function *)>; ~~~ Obviously, the first and last in the input or output mean the head pointer and the tail pointer of the array to be sorted. Next, we will create a descending sorting task. In this case, we need to pass in a comparison function. ~~~cpp auto cmp = [](int a1, int a2)->bool{ return a2 < a1; }; reverse = WFAlgoTaskFactory::create_sort_task("sort", first, last, cmp, callback); ~~~ Our usage differs slightly from **std::sort()**. Our first and last are pointers, not iterators. Similarly, you can use **create\_psort\_task()** to create a parallel sorting task. And the use of series in the sorting task is no different from that in the networking task. # About the configuration of the computing threads If you don't make any configuration, the calculation scheduler will set the number of threads as the number of the CPU cores in the machine. You can change the value with the following method: ~~~cpp #include "workflow/WFGlobal.h" int main() { struct WFGlobalSettings settings = GLOBAL_SETTINGS_DEFAULT; settings.compute_threads = 16; WORKFLOW_library_init(&settings); ... } ~~~ With the above configuration, the system will create 16 threads for computations. # About the parallel sorting algorithm The built-in parallel sorting algorithm use block+two-way merge. Its space complexity is O(1). The algorithm uses globally configured computing threads for computation, but at most 128 threads can be used. Because no extra space is used, the speedup ratio will be smaller than the number of threads, and the average CPU usage will be smaller. For the detailed implementation, please see [WFAlgoTaskFactory.inl](/src/factory/WFAlgoTaskFactory.inl). # About the name of a calculation task queue The computing task does not have priority levels. The only thing that can affect the scheduling order is the queue name of a computing task. In this example, the queue name is a string "sort". To name a queue is very simple. Please note the following items: * The queue name is a static string, and new queue names cannot be generated infinitely. For example, you cannot generate the queue name according to the request id, because each queue is allocated a small block of resources internally. * If the computing threads are not 100% occupied, all tasks are started in real time, and the queue names have no effect. * If there are multiple computing steps in a service flow and they are interspersed among multiple network communications, you can simply give each calculation step a name, which is better than using one name as a whole. * If all computing tasks use the same name, the scheduling order of all tasks is consistent with the order of submission, which will affect the average response time in some scenarios. * If each kind of computing task has an independent name, it means that they are scheduled fairly. And the same kind of tasks are scheduled sequentially, the practical effect is better. * In a word, unless the computing load of the machine is already very heavy, you do not need to pay special attention to the queue name and you can just give each kind of task a name. --- ### En Tutorial 08 Matrix Multiply (`docs/en/tutorial-08-matrix_multiply.md`) # User-defined computing tasks: matrix\_multiply # Sample code [tutorial-08-matrix\_multiply.cc](/tutorial/tutorial-08-matrix_multiply.cc) # About matrix\_multiply The program multiplies two matrices and prints the results on the screen. The main purpose of the example is to show how to implement a user-defined CPU computing task. # About computing tasks You need to provide three types of basic information when you define a computer task: INPUT, OUTPUT, and routine. INPUT and OUTPUT are two template parameters, which can be of any type. routine means the process from INPUT to OUTPUT, which is defined as follows: ~~~cpp template class __WFThreadTask { ... std::function routine; ... }; ~~~ It can be seen that routine is a simple computing process from INPUT to OUTPUT. The INPUT pointer is not necessarily be const, but you can also pass the function of const INPUT \*. For example, to implement an adding task, you can: ~~~cpp struct add_input { int x; int y; }; struct add_ouput { int res; }; void add_routine(const add_input *input, add_output *output) { output->res = input->x + input->y; } typedef WFThreadTask add_task; ~~~ In the example of matrix multiplication, the input is two matrices and the output is one matrix. They are defined as follows: ~~~cpp namespace algorithm { using Matrix = std::vector>; struct MMInput { Matrix a; Matrix b; }; struct MMOutput { int error; size_t m, n, k; Matrix c; }; void matrix_multiply(const MMInput *in, MMOutput *out) { ... } } ~~~ As the input matrices may be illegal in matrix multiplication, so there is an error field in the output to indicate errors. # Generating computing tasks After you define the types of input and output and the algorithm process, you can use WFThreadTaskFactory to generate a computing task. In [WFTaskFactory.h](/src/factory/WFTaskFactory.h), the computing task factory is defined as follows: ~~~cpp template class WFThreadTaskFactory { private: using T = WFThreadTask; public: static T *create_thread_task(const std::string& queue_name, std::function routine, std::function callback); static T *create_thread_task(time_t seconds, long nanoseconds, const std::string& queue_name, std::function routine, std::function callback); ... }; ~~~ There are two interfaces for creating tasks here. The second interface supports the user to pass in the task running time limit, we will introduce this function in the next section. Slightly different from the previous network factory class or the algorithm factory class, this factory requires two template parameters: INPUT and OUTPUT. queue\_name is explained in the previous example. routine is the computation process, and callback means the callback. In our example, we see this call: ~~~cpp using MMTask = WFThreadTask; using namespace algorithm; int main() { typedef WFThreadTaskFactory MMFactory; MMTask *task = MMFactory::create_thread_task("matrix_multiply_task", matrix_multiply, callback); MMInput *input = task->get_input(); input->a = {{1, 2, 3}, {4, 5, 6}}; input->b = {{7, 8}, {9, 10}, {11, 12}}; ... } ~~~ After the task is generated, use **get\_input()** interface to get the pointer of the input data. This is similar to the **get\_req()** in a network task. The start and the end of a task is the same as those of a network task. Similarly, the callback is very simple: ~~~cpp void callback(MMTask *task) // MMtask = WFThreadTask { MMInput *input = task->get_input(); MMOutput *output = task->get_output(); assert(task->get_state() == WFT_STATE_SUCCESS); if (output->error) printf("Error: %d %s\n", output->error, strerror(output->error)); else { printf("Matrix A\n"); print_matrix(input->a, output->m, output->k); printf("Matrix B\n"); print_matrix(input->b, output->k, output->n); printf("Matrix A * Matrix B =>\n"); print_matrix(output->c, output->m, output->n); } } ~~~ You can ignore the the possibility of failure in the ordinary computing tasks, and the end state is always SUCCESS. The callback simply prints out the input and the output. If the input data are illegal, the error will be printed out. # Computing task with running time limit Obviously, our framework can not interrupt a computing task because it's a user function, and the users have to make sure the function will terminate normally. But we support users to create a computing task with a running time limit, and if the task doesn't finish within this time, the task will callback directly: ~~~cpp template class WFThreadTaskFactory { private: using T = WFThreadTask; public: static T *create_thread_task(time_t seconds, long nanoseconds, const std::string& queue_name, std::function routine, std::function callback); ... }; ~~~ This create_thread_task function needs to pass two more parameters, seconds and nanoseconds. If the running time of func reaches the seconds+nanosconds time limit, the task callback directly, and the state is WFT_STATE_SYS_ERROR and the error is ETIMEDOUT. But the task routine will continue to run till the end. # Symmetry of the algorithm and the protocol In our system, algorithms and protocols are highly symmetrical on a very abstract level. There are thread tasks with user-defined algorithms, obviously there are network tasks with user-defined protocols. A user-defined algorithm requires the user to provide the algorithm procedure, and a user-defined protocol requires the user to provide the procedure of serialization and deserialization. You can see an introduction in [Simple client/server based on user-defined protocols](/tutorial-10-user_defined_protocol.md) For the user-defined algorithms and the user-defined protocols, both must be very pure . For example, an algorithm is just a conversion procedure from INPUT to OUPUT, and the algorithm does not know the existence of task, series, etc. The implementation of an HTTP protocol only cares about serialization and deserialization, and does not need to care about the task definition. Instead, the HTTP protocol is referred to in an http task. # Composite features of thread tasks and network tasks In this example, we use WFThreadTaskFactory to build a thread task. This is the simplest way to get a computing task, and it is sufficient in most cases. Similarly, you can simply define a server and a client with a user-defined protocol. However, in the previous example, we can use the algorithm factory to generate a parallel sorting task, which is obviously not possible with a routine. For a network task, such as a Kafka task, interactions with several machines may be required to get results, but it is completely transparent to users. Therefore, our tasks are composite. If you use our framework skillfully, you can design many composite components. --- ### En Tutorial 09 Http File Server (`docs/en/tutorial-09-http_file_server.md`) # Http server with file IO: http\_file\_server # Sample code [tutorial-09-http\_file\_server.cc](/tutorial/tutorial-09-http_file_server.cc) # About http\_file\_server http\_file\_server is a web server. You can start a web server after specifying the startup port and the root path (the default setting is the current path). You can also specify a certificate file and a key file in PEM format to start an HTTPS web server. User may access the server through command line, the request will be sent to IP address 127.0.0.1. The program mainly demonstrates how to use disk IO tasks. In the Linux system, we use the aio interface in the kernel of Linux, and the file reading is completely asynchronous. # Starting a server For starting a server, the steps are almost the same as those when starting an echo server or an HTTP proxy. There is one more way to start an SSL server here: ~~~cpp class WFServerBase { ... int start(unsigned short port, const char *cert_file, const char *key_file); ... }; ~~~ In other words, you can specify a cert file and a key file in PEM format to start an SSL server. In addition, when you define a server, you can use **std::bind()** to bind a root parameter to the process. The root parameter means the root path of the service. ~~~cpp void process(WFHttpTask *server_task, const char *root) { ... } int main(int argc, char *argv[]) { ... const char *root = (argc >= 3 ? argv[2] : "."); auto&& proc = std::bind(process, std::placeholders::_1, root); WFHttpServer server(proc); // start server ... } ~~~ # Handling requests Similar to http\_proxy, no threads are occupied in file reading. Instead, an asynchronous task is generated to read files, and a reply to the request is generated after the reading is completed. Please note again that the complete reply data should be read into the memory before the reply message is sent. Therefore, it is not suitable for transferring very large files. ~~~cpp void process(WFHttpTask *server_task, const char *root) { // generate abs path. ... int fd = open(abs_path.c_str(), O_RDONLY); if (fd >= 0) { size_t size = lseek(fd, 0, SEEK_END); void *buf = malloc(size); /* As an example, assert(buf != NULL); */ WFFileIOTask *pread_task; pread_task = WFTaskFactory::create_pread_task(fd, buf, size, 0, pread_callback); /* To implement a more complicated server, please use series' context * instead of tasks' user_data to pass/store internal data. */ pread_task->user_data = resp; /* pass resp pointer to pread task. */ server_task->user_data = buf; /* to free() in callback() */ server_task->set_callback([](WFHttpTask *t){ free(t->user_data); }); series_of(server_task)->push_back(pread_task); } else { resp->set_status_code("404"); resp->append_output_body("404 Not Found."); } } ~~~ Unlike http\_proxy that generates a new HTTP client task, here a pread task is generated by the factory. [WFAlgoTaskFactory.h](/src/factory/WFTaskFactory.h) contains the definitions of relevant interfaces. ~~~cpp struct FileIOArgs { int fd; void *buf; size_t count; off_t offset; }; ... using WFFileIOTask = WFFileTask; using fio_callback_t = std::function; ... class WFTaskFactory { public: ... static WFFileIOTask *create_pread_task(int fd, void *buf, size_t count, off_t offset, fio_callback_t callback); static WFFileIOTask *create_pwrite_task(int fd, void *buf, size_t count, off_t offset, fio_callback_t callback); /* Interface with file path name */ static WFFileIOTask *create_pread_task(const std::string& path, void *buf, size_t count, off_t offset, fio_callback_t callback); static WFFileIOTask *create_pwrite_task(const std::string& path, void *buf, size_t count, off_t offset, fio_callback_t callback); }; ~~~ Both pread and pwrite return WFFileIOTask. We do not distinguish between sort and psort, and we do not distinguish between client and server task. They all follow the same principle. In addition to these two interfaces, preadv and pwritev return WFFileVIOTask; fsync and fdatasync return WFFileSyncTask. You can see the details in the header file. The example uses the user\_data field of the task to save the global data of the service. For larger services, we recommend to use series context. You can see the [proxy examples](/tutorial/tutorial-05-http_proxy.cc) for details. # Handling file reading results ~~~cpp using namespace protocol; void pread_callback(WFFileIOTask *task) { FileIOArgs *args = task->get_args(); long ret = task->get_retval(); HttpResponse *resp = (HttpResponse *)task->user_data; /* close fd only when you created File IO task with **fd** interface. */ close(args->fd); if (ret < 0) { resp->set_status_code("503"); resp->append_output_body("503 Internal Server Error."); } else /* Use '_nocopy' carefully. */ resp->append_output_body_nocopy(args->buf, ret); } ~~~ Use **get\_args()** of the file task to get the input parameters. Here it is a FileIOArgs struct, and it's **fd** field will be -1 if the task was created with **pathname**. Use **get\_retval()** to get the return value of the operation. If ret < 0, the task fails. Otherwise, the ret is the size of the read data. In the file task, ret < 0 and task->get\_state()! = WFT\_STATE\_SUCCESS are completely equivalent. The memory of the buf domain is managed by ourselves. You can use **append\_output\_body\_nocopy()** to pass that memory to resp. After the reply is completed, we will **free()** this block of memory with this line in the process: server\_task->set\_callback(\[](WFHttpTask \*t){ free(t->user\_data); }); # Interact with the server through command line After the server is started, users may access it through command line. Simply input the file name that you want to get, or input Ctrl-D to end the program. The repeating process is implemnted by using WFRepeaterTask, which can be created by this factory function: ~~~cpp using repeated_create_t = std::function; using repeater_callback_t = std::function; class WFTaskFactory { WFRpeaterTask *create_repeater_task(repeated_create_t create, repeater_callback_t callback); }; ~~~ As above, a repeater task is created with a task creator function. The repeater calls the task creator repeatedly and run the task until the creator return a NULL pointer. When the using's input is not empty, our creator will create an HTTP task on IP 127.0.0.1 to access the server. ~~~cpp { auto&& create = [&scheme, port](WFRepeaterTask *)->SubTask *{ ... scanf("%1023s", buf); if (*buf == '\0') return NULL; std::string url = scheme + "127.0.0.1:" + std::to_string(port) + "/" + buf; WFHttpTask *task = WFTaskFactory::create_http_task(url, 0, 0, [](WFHttpTask *task) { ... }); return task; }; WFFacilities::WaitGroup wg(1); WFRepeaterTask *repeater; repeater = WFTaskFactory::create_repeater_task(create, [&wg](WFRepeaterTask *) { wg.done(); }); repeater->start(); wg.wait(); server.stop(); } ~~~ Finally, when the creator returned NULL, the repeater's callback is called and the program will be ended. # About the implementation of the file IO Linux operating system supports a set of asynchronous IO system calls with high efficiency and very little CPU occupation. If you use our framework in a Linux system, this set of interfaces are used by default. We have implemented a set of posix aio interfaces to support other UNIX systems, and used the sigevent notification method of threads, but it is no longer in use because of its low efficiency. Currently, for non-Linux systems, asynchronous IO is always simulated by multi-threading. When an IO task arrives, a thread is created in real time to execute IO tasks, and then a callback is used to return to the handler thread pool. Multi-threaded IO is also the only choice in macOS, because macOS does not have good sigevent support and posix aio will not work in macOS. Some UNIX systems do not support fdatasync. In this case, an fdatasync task is equivalent to an fsync task. --- ### En Tutorial 10 User Defined Protocol (`docs/en/tutorial-10-user_defined_protocol.md`) # A simple user-defined protocol: client/server # Sample codes [message.h](/tutorial/tutorial-10-user_defined_protocol/message.h) [message.cc](/tutorial/tutorial-10-user_defined_protocol/message.cc) [server.cc](/tutorial/tutorial-10-user_defined_protocol/server.cc) [client.cc](/tutorial/tutorial-10-user_defined_protocol/client.cc) # About user\_defined\_protocol This example designs a simple communication protocol, and builds a server and a client on that protocol. The server converts the message sent by client into uppercase and returns it to the client. # Protocol format The protocol message contains one 4-byte head and one message body. Head is an integer in network byte order, indicating the length of body. The formats of the request messages and the response messages are identical. # Protocol implementation A user-defined protocol should provide its own serialization and deserialization methods, which are virtual functions in ProtocolMeessage class. In addition, for the convenience of use, we strongly recommend users to implement the **move constructor** and **move assignment** for messages (for std::move ()). [ProtocolMessage.h](/src/protocol/ProtocolMessage.h) contains the following serialization and deserialization interfaces: ~~~cpp namespace protocol { class ProtocolMessage : public CommMessageOut, public CommMessageIn { private: virtual int encode(struct iovec vectors[], int max); /* You have to implement one of the 'append' functions, and the first one * with arguement 'size_t *size' is recommmended. */ virtual int append(const void *buf, size_t *size); virtual int append(const void *buf, size_t size); ... }; } ~~~ ### Serialization function: encode * The encode function is called before the message is sent, and it is called only once for each message. * In the encode function, you need to serialize the message into a vector array, and the number of array elements must not exceed max. Current the value of max is 2048. * For the definition of **struct iovec**, please see the system calls **readv** or **writev**. * Normally the return value of the encode function is between 0 and max, indicating how many vector are used in the message. * In case of UDP protocol, please note that the total length must not be more than 64k, and no more than 1024 vectors are used (in Linux, writev writes only 1024 vectors at one time). * The encode -1 indicates errors. To return -1, you need to set errno. If the return value is > max, you will get an EOVERFLOW error. All errors are obtained in the callback. * For performance reasons, the content pointed to by the iov\_base pointer in the vector will not be copied. So it generally points to the member of the message class. ### Deserialization function: append * The append function is called every time a data block is received. Therefore, for each message, it may be called multiple times. * buf and size are the content and the length of received data block respectively. You need to move the data content. * If the interface **append(const void \*buf, size\_t \*size)** is implemented, you can tell the framework how much length is consumed at this time by modifying \* size. remaining size = received size - consumed size, and the remaining part of the buf will be received again when the append is called next time. This function is more convenient for protocol parsing. Of course, you can also move the whole content and manage it by yourself. In this case, you do not need to modify \*size. * If the **append** function returns 0, it indicates that the message is incomplete and the transmission continues. The return value of 1 indicates the end of the message. -1 indicates errors, and you need to set errno. * In a word, the append function is used to tell the framework whether the message transmission is completed or not. Please don't perform complicated and unnecessary protocol parsing in the append. ### Setting the errno * If encode or append returns -1 or other negative numbers, it should be interpreted as failure, and you should set the errno to pass the error reason. You can obtain this error in the callback. * If the system calls or the library functions such as libc fail (for example, malloc), libc will definitely set errno, and you do not need to set it again. * Some errors of illegal messages are quite common. For example, EBADMSG or EMSGSIZE can be used to indicate that the message content is wrong and the message is too large respectively. * You can use a value that exceeds the errno range defined in the system to indicate a user-defined error. Generally, you can use a value greater than 256. * Please do not use a negative errno. Because negative numbers are used inside the framework to indicate SSL errors. In our example, the serialization and deserialization of messages are very simple. The header file [message.h](/tutorial/tutorial-10-user_defined_protocol/message.h) declares the request class and the response class. ~~~cpp namespace protocol { class TutorialMessage : public ProtocolMessage { private: virtual int encode(struct iovec vectors[], int max); virtual int append(const void *buf, size_t size); ... }; using TutorialRequest = TutorialMessage; using TutorialResponse = TutorialMessage; } ~~~ Both the request class and the response class belong to the same type of messages. You can directly introduce them with using. Note that both the request and the response can be constructed without parameters. In other words, you must provide a constructor without parameters or no constructor. In addition, the response object may be destroyed and reconstruct during communication if retrial occurs, therefore it should be a RAII class, otherwise things will be complicated). [message.cc](/tutorial/tutorial-10-user_defined_protocol/message.cc) contains the implementation of encode and append: ~~~cpp namespace protocol { int TutorialMessage::encode(struct iovec vectors[], int max/*max==8192*/) { uint32_t n = htonl(this->body_size); memcpy(this->head, &n, 4); vectors[0].iov_base = this->head; vectors[0].iov_len = 4; vectors[1].iov_base = this->body; vectors[1].iov_len = this->body_size; return 2; /* return the number of vectors used, no more then max. */ } int TutorialMessage::append(const void *buf, size_t size) { if (this->head_received < 4) { size_t head_left; void *p; p = &this->head[this->head_received]; head_left = 4 - this->head_received; if (size < 4 - this->head_received) { memcpy(p, buf, size); this->head_received += size; return 0; } memcpy(p, buf, head_left); size -= head_left; buf = (const char *)buf + head_left; p = this->head; this->body_size = ntohl(*(uint32_t *)p); if (this->body_size > this->size_limit) { errno = EMSGSIZE; return -1; } this->body = (char *)malloc(this->body_size); if (!this->body) return -1; this->body_received = 0; } size_t body_left = this->body_size - this->body_received; if (size > body_left) { errno = EBADMSG; return -1; } memcpy(this->body, buf, size); if (size < body_left) return 0; return 1; } } ~~~ The implementation of encode is very simple, in which two vectors are always, pointing to the head and the body respectively. Note that the iov\_base pointer must point to a member of the message class. When you use append, you should ensure that the 4-byte head is received completely before reading the message body. Moreover, we can't guarantee that the first append must contain a complete head, so the process is a little cumbersome. The append implements the size\_limit function, and an EMSGSIZE error will be returned if the size\_limit is exceeded. You can ignore the size_limit field if you don't need to limit the message size. Because we require the communication protocol is two way with a request and a response, users do not need to consider the so-called "TCP packet sticking" problem. The problem should be treated as an error message directly. Now, with the definition and implementation of messages, we can build a server and a client. # Server and client definitions With the request and response classes, we can build a server and a client based on this protocol. The previous example explains the type definitions related to an HTTP protocol: ~~~cpp using WFHttpTask = WFNetworkTask; using http_callback_t = std::function; using WFHttpServer = WFServer; using http_process_t = std::function; ~~~ Similarly, for the protocol in this tutorial, there is no difference in the definitions of data types: ~~~cpp using WFTutorialTask = WFNetworkTask; using tutorial_callback_t = std::function; using WFTutorialServer = WFServer; using tutorial_process_t = std::function; ~~~ # server There is no difference between this server and an ordinary HTTP server. We give priority to IPv6 startup, which does not affect the client requests in IPv4. In addition, the maximum request size is limited to 4KB. Please see [server.cc](/tutorial/tutorial-10-user_defined_protocol/server.cc) for the complete code. # client The logic of the client is to receive the user input from standard IO, construct a request, send it to the server and get the results. Here we use WFRepeaterTask to implement the repeating process, terminates if the user's input is empty. For the sake of security, we limit the packet size of the server reply to 4KB. The only thing that a client needs to know is how to generate a client task on a user-defined protocol. There are three interface options in [WFTaskFactory.h](/src/factory/WFTaskFactory.h): ~~~cpp template class WFNetworkTaskFactory { private: using T = WFNetworkTask; public: static T *create_client_task(TransportType type, const std::string& host, unsigned short port, int retry_max, std::function callback); static T *create_client_task(TransportType type, const std::string& url, int retry_max, std::function callback); static T *create_client_task(TransportType type, const ParsedURI& uri, int retry_max, std::function callback); static T *create_client_task(TransportType type, const struct sockaddr *addr, socklen_t addrlen, int retry_max, std::function callback); ... }; ~~~ Among them, TransportType specifies the transport layer protocol, and the current options include TT\_TCP, TT\_UDP, TT\_SCTP, TT\_TCP\_SSL and TT\_SCTP\_SSL. There is little difference between the interfaces. In our example, the URL is not needed for the time being. We use a domain name and a port to create a task. The actual code is shown as follows. We inherited the WFTaskFactory class, but this derivation is not required. ~~~cpp using namespace protocol; class MyFactory : public WFTaskFactory { public: static WFTutorialTask *create_tutorial_task(const std::string& host, unsigned short port, int retry_max, tutorial_callback_t callback) { using NTF = WFNetworkTaskFactory; WFTutorialTask *task = NTF::create_client_task(TT_TCP, host, port, retry_max, std::move(callback)); task->set_keep_alive(30 * 1000); return task; } }; ~~~ You can see that we used the WFNetworkTaskFactory\ class to create a client task. Next, by calling the **set\_keep\_alive()** interface of the task, the connection is kept for 30 seconds after the communication is completed. Otherwise, the short connection will be used by default. The previous examples have explained the knowledge in other codes of the above client. Please see [client.cc](/tutorial/tutorial-10-user_defined_protocol/client.cc). # How is the request on an built-in protocol generated Currently, there are five built-in protocols in the framework: HTTP, Redis, MySQL, Kafka and DNS. Can we generate an HTTP or Redis task in the same way? For example: ~~~cpp WFHttpTask *task = WFNetworkTaskFactory::create_client_task(...); ~~~ Please note that an HTTP task generated in this way will lose a lot of functions. For example, it is impossible to identify whether to use persistent connection according to the header, and it is impossible to identify redirection, etc. Similarly, if a MySQL task is generated in this way, it may not run at all, because there is no login authentication process. A Kafka request may need to have complicated interactions with multiple brokers, so the request created in this way obviously cannot complete this process. This shows that the generation of one message in each built-in protocol is far more complicated than that in this example. Similarly, if you need to implement a communication protocol with more functions, there are still many codes to write. --- ### En Tutorial 11 Graph Task (`docs/en/tutorial-11-graph_task.md`) # Direct Acyclic Graph (DAG)：graph_task # Sample code [tutorial-11-graph_task.cc](/tutorial/tutorial-11-graph_task.cc) # About graph_task The graph_task example demonstrates how to implement more complex inter-task dependencies by building a DAG. # Create tasks in the DAG In this tutorial, we create a timer task, two http fetching task, and a 'go' task. Timer task executes a delay of 1 second before fetching, http tasks fetch the home page of 'sogou' and 'baidu' in parallel, and after all of that, go task will print the fetching result. The dependencies of the tasks are: ~~~ +-------+ +---->| Http1 |-----+ | +-------+ | +-------+ +-v--+ | Timer | | Go | +-------+ +-^--+ | +-------+ | +---->| Http2 |-----+ +-------+ ~~~ # Create the graph task Graph is a kind of task as well. We can create a graph task by this function： ~~~cpp class WFTaskFactory { public: static WFGraphTask *create_graph_task(graph_callback_t callback); ... }; ~~~ The graph is a empty graph after it's creation. Of course you may run an empty graph and will get to it callback immediately. # Create graph nodes We'v got 4 orindary tasks, which can not been added to the graph directly but need to be turned into graph nodes: ~~~cpp { /* Create graph nodes */ WFGraphNode& a = graph->create_graph_node(timer); WFGraphNode& b = graph->create_graph_node(http_task1); WFGraphNode& c = graph->create_graph_node(http_task2); WFGraphNode& d = graph->create_graph_node(go_task); } ~~~ The ``create_graph_node`` interface of WFGraphTask creates a graph node that refers to a task. And we can use the references of graph nodes to specify the dependencies of them. Otherwise, they are all standalone nodes, and will run in parallel when the graph task is started. # Build the graph By using the '-->' or '<--' operators, we can specify the dependencies: ~~~cpp { /* Build the graph */ a-->b; a-->c; b-->d; c-->d; } ~~~ And now we'v built the graph that we described. And we can use it like an orindary task. Also, any of the following codes is legal and equivalent: ~~~cpp { a-->b-->d; a-->c-->d; } ~~~ ~~~cpp { d<--b<--a; d<--c<--a; } ~~~ ~~~cpp { d<--b<--a-->c-->d; } ~~~ # Canceling successors In graph tasks, we extend SeriesWork's **cancel** operation. When the series of a graph node is canceled, the operation will apply on all it's successive nodes recursively. The **cancel** operation is usually used in a task's callback: ~~~cpp int main() { WFGraphTask *graph = WFTaskFactory::create_graph_task(graph_callback); WFHttpTask *task = WFTaskFactory::create_http_task(url, 0, 0, [](WFHttpTask *t){ if (t->get_state() != WFT_STATE_SUCCESS) series_of(t)->cancel(); }); WFGraphNode& a = graph->create_graph_node(task); WFGraphNode& b = ...; WFGraphNode& c = ...; WFGraphNode& d = ...; a-->b-->c; b-->d; graph->start(); ... } ~~~ In this case, when http task failed, nodes b, c, d will all be canceled, because the operation is recursive. # Data passing Because the tasks in a graph don't share a same series, there is no general method for passing data between graph nodes. # Acknowledgement Some designs are inspired by [taskflow](https://github.com/taskflow/taskflow). --- ### En Tutorial 12 Mysql Cli (`docs/en/tutorial-12-mysql_cli.md`) # Asynchronous MySQL client: mysql\_cli # Sample code [tutorial-12-mysql\_cli.cc](/tutorial/tutorial-12-mysql_cli.cc) # About mysql\_cli The usage of mysql\_cli in the tutorial is similar to that of the official client. It is an asynchronous MySQL client with an interactive command line interface. To start the program, run the command: ./mysql_cli \ After startup, you can directly enter MySQL command in the terminal to interact with db, or enter `quit` or `Ctrl-C` to exit. # Format of MySQL URL mysql://username:password@host:port/dbname?character\_set=charset&character\_set\_results=charset - set scheme to be **mysqls://** for accessing MySQL with SSL connnection (MySQL server 5.7 or above is required). - fill in the username and the password for the MySQL database; Special characters in password need to be escaped. ~~~cpp // Password: @@@@#### std::string url = "mysql://root:" + StringUtil::url_encode_component("@@@@####") + "@127.0.0.1"; ~~~ - the default port number is 3306; - **dbname** is the name of the database to be used. It is recommended to provide a dbname if SQL statements only operates on one database; - If you have upstream selection requirements for MySQL, please see [upstream documents](/docs/en/about-upstream.md). - **character_set** indicates a character set used for the client, with the same meaning of --default-character-set in official client. The default value is utf8. For details, please see official MySQL documents [character-set.html](https://dev.mysql.com/doc/internals/en/character-set.html). - **character_set_results** indicates a character set for client, connection and results. If you wants to use `SET NAMES ` in SQL statements, please set it here. Sample MySQL URL: mysql://root:password@127.0.0.1 mysql://@test.mysql.com:3306/db1?character\_set=utf8&character_set_results=utf8 mysqls://localhost/db1?character\_set=big5 # Creating and starting a MySQL task You can use WFTaskFactory to create a MySQL task. The usage of creating interface and callback functions are similar to other tasks in workflow: ~~~cpp using mysql_callback_t = std::function; WFMySQLTask *create_mysql_task(const std::string& url, int retry_max, mysql_callback_t callback); void set_query(const std::string& query); ~~~ You can call **set\_query()** on the request to write SQL statements after creating a WFMySQLTask. If **set_query()** had **NOT** been called before the task started, the user might get **WFT_ERR_MYSQL_QUERY_NOT_SET** in callback. Other functions, including callback, series and user\_data are used in a way similar to other tasks in workflow. The following codes show some general usage: ~~~cpp int main(int argc, char *argv[]) { ... WFMySQLTask *task = WFTaskFactory::create_mysql_task(url, RETRY_MAX, mysql_callback); task->get_req()->set_query("SHOW TABLES;"); ... task->start(); ... } ~~~ # Supported commands Currently the supported command is **COM\_QUERY**, which can cover the basic requirements for adding, deleting, modifying and querying data, creating and deleting databases, creating and deleting tables, prepare, using stored procedures and using transactions. Because the program doesn't support the selection of databases (**USE** command) in our interactive commands, if there are **cross-database** operations in SQL statements, you can specify the database and table with **db\_name.table\_name**. Any other command can be **spliced together** and then passed to WFMySQLTask with `set_query()`. (including INSERT/UPDATE/SELECT/PREPARE/CALL) The spliced commands will be executed sequentially until an error occurs, and the previous commands will be executed successfully. For example: ~~~cpp req->set_query("SELECT * FROM table1; CALL procedure1(); INSERT INTO table3 (id) VALUES (1);"); ~~~ # Parsing results Similar to other tasks in workflow, you can use **task->get\_resp()** to get **MySQLResponse**. For details on the interfaces, please see [MySQLResult.h](/src/protocol/MySQLResult.h). One request will get one response, which is a 3-dimensional structure. - one response consists of one or more result sets; - the type of each result set may be **MYSQL_STATUS_GET_RESULT** or **MYSQL_STATUS_OK**; - one result set of type **MYSQL_STATUS_GET_RESULT** consists of one ore more rows; - one row consists of one or more fields, or data cells; The two types of result sets can be judged by ``cursor->get_cursor_status()``. | |MYSQL_STATUS_GET_RESULT|MYSQL_STATUS_OK| |------|-----------------------|---------------| |SQL command|SELECT(including each SELECT in PROCEDURE)|INSERT / UPDATE / DELETE / ...| |Semantics|Read. One result set consists of a 2-dimensional structure
reprecenting the response of one read operation.|Write. One result set reprecents the results of
one write operation.| |main APIs|fetch_fields();
fetch_row(&row_arr);
...|get_insert_id();
get_affected_rows();
...| When errors occur in spliced commands, you may first get the multiple result sets through **MySQLResultCursor**, the commands which have been executed successfully. Then determine whether ``resp->get_packet_type()`` equals to **MYSQL_PACKET_ERROR** and get the specific error information through ``resp->get_error_code()`` and ``resp->get_error_msg()``. A **PROCEDURE** command containing N **SELECT** statements will return N result sets of **MYSQL_STATUS_GET_RESULT** and 1 result set of **MYSQL_STATUS_OK**. The user ignores this **MYSQL_STATUS_OK** result set is fine. To get all the data, the specific steps should be: 1. checking the task state (state at communication): you can check whether the task is successfully executed by checking whether ``task->get_state()`` is equal to **WFT_STATE_SUCCESS**; 2. determining the type of the response packet (state at parsing the return packet): call ``resp->get_packet_type()`` to check the type of the last SQL query return packet. The common types include: - MYSQL\_PACKET\_OK: parsed successfully, should use cursor to get all the result sets. - MYSQL\_PACKET\_EOF: parsed successfully, should use cursor to get all the result sets. - MYSQL\_PACKET\_ERROR: requests: failed or partial failed, may use cursor to get the result sets of those successful commands. 3. traverse the result sets: you can use **MySQLResultCursor** to read the content in the result set. Because the data returned by a MySQL server contains multiple result sets, the cursor will **automatically point to the reading position of the first result set** at first. 4. checking the result set state (state at reading the result sets): **cursor->get_cursor_status()** returns the following states: - MYSQL\_STATUS\_GET\_RESULT: current result set is a READ result set; - MYSQL\_STATUS\_END: the last record of the current READ result set has been read; - MYSQL\_STATUS\_OK: current result set is a WRITE result set; - MYSQL\_STATUS\_ERROR: parsing error; 5. reading the basic content of **MYSQL_STATUS_OK** result set: - ``unsigned long long get_affected_rows() const;`` - ``unsigned long long get_insert_id() const;`` - ``int get_warnings() const;`` - ``std::string get_info() const;`` 6. reading each field and each columns of **MYSQL_STATUS_GET_RESULT** result set: - `int get_field_count() const;` - `const MySQLField *fetch_field();` - `const MySQLField *const *fetch_fields() const;` 7. reading each line of **MYSQL_STATUS_GET_RESULT** result set: you can use ``cursor->fetch_row()`` to read by row until the return value is false, in which the offset within the cursor that points to the row in the current result set will be moved: - `int get_rows_count() const;` - `bool fetch_row(std::vector& row_arr);` - `bool fetch_row(std::map& row_map);` - `bool fetch_row(std::unordered_map& row_map);` - `bool fetch_row_nocopy(const void **data, size_t *len, int *data_type);` 8. taking out all the rows in the current **MYSQL_STATUS_GET_RESULT** result set directly: you can use ``cursor->fetch_all()`` to read all rows, and the cursor that is used to record the rows internally will be moved directly to the end; The cursor state changes to **MYSQL_STATUS_END**: - `bool fetch_all(std::vector>& rows);` 9. returning to the head of the current **MYSQL_STATUS_GET_RESULT** result set: if it is necessary to read this result set again, you can use ``cursor->rewind()`` to return to the head of the current result set, and then read it via the operations in Step 7 or Step 8; 10. getting the next result set: because the data packet returned by MySQL server may contains multiple result sets (for example, each SELECT/INSERT/... statement gets one result set; or the multiple result sets returned by calling a PROCEDURE). Therefore, you can use ``cursor->next_result_set()`` to jump to the next result set. If the return value is false, it means that all result sets have been taken. 11. returning to the first result set: use **cursor->first\_result\_set()** to return to the heads of all result sets, and then you can repeat the operations from Step 4. 12. getting the data of each column (MySQLCell): the row read in Step 5 is composed of multiple columns, and the result of each column is one MySQLCell. It mainly uses the following interfaces: - `int get_data_type();` returns MYSQL\_TYPE\_LONG, MYSQL\_TYPE\_STRING, and etc. For the details, please see [mysql\_types.h](/src/protocol/mysql_types.h). - `bool is_TYPE() const;` the TYPE is int, string or ulonglong. It is used to check the data type. - `TYPE as_TYPE() const;` same as the above. It reads the data from MySQLCell in a certain type. - `void get_cell_nocopy(const void **data, size_t *len, int *data_type) const;` nocopy interface. The whole example is shown below: ~~~cpp void task_callback(WFMySQLTask *task) { // step-1. Check the status of the task if (task->get_state() != WFT_STATE_SUCCESS) { fprintf(stderr, "task error = %d\n", task->get_error()); return; } MySQLResultCursor cursor(task->get_resp()); bool test_first_result_set_flag = false; bool test_rewind_flag = false; // step-2. Check other status of repsponse packet if (resp->get_packet_type() == MYSQL_PACKET_ERROR) { fprintf(stderr, "ERROR. error_code=%d %s\n", task->get_resp()->get_error_code(), task->get_resp()->get_error_msg().c_str()); } begin: // step-3. Traverse the result sets do { // step-4. Check the status of the result set if (cursor.get_cursor_status() == MYSQL_STATUS_OK) { // step-5. Read the basic content of MYSQL_STATUS_OK result set fprintf(stderr, "OK. %llu rows affected. %d warnings. insert_id=%llu.\n", cursor.get_affected_rows(), cursor.get_warnings(), cursor.get_insert_id()); } else if (cursor.get_cursor_status() == MYSQL_STATUS_GET_RESULT) { fprintf(stderr, "field_count=%u rows_count=%u ", cursor.get_field_count(), cursor.get_rows_count()); // step-6. Read each fields. This is a nocopy api const MySQLField *const *fields = cursor.fetch_fields(); for (int i = 0; i < cursor.get_field_count(); i++) { fprintf(stderr, "db=%s table=%s name[%s] type[%s]\n", fields[i]->get_db().c_str(), fields[i]->get_table().c_str(), fields[i]->get_name().c_str(), datatype2str(fields[i]->get_data_type())); } // step-8. Read all the rows. You may use while (cursor.fetch_row(map/vector)) to get each rows accoding to step-7 std::vector> rows; cursor.fetch_all(rows); for (unsigned int j = 0; j < rows.size(); j++) { // step-12. Read each cell for (unsigned int i = 0; i < rows[j].size(); i++) { fprintf(stderr, "[%s][%s]", fields[i]->get_name().c_str(), datatype2str(rows[j][i].get_data_type())); // step-12. Check the type wih is_string()and transform the type with as_string() if (rows[j][i].is_string()) { std::string res = rows[j][i].as_string(); fprintf(stderr, "[%s]\n", res.c_str()); } else if (rows[j][i].is_int()) { fprintf(stderr, "[%d]\n", rows[j][i].as_int()); } // else if ... } } } // step-10. Get the next result set } while (cursor.next_result_set()); if (test_first_result_set_flag == false) { test_first_result_set_flag = true; // step-11. Go back to the first result set cursor.first_result_set(); goto begin; } if (test_rewind_flag == false) { test_rewind_flag = true; // step-9. Go back to the first position of the current result set cursor.rewind(); goto begin; } return; } ~~~ # WFMySQLConnection Since it is a highly concurrent asynchronous client, this means that the client may have more than one connection to the server. As both MySQL transactions and preparation are stateful, in order to ensure that one transaction or preparation ocupies one connection exclusively, you can use our encapsulated secondary factory WFMySQLConnection to create a task. Each WFMySQLConnection guarantees that one connection is occupied exclusively. For the details, please see [WFMySQLConnection.h](/src/client/WFMySQLConnection.h). ### 1\. Creating and initializing WFMySQLConnection When creating a WFMySQLConnection, you need to pass in **id**, and the subsequent calls on this WFMySQLConnection will use this id and url to find the corresponding unique connection. When initializing a WFMySQLConnection, you need to pass a URL, and then you do not need to set the URL for the task created on this connection. ~~~cpp class WFMySQLConnection { public: WFMySQLConnection(int id); int init(const std::string& url); ... }; ~~~ ### 2\. Creating a task and closing a connection With **create\_query\_task()**, you can create a task by writing an SQL request and a callback function. The task is garuanteed to be sent on this connection. Sometimes you need to close this connection manually. Because when you stop using it, this connection will be kept until MySQL server time out. During this period, if you use the same id and url to create a WFMySQLConnection, you may reuse the connection. Therefore, we suggest that if you do not want to reuse the connection, you should use **create\_disconnect\_task()** to create a task and manually close the connection. ~~~cpp class WFMySQLConnection { public: ... WFMySQLTask *create_query_task(const std::string& query, mysql_callback_t callback); WFMySQLTask *create_disconnect_task(mysql_callback_t callback); } ~~~ WFMySQLConnection is equivalent to a secondary factory. In the framework, we arrange that the life cycle of any factory object does not need to be maintained until the task ends. The following code is completely legal: ~~~cpp WFMySQLConnection *conn = new WFMySQLConnection(1234); conn->init(url); auto *task = conn->create_query_task("SELECT * from table", my_callback); conn->deinit(); delete conn; task->start(); ~~~ ### 3\. Cautions Do not generate new connection id infinitely, becauase easy id will occupy a little memory. The connection will be put to an internal connection pool if user didn't run a disconnect task, and will been reused by another WFMySQLConnection object initialized with the same id and url. Running tasks on the same connection parallelly will fail with error **EAGAIN**. If you have started `BEGIN` but have not `COMMIT` or `ROLLBACK` during the transaction and the connection has been interrupted during the transaction, the connection will be automatically reconnected internally by the framework, and you will get **ECONNRESET** error in the next task request. In this case, the transaction statements those have not been `COMMIT` would be expired and you may need to send them again. ### 4\. Preparation You can also use the WFMySQLConnection for **PREPARE**. And you can easily use it to **defend against SQL injection**. If the connection is reconnected, you also get an **ECONNRESET** error. ### 5\. Complete example ~~~cpp WFMySQLConnection conn(1); conn.init("mysql://root@127.0.0.1/test"); // test transaction const char *query = "BEGIN;"; WFMySQLTask *t1 = conn.create_query_task(query, task_callback); query = "SELECT * FROM check_tiny FOR UPDATE;"; WFMySQLTask *t2 = conn.create_query_task(query, task_callback); query = "INSERT INTO check_tiny VALUES (8);"; WFMySQLTask *t3 = conn.create_query_task(query, task_callback); query = "COMMIT;"; WFMySQLTask *t4 = conn.create_query_task(query, task_callback); WFMySQLTask *t5 = conn.create_disconnect_task(task_callback); SeriesWork *series = create_series_work(t1, nullptr); *series << t2 << t3 << t4 << t5; series->start(); ~~~ --- ### En Tutorial 13 Kafka Cli (`docs/en/tutorial-13-kafka_cli.md`) # Asynchronous Kafka Client: kafka_cli # Sample Codes [tutorial-13-kafka_cli.cc](/tutorial/tutorial-13-kafka_cli.cc) # About Compiler Because of supporting multiple compression methods of Kafka, [zlib](https://github.com/madler/zlib.git), [snappy](https://github.com/google/snappy.git), [lz4(>=1.7.5)](https://github.com/lz4/lz4.git), [zstd](https://github.com/facebook/zstd.git) and other third-party libraries are used in the compression algorithms in the Kafka protocol, and they must be installed before the compilation. It supports both CMake and Bazel for compiling. CMake: You can use **make KAFKA=y** to compile a separate library for Kafka protocol(libwfkafka.a和libwfkafka.so) and use **cd tutorial; make KAFKA=y** to compile kafka_cli. Bazel: You can use **bazel build kafka** to compile a separate library for Kafka protocol and use **bazel build kafka_cli** to compile kafka_cli. # About kafka_cli Kafka_cli is a kafka client for producing and fetching messages in Kafka. When you compile the source codes, type the command **make KAFKA=y** in the **tutorial** directory or type the command **make KAFKA=y tutorial** in the root directory of the project. The program then reads kafka broker server addresses and the current task type (produce/fetch) from the command line: ./kafka_cli \ [p/c] The program exists automatically after all the tasks are completed, and all the resources will be completedly freed. In the command, the broker_url may contain several urls seperated by comma(,). - For instance, kafka://host:port,kafka://host1:port... or: **kafkas**://host:port,**kafkas**://host1:port for kafka over SSL; - The default port is 9092 for TCP and 9093 for SSL; - Do not mix 'kafkas://' with "kafka://", otherwise the init function will fail with errno EINVAL; - If you want to use upstream policy at this layer, please refer to [upstream documents](/docs/en/about-upstream.md). The following are several Kafka broker_url samples: kafka://127.0.0.1/ kafka://kafka.host:9090/ kafka://10.160.23.23:9000,10.123.23.23,kafka://kafka.sogou kafkas://broker1.kafka.sogou,kafkas://broker2.kafka.sogou Illegal broker_url sample (The first one is SSL, and the second one is not): kafkas://broker1.kafka.sogou,broker2.kafka.sogou # Principles and Features Kafka client has no third-party dependencies internally except for the libraries used in the compression. With the high performance of Workflow, When properly configured and in fair environments, tens of thousands of Kafka requests can be processed in one second. Internally, a Kafka client divides each request into parallel tasks according to the brokers used. In parallel tasks, there is one sub-task for each broker address. In this way, the efficiency is maximized. Besides, the connection reuse mechanism in the Workflow ensures that the total number of connections is kept within a reasonable range. If there are multiple topic partitions under one broker address, you may create multiple clients and then create and start separate tasks for each topic partition to increase the throughput. # Creating and Starting Kafka Tasks To create and start a Kafka task, create a **WFKafkaClient** first and then call **init** to initialize that **WFKafkaClient**. ~~~cpp int init(const std::string& broker_url); int init(const std::string& broker_url, const std::string& group); ~~~ In the above code snippet, **broker_url** means the address of the kafka broker cluster. Its format is the same as the broker_url in the above section. **group** means the group_name of a consumer group, which is used for the consumer group in a fetch task. In the case of produce tasks or fetch tasks without any consumer groups, do not use this interface. For a consumer group, you can specify the heartbeat interval in milliseconds to keep the heartbeats. ~~~cpp void set_heartbeat_interval(size_t interval_ms); ~~~ Then you can create a Kafka task with that **WFKafkaClient**. ~~~cpp using kafka_callback_t = std::function; WFKafkaTask *create_kafka_task(const std::string& query, int retry_max, kafka_callback_t cb); WFKafkaTask *create_kafka_task(int retry_max, kafka_callback_t cb); ~~~ In the above code snippet, **query** includes the type of the task, the topic and other properties. **retry_max** means the maximum number of retries. **cb** is the user-defined callback function, which will be called after the task is completed. You can also change the default settings of the task to meet the requirements. For details, refer to [KafkaDataTypes.h](/src/protocol/KafkaDataTypes.h). ~~~cpp KafkaConfig config; config.set_client_id("workflow"); task->set_config(std::move(config)); ~~~ The supported configuration items are described below: Item name | Type | Default value | Description ------ | ---- | -------| ------- produce_timeout | int | 100ms | Maximum time for produce. produce_msg_max_bytes | int | 1000000 bytes | Maximum length for one message. produce_msgset_cnt | int | 10000 | Maximun numbers of messges in one communication set produce_msgset_max_bytes | int | 1000000 bytes | Maximum length of messages in one communication. fetch_timeout | int | 100ms | Maximum timeout for fetch. fetch_min_bytes | int | 1 byte | Minimum length of messages in one fetch communication. fetch_max_bytes | int | 50M bytes | Maximum length of messages in one fetch communication. fetch_msg_max_bytes | int | 1M bytes | Maximum length of one single message in a fetch communication. offset_timestamp | long long int | -1 | Initialized offfset in the consumer group mode when there is no offset history. -2 means the oldest offset; -1 means the latest offset. session_timeout | int | 10s | Maximum initialization timeout for joining a consumer group. rebalance_timeout | int | 10s | Maximum timeout for synchronizing a consumer group information after a client joins the consumer group. produce_acks | int | -1 | Number of brokers to ensure the successful replication of a message before the return of a produce task. -1 indicates all replica brokers. allow_auto_topic_creation | bool | true | Flag for controlling whether a topic is created automatically for the produce task if it does not exist. broker_version | char * | NULL | Version number for brokers, which should be manually specified when the version number is smaller than 0.10. compress_type | int | NoCompress | Compression type for produce messages. client_id | char * | NULL | Identifier of a client. check_crcs | bool | false | Flag for controlling whether to check crc32 in the messages for a fetch task. offset_store | int | 0 | When joining the consumer group, whether to use the last submission offset, 1 means to use the specified offset, and 0 means to use the last submission preferentially. sasl_mechanisms | char * | NULL | Sasl certification type, currently only supports plain, and is on the ongoing development of sasl support. sasl_username | char * | NULL | Username required for sasl authentication. sasl_password | char * | NULL | Password required for sasl authentication. After configuring all the parameters, you can call **start** interface to start the Kafka task. # About Produce Tasks 1\. After you create and initialize a **WFKafkaClient**, you can specify the topic or other information in the **query** to create **WFKafkaTask** tasks. For example: ~~~cpp int main(int argc, char *argv[]) { ... client = new WFKafkaClient(); client->init(url); task = client->create_kafka_task("api=fetch&topic=xxx&topic=yyy", 3, kafka_callback); ... task->start(); ... } ~~~ 2\. After the **WFKafkaTask** is created, call **set_key**, **set_value**, **add_header_pair** and other methods to build a **KafkaRecord**. For information about more interfaces on **KafkaRecord**, refer to [KafkaDataTypes.h](/src/protocol/KafkaDataTypes.h). Then you can call **add_produce_record** to add a **KafkaRecord**. For the detailed definitions of the interfaces, refer to [WFKafkaClient.h](/src/client/WFKafkaClient.h). The second parameter **partition** in **add_produce_record**, >=0 means the specified **partition**; -1 means that the **partition** is chosen randomly or the user-defined **kafka_partitioner_t** is used. For **kafka_partitioner_t**, you can call the **set_partitioner** interface to specify the user-defined rules. For example: ~~~cpp int main(int argc, char *argv[]) { ... WFKafkaClient *client_fetch = new WFKafkaClient(); client_fetch->init(url); task = client_fetch->create_kafka_task("api=produce&topic=xxx&topic=yyy", 3, kafka_callback); task->set_partitioner(partitioner); KafkaRecord record; record.set_key("key1", strlen("key1")); record.set_value(buf, sizeof(buf)); record.add_header_pair("hk1", 3, "hv1", 3); task->add_produce_record("workflow_test1", -1, std::move(record)); ... task->start(); ... } ~~~ 3\. You can use one of the four compressions supported by Kafka in the produce task by configuration. For example: ~~~cpp int main(int argc, char *argv[]) { ... WFKafkaClient *client_fetch = new WFKafkaClient(); client_fetch->init(url); task = client_fetch->create_kafka_task("api=produce&topic=xxx&topic=yyy", 3, kafka_callback); KafkaConfig config; config.set_compress_type(Kafka_Zstd); task->set_config(std::move(config)); KafkaRecord record; record.set_key("key1", strlen("key1")); record.set_value(buf, sizeof(buf)); record.add_header_pair("hk1", 3, "hv1", 3); task->add_produce_record("workflow_test1", -1, std::move(record)); ... task->start(); ... } ~~~ # About Fetch Tasks You may use consumer group mode or manual mode for fetch tasks. 1\. Manual mode In this mode, you do not need to specify consumer groups, but you must specify topic, partition and offset. For example: ~~~cpp client = new WFKafkaClient(); client->init(url); task = client->create_kafka_task("api=fetch", 3, kafka_callback); KafkaToppar toppar; toppar.set_topic_partition("workflow_test1", 0); toppar.set_offset(0); task->add_toppar(toppar); ~~~ 2\. Consumer group mode In this mode, you must specify the name of the consumer group when initializing a client. For example: ~~~cpp int main(int argc, char *argv[]) { ... WFKafkaClient *client_fetch = new WFKafkaClient(); client_fetch->init(url, cgroup_name); task = client_fetch->create_kafka_task("api=fetch&topic=xxx&topic=yyy", 3, kafka_callback); ... task->start(); ... } ~~~ 3\. Committing offset In the consumer group mode, after a message is consumed, you can create a commit task in the callback to automatically submit the consumption record. For example: ~~~cpp void kafka_callback(WFKafkaTask *task) { ... commit_task = client.create_kafka_task("api=commit", 3, kafka_callback); ... commit_task->start(); ... } ~~~ # Closing the Client In the consumer group mode, before you close a client, you must call **create_leavegroup_task** to create a **leavegroup_task**. This task will send a **leavegroup** packet. If no **leavegroup_task** is started, the group does not know that the client is leaving and will trigger rebalance. # Processing Kafka Results The data structure of the message result set is KafkaResult, and you can call **get_result()** in the **WFKafkaTask** to retrieve the results. Then you can call the **fetch_record** in the **KafkaResult** to retrieve all records of the task. The record is a two-dimensional vector. The first dimension is topic partition, and the second dimension is the **KafkaRecord** under that topic partition. [KafkaResult.h](/src/protocol/KafkaResult.h) contains the definition of **KafkaResult**. ~~~cpp void kafka_callback(WFKafkaTask *task) { int state = task->get_state(); int error = task->get_error(); // handle error states ... protocol::KafkaResult *result = task->get_result(); result->fetch_records(records); for (auto &v : records) { for (auto &w: v) { const void *value; size_t value_len; w->get_value(&value, &value_len); printf("produce\ttopic: %s, partition: %d, status: %d, offset: %lld, val_len: %zu\n", w->get_topic(), w->get_partition(), w->get_status(), w->get_offset(), value_len); } } ... protocol::KafkaResult new_result = std::move(*task->get_result()); if (new_result.fetch_records(records)) { for (auto &v : records) { if (v.empty()) continue; for (auto &w: v) { if (fp) { const void *value; size_t value_len; w->get_value(&value, &value_len); fwrite(w->get_value(), w->get_value_len(), 1, fp); } } } } ... } ~~~ --- ### En Xmake (`docs/en/xmake.md`) # xmake compiling ``` // compile workflow library xmake // compile test xmake -g test // run test xmake run -g test // compile tutorial xmake -g tutorial // compile benchmark xmake -g benchmark ``` ## running `xmake run -h` can see which targets you can run Select a target to run, for instance: ``` xmake run tutorial-06-parallel_wget ``` ## xmake install ``` sudo xmake install ``` ## Compile static / shared library ``` // compile static lib xmake f -k static xmake -r ``` ``` // compile shard lib xmake f -k shared xmake -r ``` `tips : -r means -rebuild` ## build options `xmake f --help` can see our defined options. ``` Command options (Project Configuration): --workflow_inc=WORKFLOW_INC workflow inc (default: /media/psf/pro/workflow/_include) --upstream=[y|n] build upstream component (default: y) --consul=[y|n] build consul component --workflow_lib=WORKFLOW_LIB workflow lib (default: /media/psf/pro/workflow/_lib) --redis=[y|n] build redis component (default: y) --kafka=[y|n] build kafka component --mysql=[y|n] build mysql component (default: y) ``` You can cut or integrate each components with the following commands ``` xmake f --redis=n --kafka=y --mysql=n xmake -r ``` --- --- METRICS --- - Files Extracted: 67 - Estimated Token Budget: ~99427 tokens - Canonical Reference: https://codewiki.google/github.com/sogou/workflow