haproxy Documentation

Doc/Design Thoughts/Binding Possibilities

2013/10/10 - possibilities for setting source and destination addresses

When establishing a connection to a remote device, this device is designated
as a target, which designates an entity defined in the configuration. A same
target appears only once in a configuration, and multiple targets may share
the same settings if needed.

The following types of targets are currently supported :

- listener : all connections with this type of target come from clients ;
- server : connections to such targets are for "server" lines ;
- peer : connections to such target address "peer" lines in "peers"
sections ;
- proxy : these targets are used by "dispatch", "option transparent"
or "option http_proxy" statements.

A connection might not be reused between two different targets, even if all
parameters seem similar. One of the reason is that some parameters are specific
to the target and are not easy or not cheap to compare (eg: bind to interface,
mss, ...).

A number of source and destination addresses may be set for a given target.

- listener :
- the "from" address:port is set by accept()

- the "to" address:port is set if conn_get_to_addr() is called

- peer :
- the "from" address:port is not set

- the "to" address:port is static and dependent only on the peer

- server :
- the "from" address may be set alone when "source" is used with
a forced IP address, or when "usesrc clientip" is used.

- the "from" port may be set only combined with the address when
"source" is used with IP:port, IP:port-range or "usesrc client" is
used. Note that in this case, both the address and the port may be
0, meaning that the kernel will pick the address or port and that
the final value might not match the one explicitly set (eg:
important for logging).

- the "from" address may be forced from a header which implies it
may change between two consecutive requests on the same connection.

- the "to" address and port are set together when connecting to a
regular server, or by copying the client's IP address when
"server 0.0.0.0" is used. Note that the destination port may be
an offset applied to the original destination port.

- proxy :
- the "from" address may be set alone when "source" is used with a
forced IP address or when "usesrc clientip" is used.

- the "from" port may be set only combined with the address when
"source" is used with IP:port or with "usesrc client". There is
no ip:port range for a proxy as of now. Same comment applies as
above when port and/or address are 0.

- the "from" address may be forced from a header which implies it
may change between two consecutive requests on the same connection.

- the "to" address and port are set together, either by configuration
when "dispatch" is used, or dynamically when "transparent" is used
(1:1 with client connection) or "option http_proxy" is used, where
each client request may lead to a different destination address.

At the moment, there are some limits in what might happen between multiple
concurrent requests to a same target.

- peers parameter do not change, so no problem.

- server parameters may change in this way :
- a connection may require a source bound to an IP address found in a
header, which will fall back to the "source" settings if the address
is not found in this header. This means that the source address may
switch between a dynamically forced IP address and another forced
IP and/or port range.

- if the element is not found (eg: header), the remaining "forced"
source address might very well be empty (unset), so the connection
reuse is acceptable when switching in that direction.

- it is not possible to switch between client and clientip or any of
these and hdr_ip() because they're exclusive.

- using a source address/port belonging to a port range is compatible
with connection reuse because there is a single range per target, so
switching from a range to another range means we remain in the same
range.

- destination address may currently not change since the only possible
case for dynamic destination address setting is the transparent mode,
reproducing the client's destination address.

- proxy parameters may change in this way :
- a connection may require a source bound to an IP address found in a
header, which will fall back to the "source" settings if the address
is not found in this header. This means that the source address may
switch between a dynamically forced IP address and another forced
IP and/or port range.

- if the element is not found (eg: header), the remaining "forced"
source address might very well be empty (unset), so the connection
reuse is acceptable when switching in that direction.

- it is not possible to switch between client and clientip or any of
these and hdr_ip() because they're exclusive.

- proxies do not support port ranges at the moment.

- destination address might change in the case where "option http_proxy"
is used.

So, for each source element (IP, port), we want to know :
- if the element was assigned by static configuration (eg: ":80")
- if the element was assigned from a connection-specific value (eg: usesrc clientip)
- if the element was assigned from a configuration-specific range (eg: 1024-65535)
- if the element was assigned from a request-specific value (eg: hdr_ip(xff))
- if the element was not assigned at all

For the destination, we want to know :
- if the element was assigned by static configuration (eg: ":80")
- if the element was assigned from a connection-specific value (eg: transparent)
- if the element was assigned from a request-specific value (eg: http_proxy)

We don't need to store the information about the origin of the dynamic value
since we have the value itself. So in practice we have :
- default value, unknown (not yet checked with getsockname/getpeername)
- default value, known (check done)
- forced value (known)
- forced range (known)

We can't do that on an ip:port basis because the port may be fixed regardless
of the address and conversely.

So that means :

enum {
CO_ADDR_NONE = 0, / not set, unknown value /
CO_ADDR_KNOWN = 1, / not set, known value /
CO_ADDR_FIXED = 2, / fixed value, known /
CO_ADDR_RANGE = 3, / from assigned range, known /
} conn_addr_values;

unsigned int new_l3_src_status:2;
unsigned int new_l4_src_status:2;
unsigned int new_l3_dst_status:2;
unsigned int new_l4_dst_status:2;

unsigned int cur_l3_src_status:2;
unsigned int cur_l4_src_status:2;
unsigned int cur_l3_dsp_status:2;
unsigned int cur_l4_dst_status:2;

unsigned int new_family:2;
unsigned int cur_family:2;

Note: this obsoletes CO_FL_ADDR_FROM_SET and CO_FL_ADDR_TO_SET. These flags
must be changed to individual l3+l4 checks ORed between old and new values,
or better, set to cur only which will inherit new.

In the connection, these values may be merged in the same word as err_code.

---

Doc/Design Thoughts/Connection Reuse

2015/08/06 - server connection sharing

Improvements on the connection sharing strategies
-------------------------------------------------

4 strategies are currently supported :
- never
- safe
- aggressive
- always

The "aggressive" and "always" strategies take into account the fact that the
connection has already been reused at least once or not. The principle is that
second requests can be used to safely "validate" connection reuse on newly
added connections, and that such validated connections may be used even by
first requests from other sessions. A validated connection is a connection
which has already been reused, hence proving that it definitely supports
multiple requests. Such connections are easy to verify : after processing the
response, if the txn already had the TX_NOT_FIRST flag, then it was not the
first request over that connection, and it is validated as safe for reuse.
Validated connections are put into a distinct list : server->safe_conns.

Incoming requests with TX_NOT_FIRST first pick from the regular idle_conns
list so that any new idle connection is validated as soon as possible.

Incoming requests without TX_NOT_FIRST only pick from the safe_conns list for
strategy "aggressive", guaranteeing that the server properly supports connection
reuse, or first from the safe_conns list, then from the idle_conns list for
strategy "always".

Connections are always stacked into the list (LIFO) so that there are higher
changes to convert recent connections and to use them. This will first optimize
the likeliness that the connection works, and will avoid TCP metrics from being
lost due to an idle state, and/or the congestion window to drop and the
connection going to slow start mode.

Handling connections in pools
-----------------------------

A per-server "pool-max" setting should be added to permit disposing unused idle
connections not attached anymore to a session for use by future requests. The
principle will be that attached connections are queued from the front of the
list while the detached connections will be queued from the tail of the list.

This way, most reused connections will be fairly recent and detached connections
will most often be ignored. The number of detached idle connections in the lists
should be accounted for (pool_used) and limited (pool_max).

After some time, a part of these detached idle connections should be killed.
For this, the list is walked from tail to head and connections without an owner
may be evicted. It may be useful to have a per-server pool_min setting
indicating how many idle connections should remain in the pool, ready for use
by new requests. Conversely, a pool_low metric should be kept between eviction
runs, to indicate the lowest amount of detached connections that were found in
the pool.

For eviction, the principle of a half-life is appealing. The principle is
simple : over a period of time, half of the connections between pool_min and
pool_low should be gone. Since pool_low indicates how many connections were
remaining unused over a period, it makes sense to kill some of them.

In order to avoid killing thousands of connections in one run, the purge
interval should be split into smaller batches. Let's call N the ratio of the
half-life interval and the effective interval.

The algorithm consists in walking over them from the end every interval and
killing ((pool_low - pool_min) + 2 N - 1) / (2 N). It ensures that half
of the unused connections are killed over the half-life period, in N batches
of population/2N entries at most.

Unsafe connections should be evicted first. There should be quite few of them
since most of them are probed and become safe. Since detached connections are
quickly recycled and attached to a new session, there should not be too many
detached connections in the pool, and those present there may be killed really
quickly.

Another interesting point of pools is that when a pool-max is not null, then it
makes sense to automatically enable pretend-keep-alive on non-private connections
going to the server in order to be able to feed them back into the pool. With
the "aggressive" or "always" strategies, it can allow clients making a single
request over their connection to share persistent connections to the servers.

2013/10/17 - server connection management and reuse

Current state
-------------

At the moment, a connection entity is needed to carry any address
information. This means in the following situations, we need a server
connection :

- server is elected and the server's destination address is set

- transparent mode is elected and the destination address is set from
the incoming connection

- proxy mode is enabled, and the destination's address is set during
the parsing of the HTTP request

- connection to the server fails and must be retried on the same
server using the same parameters, especially the destination
address (SN_ADDR_SET not removed)

On the accepting side, we have further requirements :

- allocate a clean connection without a stream interface

- incrementally set the accepted connection's parameters without
clearing it, and keep track of what is set (eg: getsockname).

- initialize a stream interface in established mode

- attach the accepted connection to a stream interface

This means several things :

- the connection has to be allocated on the fly the first time it is
needed to store the source or destination address ;

- the connection has to be attached to the stream interface at this
moment ;

- it must be possible to incrementally set some settings on the
connection's addresses regardless of the connection's current state

- the connection must not be released across connection retries ;

- it must be possible to clear a connection's parameters for a
redispatch without having to detach/attach the connection ;

- we need to allocate a connection without an existing stream interface

So on the accept() side, it looks like this :

fd = accept();
conn = new_conn();
get_some_addr_info(&conn->addr);
...
si = new_si();
si_attach_conn(si, conn);
si_set_state(si, SI_ST_EST);
...
get_more_addr_info(&conn->addr);

On the connect() side, it looks like this :

si = new_si();
while (!properly_connected) {
if (!(conn = si->end)) {
conn = new_conn();
conn_clear(conn);
si_attach_conn(si, conn);
}
else {
if (connected) {
f = conn->flags & CO_FL_XPRT_TRACKED;
conn->flags &= ~CO_FL_XPRT_TRACKED;
conn_close(conn);
conn->flags |= f;
}
if (!correct_dest)
conn_clear(conn);
}
set_some_addr_info(&conn->addr);
si_set_state(si, SI_ST_CON);
...
set_more_addr_info(&conn->addr);
conn->connect();
if (must_retry) {
close_conn(conn);
}
}

Note: we need to be able to set the control and transport protocols.
On outgoing connections, this is set once we know the destination address.
On incoming connections, this is set the earliest possible (once we know
the source address).

The problem analysed below was solved on 2013/10/22

| ==> the real requirement is to know whether a connection is still valid or not
| before deciding to close it. CO_FL_CONNECTED could be enough, though it
| will not indicate connections that are still waiting for a connect to occur.
| This combined with CO_FL_WAIT_L4_CONN and CO_FL_WAIT_L6_CONN should be OK.
|
| Alternatively, conn->xprt could be used for this, but needs some careful checks
| (it's used by conn_full_close at least).
|
| Right now, conn_xprt_close() checks conn->xprt and sets it to NULL.
| conn_full_close() also checks conn->xprt and sets it to NULL, except
| that the check on ctrl is performed within xprt. So conn_xprt_close()
| followed by conn_full_close() will not close the file descriptor.
| Note that conn_xprt_close() is never called, maybe we should kill it ?
|
| Note: at the moment, it's problematic to leave conn->xprt to NULL before doing
| xprt_init() because we might end up with a pending file descriptor. Or at
| least with some transport not de-initialized. We might thus need
| conn_xprt_close() when conn_xprt_init() fails.
|
| The fd should be conditioned by ->ctrl only, and the transport layer by ->xprt.
|
| - conn_prepare_ctrl(conn, ctrl)
| - conn_prepare_xprt(conn, xprt)
| - conn_prepare_data(conn, data)
|
| Note: conn_xprt_init() needs conn->xprt so it's not a problem to set it early.
|
| One problem might be with conn_xprt_close() not being able to know if xprt_init()
| was called or not. That's where it might make sense to only set ->xprt during init.
| Except that it does not fly with outgoing connections (xprt_init is called after
| connect()).
|
| => currently conn_xprt_close() is only used by ssl_sock.c and decides whether
| to do something based on ->xprt_ctx which is set by ->init() from xprt_init().
| So there is nothing to worry about. We just need to restore conn_xprt_close()
| and rely on ->ctrl to close the fd instead of ->xprt.
|
| => we have the same issue with conn_ctrl_close() : when is the fd supposed to be
| valid ? On outgoing connections, the control is set much before the fd...

---

Doc/Design Thoughts/Error Reporting

2024-10-28 - error reporting
----------------------------

- rules:
-> stream->current_rule ~= yielding rule or error
pb: not always set.
-> todo: curr_rule_in_progress points to &rule->conf (file+line)
- set on ACT_RET_ERR, ACT_RET_YIELD, ACT_RET_INV.
- sample_fetch: curr_rule

- filters:
-> strm_flt.filters[2] (1 per direction) ~= yielding filter or error
-> to check: what to do on forward filters (e.g. compression)
-> check spoe / waf (stream data)
-> sample_fetch: curr_filt

- cleanup:
- last_rule_line + last_rule_file can point to &rule->conf

- xprt:
- all handshakes use the dummy xprt "xprt_handshake" ("HS"). No data
exchange is possible there. The ctx is of type xprt_handshake_ctx
for all of them, and contains a wait_event.
=> conn->xprt_ctx->wait_event contains the sub for current handshake
if xprt points to xprt_handshake.
- at most 2 active xprt at once: top and bottom (bottom=raw_sock)

- proposal:
- combine 2 bits for muxc, 2 bits for xprt, 4 bits for fd (active,ready).
=> 8 bits for muxc and below. QUIC uses something different TBD.

- muxs uses 6 bits max (ex: h2 send_list, fctl_list, full etc; h1: full,
blocked connect...).

- 2 bits for sc's sub

- mux_sctl to retrieve a 32-bit code padded right, limited to 16 bits
for now.
=> [ 0000 | 0000 | 0000 | 0000 | SC | MUXS | MUXC | XPRT | FD ]
2 6 2 2 4
- sample-fetch for each side.

- shut / abort
- history, almost human-readable.
- event locations:
- fd (detected by rawsock)
- handshake (detected by xprt_handshake). Eg. parsing or address encoding
- xprt (ssl)
- muxc
- se: muxs / applet
- stream

< 8 total. +8 to distinguish front from back at stream level.
suggest:
- F, H, X, M, E, S front or back
- f, h, x, m, e, s back or front

- event types:
- 0 = no event yet
- 1 = timeout
- 2 = intercepted (rule, etc)
- 3 unused

// shutr / shutw: +1 if other side already shut
- 4 = aligned shutr
- 6 = aligned recv error
- 8 = early shutr (truncation)
- 10 = early error (truncation)
- 12 = shutw
- 14 = send error

- event location = MSB
event type = LSB

appending a single event:
-- if code not full --
code <<= 8;
code |= location << 4;
code |= event type;

- up to 4 events per connection in 32-bit mode stored on connection
(since raw_sock & ssl_sock need to access it).

- SE (muxs/applet) store their event log in the SD: se_event_log (64 bits).

- muxs must aggregate the connection's flags with its own:
- store last known connection state in SD: conn_event_log
- detect changes at the connection level by comparing with SD conn_event_log
- create a new SD event with difference(s) into SD se_event_log
- update connection state in SD conn_event_log

- stream
- store their event log in the stream: strm_event_log (64 bits).
- for each side:
- store last known SE state in SD: last_se_event_log
- detect changes at the SE level by comparing with SD se_event_log
- create a new STREAM event with difference(s) into STREAM strm_event_log
and patch the location depending on front vs back (+8 for back).
- update SE state in SD last_se_event_log

=> strm_event_log contains a composite of each side + stream.
- converted to string using the location letters
- if more event types needed later, can enlarge bits and use another letter.
- note: also possible to create an exhaustive enumeration of all possible codes
(types+locations).

- sample fetch to retrieve strm_event_log.

- Note that fc_err and fc_err_str are already usable

- questions:
- htx layer needed ?
- ability to map EOI/EOS etc to SE activity ?
- we'd like to detect an HTTP response before end of POST.

---

Doc/Design Thoughts/Numa Auto

2023-07-04 - automatic grouping for NUMA

Xeon: (W2145)

willy@debian:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,8
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,8
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,8
/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-15
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
/sys/devices/system/cpu/cpu0/cache/index3/type:Unified

Wtap: i7-8650U

willy@wtap:~ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,4
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,4
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,4
/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-7
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
/sys/devices/system/cpu/cpu0/cache/index3/type:Unified

pcw: i7-6700k

willy@pcw:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,4
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,4
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,4
/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-7
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
/sys/devices/system/cpu/cpu0/cache/index3/type:Unified

nfs: N5105, v5.15

willy@nfs:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-3
/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-3
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
/sys/devices/system/cpu/cpu0/cache/index3/type:Unified

eeepc: Atom N2800, 5.4 : no L3, L2 not shared.

willy@eeepc:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0-1
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0-1
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-1
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified

willy@eeepc:~$ grep '' /sys/devices/system/cpu/cpu2/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu2/cache/index0/shared_cpu_list:2-3
/sys/devices/system/cpu/cpu2/cache/index1/shared_cpu_list:2-3
/sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_list:2-3
/sys/devices/system/cpu/cpu2/cache/index0/type:Data
/sys/devices/system/cpu/cpu2/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu2/cache/index2/type:Unified

dev13: Ryzen 2700X

haproxy@dev13:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0-1
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0-1
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-1
/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-7
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
/sys/devices/system/cpu/cpu0/cache/index3/type:Unified

haproxy@dev13:~$ grep '' /sys/devices/system/cpu/cpu8/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu8/cache/index0/shared_cpu_list:8-9
/sys/devices/system/cpu/cpu8/cache/index1/shared_cpu_list:8-9
/sys/devices/system/cpu/cpu8/cache/index2/shared_cpu_list:8-9
/sys/devices/system/cpu/cpu8/cache/index3/shared_cpu_list:8-15
/sys/devices/system/cpu/cpu8/cache/index0/type:Data
/sys/devices/system/cpu/cpu8/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu8/cache/index2/type:Unified
/sys/devices/system/cpu/cpu8/cache/index3/type:Unified

dev12: Ryzen 5800X

haproxy@dev12:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,8
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,8
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,8
/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-15
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
/sys/devices/system/cpu/cpu0/cache/index3/type:Unified

amd24: EPYC 74F3

willy@mt:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,24
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,24
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,24
/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-2,24-26
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
/sys/devices/system/cpu/cpu0/cache/index3/type:Unified

willy@mt:~$ grep '' /sys/devices/system/cpu/cpu8/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu8/cache/index0/shared_cpu_list:8,32
/sys/devices/system/cpu/cpu8/cache/index1/shared_cpu_list:8,32
/sys/devices/system/cpu/cpu8/cache/index2/shared_cpu_list:8,32
/sys/devices/system/cpu/cpu8/cache/index3/shared_cpu_list:6-8,30-32
/sys/devices/system/cpu/cpu8/cache/index0/type:Data
/sys/devices/system/cpu/cpu8/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu8/cache/index2/type:Unified
/sys/devices/system/cpu/cpu8/cache/index3/type:Unified

willy@mt:~$ grep '' /sys/devices/system/cpu/cpu0/topology/*list
/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0,24
/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-47
/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0-47
/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-47
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0,24

xeon24: Gold 6212U

willy@mt01:~$ grep '' /sys/devices/system/cpu/cpu8/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu8/cache/index0/shared_cpu_list:8,32
/sys/devices/system/cpu/cpu8/cache/index1/shared_cpu_list:8,32
/sys/devices/system/cpu/cpu8/cache/index2/shared_cpu_list:8,32
/sys/devices/system/cpu/cpu8/cache/index3/shared_cpu_list:0-47
/sys/devices/system/cpu/cpu8/cache/index0/type:Data
/sys/devices/system/cpu/cpu8/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu8/cache/index2/type:Unified
/sys/devices/system/cpu/cpu8/cache/index3/type:Unified

SPR 8480+

$ grep -a '' /sys/devices/system/node/node*/cpulist
/sys/devices/system/node/node0/cpulist:0-55,112-167
/sys/devices/system/node/node1/cpulist:56-111,168-223

$ grep -a '' /sys/devices/system/cpu/cpu0/topology/*list
/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0,112
/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-55,112-167
/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0-55,112-167
/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-55,112-167
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0,112

$ grep -a '' /sys/devices/system/cpu/cpu0/cache/*/shared_cpu_list
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,112
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,112
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,112
/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-55,112-167

UP Board - Atom X5-8350 : no L3, exactly like Armada8040

willy@up1:~$ grep '' /sys/devices/system/cpu/cpu{0,1,2,3}/cache/index2/*list
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-1
/sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_list:0-1
/sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_list:2-3
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_list:2-3

willy@up1:~$ grep '' /sys/devices/system/cpu/cpu0/topology/*list
/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-3
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0

Atom D510 - kernel 2.6.33

$ strings -fn1 sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list: 0,2
sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list: 0,2
sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list: 0,2
sys/devices/system/cpu/cpu0/cache/index0/type: Data
sys/devices/system/cpu/cpu0/cache/index1/type: Instruction
sys/devices/system/cpu/cpu0/cache/index2/type: Unified

$ strings -fn1 sys/devices/system/cpu/cpu?/topology/*list
sys/devices/system/cpu/cpu0/topology/core_siblings_list: 0-3
sys/devices/system/cpu/cpu0/topology/thread_siblings_list: 0,2
sys/devices/system/cpu/cpu1/topology/core_siblings_list: 0-3
sys/devices/system/cpu/cpu1/topology/thread_siblings_list: 1,3
sys/devices/system/cpu/cpu2/topology/core_siblings_list: 0-3
sys/devices/system/cpu/cpu2/topology/thread_siblings_list: 0,2
sys/devices/system/cpu/cpu3/topology/core_siblings_list: 0-3
sys/devices/system/cpu/cpu3/topology/thread_siblings_list: 1,3

mcbin: Armada 8040 : no L3, no difference with L3 not reported

root@lg7:~# grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-1
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified

root@lg7:~# grep '' /sys/devices/system/cpu/cpu0/topology/*list
/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-3
/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0
/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-3
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0

Ampere/monolithic: Ampere Altra 80-26 : L3 not reported

willy@ampere:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified

willy@ampere:~$ grep '' /sys/devices/system/cpu/cpu0/topology/*list
/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-79
/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0
/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-79
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0

Ampere/Hemisphere: Ampere Altra 80-26 : L3 not reported

willy@ampere:~$ grep '' /sys/devices/system/node/node*/cpulist
/sys/devices/system/node/node0/cpulist:0-39
/sys/devices/system/node/node1/cpulist:40-79

LX2A: LX2160A => L3 not reported

willy@lx2a:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-1
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified

willy@lx2a:~$ grep '' /sys/devices/system/cpu/cpu2/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu2/cache/index0/shared_cpu_list:2
/sys/devices/system/cpu/cpu2/cache/index1/shared_cpu_list:2
/sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_list:2-3
/sys/devices/system/cpu/cpu2/cache/index0/type:Data
/sys/devices/system/cpu/cpu2/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu2/cache/index2/type:Unified

willy@lx2a:~$ grep '' /sys/devices/system/cpu/cpu0/topology/*list
/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-15
/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0
/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-15
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0

Rock5B: RK3588 (big-little A76+A55)

rock@rock-5b:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-7
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
/sys/devices/system/cpu/cpu0/cache/index3/type:Unified

rock@rock-5b:~$ grep '' /sys/devices/system/cpu/cpu{0,4,6}/topology/*list
/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-3
/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0
/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-3
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0
/sys/devices/system/cpu/cpu4/topology/core_cpus_list:4
/sys/devices/system/cpu/cpu4/topology/core_siblings_list:4-5
/sys/devices/system/cpu/cpu4/topology/die_cpus_list:4
/sys/devices/system/cpu/cpu4/topology/package_cpus_list:4-5
/sys/devices/system/cpu/cpu4/topology/thread_siblings_list:4
/sys/devices/system/cpu/cpu6/topology/core_cpus_list:6
/sys/devices/system/cpu/cpu6/topology/core_siblings_list:6-7
/sys/devices/system/cpu/cpu6/topology/die_cpus_list:6
/sys/devices/system/cpu/cpu6/topology/package_cpus_list:6-7
/sys/devices/system/cpu/cpu6/topology/thread_siblings_list:6

$ grep '' /sys/devices/system/cpu/cpu*/cpu_capacity
/sys/devices/system/cpu/cpu0/cpu_capacity:414
/sys/devices/system/cpu/cpu1/cpu_capacity:414
/sys/devices/system/cpu/cpu2/cpu_capacity:414
/sys/devices/system/cpu/cpu3/cpu_capacity:414
/sys/devices/system/cpu/cpu4/cpu_capacity:1024
/sys/devices/system/cpu/cpu5/cpu_capacity:1024
/sys/devices/system/cpu/cpu6/cpu_capacity:1024
/sys/devices/system/cpu/cpu7/cpu_capacity:1024

Firefly: RK3399 (2xA72 + 4xA53) kernel 6.1.28

root@firefly:~# grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
grep: /sys/devices/system/cpu/cpu0/cache/index?/shared_cpu_list: No such file or directory
grep: /sys/devices/system/cpu/cpu0/cache/index?/type: No such file or directory

root@firefly:~# grep '' /sys/devices/system/cpu/cpu*/cache/index?/{shared_cpu_list,type}
grep: /sys/devices/system/cpu/cpu*/cache/index?/shared_cpu_list: No such file or directory
grep: /sys/devices/system/cpu/cpu*/cache/index?/type: No such file or directory

root@firefly:~# dmesg|grep cacheinfo
[ 0.006290] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 0.016339] cacheinfo: Unable to detect cache hierarchy for CPU 1
[ 0.017692] cacheinfo: Unable to detect cache hierarchy for CPU 2
[ 0.019050] cacheinfo: Unable to detect cache hierarchy for CPU 3
[ 0.020478] cacheinfo: Unable to detect cache hierarchy for CPU 4
[ 0.021660] cacheinfo: Unable to detect cache hierarchy for CPU 5
[ 1.990108] cacheinfo: Unable to detect cache hierarchy for CPU 0

root@firefly:~# grep '' /sys/devices/system/cpu/cpu0/topology/*
/sys/devices/system/cpu/cpu0/topology/cluster_cpus:0f
/sys/devices/system/cpu/cpu0/topology/cluster_cpus_list:0-3
/sys/devices/system/cpu/cpu0/topology/cluster_id:0
/sys/devices/system/cpu/cpu0/topology/core_cpus:01
/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
/sys/devices/system/cpu/cpu0/topology/core_id:0
/sys/devices/system/cpu/cpu0/topology/core_siblings:3f
/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-5
/sys/devices/system/cpu/cpu0/topology/package_cpus:3f
/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-5
/sys/devices/system/cpu/cpu0/topology/physical_package_id:0
/sys/devices/system/cpu/cpu0/topology/thread_siblings:01
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0

$ grep '' /sys/devices/system/cpu/cpu*/cpu_capacity
/sys/devices/system/cpu/cpu0/cpu_capacity:381
/sys/devices/system/cpu/cpu1/cpu_capacity:381
/sys/devices/system/cpu/cpu2/cpu_capacity:381
/sys/devices/system/cpu/cpu3/cpu_capacity:381
/sys/devices/system/cpu/cpu4/cpu_capacity:1024
/sys/devices/system/cpu/cpu5/cpu_capacity:1024

VIM3L: S905D3 (4*A55), kernel 5.14.10

$ grep '' /sys/devices/system/cpu/cpu0/topology/*
/sys/devices/system/cpu/cpu0/topology/core_cpus:1
/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
/sys/devices/system/cpu/cpu0/topology/core_id:0
/sys/devices/system/cpu/cpu0/topology/core_siblings:f
/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-3
/sys/devices/system/cpu/cpu0/topology/die_cpus:1
/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0
/sys/devices/system/cpu/cpu0/topology/die_id:-1
/sys/devices/system/cpu/cpu0/topology/package_cpus:f
/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-3
/sys/devices/system/cpu/cpu0/topology/physical_package_id:0
/sys/devices/system/cpu/cpu0/topology/thread_siblings:1
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0

$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-3
/sys/devices/system/cpu/cpu0/cache/index0/type:Data
/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
/sys/devices/system/cpu/cpu0/cache/index2/type:Unified

$ grep '' /sys/devices/system/cpu/cpu*/cpu_capacity
/sys/devices/system/cpu/cpu0/cpu_capacity:1024
/sys/devices/system/cpu/cpu1/cpu_capacity:1024
/sys/devices/system/cpu/cpu2/cpu_capacity:1024
/sys/devices/system/cpu/cpu3/cpu_capacity:1024

Odroid-N2: S922X (4A73 + 2A53), kernel 4.9.254

willy@n2:~$ grep '' /sys/devices/system/cpu/cpu*/cache/index?/{shared_cpu_list,type}
grep: /sys/devices/system/cpu/cpu*/cache/index?/shared_cpu_list: No such file or directory
grep: /sys/devices/system/cpu/cpu*/cache/index?/type: No such file or directory

willy@n2:~$ sudo dmesg|grep -i 'cache hi'
[ 0.649924] Unable to detect cache hierarchy for CPU 0

No capacity.

Note that it reports 2 physical packages!

willy@n2:~$ grep '' /sys/devices/system/cpu/cpu0/topology/*
/sys/devices/system/cpu/cpu0/topology/core_id:0
/sys/devices/system/cpu/cpu0/topology/core_siblings:03
/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-1
/sys/devices/system/cpu/cpu0/topology/physical_package_id:0
/sys/devices/system/cpu/cpu0/topology/thread_siblings:01
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0

willy@n2:~$ grep '' /sys/devices/system/cpu/cpu4/topology/*
/sys/devices/system/cpu/cpu4/topology/core_id:2
/sys/devices/system/cpu/cpu4/topology/core_siblings:3c
/sys/devices/system/cpu/cpu4/topology/core_siblings_list:2-5
/sys/devices/system/cpu/cpu4/topology/physical_package_id:1
/sys/devices/system/cpu/cpu4/topology/thread_siblings:10
/sys/devices/system/cpu/cpu4/topology/thread_siblings_list:4

StarFive VisionFive2 - JH7110, kernel 5.15

willy@starfive:~/haproxy$ ./haproxy -c -f cps3.cfg
thr 0 -> cpu 0 onl=1 bnd=1 pk=00 no=-1 l3=-1 cl=000 l2=000 ts=000 l1=000
thr 1 -> cpu 1 onl=1 bnd=1 pk=00 no=-1 l3=-1 cl=000 l2=000 ts=001 l1=001
thr 2 -> cpu 2 onl=1 bnd=1 pk=00 no=-1 l3=-1 cl=000 l2=000 ts=002 l1=002
thr 3 -> cpu 3 onl=1 bnd=1 pk=00 no=-1 l3=-1 cl=000 l2=000 ts=003 l1=003
Configuration file is valid

Graviton2 / Graviton3 ?

On PPC64 not everything is available:

https://www.ibm.com/docs/en/linux-on-systems?topic=cpus-cpu-topology

/sys/devices/system/cpu/cpu<N>/topology/thread_siblings
/sys/devices/system/cpu/cpu<N>/topology/core_siblings
/sys/devices/system/cpu/cpu<N>/topology/book_siblings
/sys/devices/system/cpu/cpu<N>/topology/drawer_siblings

# lscpu -e
CPU NODE DRAWER BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED POLARIZATION ADDRESS
0 1 0 0 0 0 0:0:0:0 yes yes horizontal 0
1 1 0 0 0 0 1:1:1:1 yes yes horizontal 1
2 1 0 0 0 1 2:2:2:2 yes yes horizontal 2
3 1 0 0 0 1 3:3:3:3 yes yes horizontal 3
4 1 0 0 0 2 4:4:4:4 yes yes horizontal 4
5 1 0 0 0 2 5:5:5:5 yes yes horizontal 5
6 1 0 0 0 3 6:6:6:6 yes yes horizontal 6
7 1 0 0 0 3 7:7:7:7 yes yes horizontal 7
8 0 1 1 1 4 8:8:8:8 yes yes horizontal 8
...

Intel E5-2600v2/v3 has two L3:
https://www.enterpriseai.news/2014/09/08/intel-ups-performance-ante-haswell-xeon-chips/

More info on these, and s390's "books" (mostly L4 in fact):
https://groups.google.com/g/fa.linux.kernel/c/qgAxjYq8ohI

########################################
Analysis:
- some server ARM CPUs (Altra, LX2) do not return any L3 info though they
DO have some. They stop at L2.

- other CPUs like Atom N2800 and Armada 8040 do not have L3.

=> there's no apparent way to detect that the server CPUs do have an L3.
=> or maybe we should consider that it's more likely that there is one
than none ? Armada works much better with groups than without. It's
basically the same topology as N2800.

=> Do we really care then ? No L3 = same L3 for everyone. The problem is
that those really without L3 will make a difference on L2 while the
other ones not. Maybe we should consider that it does not make sense
to cut groups on L2 (i.e. under no circumstance we'll have one group
per core).

=> This would mean:
- regardless of L3, consider LLC. If the LLC has more than one
core per instance, it's likely the last one (not true on LX2
but better use 8 groups of 2 than nothing).

- otherwise if there's a single core per instance, it's unlikely
to be the LLC so we can imagine the LLC is unified. Note that
some systems such as LX2/Armada8K (and Neoverse-N1 devices as
well) may have 2 cores per L2, yet this doesn't allow to infer
anything regarding the absence of an L3. Core2-quad has 2 cores
per L2 with no L3, like Armada8K. LX2 has 2 cores per L2 yet does
have an L3 which is not necessarily reported.

- this needs to be done per {node,package} !
=> core_siblings and thread_siblings seem to be the only portable
ones to figure packages and threads

At the very least, when multiple nodes are possibly present, there is a
symlink "node0", "node1" etc in the cpu entry. It requires a lookup for each
cpu directory though while reading /sys/devices/system/node/node*/cpulist is
much cheaper.

There's some redundancy in this. Probably better approach:

1) if there is more than 1 CPU:
- if cache/index3 exists, use its cpulist to pre-group entries.
- else if topology or node exists, use (node,package,die,core_siblings) to
group entries
- else pre-create a single large group

2) if there is more than 1 CPU and less than max#groups:
- for each group, if no cache/index3 exists and cache/index2 exists and some
index2 entries contain at least two CPUs of different cores or a single one
for a 2-core system, then use that to re-split the group.

- if in the end there are too many groups, remerge some of them (?) or stick
to the previous layout (?)

- if in the end there are too many CPUs in a group, cut as needed, if
possible with an integral result (/2, /3, ...)

3) L1 cache / thread_siblings should be used to associate CPUs by cores in
the same groups.

Maybe instead it should be done bottom->top by collecting info and merging
groups while keeping CPU lists ordered to ease later splitting.

1) create a group per bound CPU
2) based on thread_siblings, detect CPUs that are on the same core, merge
their groups. They may not always create similarly sized groups.
=> eg: epyc keeps 24 groups such as {0,24}, ...
ryzen 2700x keeps 4 groups such as {0,1}, ...
rk3588 keeps 3 groups {0-3},{4-5},{6-7}
3) based on cache index0/1, detect CPUs that are on the same L1 cache,
merge their groups. They may not always create similarly sized groups.
4) based on cache index2, detect CPUs that are on the same L2 cache, merge
their groups. They may not always create similarly sized groups.
=> eg: mcbin now keeps 2 groups {0-1},{2,3}
5) At this point there may possibly be too many groups (still one per CPU,
e.g. when no cache info was found or there are many cores with their own
L2 like on SPR) or too large one (when all cores are indeed on the same
L2).

5.1) if there are as many groups as bound CPUs, merge them all together in
a single one => lx2, altra, mcbin
5.2) if there are still more than max#groups, merge them all together in a
single one since the splitting criterion is not relevant
5.3) if there is a group with too many CPUs, split it in two if integral,
otherwise 3, etc, trying to add the least possible number of groups.
If too difficult (e.g. result less than half the authorized max),
let's just round around N/((N+63)/64).
5.4) if at the end there are too many groups, warn that we can't optimize
the setup and are limiting ourselves to the first node or 64 CPUs.

Observations:
- lx2 definitely works better with everything bound together than by creating
8 groups (~130k rps vs ~120k rps)
=> does this mean we should assume a unified L3 if there's no L3 info, and
remerge everything ? Likely Altra would benefit from this as well. mcbin
doesn't notice any change (within noise in both directions)

- on x86 13th gen, 2 P-cores and 8 E-cores. The P-cores support HT, not the
E-cores. There's no cpu_capacity there, but the cluster_id is properly set.
=> proposal: when a machine reports both single-threaded cores and SMT,
consider the SMT ones bigger and use them.

Problems: how should auto-detection interfer with user-settings ?

- Case 1: program started with a reduced taskset
=> current: this serves to the the thread count first, and to map default
threads to CPUs if they are not affected by a cpu-map.

=> we want to keep that behavior (i.e. use all these threads) but only
change how the thread-groups are arranged.

- example: start on the first 6c12t of an EPYC74F3, should automatically
create 2 groups for the two sockets.

=> should we brute-force all thread-groups combinations to figure how the
threads will spread over cpu-map and which one is better ? Or should we
decide to ignore input mapping as soon as there's at least one cpu-map?
But then which one to use ? Or should we consider that cpu-map only works
with explicit thread-groups ?

- Case 2: taskset not involved, but nbthread and cpu-map in the config. In
fact a pretty standard 2.4-2.8 config.
=> maybe the presence of cpu-map and no thread-groups should be sufficient
to imply a single thread-group to stay compatible ? Or maybe start as
many thread-groups as are referenced in cpu-map ? Seems like cpu-map and
thread-groups work hand-in-hand regarding topology since cpu-map
designates hardware CPUs so the user knows better than haproxy. Thus
why should be try to do better ?

- Case 3: taskset not involved, nbthread not involved, cpu-map not involved,
only thread-groups
=> seems like an ideal approach. Take all online CPUs and try to cut them
into equitable thread groups ? Or rather, since nbthreads is not forced,
better sort the clusters and bind to the N first clusters only ? If too
many groups for the clusters, then try to refine them ?

- Case 4: nothing specified at all (default config, target)
=> current: uses only one thread-group with all threads (max 64).
=> desired: bind only to performance cores and cut them in a few groups
based on l3, package, cluster etc.

- Case 5: nbthread only in the config
=> might match a docker use case. No group nor cpu-map configured. Figure
the best group usage respecting the thread count.

- Case 6: some constraints are enforced in the config (e.g. threads-hard-limit,
one-thread-per-core, etc).
=> like 3, 4 or 5 but with selection adjustment.

- Case 7: thread-groups and generic cpu-map 1/all, 2/all... in the config
=> user just wants to use cpu-map as a taskset alternative
=> need to figure number of threads first, then cut them in groups like
today, and only then the cpu-map are found. Can we do better ? Not sure.
Maybe just when cpu-map is too lax (e.g. all entries reference the same
CPUs). Better use a special "cpumap all/all 0-19" for this, but not
implemented for now.

Proposal:
- if there is any cpu-map, disable automatic CPU assignment
- if there is any cpu-map, disable automatic thread group detection
- if taskset was forced, disable automatic CPU assignment

2023-07-17 ###

=> step 1: mark CPUs enabled at boot (cpu_detect_usable)
// => step 2: mark CPUs referenced in cpu-map => no, no real meaning
=> step 3: identify all CPUs topologies + NUMA (cpu_detect_topology)

=> step 4: if taskset && !cpu-map, mark all non-bound CPUs as unusable (UNAVAIL ?)
=> which is the same as saying if !cpu-map.
=> step 5: if !cpu-map, sort usable CPUs and find the best set to use
//=> step 6: if cpu-map, mark all non-covered CPUs are unusable => not necessarily possible if partial cpu-map

=> step 7: if thread-groups && cpu-map, nothing else to do
=> step 8: if cpu-map && !thread-groups, thread-groups=1
=> step 9: if thread-groups && !cpu-map, use that value to cut the thread set
=> step 10: if !cpu-map && !thread-groups, detect the optimal thread-group count

=> step 11: if !cpu-map, cut the thread set into mostly fair groups and assign
the group numbers to CPUs; create implicit cpu-maps.

Ideas:
- use minthr and maxthr.
If nbthread, minthr=maxthr=nbthread, else if taskset_forced, maxthr=taskset_thr,
minthr=1, else minthr=1, maxthr=cpus_enabled.

- use CPU_F_ALLOWED (or DISALLOWED?) and CPU_F_REFERENCED and CPU_F_EXCLUDED ?
Note: cpu-map doesn't exclude, it only includes. Taskset does exclude. Also,
cpu-map only includes the CPUs that will belong to the correct groups & threads.

- Usual startup: taskset presets the CPU sets and sets the thread count. Tgrp
defaults to 1, then threads indicated in cpu-map get their CPU assigned.
Other ones are not changed. If we say that cpu-map => tgrp==1 then it means
we can infer automatic grouping for group 1 only ?
=> it could be said that the CPUs of all enabled groups mentioned in
cpu-map are considered usable, but we don't know how many of these
will really have threads started on.

=> maybe completely ignore cpu-map instead (i.e. fall back to thread-groups 1) ?
=> automatic detection would mean:
- if !cpu-map && !nbthrgrp => must automatically detect thgrp
- if !cpu-map => must automatically detect binding
- otherwise nothing

Examples of problems:

thread-groups 4
nbthreads 128
cpu-map 1/all 0-63
cpu-map 2/all 128-191

=> 32 threads per group, hence grp 1 uses 0-63 and grp 2 128-191,
grp 3 and grp 4 unknown, in practice on boot CPUs.

=> could we demand that if one cpu-map is specified, then all groups
are covered ? Do we need really this after all ? i.e. let's just not
bind other threads and that's all (and what is written).

Calls from haproxy.c:

cpu_detect_usable()
cpu_detect_topology()

+ thread_detect_count()
=> compute nbtgroups
=> compute nbthreads

thread_assign_cpus() ?

check_config_validity()

BUGS:
- cpu_map[0].proc still used for the whole process in daemon mode (though not
in foreground mode)
-> whole process bound to thread group 1
-> binding not working in foreground

- cpu_map[x].proc ANDed with the thread's map depite thread's map apparently
never set
-> group binding ignored ?

2023-09-05
----------
Remember to make the difference between sorting (used for grouping) and
preference. We should avoid selecting the first CPUs as it encourages to
use wrong grouping criteria. E.g. CPU capacity has no business being used
for grouping, it's used for selecting. Support for HT however, does because
it allows to pack together threads of the same core.

We should also have an option to enable/disable SMT (e.g. max threads per core)
so that we can skip siblings of cores already assigned. This can be convenient
with network running on the other sibling.

2024-12-26
----------

Some interesting cases about intel 14900. The CPU has 8 P-cores and 16 E-cores.
Experiments in the lab show excellent performance by binding the network to E
cores and haproxy to P cores. Here's how the clusters are made:

$ grep -h . /sys/devices/system/cpu/cpu*/topology/package_cpus | sort |uniq -c
32 ffffffff

=> expected

$ grep -h . /sys/devices/system/cpu/cpu*/topology/die_cpus | sort |uniq -c
32 ffffffff

=> all CPUs on the same die

$ grep -h . /sys/devices/system/cpu/cpu*/topology/cluster_cpus | sort |uniq -c
2 00000003
2 0000000c
2 00000030
2 000000c0
2 00000300
2 00000c00
2 00003000
2 0000c000
4 000f0000
4 00f00000
4 0f000000
4 f0000000

=> 1 "cluster" per core on each P-core (2 threads, 8 clusters total)
=> 1 "cluster" per 4 E-cores (4 clusters total)
=> It can be difficult to split that into groups by just using this topology.

$ grep -h . /sys/devices/system/cpu/cpu*/cache/index3/shared_cpu_list | sort |uniq -c
32 0-31

=> everyone shares a uniform L3 cache

$ grep -h . /sys/devices/system/cpu/cpu*/cache/index2/shared_cpu_map | sort |uniq -c
2 00000003
2 0000000c
2 00000030
2 000000c0
2 00000300
2 00000c00
2 00003000
2 0000c000
4 000f0000
4 00f00000
4 0f000000
4 f0000000

=> L2 is split like the respective "clusters" above.

Semms like one would like to split them into 12 groups :-/ Maybe it still
remains relevant to consider L3 for grouping, and core performance for
selection (e.g. evict/prefer E-cores depending on policy).

Differences between P and E cores on 14900:

- acpi_cppc/*perf : pretty useful but not always there (e.g. aloha)
- cache index0: 48 vs 32k (bigger CPU has smaller cache)
- cache index1: 32 vs 64k (smaller CPU has bigger cache)
- cache index2: 2 vs 4M, but dedicated per core vs shared per cluster (4 cores)

=> probably that the presence of a larger "cluster" with less cache per
avg core is an indication of a smaller CPU set. Warning however, some
CPUs (e.g. S922X) have a large (4) cluster of big cores and a small (2)
cluster of little cores.

diff -urN cpu0/acpi_cppc/lowest_nonlinear_perf cpu16/acpi_cppc/lowest_nonlinear_perf
--- cpu0/acpi_cppc/lowest_nonlinear_perf 2024-12-26 18:39:27.563410317 +0100
+++ cpu16/acpi_cppc/lowest_nonlinear_perf 2024-12-26 18:40:39.531408186 +0100
@@ -1 +1 @@
-20
+15
diff -urN cpu0/acpi_cppc/nominal_perf cpu16/acpi_cppc/nominal_perf
--- cpu0/acpi_cppc/nominal_perf 2024-12-26 18:39:27.563410317 +0100
+++ cpu16/acpi_cppc/nominal_perf 2024-12-26 18:40:39.531408186 +0100
@@ -1 +1 @@
-40
+24
diff -urN cpu0/acpi_cppc/reference_perf cpu16/acpi_cppc/reference_perf
--- cpu0/acpi_cppc/reference_perf 2024-12-26 18:39:27.563410317 +0100
+++ cpu16/acpi_cppc/reference_perf 2024-12-26 18:40:39.531408186 +0100
@@ -1 +1 @@
-40
+24
diff -urN cpu0/cache/index0/size cpu16/cache/index0/size
--- cpu0/cache/index0/size 2024-12-26 18:39:27.563410317 +0100
+++ cpu16/cache/index0/size 2024-12-26 18:40:39.531408186 +0100
@@ -1 +1 @@
-48K
+32K
diff -urN cpu0/cache/index1/shared_cpu_list cpu16/cache/index1/shared_cpu_list
--- cpu0/cache/index1/shared_cpu_list 2024-12-26 18:39:27.563410317 +0100
+++ cpu16/cache/index1/shared_cpu_list 2024-12-26 18:40:39.531408186 +0100
@@ -1 +1 @@
-0-1
+16
diff -urN cpu0/cache/index1/shared_cpu_map cpu16/cache/index1/shared_cpu_map
--- cpu0/cache/index1/shared_cpu_map 2024-12-26 18:39:27.563410317 +0100
+++ cpu16/cache/index1/shared_cpu_map 2024-12-26 18:40:39.531408186 +0100
@@ -1 +1 @@
-00000003
+00010000
diff -urN cpu0/cache/index1/size cpu16/cache/index1/size
--- cpu0/cache/index1/size 2024-12-26 18:39:27.563410317 +0100
+++ cpu16/cache/index1/size 2024-12-26 18:40:39.531408186 +0100
@@ -1 +1 @@
-32K
+64K
diff -urN cpu0/cache/index2/shared_cpu_list cpu16/cache/index2/shared_cpu_list
--- cpu0/cache/index2/shared_cpu_list 2024-12-26 18:39:27.563410317 +0100
+++ cpu16/cache/index2/shared_cpu_list 2024-12-26 18:40:39.531408186 +0100
@@ -1 +1 @@
-0-1
+16-19
--- cpu0/cache/index2/size 2024-12-26 18:39:27.563410317 +0100
+++ cpu16/cache/index2/size 2024-12-26 18:40:39.531408186 +0100
@@ -1 +1 @@
-2048K
+4096K
diff -urN cpu0/topology/cluster_cpus cpu16/topology/cluster_cpus
--- cpu0/topology/cluster_cpus 2024-12-26 18:39:27.563410317 +0100
+++ cpu16/topology/cluster_cpus 2024-12-26 18:40:39.531408186 +0100
@@ -1 +1 @@
-00000003
+000f0000
diff -urN cpu0/topology/cluster_cpus_list cpu16/topology/cluster_cpus_list
--- cpu0/topology/cluster_cpus_list 2024-12-26 18:39:27.563410317 +0100
+++ cpu16/topology/cluster_cpus_list 2024-12-26 18:40:39.531408186 +0100
@@ -1 +1 @@
-0-1
+16-19

For acpi_cppc, the values differ between machines, looks like nominal_perf
is always usable:

14900k:
$ grep '' cpu8/acpi_cppc/*
cpu8/acpi_cppc/feedback_ctrs:ref:85172004640 del:143944480100
cpu8/acpi_cppc/highest_perf:255
cpu8/acpi_cppc/lowest_freq:0
cpu8/acpi_cppc/lowest_nonlinear_perf:20
cpu8/acpi_cppc/lowest_perf:1
cpu8/acpi_cppc/nominal_freq:3200
cpu8/acpi_cppc/nominal_perf:40
cpu8/acpi_cppc/reference_perf:40
cpu8/acpi_cppc/wraparound_time:18446744073709551615

$ grep '' cpu16/acpi_cppc/*
cpu16/acpi_cppc/feedback_ctrs:ref:84153776128 del:112977352354
cpu16/acpi_cppc/highest_perf:255
cpu16/acpi_cppc/lowest_freq:0
cpu16/acpi_cppc/lowest_nonlinear_perf:15
cpu16/acpi_cppc/lowest_perf:1
cpu16/acpi_cppc/nominal_freq:3200
cpu16/acpi_cppc/nominal_perf:24
cpu16/acpi_cppc/reference_perf:24
cpu16/acpi_cppc/wraparound_time:18446744073709551615

altra:
$ grep '' /sys/devices/system/cpu/cpu0/acpi_cppc/*
feedback_ctrs:ref:227098452801 del:590247062111
highest_perf:260
lowest_freq:1000
lowest_nonlinear_perf:200
lowest_perf:100
nominal_freq:2600
nominal_perf:260
reference_perf:100

w3-2345:
$ grep '' /sys/devices/system/cpu/cpu0/acpi_cppc/*
feedback_ctrs:ref:4775674480779 del:5675950973600
highest_perf:45
lowest_freq:0
lowest_nonlinear_perf:8
lowest_perf:5
nominal_freq:0
nominal_perf:31
reference_perf:31
wraparound_time:18446744073709551615

Other approaches may consist in checking the CPU's max frequency via
cpufreq, e.g on the N2:

$ grep . /sys/devices/system/cpu/cpu?/cpufreq/scaling_max_freq
/sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq:2016000
/sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq:2016000
/sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq:2400000
/sys/devices/system/cpu/cpu3/cpufreq/scaling_max_freq:2400000
/sys/devices/system/cpu/cpu4/cpufreq/scaling_max_freq:2400000
/sys/devices/system/cpu/cpu5/cpufreq/scaling_max_freq:2400000

However on x86, the cores no longer all have the same frequency, like below on
the W3-2345, so it cannot always be used to split them into groups, it may at
best be used to sort them.

$ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
/sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq:4500000
/sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq:4500000
/sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq:4300000
/sys/devices/system/cpu/cpu3/cpufreq/scaling_max_freq:4400000
/sys/devices/system/cpu/cpu4/cpufreq/scaling_max_freq:4300000
/sys/devices/system/cpu/cpu5/cpufreq/scaling_max_freq:4300000
/sys/devices/system/cpu/cpu6/cpufreq/scaling_max_freq:4400000
/sys/devices/system/cpu/cpu7/cpufreq/scaling_max_freq:4300000
/sys/devices/system/cpu/cpu8/cpufreq/scaling_max_freq:4500000
/sys/devices/system/cpu/cpu9/cpufreq/scaling_max_freq:4500000
/sys/devices/system/cpu/cpu10/cpufreq/scaling_max_freq:4300000
/sys/devices/system/cpu/cpu11/cpufreq/scaling_max_freq:4400000
/sys/devices/system/cpu/cpu12/cpufreq/scaling_max_freq:4300000
/sys/devices/system/cpu/cpu13/cpufreq/scaling_max_freq:4300000
/sys/devices/system/cpu/cpu14/cpufreq/scaling_max_freq:4400000
/sys/devices/system/cpu/cpu15/cpufreq/scaling_max_freq:4300000

On 14900, not cool either:

$ grep -h . /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq|sort|uniq -c
16 4400000
12 5700000
4 6000000

Considering that values that are within +/-10% of a cluster's min/max are still
part of it would seem to work and would make a good rule of thumb.

On x86, the model number might help, here on w3-2345:

$ grep '^model\s\s' /proc/cpuinfo |sort|uniq -c
16 model : 143

But not always (here: 14900K with 8xP and 16xE):

$ grep '^model\s\s' /proc/cpuinfo |sort|uniq -c
32 model : 183

On ARM it's rather the part number:

# a9
$ grep part /proc/cpuinfo
CPU part : 0xc09
CPU part : 0xc09

# a17
$ grep part /proc/cpuinfo
CPU part : 0xc0d
CPU part : 0xc0d
CPU part : 0xc0d
CPU part : 0xc0d

# a72
$ grep part /proc/cpuinfo
CPU part : 0xd08
CPU part : 0xd08
CPU part : 0xd08
CPU part : 0xd08

# a53+a72
$ grep part /proc/cpuinfo
CPU part : 0xd03
CPU part : 0xd03
CPU part : 0xd03
CPU part : 0xd03
CPU part : 0xd08
CPU part : 0xd08

# a53+a73
$ grep 'part' /proc/cpuinfo
CPU part : 0xd03
CPU part : 0xd03
CPU part : 0xd09
CPU part : 0xd09
CPU part : 0xd09
CPU part : 0xd09

# a55+a76
$ grep 'part' /proc/cpuinfo
CPU part : 0xd05
CPU part : 0xd05
CPU part : 0xd05
CPU part : 0xd05
CPU part : 0xd0b
CPU part : 0xd0b
CPU part : 0xd0b
CPU part : 0xd0b

2024-12-27
----------

Such machines with P+E cores are becoming increasingly common. Some like the
CIX-P1 can even provide 3 levels of performance: 4 big cores (A720-2.8G), 4
medium cores (A720-2.4G), 4 little cores (A520-1.8G). Architectures like below
will become the norm, and can be used under different policies:

+-----------------------------+
| L3 |
+---+----------+----------+---+
| | |
+---+---+ +---+---+ +---+---+
| P | P | | E | E | | E | E |
+---+---+ +---+---+ +---+---+
Policy: | P | P | | E | E | | E | E |
------- +---+---+ +---+---+ +---+---+
1 group, min: N/A 0 0
1 group, max: 0 N/A N/A
1 group, all: 0 0 0
2 groups, min: N/A 0 1
2 groups, full: 0 1 1
3 groups: 0 1 2

In dual-socket or multiple dies it can even become more complicated:

+---+---+ +---+---+ +---+---+
| P | P | | E | E | | E | E |
+---+---+ +---+---+ +---+---+
| P | P | | E | E | | E | E |
+---+---+ +---+---+ +---+---+
| | |
+---+----------+----------+---+
| L3.0 |
+-----------------------------+

+-----------------------------+
| L3.1 |
+---+----------+----------+---+
| | |
+---+---+ +---+---+ +---+---+
| P | P | | E | E | | E | E |
+---+---+ +---+---+ +---+---+
| P | P | | E | E | | E | E |
+---+---+ +---+---+ +---+---+

Setting only a thread count would yield interesting things above:
1-4T: P.0
5-8T: P.0, P.1 (2 grp)
9-16T: P.0, E.0, P.1, E.1 (3-4 grp)
17-24T: PEE.0, PEE.1 (5-6 grp)

With forced tgrp = 1:
- only fill node 0 first (P then PE, then PEE)

With forced tgrp = 2:

def: P.0, P.1
2-4T: P.0 only ?
6-8T: P.0, P.1
9-24T: PEE.0, PEE.1

With dual-socket, dual-die, it becomes:

+---+---+ +---+---+ +---+---+ ' +---+---+ +---+---+ +---+---+
| P | P | | E | E | | E | E | ' | P | P | | E | E | | E | E |
+---+---+ +---+---+ +---+---+ ' +---+---+ +---+---+ +---+---+
| P | P | | E | E | | E | E | ' | P | P | | E | E | | E | E |
+---+---+ +---+---+ +---+---+ ' +---+---+ +---+---+ +---+---+
| | | ' | | |
+---+----------+----------+---+ ' +---+----------+----------+---+
| L3.0.0 | ' | L3.1.0 |
+-----------------------------+ ' +-----------------------------+
'
+-----------------------------+ ' +-----------------------------+
| L3.0.1 | ' | L3.1.1 |
+---+----------+----------+---+ ' +---+----------+----------+---+
| | | ' | | |
+---+---+ +---+---+ +---+---+ ' +---+---+ +---+---+ +---+---+
| P | P | | E | E | | E | E | ' | P | P | | E | E | | E | E |
+---+---+ +---+---+ +---+---+ ' +---+---+ +---+---+ +---+---+
| P | P | | E | E | | E | E | ' | P | P | | E | E | | E | E |
+---+---+ +---+---+ +---+---+ ' +---+---+ +---+---+ +---+---+

In such conditions, it could make sense to first enumerate all the available
cores with all their characteristics, and distribute them between "buckets"
representing the thread groups:

1. create the min number of tgrp (tgrp.min)
2. it's possible to automatically create more until tgrp.max
-> cores are sorted by performance then by proximity. They're
distributed in order into existing buckets, and if too distant,
then new groups are created. It could allow for example to use
all P-cores in the DSDD model above, split into 4 tgrp.
-> the total number of threads is then discovered at the end.

It seems in the end that such binding policies (P, E, single/multi dies,
single/multi sockets etc) should be made more accessible to the user. What
we're missing in "cpu-map" is the ability to apply to the whole process in
fact, so that it can supersede taskset. Indeed, right now, cpu-map requires
too many details and that's why it often remains easier to deal with taskset,
particularly when dealing with thread groups.

We can revisit the situation differently. First, let's keep in mind that
cpu-map is a restriction. It means "use no more than these", it does not
mean "use all of these". So it totally makes sense to use it to replace
taskset at the process level without interfering with groups detection.
We could then have:

- "cpu-map all|process|global|? ..." to apply to the whole process
- then special keywords for the CPUs designation, among:
- package (socket) number
- die number (CCD)
- L3 number (CCX)
- cluster type (big/performant, medium, little/efficient)
- use of SMT or not, and which ones
- maybe optional numbers before these to indicate (any two of them),
e.g. "4P" to indicate "4 performance cores".

Question: how would we designate "only P cores of socket 0" ? Or
"only thread 0 of all P cores" ?

One benefit of such a declaration method is that it can make nbthread often
useless and automatic while still portable across a whole fleet of servers. E.g.
if "cpu-map all S0P*T0" would designate thread 0 of all P-cores of socket 0, it
would mean the same on all machines.

Another benefit is that we can make cpu-map and automatic detection more
exclusive:
- cpu-map all => equivalent of taskset, leaves auto-detection on
- cpu-map thr => disables auto-detection

So in the end:
- cpu-map all restricts the CPUs the process may use
-> auto-detection starts from here and sorts them
- thread-groups offers more "buckets" to arrange distant CPUs in the
same process
- nbthread limits the number of threads we'll use
-> pick the most suited ones (at least thr.min, at most thr.max)
and distribute them optimally among the number of thread groups.

One question remains: is it always possible to automatically configure
thread-groups ? Maybe it's possible after the detection to set an optimal
one between grp.min and grp.max ? (e.g. socket count, core types, etc).

It still seems that a policy such as "optimize-for resources|perfomance"
would still help quite a bit.

-> what defines a match between a CPU core and a group:
- cluster identification:
- either cluster_cpus if present (and sometimes), or:
- pkg+die+ccd number
- same LLC instance (L3 if present, L2 if no L3 etc)
- CPU core model ("model" on x86, "CPU part" on arm)
- number of SMT per core
- speed if known:
- /sys/devices/system/cpu/cpu0/acpi_cppc/nominal_perf if available
- or /sys/devices/system/cpu/cpu15/cpufreq/scaling_max_freq +/- 10%

PB: on intel P+E, clusters of E cores sharing the same L2+L3, but P cores are
alone on their L3 => poor grouping.

Maybe one approach could be to characterize how L3/L2 are used. E.g. on the
14900, we have:
- L3 0 => all cpus there
- L2 0..7 => 1C2T per L2
- L2 8..11 => 4C4T per L2
=> it's obvious that CPUs connected to L2 #8..11 are not the same as those
on L2 #0..7. We could make something with them.
=> it does not make sense to ditch the L2 distinction due to L3 being
present and the same, though it doesn't make sense to use L3 either.
Maybe elements with a cardinality of 1 should just be ignored. E.g.
cores per cache == 1 => ignore L2. Probably not true per die/pkg
though.
=> replace absent or irrelevant info with "?"

Note that for caches we have the list of CPUs, not the list of cores, so
we need to remap that invidivually to cores.

Warning: die_id, core_id etc are per socket, not per system. Worse, on Altra,
core_id has gigantic values (multiples of +1 and +256). However core_cpus_list
indicates other threads and could be a solution to create our own global core
ID. Also, cluster_id=-1 found on all cores for A8040 on kernel 6.1.

Note that LLC is always the first discriminator. But within a same LLC we can
have the issues above (e.g. 14900).

Would an intermediate approach like this work ?
-----------------------------------------------
1) first split by LLC (also test with L3-less A8040, N2800, x5-8350)
2) within LLC, check of we have different cores (model, perf, freq?)
and resplit
3) divide again so that no group has more than 64 CPUs

=> it looks like from the beginning that's what we're trying to do:
preserve locality first, then possibly trim down the number of cores
if some don't bring sufficient benefit. It possibly avoids the need
to identify dies etc. It still doesn't completely solve the 14900
though.

Multi-die CPUs worth checking:
Pentium-D (Presler, Dempsey: two 65nm dies)
Core2Quad Q6600/Q6700 (Kentsfield, Clowertown: two 65nm dual-core dies)
Core2Quad Q8xxx/Q9xxx (Yorkfield, Harpertown, Tigerton: two 45nm dual-core dies)
- atom 330 ("diamondville") is really a dual-die
- note that atom x3-z8350 ("cherry trail"), N2800 ("cedar trail") and D510
("pine trail") are single-die (verified) but have two L2 caches and no L3.
Note that these are apparently not identified as multi-die (Q6600 has die=0).

It seems that in order to form groups we'll first have to sort by topology,
and only after that sort by performance so as to choose preferred CPUs.
Otherwise we could end up trying to form inter-socket CPU groups first in
case we're forced to mix adjacent CPUs due to too many groups.

2025-01-07
----------

What is needed in fact is to act on two directions:

- binding restrictions: the user doesn't want the process to run on
second node, on efficient cores, second thread of each core, so
they're indicating where (not) to bind. This is a strict choice,
and it overrides taskset. That's the process-wide cpu-map.

- user preferences / execution profile: the user expresses their wishes
about how to allocate resources. This is only a binding order strategy
among a few existing ones that help easily decide which cores to select.
In this case CPUs are not enumerated. We can imagine choices as:

- full : use all permitted cores
- performance: use all permitted performance cores (all sockets)
- single-node: (like today): use all cores of a single node
- balanced: use a reasonable amount of perf cores (e.g. all perf
cores of a single socket)
- resources: use a single cluster of efficient cores
- minimal: use a single efficient core

By sorting CPUs first on the performance, then applying the filtering based on
the profile to eliminate more CPUs, then applying the limit on the desired max
number of threads, then sorting again on the topology, it should be possible to
draw a list of usable CPUs that can then be split in groups along the L3s.

It even sounds likely that the CPU profile or allocation strategy will affect
the first sort method. E.g:
- full: no sort needed though we'll use the same as perf so as to enable
the maximum possible high-perf threads when #threads is limited
- performance: probably that we should invert the topology so as to maximize
memory bandwidth across multiple sockets, i.e. visite node1.core0 just
after node0.core0 etc, and visit their threads later.
- bandwidth: that could be the same as "performance" one above in fact
- (low-)latency: better stay local first
- balanced: sort by perf then sockets (i.e. P0, P1, E0, E1)
- resources: sort on perf first.
- etc

The strategy will also help determine the number of threads when it's not fixed
in the configuration.

Plan:
1) make the profile configurable and implement the sort:
- option name? cpu-tuning, cpu-strategy, cpu-policy, cpu-allocation,
cpu-selection, cpu-priority, cpu-optimize-for, cpu-prefer, cpu-favor,
cpu-profile
=> cpu-selection

2) make the process-wide cpu-map configurable
3) extend cpu-map to make it possible to designate symbolic groups
(e.g. "ht0/ht1, node 0, 3*CCD, etc)

Also, offering an option to the user to see how haproxy sees the CPUs and the
bindings for various profiles would be a nice improvement helping them make
educated decisions instead of trying blindly.

2025-01-11
----------

Configuration profile: there are multiple dimensions:
- preferences between cores types
- never use a given cpu type
- never use a given cpu location

Better use something like:
- ignore-XXX -> never use XXX
- avoid-XXX -> prefer not to use XXX
- prefer-XXX -> prefer to use XXX
- restrict-XXX -> only use XXX

"XXX" could be "single-threaded", "dual-threaded", "first-thread",
"second-thread", "first-socket", "second-socket", "slowest", "fastest",
"node-XXX" etc.

We could then have:
- cpu-selection restrict-first-socket,ignore-slowest,...

Then some of the keywords could simply be shortcuts for these.

2025-01-30
----------
Problem: we need to set the restrictions first to eliminate undesired CPUs,
then sort according to the desired preferences so as to pick what
is considered the best CPUs. So the preference really looks like
a different setting.

More precisely, the final strategy involves multiple criteria. For example,
let's say that the number of threads is set to 4 and we've restricted ourselves
to using the first thread of each CPU core. We're on an EPYC74F3, there are 3
cores per CCX. One algorithm (resource) would create one group with 3 threads
on the first CCX and 1 group of 1 thread on the next one, then let each of
these threads bind to all the enabled CPU cores of their respective groups.
Another algo (performance) would avoid sharing and would want to place one
thread per CCX, causing the creation of 4 groups of 1 thread each. A third
algo (balanced) would probably say that 4 threads require 2 CCX hence 2
groups, thus there should be 2 threads per group, and it would bind 2 threads
on all cores of the first CCX and the 2 remaining ones on the second.

And if the thread count is not set, these strategies will also do their best
to figure the optimal count. Resource would probably use 1 core max, moderate
one CCX max, balanced one node max, performance all of them.

This means that these CPU selection strategries should provide multiple
functions:
- how to sort CPUs
- how to count how many is best within imposed rules

The other actions seem to only be static. This also means that "avoid" or
"prefer" should maybe not be used in the end, even in the sorting algo ?

Or maybe these are just enums or bits in a strategy and all are considered
at the same time everywhere. For example the thread counting could consider
the presence of "avoid-XXX" during the operations. But how to codify XXX is
complicated then.

Maybe a scoring system could work:
- default: all CPUs score = 1000
- ignore-XXX: foreach(XXX) set score to 0
- restrict-XXX: foreach(YYY not XXX), set score to 0
- avoid-XXX: foreach(XXX) score *= 0.8
- prefer-XXX: foreach(XXX) score *= 1.25

This supports being ignored for up to 30 different reasons before being
permanently disabled, which is sufficient.

Then sort according to score, and pick at least min_thr CPUs and continue as
long as not max_thr or score < 1000 ("avoid"). This gives the thread count. It
does not permit anything inter-CPU though. E.g. large vs medium vs small cores,
or sort by locality or frequency. But maybe these ones would use a different
strategy then and would use the score as a second sorting key (after which
one?). Or maybe there would be 2 passes, one which avoids <1000 and another
one which completes up to #min_thr including those <1000, in which case we
never sort per score.

We can do a bit better to respect the tgrp min/max as well: we can count what
it implies in terms of number of tgrps (#LLC or clusters) and decide to refrain
from adding theads which would exceed max_tgrp, but we'd possibly continue to
add score<1000 CPUs until at least enough threads to reach min_tgrp.

######## new captures ###########
CIX-P1 / radxa Orion O6 (no topology exported):
$ ~/haproxy/haproxy -dc -f /dev/null
grp=[1..12] thr=[1..12]
first node = 0
Note: threads already set to 12
going to start with nbthread=12 nbtgroups=1
[keep] thr= 0 -> cpu= 0 pk=00 no=-1 di=00 cl=000 ts=000 capa=1024
[keep] thr= 1 -> cpu= 1 pk=00 no=-1 di=00 cl=000 ts=001 capa=278
[keep] thr= 2 -> cpu= 2 pk=00 no=-1 di=00 cl=000 ts=002 capa=278
[keep] thr= 3 -> cpu= 3 pk=00 no=-1 di=00 cl=000 ts=003 capa=278
[keep] thr= 4 -> cpu= 4 pk=00 no=-1 di=00 cl=000 ts=004 capa=278
[keep] thr= 5 -> cpu= 5 pk=00 no=-1 di=00 cl=000 ts=005 capa=905
[keep] thr= 6 -> cpu= 6 pk=00 no=-1 di=00 cl=000 ts=006 capa=905
[keep] thr= 7 -> cpu= 7 pk=00 no=-1 di=00 cl=000 ts=007 capa=866
[keep] thr= 8 -> cpu= 8 pk=00 no=-1 di=00 cl=000 ts=008 capa=866
[keep] thr= 9 -> cpu= 9 pk=00 no=-1 di=00 cl=000 ts=009 capa=984
[keep] thr= 10 -> cpu= 10 pk=00 no=-1 di=00 cl=000 ts=010 capa=984
[keep] thr= 11 -> cpu= 11 pk=00 no=-1 di=00 cl=000 ts=011 capa=1024
########

2025-02-25 - clarification on the configuration
-----------------------------------------------

The "two dimensions" above can in fact be summarized like this:

- exposing the ability for the user to perform the same as "taskset",
i.e. restrict the usage to a static subset of the CPUs. We could then
have "cpu-set only-node0", "0-39", "ignore-smt1", "ignore-little", etc.
=> the user defines precise sets to be kept/evicted.

- then letting the user express what they want to do with the remaining
cores. This is a strategy/policy that is used to:
- count the optimal number of threads (when not forced), also keeping
in mind that it cannot be more than 32/64 * maxtgroups if set.
- sort CPUs by order of preference (for when threads are forced or
a thread-hard-limit is set).

It can, partially overlap with the first one. For example, the default
strategy could be to focus on a single node. If the user has limited its
usage to cores of both nodes, the policy could still further limit this.
But this time it should only be a matter of sorting and preference, i.e.
nbthread and cpuset are respected. If a policy prefers the node with more
cores first, it will sort them according to this, and its algorithm for
counting cores will only be used if nbthread is not set, otherwise it may
very well end up on two nodes to respect the user's choice.

And once all of this is done, thread groups should be formed based on the
remaining topology. Similarly, if the number of tgroups is not set, the
algorithm must try to propose one based on the topology and the maxtgroups
setting (i.e. find a divider of the #LLC that's lower than or equal to
maxtgroups), otherwise the configured number of tgroups is respected. Then
the number of LLCs will be divided by this number of tgroups, and as many
threads as enabled CPUs of each LLC will be assigned to these respective
groups.

In the end we should have groups bound to cpu sets, and threads belonging
to groups mapped to all accessible cpus of these groups.

Note: clusters may be finer than LLCs because they could report finer
information. We could have a big and a medium cluster share the same L3
for example. However not all boards report their cluster number (see CIX-P1
above). However the info about the capacity still allows to figure that and
should probably be used for that. At this point it would seem logical to say
that the cluster number is re-adjusted based on the claimed capacity, at
least to avoid accidentally mixing workloads on heterogeneous cores. But
sorting by cluster number might not necessarily work if allocated randomly.
So we might need a distinct metric that doesn't require to override the
system's numbering, like a "set", "group", "team", "bond", "bunch", "club",
"band", ... that would be first sorted based on LLC (and no finer), and
second based on capacity, then on L2 etc. This way we should be able to
respect topology when forming groups.

Note: We need to consider as LLC a level which has more than one core!
Otherwise it's supposed to exist and be unique/shared but not reported.
=> maybe this should be done very early when counting CPUs ?
We need to store the LLC level somewhere in the topo.

---

Doc/Design Thoughts/Pool Debugging

2022-02-22 - debugging options with pools

Two goals:
- help developers spot bugs as early as possible

- make the process more reliable in field, by killing sick ones as soon as
possible instead of letting them corrupt data, cause trouble, or even be
exploited.

An allocated object may exist in 5 forms:
- in use: currently referenced and used by haproxy, 100% of its size are
dedicated to the application which can do absolutely anything with it,
but it may never touch anything before nor after that area.

- in cache: the object is neither referenced nor used anymore, but it sits
in a thread's cache. The application may not touch it at all anymore, and
some parts of it could even be unmapped. Only the current thread may safely
reach it, though others might find/release it when under thread isolation.
The thread cache needs some LRU linking that may be stored anywhere, either
inside the area, or outside. The parts surrounding the <size> parts remain
invisible to the application layer, and can serve as a protection.

- in shared cache: the object is neither referenced nor used anymore, but it
may be reached by any thread. Some parts of it could be unmapped. Any
thread may pick it but only one may find it, hence once grabbed, it is
guaranteed no other one will find it. The shared cache needs to set up a
linked list and a single pointer needs to be stored anywhere, either inside
or outside the area. The parts surrounding the <size> parts remain
invisible to the application layer, and can serve as a protection.

- in the system's memory allocator: the object is not known anymore from
haproxy. It may be reassigned in parts or totally to other pools or other
subsystems (e.g. crypto library). Some or all of it may be unmapped. The
areas surrounding the <size> parts are also part of the object from the
library's point of view and may be delivered to other areas. Tampering
with these may cause any other part to malfunction in dirty ways.

- in the OS only: the memory allocator gave it back to the OS.

The following options need to be configurable:
- detect improper initialization: this is done by poisonning objects before
delivering them to the application.

- help figure where an object was allocated when in use: a pointer to the
call place will help. Pointing to the last pool_free() as well for the
same reasons when dealing with a UAF.

- detection of wrong pointer/pool when in use: a pointer to the pool before
or after the area will definitely help.

- detection of overflows when in use: a canary at the end of the area
(closest possible to <size>) will definitely help. The pool above can do
that job. Ideally, we should fill some data at the end so that even
unaligned sizes can be checked (e.g. a buffer that gets a zero appended).
If we just align on 2 pointers, writing the same pointer twice at the end
may do the job, but we won't necessarily have our bytes. Thus a particular
end-of-string pattern would be useful (e.g. ff55aa01) to fill it.

- detection of double free when in cache: similar to detection of wrong
pointer/pool when in use: the pointer at the end may simply be changed so
that it cannot match the pool anymore. By using a pointer to the caller of
the previous free() operation, we have the guarantee to see different
pointers, and this pointer can be inspected to figure where the object was
previously freed. An extra check may even distinguish a perfect double-free
(same caller) from just a wrong free (pointer differs from pool).

- detection of late corruption when in cache: keeping a copy of the
checksum of the whole area upon free() will do the job, but requires one
extra storage area for the checksum. Filling the area with a pattern also
does the job and doesn't require extra storage, but it loses the contents
and can be a bit slower. Sometimes losing the contents can be a feature,
especially when trying to detect late reads. Probably that both need to
be implemented. Note that if contents are not strictly needed, storing a
checksum inside the area does the job.

- preserve total contents in cache for debugging: losing some precious
information can be a problem.

- pattern filling of the area helps detect use-after-free in read-only mode.

- allocate cold first helps with both cases above.

Uncovered:
- overflow/underflow when in cache/shared/libc: it belongs to use-after-free
pattern and such an error during regular use ought to be caught while the
object was still in use.

- integrity when in libc: not under our control anymore, this is a libc
problem.

Arbitrable:
- integrity when in shared cache: unlikely to happen only then if it could
have happened in the local cache. Shared cache not often used anymore, thus
probably not worth the effort

- protection against double-free when in shared cache/libc: might be done for
a cheap price, probably worth being able to quickly tell that such an
object left the local cache (e.g. the mark points to the caller, but could
possibly just be incremented, hence still point to the same code location+1
byte when released. Calls are 4 bytes min on RISC, 5 on x86 so we do have
some margin by having a caller's location be +0,+1,+2 or +3.

- underflow when in use: hasn't been really needed over time but may change.

- detection of late corruption when in shared cache: checksum or area filling
are possible, but is this as relevant as it used to considering the less
common use of the shared cache ?

Design considerations:
- object allocation when in use must remain minimal

- when in cache, there are 2 lists which the compiler expect to be at least
aligned each (e.g. if/when we start to use DWCAS).

- the original "pool debugging" feature covers both pool tracking, double-
free detection, overflow detection and caller info at the cost of a single
pointer placed immediately after the area.

- preserving the contents might be done by placing the cache links and the
shared cache's list outside of the area (either before or after). Placing
it before has the merit that the allocated object preserves the 4-ptr
alignment. But when a larger alignment is desired this often does not work
anymore. Placing it after requires some dynamic adjustment depending on the
object's size. If any protection is installed, this protection must be
placed before the links so that the list doesn't get randomly corrupted and
corrupts adjacent elements. Note that if protection is desired, the extra
waste is probably less critical.

- a link to the last caller might have to be stored somewhere. Without
preservation the free() caller may be placed anywhere while the alloc()
caller may only be placed outside. With preservation, again the free()
caller may be placed either before the object or after the mark at the end.
There is no particular need that both share the same location though it may
help. Note that when debugging is enabled, the free() caller doesn't need
to be duplicated and can continue to serve as the double-free detection.
Thus maybe in the end we only need to store the caller to the last alloc()
but not the free() since if we want it it's available via the pool debug.

- use-after-free detection: contents may be erased on free() and checked on
alloc(), but they can also be checksummed on free() and rechecked on
alloc(). In the latter case we need to store a checksum somewhere. Note
that with pure checksum we don't know what part was modified, but seeing
previous contents can be useful.

Possibilities:

1) Linked lists inside the area:

V size alloc
---+------------------------------+-----------------+--
in use |##############################| (Pool) (Tracer) |
---+------------------------------+-----------------+--

---+--+--+------------------------+-----------------+--
in cache |L1|L2|########################| (Caller) (Sum) |
---+--+--+------------------------+-----------------+--
or:
---+--+--+------------------------+-----------------+--
in cache |L1|L2|###################(sum)| (Caller) |
---+--+--+------------------------+-----------------+--

---+-+----------------------------+-----------------+--
in global |N|XXXX########################| (Caller) |
---+-+----------------------------+-----------------+--

2) Linked lists before the the area leave room for tracer and pool before
the area, but the canary must remain at the end, however the area will
be more difficult to keep aligned:

V head size alloc
----+-+-+------------------------------+-----------------+--
in use |T|P|##############################| (canary) |
----+-+-+------------------------------+-----------------+--

--+-----+------------------------------+-----------------+--
in cache |L1|L2|##############################| (Caller) (Sum) |
--+-----+------------------------------+-----------------+--

------+-+------------------------------+-----------------+--
in global |N|##############################| (Caller) |
------+-+------------------------------+-----------------+--

3) Linked lists at the end of the area, might be shared with extra data
depending on the state:

V size alloc
---+------------------------------+-----------------+--
in use |##############################| (Pool) (Tracer) |
---+------------------------------+-----------------+--

---+------------------------------+--+--+-----------+--
in cache |##############################|L1|L2| (Caller) (Sum)
---+------------------------------+--+--+-----------+--

---+------------------------------+-+---------------+--
in global |##############################|N| (Caller) |
---+------------------------------+-+---------------+--

This model requires a little bit of alignment at the end of the area, which is
not incompatible with pattern filling and/or checksumming:
- preserving the area for post-mortem analysis means nothing may be placed
inside. In this case it could make sense to always store the last releaser.
- detecting late corruption may be done either with filling or checksumming,
but the simple fact of assuming a risk of corruption that needs to be
chased means we must not store the lists nor caller inside the area.

Some models imply dedicating some place when in cache:
- preserving contents forces the lists to be prefixed or appended, which
leaves unused places when in use. Thus we could systematically place the
pool pointer and the caller in this case.

- if preserving contents is not desired, almost everything can be stored
inside when not in use. Then each situation's size should be calculated
so that the allocated size is known, and entries are filled from the
beginning while not in use, or after the size when in use.

- if poisonning is requested, late corruption might be detected but then we
don't want the list to be stored inside at the risk of being corrupted.

Maybe just implement a few models:
- compact/optimal: put l1/l2 inside
- detect late corruption: fill/sum, put l1/l2 out
- preserve contents: put l1/l2 out
- corruption+preserve: do not fill, sum out
- poisonning: not needed on free if pattern filling is done.

try2:
- poison on alloc to detect missing initialization: yes/no
(note: nothing to do if filling done)
- poison on free to detect use-after-free: yes/no
(note: nothing to do if filling done)
- check on alloc for corruption-after-free: yes/no
If content-preserving => sum, otherwise pattern filling; in
any case, move L1/L2 out.
- check for overflows: yes/no: use a canary after the area. The
canary can be the pointer to the pool.
- check for alloc caller: yes/no => always after the area
- content preservation: yes/no
(disables filling, moves lists out)
- improved caller tracking: used to detect double-free, may benefit
from content-preserving but not only.

---

Doc/Design Thoughts/Ring V2

2024-02-20 - Ring buffer v2
===========================

Goals:
- improve the multi-thread performance of rings so that traces can be written
from all threads in parallel without the huge bottleneck of the lock that
is currently necessary to protect the buffer. This is important for mmapped
areas that are left as a file when the process crashes.

- keep traces synchronous within a given thread, i.e. when the TRACE() call
returns, the trace is either written into the ring or lost due to slow
readers.

- try hard to limit the cache line bounces between threads due to the use of
a shared work area.

- make waiting threads not disturb working ones

- continue to work on all supported platforms, with a particular focus on
performance for modern platforms (memory ordering, DWCAS etc can be used if
they provide any benefit), with a fallback for inferior platforms.

- do not reorder traces within a given thread.

- do not break existing features

- do not significantly increase memory usage

Analysis of the current situation
=================================

Currently, there is a read lock around the call to __sink_write() in order to
make sure that an attempt to write the number of lost messages is delivered
with highest priority and is consistent with the lost counter. This doesn't
seem to pose any problem at this point though if it were, it could possibly
be revisited.

__sink_write() calls ring_write() which first measures the input string length
from the multiple segments, and locks the ring:
- while trying to free space
- while copying the message, due to the buffer's API

Because of this, there is a huge serialization and threads wait in queue. Tests
involving a split of the lock and a release around the message copy have shown
a +60% performance increase, which is still not acceptable.

First proposed approach
=======================

The first approach would have consisted in writing messages in small parts:
1) write 0xFF in the tag to mean "size not filled yet"
2) write the message's length and write a zero tag after the message's
location
3) replace the first tag to 0xFE to indicate the size is known, but the
message is not filled yet.
4) memcpy() of the message to the area
5) replace the first tag to 0 to mark the entry as valid.

It's worth noting that doing that without any lock will allow a second thread
looping on the first tag to jump to the second tag after step 3. But the cost
is high: in a 64-thread scenario where each of them wants to send one message,
the work would look like this:
- 64 threads try to CAS the tag. One gets it, 63 fail. They loop on the byte
in question in read-only mode, waiting for the byte to change. This loop
constantly forces the cache line to switch from MODIFIED to SHARED in the
writer thread, and makes it a pain for it to write the message's length
just after it.

- once the first writer thread finally manages to write the length (step 2),
it writes 0xFE on the tag to release the waiting threads, and starts with
step 4. At this point, 63 threads try a CAS on the same entry, and this
hammering further complicates the memcpy() of step 4 for the first 63 bytes
of the message (well, 32 on avg since the tag is not necessarily aligned).
One thread wins, 62 fail. All read the size field and jump to the next tag,
waiting in read loops there. The second thread starts to write its size and
faces the same difficulty as described above, facing 62 competitors when
writing its size and the beginning of its message.

- when the first writer thread writes the end of its message, it gets close
to the final tag where the 62 waiting threads are still reading, causing
a slow down again with the loss of exclusivity on the cache line. This is
the same for the second thread etc.

Thus, on average, a writing thread is hindered by N-1 threads at the beginning
of its message area (in the first 32 bytes on avg) and by N-2 threads at the
end of its area (in the last 32 bytes on avg). Given that messages are roughly
218 bytes on avg for HTTP/1, this means that roughly 1/3 of the message is
written under severe cache contention.

In addition to this, the buffer's tail needs to be updated once all threads are
ready, something that adds the need for synchronization so that the last writing
threads (the most likely to complete fast due to less perturbations) needs to
wait for all previous ones. This also means N atomic writes to the tail.

New proposal
============

In order to address the contention scenarios above, let's try to factor the
work as much as possible. The principle is that threads that want to write will
either do it themselves or declare their intent and wait for a writing thread
to do it for them. This aims at ensuring a maximum usage of read-only data
between threads, and to leave the work area read-write between very few
threads, and exclusive for multiple messages at once, avoiding the bounces.

First, the buffer will have 2 indexes:
- head: where the valid data start
- tail: where new data need to be appended

When a thread starts to work, it will keep a copy of $tail and push it forward
by as many bytes as needed to write all the messages it has to. In order to
guarantee that neither the previous nor the new $tail point to an outdated or
overwritten location but that there is always a tag there, $tail contains a
lock bit in its highest bit that will guarantee that only one at a time will
update it. The goal here is to perform as few atomic ops as possible in the
contended path so as to later amortize the costs and make sure to limit the
number of atomic ops on the wait path to the strict minimum so that waiting
threads do not hinder the workers:

Fast path:
1 load($tail) to check the topmost bit
1 CAS($tail,$tail|BIT63) to set the bit (atomic_fetch_or / atomic_bts also work)
1 store(1 byte tag=0xFF) at the beginning to mark the area busy
1 store($tail) to update the new value
1 copy of the whole message
1 store(1 byte tag=0) at the beginning to release the message

Contented path:
N load($tail) while waiting for the bit to be zero
M CAS($tail,$tail|BIT63) to try to set the bit on tail, competing with others
1 store(1 byte tag=0xFF) at the beginning to mark the area busy
1 store($tail) to update the new value
1 copy of the whole message
1 store(1 byte tag=0) at the beginning to release the message

Queue
-----

In order to limit the contention, writers will not start to write but will wait
in a queue, announcing their message pointers/lengths and total lengths. The
queue is made of a (ptr, len) pair that points to one such descriptor, located
in the waiter thread's stack, that itself points to the next pair. In fact
messages are ordered in a LIFO fashion but that isn't important since intra-
thread ordering is preserved (and in the worst case it will also be possible
to write them from end to beginning).

The approach is the following: a writer loasd $tail and sees it's busy, there's
no point continuing, it will add itself to the queue, announcing (ptr, len +
next->len) so that by just reading the first entry, one knows the total size
of the queue. And it will wait there as long as $tail has its topmost bit set
and the queue points to itself (meaning it's the queue's leader), so that only
one thread in the queue watches $tail, limiting the number of cache line
bounces. If the queue doesn't point anymore to the current thread, it means
another thread has taken it over so there's no point continuing, this thread
just becomes passive. If the lock bit is dropped from $tail, the watching
thread needs to re-check that it's still the queue's leader before trying to
grab the lock, so that only the leading thread will attempt it. Indeed, a few
of the last leading threads might still be looping, unaware that they're no
longer leaders. A CAS(&queue, self, self) will do it. Upon failure, the thread
just becomes a passive thread. Upon success, the thread is a confirmed leader,
it must then try to grab the tail lock. Only this thread and a few potential
newcomers will compete on this one. If the leading thread wins, it brings all
the queue with it and the newcomers will queue again. If the leading thread
loses, it needs to loop back to the point above, watching $tail and the
queue. In this case a newcomer might have grabbed the lock. It will notice
the non-empty queue and will take it with it. Thus in both cases the winner
thread does a CAS(queue, queue, NULL) to reset the queue, keeping the previous
pointer.

At this point the winner thread considers its own message size plus the
retrieved queue's size as the total required size and advances $tail by as
much, and will iterate over all messages to copy them in turn. The passive
threads are released by doing XCHG(&ptr->next, ptr) for each message, that
is normally impossible otherwise. As such, a passive thread just has to
loop over its own value, stored in its own stack, reading from its L1 cache
in loops without any risk of disturbing others, hence no need for EBO.

During the time it took to update $tail, more messages will have been
accumulating in the queue from various other threads, and once $tail is
written, one thread can pick them up again.

The benefit here is that the longer it takes one thread to free some space,
the more messages add up in the queue and the larger the next batch, so that
there are always very few contenders on the ring area and on the tail index.
At worst, the queue pointer is hammered but it's not on the fast path, since
wasting time here means all waiters will be queued.

Also, if we keep the first tag unchanged after it's set to 0xFF, it allows to
avoid atomic ops inside all the message. Indeed there's no reader in the area
as long as the tag is 0xFF, so we can just write all contents at once including
the varints and subsequent message tags without ever using atomic ops, hence
not forcing ordered writes. So maybe in the end there is some value in writing
the messages backwards from end to beginning, and just writing the first tag
atomically but not the rest.

The scenario would look like this:

(without queue)

- before starting to work:
do {
while (ret=(load(&tail) & BIT63))
;
} while (!cas(&tail, &ret, ret | BIT63));

- at this point, alone on it and guaranteed not to change
- after new size is calculated, write it and drop the lock:

store(&tail, new_tail & ~BIT63);

- that's sufficient to unlock other waiters.

(with queue)

in_queue = 0;
do {
ret = load(&tail);
if (ret & BIT63) {
if (!in_queue) {
queue_this_node();
in_queue = 1;
}
while (ret & BIT63)
;
}
} while (!cas(&tail, &ret, ret | BIT63));

dequeue(in_queue) etc.

Fast path:
1 load($tail) to check the topmost bit
1 CAS($tail,$tail|BIT63) to set the bit (atomic_fetch_or / atomic_bts also work)
1 load of the queue to see that it's empty
1 store(1 byte tag=0xFF) at the beginning to mark the area busy
1 store($tail) to update the new value
1 copy of the whole message
1 store(1 byte tag=0) at the beginning to release the message

Contented path:
1 load($tail) to see the tail is changing
M CAS(queue,queue,self) to try to add the thread to the queue (avgmax nbthr/2)
N load($tail) while waiting for the lock bit to become zero
1 CAS(queue,self,self) to check the leader still is
M CAS($tail,$tail|BIT63) to try to set the bit on tail, competing with others
1 CAS(queue,queue,NULL) to reset the queue
1 store(1 byte tag=0xFF) at the beginning to mark the area busy
1 store($tail) to update the new value
1 copy of the whole message
P copies of individual messages
P stores of individual pointers to release writers
1 store(1 byte tag=0) at the beginning to release the message

Optimal approach (later if needed?): multiple queues. Each thread has one queue
assigned, either from a thread group, or using a modulo from the thread ID.
Same as above then.

Steps
-----

It looks that the queue is what allows the process to scale by amortizing a
single lock for every N messages, but that it's not a prerequisite to start,
without a queue threads can just wait on $tail.

Options
-------

It is possible to avoid the extra check on CAS(queue,self,self) by forcing
writers into the queue all the time. It would slow down the fast path but
may improve the slow path, both of which would become the same:

Contented path:
1 XCHG(queue,self) to try to add the thread to the queue
N load($tail) while waiting for the lock bit to become zero
M CAS($tail,$tail|BIT63) to try to set the bit on tail, competing with others
1 CAS(queue,self,NULL) to reset the queue
1 store(1 byte tag=0xFF) at the beginning to mark the area busy
1 store($tail) to update the new value
1 copy of the whole message
P copies of individual messages
P stores of individual pointers to release writers
1 store(1 byte tag=0) at the beginning to release the message

There seems to remain a race when resetting the queue, where a newcomer thread
would queue itself while not being the leader. It seems it can be addressed by
deciding that whoever gets the bit is not important, what matters is the thread
that manages to reset the queue. This can then be done using another XCHG:

1 XCHG(queue,self) to try to add the thread to the queue
N load($tail) while waiting for the lock bit to become zero
M CAS($tail,$tail|BIT63) to try to set the bit on tail, competing with others
1 XCHG(queue,NULL) to reset the queue
1 store(1 byte tag=0xFF) at the beginning to mark the area busy
1 store($tail) to update the new value
1 copy of the whole message
P copies of individual messages
P stores of individual pointers to release writers
1 store(1 byte tag=0) at the beginning to release the message

However this time this can cause fragmentation of multiple sub-queues that will
need to be reassembled. So finally the CAS is better, the leader thread should
recognize itself.

It seems tricky to reliably store the next pointer in each element, and a DWCAS
wouldn't help here either. Maybe uninitialized elements should just have a
special value (eg 0x1) for their next pointer, meaning "not initialized yet",
and that the thread will then replace with the previous queue pointer. A reader
would have to wait on this value when meeting it, knowing the pointer is not
filled yet but is coming.

---

Doc/Design Thoughts/Thread Group

Thread groups
#############

2021-07-13 - first draft
==========

Objective
---------
- support multi-socket systems with limited cache-line bouncing between
physical CPUs and/or L3 caches

- overcome the 64-thread limitation

- Support a reasonable number of groups. I.e. if modern CPUs arrive with
core complexes made of 8 cores, with 8 CC per chip and 2 chips in a
system, it makes sense to support 16 groups.

Non-objective
-------------
- no need to optimize to the last possible cycle. I.e. some algos like
leastconn will remain shared across all threads, servers will keep a
single queue, etc. Global information remains global.

- no stubborn enforcement of FD sharing. Per-server idle connection lists
can become per-group; listeners can (and should probably) be per-group.
Other mechanisms (like SO_REUSEADDR) can already overcome this.

- no need to go beyond 64 threads per group.

Identified tasks
================

General
-------
Everywhere tid_bit is used we absolutely need to find a complement using
either the current group or a specific one. Thread debugging will need to
be extended as masks are extensively used.

Scheduler
---------
The global run queue and global wait queue must become per-group. This
means that a task may only be queued into one of them at a time. It
sounds like tasks may only belong to a given group, but doing so would
bring back the original issue that it's impossible to perform remote wake
ups.

We could probably ignore the group if we don't need to set the thread mask
in the task. the task's thread_mask is never manipulated using atomics so
it's safe to complement it with a group.

The sleeping_thread_mask should become per-group. Thus possibly that a
wakeup may only be performed on the assigned group, meaning that either
a task is not assigned, in which case it be self-assigned (like today),
otherwise the tg to be woken up will be retrieved from the task itself.

Task creation currently takes a thread mask of either tid_bit, a specific
mask, or MAX_THREADS_MASK. How to create a task able to run anywhere
(checks, Lua, ...) ?

Profiling -> completed
---------
There should be one task_profiling_mask per thread group. Enabling or
disabling profiling should be made per group (possibly by iterating).
-> not needed anymore, one flag per thread in each thread's context.

Thread isolation
----------------
Thread isolation is difficult as we solely rely on atomic ops to figure
who can complete. Such operation is rare, maybe we could have a global
read_mostly flag containing a mask of the groups that require isolation.
Then the threads_want_rdv_mask etc can become per-group. However setting
and clearing the bits will become problematic as this will happen in two
steps hence will require careful ordering.

FD
--
Tidbit is used in a number of atomic ops on the running_mask. If we have
one fdtab[] per group, the mask implies that it's within the group.
Theoretically we should never face a situation where an FD is reported nor
manipulated for a remote group.

There will still be one poller per thread, except that this time all
operations will be related to the current thread_group. No fd may appear
in two thread_groups at once, but we can probably not prevent that (e.g.
delayed close and reopen). Should we instead have a single shared fdtab[]
(less memory usage also) ? Maybe adding the group in the fdtab entry would
work, but when does a thread know it can leave it ? Currently this is
solved by running_mask and by update_mask. Having two tables could help
with this (each table sees the FD in a different group with a different
mask) but this looks overkill.

There's polled_mask[] which needs to be decided upon. Probably that it
should be doubled as well. Note, polled_mask left fdtab[] for cacheline
alignment reasons in commit cb92f5cae4.

If we have one fdtab[] per group, what really prevents from using the
same FD in multiple groups ? _fd_delete_orphan() and fd_update_events()
need to check for no-thread usage before closing the FD. This could be
a limiting factor. Enabling could require to wake every poller.

Shouldn't we remerge fdinfo[] with fdtab[] (one pointer + one int/short,
used only during creation and close) ?

Other problem, if we have one fdtab[] per TG, disabling/enabling an FD
(e.g. pause/resume on listener) can become a problem if it's not necessarily
on the current TG. We'll then need a way to figure that one. It sounds like
FDs from listeners and receivers are very specific and suffer from problems
all other ones under high load do not suffer from. Maybe something specific
ought to be done for them, if we can guarantee there is no risk of accidental
reuse (e.g. locate the TG info in the receiver and have a "MT" bit in the
FD's flags). The risk is always that a close() can result in instant pop-up
of the same FD on any other thread of the same process.

Observations: right now fdtab[].thread_mask more or less corresponds to a
declaration of interest, it's very close to meaning "active per thread". It is
in fact located in the FD while it ought to do nothing there, as it should be
where the FD is used as it rules accesses to a shared resource that is not
the FD but what uses it. Indeed, if neither polled_mask nor running_mask have
a thread's bit, the FD is unknown to that thread and the element using it may
only be reached from above and not from the FD. As such we ought to have a
thread_mask on a listener and another one on connections. These ones will
indicate who uses them. A takeover could then be simplified (atomically set
exclusivity on the FD's running_mask, upon success, takeover the connection,
clear the running mask). Probably that the change ought to be performed on
the connection level first, not the FD level by the way. But running and
polled are the two relevant elements, one indicates userland knowledge,
the other one kernel knowledge. For listeners there's no exclusivity so it's
a bit different but the rule remains the same that we don't have to know
what threads are interested in the FD, only its holder.

Not exact in fact, see FD notes below.

activity
--------
There should be one activity array per thread group. The dump should
simply scan them all since the cumuled values are not very important
anyway.

applets
-------
They use tid_bit only for the task. It looks like the appctx's thread_mask
is never used (now removed). Furthermore, it looks like the argument is
always tid_bit.

CPU binding
-----------
This is going to be tough. It will be needed to detect that threads overlap
and are not bound (i.e. all threads on same mask). In this case, if the number
of threads is higher than the number of threads per physical socket, one must
try hard to evenly spread them among physical sockets (e.g. one thread group
per physical socket) and start as many threads as needed on each, bound to
all threads/cores of each socket. If there is a single socket, the same job
may be done based on L3 caches. Maybe it could always be done based on L3
caches. The difficulty behind this is the number of sockets to be bound: it
is not possible to bind several FDs per listener. Maybe with a new bind
keyword we can imagine to automatically duplicate listeners ? In any case,
the initially bound cpumap (via taskset) must always be respected, and
everything should probably start from there.

Frontend binding
----------------
We'll have to define a list of threads and thread-groups per frontend.
Probably that having a group mask and a same thread-mask for each group
would suffice.

Threads should have two numbers:
- the per-process number (e.g. 1..256)
- the per-group number (1..64)

The "bind-thread" lines ought to use the following syntax:
- bind 45 ## bind to process' thread 45
- bind 1/45 ## bind to group 1's thread 45
- bind all/45 ## bind to thread 45 in each group
- bind 1/all ## bind to all threads in group 1
- bind all ## bind to all threads
- bind all/all ## bind to all threads in all groups (=all)
- bind 1/65 ## rejected
- bind 65 ## OK if there are enough
- bind 35-45 ## depends. Rejected if it crosses a group boundary.

The global directive "nbthread 28" means 28 total threads for the process. The
number of groups will sub-divide this. E.g. 4 groups will very likely imply 7
threads per group. At the beginning, the nbgroup should be manual since it
implies config adjustments to bind lines.

There should be a trivial way to map a global thread to a group and local ID
and to do the opposite.

Panic handler + watchdog
------------------------
Will probably depend on what's done for thread_isolate

Per-thread arrays inside structures
-----------------------------------
- listeners have a thr_conn[] array, currently limited to MAX_THREADS. Should
we simply bump the limit ?
- same for servers with idle connections.
=> doesn't seem very practical.
- another solution might be to point to dynamically allocated arrays of
arrays (e.g. nbthread * nbgroup) or a first level per group and a second
per thread.
=> dynamic allocation based on the global number

Other
-----
- what about dynamic thread start/stop (e.g. for containers/VMs) ?
E.g. if we decide to start $MANY threads in 4 groups, and only use
one, in the end it will not be possible to use less than one thread
per group, and at most 64 will be present in each group.

FD Notes
--------
- updt_fd_polling() uses thread_mask to figure where to send the update,
the local list or a shared list, and which bits to set in update_mask.
This could be changed so that it takes the update mask in argument. The
call from the poller's fork would just have to broadcast everywhere.

- pollers use it to figure whether they're concerned or not by the activity
update. This looks important as otherwise we could re-enable polling on
an FD that changed to another thread.

- thread_mask being a per-thread active mask looks more exact and is
precisely used this way by _update_fd(). In this case using it instead
of running_mask to gauge a change or temporarily lock it during a
removal could make sense.

- running should be conditioned by thread. Polled not (since deferred
or migrated). In this case testing thread_mask can be enough most of
the time, but this requires synchronization that will have to be
extended to tgid.. But migration seems a different beast that we shouldn't
care about here: if first performed at the higher level it ought to
be safe.

In practice the update_mask can be dropped to zero by the first fd_delete()
as the only authority allowed to fd_delete() is the owner, and as soon as
all running_mask are gone, the FD will be closed, hence removed from all
pollers. This will be the only way to make sure that update_mask always
refers to the current tgid.

However, it may happen that a takeover within the same group causes a thread
to read the update_mask late, while the FD is being wiped by another thread.
That other thread may close it, causing another thread in another group to
catch it, and change the tgid and start to update the update_mask. This means
that it would be possible for a thread entering do_poll() to see the correct
tgid, then the fd would be closed, reopened and reassigned to another tgid,
and the thread would see its bit in the update_mask, being confused. Right
now this should already happen when the update_mask is not cleared, except
that upon wakeup a migration would be detected and that would be all.

Thus we might need to set the running bit to prevent the FD from migrating
before reading update_mask, which also implies closing on fd_clr_running() == 0 :-(

Also even fd_update_events() leaves a risk of updating update_mask after
clearing running, thus affecting the wrong one. Probably that update_mask
should be updated before clearing running_mask there. Also, how about not
creating an update on a close ? Not trivial if done before running, unless
thread_mask==0.

Note that one situation that is currently visible is that a thread closes a
file descriptor that it's the last one to own and to have an update for. In
fd_delete_orphan() it does call poller.clo() but this one is not sufficient
as it doesn't drop the update_mask nor does it clear the polled_mask. The
typical problem that arises is that the close() happens before processing
the last update (e.g. a close() just after a partial read), thus it still
has at least one bit set for the current thread in both update_mask and
polled_mask, and it is present in the update_list. Not handling it would
mean that the event is lost on update() from the concerned threads and
that some resource might leak. Handling it means zeroing the update_mask
and polled_mask, and deleting the update entry from the update_list, thus
losing the update event. And as indicated above, if the FD switches twice
between 2 groups, the finally called thread does not necessarily know that
the FD isn't the same anymore, thus it's difficult to decide whether to
delete it or not, because deleting the event might in fact mean deleting
something that was just re-added for the same thread with the same FD but
a different usage.

Also it really seems unrealistic to scan a single shared update_list like
this using write operations. There should likely be one per thread-group.
But in this case there is no more choice than deleting the update event
upon fd_delete_orphan(). This also means that poller->clo() must do the
job for all of the group's threads at once. This would mean a synchronous
removal before the close(), which doesn't seem ridiculously expensive. It
just requires that any thread of a group may manipulate any other thread's
status for an FD and a poller.

Note about our currently supported pollers:

- epoll: our current code base relies on the modern version which
automatically removes closed FDs, so we don't have anything to do
when closing and we don't need the update.

- kqueue: according to https://www.freebsd.org/cgi/man.cgi?query=kqueue, just
like epoll, a close() implies a removal. Our poller doesn't perform
any bookkeeping either so it's OK to directly close.

- evports: https://docs.oracle.com/cd/E86824_01/html/E54766/port-dissociate-3c.html
says the same, i.e. close() implies a removal of all events. No local
processing nor bookkeeping either, we can close.

- poll: the fd_evts[] array is global, thus shared by all threads. As such,
a single removal is needed to flush it for all threads at once. The
operation is already performed like this.

- select: works exactly like poll() above, hence already handled.

As a preliminary conclusion, it's safe to delete the event and reset
update_mask just after calling poller->clo(). If extremely unlucky (changing
thread mask due to takeover ?), the same FD may appear at the same time:
- in one or several thread-local fd_updt[] arrays. These ones are just work
queues, there's nothing to do to ignore them, just leave the holes with an
outdated FD which will be ignored once met. As a bonus, poller->clo() could
check if the last fd_updt[] points to this specific FD and decide to kill
it.

- in the global update_list. In this case, fd_rm_from_fd_list() already
performs an attachment check, so it's safe to always call it before closing
(since no one else may be in the process of changing anything).

###########################################################

Current state:

Mux / takeover / fd_delete() code ||| poller code
-------------------------------------------------|||---------------------------------------------------
\|/
mux_takeover(): | fd_set_running():
if (fd_takeover()<0) | old = {running, thread};
return fail; | new = {tid_bit, tid_bit};
... |
fd_takeover(): | do {
atomic_or(running, tid_bit); | if (!(old.thread & tid_bit))
old = {running, thread}; | return -1;
new = {tid_bit, tid_bit}; | new = { running | tid_bit, old.thread }
if (owner != expected) { | } while (!dwcas({running, thread}, &old, &new));
atomic_and(running, ~tid_bit); |
return -1; // fail | fd_clr_running():
} | return atomic_and_fetch(running, ~tid_bit);
|
while (old == {tid_bit, !=0 }) | poll():
if (dwcas({running, thread}, &old, &new)) { | if (!owner)
atomic_and(running, ~tid_bit); | continue;
return 0; // success |
} | if (!(thread_mask & tid_bit)) {
} | epoll_ctl_del();
| continue;
atomic_and(running, ~tid_bit); | }
return -1; // fail |
| // via fd_update_events()
fd_delete(): | if (fd_set_running() != -1) {
atomic_or(running, tid_bit); | iocb();
atomic_store(thread, 0); | if (fd_clr_running() == 0 && !thread_mask)
if (fd_clr_running(fd) = 0) | fd_delete_orphan();
fd_delete_orphan(); | }

The idle_conns_lock prevents the connection from being picked and released
while someone else is reading it. What it does is guarantee that on idle
connections, the caller of the IOCB will not dereference the task's context
while the connection is still in the idle list, since it might be picked then
freed at the same instant by another thread. As soon as the IOCB manages to
get that lock, it removes the connection from the list so that it cannot be
taken over anymore. Conversely, the mux's takeover() code runs under that
lock so that if it frees the connection and task, this will appear atomic
to the IOCB. The timeout task (which is another entry point for connection
deletion) does the same. Thus, when coming from the low-level (I/O or timeout):
- task always exists, but ctx checked under lock validates; conn removal
from list prevents takeover().
- t->context is stable, except during changes under takeover lock. So
h2_timeout_task may well run on a different thread than h2_io_cb().

Coming from the top:
- takeover() done under lock() clears task's ctx and possibly closes the FD
(unless some running remains present).

Unlikely but currently possible situations:
- multiple pollers (up to N) may have an idle connection's FD being
polled, if the connection was passed from thread to thread. The first
event on the connection would wake all of them. Most of them would
see fdtab[].owner set (the late ones might miss it). All but one would
see that their bit is missing from fdtab[].thread_mask and give up.
However, just after this test, others might take over the connection,
so in practice if terribly unlucky, all but 1 could see their bit in
thread_mask just before it gets removed, all of them set their bit
in running_mask, and all of them call iocb() (sock_conn_iocb()).
Thus all of them dereference the connection and touch the subscriber
with no protection, then end up in conn_notify_mux() that will call
the mux's wake().

- multiple pollers (up to N-1) might still be in fd_update_events()
manipulating fdtab[].state. The cause is that the "locked" variable
is determined by atleast2(thread_mask) but that thread_mask is read
at a random instant (i.e. it may be stolen by another one during a
takeover) since we don't yet hold running to prevent this from being
done. Thus we can arrive here with thread_mask==something_else (1bit),
locked==0 and fdtab[].state assigned non-atomically.

- it looks like nothing prevents h2_release() from being called on a
thread (e.g. from the top or task timeout) while sock_conn_iocb()
dereferences the connection on another thread. Those killing the
connection don't yet consider the fact that it's an FD that others
might currently be waking up on.

###################

pb with counter:

users count doesn't say who's using the FD and two users can do the same
close in turn. The thread_mask should define who's responsible for closing
the FD, and all those with a bit in it ought to do it.

2021-08-25 - update with minimal locking on tgid value
==========

- tgid + refcount at once using CAS
- idle_conns lock during updates
- update:
if tgid differs => close happened, thus drop update
otherwise normal stuff. Lock tgid until running if needed.
- poll report:
if tgid differs => closed
if thread differs => stop polling (migrated)
keep tgid lock until running
- test on thread_id:
if (xadd(&tgid,65536) != my_tgid) {
// was closed
sub(&tgid, 65536)
return -1
}
if !(thread_id & tidbit) => migrated/closed
set_running()
sub(tgid,65536)
- note: either fd_insert() or the final close() ought to set
polled and update to 0.

2021-09-13 - tid / tgroups etc.
==========

* tid currently is the thread's global ID. It's essentially used as an index
for arrays. It must be clearly stated that it works this way.

* tasklets use the global thread id, and __tasklet_wakeup_on() must use a
global ID as well. It's capital that tinfo[] provides instant access to
local/global bits/indexes/arrays

- tid_bit makes no sense process-wide, so it must be redefined to represent
the thread's tid within its group. The name is not much welcome though, but
there are 286 of it that are not going to be changed that fast.
=> now we have ltid and ltid_bit in thread_info. thread-local tid_bit still
not changed though. If renamed we must make sure the older one vanishes.
Why not rename "ptid, ptid_bit" for the process-wide tid and "gtid,
gtid_bit" for the group-wide ones ? This removes the ambiguity on "tid"
which is half the time not the one we expect.

* just like "ti" is the thread_info, we need to have "tg" pointing to the
thread_group.

- other less commonly used elements should be retrieved from ti->xxx. E.g.
the thread's local ID.

- lock debugging must reproduce tgid

* task profiling must be made per-group (annoying), unless we want to add a
per-thread TH_FL_* flag and have the rare places where the bit is changed
iterate over all threads if needed. Sounds preferable overall.

* an offset might be placed in the tgroup so that even with 64 threads max
we could have completely separate tid_bits over several groups.
=> base and count now

2021-09-15 - bind + listen() + rx
==========

- thread_mask (in bind_conf->rx_settings) should become an array of
MAX_TGROUP longs.
- when parsing "thread 123" or "thread 2/37", the proper bit is set,
assuming the array is either a contiguous bitfield or a tgroup array.
An option RX_O_THR_PER_GRP or RX_O_THR_PER_PROC is set depending on
how the thread num was parsed, so that we reject mixes.
- end of parsing: entries translated to the cleanest form (to be determined)
- binding: for each socket()/bind()/listen()... just perform one extra dup()
for each tgroup and store the multiple FDs into an FD array indexed on
MAX_TGROUP. => allows to use one FD per tgroup for the same socket, hence
to have multiple entries in all tgroup pollers without requiring the user
to duplicate the bind line.

2021-09-15 - global thread masks
==========

Some global variables currently expect to know about thread IDs and it's
uncertain what must be done with them:
- global_tasks_mask / Mask of threads with tasks in the global runqueue /
=> touched under the rq lock. Change it per-group ? What exact use is made ?

- sleeping_thread_mask / Threads that are about to sleep in poll() /
=> seems that it can be made per group

- all_threads_mask: a bit complicated, derived from nbthread and used with
masks and with my_ffsl() to wake threads up. Should probably be per-group
but we might miss something for global.

- stopping_thread_mask: used in combination with all_threads_mask, should
move per-group.

- threads_harmless_mask: indicates all threads that are currently harmless in
that they promise not to access a shared resource. Must be made per-group
but then we'll likely need a second stage to have the harmless groups mask.
threads_idle_mask, threads_sync_mask, threads_want_rdv_mask go with the one
above. Maybe the right approach will be to request harmless on a group mask
so that we can detect collisions and arbiter them like today, but on top of
this it becomes possible to request harmless only on the local group if
desired. The subtlety is that requesting harmless at the group level does
not mean it's achieved since the requester cannot vouch for the other ones
in the same group.

In addition, some variables are related to the global runqueue:
__decl_aligned_spinlock(rq_lock); / spin lock related to run queue /
struct eb_root rqueue; / tree constituting the global run queue, accessed under rq_lock /
unsigned int grq_total; / total number of entries in the global run queue, atomic /
static unsigned int global_rqueue_ticks; / insertion count in the grq, use rq_lock /

2022-06-14 - progress on task affinity
==========

The particularity of the current global run queue is to be usable for remote
wakeups because it's protected by a lock. There is no need for a global run
queue beyond this, and there could already be a locked queue per thread for
remote wakeups, with a random selection at wakeup time. It's just that picking
a pending task in a run queue among a number is convenient (though it
introduces some excessive locking). A task will either be tied to a single
group or will be allowed to run on any group. As such it's pretty clear that we
don't need a global run queue. When a run-anywhere task expires, either it runs
on the current group's runqueue with any thread, or a target thread is selected
during the wakeup and it's directly assigned.

A global wait queue seems important for scheduled repetitive tasks however. But
maybe it's more a task for a cron-like job and there's no need for the task
itself to wake up anywhere, because once the task wakes up, it must be tied to
one (or a set of) thread(s). One difficulty if the task is temporarily assigned
a thread group is that it's impossible to know where it's running when trying
to perform a second wakeup or when trying to kill it. Maybe we'll need to have
two tgid for a task (desired, effective). Or maybe we can restrict the ability
of such a task to stay in wait queue in case of wakeup, though that sounds
difficult. Other approaches would be to set the GID to the current one when
waking up the task, and to have a flag (or sign on the GID) indicating that the
task is still queued in the global timers queue. We already have TASK_SHARED_WQ
so it seems that antoher similar flag such as TASK_WAKE_ANYWHERE could make
sense. But when is TASK_SHARED_WQ really used, except for the "anywhere" case ?
All calls to task_new() use either 1<<thr, tid_bit, all_threads_mask, or come
from appctx_new which does exactly the same. The only real user of non-global,
non-unique task_new() call is debug_parse_cli_sched() which purposely allows to
use an arbitrary mask.

+----------------------------------------------------------------------------+
| => we don't need one WQ per group, only a global and N local ones, hence |
| the TASK_SHARED_WQ flag can continue to be used for this purpose. |
+----------------------------------------------------------------------------+

Having TASK_SHARED_WQ should indicate that a task will always be queued to the
shared queue and will always have a temporary gid and thread mask in the run
queue.

Going further, as we don't have any single case of a task bound to a small set
of threads, we could decide to wake up only expired tasks for ourselves by
looking them up using eb32sc and adopting them. Thus, there's no more need for
a shared runqueue nor a global_runqueue_ticks counter, and we can simply have
the ability to wake up a remote task. The task's thread_mask will then change
so that it's only a thread ID, except when the task has TASK_SHARED_WQ, in
which case it corresponds to the running thread. That's very close to what is
already done with tasklets in fact.

2021-09-29 - group designation and masks
==========

Neither FDs nor tasks will belong to incomplete subsets of threads spanning
over multiple thread groups. In addition there may be a difference between
configuration and operation (for FDs). This allows to fix the following rules:

group mask description
0 0 bind_conf: groups & thread not set. bind to any/all
task: it would be nice to mean "run on the same as the caller".

0 xxx bind_conf: thread set but not group: thread IDs are global
FD/task: group 0, mask xxx

G>0 0 bind_conf: only group is set: bind to all threads of group G
FD/task: mask 0 not permitted (= not owned). May be used to
mention "any thread of this group", though already covered by
G/xxx like today.

G>0 xxx bind_conf: Bind to these threads of this group
FD/task: group G, mask xxx

It looks like keeping groups starting at zero internally complicates everything
though. But forcing it to start at 1 might also require that we rescan all tasks
to replace 0 with 1 upon startup. This would also allow group 0 to be special and
be used as the default group for any new thread creation, so that group0.count
would keep the number of unassigned threads. Let's try:

group mask description
0 0 bind_conf: groups & thread not set. bind to any/all
task: "run on the same group & thread as the caller".

0 xxx bind_conf: thread set but not group: thread IDs are global
FD/task: invalid. Or maybe for a task we could use this to
mean "run on current group, thread XXX", which would cover
the need for health checks (g/t 0/0 while sleeping, 0/xxx
while running) and have wake_expired_tasks() detect 0/0 and
wake them up to a random group.

G>0 xxx bind_conf: Bind to these threads of this group
FD/task: group G, mask xxx

With a single group declared in the config, group 0 would implicitly find the
first one.

The problem with the approach above is that a task queued in one group+thread's
wait queue could very well receive a signal from another thread and/or group,
and that there is no indication about where the task is queued, nor how to
dequeue it. Thus it seems that it's up to the application itself to unbind/
rebind a task. This contradicts the principle of leaving a task waiting in a
wait queue and waking it anywhere.

Another possibility might be to decide that a task having a defined group but
a mask of zero is shared and will always be queued into its group's wait queue.
However, upon expiry, the scheduler would notice the thread-mask 0 and would
broadcast it to any group.

Right now in the code we have:
- 18 calls of task_new(tid_bit)
- 17 calls of task_new_anywhere()
- 2 calls with a single bit

Thus it looks like "task_new_anywhere()", "task_new_on()" and
"task_new_here()" would be sufficient.

---

Doc/Internals/Api/Appctx

Instantiation of applet contexts (appctx) in 2.6.

1. Background

Most applets are in fact simplified services that are called by the CLI when a
registered keyword is matched. Some of them only have a ->parse() function
which immediately returns with a final result, while others will return zero
asking for the->io_handler() one to be called till the end. For these ones, a
context is generally needed between calls to know where to restart from.

Other applets are completely autonomous applets with their init function and
an I/O handler, and these ones also need a persistent context between calls to
the I/O handler. These ones are typically instantiated by "use-service" or by
other means.

Originally a few integers were provided to keep a trivial state (st0, st1, st2)
and these ones progressively proved insufficient, leading to a "ctx.cli" sub-
context that was allowed to use extra fields of various types. Other applets
preferred to use their own context definition.

All this resulted in the appctx->ctx to contain a myriad of definitions of
various service contexts, and in some services abusing other services'
definitions by laziness, and others being extended to use their own definition
after having run for a long time on the generic types, some of which were not
noticed and mistakenly used the same storage locations by accident. A massive
cleanup was needed.

2. New approach in 2.6

In 2.6, there's an "svcctx" pointer that's initialized to NULL before any
instantiation of an applet or of a CLI keyword's function. Applets and keyword
handlers are free to make it point wherever they want, and to find it unaltered
between subsequent calls, including up to the ->release() call. The "st2" state
that was totally abused with random enums is not used anymore and was marked as
deprecated. It's still initialized to zero before the first call though.

One special area, "svc.storage[]", is large enough to contain any of the
contexts that used to be present under "appctx->ctx". The "svcctx" may be set
to point to this area so that a small structure can be allocated for free and
without requiring error checking. In order to make this easier, a specially
purposed function is provided: "applet_reserve_svcctx()". This function will
require the caller to indicate how large an area it needs, and will return a
pointer to this area after checking that it fits. If it does not, haproxy will
crash. This is purposely done so that it's known during development that if a
small structure doesn't fit, a different approach is required.

As such, for the vast majority of commands, the process is the following one:

struct foo_ctx {
int myfield1;
int myfield2;
char *myfield3;
};

int io_handler(struct appctx *appctx)
{
struct foo_ctx ctx = applet_reserve_svcctx(appctx, sizeof(ctx));

if (!ctx->myfield1) {
/ first call /
ctx->myfield1++;
}
...
}

The pointer may be directly accessed from the I/O handler if it's known that it
was already reserved by the init handler or parsing function. Otherwise it's
guaranteed to be NULL so that can also serve as a test for a first call:

int parse_handler(struct appctx *appctx)
{
struct foo_ctx ctx = applet_reserve_svcctx(appctx, sizeof(ctx));
ctx->myfield1 = 12;
return 0;
}

int io_handler(struct appctx *appctx)
{
struct foo_ctx *ctx = appctx->svcctx;

for (; !ctx->myfield1; ctx->myfield1--) {
do_something();
}
...
}

There is no need to free anything because that space is not allocated but just
points to a reserved area.

If it is too small (its size is APPLET_MAX_SVCCTX bytes), it is preferable to
use it with dynamically allocated structures (pools, malloc, etc). For example:

int io_handler(struct appctx *appctx)
{
struct foo_ctx *ctx = appctx->svcctx;

if (!ctx) {
/ first call /
ctx = pool_alloc(pool_foo_ctx);
if (!ctx)
return 1;
}
...
}

void io_release(struct appctx *appctx)
{
pool_free(pool_foo_ctx, appctx->svcctx);
}

The CLI code itself uses this mechanism for the cli_print_*() functions. Since
these functions are terminal (i.e. not meant to be used in the middle of an I/O
handler as they share the same contextual space), they always reset the svcctx
pointer to place it to the "cli_print_ctx" mapped in ->svc.storage.

3. Transition for old code

A lot of care was taken to make the transition as smooth as possible for
out-of-tree code since that's an API change. A dummy "ctx.cli" struct still
exists in the appctx struct, and it happens to map perfectly to the one set by
cli_print_*, so that if some code uses a mix of both, it will still work.
However, it will build with "deprecated" warnings allowing to spot the
remaining places. It's a good exercise to rename "ctx.cli" in "appctx" and see
if the code still compiles.

Regarding the "st2" sub-state, it will disappear as well after 2.6, but is
still provided and initialized so that code relying on it will still work even
if it builds with deprecation warnings. The correct approach is to move this
state into the newly defined applet's context, and to stop using the stats
enums STAT_ST_* that often barely match the needs and result in code that is
more complicated than desired (the STAT_ST_* enum values have also been marked
as deprecated).

The code dealing with "show fd", "show sess" and the peers applet show good
examples of how to convert a registered keyword or an applet.

All this transition code requires complex layouts that will be removed during
2.7-dev so there is no other long-term option but to update the code (or better
get it merged if it can be useful to other users).

---

Doc/Internals/Api/Buffer Api

2018-07-13 - HAProxy Internal Buffer API

1. Background

HAProxy uses a "struct buffer" internally to store data received from external
agents, as well as data to be sent to external agents. These buffers are also
used during data transformation such as compression, header insertion or
defragmentation, and are used to carry intermediary representations between the
various internal layers. They support wrapping at the end, and they carry their
own size information so that in theory it would be possible to use different
buffer sizes in parallel even though this is not currently implemented.

The format of this structure has evolved over time, to reach a point where it
is convenient and versatile enough to have permitted to make several internal
types converge into a single one (specifically the struct chunk disappeared).

2. Representation as of 1.9-dev1

The current buffer representation consists in a linear storage area of known
size, with a head position indicating the oldest data, and a total data count
expressed in bytes. The head position, data count and size are expressed as
integers and are positive or null. By convention, the head position is strictly
smaller than the buffer size and the data count is smaller than or equal to the
size, so that wrapping can be resolved with a single subtract. A buffer not
respecting these rules is said to be degenerate. Unless specified otherwise,
the various API functions will adopt an undefined behaviour when passed such a
degenerate buffer.

Buffer declaration :

struct buffer {
size_t size; // size of the storage area (wrapping point)
char *area; // start of the storage area
size_t data; // contents length after head
size_t head; // start offset of remaining data relative to area
};

Linear buffer representation :

area
|
V<--------------------------------------------------------->| size
+-----------+---------------------------------+-------------+
| |/////////////////////////////////| |
+-----------+---------------------------------+-------------+
|<--------->|<------------------------------->|
head data ^
|
tail

Wrapping buffer representation :

area
|
V<--------------------------------------------------------->| size
+---------------+------------------------+------------------+
|///////////////| |//////////////////|
+---------------+------------------------+------------------+
|<-------------------------------------->| head
|-------------->| ...data data...|<-----------------|
^
|
tail

3. Terminology

Manipulating a buffer just based on a head and a wrapping data count is not
very convenient, so we define a certain number of terms for important elements
characterizing a buffer :

- origin : pointer to relative position 0 in the storage area. Undefined
when the buffer is not allocated.

- size : the allocated size of the storage area starting at the origin,
expressed in bytes. A buffer whose size is zero is said not to
be allocated, and its origin in this case is undefined.

- data : the amount of data the buffer contains, in bytes. It is always
lower than or equal to the buffer's size, hence it is always 0
for an unallocated buffer.

- emptiness : a buffer is said to be empty when it contains no data, hence
data == 0. It is possible for such buffers not to be allocated
and to have size == 0 as well.

- room : the available space in the buffer. This is its size minus data.

- head : position relative to origin where the oldest data byte is found
(it typically is what send() uses to pick outgoing data). The
head is strictly smaller than the size.

- tail : position relative to origin where the first spare byte is found
(it typically is what recv() uses to store incoming data). It
is always equal to the buffer's data added to its head modulo
the buffer's size.

- wrapping : the byte following the last one of the storage area loops back
to position 0. This is called wrapping. The wrapping point is
the first position relative to origin which doesn't belong to
the storage area. There is no wrapping when a buffer is not
allocated. Wrapping requires special care and means that the
regular string manipulation functions are not usable on most
buffers, unless it is known that no wrapping happens. Free
space may wrap as well if the buffer only contains data in the
middle.

- alignment : a buffer is said to be aligned if its data do not wrap. That
is, its head is strictly before the tail, or the buffer is
empty and the head is null. Aligning a buffer may be required
to use regular string manipulation functions which have no
support for wrapping.

A buffer may be in three different states :
- unallocated : size == 0, area == 0 (b_is_null() is true)
- waiting : size == 0, area != 0
- allocated : size > 0, area > 0

It is not permitted to have area == 0 with a non-null size. In addition, the
waiting state may also be used to indicate a read-only buffer which does not
wrap and which must not be freed (e.g. for use with error messages).

The basic API only covers allocated buffers. Switching to/from the other states
is covered by the management API since it requires specific allocation and free
calls.

4. Using buffers

Buffers are defined in a few files :
- include/common/buf.h : structure definition, and manipulation functions
- include/common/buffer.h : resource management (alloc/free/wait lists)
- include/common/istbuf.h : advanced string manipulation

4.1. Basic API

The basic API is made of the functions which abstract accesses to the buffers
and which help calculating their state, free space or used space.

4.2. String API

The string API aims at providing both convenient and efficient ways to read and
write to/from buffers using indirect strings (ist). These strings and some
associated functions are defined in ist.h.

4.3. Management API

The management API makes a distinction between an empty buffer, which by
definition is not allocated but is ready to be allocated at any time, and a
buffer which failed an allocation and is waiting for an available area to be
offered. The functions allow to register on a list to be notified about buffer
availability, to notify others of a number of buffers just released, and to be
and to be notified of buffer availability. All allocations are made through the
standard buffer pools.

5. Porting code from older versions

The previous buffer API introduced in 1.5-dev9 (May 2012) used to look like the
following (with the struct renamed to old_buffer here to avoid confusion during
quick lookups at the doc). It's worth noting that the "data" field used to be
part of the struct but with a different type and meaning. It's important to be
careful about potential code making use of &b->data as it will silently compile
but fail.

Previous buffer declaration :

struct old_buffer {
char p; / buffer's start pointer, separates in and out data */
unsigned int size; / buffer size in bytes /
unsigned int i; / number of input bytes pending for analysis in the buffer /
unsigned int o; / number of out bytes the sender can consume from this buffer /
char data[0]; / <size> bytes /
};

Previous linear buffer representation :

data p
| |
V V
+-----------+--------------------+------------+-------------+
| |////////////////////|////////////| |
+-----------+--------------------+------------+-------------+
<---------------------------------------------------------> size
<------------------> <---------->
o i

There is this correspondence between old and new fields (some will involve a
knowledge of a channel when the output byte count is required) :

Then some common expressions can be mapped like this :

The buffer modification functions are less straightforward and depend a lot on
the context where they are used. It is strongly advised to figure in the list
of functions above what is available based on what is attempted to be done in
the existing code.

Note that it is very likely that any out-of-tree code relying on buffers will
not use both ->i and ->o but instead will use exclusively ->i on the side
producing data and use exclusively ->o on the side consuming data (such as in a
mux or in an applet). In both cases, it should be assumed that the other side
is always zero and that either ->i or ->o is replaced with ->data, making the
remaining code much simpler (no more code duplication based on the data
direction).

---

Doc/Internals/Api/Buffer List Api

2024-09-30 - Buffer List API

1. Use case

The buffer list API allows one to share a certain amount of buffers between
multiple entities, which will each see their own as lists of buffers, while
keeping a shared free list. The immediate use case is for muxes, which may
want to allocate up to a certain number of buffers per connection, shared
among all streams. In this case, each stream will first request a new list
for its own use, then may request extra entries from the free list. At any
moment it will be possible to enumerate all allocated lists and to know which
buffer follows which one.

2. Representation

The buffer list is an array of struct bl_elem. It can hold up to N-1 buffers
for N elements. The first one serves as the bookkeeping head and creates the
free list.

Each bl_elem contains a struct buffer, a pointer to the next cell, and a few
flags. The struct buffer is a real struct buffer for all cells, except the
first one where it holds useful data to describe the state of the array:

struct bl_elem {
struct buffer {
size_t size; // head: size of the array in number of elements
char *area; // head: not used (0)
size_t data; // head: number of elements allocated
size_t head; // head: number of users
} buf;
uint32_t next;
uint32_t flags;
};

There are a few important properties here:

- for the free list, the first element isn't part of the list, otherwise
there wouldn't be any head storage anymore.

- the head's buf.data doesn't include the first cell of the array, thus its
maximum value is buf.size - 1.

- allocations are always made by appending to end of the existing list

- releases are always made by releasing the beginning of the existing list

- next == 0 for an allocatable cell implies that all the cells from this
element to the last one of the array are free. This allows to simply
initialize a whole new array with memset(array, 0, sizeof(array))

- next == ~0 for an allocated cell indicates we've reached the last element
of the current list.

- for the head of the list, next points to the first available cell, or 0 if
the free list is depleted.

3. Example

The array starts like this, created with a calloc() and having size initialized
to the total number of cells. The number represented is the 'next' value. "~"
here standands for ~0 (i.e. end marker).

[1|0|0|0|0|0|0|0|0|0] => array entirely free

strm1: bl_get(0) -> 1 = assign 1 to strm1's first cell

[2|~|0|0|0|0|0|0|0|0] => strm1 allocated at [1]
1

strm1: bl_get(1) -> 2 = allocate one cell after cell 1

[3|2|~|0|0|0|0|0|0|0]
1

strm1: bl_get(2) -> 3 = allocate one cell after cell 2

[4|2|3|~|0|0|0|0|0|0]
1

strm2: bl_get(0) -> 4 = assign 4 to strm2's first cell

[5|2|3|~|~|0|0|0|0|0]
1 2

strm1: bl_put(1) -> 2 = release cell 1, jump to next one (2)

[1|5|3|~|~|0|0|0|0|0]
1 2

4. Manipulating buffer lists

The API is very simple, it allows to reserve a buffer for a new stream or for
an existing one, to release a stream's first buffer or release the entire
stream, and to initialize / release the whole array.

---

Doc/Internals/Api/Event Hdl

-----------------------------------------
event_hdl Guide - version 3.1
( Last update: 2024-06-21 )
------------------------------------------

ABSTRACT
--------

The event_hdl support is a new feature of HAProxy 2.8. It is a way to easily
handle general events in a simple to maintain fashion, while keeping core code
impact to the bare minimum.

This document first describes how to use already supported events,
then how to add support for your very own events.

This feature is quite new for now. The API is not frozen and will be
updated/modified/improved/extended as needed.

SUMMARY
-------

1. event_hdl introduction
2. How to handle existing events
2.1 SYNC mode
2.2 ASYNC mode
2.2.1 normal version
2.2.2 task version
2.3 Advanced features
2.3.1 sub_mgmt
2.3.2 subscription external lookups
2.3.3 subscription ptr
2.3.4 private_free
3. How to add support for new events
3.1 Declaring a new event data structure
3.2 Publishing an event
4. Subscription lists
5. misc/helper functions

1. EVENT_HDL INTRODUCTION
-------------------------

EVENT_HDL provides two complementary APIs, both are implemented
in src/event_hdl.c and include/haproxy/event_hdl(-t).h:

One API targeting developers that want to register event
handlers that will be notified when specific events occur in the process.
(See section 2.)

One API targeting developers that want to notify registered handlers about
an event that is happening in the process.
(See section 3.)

2. HOW TO HANDLE EXISTING EVENTS
--------------------------------

To handle existing events, you must first decide which events you're
interested in.

event types are defined as follow:

text

/ type for storing event subscription type /
	typedef struct event_hdl_sub_type
	{
		/ up to 256 families, non cumulative, adjust if needed /
		uint8_t family;
		/ up to 16 sub types using bitmasks, adjust if needed /
		uint16_t subtype;
	} event_hdl_sub_type;

For an up to date list of already supported events,
please refer to include/haproxy/event_hdl-t.h
At the end of the file you will find existing event types.

Each event family provides an unique data structure that will
be provided to the event handler (registered to one or more
event subtypes) when such events occur.

An event handler can subscribe to a single event family type at a time, but
within the family type it can subscribe to multiple event subtypes.

For example, let's consider the SERVER family type.

Let's assume it provides the event_hdl_cb_data_server data structure.

We can register a handler that will be notified for
every SERVER event types using:
EVENT_HDL_SUB_SERVER

This will include EVENT_HDL_SUB_SERVER_ADD,
EVENT_HDL_SUB_SERVER_DEL [...]

But we can also subscribe to a specific subtype only,
for example server deletion:
EVENT_HDL_SUB_SERVER_DEL

You can even combine multiple SERVER subtypes using
event_hdl_sub_type_add function helper:
event_hdl_sub_type_add(EVENT_HDL_SUB_SERVER_DEL,
EVENT_HDL_SUB_SERVER_ADD)

(will refer to server deletion as well as server addition)

Registering a handler comes into multiple flavors:

SYNC mode:
handler is called in a blocking manner directly from the
thread that publishes the event.
This mode should be used with precaution because it could
slow the caller or cause deadlocks if used improperly.

Sync mode is useful when you directly depend on data or
state consistency from the caller.

Sync mode gives you access to unsafe elements in the data structure
provided by the caller (again, see event_hdl-t.h for more details).
The data structure may provide lock hints in the unsafe section
so that you know which locks are already held within the
calling context, hopefully preventing you from relocking
an already locked element and preventing deadlocks.

ASYNC mode:
handler is called in a non-blocking manner
(in a dedicated tasklet),
thus, the caller (that published the event) is not affected
by the handler. (time wise and data wise)

This is the safest way to handle events,
but it also comes with a limitation:

unsafe elements in the data structure provided by
the caller SHOULD be used under NO circumstances.
Indeed, only safe elements are meant to be used
when handling the event in async mode.

ASYNC mode is declined in 2 different versions:
normal:
handler is simply a function pointer
(same prototype as sync mode),
that is called asynchronously with relevant data
when the event is published. Only difference with
sync mode here is that 'unsafe' data provided
by the data structure may not be used.
task:
handler is a user defined task(let) that uses an event
queue to consume pending events.
This mode is interesting when you need to perform
advanced operations or you need to handle the event
in an already existing task context.
It is a bit more complicated to setup, but really
nothing to worry about, some examples will be
provided later in this document.

event subscription is performed using the function:

event_hdl_subscribe(list, event, hdl);

The function returns 1 in case of success,
and 0 in case of failure (bad arguments, or memory error)

The function may BUG_ON if used improperly (invalid arguments)

<list> is either user specified list used to store the
new subscription, or NULL if you want to store the subscription
in the process global list.

<list> is also asked when publishing an event,
so specifying list could be useful, if, for example,
you only want to subscribe to a specific subscription list
(see this as a scope for example, NULL being full scope,
and specific list being limited scope)

We will use server events as an example:

You could register to events for ALL servers by using the
global list (NULL), or only to a specific server events
by using the subscription list dedicated to a single server.

<event> are the events (family.subtypes) you're subscribing to

<hdl> contains required handler options, it must be provided using
EVENT_HDL_(TASK_)(A)SYNC() and EVENT_HDL_ID_(TASK_)(A)SYNC()
helper macros.

See include/haproxy/event_hdl.h or below to know which macro
best suits your needs.

When registering a handler, you have the ability to provide an
unique ID (using EVENT_HDL_ID_ macro family) that could be used
later to perform lookups on the subscription.
ID is stored as an uint64_t hash that is expected to be computed using
general purpose event_hdl_id inline function provided by event_hdl.h.
Not providing an ID (using EVENT_HDL_ macro family)
results in the subscription being considered as anonymous.
As the name implies, anonymous subscriptions don't support lookups.

2.1 SYNC MODE
-------------

Example, you want to register a sync handler that will be called when
a new server is added.

Here is what the handler function will look like:

text

void my_sync_handler(const struct event_hdl_cb cb, void private)
{
	const struct event_hdl_cb_data_server *server = cb->e_data;
	/* using EVENT_HDL_ASSERT_SYNC is a good practice to ensure
	 * that the function breaks if used in async mode
	 * (because we will access unsafe data in this function that
	 * is sync mode only)
	 */
	EVENT_HDL_ASSERT_SYNC(cb);
	printf("I've been called for '%s', private = %p\n",
	       event_hdl_sub_type_to_string(cb->e_type), private);
	printf("server name is '%s'\n", server->safe.name);
	/ here it is safe to use unsafe data /
	printf("server ptr is '%p'\n", server->unsafe.ptr);	/* from here you have the possibility to manage the subscription
	 * cb->sub_mgmt->unsub(cb->sub_mgmt);
	 * // hdl will be removed from the subscription list
	 */
}

Here is how you perform the subscription:

anonymous subscription:

text

int private = 10;	event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD,
			    EVENT_HDL_SYNC(my_sync_handler, &private, NULL));

identified subscription:

text

int private = 10;
	uint64_t id = event_hdl_id("test", "sync");	event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD,
			    EVENT_HDL_ID_SYNC(id,
					      my_sync_handler,
					      &private,
					      NULL));

identified subscription where freeing private is required when subscription ends:
(also works for anonymous)
(more on this feature in 2.3.4)

text

int private = malloc(sizeof(private));
	uint64_t id = event_hdl_id("test", "sync_free");
	BUG_ON(!private);
	*private = 10;
	/* passing free as 'private_free' function so that
	 * private can be freed when unregistering is performed
	 */
	event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD,
			    EVENT_HDL_ID_SYNC(id,
					      my_sync_handler,
					      private,
					      free));

	/ ... /	// unregistering the identified hdl
	if (event_hdl_lookup_unsubscribe(NULL, id)) {
		printf("private will automatically be freed!\n");
	}

2.2 ASYNC MODE
--------------

As mentioned before, async mode comes in 2 flavors, normal and task.

2.2.1 NORMAL VERSION
--------------------

Normal is meant to be really easy to use, and highly compatible with sync mode.

(Handler can easily be converted or copy pasted from async to sync mode
and vice versa)

Quick warning about sync to async handler conversion:

please always use EVENT_HDL_ASSERT_SYNC whenever you develop a
sync handler that performs unsafe data access.

This way, if the handler were to be converted or copy pasted as is to
async mode without removing unsafe data accesses,
the handler will forcefully fail to indicate an error so that you
know something has to be fixed in your handler code.

Back to our async handler, let's say you want to declare an
async handler that will be called when a new server is added.

Here is what the handler function will look like:

text

void my_async_handler(const struct event_hdl_cb cb, void private)
{
	const struct event_hdl_cb_data_server *server = cb->e_data;
	printf("I've been called for '%s', private = %p\n",
	       event_hdl_sub_type_to_string(cb->e_type), private);
	printf("server name is '%s'\n", server->safe.name);
	/ here it is not safe to use unsafe data /	/* from here you have the possibility to manage the subscription
	 * cb->sub_mgmt->unsub(cb->sub_mgmt);
	 * // hdl will be removed from the subscription list
	 */
}

Note that it is pretty similar to sync handler, except
for unsafe data access.

Here is how you declare the subscription:

anonymous subscription:

text

int private = 10;	event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD,
			    EVENT_HDL_ASYNC(my_async_handler, &private, NULL));

identified subscription:

text

int private = 10;
	uint64_t id = event_hdl_id("test", "async");	event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD,
			    EVENT_HDL_ID_ASYNC(id,
					       my_async_handler,
					       &private,
					       NULL));

identified subscription where freeing private is required when subscription ends:
(also works for anonymous)

text

int private = malloc(sizeof(private));
	uint64_t id = event_hdl_id("test", "async_free");
	BUG_ON(!private);
	*private = 10;
	/* passing free as 'private_free' function so that
	 * private can be freed when unregistering is performed
	 */
	event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD,
			    EVENT_HDL_ID_ASYNC(id,
					       my_async_handler,
					       private,
					       free));
	/ ... /	// unregistering the identified hdl
	if (event_hdl_lookup_unsubscribe(NULL, id)) {
		printf("private will automatically be freed when "
		       "all pending events referencing private "
		       "are consumed!\n");
	}

2.2.2 TASK VERSION
------------------

task version requires a bit more setup, but it's pretty
straightforward actually.

First, you need to initialize an event queue that will be used
by event_hdl facility to push you events according to your subscription:

text

event_hdl_async_equeue my_q;	event_hdl_async_equeue_init(&my_q);

Then, you need to declare a task(let) (or reuse existing task(let))

It is your responsibility to make sure that the task(let) still exists
(is not freed) when calling the subscribe function
(and that the task remains valid as long as the subscription is).

When a subscription referencing your task is over
(either ended because of list purge, external code or from the handler itself),
you will receive the EVENT_HDL_SUB_END event.
When you receive this event, you must free it as usual and you can safely
assume that the related subscription won't be sending you any more events.

Here is what your task will look like (involving a single event queue):

text

struct task event_hdl_async_task_my(struct task task,
                                     void *ctx, unsigned int state)
{
	struct tasklet tl = (struct tasklet )task;
	event_hdl_async_equeue *queue = ctx;
	struct event_hdl_async_event *event;
	struct event_hdl_cb_data_server *srv;
	uint8_t done = 0;
	while ((event = event_hdl_async_equeue_pop(queue)))
	{
		if (event_hdl_sub_type_equal(event->type, EVENT_HDL_SUB_END)) {
			done = 1;
			event_hdl_async_free_event(event);
			printf("no more events to come, "
			       "subscription is over\n");
			break;
		}
		srv = event->data;
	        printf("task event %s, %d (name = %s)\n",
		       event_hdl_sub_type_to_string(event->type),
		       ((int )event->private), srv->safe.name);
		event_hdl_async_free_event(event);
	}	if (done) {
		/* our job is done, subscription is over:
		 * no more events to come
		 */
		tasklet_free(tl);
		return NULL;
	}
	return task;
}

Here is how we would initialize the task event_hdl_async_task_my:

text

struct tasklet *my_task;	my_task = tasklet_new();
	BUG_ON(!my_task);
	my_task->context = &my_q; // we declared my_q previously in this example
	/* we declared event_hdl_async_task_my previously
	 * in this example
	 */
	my_task->process = event_hdl_async_task_my;

Given our task and our previously initialized event queue, here is how
to perform the subscription:

text

int test_val = 11;
	uint64_t id = event_hdl_id("test", "my_task");	/ anonymous variant /
	event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD,
			    EVENT_HDL_ASYNC_TASK(&my_q,
						 my_task,
						 &test_val,
						 NULL));
	/ identified variant /
	event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD,
			    EVENT_HDL_ID_ASYNC_TASK(id,
						    &my_q,
						    my_task,
						    &test_val,
						    NULL));

Note: it is not recommended to perform multiple subscriptions
that share the same event queue or same task(let) (or both)

That is, having more than one subscription waking a task(let)
and/or feeding the same event queue.

No check is performed on this when registering, so the API
won't prevent you from doing it.

If you are going to do this anyway despite this warning:

In the case you need to stop the task prematurely
(if this is not going to happen please skip this paragraph):
You are responsible for acknowledging the end of every
active subscriptions that refer to your task or
your event queue(s).
And you really don't want a subscription associated with
your task or event queue to keep going when the task
is not active anymore because:
1: there will be memory leak
(event queue might continue to receive new events)
2: there is a 100% chance of process crash in case of event
because we will try to wake a task (your task)
that might already be freed. Thus UAF will occur.

2.3 ADVANCED FEATURES
---------------------

We've already covered some of these features in the previous examples.
Here is a documented recap.

2.3.1 SUB MGMT
--------------

From an event handler context, either sync or async mode:
You have the ability to directly manage the subscription
that provided the event.

As of today, these actions are supported:
- Consulting the subscription.
- Modifying the subscription (resubscribing within same family)
- Unregistering the subscription (unsubscribing).

To do this, consider the following structure:

text

struct event_hdl_sub_mgmt
	{
		/* manage subscriptions from event
		 * this must not be used directly because
		 * locking might be required
		 */
		struct event_hdl_sub *this;
		/* safe functions than can be used from
		 * event context (sync and async mode)
		 */
		struct event_hdl_sub_type (getsub)(const struct event_hdl_sub_mgmt );
		int (resub)(const struct event_hdl_sub_mgmt , struct event_hdl_sub_type);
		void (unsub)(const struct event_hdl_sub_mgmt );
	};

A reference to this structure is provided in every handler mode.

Sync mode and normal async mode (directly from the callback data pointer):

text

const struct event_hdl_cb *cb;
	// cb->sub_mgmt
	// cb->sub_mgmt->getsub(cb->sub_mgmt);
	// cb->sub_mgmt->unsub(cb->sub_mgmt);

task and notify async modes (from the event):

text

struct event_hdl_async_event *event;
	// event->sub_mgmt
	// event->sub_mgmt.getsub(&event->sub_mgmt);
	// event->sub_mgmt.unsub(&event->sub_mgmt);

2.3.2 SUBSCRIPTION EXTERNAL LOOKUPS
-----------------------------------

As you've seen in 2.3.1, managing the subscription directly
from the handler is a possibility.

But for identified subscriptions, you also have the ability to
perform lookups and management operations on specific subscriptions
within a list based on their ID, anywhere in the code.

/!\ This feature is not available for anonymous subscriptions /!\

Here are the actions already supported:

- unregistering a subscription (unsubscribing)
- updating a subscription (resubscribing within same family)
- getting a ptr/reference to the subscription

Those functions are documented in event_hdl.h
(search for EVENT_HDL_LOOKUP section).

To select a specific subscription, you must provide
the unique identifier (uint64_t hash) that was provided when subscribing.
(using event_hdl_id(scope, name) function)

Notes:
"id" is only unique within a given subscription list.

When using event_hdl_id to provide the id:
It is your responsibility to make sure that you "own"
the scope if you rely on name to be "free".

As ID computation is backed by xxhash hash API,
you should be aware that hash collisions could occur,
but are extremely rare and are thus considered safe
enough for this usage.
(see event_hdl.h for implementation details)

Please consider ptr based subscription management if
these limitations don't fit your requirements.

Here are some examples:

unsubscribing:

text

/ registering "scope":"name" subscription /
	event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD,
	  EVENT_HDL_ID_SYNC(event_hdl_id("scope", "name"),
			    my_sync_handler,
			    NULL,
			    NULL));
	/ unregistering "scope":"name" subscription /
	event_hdl_lookup_unsubscribe(NULL, event_hdl_id("scope", "name"));

2.3.3 SUBSCRIPTION PTR
----------------------

To manage existing subscriptions from external code,
we already talked about identified subscriptions that
allow lookups within list.

But there is another way to accomplish this.

When subscribing, you can use the event_hdl_subscribe_ptr() function
variant (same arguments as event_hdl_subscribe()).

What this function does, is instead of returning 1 in case of
success and 0 in case of failure: it returns a valid subscription ptr
for success and NULL for failure.

Returned ptr is guaranteed to remain valid even if subscription
is ended meanwhile because the ptr is internally guarded with a refcount.

Thus, as long as you don't explicitly unregister the subscription with
event_hdl_unsubscribe() or drop the reference using event_hdl_drop(),
subscription ptr won't be freed.

This ptr will allow you to use the following subscription
management functions from external code:

- event_hdl_take() to increment subscription ptr refcount
(automatically incremented when using event_hdl_subscribe_ptr)
- event_hdl_drop() to decrement subscription ptr refcount
- event_hdl_resubscribe() to modify subscription subtype
- event_hdl_unsubscribe() to end the subscription
(refcount will be automatically decremented)

Here is an example:

text

struct event_hdl_sub *sub_ptr;
	/ registering a subscription with subscribe_ptr /
	sub_ptr = event_hdl_subscribe_ptr(NULL, EVENT_HDL_SUB_SERVER_ADD,
			    EVENT_HDL_SYNC(my_sync_handler,
					   NULL,
					   NULL));
	/ ... /	/ unregistering the subscription /
	event_hdl_unsubscribe(sub_ptr);

Regarding identified subscriptions that were registered using the non ptr
subscribe function:

You still have the ability to get a reference to the related subscription
(if it still exists), by using event_hdl_lookup_take(list, id) function.
event_hdl_lookup_take will return a subscription ptr in case of success
and NULL in case of failure.
Returned ptr reference is automatically incremented, so it is safe to use.

Please don't forget to drop the reference
when holding the ptr is no longer needed.

Example:

text

struct event_hdl_sub *sub_ptr = NULL;
	/ registering subscription id "test":"ptr" with normal subscribe /
	if (event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_ADD,
	    EVENT_HDL_ID_SYNC(event_hdl_id("test", "ptr"),
			      my_sync_handler,
			      NULL,
			      NULL))) {
		/ fetch ref to subscription "test":"ptr" /
		sub_ptr = event_hdl_lookup_take(NULL,
						event_hdl_id("test", "ptr"));
		/ unregister the subscription using lookup /
		event_hdl_lookup_unsubscribe(NULL,
					     event_hdl_id("test", "ptr"));
	}
	/ ... /	/* unregistering the subscription with ptr
	 * will do nothing because subscription was
	 * already ended by lookup_unsubscribe, but
	 * here the catch is that sub_ptr is still
	 * valid so this won't crash the program
	 */
	if (sub_ptr) {
		event_hdl_unsubscribe(sub_ptr);
		/* unsubscribe will also result in subscription
		 * reference drop, thus subscription will be freed here
		 * because sub_ptr was the last active reference.
		 * You must not use sub_ptr anymore past this point
		 * or UAF could occur
		 */
	}

2.3.4 PRIVATE FREE
------------------

Upon handler subscription, you have the ability to provide
a private data pointer that will be passed to the handler
when subscribed events occur.

Sometimes this private data pointer will rely on dynamically allocated memory.
And in such cases, you have no way of knowing when
freeing this pointer can be done safely.

You could be tempted to think that freeing right after performing
the unsubscription could be safe.
But this is not the case, remember we could be dealing with async handlers
that might still consume pending events even though unsubscription
has been performed from external code.

To deal with this, you may want to provide the private_free
function pointer upon subscription.
This way, private_free function will automatically be called
(with private as argument) when private is no longer be used.

Example:
First we declare our private free function:

text

void my_private_free(void *my_private_data) {
	/* here we only call free,
	 * but you could do more sophisticated stuff
	 */
	free(my_private_data);
}

Then:

text

char *my_private_data = strdup("this string needs to be freed");
	BUG_ON(!my_private_data);
	event_hdl_subscribe(NULL, EVENT_HDL_SUB_SERVER_DEL,
	    EVENT_HDL_ID_ASYNC(event_hdl_id("test", "private"),
			       my_async_handler,
			       my_private_data,
			       my_private_free));
	/* freeing my_private_data is not required anymore,
	 * it will be automatically freed by our private free
	 * function when subscription ends
	 */
	/ unregistering "test":"private" subscription /
	event_hdl_lookup_unsubscribe(NULL, event_hdl_id("test", "private"));	/* my_private_free will be automatically summoned when my_private_data
	 * is not referenced anymore
	 */

3 HOW TO ADD SUPPORT FOR NEW EVENTS
-----------------------------------

Adding support for a new event is pretty straightforward.

First, you need to declare a new event subtype in event_hdl-t.h file
(bottom of the file).

You might want to declare a whole new event family, in which case
you declare both the new family and the associated subtypes (if any).

Up to 256 families containing 16 subtypes each are supported by the API.
Family 0 is reserved for special events, which means there are 255 usable
families.

You can declare a family using EVENT_HDL_SUB_FAMILY(x) where x is the
family.

You can declare a subtype using EVENT_HDL_SUB_TYPE(x, y) where x is the
family previously declared and y the subtype, Subtypes range from 1 to
16 (included), 0 is not a valid subtype.

text

#define EVENT_HDL_SUB_NEW_FAMILY                EVENT_HDL_SUB_FAMILY(4)
	#define EVENT_HDL_SUB_NEW_FAMILY_SUBTYPE_1      EVENT_HDL_SUB_TYPE(4,1)

Then, you need to update the event_hdl_sub_type_map map,
defined in src/event_hdl.c file (top of the file)
to add string to event type and event type to string conversion support.
You just need to add the missing entries corresponding to
the event family / subtypes you've defined.

Please follow this procedure:
You only added a new subtype to existing family: go to section 3.2
You added a new family: go to section 3.1

3.1 DECLARING A NEW EVENT DATA STRUCTURE
----------------------------------------

You have the ability to provide additional data for a given
event family when such events occur.

Note that it is not mandatory: you could simply declare a new event family
that does not provide any data.
If this is your case, you can skip this section and go to 3.2 section.

Now, take a look at this event data structure template
(also defined at the top of event_hdl-t.h file):

text

/ event data struct are defined as followed /
	struct event_hdl_cb_data_template {
		struct {
			/* safe data can be safely used from both
			 * sync and async functions
			 * data consistency is guaranteed
			 */
		} safe;
		struct {
			/* unsafe data may only be used from sync functions:
			 * in async mode, data consistency cannot be guaranteed
			 * and unsafe data may already be stale, thus using
			 * it is highly discouraged because it
			 * could lead to undefined behavior
			 * (UAF, null dereference...)
			 */
		} unsafe;
	};

This structure template allows you to easily create a new event
data structure that can be provided with your new event family.

You should name it after 'struct event_hdl_cb_data_new_family' so that it is
easy to guess the event family it relates to.

Indeed, each event data structure is to be associated with an
unique event family type.
For each subtypes within a family type, the associated data structure
should be provided when publishing the event.

The event data struct declaration should not be performed
directly under event_hdl-t.h file:

It should be done in the header files of the corresponding
facility that will publish/provide this event.

Example: struct event_hdl_cb_data_server, provided for the
EVENT_HDL_SUB_SERVER event family, is going to be declared in
include/haproxy/server-t.h file.

However, in event_hdl-t.h, where you declare event family/subtypes,
you should add comments or links to the file containing the relevant
data struct declaration. This way we make sure all events related
information is centralized in event_hdl-t.h while keeping it clean
and not depending on any additional includes (you are free to
depend on specific data types within your custom event data structure).

Please make sure that EVENT_HDL_ASYNC_EVENT_DATA (defined in event_hdl-t.h)
is greater than sizeof(event_hdl_cb_data_new_family).

It is required for async handlers to properly consume event data.

You are free to adjust EVENT_HDL_ASYNC_EVENT_DATA size if needed.

If EVENT_HDL_ASYNC_EVENT_DATA is not big enough to store your new
event family struct, a compilation assert triggered by EVENT_HDL_CB_DATA
will occur. In addition to this, an extra runtime BUG_ON will make
sure the condition is met when publishing the event.
The goal here is to force haproxy to fail explicitly so you know that
something must be done on your side.

3.1 PUBLISHING AN EVENT
-----------------------

Publishing an event is really simple.
It relies on the event_hdl_publish function.

The function is defined as follow:

text

int event_hdl_publish(event_hdl_sub_list *sub_list,
	                      event_hdl_sub_type e_type,
	                      const struct event_hdl_cb_data *data);

We will ignore sub_list argument for now.
In the examples below, we will use sub_list = NULL.
Go to section 4 for a full picture about this feature.

<e_type>: the event type that should be published.
All subscriptions referring to this event within
a subscription list context will be notified about the event.
<data>: data provided for the event family of <e_type>
If <e_type>.family does not provide additional data,
data should be set to NULL.
If <e_type>.family does provide additional data, data should be set
using EVENT_HDL_CB_DATA macro.
(see the example below)

The function returns 1 in case of SUCCESS (handlers successfully notified)
and 0 in case of FAILURE (no handlers notified, because of memory error).

Event publishing can be performed from anywhere in the code.
(this example does not compile)

text

struct event_hdl_cb_data_new_family event_data;
	/* first we need to prepare event data
	 * that will be provided to event handlers
	 */
	/ safe data, available from both sync and async contexts /
	event_data.safe.my_custom_data = x;
	/ unsafe data, only available from sync contexts /
	event_data.unsafe.my_unsafe_data = y;
	/ once data is prepared, we can publish the event /
	event_hdl_publish(NULL,
			  EVENT_HDL_SUB_NEW_FAMILY_SUBTYPE_1,
			  EVENT_HDL_CB_DATA(&event_data));	/* EVENT_HDL_SUB_NEW_FAMILY_SUBTYPE_1 event was
	 * successfully published in global subscription list
	 */

--------------------------------------------------------------------------------
|You should know that there is currently a limitation about publish function: |
|The function should not be used from critical places |
|(where the calling frequency is high |
|or where timing sensitivity is high). |
| |
|Because in current implementation, subscription list lookups are not |
|optimized for such uses cases. |
--------------------------------------------------------------------------------

4 SUBSCRIPTION LISTS
--------------------

As you may already know, EVENT_HDL API main functions rely on
subscription lists.
Providing NULL where subscription list argument is required
allows to use the implicit global subscription list.

But you can also provide a specific subscription list, example:
subscription list associated with a single entity so that you only
subscribe to events of this single entity

A subscription list is of type event_hdl_sub_list.
It is defined in event_hdl-t.h

To make use of this feature, you should know about these 2 functions:

event_hdl_sub_list_init(list): use this fcn to initialize
a new subscription list.

Example:

text

event_hdl_sub_list my_custom_list;	event_hdl_sub_list_init(&my_custom_list);

event_hdl_sub_list_destroy(list): use this fcn to destroy
an existing subscription list.

Example:

text

event_hdl_sub_list_init(&my_custom_list);

Using this function will cause all the existing subscriptions
within the provided sub_list to be properly unregistered
and deleted according to their types.

Now we'll take another quick look at event_hdl_publish() function:

Remember that the function is defined as follow:

text

int event_hdl_publish(event_hdl_sub_list *sub_list,
	                      event_hdl_sub_type e_type,
	                      const struct event_hdl_cb_data *data);

In the previous examples, we used sub_list = NULL.

if sub_list is NULL:
event will be published in in global list
else
event will be published in user specified sub_list

5 MISC/HELPER FUNCTIONS
-----------------------

Don't forget to take a look at MISC/HELPER FUNCTIONS in
include/haproxy/event_hdl.h (end of the file) for a
complete list of helper functions / macros.

We've already used some, if not the vast majority
in the examples shown in this document.

This includes, to name a few:
- event types manipulation
- event types comparison
- lookup id computing
- subscriber list management (covered in section 4)
- sync/async handler helpers

---

Doc/Design Thoughts/Binding Possibilities Copy Section

Doc/Design Thoughts/Connection Reuse Copy Section

Doc/Design Thoughts/Error Reporting Copy Section

Doc/Design Thoughts/Numa Auto Copy Section

2023-07-17 ### Copy Section

Doc/Design Thoughts/Pool Debugging Copy Section

Doc/Design Thoughts/Ring V2 Copy Section

Doc/Design Thoughts/Thread Group Copy Section

Doc/Internals/Api/Appctx Copy Section

Doc/Internals/Api/Buffer Api Copy Section

Doc/Internals/Api/Buffer List Api Copy Section

Doc/Internals/Api/Event Hdl Copy Section

Explore Related Technologies & Topics

Doc/Design Thoughts/Binding Possibilities

Doc/Design Thoughts/Connection Reuse

Doc/Design Thoughts/Error Reporting

Doc/Design Thoughts/Numa Auto

2023-07-17 ###

Doc/Design Thoughts/Pool Debugging

Doc/Design Thoughts/Ring V2

Doc/Design Thoughts/Thread Group

Doc/Internals/Api/Appctx

Doc/Internals/Api/Buffer Api

Doc/Internals/Api/Buffer List Api

Doc/Internals/Api/Event Hdl