## 1. Project Overview & Quickstart (GothenburgBitFactory/coding_style) # coding_style Open-source repository GothenburgBitFactory/coding_style ### Repository Details - **Repository:** [GothenburgBitFactory/coding_style](https://github.com/GothenburgBitFactory/coding_style) - **Primary Language:** Code *Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.* ## 2. Official Technical Reference & Guides (GothenburgBitFactory/GothenburgBitFactory.github.io) ## File: README.md # GBF Site Repository This repository contains the content of [gothenburgbitfactory.org](https://gothenburgbitfactory.org). ## Contributing To clone this repository, run ```shell git clone https://github.com/GothenburgBitFactory/gbf.org ``` For minor changes, it is perfectly fine to simply change the relevant Markdown files and make a pull request. For more complex changes, here is how to run the site locally: 1. Install [Hugo](https://gohugo.io/getting-started/quick-start/). 2. Within the repository, run ```shell hugo server -D -w ``` This defaults to running a web server on `http://localhost:1313`. Now you can inspect your changes locally before opening a pull request. --- ## File: content/anomaly/_index.md --- title: "Anomaly" language: en layout: single source: "https://github.com/GothenburgBitFactory/anomaly" rank: 1 menu: main aliases: - /projects/anomaly --- # Anomaly Anomaly can detect anomalous data in a numeric stream. {{< project_facts anomaly >}} ## About In order to do this, anomaly needs to see a stream of numeric data, and apply one of its detection methods. If an anomaly is detected, a response is made, chosen from one or more built in methods. ## Numeric Stream Anomaly works best in a pipe, and will read only numeric data from its input. As a simple example, suppose you wish to monitor load average and look for unusual spikes. The load average can be obtained from the 'uptime' command: ``` $ uptime 11:40 up 15 days, 4:04, 6 users, load averages: 0.38 0.32 0.32 ``` We can extract the 5-minute load (the second of the three numbers) using this: ``` $ uptime | cut -f 13 -d ' ' 0.29 ``` That number can be extracted once a minute, using this: ``` $ while [ 1 ]; do uptime | cut -f 13 -d ' '; sleep 60; done 0.29 0.26 0.19 ``` That is the kind of data stream that anomaly monitors. White space (spaces, tabs, newlines) between the numbers are ignored, so we can simulate the above stream like this: ``` $ echo 0.29 0.26 0.19 ``` This is a convenient way to demonstrate anomaly, shown below. ## Detection - Threshold The simplest detection method is threshold, which compares the data to an absolute value. This method can use a minimum and a maximum value for comparison. These alternatives are all valid, and make use of `--min`, `--max` or both: ``` anomaly --threshold --min 1.22 --max 9.75 anomaly --threshold --min 1.22 anomaly --threshold --max 9.75 ``` In the following example, the values '1' and '10' would be detected as anomalies: ``` $ echo 2 1 3 6 10 5 | anomaly --threshold --min 1.5 --max 8 Anomalous data detected. The value 1 is below the minimum of 1.5. Anomalous data detected. The value 10 is above the maximum of 8. ``` ## Detection - Standard Deviation Standard deviation measures differences from the mean value of a sample of data, and is useful for detecting extraordinary values. The sample size can be chosen such that there is enough data to determine a good mean value. The limited sample size means that a rolling window of data is used, and therefore the mean and standard deviation is updated for the current window. This makes the monitoring somewhat adaptive. Here is an example: ``` anomaly --stddev --sample 20 ``` This uses a sample size of the 20 most recent values, and will detect any values that are +/- 1 standard deviation from the mean. An example: ``` $ echo 1 2 3 4 5 6 | anomaly --stddev --sample 5 Anomalous data detected. The value 6 is more than 1 sigma(s) above the mean value 3, with a sample size of 5. ``` With a sample size of 5, comparisons begin only after the 6th value is seen. In the example, the mean value of \[1 2 3 4 5\] is 3, and the standard deviation is 1.58. This means that the 6th value is considered an anomaly if it is outside the range (3 +/- 1.58), which is from 1.42 to 4.58. To make this less sensitive, a coefficient is introduced, which defaults to 1.0 (as above) but can be overridden: ``` $ echo 1 2 3 4 5 6 | anomaly --stddev --sample 5 --coefficient 1.9 $ ``` In this example, the 6th value is not considered an anomaly because it is within the range (3 +/- (1.9 * 1.58)), which is between -0.002 and 6.002. ## Response - Message The message response is the default, and consists of a single line of printed text. It is a description of why the data value is considered an anomaly. Here is an example: ``` $ echo 1 2 3 | anomaly --threshold --max 2.5 Anomalous data detected. The value 3 is above the maximum of 2.5. ``` The message can be suppressed, but another response must be specified: ``` $ echo 1 2 3 | anomaly --threshold --max 2.5 --quiet ... ``` ## Response - Execute Anomaly can execute a program in response to detection. Here an example uses the 'date' command, but any program can be used: ``` $ echo 1 2 3 | anomaly --threshold --max 2.5 --quiet --execute '/bin/date +%s' 1361727327 ``` ## Response - Signal Anomaly can send a USR1 signal to a program in response to detection: ``` $ echo 1 2 3 | anomaly --threshold --max 2.5 --quiet --pid 12345 ``` This sends the USR1 signal to the process with PID 12345. The receiving program would need to respond accordingly. --- ## File: content/anomaly/latest.md --- layout: plain aliases: - /latest/anomaly --- {{< latest_version anomaly >}} --- ## File: content/clog/docs/_index.md --- title: "Docs" language: en layout: single menu: main --- {{< lead >}}Here is the complete set of Clog documentation.{{< /lead >}} ## Getting Started - [What is Clog?](clog/docs/what) - [How does Clog work?](clog/docs/how) - [Download and build the tarball](clog/docs/download) - [Clone the code repository and build the development snapshot](clog/docs/clone) ## Usage - [Getting Help](clog/docs/help) - [Setup](clog/docs/setup) - [Rule Syntax](clog/docs/syntax) - [Rule Sections](clog/docs/sections) - [Rule Patterns](clog/docs/patterns) - [Rule Actions](clog/docs/actions) - [Clog Options](clog/docs/options) - [Color](clog/docs/color) ## Example Rule Sets - [Syslog](clog/docs/example_syslog) - [Apache](clog/docs/example_apache) --- ## File: content/clog/docs/actions.md --- title: "Clog - Rule Actions" --- # Rule Actions When a rules matches, an action is taken. There are currently four supported actions: - `line` - `match` - `blank` - `suppress` ## The `line` Action The `line` action instructs Clog to color the entire line. This makes the whole line more visible. An example might be to color a line in a log file if that line reports a critical error. ## The `match` Action The `match` action instructs clog to only color the matching pattern and not the entire line. This make individual keywords more visible. An example might be to highlight simply an error code, or server name. ## The `blank` Action The `blank` action causes a blank line to be written both before and after the matching line. This is another form of emphasis. ## The `suppress` Action The `suppress` action causes the whole line to be removed from the output. You will not see a suppressed line at all. This is useful for stripping out noise from a log file. ## Example Here is a default rule set that includes all the different actions. The first rule will make any line completely blue if it contains the word \'blue\'. The second rule colors the word \'red\' red. The third puts blank lines around any line containing the word \'emphasize\'. The fourth rule suppresses any line containing the word \'ignore\'. Here is a sample file that triggers every rule, so we should see the actions of all rules taken. Can you predict what will happen? Here is the output: This shows rules precedence also - the second rule finds the word \'red\' in the blue line. --- ## File: content/clog/docs/clone.md --- title: "Clog - Clone" --- # How To Build Clog This is for developers. Specifically those who know how to use tools, satisfy dependencies, and want to set up a development environment. It is not user-friendly. You\'ll need these tools: - [Git](https://git-scm.com/) - [CMake](https://cmake.org) - Make - C++ compiler, currently gcc 4.7+ or clang 3.3+ for full C++11 support - Python 2.7 or later (for tests) ## Cloning the Repo ``` $ git clone --recursive https://github.com/GothenburgBitFactory/clog.git clog.git ``` ## Building the Stable Version The master branch always represents the more recently released version, and should be your preferred choice. ``` $ cd clog.git $ git checkout master # Master is the stable branch. $ cmake -DCMAKE_BUILD_TYPE=release . # 'release' for performance. $ make # Just build it. ``` ## Building the Dev Branch The dev branch is always the highest numbered branch, in this example, `1.4.0`. ``` $ cd clog.git $ git checkout 1.4.0 # Dev branch $ git submodule init && git submodule update # Fetch the submodule $ cmake -DCMAKE_BUILD_TYPE=debug . # debug or release. ``` Default: neither. ``` $ make VERBOSE=1 # Shows details ``` Build the debug type if you want symbols in the binary for debugging. Note: the development branch is not always stable, doesn\'t always pass tests, and sometimes doesn\'t even compile. Choose wisely. ## Running the Test Suite There are C++ and Python, build and run them: ``` $ cd clog.git/test $ make VERBOSE=1 # Shows details $ ./run_all # Runs all tests silently > all.log $ ./problems # Find errors in all.log ``` ## Submitting a Patch Talk to us first - make sure you are working on something that is wanted. Patches will not be applied simply because you did the work, or because it was easy, or because you want it. Instead, the change has to be considered right for the project. Remember the various forms of documentation involved, and the test suite. Work on the dev branch, not `master`. When you are ready to submit, do this: ``` $ git commit ``` Follow the standard form for commit messages, which looks like this: ``` Category: Short message - Details - Details ``` Here is a good example: ``` TW-1636: UUID with numeric-only first segment is not parsed properly - Switched Nibbler::getUUID to Nibbler::getPartialUUID, which caused partial UUID matching to fail sometimes. - Changed precedence to search for UUID before ID, which solves the numeric UUID problem. ``` Create the patch using this: ``` $ git format-patch HEAD^ ``` Mail the patch to or attach it to the appropriate ticket in the [bug tracker](https://github.com/GothenburgBitFactory/clog/issues). If you do the latter, make sure someone knows about it, or it could go unnoticed. Expect feedback. It is unlikely your patch will be accepted unmodified. Usually this is because you violated the coding style, worked in the wrong branch, or *forgot* about documentation and unit tests. --- ## File: content/clog/docs/color.md --- title: "Clog - Color" --- # Color Clog uses the same color model as Taskwarrior. A color iѕ described with one or more words. You can specify the foreground color, the background color, and some text effects. Color can be both simple and complex. You can completely ignore the color models below, and just use color in one of these ways: ``` red white on red on red bold blue bold yellow on bright green underline ``` Or you can have full control using a 256-color model. But sometimes all you need is red and green. ## 16-Color Model The basic color support is provided through named colors: ``` black, red, blue, green, magenta, cyan, yellow, white ``` Shown here used as foreground and background colors: Foreground color (for text) is simply specified as one of the above colors, or not specified at all to use the default terminal text color. Background color is specified by using the word `on`, and one of the above colors. Some examples: ``` green green on yellow on yellow ``` These colors can be modified further, by making the foreground `bold`, or by making the background `bright`. Some examples: ``` bold green bold white on bright red on bright cyan ``` Bold is not uniformly supported by all terminal emulators. Sometimes it results in a slightly brighter color, a lightly heavier typeface, or no difference. The order of the words is not important, so the following are equivalent: ``` bold green green bold ``` But the `on` is important - colors before the `on` are foreground, and colors after `on` are background. There is an additional `underline` attribute that may be used: ``` underline bold red on black ``` Much like `bold`, you may see different behavior among different terminal emulators. There is also an `inverse` attribute that flips foreground and background colors: ``` inverse red ``` Some combinations look very nice, some look terrible. Different terminal programs do implement slightly different versions of \'red\', for example, so you may see some unexpected variation across machines. The brightness of your display is also a factor. ## 256-Color Model Using 256 colors follows the same form, but the names are different, and some colors can be referenced in different ways. First there is by color ordinal, which is like this: ``` color0 color1 color2 ... color255 ``` This gives you access to all 256 colors, but doesn\'t help you much. This range is a combination of 8 basic colors (color0 - color7), then 8 brighter variations (color8 - color15). Then a block of 216 colors (color16 - color231). Then a block of 24 gray colors (color232 - color255). The large block of 216 colors (6x6x6 = 216) represents a color cube, which can be addressed via RGB values from 0 to 5 for each component color. A value of 0 means none of this component color, and a value of 5 means the most intense component color. For example, a bright red is specified as: ``` rgb500 ``` And a darker red would be: ``` rgb300 ``` Note that the three digits represent the three component values, so in this example the 5, 0 and 0 represent red=5, green=0, blue=0. Combining intense red with no green and no blue yields red. Similarly, blue and green are: ``` rgb005 rgb050 ``` Another example - bright yellow - is a mix of bright red and bright green, but no blue component, so bright yellow is addressed as: ``` rgb550 ``` A soft pink would be specified as: ``` rgb515 ``` You may notice that the large color block is represented as 6 squares. All colors in the first square have a red value of 0. All colors in the 6th square have a red value of 5. Within each square, blue ranges from 0 to 5 left to right, and within each square green ranges from 0 to 5, top to bottom. This scheme takes some getting used to. The block of 24 gray colors can also be accessed as gray0 - gray23, in a continuous ramp from black to white. ## True Color Model This is true 24-bit color, and is not yet supported. ## Mixing Color Models If you specify 16-color colors, and view on a 256-color terminal, no problem. If you try the reverse, specifying 256-color colors and viewing on a 16-color terminal, you will be disappointed, perhaps even appalled. The results are usually that only the 16-color model is shown, and the rest is just blank. There is some limited color mapping - for example, if you were to specify this combination: ``` red on gray3 ``` Then you are mixing a 16-color and 256-color specifications. Taskwarrior will map `red` to `color1`, and proceed. Note that `red` and `color1` are not quite the same tone. Note also that there are no `bold` or `bright` attributes when dealing with 256 colors, but there is still underline available. ## Terminal Settings All of the above directly emit Xterm escape sequences to control color and have absolutely nothing to do with your `$TERM` setting. --- ## File: content/clog/docs/download.md --- title: "Clog - Download" --- # How To Build Clog This is for developers. Specifically those who know how to use tools, satisfy dependencies, and want to set up a development environment. It is not user-friendly. You\'ll need these tools: - [Git](https://git-scm.com/) - [CMake](https://cmake.org) - Make - C++ compiler, currently gcc 4.7+ or clang 3.3+, for full C++11 support ## Downloading the Repo The next step is to obtain the code. This means getting the released tarball. You can download the tarball with `curl`, as an example of just one of many ways to download the tarball. ``` $ curl -L -O https://github.com/GothenburgBitFactory/clog/releases/download/v{{< latest_version clog >}}/clog-{{< latest_version clog >}}.tar.gz ``` ## Building the Tarball Expand the tarball, build Clog, and install it. This copies files into the right place, and installs the man page. ``` $ tar xzf clog-{{< latest_version clog >}}.tar.gz $ cd clog-{{< latest_version clog >}} $ cmake -DCMAKE_BUILD_TYPE=release . ... $ make ... $ sudo make install ``` ## Submitting a Patch [See here for details](clog/docs/clone/#submitting-a-patch). --- ## File: content/clog/docs/example_apache.md --- title: "Clog - Apache Rules" --- # Apache Rules This example shows color highlighting for HTTP status code. ``` # Apache access.log status codes apache rule / 2[0-9][0-9] / --> green match apache rule / 3[0-9][0-9] / --> yellow match apache rule / 4[0-9][0-9] / --> red match apache rule / 5[0-9][0-9] / --> bold red match ``` --- ## File: content/clog/docs/example_syslog.md --- title: "Clog - Syslog Rules" --- # Syslog Rules This example shows color highlighting for `syslog` severities. ``` # Standard syslog entries. default rule /warn|debug/ --> yellow line default rule /error|severe/ --> red line default rule "critical" --> bold red line default rule "critical" --> blank default rule "ignore" --> suppress ```