1. Project Overview & Quickstart (itzg/docker-minecraft-server)
File: README.md
There you will find things like
- Quick start with Docker Compose
- Running different versions of Minecraft and using various server types for Java Edition
- Setting server properties via container environment variables
- Managing mods and plugins with automated downloads and cleanup
- Using various modpack providers/platforms
- ...and much more
There are also many examples located in the examples directory of this repo.
This image only supports Java edition natively; however, if looking for a server that is compatible with Bedrock edition, then use itzg/minecraft-bedrock-server or refer to this section to add Bedrock compatibility to a Java edition server.
SpawnBox - Powered by itzg/minecraft-server, it's a Windows desktop app for parents, teens, and friend groups who want a Minecraft server on their own PC without learning Docker, WSL2, or networking.
Server.pro - A game server hosting platform offering one-click Minecraft server deployment powered by itzg/minecraft-server, with global locations and an easy-to-use control panel.
Seedloaf - Built from scratch (by two nerds) on top of itzg/minecraft-server. Seedloaf offers free, and premium Minecraft servers where you have full control!
File: docs/configuration/auto-rcon-commands.md
Auto-execute RCON commands
RCON commands can be configured to execute when the server starts, a client connects, or a client disconnects.
!!! note
When declaring several commands within a compose file environment variable, it's easiest to use YAML's `|-` [block style indicator](https://yaml-multiline.info/).On Server Start:
RCON_CMDS_STARTUP: |-
gamerule doFireTick false
pregen start 200On Client Connection:
RCON_CMDS_ON_CONNECT: |-
team join New @a[team=]Note:
- On client connect we only know there was a connection, and not who connected. RCON commands will need to be used for that.
On Client Disconnect:
RCON_CMDS_ON_DISCONNECT: |-
gamerule doFireTick trueOn First Client Connect
RCON_CMDS_FIRST_CONNECT: |-
pregen stopOn Last Client Disconnect
RCON_CMDS_LAST_DISCONNECT: |-
kill @e[type=minecraft:boat]
pregen start 200Example of rules for new players
Uses team NEW and team OLD to track players on the server. So move player with no team to NEW, run a command, move them to team OLD.
Reference Article
RCON_CMDS_STARTUP: |-
/pregen start 200
/gamerule doFireTick false
/team add New
/team add Old
RCON_CMDS_ON_CONNECT: |-
/team join New @a[team=]
/give @a[team=New] birch_boat
/team join Old @a[team=New]
RCON_CMDS_FIRST_CONNECT: |-
/pregen stop
RCON_CMDS_LAST_DISCONNECT: |-
/kill @e[type=minecraft:boat]
/pregen start 200File: docs/configuration/interpolating.md
title: Modifying config files
Replacing variables inside configs
Sometimes you have mods or plugins that require configuration information that is only available at deploy-time. For example if you need to configure a plugin to connect to a database, you don't want to include this information in your Git repository or Docker image.
Or maybe you have some runtime information like the server name that needs to be set in your config files after the container starts.
For those cases there is the option to replace defined variables inside your configs with environment variables defined at container runtime.
When the environment variable REPLACE_ENV_IN_PLACE is set to true (the default), the startup script will go through all files inside the container's /data path and replace variables that match the container's environment variables. Variables can instead (or in addition to) be replaced in files sync'ed from /plugins, /mods, and /config by setting REPLACE_ENV_DURING_SYNC to true (defaults to false).
Variables that you want to replace need to be declared inside curly brackets and prefixed with a dollar sign, such as ${CFG_YOUR_VARIABLE}, which is same as many scripting languages.
You can also change REPLACE_ENV_VARIABLE_PREFIX, which defaults to "CFG_", to limit which environment variables are allowed to be used. For example, with "CFG_" as the prefix, the variable ${CFG_DB_HOST} would be substituted, but not ${DB_HOST}. The prefix can be set to an empty string to allow for matching any variable name.
If you want to use a file's content for value, such as when using secrets mounted as files, declare the placeholder named like normal in the file and declare an environment variable named the same but with the suffix _FILE.
For example, a my.cnf file could contain:
[client]
password = ${CFG_DB_PASSWORD}...a secret declared in the compose file with:
secrets:
db_password:
external: true...and finally the environment variable would be named with a _FILE suffix and point to the mounted secret:
environment:
CFG_DB_PASSWORD_FILE: /run/secrets/db_passwordVariables will be replaced in files with the following extensions:.yml, .yaml, .txt, .cfg, .conf, .properties.
Specific files can be excluded by listing their name (without path) in the variable REPLACE_ENV_VARIABLES_EXCLUDES.
Paths can be excluded by listing them in the variable REPLACE_ENV_VARIABLES_EXCLUDE_PATHS. Path
excludes are recursive. Here is an example:
REPLACE_ENV_VARIABLES_EXCLUDE_PATHS="/data/plugins/Essentials/userdata /data/plugins/MyPlugin"Here is a full example where we want to replace values inside a database.yml.
---
database:
host: ${CFG_DB_HOST}
name: ${CFG_DB_NAME}
password: ${CFG_DB_PASSWORD}This is how your compose.yaml file could look like:
# Other docker compose examples in /examples
services:
minecraft:
image: itzg/minecraft-server
ports:
- "25565:25565"
volumes:
- "mc:/data"
environment:
EULA: "TRUE"
ENABLE_RCON: "true"
RCON_PASSWORD: "testing"
RCON_PORT: 28016
# enable env variable replacement
REPLACE_ENV_VARIABLES: "TRUE"
# define an optional prefix for your env variables you want to replace
ENV_VARIABLE_PREFIX: "CFG_"
# and here are the actual variables
CFG_DB_HOST: "http://localhost:3306"
CFG_DB_NAME: "minecraft"
CFG_DB_PASSWORD_FILE: "/run/secrets/db_password"
volumes:
mc:
rcon:
secrets:
db_password:
file: ./db_passwordPatching existing files
JSON path based patches can be applied to one or more existing files by setting the variable PATCH_DEFINITIONS to the path of a directory that contains one or more patch definition json files or a patch set json file.
The file and value fields of the patch definitions may contain ${...} variable placeholders. The allowed environment variables in placeholders can be restricted by setting REPLACE_ENV_VARIABLE_PREFIX, which defaults to "CFG_".
The following example shows a patch-set file where various fields in the paper.yaml configuration file can be modified and added:
{
"patches": [
{
"file": "/data/paper.yml",
"ops": [
{
"$set": {
"path": "$.verbose",
"value": true
}
},
{
"$set": {
"path": "$.settings['velocity-support'].enabled",
"value": "${CFG_VELOCITY_ENABLED}",
"value-type": "bool"
}
},
{
"$put": {
"path": "$.settings",
"key": "my-test-setting",
"value": "testing"
}
}
]
}
]
}Supports the file formats:
- JSON
- JSON5
- Yaml
- TOML, but processed output is not pretty
File: docs/configuration/jvm-options.md
JVM Options
Memory Limit
By default, the image declares an initial and maximum Java memory-heap limit of 1 GB. There are several ways to adjust the memory settings:
MEMORY: "1G" by default, can be used to adjust both initial (Xms) and max (Xmx) memory heap settings of the JVMINIT_MEMORY: independently sets the initial heap sizeMAX_MEMORY: independently sets the max heap size
The values of all three are passed directly to the JVM and support format/units as <size>[g|G|m|M|k|K].
To have control over heap size, without relying on absolute memory sizes percentages are also supported using <size>%.
!!! info "RAMPercentage parameters"
Percentage based heap sizing is enabled using -XX:InitialRAMPercentage for INIT_MEMORY and -XX:MaxRAMPercentage for MAX_MEMORY.
For details on the function of these parameters look here.
!!! example "Using docker run"
```
-e MEMORY=2G
```
or to use init and max memory:
```
-e INIT_MEMORY=1G -e MAX_MEMORY=4G
```!!! example "Using compose file"
```
environment:
MEMORY: 2G
```
or to use init and max memory:
```
environment:
INIT_MEMORY: 1G
MAX_MEMORY: 4G
```To let the JVM calculate the heap size from the container declared memory limit, unset MEMORY with an empty value, such as -e MEMORY="". By default, the JVM will use 25% of the container memory limit as the heap limit.
!!! important
The settings above only set the Java heap limits. Memory resource requests and limits on the overall container should also account for non-heap memory usage. An extra 25% is a general best practice.
Extra JVM Options
General JVM options can be passed to the Minecraft Server invocation by passing a JVM_OPTS
environment variable. If needing to set -XX options, then pass those with the JVM_XX_OPTS variable instead. Both variables accept space-delimited, raw JVM arguments.
docker run ... -e JVM_OPTS="-XsomeJVMOption -DpropName=value" ...NOTE When declaring JVM_OPTS in a compose file's environment section with list syntax, do not include the quotes:
environment:
- EULA=true
- JVM_OPTS=-XsomeJVMOption -DpropName=valueUsing object syntax is recommended and more intuitive:
environment:
EULA: "true"
JVM_OPTS: "-XsomeJVMOption -DpropName=value"
# or without quotes
# JVM_OPTS: -XsomeJVMOption -DpropName=valueAs a shorthand for passing several system properties as -D arguments, you can instead pass a comma separated list of name=value or name:value pairs with JVM_DD_OPTS. (The colon syntax is provided for management platforms like Plesk that don't allow = inside a value.)
For example, instead of passing
JVM_OPTS: -Dfml.queryResult=confirm -Dname=valueyou can use
JVM_DD_OPTS: fml.queryResult=confirm,name=valueEnable Remote JMX for Profiling
To enable remote JMX, such as for profiling with VisualVM or JMC, set the environment variable ENABLE_JMX to "true", set JMX_HOST to the IP/host running the Docker container, and add a port forwarding of TCP port 7091, such as:
!!! example
With `docker run`
```
-e ENABLE_JMX=true -e JMX_HOST=$HOSTNAME -p 7091:7091
```If needing to map to a different port, then also set the environment variable JMX_PORT to the desired host port.
!!! example
With a compose file:
```yaml
environment:
ENABLE_JMX: true
JMX_HOST: ${HOSTNAME}
JMX_PORT: "7092"
ports:
- "7092:7092"
```Enable Aikar's Flags
Aikar has done some research into finding the optimal JVM flags for GC tuning, which becomes more important as more users are connected concurrently. PaperMC also has an explanation of what the JVM flags are doing.
The set of flags documented there can be added using
-e USE_AIKAR_FLAGS=trueWhen MEMORY is greater than or equal to 12G, then the Aikar flags will be adjusted according to the article.
Enable MeowIce's Flags
MeowIce has created an updated set of JVM flags based on Aikar's flags but with support for optimizations for Java 17 and above
The set of flags documented there can be added by setting the environment variable USE_MEOWICE_FLAGS to true. There is an optional USE_MEOWICE_GRAALVM_FLAGS variable to enable GraalVM specific optimizations, defaults to FALSE.
File: docs/configuration/misc-options.md
Running with a custom server JAR
If you would like to run a custom server JAR, set -e TYPE=CUSTOM and pass the custom server
JAR via CUSTOM_SERVER. It can either be a URL or a container path to an existing JAR file.
If it is a URL, it will only be downloaded into the /data directory if it wasn't already. As
such, if you need to upgrade or re-download the JAR, then you will need to stop the container,
remove the file from the container's /data directory, and start again.
Force re-download of the server file
For VANILLA, FORGE, BUKKIT, SPIGOT, PAPER, CURSEFORGE, SPONGEVANILLA server types, set$FORCE_REDOWNLOAD to some value (e.g. 'true) to force a re-download of the server file for
the particular server type. by adding a -e FORCE_REDOWNLOAD=true to your command-line.
For example, with Paper, it would look something like this:
docker run -d --pull=always \
-v /path/on/host:/data \
-e TYPE=PAPER -e FORCE_REDOWNLOAD=true \
-p 25565:25565 -e EULA=TRUE --name mc itzg/minecraft-serverClean server libraries
By default, supported server types remove stale server libraries during installation by setting CLEAN_SERVER_LIBRARIES to true. Set CLEAN_SERVER_LIBRARIES to false to disable this cleanup if it causes unexpected behavior.
Currently, library cleanup is supported by
TYPE=PAPERTYPE=PURPUR
Running as alternate user/group ID
By default, the container will switch to and run the Minecraft server as user ID 1000 and group ID 1000; however, that can be changed by setting the environment variables UID and GID.
The startup will also skip user switching if the --user/-u argument is passed to docker run or user is set on the compose service.
Extra Arguments
Arguments that would usually be passed to the jar file (those which are written after the filename) can be passed via the EXTRA_ARGS environment variable.
See Custom worlds directory path for an example.
Interactive and Color Console
When RCON is enabled, which is the default, and TTY is enabled on the container, then some server types will output colorized logs and provide a fully interactive console. To access the interactive console, use docker attach (not exec). When finished, make sure to use the sequence Control-P, Control-Q to detach without stopping the container.
If this behavior interferes with the log content, then disable TTY or remove the setting entirely since the default is disabled. In a compose file, set the service's tty parameter to false. On the docker run command-line remove the -t argument.
Server Shutdown Options
To allow time for players to finish what they're doing during a graceful server shutdown, set STOP_SERVER_ANNOUNCE_DELAY to a number of seconds to delay after an announcement is posted by the server.
To set a custom command to run at the start of this delay period, set STOP_SERVER_DELAY_COMMAND to the full command. This will run in place of the announcement.
!!! warning "Increase stop grace period"
The Docker stop grace period must be increased to a value longer than the announce delay. The value to use that is longer than announce delay will vary based upon the amount of time it takes for final world data saving. If the container exits with exit code 137, then that indicates a longer grace period is needed.
The grace period can be increased using [the -t option on `docker compose down`](https://docs.docker.com/compose/reference/down/) or set the [stop_grace_period](https://docs.docker.com/compose/compose-file/05-services/#stop_grace_period) in the compose file.The STOP_SERVER_ANNOUNCE_DELAY can be bypassed by sending a SIGUSR1 signal to the mc-server-runner process.
docker:
docker stop --signal SIGUSR1 mcdocker compose:
docker compose kill --signal SIGUSR1Configuration Options for Minecraft Server Health Monitoring
The image tags include specific variables to simplify configuration for monitoring the health of a Minecraft server:
-e SERVER_HOST=localhost: This variable sets the host address of the Minecraft server to be monitored. By default, it is set tolocalhost, but you can replace it with the actual hostname or IP address of your Minecraft server.-e SERVER_PORT=25565: This variable sets the port number on which the Minecraft server is running. By default, Minecraft servers run on port 25565, but if your server is configured to use a different port, you should replace25565with the correct port number. This helps the monitoring system to accurately check the health status of the Minecraft server on the specified port.
OpenJ9 Specific Options
The openj9 image tags include specific variables to simplify configuration:
-e TUNE_VIRTUALIZED=TRUE: enables the option to
optimize for virtualized environments-e TUNE_NURSERY_SIZES=TRUE: configures nursery sizes where the initial size is 50%
of theMAX_MEMORYand the max size is 80%.
Customize Minecraft server logging
The image can use a templated log4j2 configuration based on PaperMC's logging setup, which is automatically applied for versions that don't require Log4j security patches. This configuration provides rolling logs and advanced logging features by default.
Set the environment variable GENERATE_LOG4J2_CONFIG to "true" to enable the following features.
!!! note
This configures the server logging only. To diagnose container startup, set the `DEBUG` environment variable to "true".Customization via environment variables
You can customize various aspects of the logging behavior using environment variables:
LOG_LEVEL: Root logger level (default:info)text-e LOG_LEVEL=debugROLLING_LOG_FILE_PATTERN: Pattern for rolled log file names (default:logs/%d{yyyy-MM-dd}-%i.log.gz)text-e ROLLING_LOG_FILE_PATTERN="logs/archive/%d{yyyy-MM-dd}-%i.log.gz"ROLLING_LOG_MAX_FILES: Maximum number of archived log files to keep (default:1000)text-e ROLLING_LOG_MAX_FILES=30
Customizing log message formats
For full control over how log messages are formatted, you can customize the Log4j2 pattern layouts using these variables. These use Log4j2 Pattern Layout syntax:
LOG_CONSOLE_FORMAT: Format for console output (what you see indocker logs)
Default:[%d{HH:mm:ss}] [%t/%level]: %msg%nLOG_FILE_FORMAT: Format for file logs (written tologs/latest.log)
Default:[%d{HH:mm:ss}] [%t/%level]: %msg%nLOG_TERMINAL_FORMAT: Format for interactive terminal console (used withdocker attach)
Default:[%d{HH:mm:ss} %level]: %msg%n
Example configurations
Simple timestamp customization (most common use case):
environment:
# What you see in docker logs
LOG_CONSOLE_FORMAT: "[%d{yyyy-MM-dd HH:mm:ss.SSS}] [%t/%level]: %msg%n"
# What gets written to logs/latest.log
LOG_FILE_FORMAT: "[%d{yyyy-MM-dd HH:mm:ss.SSS}] [%t/%level]: %msg%n"Advanced customization:
environment:
LOG_LEVEL: debug
# Custom ISO8601 format with logger names
LOG_CONSOLE_FORMAT: "%d{ISO8601} %-5level [%t] %logger{36} - %msg%n"
LOG_FILE_FORMAT: "%d{ISO8601} %-5level [%t] %logger{36} - %msg%n"
ROLLING_LOG_MAX_FILES: 50Legacy ENABLE_ROLLING_LOGS option
The ENABLE_ROLLING_LOGS environment variable is no longer needed for most use cases, as rolling logs are now enabled by default through the templated configuration. This option is maintained for backward compatibility but is only checked for error reporting when rolling logs cannot be used due to Log4j security patches.
NOTE The templated log4j2 configuration may interfere with interactive/color consoles as described in the section above
Timezone Configuration
You can configure the timezone to match yours by setting the TZ environment variable:
-e TZ=Europe/Londonsuch as:
docker run -d -it --pull=always -e TZ=Europe/London -p 25565:25565 --name mc itzg/minecraft-serverOr mounting /etc/timezone as readonly (not supported on Windows):
-v /etc/timezone:/etc/timezone:rosuch as:
docker run -d -it --pull=always -v /etc/timezone:/etc/timezone:ro -p 25565:25565 --name mc itzg/minecraft-serverHTTP Proxy
You may configure the use of an HTTP/HTTPS proxy by passing the proxy's "host:port" via the environment variable PROXY. In the example compose file it references a Squid proxy. The host and port can be separately passed via the environment variables PROXY_HOST and PROXY_PORT. A | delimited list of hosts to exclude from proxying can be passed via PROXY_NON_PROXY_HOSTS.
Prefer IPv6 Addresses and Stack
The image can be configured to prefer IPv6 addresses and stack by setting the environment variable PREFER_IPV6 to "true". This will ensure the Java properties java.net.preferIPv6Addresses and java.net.preferIPv6Stack are set accordingly for mc-image-helper and the Minecraft server process.
Using "noconsole" option
Some older versions (pre-1.14) of Spigot required --noconsole to be passed when detaching stdin, which can be done by setting -e CONSOLE=FALSE.
Explicitly disable GUI
Some older servers get confused and think that the GUI interface is enabled. You can explicitly
disable that by passing -e GUI=FALSE.
Stop Duration
When the container is signaled to stop, the Minecraft process wrapper will attempt to send a "stop" command via RCON or console and waits for the process to gracefully finish. By default, it waits 60 seconds, but that duration can be configured by setting the environment variable STOP_DURATION to the number of seconds.
Be sure to also increase the shutdown timeout described here for docker compose down and here for docker stop.
Setup only
If you are using a host-attached data directory, then you can have the image setup the Minecraft server files and stop prior to launching the server process by setting SETUP_ONLY to true.
Custom server runner
By default, the image finishes startup by exec'ing mc-server-runner to run the Minecraft server itself. This can be replaced by setting the environment variable SERVER_RUNNER. The arguments passed will start with ones specific to mc-server-runner, followed by --, and then the actual java -jar arguments or equivalent server entry script, such as run.sh for Forge.
[!TIP]
TheSERVER_RUNNERvariable is expanded when passed toexecso thatmc-server-runnercan be wrapped by another executable that is mounted into the container.
Enable Flare Flags
To enable the JVM flags required to fully support the Flare profiling suite, set the following variable:
-e USE_FLARE_FLAGS=trueFlare is built-in to Pufferfish/Purpur, and is available in plugin form for other server types.
Enable support for optimized SIMD operations
To enable support for optimized SIMD operations, the JVM flag can be set with the following variable:
-e USE_SIMD_FLAGS=trueSIMD optimized operations are supported by Pufferfish and Purpur.
For example, to download configuration files for plugins:
environment:
DOWNLOAD_EXTRA_CONFIGS: |
plugins/WorldEdit<https://raw.githubusercontent.com/EngineHub/WorldEdit/refs/heads/version/7.3.x/worldedit-bukkit/src/main/resources/defaults/config.yml
plugins/EssentialsX<https://raw.githubusercontent.com/EssentialsX/Essentials/refs/heads/2.x/Essentials/src/main/resources/config.ymlOr as a single line:
environment:
DOWNLOAD_EXTRA_CONFIGS: "plugins/WorldEdit<https://example.com/worldedit.yml,config<https://example.com/another.yml"The files will be downloaded to /data/ relative paths, so plugins/WorldEdit will be saved as /data/plugins/WorldEdit/config.yml.
!!! note
The downloaded files can be further processed using environment variable replacement or patch definitions
Enable timestamps in init logs
Before the container starts the Minecraft Server its output is prefixed with [init], such as
[init] Starting the Minecraft server...To also include the timestamp with each log, set LOG_TIMESTAMP to "true". The log output will then look like:
[init] 2022-02-05 16:58:33+00:00 Starting the Minecraft server...File: docs/configuration/server-properties.md
To unify management of the Minecraft server container, all known server.properties entries can be managed by the environment variables described in the sections below. Some mappings provide additional functionality above and beyond the properties file and will be described in the sections below.
!!! warning "Version compatibility"
Not all server properties are supported by all versions of Minecraft. Since this image supports a wide range of versions, please consult the [server properties documentation](https://minecraft.wiki/w/Server.properties) for the version you are using.If you prefer to manually manage the server.properties file, set OVERRIDE_SERVER_PROPERTIES to "false". Similarly, you can entirely skip the startup script's creation of server.properties by setting SKIP_SERVER_PROPERTIES to "true".
!!! note
To clear a server property, set the variable to an empty string, such as `-e RESOURCE_PACK=""`. An unset variable is ignored and the existing server property is left unchanged.To see what server.properties will get used by the server, set the environment variable DUMP_SERVER_PROPERTIES to "true" and the contents of server.properties will get output before the server starts.
Placeholders
When declaring a server properties via container environment variables, those values may contain placeholders that are processed when the server.properties file is updated.
The syntax of placeholders is DOS-style, %VAR%, to avoid being processed by Docker or the shell and the following options are available:
%VAR% or %env:VAR%
: Replaced with the resolved value or the environment variable VAR
%date:FMT%
: Formats the current date/time with the given FMT string processed by Java's DateTimeFormatter.
Any declared or resolved environment variable may be referenced, such as VERSION and TYPE. Additionally, Modrinth and Auto CurseForge modpacks will expose the environment variables MODPACK_NAME and MODPACK_VERSION. The originally declared version, such as "LATEST" or "SNAPSHOT", is available in the variable DECLARED_VERSION
!!! example
As a compose file environment entry:
```yaml
MOTD: Running %MODPACK_NAME% version %env:MODPACK_VERSION%
LEVEL: world-%date:yyyy-MM-dd%
```!!! warning
Placeholders are not supported when manually managing `server.properties`Properties
Message of the Day
The message of the day, shown below each server entry in the client UI, can be changed with the MOTD environment variable or a default is computed from the server type and version, such as
A Paper Minecraft Server powered by DockerThat way you can easily differentiate between several server types you may have started.
The section symbol (§) and other unicode characters are automatically converted to allow formatting codes to be used consistently with all server versions. For Minecraft versions less than 1.20, unicode characters in server.properties will be escaped as \uXXXX, by default. That behavior can be altered by setting SERVER_PROPERTIES_ESCAPE_UNICODE to "true" or "false".
!!! example
With `docker run`
-e MOTD="A §l§cMinecraft§r §nserver"
or within a compose file
environment:
MOTD: "A §l§cMinecraft§r §nserver"
rendersTo produce a multi-line MOTD, embed a newline character as \n in the string, such as the following example.
!!! example "Multi-line MOTD"
With `docker run`
```
-e MOTD="Line one\nLine two"
```
From bash shell
```
-e MOTD=$'Line one\nLine two'
```
or within a compose file
```yaml
MOTD: |
line one
line two
# or
# MOTD: "line one\nline two"
```The following example combines a multi-line MOTD with placeholders from the latest version of the installed modpack:
!!! example
```yaml
MOD_PLATFORM: AUTO_CURSEFORGE
CF_SLUG: craftoria
MOTD: |
A %TYPE% server on %VERSION%
running %MODPACK_NAME% %MODPACK_VERSION%
```Difficulty
The difficulty level (default: easy) can be set like:
docker run -d -e DIFFICULTY=hard ...Valid values are: peaceful, easy, normal, and hard, and an
error message will be output in the logs if it's not one of these
values.
Refer to the Minecraft wiki
Whitelist Players
!!! warning "For public servers"
It is very important to consider setting a whitelist of expected players.To whitelist players for your Minecraft server, you can:
- Provide a list of usernames and/or UUIDs separated by commas or newlines via the
WHITELISTenvironment variable - Provide the URL or container path to a whitelist file via
WHITELIST_FILEthat will be retrieved/copied into the standard location
!!! example
In a compose file, a text block can be used to improve maintainability, such as
```yaml
WHITELIST: |
user1
user2
user3
```When either is set, whitelisting of connecting users is enabled.
To change the behavior when the whitelist file already exists, set the variable EXISTING_WHITELIST_FILE to one of the following options:
SKIP
: Skip processing of the whitelist file when one is already present. This is the same as setting the legacy variable OVERRIDE_WHITELIST to "false".
SYNCHRONIZE
: Synchronize the list of users in the file with the WHITELIST or WHITELIST_FILE provided. When using both, WHITELIST will take precedence. This is the same as setting the legacy variable OVERRIDE_WHITELIST to "true".
MERGE
: Merge the list of users from WHITELIST into the existing file. WHITELIST_FILE cannot be used with this option.
SYNC_FILE_MERGE_LIST (default)
: When WHITELIST_FILE is provided it will overwrite an existing whitelist file. Also, if WHITELIST is provided, then those users will be merged into the newly copied file.
!!! note
For versions prior to 1.7.3, `white-list.txt` will be maintained instead. Only usernames are supported for those versions.To enforce the whitelist changes immediately when whitelist commands are used , set ENFORCE_WHITELIST to "true". If managing the whitelist file manually, ENABLE_WHITELIST can be set to "true" to set the white-list property.
!!! tip "Changing user API provider"
When specifying usernames for whitelist and ops processing, use Minecraft: Java Edition profile names, not Xbox gamertags. They can differ even when they belong to the same Microsoft account. A player can review or change their Java profile name on the [Minecraft profile page](https://www.minecraft.net/en-us/msaprofile/mygames/editprofile).
Usernames are resolved using either [PlayerDB](https://playerdb.co/) or [Mojang's API](https://wiki.vg/Mojang_API#Username_to_UUID). The default uses PlayerDB, but can be changed by setting the environment variable `USER_API_PROVIDER` to "mojang". Sometimes one or the other service can become overloaded, which is why there is the ability to switch providers.Op/Administrator Players
Similar to the whitelist, users can be provisioned as operators (aka administrators) to your Minecraft server by
- Providing a list of usernames and/or UUIDs separated by commas or newlines via the
OPSenvironment variable - Providing the URL or container path to an ops file via
OPS_FILEthat will be retrieved/copied into the standard location
!!! example
In a compose file, a text block can be used to improve maintainability, such as
```yaml
OPS: |
user1
user2
user3
```To change the behavior when the ops file already exists, set the variable EXISTING_OPS_FILE to one of the following options:
SKIP
: Skip processing of the ops file when one is already present. This is the same as setting the legacy variable OVERRIDE_OPS to "false".
SYNCHRONIZE
: Synchronize the list of users in the file with the OPS or OPS_FILE provided. When using both, OPS will take precedence. The level and bypassesPlayerLimit will be retained from previous entries. This is the same as setting the legacy variable OVERRIDE_OPS to "true".
MERGE
: Merge the list of users from OPS into the existing file. OPS_FILE cannot be used with this option.
SYNC_FILE_MERGE_LIST (default)
: When OPS_FILE is provided it will overwrite an existing ops file. Also, if OPS is provided, then those users will be merged into the newly copied file.
!!! note
For versions prior to 1.7.3, `ops.txt` will be maintained instead. Only usernames are supported for those versions.Enable/disable initial selection of datapacks
New to 22W42A, datapacks with feature flags can be controlled before initial world creation by setting these to a comma separated list of packs:
INITIAL_ENABLED_PACKS
such as "update_1_20,bundle"INITIAL_DISABLED_PACKS
Server icon
A server icon can be configured by setting the ICON variable to a URL to download or a container path. The image will be automatically downloaded (if a URL), scaled, and converted from any other image format:
!!! example
Using `docker run`:
```
docker run -d -e ICON=http://..../some/image.png ...
```
In compose file:
```yaml
environment:
ICON: http://..../some/image.png
```
Using a file from host filesystem:
```yaml
environment:
ICON: /icon.png
OVERRIDE_ICON: true
volumes:
./icon.png:/icon.png
```By default an existing server-icon.png file will not be replaced, that can be changed by setting OVERRIDE_ICON to "true".
RCON
RCON is enabled by default to allow for graceful shut down of the server and coordination of save state during backups. RCON can be disabled by setting ENABLE_RCON to "false".
!!! warning
Disabling RCON will remove and limit some features, such as interactive and color console support.RCON Password
The default password is randomly generated on each startup. However, you can specify a password using one of the following environment variables:
- Set
RCON_PASSWORDto your desired password. - Set
RCON_PASSWORD_FILEto the path of a file containing the password.
Using RCON_PASSWORD_FILE is the recommended method for managing sensitive data, as it allows full support for Docker Secrets.
??? example
```yaml title="compose.yaml"
services:
mc:
image: itzg/minecraft-server:latest
pull_policy: daily
tty: true
stdin_open: true
ports:
- "25565:25565"
environment:
EULA: "TRUE"
RCON_PASSWORD_FILE: /run/secrets/rcon_pass # Points to the path where the secret is mounted
volumes:
# attach the relative directory 'data' to the container's /data path
- ./data:/data
secrets:
- rcon_pass
secrets:
rcon_pass:
file: ./rcon_password # local file containing the password
```!!! warning
BE CAUTIOUS OF MAPPING THE RCON PORT EXTERNALLY unless you are aware of all the consequences and have set a secure password.
!!! info
Mapping ports (`-p` command line or `ports` in compose) outside the container and docker networking needs to be a purposeful choice. Most production Docker deployments do not need any of the Minecraft ports mapped externally from the server itself.By default, the server listens for RCON on port 25575 within the container. It can be changed with RCON_PORT but only do this if you have a very good reason. DO NOT change rcon.port via server.properties or integrations will break.
Query
Set the environment variable ENABLE_QUERY to "true" to enable the gamespy query protocol. Maps to the server property enable-query. By default, the query port will be 25565 (UDP) but can be changed with the QUERY_PORT environment variable.
Level Seed
If you want to create the Minecraft level with a specific seed, set the environment variable SEED, which maps to the level-seed property.
If using a negative value for the seed, make sure to quote the value such as:
!!! example "Using docker run"
```
-e SEED="-1785852800490497919"
```!!! example "Using compose"
```yaml
environment:
SEED: "-1785852800490497919"
```Game Mode
By default, Minecraft servers are configured to run in Survival mode. You can
change the mode using MODE where you can either provide the standard
numerical values or the
shortcut values:
- creative
- survival
- adventure
- spectator (only for Minecraft 1.8 or later)
For example:
docker run -d -e MODE=creative ...Level Type and Generator Settings
By default, a standard world is generated with hills, valleys, water, etc. A different level type can
be configured by setting LEVEL_TYPE to an expected type listed here.
For some of the level types, GENERATOR_SETTINGS can be used to further customize the world generation.
To configure the GENERATOR_SETTINGS you need to add the appropriate GeneratorOptions JSON configuration. In the case of a superflat world, you may omit the flat_world_options.
The layers are applied from -64 and up and are added in the order of the list
Example for a superflat world:
- 1x bedrock
- 2x stone
- 15x sandstone
- Desert biome
environment:
LEVEL_TYPE: FLAT
GENERATOR_SETTINGS: >-
{
"layers": [
{
"block": "minecraft:bedrock",
"height": 1
},
{
"block": "minecraft:stone",
"height": 2
},
{
"block": "minecraft:sandstone",
"height": 15
}
],
"biome": "minecraft:desert"
}For more details, refer to the Minecraft Wiki sections for Superflat Multiplayer and generator options tag format.
Custom Server Resource Pack
You can set a link to a custom resource pack and set it's checksum using the RESOURCE_PACK and RESOURCE_PACK_SHA1 options respectively, the default is blank:
docker run -d -e 'RESOURCE_PACK=http://link.com/to/pack.zip?=1' -e 'RESOURCE_PACK_SHA1=d5db29cd03a2ed055086cef9c31c252b4587d6d0'You can enforce the resource pack on clients by setting RESOURCE_PACK_ENFORCE to TRUE (default: FALSE).
Level / World Save Name
You can either switch between world saves or run multiple containers with different saves by using the LEVEL option,
where the default is "world":
docker run -d -e LEVEL=bonus ...NOTE: if running multiple containers be sure to either specify a different
-vhost directory for eachLEVELin use or don't use-vand the container's filesystem will keep things encapsulated.
INFO Refer to the data directory section for a visual description of where the
$LEVELdirectory is situated.
Server port
WARNING: only change this value if you know what you're doing. It only needs to be changed when using host-networking and it is rare that host networking should be used. Use
-pport mappings instead.
If you must, the server port can be set like:
!!! example "Using docker run"
```
docker run -d -e SERVER_PORT=25566 ...
```!!! example "Using compose"
```yaml
environment:
SERVER_PORT: 25566
```however, be sure to change your port mapping accordingly and be prepared for some features to break.
Custom server properties
Some mods/plugins utilize custom server.properties entries which can be declared via the CUSTOM_SERVER_PROPERTIES environment variable. The contents must be newline delimited name=value pairs.
Within a compose file, newline delimited entries can be declared as shown here:
CUSTOM_SERVER_PROPERTIES: |
custom1=value1
defaultworldgenerator-port=f8c04631-f744-11ec-b260-f02f74b094e0When using docker run from a bash shell, the entries must be quoted with the $' syntax, such as
-e CUSTOM_SERVER_PROPERTIES=$'k1=v1\nk2=v2'Other server property mappings
!!! warning "Version compatibility"
Not all server properties are supported by all versions of Minecraft. Since this image supports a wide range of versions, please consult the [server properties documentation](https://minecraft.wiki/w/Server.properties) for the version you are using.| Environment Variable | Server Property |
|---|---|
| ACCEPTS_TRANSFERS | accepts-transfers |
| ALLOW_FLIGHT | allow-flight |
| ALLOW_NETHER | allow-nether |
| ANNOUNCE_PLAYER_ACHIEVEMENTS | announce-player-achievements |
| BROADCAST_CONSOLE_TO_OPS | broadcast-console-to-ops |
| BROADCAST_RCON_TO_OPS | broadcast-rcon-to-ops |
| BUG_REPORT_LINK | bug-report-link |
| ENABLE_COMMAND_BLOCK | enable-command-block |
| ENABLE_STATUS | enable-status |
| ENFORCE_SECURE_PROFILE | enforce-secure-profile |
| ENTITY_BROADCAST_RANGE_PERCENTAGE | entity-broadcast-range-percentage |
| FORCE_GAMEMODE | force-gamemode |
| FUNCTION_PERMISSION_LEVEL | function-permission-level |
| GENERATE_STRUCTURES | generate-structures |
| GENERATOR_SETTINGS | generator-settings |
| HARDCORE | hardcore |
| HIDE_ONLINE_PLAYERS | hide-online-players |
| LOG_IPS | log-ips |
| MANAGEMENT_SERVER_ALLOWED_ORIGINS | management-server-allowed-origins |
| MANAGEMENT_SERVER_ENABLED | management-server-enabled |
| MANAGEMENT_SERVER_HOST | management-server-host |
| MANAGEMENT_SERVER_PORT | management-server-port |
| MANAGEMENT_SERVER_SECRET | management-server-secret |
| MANAGEMENT_SERVER_TLS_ENABLED | management-server-tls-enabled |
| MANAGEMENT_SERVER_TLS_KEYSTORE | management-server-tls-keystore |
| MANAGEMENT_SERVER_TLS_KEYSTORE_PASSWORD | management-server-tls-keystore-password |
| MAX_BUILD_HEIGHT | max-build-height |
| MAX_COMMAND_CHAIN_LENGTH | max-command-chain-length |
| MAX_ENTITY_CRAMMING | max-entity-cramming |
| MAX_ENTITY_COLLISION_RADIUS | max-entity-collision-radius |
| MAX_FUNCTION_CHAIN_DEPTH | max-function-chain-depth |
| MAX_NEIGHBORS | max-neighbors |
| MAX_CHAINED_NEIGHBOR_UPDATES | max-chained-neighbor-updates |
| MAX_PLAYERS | max-players |
| MAX_TICK_TIME | max-tick-time |
| MAX_WORLD_SIZE | max-world-size |
| NETWORK_COMPRESSION_THRESHOLD | network-compression-threshold |
| ONLINE_MODE | online-mode |
| OP_PERMISSION_LEVEL | op-permission-level |
| PAUSE_WHEN_EMPTY_SECONDS | pause-when-empty-seconds |
| PLAYER_IDLE_TIMEOUT | player-idle-timeout |
| PREVENT_PROXY_CONNECTIONS | prevent-proxy-connections |
| PVP | pvp |
| RATE_LIMIT | rate-limit |
| REGION_FILE_COMPRESSION | region-file-compression |
| RESOURCE_PACK_ID | resource-pack-id |
| RESOURCE_PACK_PROMPT | resource-pack-prompt |
| SERVER_NAME | server-name |
| SIMULATION_DISTANCE | simulation-distance |
| SNOOPER_ENABLED | snooper-enabled |
| SPAWN_ANIMALS | spawn-animals |
| SPAWN_MONSTERS | spawn-monsters |
| SPAWN_NPCS | spawn-npcs |
| SPAWN_PROTECTION | spawn-protection |
| STATUS_HEARTBEAT_INTERVAL | status-heartbeat-interval |
| SYNC_CHUNK_WRITES | sync-chunk-writes |
| USE_NATIVE_TRANSPORT | use-native-transport |
| VIEW_DISTANCE | view-distance |
File: docs/misc/deployment/index.md
More Deployment Info
On Kubernetes
Using Helm
- itzg Helm Chart:
- mcsh/server-deployment
Using Shulker
Shulker is a Kubernetes operator for managing complex and dynamic Minecraft infrastructures, including game servers and proxies. It uses the docker-minecraft-server and docker-bungeecord images under-the-hood.
With Ansible
Ansible is an open-source task automation tool built in Python. Ansible playbooks can be used to automate all kinds of tasks, including deploying remote Minecraft servers.
Using the MASH playbook
The MASH Playbook is a premade playbook with the option to deploy a wide variety of open-source services to your server(s), including docker-minecraft-server, making it a good option if you want all the bells and whistles alongside your Minecraft server.
- Check out the Installation Guide for the MASH playbook to get started.
- You can then enable Minecraft in your vars.yml.
- Enable any supporting services that you might find useful, such as user authentication, remote backups, email relay, cron monitoring, audio and video conferencing, databases (MariaDB, PostgresSQL), push notifications, uptime monitoring, a website, as well as installing any extra files, folders, applications, services and running commands automatically on deployment.
On CloudFormation (AWS)
If you're looking for a simple way to deploy this to the Amazon Web Services Cloud, check out the Minecraft Server Deployment (CloudFormation) repository. This repository contains a CloudFormation template that will get you up and running in AWS in a matter of minutes. Optionally it uses Spot Pricing so the server is very cheap, and you can easily turn it off when not in use.
Supporting Articles
Below are supporting articles for server deployment.
"Zero to Minecraft Server with Docker Desktop and Compose"
https://dev.to/rela-v/zero-to-minecraft-server-with-docker-desktop-and-compose-500a
- This is a reference guide/tutorial on how to set up a vanilla Minecraft server using this project, including step-by-step instructions, and information on topics such as port-forwarding.
File: docs/misc/contributing/development.md
Adding a server type
Adding a new server TYPE can vary due to the complexity of obtaining and configuring each type; however, the addition of any server type includes at least the following steps:
- Copy an existing "start-deploy*" script, such as start-deployFabric and rename it accordingly making sure to retain the "start-deploy" prefix
- Modify the type-specific behavior between the "start-utils" preamble and the hand-off to
start-setupWorldat the end of the script - Develop and test the changes using the iterative process described below
- Add a case-entry to the
case "${TYPE^^}"in start-configuration - Add a section to the docs. It is recommended to copy-modify an existing section to retain a similar wording and level of detail
- Submit a pull request
Iterative script development
The included compose-dev.yml will mount the local scripts code into the container and allow for iterative development. Replace [-e key=value] with any environment variables you wish to set for testing the modified scripts.
docker compose -f compose-dev.yml run --rm -it [-e key=value] mc-dev!!! tip
To speed up the development cycle, it is recommended to set `SETUP_ONLY` to `true` as part of the run command above.Building the image with a new release of a tool
In this exapmle, let's say that mc-image-helper has been released at 1.56.0, but the corresponding changes in the image scripts need to be tested against that version while updating the Dockerfile.
services:
mc:
build:
# ...or wherever you cloned the docker-minecraft-server repo
context: ../../..
args:
MC_HELPER_VERSION: 1.56.0
environment:
EULA: true
TYPE: "FABRIC"
MODRINTH_PROJECTS: |
fabric-api
pl3xmap?:beta
ports:
- "25565:25565/tcp"
volumes:
- ./data:/dataUsing development copy of tools
In the cloned repo, such as mc-image-helper, install the distribution locally by running:
./gradlew installDistThe distribution will be installed in the project's build/install/mc-image-helper. Obtain the absolute path to that directory use in the next step.
Refer to the instructions above to mount any locally modified image scripts or build a local copy of the image using or with alternate BASE_IMAGE, as described above:
docker build -t itzg/minecraft-server .Mount the local mc-image-helper distribution directory as a volume in the container at the path /usr/share/mc-image-helper, such as
docker run -it --rm \
-v /path/to/mc-image-helper/build/install/mc-image-helper:/usr/share/mc-image-helper \
-e EULA=true \
itzg/minecraft-serverIn a compose file, include the volume mount in the volumes section of the container definition:
services:
mc:
# ... usual container definition
volumes:
- /path/to/mc-image-helper/build/install/mc-image-helper:/usr/share/mc-image-helper:roFor Go base tools, run
goreleaser release --snapshot --cleanClone itzg/github-releases-proxy and run it according to the instructions shown there.
In the Docker build, configure the following
--build-arg GITHUB_BASEURL=http://host.docker.internal:8080 \
--build-arg APPS_REV=1and declare one or more version overrides, such as
--build-arg MC_HELPER_VERSION=1.8.1-SNAPSHOTGenerating release notes
The following git command can be used to provide the bulk of release notes content:
git log --invert-grep --grep "^ci:" --grep "^misc:" --grep "^docs:" --pretty="* %s" 1.1.0..1.2.0File: docs/misc/contributing/docs.md
Site documentation
The documentation for this image/repository is written in markdown and built using Zensical into a documentation website hosted at Read the Docs. Here is general information about writing Zensical markdown.
!!! note
The README.md rarely needs to be modified and only serves as a brief introduction to the project.
The documentation source is maintained in the docs folder and is organized into sections by directory and files. Look through the existing content to determine if an existing file should be updated or a new file/directory added.
It will be very helpful to view the rendered documentation as you're editing. To do that run the following from the top-level directory:
docker compose -f compose-docs.yml -p zensical upYou can access the live documentation rendering at http://localhost:8000.
2. Official Technical Reference & Guides (itzg/docs)
File: README.md
GRAKN.AI Documentation
Dependencies
You need to install the following dependencies to be able to build HTML pages in the documentation repository. NOTE: this is not necessary for updating the documentation itself.
Bundle; you will need to install
bundlerthrough your package manager of choice.Arch Linux
text$ yaourt -S ruby-bundlerOSX
text$ brew install bundlerRuby Gems (generic)
text$ gem install bundlerRake; this is used to automate the rest of the site building process.
text$ gem install rake
With rake installed you can now install all other dependencies:
$ rake dependenciesEnvironment variables
There is an environment variable that needs to be set depending on the build you wish to commence: urlprefix. If you only wish to preview the generated documentation itself; as server from this repository rather than integrated into the GRAKN.AI website you do not need to do anything.
`urlprefix`
For local builds (local web server with documentation only), do not set this variable, or make sure it has been cleared with:
$ export urlprefix=""For integrating to GRAKN.AI website, or elsewhere set as needed. For example; www.grakn.ai/docs/
$ export urlprefix=/docsBuilding
You can generate the documentation HTML by running the following in the repository top level.
$ rake build
Generating...
done in 1.503 seconds.
Auto-regeneration: disabled. Use --watch to enable.
$This will build the documentation site in _jekyll and create a symlink _site in the repository top level directory which will contain all the generated content.
Cleaning
Clean by running the following command in the repository top level:
$ rake cleanThis will remove all generated files.
Serving
You can also build and server the generated HTML files in one command. A web
server will be started listening on localhost (127.0.0.1) on port 4005
$ rake serve
Server address: http://127.0.0.1:4005/
Server running... press ctrl-c to stop.
$You can now view the documentation by navigating your web browser to http://127.0.0.1:4005/
File: pages/documentation/11-examples/analytics.md
title: Using Grakn for Statistical Analysis keywords: analytics, machine-learning last_updated: March 2017 tags: [analytics, examples] summary: "How to use Grakn for statistical analysis" sidebar: documentation_sidebar permalink: /documentation/examples/analytics.html folder: documentation comment_issue_id: 27
Introduction
This example illustrates the use of aggregate and compute queries in Graql using a simple dataset to calculate statistics. The code for the example can be found on our sample-projects repository, and is also included in the examples folder of the Grakn distribution from v0.12.0.
For a detailed overview of calculating statistics using Graql, we recommend that you take a look at the documentation for:
Data
This example takes a dataset that will be familiar to students of R - mtcars (Motor Trend Car Road Tests) data. The data was extracted from the 1974 Motor Trend US magazine, and comprises fuel consumption and 10 other aspects of automobile design and performance for 32 automobiles (1973–74 models). We have created a csv file of the data and added two columns to indicate the car maker's name and region that the car was made in (Europe, Japan or North America). The readme file in the repository gives further information for anyone who wishes to migrate the mtcars data directly into a graph, but for the purposes of this example we provide a single data file that you can load to populate a graph.
Ontology
We have provided the following ontology to represent the data, although many other variations are possible:
insert
# Entities
vehicle sub entity
is-abstract;
car sub vehicle
is-abstract
has-resource model
has-resource mpg
has-resource cyl
has-resource disp
has-resource hp
has-resource wt
has-resource gear
has-resource carb
plays-role made;
automatic-car sub car;
manual-car sub car;
carmaker sub entity
is-abstract
has-resource maker-name
plays-role maker;
japanese-maker sub carmaker;
american-maker sub carmaker;
european-maker sub carmaker;
# Resources
model sub resource datatype string;
maker-name sub resource datatype string;
mpg sub resource datatype double;
cyl sub resource datatype long;
disp sub resource datatype double;
hp sub resource datatype long;
wt sub resource datatype double;
gear sub resource datatype long;
carb sub resource datatype long;
# Roles and Relations
manufactured sub relation
has-role maker
has-role made;
maker sub role;
made sub role;To load ontology.gql into Grakn, make sure the engine is running and choose a clean keyspace in which to work (here we use the default keyspace, so we are cleaning it before we get started).
<relative-path-to-Grakn>/bin/grakn.sh clean
<relative-path-to-Grakn>/bin/grakn.sh start
<relative-path-to-Grakn>/bin/graql.sh -f ./ontology.gql
```
## Data Migration
We migrated the CSV data using template Graql files, but for ease of use, we provide a single data file that you can load to populate a graph.
```bash
<relative-path-to-Grakn>/bin/graql.sh -b ./data.gql
```
Spin up the [Grakn visualiser](../grakn-dashboard/visualiser.html) by pointing your browser to [http://localhost:4567/](http://localhost:4567/). You can submit queries to check the data, or explore it using the Types dropdown menu.
Some sample queries:
```graql
# Cars where the model name contains "Merc"
match $x has model contains "Merc";
# Cars with more than 4 gears
match $x has gear > 4;
# Japanese-made cars that are manual
match $x isa manual-car; $y isa japanese-maker;
# European cars that are automatic
match $x isa automatic-car; $y isa european-maker;At this point, you are ready to start investigating statistics within the data using Graql aggregate and compute queries:
`aggregate`
You cannot make aggregate queries from within the Graph view in the Grakn visualiser, so you will need to switch views using the left hand navigation pane, from Graph to Console. This shows a read-write view on Grakn, and you can now submit queries in the usual way, via the form. Alternatively, from your terminal, you can start the Graql shell in its interactive (REPL) mode by typing graql.sh at the terminal, from within the bin directory of the Grakn installation.
Here are some example aggregate queries to try:
# Count of all cars
match $x isa car; aggregate count; # 32
# Count American car makers
match $x isa american-maker; aggregate count; # 6
# Maximum MPG for an automatic car
match $x isa automatic-car, has mpg $a; aggregate max $a;
# Minimum HP for all cars
match $x isa car, has hp $hp; aggregate min $hp;
# Mean MPG for manual and automatic cars
match $x isa manual-car has mpg $mpg; aggregate mean $mpg; # 24.39
match $x isa automatic-car has mpg $mpg; aggregate mean $mpg; # 17.15
# Median number of cylinders (all Mercedes cars)
match $x isa carmaker has maker-name contains "Mercedes";
$y isa car has cyl $c; (maker:$x, made:$y); aggregate median $c; # 6
# Or...
match $x has model contains "Merc", has cyl $c; aggregate median $c;# 6
# Maximum number of carburetors (all Chrysler cars)
match $x isa carmaker has maker-name contains "Chrysler"; $y isa car has carb $c; (maker:$x, made:$y); aggregate max $c; # 4
# Minimum number of gears (all cars)
match $x isa car, has gear $g; aggregate min $g; # 3`compute`
Graql also provides compute queries that can be used to determine values such as mean, minimum and maximum. These can be submitted using the Graph view on the Visualiser. For example, type each of the following into the form and submit:
# Number of automatic and manual cars
compute count in automatic-car; # 19
compute count in manual-car; # 13
# Number of Japanese car makers
compute count in japanese-maker; # 4
# Median number of cylinders (all cars)
compute median of cyl; # 6
# Minimum number of gears (all cars)
compute min of gear; # 3
# Maximum number of carburetors (all cars)
compute max of carb; # 8When to Use `aggregate` and When to Use `compute`?
Aggregate queries are computationally light and run single-threaded on a single machine, and are more flexible than the equivalent compute query (for example, you can use an aggregate query to filter results by resource).
match $x has identifier contains "Elizabeth"; aggregate count;Compute queries are computationally intensive and run in parallel on a cluster, so are good for big data and can be used to calculate results very fast. However, you can't filter the results by resource in the same way as you can for an aggregate query.
Inference
match $x has powerful contains "TRUE" has economical contains "TRUE";
match $x has powerful contains "TRUE" has economical contains "TRUE" has model $y;
$x id "106584" isa manual-car; $y value "Ferrari Dino" isa model;
$x id "254120" isa automatic-car; $y value "Pontiac Firebird" isa model;Where Next?
If you haven't already, we recommend that you review the documentation about Graql analytics, since there is more to compute than just statistical analysis. Unfortunately, this example is not a good one to illustrate clusters, degrees or shortest path analytics, which is why it isn't described here.
This example was based on CSV data migrated into Grakn. Having read it, Yyou may want to further study our documentation about CSV migration and Graql templating.
{% include links.html %}
Comments
Want to leave a comment? Visit the issues on Github for this page (you'll need a GitHub account). You are also welcome to contribute to our documentation directly via the "Edit me" button at the top of the page.
File: pages/documentation/11-examples/CSV-migration.md
title: An Example of Migrating CSV to Grakn keywords: migration last_updated: January 2017 tags: [migration, examples] summary: "An example to illustrate migration of CSV to Grakn" sidebar: documentation_sidebar permalink: /documentation/examples/CSV-migration.html folder: documentation comment_issue_id: 27
Introduction
This example looks at the migration of genealogy data in CSV format to build a graph in GRAKN.AI. The data is used as the basis of a blog post that illustrates the fundamentals of the Grakn visualiser, reasoner and analytics components.
As the blog post explains, the original data was a document from Lenzen Research that described the family history of Catherine Niesz Titus for three generations of her maternal lineage.
In this example, we will walk through how to migrate the CSV data into Grakn, and confirm that we have succeeded using the Grakn visualiser.
For a detailed overview of CSV migration, we recommend that you take a look at the Grakn documentation on CSV Migration and Graql templating.
Genealogy Data
The data for this example can be found as a set of CSV files in the sample-projects repository on Github, which is also included in the Grakn distribution zip file. The data was put together by our team from narrative information gleaned from the original Lenzen Research document, with some minor additions to generate some interesting queries for Grakn's reasoner.
Let's take a look at the raw-data directory in the example project, which contains the CSV files. These files were put together by hand by our team, mostly by Michelangelo.
|filename|description|
|people.csv| This comprehensive CSV contains information about all the individuals discussed in the Lenzen document. For each row, it lists the available information about individuals' names, gender, birth and death dates and age at death. It also assigns each person a person ID ("pid"), which is a string containing the full name of each individual, and is used for identification of individuals in the other CSV files.|
|births.csv| This CSV lists the person IDs of a child and each of its parents|
|weddings.csv| This CSV comprises a row for each marriage, identifying each by a wedding ID (wid). The rows contain the person IDs of each spouse and the date of their wedding, where it is known.|
Ontology
The ontology is a way to describe the entities and their relationships, so the underlying graph can store them as nodes and edges. You can find out more in our guide to the Grakn Knowledge Model. The ontology allows Grakn to perform:
- logical reasoning over the represented knowledge, such as the extraction of implicit information from explicit data (inference)
- discovery of inconsistencies in the data (validation).
The ontology is shown below. There is a single entity, person, which has a number of resources and can play various roles (parent, child, spouse1 and spouse2) in two possible relations (parentship and marriage).
insert
# Entities
person sub entity
plays-role parent
plays-role child
plays-role spouse1
plays-role spouse2
has-resource identifier
has-resource firstname
has-resource surname
has-resource middlename
has-resource picture
has-resource age
has-resource birth-date
has-resource death-date
has-resource gender;
# Roles and Relations
marriage sub relation
has-role spouse1
has-role spouse2
has-resource picture;
spouse1 sub role;
spouse2 sub role;
parentship sub relation
has-role parent
has-role child;
parent sub role;
child sub role;
# Resources
identifier sub resource datatype string;
firstname sub resource datatype string;
surname sub resource datatype string;
middlename sub resource datatype string;
picture sub resource datatype string;
age sub resource datatype long;
birth-date sub resource datatype string;
death-date sub resource datatype string;
gender sub resource datatype string;To load ontology.gql into Grakn, make sure the engine is running and choose a clean keyspace in which to work (here we use the default keyspace, so we are cleaning it before we get started).
<relative-path-to-Grakn>/bin/grakn.sh clean
<relative-path-to-Grakn>/bin/grakn.sh start
<relative-path-to-Grakn>/bin/graql.sh -f ./ontology.gqlData Migration
Having loaded the ontology, the next steps are to populate the graph by migrating data into Grakn from CSV.
We will consider three CSV files that contain data to migrate into Grakn.
people.csv
The people.csv file contains details of the people that we will use to create seven person entities. Note that not all fields are available for each person, but at the very least, each row is expected to have the following:
- pid (this is the person identifier, and is a string representing their full name)
- first name
- gender
name1,name2,surname,gender,born,dead,pid,age,picture
Timothy,,Titus, male,,, Timothy Titus,,
Mary,,Guthrie,female,,,Mary Guthrie,,
John,,Niesz,male,1798-01-02,1872-03-06,John Niesz,74,
Mary,,Young,female,1798-04-09,1868-10-28,Mary Young,70,
William,Sanford,Titus,male,1818-03-23,01/01/1905,William Sanford Titus,76,
Elizabeth,,Niesz,female,1820-08-27,1891-12-08,Elizabeth Niesz,71,
Mary,Melissa,Titus,female,1847-08-12,10/05/1946,Mary Melissa Titus,98,
...The migrator is a set of template Graql statements that instruct the Grakn migrator on how the CSV data can be mapped to the ontology. The Grakn migrator applies the template to each row of data in a CSV file, replacing the indicated sections in the template with the value from a specific cell, identified by the column header (the key). This sounds complicated, but isn't really, as we will show.
The Graql template code for the people migrator is as follows:
insert
$p isa person has identifier
has firstname <name1>,
if (<surname> != "") do
{
has surname <surname>,
}
if (<name2> != "") do
{
has middlename <name2>,
}
if ( != "") do
{
has picture ,
}
if (<age> != "") do
{
has age @long(<age>),
}
if ( != "") do
{
has birth-date ,
}
if (<dead> != "") do
{
has death-date <dead>,
}
has gender <gender>;For each row in the CSV file, the template inserts a person entity with resources that take the value of the cells in that row. Where data is optional, the template checks to see if it is present before adding the resources for middlename, surname, picture, age, birth and death dates.
Calling the Grakn migrator on the people.csv file using the above template (named people-migrator.gql) is performed as follows:
<relative-path-to-Grakn>/bin/migration.sh csv -i ./people.csv -t ./migrators/people-migrator.gql -k grakn{% include note.html content="Usage instructions for the Grakn migrator are available as part of our migration documentation." %}
The data insertion generated by the migrator is as follows:
insert $p0 has death-date "1891-12-08" isa person has gender "female" has identifier "Elizabeth Niesz" has surname "Niesz" has age 71 has firstname "Elizabeth" has birth-date "1820-08-27";
insert $p0 has identifier "William Sanford Titus" has age 76 isa person has firstname "William" has surname "Titus" has gender "male" has birth-date "1818-03-23" has middlename "Sanford" has death-date "1905-01-01";
insert $p0 isa person has surname "Titus" has firstname "Timothy" has identifier "Timothy Titus" has gender "male";
insert $p0 isa person has firstname "Mary" has identifier "Mary Guthrie" has surname "Guthrie" has gender "female";
insert $p0 isa person has firstname "Mary" has death-date "1946-05-10" has surname "Titus" has age 98 has identifier "Mary Melissa Titus" has middlename "Melissa" has gender "female" has birth-date "1847-08-12";
insert $p0 has death-date "1872-03-06" has age 74 has identifier "John Niesz" isa person has birth-date "1798-01-02" has firstname "John" has gender "male" has surname "Niesz";
insert $p0 has identifier "Mary Young" has birth-date "1798-04-09" isa person has firstname "Mary" has death-date "1868-10-28" has surname "Young" has gender "female" has age 70;
...births.csv
Each row of births.csv records a parent and child, with two rows for each of the three children listed:
parent,child
Timothy Titus,William Sanford Titus
Mary Guthrie, William Sanford Titus
John Niesz,Elizabeth Niesz
Mary Young,Elizabeth Niesz
Elizabeth Niesz,Mary Melissa Titus
William Sanford Titus,Mary Melissa Titus
...The Graql template code for the Grakn migrator is as follows:
match
$c isa person has identifier <child>;
$p isa person has identifier ;
insert
(child: $c, parent: $p) isa parentship;For each row in the CSV file, the template matches the child and parent cells to their corresponding person entities, and then inserts a parentship relation, placing the entities it has matched into the child and parent roles.
{% include note.html content="You must ensure that all entities exist in a graph before inserting relations." %}
Calling the Grakn migrator on the births.csv file using the above template (named births-migrator.gql) is performed as follows:
<relative-path-to-Grakn>/bin/migration.sh csv -i ./births.csv -t ./migrators/births-migrator.gql -k graknThe data insertion generated by the migrator is as follows:
match $c0 has identifier "William Sanford Titus" isa person; $p0 isa person has identifier "Timothy Titus";
insert (child: $c0, parent: $p0) isa parentship;
match $p0 isa person has identifier "Mary Guthrie"; $c0 has identifier "William Sanford Titus" isa person;
insert (child: $c0, parent: $p0) isa parentship;
match $p0 isa person has identifier "Elizabeth Niesz"; $c0 isa person has identifier "Mary Melissa Titus";
insert (child: $c0, parent: $p0) isa parentship;
match $c0 isa person has identifier "Mary Melissa Titus"; $p0 has identifier "William Sanford Titus" isa person;
insert (child: $c0, parent: $p0) isa parentship;
match $c0 isa person has identifier "Elizabeth Niesz"; $p0 has identifier "John Niesz" isa person;
insert (child: $c0, parent: $p0) isa parentship;
match $c0 isa person has identifier "Elizabeth Niesz"; $p0 has identifier "Mary Young" isa person;
insert (child: $c0, parent: $p0) isa parentship;
...weddings.csv
The weddings.csv file contains two columns that correspond to both spouses in a marriage, and an optional column for a photograph of the happy couple:
spouse1,spouse2,picture
Timothy Titus,Mary Guthrie,
John Niesz,Mary Young,http://1.bp.blogspot.com/-Ty9Ox8v7LUw/VKoGzIlsMII/AAAAAAAAAZw/UtkUvrujvBQ/s1600/johnandmary.jpg
Elizabeth Niesz,William Sanford Titus,The Graql template code for the migrator is as follows:
match
$x has identifier <spouse1>;
$y has identifier <spouse2>;
insert
(spouse1: $x, spouse2: $y) isa marriage
if ( != "") do
{
has picture
};For each row in the CSV file, the template matches the two spouse cells to their corresponding person entities, and then inserts a marriage relation, placing the entities it has matched into the spouse1 and spouse2 roles. If there is data in the picture cell, a picture resource is also created for the marriage relation.
Calling the Grakn migrator on the weddings.csv file using the above template (named weddings-migrator.gql) is performed as follows:
<relative-path-to-Grakn>/bin/migration.sh csv -i ./weddings.csv -t ./migrators/weddings-migrator.gql -k graknThe Graql insertion code is as follows:
match $x0 has identifier "Timothy Titus"; $y0 has identifier "Mary Guthrie";
insert (spouse1: $x0, spouse2: $y0) isa marriage;
match $x0 has identifier "John Niesz"; $y0 has identifier "Mary Young";
insert has picture "http:\/\/1.bp.blogspot.com\/-Ty9Ox8v7LUw\/VKoGzIlsMII\/AAAAAAAAAZw\/UtkUvrujvBQ\/s1600\/johnandmary.jpg" (spouse1: $x0, spouse2: $y0) isa marriage;
match $y0 has identifier "William Sanford Titus"; $x0 has identifier "Elizabeth Niesz";
insert (spouse1: $x0, spouse2: $y0) isa marriage;
...Migration Script
For simplicity, the /raw-data/ directory of the example project contains a script called loader.sh that calls each migration script in turn, so you can simply call the script from the terminal, passing in the path to the Grakn /bin/ directory.
./loader.sh <relative-path-to-Grakn>/binThe migration will take a minute or two, and the terminal will report which file it is migrating at each step. When it is complete, it will report that it is "Done migrating data". To check that it has been successful, open the Grakn visualiser and select Types, then Entities, and choose one of those presented to you (the entities should be those described above]. The visualiser will display the entities it has imported. The screenshot below illustrates the result from selecting to see all person entities.
We have completed the data import, and the graph can now be queried. For example, from the Graql shell:
match $x isa person, has identifier $i; aggregate count;There should be 60 people in the dataset.
Data Export
In this example, we have imported a dataset stored in three separate CSV files into Grakn to build a simple graph. We have discussed the ontology and migration templates, and shown how to apply the templates to the CSV data using the shell migrator, using a script file loader.sh to automate calling the migrator on each file It is possible to export the data from Grakn, in .gql format, so that it can easily be loaded to a graph again without the need to migrate from CSV.
To export the data, we use the Grakn migration.sh shell script again, as described in the migration documentation:
# Export the ontology
<relative-path-to-Grakn>/migration.sh export -ontology > ontology-export.gql
# Export the data
<relative-path-to-Grakn>/migration.sh export -data > data-export.gqlExporting data or the ontology from Grakn, into Graql, will always redirect to standard out, so above we are sending the output to an appropriately named file.
Where Next?
This example has illustrated how to migrate CSV data into Grakn. Having read it, you may want to further study our documentation about CSV migration and Graql templating.
{% include links.html %}
Comments
Want to leave a comment? Visit the issues on Github for this page (you'll need a GitHub account). You are also welcome to contribute to our documentation directly via the "Edit me" button at the top of the page.
File: pages/documentation/11-examples/examples-overview.md
title: Examples Overview keywords: examples last_updated: September 23, 2016 tags: [getting-started, examples] summary: "Landing page for Grakn examples." sidebar: documentation_sidebar permalink: /documentation/examples/examples-overview.html folder: documentation comment_issue_id: 27
Introduction
This page lists the examples of Grakn that we suggest you study to learn how to work with our stack. We plan to continue to expand our set, and we also encourage you to let us know if you have example code to share, so we can link to it (for example, to your repo or a blog post).
If you would like to request a particular example, please get in touch with us by leaving a comment on this page or posting a question on the discussion boards. Our Community page lists other ways you can talk to us.
Genealogy Dataset
The genealogy dataset is widely used across our documentation about GRAKN.AI, because it allows for a simple, yet powerful, illustration of many key features. As described below, it is used to illustrate CSV migration and the Grakn Reasoner.
It is available on the sample-datasets repo on Github, and is also discussed in the "Family Matters" blog post.
Use Cases
CSV Migration
There are several examples available:
We use the genealogy dataset to show how to migrate genealogy data into Grakn from CSV.
Our sample-projects repository on Github also contains an example that takes a simple CSV data file of pets. Please see the readme file to get started.
CSV migration is also covered in a blog post.
JSON Migration
There is an example of using the Java Migration API for JSON migration on the sample-projects repository on Github.
SQL Migration
There are several examples available:
A common use-case is to migrate existing SQL data to a graph in Grakn. We walk through a simple example of using the migration script as part of the documentation about SQL migration
There is a an additional example of SQL migration using the Java API.
We also cover SQL migration in a blog post.
OWL Migration
For more information about migrating OWL to Grakn using the Java migration API, please see the OWL migration example.
Reasoning with Graql
We use the genealogy dataset to illustrate how to write rules to infer new information from a dataset. You can find the example here.
Learn Graql
- The Modern example is a simple one, designed to test your knowledge of Graql.
- We have a simple Pokemon example to illustrate how to form a range of different Graql queries.
- The philosophers.gql file, also distributed in the Grakn release zip, contains a simple ontology and data, for use as an example.
Languages
Haskell, R and Python Bindings
It is possible to extract data from Grakn and use it as a data science tool for analysis. You can take the results of a Graql query and store the results in a dataframe or similar structure, for use with Haskell, R or Python.
- Haskell: This blog post is the first in a series of posts about combining GRAKN.AI and Haskell.
- R and Python: This blog post explains and gives a simple example.
- Python: A further blog post uses the Python driver to examine our example movie dataset.
Java Examples
- Use of the core APIs for creating an ontology, adding data and making basic queries is covered in a blog post.
- JSON migration: There is an example of using the Java Migration API for JSON migration on the sample-projects repository on Github.
- SQL migration: We have documented an example of SQL migration using the Java API.
- Pokemon: The sample-projects repo on Github contains a Java project that uses the Java Graph API on Pokemon data and an ontology.
- Philosophers: The sample-projects repo on Github contains a Java project that uses the Java Graph API on a Philosophers dataset and ontology.
Moogi Movie Database
Moogi is a large database of information about movies. We have provided a subset of its data to try out in our sample-datasets repository on Github.
Where Next?
If you are interested in writing an example on Grakn, maybe as a way of trying it out, please take a look at the Example Projects page, which lists some ideas that we have for potential examples or research projects.
{% include links.html %}
Comments
Want to leave a comment? Visit the issues on Github for this page (you'll need a GitHub account). You are also welcome to contribute to our documentation directly via the "Edit me" button at the top of the page.
File: pages/documentation/11-examples/graql-rules.md
title: Reasoning with Graql keywords: migration last_updated: February 2017 tags: [reasoning, examples] summary: "An example to illustrate inference using genealogy data." sidebar: documentation_sidebar permalink: /documentation/examples/graql-reasoning.html folder: documentation comment_issue_id: 27
Introduction
This is an example of how to use the reasoning facilities of Graql to infer information about family relationships from a simple dataset of genealogy data. The data has been used previously as the basis of a blog post to illustrate the fundamentals of the Grakn visualiser as well as reasoning and analytics capabilities.
As the blog post explained, the original data was a document from Lenzen Research that described the family history of Catherine Niesz Titus for three generations of her maternal lineage. Our team gathered together a set of CSV files containing basic information about the family, such as names, dates of birth, death and marriage, who was a parent of who, and who married who.
The full genealogy-graph project can be found on Grakn's sample-datasets repository on Github. However, we will use a simpler ontology for this example, which is found in basic-genealogy.gql, located within the /examples folder of the Grakn installation. The file can also be found on Github.
In this example, we will explore how to use Grakn to make inferences and find information from the data that is not explicit in the dataset. You can find documentation about writing rules in Graql here.
Ontology and Data
On GRAKN.AI, the first step when working with a dataset is to define its ontology in Graql. The ontology is a way to describe the entities and their relations, so the underlying graph can store them as nodes and edges. You can find out more in our guide to the Grakn Knowledge Model. The ontology allows Grakn to perform:
- logical reasoning over the represented knowledge, such as the extraction of implicit information from explicit data (inference)
- discovery of inconsistencies in the data (validation).
The complete ontology for the genealogy-graph demo is in basic-genealogy.gql.
The ontology contains a person entity, and a number of possible family relationships between person entities, such as parent/child, siblings, grandparent/grandchild, which we will discuss shortly. The data for this example is discussed more fully in our example on CSV migration, which discusses how the original data is laid out in CSV files and migrated into GRAKN.AI. However, for this example, you do not need any familiarity with CSV migration or the origin of the data we are using.
Loading the Example
The basic-genealogy.gql file contains the ontology, data and rules needed for this example. To load it into a graph, make sure the engine is running and choose a clean keyspace in which to work (in the example below we use the default keyspace, so we are cleaning it before we get started):
<relative-path-to-Grakn>/bin/grakn.sh clean
<relative-path-to-Grakn>/bin/grakn.sh start
<relative-path-to-Grakn>/bin/graql.sh -f <relative-path-to-Grakn>/examples/basic-genealogy.gqlWhen the terminal prompt returns, the data will have been loaded into the default keyspace, and you can start to look it using the Grakn visualiser, by navigating to http://localhost:4567 in your browser. Submit a query, such as match $x isa person to check that all is well with your graph.
Inferring Family Relations
By default, inference is switched off, and the only information you can query Grakn about is what was directed loaded from the data. For example, if you submit the following query to the visualiser:
match (child: $c, parent: $p) isa parentship;You will receive parentship results, but if you clear the query and then submit a new query for gender-specific results, say mother-daughter relationships:
match (mother: $c, daughter: $p) isa parentship;You will receive no results at all. To find inferred relations between the people in our dataset, you need to activate inference in the Grakn visualiser. Open the Query settings under the cog button, which is on the far right hand side of the horizontal icon menu (at the top of the screen).
You will see the "Activate inference" checkbox. Check it, and Grakn is ready to start building some new information about the family. Try the query again:
match (mother: $c, daughter: $p) isa parentship;You should see something similar to the screenshot below in your visualiser window.
{% include note.html content="You can alternatively make queries in the graql shell. You will need to use ./graql.sh -n to enable inference when you start the shell." %}
Inference Rules
Inference is the process of deducing new information from available data. For example, given the following statements:
If grass is not an animal.
If vegetarians only eat things which are not animals.
If sheep only eat grass.It is possible to infer the following:
Then sheep are vegetarians.The initial statements can be seen as a set of premises. If all the premises are met we can infer a new fact (that sheep are vegatarians). If we hypothetise that sheep are vegetarians then the whole example can be expressed with a particular two-block structure: IF some premises are met, THEN a given hypothesis is true.
This is how reasoning in Graql works. It checks whether the statements in the first block can be verified and, if they can, infers the statement in the second block. The rules are written in Graql, and we call the first set of statements (the IF part or, if you prefer, the antecedent) simply the left hand side (LHS). The second part, not surprisingly, is the right hand side (RHS). Using Graql, both sides of the rule are enclosed in curly braces and preceded by, respectively, the keywords lhs and rhs.
{% include note.html content="The full documentation for writing rules in Graql is available from here." %}
Example 1: Specific relations between parents and children
As we saw above, it is possible for Grakn to infer the gender-specific roles (mother, father, daughter, son) that a person entity plays. It does this by applying the following rules:
$genderizeParentships1 isa inference-rule
lhs
{(parent: $p, child: $c) isa parentship;
$p has gender "male";
$c has gender "male";
}
rhs
{(father: $p, son: $c) isa parentship;};
$genderizeParentships2 isa inference-rule
lhs
{(parent: $p, child: $c) isa parentship;
$p has gender "male";
$c has gender "female";
}
rhs
{(father: $p, daughter: $c) isa parentship;};
$genderizeParentships3 isa inference-rule
lhs
{(parent: $p, child: $c) isa parentship;
$p has gender "female";
$c has gender "male";
}
rhs
{(mother: $p, son: $c) isa parentship;};
$genderizeParentships4 isa inference-rule
lhs
{(parent: $p, child: $c) isa parentship;
$p has gender "female";
$c has gender "female";
}
rhs
{(mother: $p, daughter: $c) isa parentship;};The four rules can be broken down as follows:
- LHS: In the
parentshiprelation between child$cand parent$p, do they both have agenderresource that ismale?- RHS: The
parentshiprelation is betweenfatherandson
- RHS: The
- LHS: In the
parentshiprelation between child$cand parent$p, does$chave agenderresource that isfemaleand$phave agenderresource that ismale?- RHS: The
parentshiprelation is betweenfatheranddaughter
- RHS: The
- LHS: In the
parentshiprelation between child$cand parent$p, does$chave agenderresource that ismaleand$phave agenderresource that isfemale?- RHS: The
parentshiprelation is betweenmotherandson
- RHS: The
- LHS: In the
parentshiprelation between child$cand parent$p, do both have agenderresource that isfemale?- RHS: The
parentshiprelation is betweenmotheranddaughter
- RHS: The
We can use the rule to easily discover the sons who have the same name as their fathers:
match (father: $p, son: $c) isa parentship; $p has firstname $n; $c has firstname $n;In the genealogy-graph example, there should be two results returned. William and John are names shared between father/son pairs.
Example 2: A `grandparentship` relation
The basic-genealogy file contains a number of rules for setting up family relationships, such as siblings, cousins, in-laws and the following, which sets up a relation called grandparentship:
$parentsOfParentsAreGrandparents isa inference-rule
lhs
{(parent:$p, child: $gc) isa parentship;
(parent: $gp, child: $p) isa parentship;
}
rhs
{(grandparent: $gp, grandchild: $gc) isa grandparentship;};Here, the left hand side rules check two statements:
- does
$pplay theparentrole in aparentshiprelation, with ($gc) playing thechildrole? - does
$palso play thechildrole in aparentshiprelation, with ($gp) playing theparentrole?
If so, the right hand side of the rules state that:
$gpand$gcare in agrandparentshiprelation, where$gpplays thegrandparentrole and$gcplays thegrandchildrole.
Some additional rules can add more specifics to the grandparentship and assign the entities to the roles grandson, granddaughter, grandmother and grandfather:
$grandParents1 isa inference-rule
lhs
{($p, son: $gc) isa parentship;
(father: $gp, $p) isa parentship;
}
rhs
{(grandfather: $gp, grandson: $gc) isa grandparentship;};
$grandParents2 isa inference-rule
lhs
{($p, daughter: $gc) isa parentship;
(father: $gp, $p) isa parentship;
}
rhs
{(grandfather: $gp, granddaughter: $gc) isa grandparentship;};
$grandParents3 isa inference-rule
lhs
{($p, daughter: $gc) isa parentship;
(mother: $gp, $p) isa parentship;
}
rhs
{(grandmother: $gp, granddaughter: $gc) isa grandparentship;};
$grandParents4 isa inference-rule
lhs
{($p, son: $gc) isa parentship;
(mother: $gp, $p) isa parentship;
}
rhs
{(grandmother: $gp, grandson: $gc) isa grandparentship;};Much as above for the parentship relation, the rules can be broken down as follows:
- LHS: There is a
parentshiprelation between son$gcand parent$p, and anotherparentshiprelation where$pis now in the child role and$gpis a father.- RHS: The
grandparentshiprelation is betweengrandfatherandgrandson
- RHS: The
- LHS: There is a
parentshiprelation between daughter$gcand parent$p, and anotherparentshiprelation where$pis now in the child role and$gpis a father.- RHS: The
grandparentshiprelation is betweengrandfatherandgranddaughter
- RHS: The
- LHS: There is a
parentshiprelation between daughter$gcand parent$p, and anotherparentshiprelation where$pis now in the child role and$gpis a mother.- RHS: The
grandparentshiprelation is betweengrandmotherandgranddaughter
- RHS: The
- LHS: There is a
parentshiprelation between son$gcand parent$p, and anotherparentshiprelation where$pis now in the child role and$gpis a mother.- RHS: The
grandparentshiprelation is betweengrandmotherandgrandson
- RHS: The
These rules allow us to find all grandparentship relations, and further, it allows us to query, for example, which grandfather/grandson pairs share the same name:
match (grandfather: $x, grandson: $y) isa grandparentship; $x has firstname $n; $y has firstname $n;In the genealogy-graph example, there should be three results returned. George, Jacob and John are names shared between grandfather/grandson pairs.
Example 3: A `cousins` relation
Another rule can be used to infer person entities who are cousins:
$peopleWithSiblingsParentsAreCousins isa inference-rule
lhs
{
(parent: $p, child: $c1) isa parentship;
($p, $p2) isa siblings;
(parent: $p2, child: $c2) isa parentship;
}
rhs
{(cousin1: $c1, cousin2: $c2) isa cousins;};$pis aparentand has achild$c1$pis a sibling to$p2$p2has achild$c2- Then
$c1and$c2are in acousinsrelationship.
It would also be possible to infer uncle/aunt/nephew/niece relationships, using a similar approach. We will leave that as an exercise for the reader!
Other Queries
There are some queries that are currently not possible. For example:
- Who has no children?
- Who has never married?
- Who has the most grandchildren?
- Who has the most brothers?
If you want to infer information from a dataset such as this example, and want to check whether it is possible, please reach out to us via Slack or our discussion forum.
Where Next?
This example has illustrated the basics of the inference rules used by Graql. Having read it, you may want to further study our documentation about Graql Rules and further investigate the example that imports the genealogy data from CSV format into Grakn.
We will be using the genealogy data throughout our documentation. For an overview, please see our Family Matters blog post.
Comments
Want to leave a comment? Visit the issues on Github for this page (you'll need a GitHub account). You are also welcome to contribute to our documentation directly via the "Edit me" button at the top of the page.
{% include links.html %}
File: pages/documentation/11-examples/JSON-migration-NOT-PUBLISHED.md
title: An Example of Migrating JSON to Grakn keywords: migration last_updated: February 2017 tags: [migration, examples] summary: "An example to illustrate migration of JSON to Grakn" sidebar: documentation_sidebar permalink: /documentation/examples/JSON-migration.html folder: documentation comment_issue_id: 27
Introduction
This example looks at the migration of simple JSON data to build a graph in GRAKN.AI. The data is an invented library catalogue of a very small number of books, and illustrates how to migrate simple data and handle issues such as preventing the creation of duplicate entities. You can find the data, ontology and templates for this example in our sample projects repo on Github.
For a detailed overview of CSV migration, we recommend that you take a look at the Grakn documentation on JSON Migration and Graql templating.
Data
The JSON data for the book catalogue is as follows:
{
"books": [
{
"title":"The Gruffalo",
"id":"1",
"author":"Julia Donaldson",
"subject":"Children's Fiction"
},
{
"title":"The Tear Thief",
"id":"2",
"author":"Carol Ann Duffy",
"subject":"Children's Fiction"
},
{
"title":"Selected Poems",
"id":"3",
"author":"Carol Ann Duffy",
"subject":"Poetry"
},
...
]
}The id value can be considered to be unique and identify each book. However, other values cannot be assumed to be unique. In the snippet shown above, note that the author Carol Ann Duffy and subject Children's Fiction are both duplicated. It is possible that a book title may also be duplicated, although this is not shown.
Ontology
The ontology for the book catalogue is as follows:
insert
# Entities
book sub entity
has-resource bookId
has-resource title
plays-role publication-item;
author sub entity
has-resource authorName
plays-role publication-author;
subject sub entity
has-resource subjectName
plays-role publication-subject;
# Resources
bookId sub resource datatype string;
title sub resource datatype string;
authorName sub resource datatype string;
subjectName sub resource datatype string;
# Relations and Roles
publication sub relation
has-role publication-item
has-role publication-author
has-role publication-subject;
publication-item sub role;
publication-author sub role;
publication-subject sub role;Here, there are three entities, to reflect the book, author of the book and possible book subjects. There is one relation, publication which between all three entities.
To load ontology.gql into Grakn, make sure the engine is running and choose a clean keyspace in which to work (here we use the default keyspace, so we are cleaning it before we get started).
<relative-path-to-Grakn>/bin/grakn.sh clean
<relative-path-to-Grakn>/bin/grakn.sh start
<relative-path-to-Grakn>/bin/graql.sh -f ./ontology.gqlData Migration
Having loaded the ontology, the next steps are to populate the graph by migrating data into Grakn, using Graql templates. There are separate templates for each entity. First, for the books:
insert
for() do {
$book isa book
has bookId
has title <title>;
}The authors:
insert
for() do {
$author isa author
has authorName <author>;
}The subjects:
insert
for() do {
$subject isa subject
has subjectName <subject>;
}Finally, a template to build the relations between the entities:
To call the migration script on each template:
./bin/migration.sh json -t ./json/book-template.gql -i ./json/library-data.json -k grakn
./bin/migration.sh json -t ./json/author-template.gql -i ./json/library-data.json -k grakn
./bin/migration.sh json -t ./json/subject-template.gql -i ./json/library-data.json -k grakn
./bin/migration.sh json -t ./json/publication-template.gql -i ./json/library-data.json -k graknThe resultant Graql for the migration:
Visualise the Data
Add Borrower Data
Data Export
# Export the ontology
<relative-path-to-Grakn>/migration.sh export -ontology > ontology-export.gql
# Export the data
<relative-path-to-Grakn>/migration.sh export -data > data-export.gqlExporting data or the ontology from Grakn, into Graql, will always redirect to standard out, so above we are sending the output to an appropriately named file.
Where Next?
{% include links.html %}
Comments
Want to leave a comment? Visit the issues on Github for this page (you'll need a GitHub account). You are also welcome to contribute to our documentation directly via the "Edit me" button at the top of the page.
File: pages/documentation/11-examples/modern.md
title: Modern Example in Graql keywords: graql, query last_updated: September 19, 2016 tags: [graql, examples] summary: "A short example to illustrate Graql queries" sidebar: documentation_sidebar permalink: /documentation/examples/modern.html folder: documentation comment_issue_id: 27
If you have not yet set up the Grakn environment, please see the Setup guide. For a comprehensive guide to all Graql keywords, please see the Graql documentation.
Introduction
We have a few examples on how to work with Graql using different datasets. For example, see our blog posts and the Quickstart Tutorial that introduces how to make Graql queries.
This example takes a very simple example from TinkerPop3 Documentation, called TinkerPop Modern. Here it is, shown diagrammatically:
The image above is used from the documentation provided for TinkerPop3, and licensed by the Apache Software Foundation.
We have chosen this example as it may already be familiar, and is simple enough to demonstrate some of the fundamentals of Graql. We walk through the entities ("things") and relationships between them and show how to represent them using Graql to define a schema. We then use Graql to add the data to the graph. The main purpose of this example, however, is to use it for practice at making sample queries on the graph.
Starting Graql
If it is not already running, start Grakn, then open a Graql shell:
cd [your Grakn install directory]
bin/grakn.sh start
bin/graql.shDefining a Schema
In the example, we have a number of entities:
- people (marko, vadas, josh and peter), each of which has an age.
- software (lop and ripple), each of which has an associated programming language.
The relationships between the entities are straightforward:
- there are two connections between marko and other people who he "knows" (josh and vadas).
- there are four connections between people and software, such that the people "created" the software. Lop was created by marko, peter and josh, which ripple was created by josh.
Entities
Here, we add person and software entities, via the Graql shell:
insert person sub entity;
insert software sub entity;Resources
To assign resources to the entities, which you can think of as attributes, we use resources. First, we define what they are (age is a number, programming language is a string that represents the language's name), then we allocate them to the entity in question:
insert age sub resource datatype long;
insert name sub resource datatype string;
insert person has-resource age, has-resource name;
insert lang sub resource datatype string;
insert software has-resource lang has-resource name;
insert weight sub resource datatype double;Relations
Let's first define the relationship between people. The diagram shows that marko knows vadas, but we don't have any information about whether the inverse is true (though it seems likely that vadas probably also knows marko). Let's set up a relationship called knows, which has two roles - knower (for marko) and known-about (for vadas):
insert knower sub role;
insert known-about sub role;
insert person plays-role knower;
insert person plays-role known-about;
insert knows sub relation, has-role knower, has-role known-about, has-resource weight;Note that the knows relation also has an attribute, in the form of a resource called weight (though it's not clear from the TinkerPop example what this represents).
We can set up a similar relationship between software and the people that created it:
insert programmer sub role;
insert programmed sub role;
insert person plays-role programmer;
insert software plays-role programmed;
insert programming sub relation, has-role programmer, has-role programmed, has-resource weight;And that's it. At this point, we have defined the schema of the graph.
Adding the Data
Now we have a schema, we can move on to adding in the data, which is pretty much just a typing exercise:
insert $marko isa person, has name "marko", has age 29;
insert $vadas isa person, has name "vadas", has age 27;
insert $josh isa person, has name "josh", has age 32;
insert $peter isa person, has name "peter", has age 35;
match $marko has name "marko"; $josh has name "josh"; insert (knower: $marko, known-about: $josh) isa knows has weight 1.0;
match $marko has name "marko"; $vadas has name "vadas"; insert (knower: $marko, known-about: $vadas) isa knows has weight 0.5;insert $lop isa software, has lang "java", has name "lop";
insert $ripple isa software, has lang "java", has name "ripple";
match $marko has name "marko"; $lop has name "lop"; insert (programmer: $marko, programmed: $lop) isa programming has weight 0.4;
match $peter has name "peter"; $lop has name "lop"; insert (programmer: $peter, programmed: $lop) isa programming has weight 0.2;
match $josh has name "josh"; $lop has name "lop"; insert (programmer: $josh, programmed: $lop) isa programming has weight 0.4;
match $josh has name "josh"; $ripple has name "ripple"; insert (programmer: $josh, programmed: $ripple) isa programming has weight 1.0;Querying
This example is designed to get you up close and personal with Graql queries. It will run through a few basic examples, then ask you a set of "Test Yourself" questions.
OK, so if you've followed the above, you should now have a schema and some data in a graph. How do you go about using the graph to answer your queries? That's where the match statement comes in.
As with any query language, you use a variable to receive the results of the match query, which you must prefix with a $. So, to make the query "List every person in the graph", you would use the following in Graql:
>>> match $x isa person, has name $n; select $n;
$n value "vadas" isa name;
$n value "marko" isa name;
$n value "josh" isa name;
$n value "peter" isa name;In Graql, a match is formed of three parts: the match statement, an optional select statement, and any other optional modifiers that you choose to apply to the listing of results. Only the first part of a match query is needed: the modifier parts are optional.
In the match $x isa person query we are not using any select or delimiters, so let's add some now. We can add a select statement to ask Graql to list out every person and to include their id (which is their name) and age. We use order by to modify how the results are listed out - in this case, we order them by ascending age, so the youngest person is shown first.
>>> match $x isa person, has name $n, has age $a; select $n, $a; order by $a asc;
$x id "vadas" has age "27";
$x id "marko" has age "29";
$x id "josh" has age "32";
$x id "peter" has age "35";Complete Example
Here is the complete example - the code to define the schema and insert the data into a graph. You can load this directly into Graql, if you don't want to type it out for yourself. Cut and paste the Graql below and start Graql:
bin/graql.shThen type edit, which will open up the systems default text editor where you can paste your chunk of text. Upon exiting the editor, the Graql will execute.
insert
age sub resource datatype long;
name sub resource datatype string;
person sub entity;
person has-resource age, has-resource name;
$marko isa person;
$vadas isa person;
$josh isa person;
$peter isa person;
$marko has age 29, has name "marko";
$josh has age 32, has name "josh";
$vadas has age 27, has name "vadas";
$peter has age 35, has name "peter";
weight sub resource datatype double;
knower sub role;
known-about sub role;
person plays-role knower;
person plays-role known-about;
knows sub relation
has-role knower
has-role known-about
has-resource weight;
(knower: $marko, known-about: $josh) isa knows has weight 1.0;
(knower: $marko, known-about: $vadas) isa knows has weight 0.5;
lang sub resource datatype string;
software sub entity;
software has-resource lang, has-resource name;
$lop isa software;
$ripple isa software;
$lop has lang "java", has name "lop";
$ripple has lang "java", has name "ripple";
programmer sub role;
programmed sub role;
person plays-role programmer;
software plays-role programmed;
programming sub relation
has-role programmer
has-role programmed
has-resource weight;
(programmer: $marko, programmed: $lop) isa programming has weight 0.4;
(programmer: $peter, programmed: $lop) isa programming has weight 0.2;
(programmer: $josh, programmed: $lop) isa programming has weight 0.4;
(programmer: $josh, programmed: $ripple) isa programming has weight 1.0;Test Yourself
List every person with their name and age in ascending age order
List every person who has an age over 30
List every person who knows someone else, and every person who is known about
List every person that Marko knows
List every item of software and the language associated with it
List everything that Josh has programmed
List everyone who has programmed Lop
List everything you know about Marko
{% include links.html %}
Comments
Want to leave a comment? Visit the issues on Github for this page (you'll need a GitHub account). You are also welcome to contribute to our documentation directly via the "Edit me" button at the top of the page.
File: pages/documentation/11-examples/OWL-migration.md
title: An Example of Migrating OWL to Grakn keywords: migration last_updated: September 19, 2016 tags: [migration, examples] summary: "A short example to illustrate migration of OWL to Grakn" sidebar: documentation_sidebar permalink: /documentation/examples/OWL-migration.html folder: documentation comment_issue_id: 27
For a comprehensive guide to migration, please see both our Migration Tutorial and our additional documentation specific to OWL Migration.
Introduction
This example addresses the topic of OWL interoperability through a migration of a family tree OWL ontology. The code for this example can be found in our github repository.
The sole prerequisite of this example is having the Grakn environment installed and the Engine running. If you need help starting Grakn Engine, please see the setup guide.
Running the example
You can run this example by running the Main class. Check out the OWLResourceMigrator class for the bulk of the migration code.
We run a few queries in the example to prove that the data has been migrated. After running the example, you should be able to answer the following questions (see the bottom of the page for answers - but please don't peek until you've tried it!):
- What are the Types in the Family Tree graph in Grakn?
- How many people are in the family tree?
- How many descendants does Eleanor Pringle have?
- Who are the great uncles of Ethel Archer?
Test Yourself Answers
What are the Types in the Family Tree graph?
Answer:
tThing
tPerson
tMan
tWoman
```
**How many people are in the family tree?**
Answer: `411`
**How many descendants does Eleanor Pringle have?**
Answer: `55`
**Who are the great uncles of Ethel Archer?**
Answer:William Whitfield
Harry Whitfield
George Whitfield
Walter Whitfield
James Whitfield
## Where next?
After running this OWL migration, check out the [Graql documentation](../graql/graql-overview.html) and the [Graph API documentation](../developing-with-java/graph-api.html) for more instructions on how you can explore the family tree.
{% include links.html %}
## Comments
Want to leave a comment? Visit [the issues on Github for this page](https://github.com/graknlabs/docs/issues/27) (you'll need a GitHub account). You are also welcome to contribute to our documentation directly via the "Edit me" button at the top of the page.
---
## File: pages/documentation/11-examples/pokemon.md
---
title: Pokemon Example
keywords: graql, query
last_updated: January 2017
tags: [graql, examples]
summary: "A short example to illustrate Graql queries"
sidebar: documentation_sidebar
permalink: /documentation/examples/pokemon.html
folder: documentation
comment_issue_id: 27
---
This example uses the pokemon ontology and dataset, which is available from [github](https://github.com/graknlabs/grakn/blob/master/grakn-dist/src/examples/pokemon.gql) and is also in the *examples* folder in the Grakn distribution zip. You can use it as a primer to work out if you're building a Graql query correctly, to revise your knowledge or just walk through it as you learn Graql.
First up, you need to load the ontology and dataset (which are all in one file) as follows. Here we will use the default keyspace, so will clean it before we start Grakn engine and open the Graql shell.
```bash
<relative-path-to-Grakn>/bin/grakn.sh clean
<relative-path-to-Grakn>/bin/grakn.sh start
<relative-path-to-Grakn>/bin/graql.sh -f <relative-path-to-Grakn>/examples/pokemon.gql
<relative-path-to-Grakn>/bin/graql.shHere are the queries. Type each one in a separate line and see what happens. Comments are marked with a #, so you can ignore them.
/* Detailed source-code truncated for AI context efficiency. */Comments
Want to leave a comment? Visit the issues on Github for this page (you'll need a GitHub account). You are also welcome to contribute to our documentation directly via the "Edit me" button at the top of the page.
File: pages/documentation/11-examples/projects.md
title: Example Projects
keywords: examples, projects
last_updated: October 7, 2016
tags:
summary: "A list of potential projects that use Grakn."
sidebar: documentation_sidebar
permalink: /documentation/examples/projects.html
folder: documentation
Introduction
This page lists potential research projects that could be based upon Grakn. We have estimated the amount of work involved in them as T-shirt sizes (small, medium and large) and indicated the level of expertise we think would be required. If you are unsure of whether an idea would be suitable for you, please do talk to us. We are always happy to discuss your ideas, either based on these suggestions or for other projects. Just see our Community page for ways to get in touch with us!
Small
The following would particularly suit an undergraduate project.
Rule-based Game Solver
Define a simple ontology, which is just a specification of the components of a system and how they relate, and then devise logical rules to construct a simple application. An example may be to produce a set of rules that can validate that a Sudoku puzzle has been filled correctly.
Links Between Likes
Add a new feature to Moogi that allows users to “like” concepts. Then produce a visualisation that displays the link between the things that a user likes. For more information about Moogi, the film discovery engine that is built on Grakn, please see https://moogi.co/.
Moogi Data Cleanup
Clean Moogi data and port it to Mindmaps. This requires dabbling a bit with Gremlin, scripting, exploring the graph and ETL probably.
Graph Database Pros and Cons
Write a review of Graph Database technologies with pros and cons and example apps.
JavaScript Driver for Grakn
Create a prototype to interface with the Grakn REST API in a simple manner.
Create a Graph Representation of a Website
Write a simple tool that follows links in a website and constructs a graph representation of it in Mindmaps. This could simply be pages and links, but could even pull out structural information such as containers and side panes inside pages. Follow on projects could be:
- mapping click stream data onto the graph - and analysing
- analysing clusters and shortest paths through the website.
Graph a Subject's (e.g. Computer Science) Publications
Build a graph of academic publications on a particular subject for analysis.
Sports Results
For example, a knowledge graph of the Rio Olympics stats dataset, Premier League results etc.
Recommendation Engine
Shows how to leverage connections in your data to gather insights and start recommending currently unrelated but relevant facts.
Medium
The following would particularly suit an undergraduate project.
Game Search Engine
Develop an application that shows how a graph database can excel over RDBMS for certain domains. For example, a game search engine that has features like search by: genre, ratings, price, keywords, etc. You could pick any area you are interested in (books, music, recipes - the list goes on - the main requirement is something with a lot of highly connected data).
Once this project has been tried and tested, it may be interesting to abstract it further and look at building a factory that creates the basic search engine, and can then be customised according to subject area and features.
Large
The following would be most suitable for postgraduate investigation.
Semantic Machine Learning
Using Machine Learning to work with the semantics of the graph.
Publication Topology Based Ranking
Semantic search engine for computer science publications. How do ranking results based on topology compare to user data? Can we use an ensemble measure to get “better” results?
Life Sciences Knowledge Graph
There is potential to use the open data from Elixir and Bioschemas.org to build a life science graph.
European Broadcast Union
The European Broadcast Union are interested interested in semantics and have documented what they are pursuing here.
Comments
Want to leave a comment? Visit the issues on Github for this page (you'll need a GitHub account). You are also welcome to contribute to our documentation directly via the "Edit me" button at the top of the page.
{% include links.html %}