cayley Documentation

File: README.md Copy Chapter

Cayley is an open-source database for Linked Data. It is inspired by the graph database behind Google's Knowledge Graph (formerly Freebase).

[Documentation](https://cayley.gitbook.io/cayley/) Copy Chapter

Features Copy Chapter

• Built-in query editor, visualizer and REPL
• Multiple query languages:
  • Gizmo: query language inspired by Gremlin
  • GraphQL-inspired query language
  • MQL: simplified version for Freebase fans
• Modular: easy to connect to your favorite programming languages and back-end stores
• Production ready: well tested and used by various companies for their production workloads
• Fast: optimized specifically for usage in applications

Performance Copy Section

Rough performance testing shows that, on 2014 consumer hardware and an average disk, 134m quads in LevelDB is no problem and a multi-hop intersection query -- films starring X and Y -- takes ~150ms.

Community Copy Chapter

• Website: cayley.io
• Slack: cayleygraph.slack.com -- Invite here
• Discourse list: discourse.cayley.io (Also acts as mailing list, enable mailing list mode)
• Twitter: @cayleygraph
• Involvement: Contribute

File: appengine/README.md Copy Chapter

About appengine

This directory contains a appengine deployable cayley graph instance.

Running/testing traditional appengine locally

Install the latest appengine sdk, and ensure that your local version of go
matches the same version obtained from goapp.

Note: if you use a more recent version of Go locally, goapp get might
accidentally copy the wrong files from the local $GOPATH.

$ cd $GOPATH/src/github.com/cayleygraph/cayley/appengine

# check go version
$ go version

# check goapp version
$ goapp version

# install dependencies
$ goapp get

# ensure that goapp can build
$ goapp build

# test locally
$ goapp serve

Running/testing flexible appengine locally

The latest appengine flexible environment can be tested instead of the
traditional environment by doing the following:

$ go generate

$ goapp serve app.flexible.yaml

Deploying to traditional appengine environment

$ goapp deploy -version 1 -application <MY_PROJECT_ID>

Deploying to flexible appengine environment

# install aedeploy if not installed
$ go get -u google.golang.org/appengine/cmd/aedeploy

$ go generate

# deploy
$ aedeploy gcloud app deploy app.flexible.yaml

File: docs/deployment/container.md Copy Chapter

Running in Docker

Running in Kubernetes Copy Chapter

To run Cayley in K8S check this docs section.

Running in a container Copy Chapter

A container exposing the HTTP API of Cayley is available.

Running with default configuration Copy Section

Container is configured to use BoltDB as a backend by default.

docker run -p 64210:64210 -d ghcr.io/cayleygraph/cayley

New database will be available at http://localhost:64210.

Custom configuration Copy Section

To run the container one must first setup a data directory that contains the configuration file and optionally contains persistent files (i.e. a boltdb database file).

mkdir data
cp cayley_example.yml data/cayley.yml
cp data/testdata.nq data/my_data.nq
# initialize and serve database
docker run -v $PWD/data:/data -p 64210:64210 -d ghcr.io/cayleygraph/cayley -c /data/cayley.yml --init -i /data/my_data.nq
# serve existing database
docker run -v $PWD/data:/data -p 64210:64210 -d ghcr.io/cayleygraph/cayley -c /data/cayley.yml

Other commands Copy Section

Container runs cayley http command by default. To run any other Cayley command reset the entry point for container:

docker run -v $PWD/data:/data ghcr.io/cayleygraph/cayley --entrypoint=cayley version

File: docs/deployment/k8s-1.md Copy Chapter

Running in Kubernetes

Most examples requires Kubernetes 1.5+ and PersistentVolumes are configured.

After running scripts namespace cayley will be created and service with the same name will be available in cluster. Service is of type ClusterIP by default. If you want to expose it, consider changing type to LoadBalancer.

Single instance \(Bolt\) Copy Chapter

This is a simplest possible configuration: single Cayley instance with persistent storage, using Bolt as a backend.

kubectl create -f ./docs/k8s/cayley-single.yml

Distributed \(MongoDB cluster\) Copy Chapter

This example is based on thesandlord/mongo-k8s-sidecar and runs Cayley on top of 3-node Mongo cluster.

kubectl create -f ./docs/k8s/cayley-mongo.yml

Distributed Copy Chapter

TODO: PostgreSQL, CockroachDB

File: docs/getting-involved/contributing.md Copy Chapter

Contributing

Contributing Copy Chapter

Community Involvement Copy Section

Join our community on discourse.cayley.io or other Locations.

Simply building Cayley Copy Section

If your version of Go < 1.13, you need to run:

export GO111MODULE=on

Follow the instructions for running Cayley locally:

# clone project
git clone https://github.com/cayleygraph/cayley
cd cayley

# Download dependencies
go mod download

# Update web files (optional)

go run cmd/download_ui/download_ui.go

Generate static files go modules Copy Chapter

packr2

build the binary Copy Chapter

go build ./cmd/cayley

try the generated binary Copy Chapter

./cayley help

Give it a quick test with:

./cayley repl -i data/testdata.nq

To run the web frontend, replace the "repl" command with "http"

./cayley http -i data/testdata.nq

You can now open the WebUI in your browser: http://127.0.0.1:64210

Hacking on Cayley Copy Section

First, you'll need Go (version 1.11.x or greater) and a Go workspace. This is outlined by the Go team at http://golang.org/doc/code.html and is sort of the official way of going about it.

If your version of Go < 1.13, you need to run:

export GO111MODULE=on

If you just want to build Cayley and check out the source, or use it as a library, a simple go get github.com/cayleygraph/cayley will work!

But suppose you want to contribute back on your own fork (and pull requests are welcome!). A good way to do this is to set up your $GOPATH and then...

mkdir -p $GOPATH/src/github.com/cayleygraph
cd $GOPATH/src/github.com/cayleygraph
git clone https://github.com/$GITHUBUSERNAME/cayley

...where $GITHUBUSERNAME is, well, your GitHub username :) You'll probably want to add

cd cayley
git remote add upstream http://github.com/cayleygraph/cayley

So that you can keep up with the latest changes by periodically running

git pull --rebase upstream

With that in place, that folder will reflect your local fork, be able to take changes from the official fork, and build in the Go style.

For iterating, it can be helpful to, from the directory, run

packr2 && go build ./cmd/cayley && ./cayley <subcommand> <your options>

Which will also resolve the relevant static content paths for serving HTTP.

Reminder: add yourself to CONTRIBUTORS and AUTHORS.

Running Unit Tests Copy Section

If your version of Go < 1.13, you need to run:

export GO111MODULE=on

First, cd into the cayley project folder and run:

go test ./...

If you have a Docker installed, you can also run tests for remote backend implementations:

go test -tags docker ./...

If you have a Docker installed, you only want to run tests for a specific backend implementations eg. mongodb

go test -tags docker ./graph/nosql/mongo

Integration tests can be enabled with environment variable:

RUN_INTEGRATION=true go test ./...

File: docs/getting-involved/glossary.md Copy Chapter

Glossary of Terms

Note: this definitions in this glossary are sequenced so that they build on each other, one to the next, rather than alphabetically.

triple Copy Chapter

1. a data entity composed of subject-predicate-object, like "Bob is 35" or "Bob knows Fred". (A predicate in traditional grammar...is seen as a property that a subject has or is characterized by.) source and [source](https://en.wikipedia.org/wiki/Predicate_(grammar)#Predicates_in_traditional_grammar)

triplestore Copy Chapter

1. a purpose-built database for the storage and retrieval of triples... source

quad Copy Chapter

1. where triples have the form {subject, predicate, object}, quads would have a form along the lines of {subject, predicate, object, context} source

2. You can add context or extra values to triples that identifies them and makes it easy to define subgraphs, or named properties. source

3. From Cayley godoc:

   go Copy

   type Quad struct {
    Subject   Value `json:“subject”`
    Predicate Value `json:“predicate”`
    Object    Value `json:“object”`
    Label     Value `json:“label,omitempty”`
   }

link Copy Chapter

1. Another name for a triple, since it "links" any two nodes.
2. Given the triple {A, knows, C} you would say in graph terminology that A and C are "vertices" while knows is an "edge". You would also say that A, knows, and C are all "nodes", and they are "linked" to one another by the triple.

IRI Copy Chapter

1. IRI is an RDF Internationalized Resource Identifier. source
2. An IRI (Internationalized Resource Identifier) within an RDF graph is a Unicode string that conforms to the syntax defined in RFC 3987. source
3. IRIs are a generalization of URIs that permits a wider range of Unicode characters. Every absolute URI and URL is an IRI, but not every IRI is an URI. source

RDF Copy Chapter

1. Resource Description Framework, basically a set of standards defined around quads
2. An RDF triple consists of three components:
   1. the subject, which is an IRI or a blank node
   2. the predicate, which is an IRI
   3. the object, which is an IRI, a literal or a blank node source

RDF store, quad store, named graph, semantic graph database Copy Chapter

1. ...persisting RDF — storing it — became a thing, and these stores were called triple stores. Next they were called quad stores and included information about context and named graphs, then RDF stores, and most recently they call themselves “semantic graph database.” source
2. Adding a name to the triple makes a "quad store" or named graph. source

Cayley Copy Chapter

1. Cayley is a quad store that supports multiple storage backends. It supports multiple query languages for traversing and filtering the named graphs formed by its quads, and it has associated tooling such as a CLI, HTTP server, and so on.

Gizmo Copy Chapter

1. A Gremlin/TinkerPop-inspired query language for Cayley. Looks a lot like JavaScript, the syntax is documented here.

g.V\(\) Copy Chapter

1. For Gremlin/TinkerPop, g.V() returns a list of all the vertices in the graph
2. .v() is for "Vertex" in Gizmo, and it is used like pathObject = graph.Vertex([nodeId],[nodeId]...) (see [[path|#path]])

inbound/outbound predicate Copy Chapter

1. Inbound/outbound refers to the direction of a relation via a predicate. In the case of the triple "A follows B", "follows" is an outbound predicate for A and an inbound predicate for B.

   In/out predicates can be expressed in a query language, for example using the format resultSet = subject.out(predicate) to discover matching Objects. In the case of the triple "A follows B", A.out(“follows”) would return a set of nodes which contains B. An excellent example of this sort of query format is given in the Gremlin/TinkerPop homepage example:

   javascript Copy

   What are the names of projects that were created by two friends?
    g.V().match(
      as(“a”).out(“knows”).as(“b”),
      as(“a”).out(“created”).as(“c”),
      as(“b”).out(“created”).as(“c”),
      as(“c”).in(“created”).count().is(2)).
        select(“c”).by(“name”)

direction Copy Chapter

1. Direction specifies a node's position within a quad. source

   go Copy

   const (
    Any Direction = iota
    Subject
    Predicate
    Object
    Label
   )

2. Direction is passed to the Get method of a quad to access one of its four parts, see quad.Get(d Direction) Value

3. The term "Direction" comes about from the concept of traversing a graph. Take for example the triple {A, follows, B} and supposing you "select" the predicate follows. Now you want to traverse the graph, so you move in the Object direction, and you now have B selected. Whereas the high-level path abstraction for queries uses inbound/outbound predicates to represent movement on the graph, the bottom-level iterator mechanic uses Direction.

path Copy Chapter

1. Paths are just a set of helpers to build a query, but they are not that good for building something more complex. You can try using Shapes for this - it will give you a full control of what the query actually does. source

2. Path represents either a morphism (a pre-defined path stored for later use), or a concrete path, consisting of a morphism and an underlying QuadStore. source

3. Underlying code:

   go Copy

   type Path struct {
      stack       []morphism
      qs          graph.QuadStore
      baseContext pathContext
   }
   
   type morphism struct {
      IsTag    bool
      Reversal func(*pathContext) (morphism, *pathContext)
      Apply    applyMorphism
      tags     []string
   }
   
   type applyMorphism func(shape.Shape, *pathContext) (shape.Shape, *pathContext)

   So, as previously stated, the path package is just helper methods on top of the shape package.

morphism Copy Chapter

1. Morphism is basically a path that is not attached to any particular quadstore or a particular starting point in the graph. Morphisms are meant to be used as a query part that can be applied to other queries to follow a path specified in the Morphism.

   A good example will be a FollowRecursive function that will apply a single morphism multiple times to get to all nodes that can be traversed recursively. source

iterator Copy Chapter

1. So a graph query is roughly represented as a tree of iterators – things

   that implement graph.Iterator. An iterator is (loosely) a stand-in for a

   set of things that match a particular portion of the graph. source

subiterator Copy Chapter

1. So a graph query is roughly represented as a tree of iterators...Evaluation is merely calling Next() repeatedly on the iterator at the top of the tree. Subiterators, then, are the branches and leaves of the tree. source
2. Example of converting the Cayley-Gremlin-Go-API query g.V(“B”).In(“follows”).All() into an iterator tree:
   • HasA (subject) – gets the things in the subject field for:
     • And – the intersection of:
       • LinksTo (predicate) links that have the predicate of…:
         • Fixed iterator containing “follows” – … just the node “follows”.
       • LinksTo (object) links that have the object field of:
         • Fixed iterator containing “B” – … just the node “B”

LinkTo iterator Copy Chapter

1. A LinksTo takes a subiterator of nodes, and contains an iteration of links which "link to" those nodes in a given direction. ... Can be seen as the dual of the HasA iterator. source
   • Next()ing a LinksTo is straightforward -- iterate through all links to things in the subiterator, and then advance the subiterator, and do it again.

     • To restate in pseudo-code; results is what would be returned in successive Next() calls:

       go Copy

       var results []quad.Quad
       for _, node := range linkTo.subIterator {
         for _, quad := range allQuads {
             if quad.Get(linkTo.direction) == node {
                 results = append(results, quad)
             }
         }
       }

   • Contains()ing a LinksTo means, given a link, take the direction we care about and check if it's in our subiterator.

     • To restate in pseudo-code:

       go Copy

       for _, node := range linkTo.subIterator {
         if theLink.Get(linkTo.direction) == node {
             return true
         }
       }
       return false

HasA iterator Copy Chapter

1. The HasA takes a subiterator of links, and acts as an iterator of nodes in the given direction. The name comes from the idea that a "link HasA subject" or a "link HasA predicate". source

   • Next(), We have a subiterator we can get a value from, and we can take that resultant quad, pull our direction out of it, and return that.

     go Copy

     var results []quad.Value
     for _, quad := range hasA.subIterator {
       results = append(results, quad.Get(hasA.direction))
     }

   • Contains()

     go Copy

     for _, quad := range hasA.subIterator {
       if quad.Get(hasA.direction) == theNode {
           return true
       }
     }
     return false

shape Copy Chapter

1. Shape represent a query tree shape. source

   go Copy

   type Shape interface {
        BuildIterator(qs graph.QuadStore) graph.Iterator
        Optimize(r Optimizer) (Shape, bool)
    }

2. This is the most interesting part of the query system - it describes how exactly the query looks like. ... This package also describes different query optimizations that are not specific to a backend. ... You can write a query using either Paths, Shapes or raw Iterators... source

3. A Shape seems to be an abstract representation of a query, a level above Iterators and a level below Paths. You can perform various operations on it (traverse inbound/outbound predicates, find unions and intersections, etc.) and most importantly build a tree of Iterators from it, which will do the mechanical act of processing quads to find results.

token Copy Chapter

1. In the context of a quad store, a graph.Value. However the backend wishes to implement it, a Value is merely a token to a quad or a node that the backing store itself understands, and the base iterators pass around.

   For example, in a very traditional, graphd-style graph, these are int64s (guids of the primitives). In a very direct sort of graph, these could be pointers to structs, or merely quads, or whatever works best for the backing store.

reification Copy Chapter

1. “With reification, we create a metagraph on top of our graph that represents the statement that we have here. We create a new node that represents a statement and points at the subject...” source

2. Reifying a relationship means viewing it as an entity. The purpose of reifying a relationship is to make it explicit, when additional information needs to be added to it.

   Viewing a relationship as an entity, one can say that the entity reifies the relationship. This is called reification of a relationship. Like any other entity, it must be an instance of an entity type. source

File: docs/getting-involved/locations.md Copy Chapter

Locations of parts of Cayley

Community Copy Chapter

• Where is the beating heart of this community? https://discourse.cayley.io/
• Where is the mailing list? https://discourse.cayley.io/ (enable mailing list mode under options)
• Where is the chat room? cayleygraph.slack.com -- Invite here
• Where should I start contributing? Contributing.md (Also: How to get involved!)
• Where is the site-content? https://github.com/cayleygraph/site-content
• Where are issues? https://github.com/cayleygraph/cayley/issues
• Where is the graph source code? https://github.com/cayleygraph/cayley
• Where are pull requests? https://github.com/cayleygraph/cayley/pulls
• Where is the wiki? It was at https://github.com/cayleygraph/cayley/wiki, but we are deprecating it.
• Where is the (old) mailing list? It was at https://groups.google.com/forum/#!forum/cayley-users, but we are deprecating it for this forum (and its mailing list mode).

File: docs/getting-involved/todo.md Copy Chapter

TODOs

The main source of our TODO list is our Github Issues, so we are going to try to avoid duplicating them here.

Anything marked "TODO" in the code. Copy Chapter

Usually something that should be taken care of.

File: docs/k8s/k8s.md Copy Chapter

Running in Kubernetes

Most examples requires Kubernetes 1.5+ and PersistentVolumes are configured.

After running scripts namespace cayley will be created and service with the same name will be available in cluster. Service is of type ClusterIP by default. If you want to expose it, consider changing type to LoadBalancer.

Single instance \(Bolt\) Copy Chapter

This is a simplest possible configuration: single Cayley instance with persistent storage, using Bolt as a backend.

kubectl create -f ./docs/k8s/cayley-single.yml

Distributed \(MongoDB cluster\) Copy Chapter

This example is based on thesandlord/mongo-k8s-sidecar and runs Cayley on top of 3-node Mongo cluster.

kubectl create -f ./docs/k8s/cayley-mongo.yml

Distributed Copy Chapter

TODO: PostgreSQL, CockroachDB