## 1. Project Overview & Quickstart (schollz/Dockerfile)

# Dockerfile

Open-source repository schollz/Dockerfile

### Repository Details
- **Repository:** [schollz/Dockerfile](https://github.com/schollz/Dockerfile)
- **Primary Language:** Code

*Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.*

## 2. Official Technical Reference & Guides (schollz/docs)

## File: README.md

## Dat Project Documentation

Repository for the documentation of the Dat Project ecosystem. View the docs at [docs.datproject.org](https://docs.datproject.org/).

[](http://webchat.freenode.net/?channels=dat)
[](https://gitter.im/datproject/discussions?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[](http://docs.dat-data.com)

## Writing & Editing Docs

[See docs folder](docs/readme.md) for information on editing and adding docs. Once you finish editing the docs, send a PR to the `master` branch to get the edits automatically deployed.

### Creating + Generating Paper from Markdown

[See this gist](https://gist.github.com/maxogden/97190db73ac19fc6c1d9beee1a6e4fc8) for more information on how the paper is created with a basic example.

To generate the paper again, make sure you have `pandoc` and `pandoc-citeproc`:

```
brew install pandoc pandoc-citeproc
```

Then run the build script in `papers`

## Development

This documentation uses [minidocs](https://github.com/freeman-lab/minidocs) for layout, [ecosystem-docs](https://github.com/hughsk/ecosystem-docs) to get documentation from other modules, and ideas from [pull-stream-docs](https://github.com/pull-stream/pull-stream-docs) for auto travis deployment.

### Deployment

This repository uses [netlify](https://www.netlify.com/) for deployment. Deployment will happen automatically.

It works this way:

* Git webhook tells netlify there is new content
* netlify pulls latest repo
* netlify automatically runs `npm install`
* netlify runs the build script `sh scripts/netlify.sh`, which:
  * sets git config so we can use `ecosystem-docs` and pull latests readme files
  * runs `npm run netlify` which updates remote repos and builds.
* deploys `/dist` to the web

### Viewing Docs Locally

1. Clone Repository
2. `npm install`
3. `npm run update` to update external modules
4. `npm run build:local` to build the docs for local viewing
5. `npm run start` to view the docs in browser

### Ecosystem-docs Modules

Ecosystem-docs downloads readme files from the following modules to `docs/modules/`. Add another module to the documentation by adding the module below and updating `contents.json`: `"dat-node": "modules/dat-node.md"`. The 📔 makes it easier to parse (see `package.json`).

* 📔 [dat](https://github.com/datproject/dat)
* 📔 [hyperdrive](https://github.com/mafintosh/hyperdrive)
* 📔 [hypercore](https://github.com/mafintosh/hypercore)
* 📔 [dat-node](https://github.com/datproject/dat-node)
* 📔 [hyperdiscovery](https://github.com/karissa/hyperdiscovery)

---

## File: docs/cookbook/browser.md

# Browser Dat

Dat is written in JavaScript, so naturally, it can work entirely in the browser! The great part about this is that as more peers connect to each other in their client, the site assets will be shared between users rather hitting any server.

This approach is similar to that used in Feross' [Web Torrent](http://webtorrent.io). The difference is that Dat links can be rendered live and read dynamically, whereas BitTorrent links are static. The original owner can update the files in the directory and all peers will receive the updates automatically.

## WebRTC Usage Notes

**Important**: dat-js uses WebRTC, so it can only connect to other WebRTC clients. It is not possible for the dat-js library to connect directly clients using other protocols. All other Dat applications use non-WebRTC protocols ([see this FAQ for more info](https://docs.datproject.org/faq#does-dat-use-webrtc)). Non-browser clients can connect dats peer-to-peer via webrtc modules, such as [electron-webrtc](https://github.com/mappum/electron-webrtc), or use proxies via websockets, http, or other client-server protocols.

Due to WebRTC's less than stellar performance - Dat has focused on creating solid networking using other protocols. We may integrate WebRTC if performance improves and it becomes easier to run in non-browser interfaces (though we'd prefer using [more performant options](https://github.com/noffle/web-udp) in the browser, if they develop).

OK, now for the goods.

## Install

Embed the following script [dat.min.js](https://cdn.jsdelivr.net/dat/6.2.0/dat.min.js) on the page:
```
<script type="text/javascript" src="dat.min.js"></script>
```

You can also use the jsdelivr CDN for faster load speeds:

```
<script type="text/javascript" src="https://cdn.jsdelivr.net/dat/6.2.0/dat.min.js"></script>
```

This provides a `Dat` prototype on the `window` object.

### Browserify

Or use Node.js in the browser with [browserify](http://github.com/substack/node-browserify), which lets you use node.js style require() to organize your browser code using npm.

```
npm install dat-js
```

Then use `dat-js` like this:

```
var Dat = require('dat-js')
```

## Quick Examples

### Sharing data

```js
var dat = Dat()
dat.add(function (repo) {
  var writer = repo.archive.createFileWriteStream('hello.txt')
  writer.write('world')
  writer.end(function () { replicate(repo.key) })
})
```

### Downloading data

```js
var Dat = require('dat-js')

var clone = Dat()
clone.add(key, function (repo) {
  repo.archive.readFile('hello.txt', function (err, data) {
    console.log(data.toString()) // prints 'world'
  })
})
```

The `repo.archive` is a [hyperdrive](http://github.com/mafintosh/hyperdrive) instance, which manages all of the files. A hyperdrive archive has a bunch of simple methods including only getting the files and byte ranges you want from a particular repository.

For the full hyperdrive API and more examples, see the full [hyperdrive documentation](/hyperdrive).

## Downloading only what you need

You might be asking 'Is it possible to index into a subset of a dat dataset?' Most datasets are too large for browsers, and we probably only want a subset of them.

You can do this by using `sparse` mode, which make it only download content that the peer asks for. To do this, simply pass `{sparse: true}` when you create the dat:

```js
var Dat = require('dat-js')

var dat= Dat()
dat.add(key, {sparse: true}, function (repo) {
  // etc..
})
```

## Under the hood

Let's look under the hood of `dat-js` to see how a simple lower-level implementation can be built to create a browser-based dat.

Here's the most simple example using the underlying modules directly:

```js
var webrtc = require('webrtc-swarm')
var signalhub = require('signalhub')
var hyperdrive = require('hyperdrive')
var memdb = require('memdb')
var pump = require('pump')

var DEFAULT_SIGNALHUBS = 'https://signalhub.mafintosh.com'

var archive = hyperdrive()
var link = archive.discoveryKey.toString('hex')

var swarm = webrtc(signalhub(link, DEFAULT_SIGNALHUBS))
swarm.on('peer', function (conn) {
  var peer = archive.replicate({
    upload: true,
    download: true
  })
  pump(conn, peer, conn)
})
```

That's it. Now you are serving a dat-compatible hyperdrive from the browser. In another browser tab, you can connect to the swarm and download the data by using the same code as above. Just make sure to reference the archive you created before by using `archive.key` as the first argument:

## Storage API for metadata and content

Hyperdrive is the underlying database that runs dat.

Hyperdrive will save the metadata (small) and the content (potentially large) separately. You can control where both of these are saved and how they are retrieved. These tweaks have huge impact on performance, stability, and user experience, so it's important to understand the tradeoffs.

There are a million different ways to store and retrieve data in the browser, and all have their pros and cons depending on the use case. We've compiled a variety of examples here to try to make it as clear as possible.

The first argument to `hyperdrive` will be the main database for all metadata and content. The `file` option can be supplied to specify how to read and write content data. If a `file` option is not supplied, the content will also be stored in the main database.

There are many different ways to piece modules together to create the storage infrastructure for a hyperdrive -- here are some tested examples:

### Writing large files from the filesystem to the browser

File writes are limited to the available memory on the machine. Files are buffered (read: copied) *into memory* while being written to the hyperdrive instance. This isn't ideal, but works as long as file sizes stay below system RAM limits.

To fix this problem, you can use [random-access-file-reader](https://github.com/mafintosh/random-access-file-reader) to read the files directly from the filesystem instead of buffering them into memory.

Come over to our community channels and ask a question. It's probably a good one and we should cover it in the documentation. Thanks for trying it out, and PRs always welcome!

[](http://webchat.freenode.net/?channels=dat)
[](https://gitter.im/datproject/discussions?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

---

## File: docs/cookbook/diy-dat.md

# Build with Dat

In this guide, we will show how to develop applications with the Dat ecosystem. The Dat ecosystem is very modular making it easy to develop custom applications using Dat.

Dat comes with a built in javascript API we use in Dat Desktop and dat command line. For custom applications, or more control, you can also use the core Dat modules separately.

Use Dat in your JS Application:

1. `require('dat')`: use the [high-level Dat JS API](https://github.com/datproject/dat-node).
2. Build your own!

This tutorial will cover the second option and get you familiar with the core Dat modules.

### The Dat Core Modules

This tutorial will follow the steps for sharing and downloading files using Dat. In practice, we implement these in [dat-node](https://github.com/datproject/dat-node), a high-level module for using Dat that provides easy access to the core Dat modules.

For any Dat application, there are two essential modules you will start with:

1. [hyperdrive](https://npmjs.org/hyperdrive) for file synchronization and versioning
2. [hyperdiscovery](https://npmjs.org/hyperdiscovery) helps discover and connect to peers over local networks and the internet

The [Dat CLI](https://npmjs.org/dat) module itself combines these modules and wraps them in a command-line API. We also use the [dat-storage](https://github.com/datproject/dat-storage) module to handle file and key storage. These modules can be swapped out for a similarly compatible module, such as switching storage for [random-access-memory](https://github.com/mafintosh/random-access-memory).

## Getting Started

You will need node and npm installed to build with Dat. [Read more](https://github.com/datproject/dat/blob/master/CONTRIBUTING.md#development-workflow) about our development work flow to learn how we manage our module dependencies during development.

## Download a File

Our first module will download files from a Dat link entered by the user.

```bash
mkdir module-1 && cd module-1
npm init
npm install --save hyperdrive random-access-memory hyperdiscovery
touch index.js
```

For this example, we will use random-access-memory for our database (keeping the metadata in memory rather than on the file system). In your `index.js` file, require the main modules and set them up:

```js
var ram = require('random-access-memory')
var hyperdrive = require('hyperdrive')
var discovery = require('hyperdiscovery')

var link = process.argv[2] // user inputs the dat link
var key =  link.replace('dat://', '') // extract the key

var archive = hyperdrive(ram, key)
archive.ready(function () {
  discovery(archive)
})
```

Notice, the user will input the link for the second argument The easiest way to get a file from a hyperdrive archive is to make a read stream. `archive.readFile` accepts the index number of filename for the first argument. To display the file, we can create a file stream and pipe it to `process.stdout`.

```js
// Make sure your archive has a dat.json file!
var stream = archive.readFile('dat.json', 'utf-8', function (err, data) {
  if (err) throw err
  console.log(data)
})
```

Now, you can run the module! To download the `dat.json` file from an archive:

```
node index.js dat://<link>
```

You should see the `dat.json` file.

#### Bonus: Display any file in the Dat

With a few more lines of code, the user can enter a file to display from the Dat link.

Challenge: create a module that will allow the user to input a Dat link and a filename: `node bonus.js <dat-link> <filename>`. The module will print out that file from the link, as we did above:

```js
var stream = archive.readFile(fileName)
```

Once you are finished, see if you can view this file by running:

```bash
node bonus.js 395e3467bb5b2fa083ee8a4a17a706c5574b740b5e1be6efd65754d4ab7328c2 readme.md
```

## Download all files to computer

This module will build on the last module. Instead of displaying a single file, we will download all of the files from a Dat into a local directory.

To download the files to the file system, we are going to use [mirror-folder](https://github.com/mafintosh/mirror-folder). [Read more](/using-fs) about how mirror-folder works with hyperdrive.

In practice, you should use [dat-storage](https://github.com/datproject/dat-storage) to do this as it'll be more efficient and keep the metadata on disk.

Setup will be the same as before (make sure you install `mirror-folder`). The first part of the module will look the same.

```js
var path = require('path')
var ram = require('random-access-memory')
var hyperdrive = require('hyperdrive')
var discovery = require('hyperdiscovery')
var mirror = require('mirror-folder')
var mkdirp = require('mkdirp')

var link = process.argv[2] // user inputs the dat link
var key = link.replace('dat://', '') // extract the key
var dir = path.join(process.cwd(), 'dat-download') // make a download folder
mkdirp.sync(dir)

var archive = hyperdrive(ram, key)
archive.ready(function () {
  discovery(archive)

  var progress = mirror({name: '/', fs: archive}, dir, function (err) {
    console.log('done downloading!')
  })
  progress.on('put', function (src) {
    console.log(src.name, 'downloaded')
  })
})
```

You should be able to run the module and see all our docs files in the `download` folder:

```bash
node index.js dat://<link>
```

---

## File: docs/cookbook/http.md

# Sharing files over HTTP

The Dat command line comes with a built in HTTP server. This is a cool demo because we can also see how version history works! The `--http` option works for files you are sharing *or* downloading.

(If you don't have dat command line installed, run `npm install dat`, or [see more info](intro#installation).)

## Serve over HTTP

Serve dat files on http by adding the `--http` option. For example, you can sync an existing dat:

```
❯ dat sync --http
dat://778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639
Sharing dat: 2 files (1.4 MB)
Serving files over http at http://localhost:8080

2 connections | Download 0 B/s Upload 0 B/s
```

Now visit [http://localhost:8080]() to see the files in your browser! The default http port is 8080. You should see a directory listing:



If your dat has an `index.html` page, that will be shown instead.

You can combine Dat's http support with our server tools to create a live updating website or place to share files publicly.

## Built-in Versioning

As you may know, Dat automatically versions all files. The HTTP display is an easy way to view version history:

**Use [localhost:8080/?version=2]() to view a specific version.**

## Live reloading

The Dat http viewer also comes with live reloading. If it detects a new version it will automatically reload with the new directory listing or page (as long as you aren't viewing a specific version in the url).

## Sparse Downloading

Dat supports *sparse*, or partial downloads, of datasets. This is really useful if you only want a single file from a large dat. Unfortunately, we haven't quite built a user interface for this into our applications. So you can hack around it!

This will allow you to download a single file from a larger dat, without downloading metadata or any other files.

First, start downloading our demo dat, make sure you include both the flags (`--http`, `--sparse`).

```
❯ dat dat://778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639 ./demo --http --sparse
Cloning: 2 files (1.4 MB)
Serving files over http at http://localhost:8080

3 connections | Download 0 B/s Upload 0 B/s
```

The `--sparse` option tells Dat to only download files you specifically request. See how it works:

1. Check out your `./demo` folder, it should be empty.
2. [Open the Dat](http://localhost:8080) in your browser.
3. Click on a file to download.
4. It should be in your folder now!

Pretty cool! You can use this hack to download only specific files or even older versions of files (if they've been saved somewhere).

---

## File: docs/cookbook/publish.md

# Publishing a Dat

Dat has privacy by default, but what if you want to share your dat with the world publicly, or within your organization? There are a variety of ways to do this, which we describe here.

## Create dat.json file
When referencing your dat, you want to be able to have some way to specify what the dat is -- like a title, description, and url. To do this, we recommend creating a `dat.json` file as well as a README in your dat archive. The `dat create` command creates a `dat.json` file for you by default:

```
$ dat create
Welcome to dat program!
You can turn any folder on your computer into a Dat.
A Dat is a folder with some magic.

Your dat is ready!
We will walk you creating a 'dat.json' file.
(You can skip dat.json and get started now.)

Learn more about dat.json: https://github.com/datprotocol/dat.json

Ctrl+C to exit at any time
Title: More Tweets, More Votes
```

This will generate a `dat.json` file which you can use to reference your dat in the future. For example:

`dat.json`

```json
{
  "url": "dat://96cf4957539aff4fc856fa0804e613181064ed193e5f7882c9623ec7bed38deb",
  "title": "More Tweets More Votes",
  "description": "An analysis of tweets from the peroid before 2012 U.S. House election combined with census data."
}
```

Dat can read these files to know how to download and catalogue your dat. This is the most simple version, but they can get more complicated if you want. You can use this file in other applications such as a registry, data portal, or git repository to reference your dat.

## Dat Registry

We have built a web server that allows you to create a user account and build your list of datasets, using the dat.json file to give them names, titles, and descriptions that make it easy to share your data with the general public. The registry does not take care of long-term hosting of your data -- the registry only acts as a way to easily surface available dats in a searchable list. A dat registry is similar to a torrent tracker, if you're familiar with those.

We have hosted a global public registry at [https://datproject.org](https://datproject.org). First, you must create an account:

```bash
$ dat register
Welcome to dat program!
Create a new account with a Dat registry.

Dat registry:  (datproject.org)
```

By default, `dat` will use `datproject.org` as the registry. If you'd like to host your own registry, you can type in your own server name. Once you have an account, you can use `dat publish` to publish your dat to the registry.

First, `dat publish` will ask you for a name for your dat. This is the short name that will be part of the URL, for example, more-tweets-more-votes would be `myusername/more-tweets-more-votes`:

```bash
cd /path/to/my/dat
$ dat publish
Name: more-tweets-more-votes
Published successfully!
```

Once the dat is published, you can visit it. For this example: [https://datproject.org/karissa/more-tweets-more-votes](https://datproject.org/karissa/more-tweets-more-votes).

In order for anyone to see the dat, you'll need to have `dat share` running on your current dat. If you want to turn off your desktop, we recommend setting up a server to host your dats.

[Learn how to set up a long-term server for hosting dats](/server)

## Using Git with `dat.json`

You might want to be able to get the collaborative benefits of git along with the benefits of fast data download, versioning, p2p, and deduplication with dat. All you have to do is include the `dat.json` file in your git repository, and tell people to type `dat clone .` for them to get the latest version of your data. Now, when someone clones your git repository, they can download your data by simply typing `dat clone .` or `dat clone dat.json`, and the url in the dat.json will be used to download the dat.

To save space in your git repo, you probably want to add `.dat` to your `.gitignore` file in your git repository to make sure none of dat's metadata is included. Dat by default ignores your `.git` folder, so you don't have to worry about that.

---

## File: docs/cookbook/server.md

# Dats on a Server

Since Dat is a distributed (peer-to-peer) data sharing tool, a computer must be actively sharing a dat for it to be available. If you're sharing files over Dat, you might want to set up a dedicated server that re-hosts your dat. This means that it'll still be available even after you turn off your personal computer.

Running Dat on a server can also be used for live backups. As long as you are connected to your server and syncing changes, your server can backup all of your content history - allowing you to view old content later.

## Short Instructions

We have built a simple tool to host multiple dats using the commandline. The tool is called `hypercored`. Hypercored reads a file that contains all of the dats that you want to share.

Install it on your server, create a `feeds` file with dats separated by newlines, and run `hypercored`.

```
npm install hypercored
echo 'dat://64375abb733a62fa301b1f124427e825d292a6d3ba25a26c9d4303a7987bec65' >> feeds
echo 'dat://another-dat-link-here' >> feeds
hypercored
```

That's it. Now it will download and host the data for the each dat in the `feeds` file. Hypercored uses [hypercore-archiver](https://github.com/mafintosh/hypercore-archiver) for efficient sharing of many dats and full content history backup.

See below for more detailed instructions.

## Detailed Instructions

### Node Version

Check your node version, you should have version 4.0 or higher, but 6.10.3 or higher is preferred.

```
$ node -v
```

Then, go to your server (using `ssh username@hostname.com`) and install `hypercored`:

```
npm install hypercored
```

If you have installation trouble due to a permissions error, please see [this tutorial for fixing permissions in node.js](https://docs.npmjs.com/getting-started/fixing-npm-permissions).

Now, create a file called 'feeds' with the list of dats you want to share.

feeds
```
dat://one-hash
two-hash
website.com/three-hash
```

So then if you `ls` you should see `feeds` in the list of files.

```
$ ls
feeds
```

Now, to share these dats simply type `hypercored` in the same directory.

```
~/dat $ hypercored
Watching ~/dat/feeds for a list of active feeds
Archiver key is 42471e32d36be3cb617ec1df382372532aac1d1ce683982962fb3594c5f9532a
Swarm listening on port 58184
```

That's great! Now all of the dats in `feeds` will be downloaded and re-hosted. However, it's running in the foreground -- you probably want to use a process manager to run and watch the process so that it never goes down.

## Run it Forever

We recommend using `lil-pids` and `add-to-systemd` for long-term dat hosting from a linux server.

```
npm install add-to-systemd lil-pids
mkdir ~/dat
echo "hypercored --cwd ~/dat" > ~/dat/services
sudo add-to-systemd dat-lil-pids $(which lil-pids) ~/dat/services ~/dat/pids
```

Replace `~` with the path where you want to store your dats.

---

## File: docs/cookbook/using-fs.md

# Using the Hyperdrive FS

[Hyperdrive](https://github.com/mafintosh/hyperdrive), the core file system module in dat, exposes an API that mimics the Node fs API. This allows you to create modules that act on hyperdrive or a regular fs with the same API. We have several modules that we make use of this custom fs, such as [mirror-folder](https://github.com/mafintosh/mirror-folder). 

Mirror folder can copy a regular directory to another regular directory:

```js
var mirror = require('mirror-folder')

mirror('/source', '/dest', function (err) {
  console.log('mirror complete')
})
```

You can also copy a folder to an archive (this is how we do importing in Dat):

```js
var archive = hyperdrive('/dir')
mirror('/source', {name: '/', fs: archive}, function (err) {
  console.log('mirror complete')
})
```

### Creating Custom FS Modules

To create a module that uses a custom fs, you can default to the regular `fs` but also accept `fs` as an argument. For example, to print a file you could write this function:

```js
function printFile(file, fs) {
  if (!fs) fs = require('fs')

  fs.readFile(file, 'utf-8', function (err, data) {
    console.log(data)
  }) 
}
```

Then you could use this to print a file from a regular fs:

```js
printFile('/data/hello-world.txt')
```

Or from a hyperdrive archive:

```js
var archive = hyperdrive('/data')
printFile('/hello-world.txt', archive) // pass archive as the fs!
```

## Modules!

See more examples of custom-fs modules:

* [mirror-folder](https://github.com/mafintosh/mirror-folder) - copy files from one fs to another fs (regular or custom)
* [count-files](https://github.com/joehand/count-files) - count files in regular or custom fs.
* [ftpfs](https://github.com/maxogden/ftpfs) - custom fs for FTPs
* [bagit-fs](https://github.com/joehand/bagit-fs) - custom fs module for the BagIt spec.

---

## File: docs/concepts.md

# Key Concepts

## In Place Archiving

You can turn any folder on your computer into *a dat*. We call this *in place archiving*. A dat is a regular folder with some magic attached. The magic is a set of metadata files, in a `.dat` folder. Dat uses the metadata to track file history and securely share your files. Your files and the `.dat` folder can be instantly synced to anywhere.



Once you have installed Dat, you can use a single command to live sync your files to friends, backup to an external drive, and publish to a website (so people can download over http too!). The cool part is this all happens at the same time. If you go offline for a bit, no worries. Dat shares the latest files and any saved history once you are back online. These data transfers happen between the computers, forgoing any centralized source.

In place archiving in Dat really means **any place**. Dat seamlessly syncs your files where you want and when you want. Dat's decentralized technology and automatic versioning will improve data availability and data quality without sacrificing ease of use.

## Distributed Network

Dat goes beyond regular archiving through its *distributed network*. When you share data, Dat sends data to many download locations at once, and they can sync the same data with each other! By connecting users directly Dat transfers files faster, especially sharing on a local network. Distributed syncing allows robust global archiving for public data.



To maintain privacy, the dat link controls access to your data. Any data shared in the network is encrypted using your link as the password. Learn more about Dat's securtiy and privacy below or in [the faqs](faq#security-and-privacy). We are also investigating ways to improve [reader privacy](https://blog.datproject.org/2016/12/12/reader-privacy-on-the-p2p-web/) for public data.

## Version History

Dat automatically maintains a built in version history whenever files are added. Dat uses this history to allow partial downloads of files, for example only getting the latest files. There are two types of versioning performed automatically by Dat. Metadata is stored in a folder called `.dat` in the main folder of a repository, and data is stored as normal files in the main folder.

Dat uses append-only registers to store version history. This means all changes are written to the end of the file, growing over time.

### Metadata Versioning

Dat acts as a one-to-one mirror of the state of a folder and all it's contents. When importing files, Dat grabs the filesystem metadata for each file and checks if there is already an entry for this filename. If the file with this metadata matches exactly the newest version of the file metadata stored in Dat, then this file will be skipped (no change).

If the metadata differs or does not exist, then this new metadata entry will be appended as the new 'latest' version for this file in the append-only SLEEP metadata content register.

### Content Versioning

The metadata only tells you if or when a file is changed, not how it changed. In addition to the metadata, Dat tracks changes in the content in a similar manner.

The default storage system used in Dat stores the files as files. This has the advantage of being very straightforward for users to understand, but the downside of not storing old versions of content by default.

In contrast to other version control systems, like Git, Dat only stores the current set of files, not older versions. Git, for example, stores all previous content versions and all previous metadata versions in the `.git` folder. But Dat is designed for larger datasets.

Storing all history on content could easily fill up the users hard drive. Dat has multiple storage modes based on usage. With Dat's dynamic storage, you can store the content history on a local external hard drive or on a remote server (or both!).

## Dat Privacy

Files shared with Dat are encrypted (using the link) so *only* users with your unique link can access your files. The link acts as a kind of password meaning, generally, you should assume *anyone* with the link will have access to your files.

The link allows users to download, and re-share, your files, whether you intended them to have the link or not (with some hand waiving assumptions about them being able to connect to you, which can be limited, see more in [security & privacy faq](faq#security-and-privacy)).

Make sure you are thoughtful about who you share links with and how. Dat ensures links cannot be intercepted through the Dat network. If you share your links over other channels, ensure the privacy & security matches or exceeds your data security needs. We try to limit times when Dat displays full links to avoid accidental sharing.

## dat:// links

Dat links have some special properties that are helpful to understand.

Traditionally, http links point to a specific server, e.g. datproject.org's server, and/or a specific resource on that server. Unfortunately, links often break or the content changes without notification (this makes it impossible to cite `nytimes.com`, for example, because the link is meaningless without a reference to what content was there at citation time). Dat links, on the other hand, never change. You can update data in a dat and use the same link to download the changes.

Here is an example dat link:

```
dat://ff34725120b2f3c5bd5028e4f61d14a45a22af48a7b12126d5d588becde88a93
```

What is with the weird long string of characters? Let's break it down!

**`dat://` - the protocol**

The first part of the link is the link protocol, Dat (read about the Dat protocol at [datprotocol.com](http://www.datprotocol.com)). The protocol describes what "language" the link is in and what type of applications can open it. You do not always need this part with Dat but it is helpful context.

**`ff34725120b2f3c5bd5028e4f61d14a45a22af48a7b12126d5d588becde88a93` - the unique identifier**

The second part of the link is a 64-character hex strings ([ed25519 public-keys](https://ed25519.cr.yp.to/) to be precise). Each Dat archive gets a public key link to identify it. With the hex string as a link we can a few things:

1. Encrypt the data transfer
2. Create a persistent identifier, an ID that never changes, even as file are updated (as opposed to a checksum which is based on the file contents).

**`dat://ff34725120b2f3c5bd5028e4f61d14a45a22af48a7b12126d5d588becde88a93`**

All together, the links can be thought of similarly to a web URL, as a place to get content, but with some extra special properties. When you download a dat link:

1. You do not have to worry about where the files are stored.
2. You can always get the latest files available.
3. You can view the version history or add version numbers to links to get an permanent link to a specific version.

[If you'd like you read more about how dat works, see our whitepaper.](https://github.com/datproject/docs/blob/master/papers/dat-paper.pdf)

---

## File: docs/dat-powered.md

# Dat Powered Apps

## Beaker Browser

## Science Fair

## Data Archiving

* Project Svalbard

## Distributed Tools

* Hyperpipe

---

## File: docs/ecosystem.md

# Open Source Tools

We have built and contributed to a variety of modules that support our work on Dat as well as the larger data and code ecosystem. Feel free to go deeper and see the implementations we are using in the [Dat command-line tool](https://github.com/datproject/dat) and the [Dat Desktop](https://github.com/datproject/dat-desktop).

Dat embraces the Unix philosophy: a modular design with composable parts. All of the pieces can be replaced with alternative implementations as long as they implement the abstract API. We believe this creates better end-user software, but more importantly, will create more sustainable and impactful open source tools.

## User Software

* [dat](https://github.com/datproject/dat) - The command line interface for Dat.
* [Dat Desktop](https://github.com/datproject/dat-desktop) - Desktop application for Dat.
* [datproject.org](https://github.com/datproject/datproject.org/) - Repository for the Dat project website, a public data registry and sharing.

## Specifications

* [Dat Whitepaper](https://github.com/datproject/docs/tree/master/papers) - The Dat whitepaper
* [Dat Protocol](https://www.datprotocol.com/) - Site for the Dat protocol

## Core Modules

These modules form the backbone of Dat software:

* [hypercore](https://github.com/mafintosh/hypercore) - A secure, distributed append-only log.
* [hyperdrive](https://github.com/mafintosh/hyperdrive) - A secure, real time distributed file system (built on hypercore).
* [dat-node](https://github.com/datproject/dat-node) - High-level module for building Dat applications on the file system.
* [hyperdiscovery](https://github.com/karissa/hyperdiscovery) - Defaults for networking discovery and connection management.
* [dat-storage](https://github.com/datproject/dat-storage) - Default storage module for Dat.

View [more on Github](https://github.com/search?utf8=%E2%9C%93&q=topic%3Adat&type=Repositories).

## Modules We Like & Use

These modules we use throughout our applications:

* [Choo](https://github.com/yoshuawuyts/choo) - A 4kb framework for creating sturdy frontend applications.
* [neat-log](https://github.com/joehand/neat-log) - A neat cli logger for stateful command line applications Edit
Add topics
* [mirror-folder](https://github.com/mafintosh/mirror-folder) - Mirror a folder to another folder, supports hyperdrive and live file watching.
* [toiletdb](https://github.com/maxogden/toiletdb) - CRUD database using a JSON file for storage.