Go-SSB

WARNING: Project is still in alpha, backwards incompatible changes will be made.

A full-stack implementation of secure-scuttlebutt using the Go programming language.

If you encounter a bug, please refer to our public issue tracker.

See the FAQ for more. See this project for current focus.

Developing

Want to contribute patches to go-ssb? Read the developer documentation hosted at dev.scuttlebutt.nz. If you have a large change you want to make, reach out first and we'll together make sure that the resulting PR will be accepted 🖤

Server Features

• Follow-graph implementation (based on gonum) to authorize incoming connections
• Blobs store and replication
• Publishing new messages to the log
• Legacy feed replication
• Epidemic Broadcast Trees (EBT) feed replication in beta (use go-sbot -enable-ebt)
• Invite mechanics (peer-invites partially done, too. See Issue 45 for more.)

Observability

go-ssb has built-in support for OpenTelemetry tracing and Prometheus metrics.

Prometheus Metrics

Metrics are exposed at http://localhost:6078/metrics when running with -debugAddr localhost:6078 (or your chosen debug address). These include connection counts, bytes tx/rx, and repository stats.

OpenTelemetry Tracing

Tracing is available via the standard OTel SDK with an OTLP exporter. When tracing is not enabled, all calls are no-ops with zero overhead.

Enable tracing via any of:

# Flag
go-sbot -enable-otel

# Environment variable
SSB_OTEL_ENABLED=yes go-sbot

# Config file (~/.ssb-go/config.toml)
# [go-sbot]
# enable-otel = true

The OTLP exporter endpoint is controlled by the standard OTEL_EXPORTER_OTLP_ENDPOINT variable (defaults to http://localhost:4318).

Quick start with Jaeger:

# 1. Run Jaeger (all-in-one, with OTLP receiver)
docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

# 2. Start go-sbot with tracing enabled
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 go-sbot -enable-otel

# 3. Open Jaeger UI
open http://localhost:16686
# Search for service "go-sbot"

What you'll see in traces:

Span	Shows you
ssb.query.subset	Query execution: which bitmap operations ran, result cardinality, time per operation
ssb.ebt.session	EBT sync sessions: messages received per peer, frontier exchanges, verification time
ssb.gossip.fetch	Legacy gossip: per-feed fetch with batch verification and storage timing
ssb.index.batch	Index processing: batch sizes, which index is the bottleneck

All SSB-specific attributes use the ssb. prefix (e.g. ssb.peer.id, ssb.feed, ssb.query.op, ssb.frontier.size). See internal/tracing/tracing.go for the full list.

Frontiers (Vector Clocks)

go-ssb tracks frontiers — vector clocks over SSB feeds that represent "what data has this node seen?" Frontiers enable:

• Cache validity: "Has my timeline changed since last render?" → compare frontiers with HappenedBefore()
• Incremental updates: Diff() tells you exactly which feeds advanced and by how many messages
• Fork detection: A ForkProof pairs a frontier with two conflicting messages from the same feed

Frontiers are scoped to a concern (your follows, a thread's participants, a git-ssb repo's contributors) — typically under 100 feeds, small enough to embed in an SSB message.

// Check if a cached result is still valid
if cachedFrontier.HappenedBefore(currentFrontier) {
    // result is still good — no new data in the feeds that matter
} else {
    // something changed — Diff() tells you what
    changes := cachedFrontier.Diff(currentFrontier)
    for _, adv := range changes {
        fmt.Printf("feed %s advanced from %d to %d\n",
            adv.Feed.ShortSigil(), adv.OldSeq, adv.NewSeq)
    }
}

See docs/design-frontier-vector-clock.md for the full design and docs/design-otel-tracing.md for how frontiers integrate with tracing.

Installation

You can install the project using Golang's install command which will place the commands into the directory pointed to by the GOBIN environment variable.

git clone https://github.com/ssbc/go-ssb
cd go-ssb
go install ./cmd/go-sbot
go install ./cmd/sbotcli

Requirements:

• Golang version 1.17 or higher

Running go-sbot

The tool in cmd/go-sbot is similar to ssb-server (previously called scuttlebot or sbot for short).

See the quick start document for a walkthrough and getting started tour. You may also be interested in the ways you can configure a running go-sbot, see the configuration guide.

Bootstrapping from an existing key-pair

If you have an existing feed with published contact messages, you can just resync it from another go or js server. To get this going you copy the key-pair ($HOME/.ssb/secret by default) to $HOME/.ssb-go/secret, start the program and connect to the server (using the multiserver address format).

mkdir $HOME/.ssb-go
cp $HOME/.ssb/secret $HOME/.ssb-go
go-sbot &
sbotcli connect "net:some.ho.st:8008~shs:SomeActuallyValidPubKey="

Publishing

This currently constructs legacy SSB messages, that still have the signature inside the signed value:

{
  "key": "%EMr6LTquV6Y8qkSaQ96ncL6oymbx4IddLdQKVGqYgGI=.sha256",
  "value": {
    "previous": "%rkJMoEspdU75c1RpGbwjEH7eZxM/PJPFubpZTtynhsg=.sha256",
    "author": "@iL6NzQoOLFP18pCpprkbY80DMtiG4JFFtVSVUaoGsOQ=.ed25519",
    "sequence": 793,
    "timestamp": 1457694632215,
    "hash": "sha256",
    "content": {
      "type": "post",
      "text": "@dust \n> this feels like the cultural opposite of self-dogfooding, and naturally, leaves a bad taste in my mouth \n \n\"This\" meaning this thread? Or something in particular in this thread? And if this thread or something in it, how so? I don't want to leave a bad taste in your mouth.",
      "root": "%I3yWHMF2kqC7fLZrC8FB+Kuu/6MQZIKzJGIjR3fVv9g=.sha256",
      "branch": "%cNJgO+1R4ci/jgTup4LLACoaKZRtYtsO7BzRCDJh6Gg=.sha256",
      "mentions": [
        {
          "link": "@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519",
          "name": "dust"
        }
      ],
      "channel": "patchwork-dev"
    },
    "signature": "bbjj+zyNubLNEV+hhUf6Of4KYOlQBavQnvdW9rF2nKqTHQTBiFBnRehfveCft3OGSIIr4VgD4ePICCTlBuTdAg==.sig.ed25519"
  },
  "timestamp": 1550074432723.0059
}

The problem with this (for Go and others) is removing the signature field from value without changing any of the values or field ordering of the object, which is required to compute the exact same bytes that were used for creating the signature. Signing JSON was a bad idea. There is also other problems around this (like producing the same byte/string encoding for floats that v8 produces) and a new, canonical format is badly needed.

What you are free to input is the content object, the rest is filled in for you. The author is determined by the keypair used by go-sbot. Multiple identities are supported through the API.

Over muxrpc

go-sbot also exposes the same async publish method that ssb-server has. So you can also use it with ssb-client!

Through Go API

To do this programatically in go, you construct a margaret.Log using multilogs.OpenPublishLog (godoc) that publishes the content portion you Append() to it the feed of the keypair.

Example:

package main

import (
	"log"
	"fmt"

	"github.com/ssbc/go-ssb"
	"github.com/ssbc/go-ssb/multilogs"
	"github.com/ssbc/go-ssb/sbot"
)

func main() {
	sbot, err := sbot.New()
	check(err)

	publish, err := multilogs.OpenPublishLog(sbot.ReceiveLog, sbot.UserFeeds, *sbot.KeyPair)
	check(err)

	alice, err := refs.ParseFeedRef("@alicesKeyInActualBase64Bytes.ed25519")
	check(err)

	var someMsgs = []interface{}{
		map[string]interface{}{
			"type":  "about",
			"about": sbot.KeyPair.ID().Ref(),
			"name":  "my user",
		},
		map[string]interface{}{
			"type":      "contact",
			"contact":   alice.Ref(),
			"following": true,
		},
		map[string]interface{}{
			"type": "post",
			"text": `# hello world!`,
		},
		map[string]interface{}{
			"type":  "about",
			"about": alice.Ref(),
			"name":  "test alice",
		},
	}
	for i, msg := range someMsgs {
		newSeq, err := publish.Append(msg)
		check(fmt.Errorf("failed to publish test message %d: %w", i, err))
		log.Println("new message:", newSeq)
	}

	err = sbot.Close()
	check(err)
}

func check(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

sbotcli

Has some commands to publish frequently used messages like post, vote and contact:

sbotcli publish contact --following '@p13zSAiOpguI9nsawkGijsnMfWmFd5rlUNpzekEE+vI=.ed25519'
sbotcli publish contact --blocking '@p13zSAiOpguI9nsawkGijsnMfWmFd5rlUNpzekEE+vI=.ed25519'
sbotcli publish about --name "cryptix" '@p13zSAiOpguI9nsawkGijsnMfWmFd5rlUNpzekEE+vI=.ed25519'

They all support passing multiple --recps flags to publish private messages as well:

sbotcli publish post --recps "@key1" --recps "@key2" "what's up?"

For more dynamic use, you can also just pipe JSON into stdin:

cat some.json | sbotcli publish raw

Building

There are two binary executable in this project that are useful right now, both located in the cmd folder. go-sbot is the database server, handling incoming connections and supplying replication to other peers. sbotcli is a command line interface to query feeds and instruct actions like connect to X. This also works against the JS implementation.

If you just want to build the server and play without contributing to the code (and are using a recent go version > 1.17), you can do this:

# clone the repo
git clone https://github.com/ssbc/go-ssb
# go into the servers folder
cd go-ssb/cmd/go-sbot
# build the binary (also fetches pinned dependencies)
go build -v -i
# test the executable works by printing it's help listing
./go-sbot -h
# (optional) install it somwhere on your $PATH
sudo cp go-sbot /usr/local/bin

If you want to hack on the other dependencies of the stack, you can use the replace statement.

E.g. for hacking on a locally cloned copy of go-muxrpc, you'd do:

diff --git a/go.mod b/go.mod
index c4d475f..51c2757 100644
--- a/go.mod
+++ b/go.mod
@@ -6,6 +6,8 @@ module github.com/ssbc/go-ssb

+replace github.com/ssbc/go-muxrpc/v3 => ./../go-muxrpc
+

Which points the new build of go-sbot/sbotcli to use your local copy of go-muxrpc.

Testing

Once you have configured your environment set up to build the binaries, you can also run the tests. We have unit tests for most of the modules, most importantly message, blobstore and the replication plugins (gossip and blobs). There are also interoperability tests with the nodejs implementation (this requires recent versions of node and npm).

$ go test -v ./message
2019/01/08 12:21:55 loaded 236 messages from testdata.zip
=== RUN   TestPreserveOrder
--- PASS: TestPreserveOrder (0.00s)
=== RUN   TestComparePreserve
--- PASS: TestComparePreserve (0.02s)
=== RUN   TestExtractSignature
--- PASS: TestExtractSignature (0.00s)
=== RUN   TestStripSignature
--- PASS: TestStripSignature (0.00s)
=== RUN   TestUnicodeFind
--- PASS: TestUnicodeFind (0.00s)
=== RUN   TestInternalV8String
--- PASS: TestInternalV8String (0.00s)
=== RUN   TestSignatureVerify
--- PASS: TestSignatureVerify (0.06s)
=== RUN   TestVerify
--- PASS: TestVerify (0.06s)
=== RUN   TestVerifyBugs
--- PASS: TestVerifyBugs (0.00s)
PASS
ok  	github.com/ssbc/go-ssb/message	0.180s

If you encounter a feed that can't be validated with our code, there is a encode_test.js script to create the testdata.zip from a local sbot. Call it like this cd message && node encode_test.js @feedPubKey.ed25519 and re-run go test.

$ go test ./plugins/...
ok  	github.com/ssbc/go-ssb/plugins/blobs	0.021s
?   	github.com/ssbc/go-ssb/plugins/control	[no test files]
ok  	github.com/ssbc/go-ssb/plugins/gossip	0.667s
?   	github.com/ssbc/go-ssb/plugins/test	[no test files]
?   	github.com/ssbc/go-ssb/plugins/whoami	[no test files]

(Sometimes the gossip test blocks indefinitely. This is a bug in go-muxrpcs closing behavior. See the FAQ for more information.)

To run the interop tests you need to install the dependencies first and then run the tests. Diagnosing a failure might require adding the -v flag to get the stderr output from the nodejs process.

$ cd message/legacy && npm ci
$ go test -v
$ cd -
$ cd tests && npm ci
$ go test -v

CI

The tests run under this Github Actions configuration.

We cache both the global ~/.npm and **/node_modules to reduce build times. This seems like a reasonable approach because we're testing against a single Node.js version and a locked package set. Go dependencies are also cached.

Stack links

• secret-handshake key exchange using secretstream
• JS interoparability by using go-muxprc
• Embedded datastore, no external database required (margaret abstraction with BadgerDB backend, similar to flumedb)
• pull-stream-like abstraction (called luigi) to pipe between rpc and database.

Contact

• Post to the #go-ssb / #go-ssb-dev channels on ssb
• Raise an issue
• Or mention us individually on ssb
  • cryptix: @p13zSAiOpguI9nsawkGijsnMfWmFd5rlUNpzekEE+vI=.ed25519
  • keks: @YXkE3TikkY4GFMX3lzXUllRkNTbj5E+604AkaO1xbz8=.ed25519
  • decentral1se @i8OXtTYaK0PrF002pd4vpXmrlg98As7ZMaHGKoXixdM=.ed25519