Peergos release v0.1.3
We've been hard at work, so hard in fact that we're only now writing another blog post. We've sped everything up, enabled native builds, added garbage collection, added S3 storage support and fixed several bugs.
The most obvious change is that we've sped most things up by ~2.5X, notably uploads and downloads. Our download speeds are now comparable to Dropbox, despite the extra overhead of decryption!
2.5X faster downloads
The next big feature is transaction-aware garbage collection. This is critical to ensure our disks don't unnecessarily fill up.
If you change 1 byte of a file bigger than 5 MiB, then we reencrypt that 5 MiB section of file. The new cipher text for this section has nothing in common with the old cipher text (otherwise it wouldn't be very good encryption!!). This means that now 5 MiB of blocks are unused and wasting space. Eventually, with enough modifcations you'd fill up the disk with all this garbage.
Too much garbage
The operation that cleans up these unreferenced blocks of data is called garbage collection. The difficult part is making sure we don't garbage collect data that someone is in the middle of writing (before they commit it). At the moment, this is achieved with a process wide lock. Once IPFS implements native transactions we can remove this global lock and get much better throughput.
The next addition was statically compiled native binaries. Until now, to run Peergos yourself you had to independently install Java, which is ~100 MiB. This is an extra barrier that makes it harder to use, and which some people aren't willing to do. The native builds:
- are smaller in total size (32 MiB compared to a 13 MiB jar plus a 100 MiB JVM);
- use much less RAM than a full JVM;
- start much faster than a JVM (4 ms compared to 660 ms).
The last feature in particular enables us to have an interactive command line interface which doesn't require running a daemon. We'll be writing more about this soon! The only negative of native image builds is we don't yet have reproducible builds for it.
The final feature in this release is support for using an S3 compatible service for data storage instead of the local hard disk.
As well as all these new features we've fixed a host of bugs in both the server and the client. So head on over to our alpha and check it out!
Applications on Peergos, August 2019
They're here - the first apps for Peergos!
The first application we've made available is a read-only viewer for PDF files. This is a super-simple app which adds very useful functionality to Peergos. Now you can securely and privately read PDF files within Peergos.
For the PDF viewer, we simply imported pdf.js from Mozilla and implemented a one-way message from the external context which sends the content of the PDF to render. For the text editor, we used a great library called Codemirror. This app required a slightly more complicated protocol than the one-way message system of the PDF viewer. We just needed a message to send the modified content back to the external context, and the privileged code could then save the modified file contents.
Before we open up the sandbox to truly untrusted apps, there is one prerequisite. Currently, browsers don't put sandboxed iframes into a different operating system process, which is necessary to be fully protected from CPU vulnerabilities such as Spectre or Meltdown. Both Firefox and Chrome are in the process of moving iframes to separate processes and once that lands, we can enable 100% safe sandboxes for Peergos applications!
In the meantime we can use it to curate semi-trusted apps like the PDF viewer and text editor. We'd love to hear what you use the text editor for!
-The Peergos team
Fast seeking and encrypted history
Another week, another set of cool new Peergos features. We've been hard at work and wanted to tell you about our latest new features: super fast seeking within a file, encrypted history (fix the back button), and storage usage and quota display.
Imagine you were watching a long movie in Peergos using our built in streaming video player, and wanted to skip ahead to where you left off last time, around two hours in. Previously this involved downloading the encrypted metadata for every 5MiB chunk of the file (a few network requests), decrypting the location of the next chunk, and repeating this until you reach the desired offset. The links between the chunks are encrypted so that the outside world (or the server) can't deduce the size of your files. This was SLOOOOOW. To seek a gigabyte or so in took about 5 minutes, or more depending on your internet connection latency! No one wants to wait around that long. so we came up with a new scheme for deterministic, but still private subsequent chunk locations. Before this change each chunk location was 32 random bytes. In the new scheme we calculate the location of a subsequent chunk as the following, where the secret is 32 random bytes that are stored in the encrypted metadata of the first chunk of the file:
How we now calculate the locations of later parts of a file
This means that to seek ahead we don't need to do any IO at all. We just do a bunch of hashing locally. This is ridiculously fast and doesn't leak to the world the connections between your chunks. We can now seek many gigabytes into a file in about 5s, which is mostly the time to retrive the eventual chunk of the file.
A URL with the encrypted path in the fragment.
The second feature this week is encrypted history. Until now, if you press the browser back button you get logged out of Peergos. Super annoying! Now, every time you change directory we store the path, padded and encrypted, in the url fragment. If we didn't encrypt it then there'd be a log of every directory you visited in your browser history. The other requirement is that we catch changes to the url (e.g. caused by pressing the back button) and try to decrypt and interpet the resulting path. This makes navigation much more intuitive.
We also add to the history if you open a file in Peergos, and if you log in through one of these links then we will restore you to the correct directory, and open any file you had open. This allows you to store "deep bookmarks" to things in Peergos which don't reveal anything to someone who get's access you your bookmarks!
Space usage display
The third feature is a display of your current space usage and allowance. This lets you see at a glance how close to your limit you are, and you can request more from the admin for your storage node. That could be a friend who's running a Peergos server for you and gives you more storage for free. Or later down the line we will add the ability for a hosting provider to charge money for storage. After you've requested more storage an admin can approve it next time they log in.
Request more space
-The Peergos team
IPFS Camp, new features, July 2019
At Peergos we care deeply about your privacy and security online. We are super-excited to be building the future of the web where individuals have total control over their data, and who can see it. This is why we prioritised getting a full security audit and why we continue to build a privacy-first platform where you grant applications and people access only to the data you want them to have.
Recently we attended the inaugural IPFS camp organised by Protocol labs from 27th-30th June. Set in magnificent surroundings in the Barcelonian hills, the situation was primed for productivity with 160 pioneers of the decentralised web gathered to meet, discuss, learn and solve. The schedule was packed, with core courses, electives, lightning talks, deep dives, unconf sessions and a sci-fi fair. It was a great experience and hugely useful for Peergos and our mission.
Best deep dive trophy
Our team member Jonathan and his group won the best deep dive, for a product analysis of Peergos. This deep dive was a guided user testing of Peergos which resulted in a ton of feedback and suggestions. One of the suggestions was to implement friend verification in person or over the phone via a qr code or safety number. In fact, one member of the group said they would pay right now for Peergos if it had this feature! So now, a few weeks later, we present friend verification! It works directly from the browser using the webcam, including on mobile. Just scan your friends qr code in person for an added layer of confidence!
QR code based verification of friends
We already have many defenses against a man-in-the-middle attack including mirrored append-only certificate transparency for all user identities, but this adds another layer for the hyper paranoid users.
Another suggestion from our deep dive was that uploading a folder should work. We've attempted this in the past but hit problems with the browser APIs. However, now we have solved the problems and you can drag and drop a folder into Peergos and it will recusively upload all its contents!
We also had a great deep dive session with Berty on a secure decentralised group chat protocol. We were investigating if Messaging Layer Security can be adapted to work in a more decentralised setting. Berty also presented their implementation of a libp2p transport over bluetooth low energy. Very cool!
Until next time, happy Peergossing!
-The Peergos team
Security Audit, June 2019
Thanks to some very generous support from Protocol Labs we have had our first security audit! We contracted the firm Cure53 to give us a large scale security asessment of our cryptography, protocols, architecture and code. We're extremely pleased to be able to announce that no fundamental architectural or cryptographic problems were found, and all the issues that were found have been fixed.
This means that you can rest assured that your data is safe with Peergos.
Read the full report here.
Alpha Release, June 2019
The time you've all been waiting for has arrived... The Peergos alpha is now officially open! You can now have your own private shareable space online.
We have a strictly limited number of accounts on our alpha available for free with an initial 200MB of storage, so get in fast. All we ask in return is that you use it and tell us what we can improve. You can do this in Peergos itself using the feedback button, on Reddit, or via email or Twitter. If you run out of space in Peergos then just message us and we'll increase your storage quota, subject to availability.
Some things to bear in mind:
- We haven't had an external security audit (coming soon), so we advise you not to store anything sensitive in Peergos yet.
- For now, we recommend Chrome for files larger than 50MB. We'd love to be able to recommend Firefox, but they haven't implemented a critical feature we need to safely stream from encrypted data.
- There are still a lot of rough edges and things that can be optimised, but we wanted to get it out there and start getting feedback on what people actually want, now that the foundations are in place.
You can sign up to the alpha here.
Development update, Feb 2019
This is our first development update since we started, in July 2013. For most of that period we've been working on Peergos for free in our spare time. Everything we do is open source and pro bono - If you would like to support our development then please make a recurring donation of less than 100 euros per week or a larger one off donation.
The browser's capabilities have come a long way since we started. Things that we can do now were not possible 5 years ago. The Inter-Planetary File system (IPFS) has come a long way in solving many of the peer-to-peer networking problems and laying a solid structural foundation to build on. That's why very early on after IPFS started we switched from our own networking and distributed hash table to IPFS, allowing us to focus on our core principles of privacy and security.
Thanks to some very generous support from Protocol Labs (the initial creators of IPFS) we have been able to accelerate development recently. We've released a bunch of cool new features and are much closer to our public alpha. Let's go through a few of the big ones. You can see the rest in more detail in our book.
The first new feature is proper decentralization. Now you can self host your own Peergos instance and transparently interact with users on other servers, whilst being independent of the domain name system (DNS) and the SSL certificate authorities (central points of failure outside our control). You can even log in to your Peergos account through someone else's Peergos server. The Peergos interface can actually be entirely self hosted from within IPFS itself (i.e. you can log in through a standard ipfs instance rather than a Peergos instance)! This was all achieved with IPFS p2p streams. p2p streams are a new feature in IPFS which let you create a tcp socket between any two IPFS instances. This stream is end-to-end encrypted with secio. Secio is basically TLS with a different handshake because the end points are not location addressed - instead the addresses are the hash of each node's public key. Like all other IPFS connections, this stream will tunnel through NATs and firewalls transparently.
We implemented a new http endpoint on the IPFS gateway which allows you to proxy a http request to any target IPFS instance (who is listening) using these p2p streams.
This new endpoint is accessed through the gateway (after enabling it - it's an experimental feature for now) at:
For each such request, IPFS will open a p2p stream to $ipfs-node-id and proxy the request to it with a path of $path. Note that the $ipfs-node-id is the hash of the IPFS node's public key. How this works is illustrated below. The beauty of this http endpoint is you don't need to manually open p2p streams for every different endpointdestination, IPFS handles that for you automatically.
Using this key ingredient, we can separate out each user's data into their own server. Most of the data is already in IPFS, but a user's storage server is responsible for pinning it, and for storing the mutable pointers (our equivalent of IPNS), and pending follow requests. This means the only centralized component is the pki, which stores mappings from username to public identity key, and public storage server key. The job of the pki server is to ensure unique usernames and thus it is logically centralized, but we take various efforts to mitigate the effect of this. For example the pki is mirrored on every Peergos server. This has threefold benefits: first it allows users to do private lookups for friends' public keys without announcing to the network who they are looking up; second, it makes the whole system much more scalable - you only need to contact the real pki when you are signing up, or changing your identity or storage server; and third it improves resiliency, if the pki server were to permanently be taken down the network could move to a new one without any loss of data. The pki data itself is stored in IPFS as an append only data structure in a compressed hash array mapped trie (champ).
Granting write access
We've had the ability to grant read access for years, but only this year have we finally implemented granting write access to other users. Every write is signed by a signing keypair. Initially you only have one signing keypair for your entire filestystem, which means that to only grant write access to a subtree, it needs to be moved to a new signing keypair. To explain how this works we're going to take a few steps back. Every directory or 5 MiB section (chunk) of a file requires a unique capability to access it consisting of:
(owner, signer, label, read base key, write base key)
Here the owner and signer are (hashes of) public signing keys, label is a random 32 byte label, and the read and write base keys are symmetric keys. If someone has the first 4, which amount to a location and a key, then they can read the file or folder that it points to. If they also have the write base key, then they can also make modifications.
From a network visible point of view, every signing key pair points to a champ root (by signing the root). This champ is just a look up from the label to the value, which is a cryptree node. You can read more about cryptree in our book, or the original paper describing it. If the thing pointed to is a file, then the cryptree node itself will have merkle links to the encrypted fragments of the chunk. Remember that merkle links are just a hash that points to another ipfs object. This is illustrated below.
When you grant write access to a file or folder then you are just revealing the write base key to them. This enables them to extract the private signing key and thus make modifications. Initially your entire filesystem is under the same signing key pair. This means were we to naively grant write access by sharing this key then the recipient could delete (though not read) all your files. To avoid this we first move the file or directory to which we want to grant write access to a new signing key pair. This allows us to only grant the friend write access to the particular part of our filesystem we want to share. Voila!
If you grant write access to a folder, then that means that the recipient can upload new files or make new directories inside that folder, or delete all the contents. Granting write access to an individual file means the recipient can only modify that file.
We have lots of work in the pipeline, so stay tuned! These include an in Peergos text editor, for a google docs like experience with collaborative editing, a secure messaging system inside Peergos and a sandbox and API for third-party applications inside Peergos.