Storj basically refers to an open-source, blockchain-based, or more specifically, decentralized file storage solution. The platform implements encryption, file sharding, and a blockchain-based hash table for the storage of files on a p2p network. The objective in this case is to make storage of files on the cloud private, faster, and cheaper.
Conventional cloud storage solutions, for instance Google Drive and Dropbox, have without doubt exceeded our expectations insofar as cloud storage is concerned. Nonetheless, they come bundled with a number of limitations. As much as with the aforementioned solutions there is redundant backup of files on the cloud, there is the possibility of restricted access to a user’s files due to bandwidth from a data center, given that the storage is centralized, or unanticipated outages. In addition, there arises the issue of privacy. These organizations have absolute control over your files, not to mention your ability to access them.
Thanks to the Storj project, the above mentioned flaws can be mitigated against. Storj implements blockchain and p2p networks to solve these issues. The platform distributes the files in such a way that redundancy is well established. It as well guarantees no one but the user has access to their files.
Before we proceed, it’s fundamental that we point a key disparity between the open-source Storj and Storj Labs, the for-profit organization. Whilst anyone can create their own instance of the open-source software that runs Storj, Storj Labs automatically does this for its users, and it’s worth mentioning that their network already comprises thousands of users. Also, most noteworthy, Storj Labs charges users for using the network.
This post dives deeper into the manner in which the Storj project operates and the success it has so far seen. In the end, having read this post, it will be upon you to establish whether or not the Storj project warrants the acclamation it has so far garnered. Maybe, just maybe, you may decide to start using it for storing your files.
Torrents. Remember Them?
A basic understanding of how torrents work will possibly make it a lot easier for you to start understanding how the Storj project is structured to work. In the early 2000s, torrents became a universal buzzword. It came to the rescue of those who wanted to download TV shows, music, and movies – though illegally, for free.
Torrents work on a p2p network and below is a brief outline of how they work:
- Several users retain copies of a particular file.
- In the event anyone needs a copy of the file, they send a request to the p2p network.
- Users who are in possession of the file, normally referred to as seeds, send fragments of the file to the user who requested it.
- The requester as a result receives several fragments from a variety of seeds, and the torrent software recompiles these fragments of the file to come up with the original file.
The advantage that comes with using torrents regards the fact that a user can download a file’s fragments from multiple sources in parallel. This basically implies that the transfer of the file could be passably quicker as compared to downloading the entire package all at once from a particular source.
What’s more, so far as pirated content goes, the torrents run in a decentralized network. No particular organization has control over the content of the torrent. As such, it is almost impossible to shut down a torrent, on the grounds that you would have to shut down each individual seed.
Storj operates in a typical manner, except it’s not only limited to pirated digital content. Here’s how.
File Sharding is one of the characteristics typical to both torrents and the Storj project. This basically implies that to store a file on the Storj platform, one has to first of all subdivide it into several tinier fragments.
File sharding has a two-fold benefit. First, it enables a user to send and recall shards of the file in parallel. As such, the file transfer process is further quickened. The other advantage is with regard to the fact that no particular entity holds your file’s entirety. As a user, you are the only one with the knowledge of where all the shards of your file are located.
Shard location is a key distinction between torrents and Storj. Torrents are known to publish shard locations publicly. The objective in this case is to make it as simple as possible for anyone to download the files. Storj, as a provider of cloud storage services, obviously gives more priority to the privacy of its users. The person who uploads a file should as such be the only one who knows where all the shards of their file went to or are allocated.
It is at this point that the ideas surrounding cryptography and blockchain come into play. Storj implements what is referred to as a distributed hash table. As such, users are in a position to locate all the shards that form their original file. This distributed hash table necessitates users to possess a private key to realize the shards. It would be almost beyond the bounds of possibility to aptly guess a sharded file’s location without this private key.
Storj utilizes a distributed hash table commonly known as Kademlia. Kademlia is one among many of the fundamental pieces of Storj’s architecture.
Parity Shards and Erasure Coding
The individual file shards are transferred to ordinary computers throughout Storj’s decentralized network. Be that as it may, let’s consider a case where one among the connected computers gets shut down or ceases to run Storj. The question you may most probably ask yourself is whether the shards stored on that particular computer will as such get lost. Well, thanks to the robust technology behind Storj, this question is satisfactorily answerable.
Storj has implemented some sort of redundancy into their system. Most significantly, they do so with parity shards. When a user uploads a file, they are at a position to select the redundancy level they require for their file. Though, Storj will as well help them set this up themselves. With sufficient parity shards, a user can immensely narrow down the possibilities of losing a shard of data from their file.
Having pointed that out, over an elongated period of time, the possibility of losing a shard of data maximizes. Consequently, Storj carries out consistent audits as well as other validation processes to make certain the latter does not go unmitigated. All the same, a best practice would entail users recalling and rebuilding their files periodically prior to reuploading them to Storj.
Needless to say, the reverse is as well an issue as a lot of redundancy impedes the network. Storj has set up erasure coding rules which are responsible for reducing the redundancy of shards that have been duplicated too often. These erasure coding rules as well help Storj point out unique data that requires increased redundancy.
It is at this point that data privacy goes a notch higher. Thus far, sharding adds one layer of user privacy. The latter owes to the fact that no one host of data (also referred to as farmers) can read the file in its entirety. Nonetheless, even being able to read a shard of a file poses a problem, especially in cases where the file contains information deemed sensitive.
In a bid to mitigate against the latter, Storj helps users who upload information to the platform, also referred to as tenants, have their files compressed and encrypted prior to sharding. Each encrypted file comes with only a single key, and the uploader of the file, or rather the tenant, keeps the key locally on their PC or on the Bridge as we will in a short while discuss.
As the encryption key’s owner, the tenant is the only one with the ability to read the file they have uploaded. By the time a farmer receives a shard, it has thus far been encrypted as a fragment of a larger file. The data hosted by a farmer is futile without the encryption key and all the other shards.
Hacking the Storj platform and thusly gaining access to a file would require one to have the locations of every individual shard that makes up a file. The latter is just about implausible without the private key to the Kademlia hash table. You would then have to persuade the farmers hosting the shards to send you the shards of the file without the appropriate signature. In the end, you would have to form an opinion of the encryption key (which is nearly impossible), or steal the encryption key from the uploader of the file (or tenant).
Fingers crossed, you now realize that file storage on a decentralized platform comes with a lot more security as compared to doing so on conventional centralized platforms such as Google Drive or Dropbox.
Verification of Files
The one question that still lingers is, how does a user know that their files are really existent on the network? In addition, let us as well consider a case where a farmer simply deletes their own shards or turns their computer off. What happens then?
To respond to the above mentioned concern, Storj carries out and completes a file validation audit every single hour. So as to get recompensed, farmers are necessitated to show beyond reasonable doubt that they have the shards sent to them. Storj sends a request to the farmers, and in case they have changed or deleted the encrypted shard, they will not be at a position to answer the request.
In the event the farmer holds the file at that particular moment, they can then appropriately answer the request sent by Storj. Significantly, the farmer is recompensed a micropayment for the storage and maintenance of the file. On this account, Storj incentivizes farmers to store the files as well as retain activity on the Storj network.
Insofar as future releases go, Storj is considering the implementation of a reputation system for farmer nodes. This will as a result help give priority to nodes whose operations are on the level and have higher bandwidths.
Storj’s most recent initiative regards the Bridge server. Before the establishment of the Bridge server, tenants would store their private encryption keys on their local PCs. This was fine particularly in cases where users would want to access their files from the same PC. Nonetheless, an inconvenience arises at the point where the tenant has to switch devices.
Bridge is a server that caches users’ encryption keys without centralizing their access. The server securely stores the keys in such a manner that their bearers can easily access their files from multiple gadgets.
STORJ, the Storj token (coin), serves as a payment means on the platform. Fees paid by tenants go to the farmers/miners who contribute space for storage as well as bandwidth to the network.
Whilst Storj Labs’ implementation of the Storj particularly uses the STORJ token, open-source Storj’s payment system is agnostic. Other than STORJ, such crypto coins as BTC or ETH can as well be implemented.
- Token supply: 500 million
- Distributed in ICO: Up to 25% (June 2017)
- Emission rate: No new coins created.
- Blockchain: Ethereum
- Consensus: Proof of Work
The team at Storj comprises of established startup executives. The official team comprises of roughly forty employees, with a broader community in support of the open-source initiatives.
Decentralized cloud storage is an entrancing use-case for both p2p networks and blockchain technology. Other than Storj, Filecoin, Sia, and Maidsafe are among the other platforms that offer these services. For Storj enthusiasts, the good news is that the platform is seemingly near the front of the pack insofar as usability, adoption, and the underlying technology go.
Here is what Vitalik Buterin, Ethereum’s Chief Executive Officer, had to say about decentralized file storage systems:
“Decentralized file storage systems like Storj have the potential to eliminate high markup costs and market inefficiencies and provide a much higher level of privacy, reliability, and quality of service than we see today.”