Welcome to Tahoe

Welcome to Tahoe
A "storage grid" is made up of a number of storage servers. A storage server has local attached storage (typically one or more hard disks). A "gateway" uses the storage servers and provides to its clients a filesystem over a standard protocol such as HTTP(S), FUSE, or SMB.

Users do not rely on storage servers to provide confidentiality nor integrity for the data -- instead all of the data is encrypted and integrity-checked by the gateway, so that the servers can neither read nor alter the contents of the files.

Users do rely on storage servers for availability. The ciphertext is erasure-coded and distributed across N storage servers (the default value for N is 10) so that it can be recovered from any K of these servers (the default value of K is 3). Therefore only the simultaneous failure of N-K+1 (with the defaults, 8) servers can make the data unavailable. Phrasing this in terms of reliance, we say that the users rely on the gateway for the confidentiality and integrity of the data, and on any 3 of the 10 servers for the availability of the data.

In the typical deployment mode each user runs her own gateway on her own machine. This way she relies on only her own machine for the confidentiality and integrity of the data, and she can take advantage of filesystem integration using FUSE or SMB.

An alternate deployment mode is that the gateway runs on a remote machine and the user connects to it over HTTPS. This means that the operator of the gateway can view and modify the user's data (the user relies on the gateway for confidentiality and integrity), but the user can access the filesystem with a client that doesn't have the gateway software installed, such as an Internet kiosk or cell phone.

Your own appengine

Your own appengine

My main gripe with appengine is that while you're sticking your app on a lot of iron, it's pretty much stuck there forever because of the infrastructure. Your app is not portable. You can't just take it off of Google's iron and host it yourself.

So, when I was listening to part 3 of the google app engine intro, I heard him describe BigTable as a "a distributed, fault-tolerant and schema-free", I knew I heard that before. I heard that from the CouchDB project. I never saw a need for CouchDB, but it's looking interesting now.

After looking at the SDK for appengine, the datastore interface is pretty simple, there's no reason it couldn't be implemented with CouchDB as the backend. Giving you your own "distributed, fault-tolerant and schema-free" datastore.

Also in part 3 that they run your python code on a low overhead, distributed, fault-tolerant infrastructure.. I knew I heard that before too. I heard it with the Parallel Python project.

Need a decentralized, fault-tolerant file system? There's Tahoe. Tahoo provides that too.

So let's say you have 20 servers, a mix match of database, media and http servers. Now instead of partitioning them to have their own roles, they become nodes in a cloud all handling http, appserving, media file serving and data storage.

If you create a webserver inside of parallel python that basically brokers requests for different sites to the parallel python cloud, a node in llpy then executes the request. llpy may contact the couchdb cloud. All using the power of the cloud. Therefore if you have 22 sites and only 5 get heavy traffic, you don't have to waste the power of the other 17 machines on the slow sites.

The appengine SDK looks like a start to create such a system. The datastore modules provide a way to interface CouchDB. Recreating the http brokering system with llpy shouldn't be that hard.

And once the django community finds a way to work it's ORM onto Google's datastore api (which I know they will), implementing Django inside your own parellel python/couchdb appengine wouldn't be hard.

Creating your own appengine wouldn't be for the average joe schmoe, It would be for big companies with some iron.