Welcome to Tahoe
A "storage grid" is made up of a number of storage servers. A storage server has local attached storage (typically one or more hard disks). A "gateway" uses the storage servers and provides to its clients a filesystem over a standard protocol such as HTTP(S), FUSE, or SMB.
Users do not rely on storage servers to provide confidentiality nor integrity for the data -- instead all of the data is encrypted and integrity-checked by the gateway, so that the servers can neither read nor alter the contents of the files.
Users do rely on storage servers for availability. The ciphertext is erasure-coded and distributed across N storage servers (the default value for N is 10) so that it can be recovered from any K of these servers (the default value of K is 3). Therefore only the simultaneous failure of N-K+1 (with the defaults, 8) servers can make the data unavailable. Phrasing this in terms of reliance, we say that the users rely on the gateway for the confidentiality and integrity of the data, and on any 3 of the 10 servers for the availability of the data.
In the typical deployment mode each user runs her own gateway on her own machine. This way she relies on only her own machine for the confidentiality and integrity of the data, and she can take advantage of filesystem integration using FUSE or SMB.
An alternate deployment mode is that the gateway runs on a remote machine and the user connects to it over HTTPS. This means that the operator of the gateway can view and modify the user's data (the user relies on the gateway for confidentiality and integrity), but the user can access the filesystem with a client that doesn't have the gateway software installed, such as an Internet kiosk or cell phone.
Saturday, April 19, 2008
Welcome to Tahoe
Parallel Python Software
Overview:
PP is a python module which provides mechanism for parallel execution of python code on SMP (systems with multiple processors or cores) and clusters (computers connected via network).
It is light, easy to install and integrate with other python software.
PP is an open source and cross-platform module written in pure python
Features:
- Parallel execution of python code on SMP and clusters
- Easy to understand and implement job-based parallelization technique (easy to convert serial application in parallel)
- Automatic detection of the optimal configuration (by default the number of worker processes is set to the number of effective processors)
- Dynamic processors allocation (number of worker processes can be changed at runtime)
- Low overhead for subsequent jobs with the same function (transparent caching is implemented to decrease the overhead)
- Dynamic load balancing (jobs are distributed between processors at runtime)
- Fault-tolerance (if one of the nodes fails tasks are rescheduled on others)
- Auto-discovery of computational resources
- Dynamic allocation of computational resources (consequence of auto-discovery and fault-tolerance)
- SHA based authentication for network connections
- Cross-platform portability and interoperability (Windows, Linux, Unix, Mac OS X)
- Cross-architecture portability and interoperability (x86, x86-64, etc.)
- Open source
Apache CouchDB: The CouchDB Project
Apache CouchDB: The CouchDB Project: "The CouchDB Project
CouchDB Server Components
Apache CouchDB is a distributed, fault-tolerant and schema-free document-oriented database accessible via a RESTful HTTP/JSON API. Among other features, it provides robust, incremental replication with bi-directional conflict detection and resolution, and is queryable and indexable using a table-oriented view engine with JavaScript acting as the default view definition language."
Your own appengine
Your own appengine
My main gripe with appengine is that while you're sticking your app on a lot of iron, it's pretty much stuck there forever because of the infrastructure. Your app is not portable. You can't just take it off of Google's iron and host it yourself.
So, when I was listening to part 3 of the google app engine intro, I heard him describe BigTable as a "a distributed, fault-tolerant and schema-free", I knew I heard that before. I heard that from the CouchDB project. I never saw a need for CouchDB, but it's looking interesting now.
After looking at the SDK for appengine, the datastore interface is pretty simple, there's no reason it couldn't be implemented with CouchDB as the backend. Giving you your own "distributed, fault-tolerant and schema-free" datastore.
Also in part 3 that they run your python code on a low overhead, distributed, fault-tolerant infrastructure.. I knew I heard that before too. I heard it with the Parallel Python project.
Need a decentralized, fault-tolerant file system? There's Tahoe. Tahoo provides that too.
So let's say you have 20 servers, a mix match of database, media and http servers. Now instead of partitioning them to have their own roles, they become nodes in a cloud all handling http, appserving, media file serving and data storage.
If you create a webserver inside of parallel python that basically brokers requests for different sites to the parallel python cloud, a node in llpy then executes the request. llpy may contact the couchdb cloud. All using the power of the cloud. Therefore if you have 22 sites and only 5 get heavy traffic, you don't have to waste the power of the other 17 machines on the slow sites.
The appengine SDK looks like a start to create such a system. The datastore modules provide a way to interface CouchDB. Recreating the http brokering system with llpy shouldn't be that hard.
And once the django community finds a way to work it's ORM onto Google's datastore api (which I know they will), implementing Django inside your own parellel python/couchdb appengine wouldn't be hard.
Creating your own appengine wouldn't be for the average joe schmoe, It would be for big companies with some iron.
Worries about using GAE as a commercial platform?
18 April 2008 - Mark McLaren's Weblog: "Worries about using GAE as a commercial platform?
If I were a business with an interest in using GAE commercially, storing my valuable data in proprietary Google storage would worry me somewhat. There are open source versions of BigTable available (e.g. HBase, HyperTable) and these owe there existence to Google but as there is not yet a standard 'GQL' mechanism, data exit strategies seem limited.
I would be greatly concerned that my applications are hosted by Google and that my Python developers were exposed to Google. GAE could be described as a lobster pot. Getting Google to host your business means that Google are in a prime position to absorb successful businesses, steal talented developers, learn more about your business quicker than you can yourself and learn from your mistakes."
Django | The Web framework for perfectionists with deadlines
Django | The Web framework for perfectionists with deadlines: "Meet Django
Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.
Developed and used over two years by a fast-moving online-news operation, Django was designed to handle two challenges: the intensive deadlines of a newsroom and the stringent requirements of the experienced Web developers who wrote it. It lets you build high-performing, elegant Web applications quickly.
Django focuses on automating as much as possible and adhering to the DRY principle.
Dive in by reading the overview →
When you're ready to code, read the installation guide and tutorial."
Twitter Tracking
This is what I'm currently tracking: "gae", "google app engine", "dojo", "yui", "bigtable", "mapreduce", "gdata", "gears", "google gadgets", "opensocial", "silverlight", "hadoop".
Money for Nothing
Yahoo! 360� - Douglas Crockford's The Department of Style: "Money for Nothing
In the realm of music, it is pretty clear now, even to the so-called Music Industry, that DRM is not a viable business model. Apple's thralls still shop at iTunes, but The Rest of Us aren't buying it.
So the cartel is exploring a taxation model. They want a piece of all music sales and/or a piece of all music player sales, which would probably include all computers and cellphones, to compensate them for their failures to adapt to the changing technological landscape. The cartel is not entitled to your money and should not get it.
Your Congressional Representative is up for reelection this year, and maybe one of your Senators. So ask them where they stand on the Music Tax. Make it an issue. Remind them who they work for."
Friday, April 18, 2008
The GAE SWF Project at Aral Balkan
The GAE SWF Project at Aral Balkan: "Today I'm releasing The GAE SWF Project, a resource of Flash and Flex-related knowledge specifically aimed at getting you up and running quickly with Google App Engine. If you want to skip all the details and start playing with it, skip to the Getting Started section.
Everyone else, read on!
When Google announced Google App Engine last week, I couldn't believe what I was hearing. It was as if someone had said, 'Hey, Aral, tell us what your perfect development environment is and we'll build it for you.'"
Hadoop Summit Slides and Video Available
Hadoop Summit Slides and Video Available from Hadoop and Distributed Computing at Yahoo!
It's been a few weeks since the Hadoop Summit in Santa Clara, and we hope everyone had a good time and learned a lot. Feedback has been quite good so far, but don't be shy about sending us comments. The Yahoo! Research team has assembled a single page containing links to all the presentation slides and video from both the Hadoop Summit and the Data Intensive Computing Symposium. As a sample, here's the opening presentation that Doug and Eric gave: Update: Videos are currently unavailable outside of Yahoo! We're working on the problem..."
Google Datastore and the shift from a RDBMS
Google Datastore and the shift from a RDBMS: "Diving into the Datastore docs to get a grip on what’s the best way to implement it shed some light on the transition any developer thinking about writing data-backed apps for GAE (Google App Engine) will need to tackle.
Some notes on terminology, Google has Entities, Kinds, and Properties. These correspond roughly to Rows, Tables, and Columns in RDBMS-speak. Kinds can also be called classes, because in the Python API, you create a class and inherit from the appropriate datastore class. Entities may also be referred to as instances, since performing a query returns a list of objects (instances)."
Google AppEngine - A Second Look | High Scalability
Google AppEngine - A Second Look | High Scalability: "It's been a few days now since GAE (Google App Engine) was released and we had our First Look. It's high time for a retrospective. Too soon? Hey, this is Internet time baby. So how is GAE doing? I did get an invite so hopefully I'll have a more experience grounded take a little later. I don't know Python and being the more methodical type it may take me a while. To perform our retrospective we'll take a look at the three sources of information available to us: actual applications in the AppGallery, blogspew, and developer issues in the forum."
RescueTime
Frequently Asked Questions | RescueTime: "RescueTime is a web-based time-management tool that allows you to easily understand how you spend your time. One of the coolest things about RescueTime is that there is NO DATA ENTRY. You install a doohicky on your computer and we magically track all of your time usage. For more info, check out the product tour.
Cool idea, but I always have a lot of different applications and sites open at any given time. How does RescueTime handle that ?
RescueTime doesn't really care which applications you have open, but rather pays attention to which applicaiton or site is currently 'in focus'. In other words, we're measuring what's what you are paying attention to, not what you have open."
Thursday, April 17, 2008
twitter.zappos.com
I must confess that I've never really "got" Twitter, so today I went on a campaign to follow as many participants as I can to try and get a sense of the zeitgeist. The article belong is the best into I've seen and the first thing I noticed was that it seemed more useful via a mobile phone that a web browser. I've started using Google Talk as my main interface. When you type help into it you get a menu:
Reply with what you're doing.
'invite' to invite a friend.
'follow' to receive updates.
'track' to track interests.
'whois' for info.
'off' to silence.
Typing the commands gives more help. You can follow me by typing: follow CardHolder. You can silence me with: off CardHolder and unfollow me with: leave CardHolder.
Perhaps the most interesting feature was one i wasn't previously aware of, you can track keywords that you are interested in as they come up in peoples tweets. To track "yui" for instance you type: track yui. The opposite operation is: untrack yui.
I've set up a series of tracks on topics of interest to this blog, we'll see how it goes.
---MG
twitter.zappos.com: "I know that I've been spending about half an hour every time I try to convince my friends to sign up for Twitter. At first, they think it sounds interesting but aren't really motivated to sign up. Sometimes it's been a multi-week long process. But finally they relent and sign up, probably just so they can shut me up. I walk them through the signup process, step by step, and then slowly but surely, they become addicted and their lives are never the same again.
So to save me from having to give the same spiel over and over again, I decided to create this page for anyone new to Twitter."
Todd Bishop's Microsoft Blog
Todd Bishop's Microsoft Blog: "Ballmer says Vista is a 'work in progress'
More than a year after Windows Vista's release, Microsoft CEO Steve Ballmer this morning acknowledged some of the operating system's shortcomings -- calling it a 'work in progress' during a speech to the company's Most Valuable Professionals conference in Seattle. He also acknowledged Windows XP's fans, but he stopped short of committing to further extend the older operating system's life.
Here's an extended excerpt from his comments on the topic:
Windows Vista: A work in progress. [Laughter, applause.] A very important piece of work, and I think we did a lot of things right, and I think we have a lot of things we need to learn from. Certainly, you never want to let five years go between releases. Can we just sort of kiss that stone and move on? Because it turns out many things become problematic when you have those long release cycles. The design point, what you should be targeting. We can't ever let that happen again. We had some things that we can't just set the dial back, but I think people wish we could. Vista is bigger than XP. It's going to stay bigger than XP. We have to make sure it doesn't get bigger still, and that the performance and that the battery life and that the compatibility, we're driving on the things that we need to drive hard to improve."
How to Add Google Analytics to Your Blogger Blog
How to Add Google Analytics to Your Blogger Blog: "How to Add Google Analytics to Your Blogger Blog
Yesterday we talked about Google Analytics, Google’s free service that lets you track the performance of your websites and blogs from one simple, easy-to-use interface.
Grab Your Google Analytics Code Block
1. Login to Google Analytics at http://google.com/analytics/. The main Settings page loads.
2. Click on Add Website Profile. A form displays.
3. Select Add a Profile for a New Domain.
4. Enter the URL of your site or blog.
5. Select your country and time zone. Click Finish.
6. Analytics provides you with a code block - a swatch of HTML - to add to your site’s pages.
7. Highlight the code block and then copy it by selecting Edit > Copy or Ctrl-C or Command-C."
Add the Google Analytics Code Block to Your Blogger Blog
- Login to http://www.blogger.com/. The Dashboard loads.
- Under the blog you want to add Analytics tracking to, click on Layout or Template.
- Click on Edit HTML. An editing screen for your blog template’s HTML displays. Don’t freak out. Just scroll to the bottom.
- Look for the end of the template. It’ll look like:
(Google Analytics Code Block is going to go here!!!)
- Put your cursor right before that tag.
- Paste the Google Analytics Code Block by selecting Edit > Paste, Ctrl -V or Command-V.
- Click Save Changes.
You have now added the Google Analytics Code Block to Your Blogger Blog.
Check Your Work
- To ensure that you have successfully added the Google Analytics Code Block to your Blogger blog, go back to http://google.com/analytics/.
- Next to your blog’s URL it will say either Receiving Data (you were successful) or Tracking Not Installed (something is amiss).
- If it said Tracking Not Installed, click on Check Status. Google then checks your blog for the Analytics Code Block and reports back if it find it or not.
- If not, try re-pasting the Code Block in.