Parallel Python Software

Parallel Python Software

Overview:

PP is a python module which provides mechanism for parallel execution of python code on SMP (systems with multiple processors or cores) and clusters (computers connected via network).

It is light, easy to install and integrate with other python software.

PP is an open source and cross-platform module written in pure python

Features:

• Parallel execution of python code on SMP and clusters
• Easy to understand and implement job-based parallelization technique (easy to convert serial application in parallel)
• Automatic detection of the optimal configuration (by default the number of worker processes is set to the number of effective processors)
• Dynamic processors allocation (number of worker processes can be changed at runtime)
• Low overhead for subsequent jobs with the same function (transparent caching is implemented to decrease the overhead)
• Dynamic load balancing (jobs are distributed between processors at runtime)
• Fault-tolerance (if one of the nodes fails tasks are rescheduled on others)
• Auto-discovery of computational resources
• Dynamic allocation of computational resources (consequence of auto-discovery and fault-tolerance)
• SHA based authentication for network connections
• Cross-platform portability and interoperability (Windows, Linux, Unix, Mac OS X)
• Cross-architecture portability and interoperability (x86, x86-64, etc.)
  
• Open source

Your own appengine

Your own appengine

My main gripe with appengine is that while you're sticking your app on a lot of iron, it's pretty much stuck there forever because of the infrastructure. Your app is not portable. You can't just take it off of Google's iron and host it yourself.

So, when I was listening to part 3 of the google app engine intro, I heard him describe BigTable as a "a distributed, fault-tolerant and schema-free", I knew I heard that before. I heard that from the CouchDB project. I never saw a need for CouchDB, but it's looking interesting now.

After looking at the SDK for appengine, the datastore interface is pretty simple, there's no reason it couldn't be implemented with CouchDB as the backend. Giving you your own "distributed, fault-tolerant and schema-free" datastore.

Also in part 3 that they run your python code on a low overhead, distributed, fault-tolerant infrastructure.. I knew I heard that before too. I heard it with the Parallel Python project.

Need a decentralized, fault-tolerant file system? There's Tahoe. Tahoo provides that too.

So let's say you have 20 servers, a mix match of database, media and http servers. Now instead of partitioning them to have their own roles, they become nodes in a cloud all handling http, appserving, media file serving and data storage.

If you create a webserver inside of parallel python that basically brokers requests for different sites to the parallel python cloud, a node in llpy then executes the request. llpy may contact the couchdb cloud. All using the power of the cloud. Therefore if you have 22 sites and only 5 get heavy traffic, you don't have to waste the power of the other 17 machines on the slow sites.

The appengine SDK looks like a start to create such a system. The datastore modules provide a way to interface CouchDB. Recreating the http brokering system with llpy shouldn't be that hard.

And once the django community finds a way to work it's ORM onto Google's datastore api (which I know they will), implementing Django inside your own parellel python/couchdb appengine wouldn't be hard.

Creating your own appengine wouldn't be for the average joe schmoe, It would be for big companies with some iron.

Worries about using GAE as a commercial platform?

18 April 2008 - Mark McLaren's Weblog: "Worries about using GAE as a commercial platform?

If I were a business with an interest in using GAE commercially, storing my valuable data in proprietary Google storage would worry me somewhat. There are open source versions of BigTable available (e.g. HBase, HyperTable) and these owe there existence to Google but as there is not yet a standard 'GQL' mechanism, data exit strategies seem limited.

I would be greatly concerned that my applications are hosted by Google and that my Python developers were exposed to Google. GAE could be described as a lobster pot. Getting Google to host your business means that Google are in a prime position to absorb successful businesses, steal talented developers, learn more about your business quicker than you can yourself and learn from your mistakes."

Django | The Web framework for perfectionists with deadlines

Django | The Web framework for perfectionists with deadlines: "Meet Django
Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.

Developed and used over two years by a fast-moving online-news operation, Django was designed to handle two challenges: the intensive deadlines of a newsroom and the stringent requirements of the experienced Web developers who wrote it. It lets you build high-performing, elegant Web applications quickly.

Django focuses on automating as much as possible and adhering to the DRY principle.

Dive in by reading the overview →

When you're ready to code, read the installation guide and tutorial."

Python 3000