Thinking until the 2147483648'th second

About This

I think all of the files I linked to in some of my older posts are gone now. I am working on fixing them.

Sunday, July 8, 2007

All Things Distributed

One of the first languages I'm researching is Erlang. There are various solutions for distributed programming for other languages, specifically Termite for Scheme, however Erlang has proven itself to be an industrial standard. After a little research with Erlang, I can see why, and I'm definitely using Erlang for distributed processing.

Termite is great, and it may prove itself to be the industrial standard distribution system for Scheme in the next couple years. However, it has a ways to go until then. I believe its Windows support is shaky, there's only a few guys behind all the code, there aren't any libraries written with it in mind, and it doesn't have a simple c interface. Not to mention that possibly the only documentation for Termite is Marc Feeley's post about it with some examples.

With Erlang, it was extremely easy for me to sit down, get a distributed system up and running, and even add some c nodes to the network. The c interface to Erlang is what really excites me, because that means you can have a node written in any language that supports a c interface. Other nodes see the c node as an Erlang node, but it's actually completely written in C (or a language supporting native C calls). That means I could have a huge network with various nodes of Scheme, Erlang, Python, etc., but more on this later.

On top of that, you get all the benefits outlined in this post, most importantly the possibility of embedding Mnesia in any of these nodes. Also, ssh into nodes? That would be pretty cool.

It's still vague exactly how much Erlang code I will write, but it will be minimal. I plan to use Erlang for exactly what it's good for - distributing. And I'm realizing more and more that it's not just network distributing, but even local code distribution. Going back to having a network of local nodes written in different languages, each node could be written in the language that fits best for the problem that node is designed to solve. And the the language translation delay (where it's slow to call across language barriers) becomes a very small constant because each node continually runs in its own environment. It never has to leave, it just communicates with messages.

And then I'm free to distribute the system however I want, whether if its across several computers or all locally. And Erlang makes it really easy to implement fault tolerance.

I have more prototyping to do with Erlang, but this looks like it could be a very cool language to use (whether or not I'm actually writing in it :))!

My philosophy behind this project is that many recurring problems in computer programming have been solved already, and I should not re-implement anything that has a good solution already. The difficulty lies in the fact that it may be impractical, and in fact contradictory, to use certain solutions together because of the environment/nature that each involves. It's my job to figure out how closely I can integrate existing well-written solutions without sacrificing performance or design, and deciding exactly what code I need to write.

For example, I'm hoping I can do the following while keeping high performance:
  1. Use Erlang for distribution of code
  2. Use Scheme for its elegant syntax and advanced flow structures (like continuations) and speed (with Gambit)
  3. Use Javascript for graphical 2d UI interaction because of its wealth of libraries
  4. Possibly use Nebula Device 2 for rendering, depending on how hard a Scheme FFI will be
And I'm still heavily researching many other languages/libraries!

Random Notes

I work at Coptix

Ideas / To-do

  • Research .Mac style photo gallery for screenshots