Thinking until the 2147483648'th second

About This

I think all of the files I linked to in some of my older posts are gone now. I am working on fixing them.

Sunday, July 29, 2007

Yay for simplicity! Gambit's namespacing

For a while now I've wondered how people usually organize Scheme code in large projects. There are various module systems for Scheme implementations, and some are rather bulky. One of my main questions is how object systems relate to modules; I haven't been able to find much about this.

I like Gambit's namespace directive; beauty comes in simplicity. I know the Scheme purist guy throws up a little in his mouth whenever he thinks about it, and here's why: it uses name mangling. It seems that there are two types of Scheme programmers; those that care more about practical applications of using Scheme, and those that care more about being theoretically correct. It's rather non-deterministic which one is better. Often the purist comes out in the end when his app behaves the way it should, but many times the practical guy wins when he gets things done quick or his app runs faster.

In general, I love how dedicated the Scheme community is with being algorithmically correct. I think we should almost always favor theoretically-correct design over hacked-for-optimization design. Almost, though, because there are cases where we want to apply the greatness of Scheme to areas which require high performance, such as graphics engines. And this is where I am (designing a graphics engine), so, however imperfect it may seem, I will tend to hack the design of my scheme code to favor critical performance.

On a quick side note, it seems that Gambit represents a system which strives to be theoretically correct, but recognizes the "need for speed." Scheme48, on the other hand, will always make decisions purely based on the formal theory behind it.

Back to Gambit's module system. The namespace directive simply rewrites definitions with the supplied prefix. This maps well to Gambit's builtin define-type, which is a basic data type definition. define-type is a macro which writes out several functions which make the object, set/get its properties, etc. Using the namespace directive, these will get properly renamed.

Well, kind of. You don't exactly import and export symbols, but you simply pass a list of symbols along with the namespace declaration which claim "this namespace owns these symbols." Any reference to those variables will refer to (namespace)#(variable). If you don't pass any variables, you are claiming your namespace owns everything, which is why you lose all the standard definitions:
> (namespace ("foo#"))
> (define a 5)
*** ERROR IN (console)@2.2 -- Unbound variable: foo#define
Most of the time people explicitly list the variables they want associated with the namespace:
> (namespace ("foo#" a))
> (define a 5)
> (define b 7)
> a
5
> foo#a
5
> foo#b
*** ERROR IN (console)@7.1 -- Unbound variable: foo#b
This presents a problem though: for rewriting macros such as define-type we don't always know the real symbols to export (even if we do, it would be tedious to list them, that's the whole point of the macro). Are these the kind of problems syntax-case solves? I've heard that a proper module system works with syntax-case and this kind of thing can be done.

I want to stick with name mangling and something similar to define-type however because of the performance gain in Gambit with top-level functions. I'm not sure what to do about exporting types (aka objects), I'll have to prototype and see how much I need to use types outside of modules. I could probably get by with defining constructor/getter/setter functions that should be used by the outside world, and manually export them. That may be better anyway; only let the local namespace modify the type data directly.

Of course, I could write a tool that analyzes the code and generating a proper namespace header file. I'll have to see if that's necessary though.

We can still get macros like define-type to bind the created function to the local namespace by doing something like this:
> (namespace ("graphics#"))
> (##include "~~/lib/gambit#.scm")
> (define-type point x y z)
> (namespace (""))
> (make-point 1 2 3)
*** ERROR IN (console)@5.2 -- Unbound variable: make-point
1> ,d
> (graphics#make-point 1 2 3)
> #
Another solution might be to always use fully qualified names. I'll have to take a look at Christian Jaeger's chjmodule system to see how he handles some of this. I may look into using his system in the future.

Although it seems somewhat primitive, I like the simplicity that Gambit offers. It seems that exporting special types such as objects is an inherent problem with any module system, where special support must be built. I'd rather write that support so that I can use objects that way I want to.

Tuesday, July 24, 2007

Project Flow

As any reader would have guessed from the previous posts, I am starting a project. Up until now, it's been a vague collection of varying ideas. I'm "announcing" (quotes because nobody read this right now) the project now, in hopes to clarify requirements and get some work done on it.

It will be "code"-named Flow. "Code" sounds quite secret, while it's only meant to convey the temporary nature of the name. I can't think of a better one, and no, it's not named after the Spore-inspired flash game (seriously, I've had it in my head ever since I got the idea of a Scheme-based graphics engine, which was a while ago).

So, what will Flow be? It will be a collection of tools written in Scheme for developing 3d graphics and artificial intelligence. It isn't exactly focused on game creation, but rather some of the more specific and important aspects of it. And it could certainly be used for game creation, but there won't be any devices for handling game logic.

The first step is to create a graphics engine completely written in Scheme. From there I will start working on some artificial intelligence applications which will use the renderer for output, but that's for later.

'Flow' indicates how flexible the system should be. The system should feel smooth and easily configurable, while being fast.

I am still in the research/design phase of all this. I have decided on a few things though: it will use Gambit Scheme with Termite, and Javascript for UI.

In an effort to organize my research, I'm compiling a list of modules that I will start with:

Types:
  1. Native (to scheme)
  2. FFI (os-specific)
  3. FFI (general)

Window
Type: FFI (os-specific)
Function: Will provide functions for creating a window and managing it, as well as functions to query display info
Dependencies: None

OpenGL Implementation
Type: FFI (os-specific)
Function: Will provide functions for calling OS specific OpenGL libraries (AGL, CGL, WGL, etc.)
Dependencies: Window

OpenGL
Type: FFI (general)
Function: Will provide methods for rendering and other various graphical operations
Dependencies: None

Math
Type: Unknown (thoughts?)
Function: Will provide methods for calculating linear algebar equations
Dependencies: None

Renderer
Type: Native
Function: Will provide higher-level functions for rendering operations
Dependencies: OpenGL, OpenGL Implementation

Scene
Type: Native
Function: Will provide functions and structures to provide scene structure and querying
Dependencies: None

I'm relatively new to functional programming, so figuring out how to structure these modules and their communication will be fun.

Monday, July 16, 2007

Gambit-C Erlang FFI

I have finished up a prototype Gambit-C Erlang FFI that I have been working on. It's a neat technique and could be very valid in a situation where you want to use Scheme for various tasks in an environment like Yaws. However, if using Scheme for more than achieving various tasks, and using it for some of its mind-blowingly weird and cool stuff and basing an application around this, you're better off using Termite.

If you're interested, you can download it.

To be able to take advantage of all of Erlang's distributing
mechanisms, we need to use a proxy Erlang process and communicate through ports, as described here:

http://www.erlang.org/doc/tutorial/erl_interface.html#5

This is what my FFI is based on. It's a very basic implementation, but it allows to write a Scheme app like this:

(load "erlang")

(define (bar num)
(+ num 2))

(define (foo num)
(* num 2))

(define (start)
(start-erl)
(let loop ((term (erl-receive)))
(if (not (eqv? term 'close))
(begin
(let* ((fun (car term))
(args (cdr term))
(res (apply (eval fun) args)))
(erl-send (erl-element res))
(loop (erl-receive)))))))

(start)
And then in Erlang do this:
 2> test:start("./test_process").
<0.38.0>
3> test:call({foo,10}).
20
4> test:call({foo,12}).
24
5> test:call({foo,25.2}).
50.4000
6> test:call({foo,12.25}).
24.5000
7> test:call({bar,100}).
102
8> test:call({bar,100.123}).
102.123
9>
Pretty neat.

Sunday, July 8, 2007

All Things Distributed

One of the first languages I'm researching is Erlang. There are various solutions for distributed programming for other languages, specifically Termite for Scheme, however Erlang has proven itself to be an industrial standard. After a little research with Erlang, I can see why, and I'm definitely using Erlang for distributed processing.

Termite is great, and it may prove itself to be the industrial standard distribution system for Scheme in the next couple years. However, it has a ways to go until then. I believe its Windows support is shaky, there's only a few guys behind all the code, there aren't any libraries written with it in mind, and it doesn't have a simple c interface. Not to mention that possibly the only documentation for Termite is Marc Feeley's post about it with some examples.

With Erlang, it was extremely easy for me to sit down, get a distributed system up and running, and even add some c nodes to the network. The c interface to Erlang is what really excites me, because that means you can have a node written in any language that supports a c interface. Other nodes see the c node as an Erlang node, but it's actually completely written in C (or a language supporting native C calls). That means I could have a huge network with various nodes of Scheme, Erlang, Python, etc., but more on this later.

On top of that, you get all the benefits outlined in this post, most importantly the possibility of embedding Mnesia in any of these nodes. Also, ssh into nodes? That would be pretty cool.

It's still vague exactly how much Erlang code I will write, but it will be minimal. I plan to use Erlang for exactly what it's good for - distributing. And I'm realizing more and more that it's not just network distributing, but even local code distribution. Going back to having a network of local nodes written in different languages, each node could be written in the language that fits best for the problem that node is designed to solve. And the the language translation delay (where it's slow to call across language barriers) becomes a very small constant because each node continually runs in its own environment. It never has to leave, it just communicates with messages.

And then I'm free to distribute the system however I want, whether if its across several computers or all locally. And Erlang makes it really easy to implement fault tolerance.

I have more prototyping to do with Erlang, but this looks like it could be a very cool language to use (whether or not I'm actually writing in it :))!

My philosophy behind this project is that many recurring problems in computer programming have been solved already, and I should not re-implement anything that has a good solution already. The difficulty lies in the fact that it may be impractical, and in fact contradictory, to use certain solutions together because of the environment/nature that each involves. It's my job to figure out how closely I can integrate existing well-written solutions without sacrificing performance or design, and deciding exactly what code I need to write.

For example, I'm hoping I can do the following while keeping high performance:
  1. Use Erlang for distribution of code
  2. Use Scheme for its elegant syntax and advanced flow structures (like continuations) and speed (with Gambit)
  3. Use Javascript for graphical 2d UI interaction because of its wealth of libraries
  4. Possibly use Nebula Device 2 for rendering, depending on how hard a Scheme FFI will be
And I'm still heavily researching many other languages/libraries!

Wednesday, July 4, 2007

Introduction

So here I am, finally using a blog to keep track of different things I'm researching. I've been meaning to do this for a while.

Basically, I'm starting to research a broad amount of ways to implement a flexible development environment for interactive 3d worlds. I know that's extremely vague, but I don't even know exactly what my goal is yet. I'm sure that it won't be the pure 'gameplay' type programming because honestly that doesn't interest me as much as other things. I'm fascinated by AI and self-populating worlds, as well as advanced graphical techniques. Somewhere this will merge into an interesting place where I think the term 'game' might be applied, but only by a natural progression from an incredible amount of interaction with some smart AI creating the rules.

This blog will serve as my platform for logging technical details and personal opinions about everything I research. I will describe the process I'm going through and reflect on different ways of achieving my goal. Something I need to do soon is actually state my goal, but I can still look around at a couple things first. :)

So here's where I am now: I need to get familiar with more programming languages to get a feel for what's really possible, specifically functional (ish) languages. I will do this by researching several languages, pick a few out and develop prototype systems using those languages. On my list of languages are Lisp/Scheme, Erlang, Smalltalk, C, C++, maybe Haskell, and others. I'd love to hear peoples' comments about these languages as I'm researching each of them.

To start off, I found an interesting article about Erlang and multiplayer gaming (Writing low-pain massively scalable multiplayer servers). I'm very interested in Erlang as possibly the 'controlling' language that manages the flow of the program, while most of the actual program is written in Scheme/C++. More research is required on this though.

I really need to modularize this project and figure out how deep I want to go in each module. For example, I haven't decided if I want to develop a rendering engine from scratch or try something like Nebula Device (which looks fantastic). The deciding factor will be the languages I choose to work in and how hard that makes it to interface into an engine, but I'll most likely write the lower level modules in C++ anyway so an interface is going to be required.

Nebula Device looks awesome though, and it's worth at least a prototype to see if I should use it or not.

Random Notes

I work at Coptix

Ideas / To-do

  • Research .Mac style photo gallery for screenshots