How To Interface Gambas With External Libraries

Introduction

There are lots of shared libraries available in a Linux system, capable of doing a lot of useful things, and many of those libraries can be used from Gambas using some of its features.

The first step to do is to find a suitable library to use for a given purpose; not all the libraries can be used, but the vast majority can. The prerequisite is that the library is written in plain C. Most libraries are written in C, while others are written in C++ and, perhaps, still other languages. This document will focus only about C libraries.

Once the intended library is found, its general logic must be understood in order to determine what is needed, and how this new stuff will be used by the final program. The full documentation of the library, and possibly some example, should be read carefully. Libraries are not programs, and hence their philosophy is quite different. A program tends to contain only the subroutines required to do its job, while a library is what the name implies: a collection of subroutines, to be used many times, by many different programs. It is not uncommon to find, inside libraries, two or more subroutines which do the same thing, in a slightly different way. Libraries sometimes are written keeping in mind that they will be used by different languages, not only C: python, ruby, ocaml, many others, Gambas included. Libraries tend to encapsulate the details of a task inside a "handle", in a way similar to a Gambas object; but, they don't have properties and methods - everything is carried out by calling functions which these handles are passed to. If you think of a Gambas class with three properties and four methods, an external library implementing the same thing will have at least seven (three+four) subroutines, and probably a couple more to create and destroy the object. Other than this, there is not a big difference from setting a property and calling a function, the latter is simply a bit longer to write. To search for what we need, we must be prepared to read a lot of documentation, too often badly written (hey, apropos, how well do we document our software?).

The Extern Declaration

Extern declaration is simple. It is like a normal Gambas subroutine declaration, but preceded by "EXTERN". The EXTERN keyword says to gambas that the body of the procedure is not defined by the gambas program we are writing, but somewhere else (an external library). We must also specify which library to use: this is done by the clause "IN libraryXXX". Alternatively, you can use a separate LIBRARY statement: all the subsequent extern declarations will refer to this statement. Either "IN library" or "LIBRARY xxx" can specify a version number after a colon, and this is recommended.

Let's see an example, choosen because of its simplicity:

These two lines say that a function named "getgid" exists in the library "libc" version 6. This function takes no parameters, and returns an integer (the group ID).

The same thing can be written like this:

Another example, slightly more complicated. This time we have parameters, and the last thing to say about the formal declaration:

The first line (the comment) shows the original declaration, and the second line the gambas one. We can note a number of things. (The difference in function names will be explained shortly.)

First, what does the original declaration mean? It means "a function called kill returns an int, and it accepts two parameters. The first is named pid and its type is pid_t; the second is named sig and its type is int". Contrary to Gambas, the C language puts the type of a variable before the variable instead of after.

Second, what in Gambas is called "Integer", in C is called "int".

Third, what is "pid_t"? It's a type; we can understand it because it is written in a place where a type specifier is expected, and because it ends with "_t" (underscore t).

Fourth, a new clause EXEC "kill" is used in the Gambas declaration. This is necessary because we want to use a function named kill, but KILL is a name already used by Gambas. So, in Gambas, we must name the function differently, but anyway we must indicate its true name inside the library. The declaration says "I declare an external function named killme, but its real name is kill". I chose the name killme because in the attached gambas example this function is used to kill the running program.

To be sincere I noticed that, even without renaming the function from kill to killme, the program was working the same. May be that this has something to do with case sensitivity - C is sensitive, and Gambas not, so there is still a difference between kill (lowercase) and KILL (uppercase). Anyway, when there is a possible name conflict, it is best to use this renaming technique.

Warning --- End Of The Easy Part

(just joking)

At this point, several things must be noted. Most of the suitable libraries are written in C, which is a different language than Gambas. We will need to know at least a little of C declarations, in order to translate them to gambas. Referring to the last example, one could ask why I translated the pid_t type to integer. The simple and correct answer is "because on my system the pid_t type is actually an integer". This answer is really correct but must be explained better. We can think about lemons and oranges, which both are citrus fruits, and are very similar: they weight more or less the same, and often can be interchanged; you can eat them directly, or squeeze them to drink their juice, but it is unlikely that you will put orange juice on your fried fish. In C, this is expressed by the fact that it is unlikely that you want to use the kill() function passing it an arbitrary integer. Surely, you will pass a process identifier (PID), which actually is an integer, but it is indicated more precisely as pid_t. In my motivation, I also said "on my system the pid_t type...". Yes, on my system - on most systems, the type pid_t is an integer, but this could be different.

The final answer can be found by typing these two commands in a terminal:

which will show the involuted way types are managed in C. This argument is way too complicated to go further; giving that Gambas runs on Linux, and presumably on desktop systems, we can consider that all the parameters passed to a function, and returned by it, will be either integer, or pointers - which are integer too, or strings - which are pointers that are integers. There can be also floating point numbers - float and double. The following table lists some of the types you can encounter in a C declaration, and the suitable type to be used in gambas:

We will start to briefly introduce pointers, which are little used in Gambas. A pointer is an integer, but used very differently. The thing that more closely resembles a pointer in Gambas is a class instance. When you create, say, a Form in Gambas, a lot of data is stored somewhere in memory. That memory will hold all the specific settings of the form: its caption, its color, the list of all its children, and so on. The address of that block of memory is returned to your program, and stored into the variable which refers to the just created form:

That variable MainForm is really a pointer: in only 4 (or 8) bytes it tracks a lot of data, stored somewhere in memory at a specific location (address). The memory is a long sequence of cells (bytes), each identified by a progressive number. A pointer contains the identifying number of a cell of memory (its address). In C, pointers are used for two reasons: the first is that passing only an address (a pointer contains an address), is much quicker than passing a lot of data; this is the same reason why Gambas instance variables like MainForm are similar to pointers.

The second reason is when the called function should modify the variable we pass. For example, if we in Gambas wrote:

in C we would write:

The reason is that we want our INPUT command to fill our variable "a". In C, we must call input() and say to it where our variable is, in order to let it fill the variable. The ampersand "&" takes the address of the variable, and passes it to the function. The declaration of input() says "int *a", which states that "a" is not an integer, but a pointer to an integer, ie, the parameter says where to find the value, not the value itself.

Gambas Implementation Of Pointers

Gambas has the datatype Pointer, and a set of operations suitable for it. To use a pointer, a normal declaration is required like any other variable. Then, a value must be assigned to it. When using a normal variable, often you can assign a literal, for example you can write "a=3". With pointers, this is not advisable. A pointer gives access to any cell location in memory, but you should know in advance what location you are interested in, and that location must be the correct one for the intended purpose, otherwise Gambas or the operating system will get angry. This is much the same as saying that you can not write "MainForm = 3". You can write "MainForm = NEW()", or "MainForm = AnotherExistentForm", or "MainForm = NULL". So, a direct assignment to a pointer, just as a class instance variable like MainForm, will always be to NULL, to another pointer, or to a call to some function returning a pointer.

Sometimes an external function returns a pointer, and this pointer will be required in order to invoke subsequent calls to the external library. This case is much like creating a Form, and using its reference to operate on the form itself. In this case, the data behind (pointed by) the pointer is said to be "opaque": we can not see through something opaque, so we don't know, and we don't want to know. This is the simplest case; an example about this is the LDAP library. The first thing to do to interact with LDAP is to open a connection to a server. All the subsequent operations will be made on the connection created by a specific call. Things go like this:

As already seen, a LIBRARY is specified. Then, an EXTERN function is declared; this function is the one which must be called in order to do anything with ldap. The last two lines are the ones that, when executed, will open the connection and store its handle, or instance, for this connection. In this specific case, ldap_init() returns NULL if something goes wrong, so we can test for NULL to raise an error. Once obtained a handle to the connection, this handle must be specified on every subsequent call to the ldap library. For example, to delete an entry in the database, the following must be used:

Unfortunately, things are not always so simple. One of the reasons C uses a pointer, is to let the subroutine write some data in the location indicated by the calling parameters. Remaining in the initialization of a library, ALSA for example is different. To initiate a dialog with the ALSA sequencer, a handle for the sequencer is needed. The C declaration for this function is:

Hep! what is this "snd_seq_t **seqp"? We know that the asterisk is used to indicate a pointer - so what could mean a double asterisk? It's easy: a pointer to a pointer. This function snd_seq_open() uses the pointer notation to fill a value; this value is a pointer itself. Differently from the case of LDAP, where the function ldap_init() returns only one value, here this function returns two values. The return result of the function is an error code - all the ALSA functions use this scheme. A return result of zero means success. So to return more than a value, the function can only write some data to some location we specify, by using a pointer. The value it writes has type pointer, so the notation "double pointer" is used. So far so good. But can we translate this to Gambas? Yes and no. We need a pointer, and this is not a problem. Then we must take the address of this pointer, in order to obtain "a pointer to a pointer". Gambas3 can do that, Gambas2 can not.

Let see the simpler way, only available in Gambas3. The VarPtr() function returns the address of a variable or, in other words, a pointer to that variable - and its name says so: VAR-PTR, "variable pointer". In gambas3 we would write:

The EXTERN declaration says that snd_seq_open() expects a pointer, which is true: snd_seq_open() expects a pointer to a pointer, which is anyway a pointer. So we declare a variable Alsahandler as pointer, and pass its address using VarPtr() which returns a pointer to the variable.

In Gambas 2 this is not possible - we don't have VarPtr(). We must anyway declare a variable to hold the handle, like before, but then we can not get its address, or a pointer to it. We will attack the problem from another side. We need to find a location in memory to pass to ALSA and, after that, go to peek in that location. In Gambas 2 the only way is the Alloc() function. By using Alloc(), we reserve a piece of memory somewhere, and obtain its address. This address is what we need to pass to snd_seq_open(): a pointer contains an address. Well, can we start to write something? Yes:

When we want to open the connection and obtain a handler, we reserve some memory and pass its address, in order to have snd_seq_open() write useful data there. But then, how can we read that location to retrieve the handler? Here come the functionality of pointers in gambas. Pointers can work like streams - you can read from them and write to them. Actually, the memory of the computer is a file of memory cells, right? We can read a value from a pointer with:

At this point, we succeeded. It's a little like an Odissey, but it is worth! We only must release the memory we reserved with Alloc(), so the Odissey is not yet over. That memory has been used in a temporary way, and we could neglect it, but if this operation was made many many times in a program, the program continues to eat memory. Normally Gambas has automatic memory management, but in this case it cannot help because it doesn't know what we are doing with the memory, so we are responsible to free the memory when we are done with it:

There are other reasons to use a pointer. Take the declaration of getloadavg(), a nice function that tells us how much our CPU has been busy in the last minute:

This nice C language can even pass arrays to functions? Yes. And try to guess how it does it? Pointers again... In this case, the array passed to the function will be filled with one or more values, each signifying a different kind of load average; each value will be put in consecutive locations in the array. But C is not smart enough to know how big an array is, so the function can not know how many values to write. We have to tell the function, through the "nelem" parameter. To make it short, the correct declaration for this situation is this:

We need to pass a pointer, because the function getloadavg expects a pointer, even if this could not be obvious by looking at its declaration. The pointer must point to free ram, because the function will fill the memory pointed to by this pointer. Then, we will read the values and, lastly, we will fre the memory. Ax example usage is:

The subroutine is straightforward: we allocate 8 bytes because a gambas float is 8 bytes long. Then we call the getloadavg(), which will fill these 8 bytes. Whether the operation succeeds or not, we must free the allocated memory. But, If the operation succeeded, first we must read the memory. This is why we have two "free(p)" in the subroutine. A more elegant way could be to use a FINALLY clause, but this way we are more close to the C spirit...

getloadavg() returns the number of values read. Asking for only one value, it is legitimate to interpret a return result different from one as an error. If we asked for three, and only obtained two, we would have had a strange situation - something in between from a correct result and a failure. This and other funny things can be seen when trying to use some historical interfaces. For example, in some version of Unix there is not a clear method to read a file name. The function returns the number of character written, but no indication that the name is shorter than that. So you are only sure to have read the full name when you passed a buffer longer than the function result. But you have the function result after the call, not before! The typical usage is to take an arbitrary value, say 256, and do the first try. If it fails, you add another 256 bytes, and try again. And so on...

Back to our getloadavg(), anyway. We used Alloc(8) because a gambas float is 8 bytes long. And we used a gambas float in order to interface with a C double. But where is stated that a C double is 8 bytes long? In fact, there are out there machines where a double is 10 bytes. This is a serious issue, because the above subroutine will not work. We could allocate more memory, perhaps 64 bytes instead of 8: I am pretty sure that no computer exists which use more than 64 bytes for a floating point number. But anyway, trying to read a 8-bytes value out of a 64-bytes value would yeld a nonsense. Perhaps is better to let a program crash, instead of giving the impression that it works. One question could arise... How can a C program work on so many different architectures? The answer is the following: because a new, perhaps different architecture, must have an homogeneous set of kernel, include files and compiler. In a real C program you will never see a statement like "alloc(8)", but instead something like "alloc(sizeof(double))". The compiler knows the size of a double, and the keyword "sizeof" puts the knowledge of the compiler into the source program.

More On Pointers

Some better explanation is needed, at this point. The instruction READ #ret,... reads something from the location pointed to by the pointer "ret". It is important to stress once again that this kind of things must be designed carefully. Working badly with pointers is one of the most common cause of failure of C programs and, when using pointers, Gambas can do no differently. In this case is easy, because we made our job in a few lines in a row.

The semantics of the READ instruction in pointers resembles the one of stream, but with an important difference: while the stream is advanced automatically after a read or a write, the same operation on pointer does not. If our memory contained two variables to be read, one after the other, we had to advance the pointer by ourselves:

As you can see, it is possible to treat a pointer like an integer. Using this mechanism, one can walk forward and back in memory to emulate what in C are called "struct". A "C struct" is a group of heterogeneous variables put side by side, which then can be treated like a single variable. Its closest counterpart in Gambas is, again, a class. Structures are often referenced by a pointer, especially when they are to be passed to a function. We will see later an alternative method to implement this in gambas, but now we are talking about pointers, so we will finish this topic. The C language has also "unions", which are an unknown thing to Gambas, and therefore they must be emulated using pointers (not completely true). Unions are composed of two or more variables that share the same memory: writing to one variable modifies implicitly the others too: they are overlapped. The reason for this is to describe in a unique type different layouts. By combining struct and union, complex configurations can be generated, and these layouts are difficult not only to manage, but even to understand. To give an example, we will talk again about the ALSA sequencer. The sequencer works with events (mostly notes to be played) that have a time stamp to indicate "when" these events are to be played or carried out. This time stamp can be expressed in ticks, which is the traditional way related to the metronome. Ticks are normal integers. But ALSA goes further, and permits to use real-time time stamps, a much more precise indication, useful to synchronize music with other things (video, for example). This measurement is more precise, therefore it needs more memory to hold the bigger precision (two integers). So there are events having time specified with 4 bytes (an integer), and events having time specified with 8 bytes. They could have used simply two fields, respectively of 4 and 8 bytes, one after the other. But by using a union, they saved 4 bytes. The real memory reserved for time stamp is 8 byte, big enough to hold either of the two values, but at a logic level these two values are mutually exclusive. All this is handled automatically by the C compiler. When playing with unions in Gambas, we must do all this by ourself.

A Gambas Drum Machine

A concrete example about all we have seen until now is the ALSA library: a simple, very basic drum machine will be implemented in Gambas. First of all, what is a drum machine? It is a machine which emulates the combination of a drum player and a drum set. Many musicians use it, especially those who produce music all by themself. In the Gambas world, this is accomplished by using ALSA. ALSA is the Advanced Linux Sound Architecture, and its aim is to offer a complete set of functions to produce sounds and, hence, music. From the point of view of a computer, generic sound and music are two different things. If you play an MP3 file, ALSA will move the speakers as directed by the MP3 data, without knowing or analyzing anything. We are interested in another kind of interface - the sequencer interface. A sequencer copes with "events", which are "played" at the right times, using suitable parameters (or properties) for the event. If we think at a piano player, we can see that he presses his keys, one or more together, at different moments. Simply, every keypress it's an event. The three most important things when pressing a key on a piano are: 1) when; 2) which key; 3) how strong. If you want to play two notes at the same time, you create two events having the same "timestamp". If you want to play a chord, you create three events having the same time, three different notes, and (probably) the same strongness. Then, you feed these events to the sequencer, and it will send them to something else which will produce the sounds. The sequencer does not care about producing sound: this can be produced by some software, or be outputted by a MIDI interface to some external musical instrument. If, instead of a piano, you say "I want trumpets", the three notes will be played by three trumpets. This interface does not specify "how long the notes play": they will sound until another event will say to stop. So a single note is actually done with two events: a NOTE-ON and a NOTE-OFF. In the case of drums, a note identifies a different piece of percussion: bass drum, snare drum, cymbals, maracas, bells, even whistles and much more.

Music and computers have much in common. For example, a typical musical measure is divided in 4 quarters. Is the number four uncommon in computers? Keys on piano are numbered, and the strongness ("velocity") of a keypress can be expressed by a number, as well as the duration of a note. There are other values involved, for example the force a flute player uses to blow in his instrument (after the note has been started), but we will not go so deep. Only let me say that a good sequencer, combined with good hardware, can simulate surprisingly well an entire orchestra.

The simple drum machine has a grid on the screen, and every cell of the grid represents a note: its row number specifies the note to be played (on a drum machine, different notes correspond to different pieces, or instruments). The column of a cell represents the time when the note will be played. The grid contains two measures which are played over and over - this is enough to construct a normal rythm. Every measure is divided in 4 quarters, and every quarter is divided further in four 16th's. The top row is a visual ruler, and the leftmost column is used to hear an instrument. Clicking in a cell toggles an "o" marker; to play the pattern click the button "Play grid". Other buttons produce some other sound, just to show simpler things like chords, legato's, arpeggio.

To have the program produce sounds, the correct client/device and port (ALSA terminology) must be written in the first two lines of FMain.class, and depends on the hardware installed. Issuing an "aconnect -ol" in a terminal shows the suitable devices. If a software synthetizer is present, like Timidity, probably it will show as "client 128". The MIDI out device could be number 16. The port number can probably be always 0. Another way to find out the correct numbers is to use an already working sequencer or MIDI player, like Kmidi, and peek at its midi configuration.

The main reason to analyze this program is to look at a complete interface with an external library. Most of the issues have been presented already, but an important part not yet covered is how to cope with C structures using pointers. Instead of using pointers, an alternative way is to use declare variables in a class, and then pass an instance of that class to an external function; this is not covered here: the method would be better and clearer, but the pointers are more versatile. A yet better approach is possible in gambas3, which has native structures.

Because the program is very ALSA-specific, we will skip everything but the "event structure". Once all the things required by ALSA are done (opening ALSA, creating queues, ports, starting them and so on), only remains to construct events and send them to ALSA, which will play them at the correct time. An event is defined by ALSA like this:

The first lines, before "union", are common to every kind of events; in fact, they contain the event "type", some "flags", a "tag", the "queue" where to enqueue the event, the "source" (who created the event?) and the "dest" (to whom send this event?). Let's look at the firt line: "snd_seq_event_type_t type". The field is named "type", and its type is "snd_seq_event_type_t type". So we must inspect the documentation to find out how this type is made. We find:

The line above says that "snd_seq_event_type_t" is an alias for "unsigned char". An unsigned char is a byte. The next three fields in the struct are flags tag and queue, all of type unsigned char, hence byte. Then the timestamp "time" is declared as "snd_seq_timestamp_t"; searching again for declaration, we find that it is a union containing either a midi tick (an unsigned int) or a struct which is composed by two unsigned int. The net result is that the length of this field is 2 unsigned ints, or 8 bytes on 32-bit systems. The first part of an event is composed, by our point of view, of the following fields:

If we want to fill the field "tick", we must point a pointer to the beginning of the event memory, then advance the pointer by 4 bytes, then write to the pointer the intended value (an integer).

The CAlsa class allocates memory for just an event:

and then manipulates this memory over and over before passing the event to ALSA. The subroutine prepareev() clears the event and fills the common part. Here is its declaration:

The parameters of the function reflect what we are interested in - for example, we are not interested in the "tag" field, so we don't pass a value for it. The first step is to clear the event, to make sure that no unwanted data is there from before:

The pointer "p" is pointed to the beginning of the event with "p = ev". To be sure to write a single byte instead of two or four in every iteration, we use a temporary variable declared as byte: b=0. Using a for-next, a stream of zeroes is written out. Instead of using a temporary variable, we could also have used the instruction "WRITE #p, chr(0), 1". This latter instruction is a special form available for strings, that specifies how many bytes from the string have to be written out. Withouth specifying the number of bytes, ie using "WRITE #p, astring", Gambas writes out the whole string, but precedes it with a binary representation of the string length; probably not what one desires in this situation.

A better algorithm to clear the event would be to write 4 bytes at a time, and reduce the loop to 1/4. A still better way would be to simply rewrite fields we are interested in, and clear only the fields we know that are dirty.

After clearing the event, we start to fill the relevant fields. Again, we point our pointer "p" to the correct place (we moved it, remember?), write a value, and move the pointer afterward:

The rest of the routine is a repetition of what we have already seen. At the point of writing the timestamp, which is a C union, there is the following code:

This single instruction writes the first of the two integers of "snd_seq_timestamp_t time". Then:

Well, these three lines are not needed. We cleared all the memory before, so there is no need to set any field to zero. But we should anyway move the pointer. The two previous blocks of code could be as follows:

The subroutine prepareev() is called by noteon() and noteoff(), which then continue to fill the event with own data. The noteon() subroutine is this:

The subroutine accepts a timestamp "ts", which says -when- to play the note, and refers to the time when the queue was started. A timestamp of zero, or anyway a value less than the current time of the queue, is played immediately. If the timestamp is greater than the current queue time, then the event will be played in the future, when the queue will reach the correct time. But ALSA can also use relative timestamps, setting a flag in the event. The gambas CAlsa class uses this possibility embedding it in the timestamp directly. If the ts parameter to prepareev() is set to a negative value, then the routine inverts its sign and sets the relative flag. In the main program, the routine btArpeggio_Click() produces three notes in succession by using this possibility:

To subroutine starts the first note at time 0, which means "now". After 100 ticks, the note is shut off, and a new one is started (timestamp=-100 means "100 ticks after 0 -- 100 ticks after now"). And so on for the next notes.

Given that playing notes is the most common task, and that to play a note one must create a note-on and a note-off, the CAlsa class implements a routine to do that with a single call. The following routine:

uses a single call to playnote(), which in turn generates internally two events.

Lastly, to be precise, here is an explanation of the drum machine algorithm.

The whole thing is to produce a stream of events, which will be played later. We must prepare a bunch of events in advance, so the hardware has data to work on. But a drum machine can be kept on for a long time, and we can not buffer all the needed events - we must supply a certain number of events ahead of time, not too much and not too little. The internal "pointer" of the drum machine always is a musical measure ahead of the sound we are hearing. We can not predict reliably "when" new data will be needed, because the sequencer consumes the events basing itself on a timer that can be different from our. Without a feedback from the sequencer, it is nearly impossible to keep in sync. The problem is even more difficult if we want to have a visual feedback on what the sequencer is playing in a given moment. This is solved by adding, in the event stream, some event whose purpose is not to make sounds, but to come back to us: an echo. The sequencer receives this added events, and sends them back to us at the correct moment. When we see this echoes, we know at which point the sequencer is. The gambas drum machine sends an echo for every quarter, and uses this information, when it comes back, to give a visual feedback. These echo events can contain some user data - that datum is used to distinguish them; in the program, this can reveal a start of a measure, and in that condition a new measure is loaded (buffered in advance) to the sequencer.

The problem, in this program, is that the normal ALSA interface does not provide a callback to signal when an event is ready to be read. This is simulated by the CAlsa class, which raises a Gambas event when an ALSA event of type "echo" has been read, but the CAlsa class itself uses polling to interrogate ALSA. The frequency of this polling can be varied using the slider "poll freq" in the main program. Setting this frequency to low values should show an imprecise visual feedback of the drum machine, but the precision of the music should be unaffected. In reality, at least on my slow machine, it seems that low poll rates work better than high ones. This can be explained, perhaps, considering that doing the polling is a cost in term of CPU: by polling too often, my machine uses too much CPU, loosing its readyness.