Dynamically loading C/RTL code

Object Icon incorporates some enhancements to Icon’s dynamic loading facility which allow dynamically loaded functions to be written in RTL rather than plain C. This means that dynamically loaded code can take the same form as builtin code, and can do the same things.

A sample RTL file

Here is a not-very-useful function to generate the bit positions of ones in an integer (for example 6 is 110, so generates 2, 3).

function bits(v)
    if !cnv:C_integer(v) then
       runerr(101, v)
    body {
       int i = 1;
       while (v) {
           if (v & 1)
               suspend C_integer i;
           v /= 2;
           ++i;
       }
       fail;
    }
end

To create a dynamic library to use this function we would firstly place it in a file with the suffix “.r”, say testlib.r.

Compilation

Next, create a Makefile to build the library.

include $(OI_HOME)/Makedefs

all:    testlib.so prog

clean :
    rm -f *.c *.u *.o *.so prog

prog :  prog.icn
    oit -s prog.icn

testlib.o : testlib.r
    rtt testlib.r
    $(CC) $(CPPFLAGS) $(CFLAGS) $(DYNAMIC_LIB_CFLAGS) -c testlib.c -o testlib.o

testlib.so : testlib.o
    $(CC) $(DYNAMIC_LIB_LDFLAGS) -o testlib.so testlib.o

Notes :-

• The Makedefs file defines various variables, in particular the flags for C compilation.
• The flags to the $(CC) command may need to be tweaked.

Compiling on Windows

The above relates to Unix-like platforms. For Windows, the commands are of course slightly different. If Cygwin is being used, the following Makefile could be used. Note that rtt is given the -x option, which is explained below.

include $(OI_HOME)/Makedefs

all:    testlib.dll prog.exe

clean :
    rm -f *.c *.u *.obj *.dll prog.exe

prog.exe :  prog.icn
    oit -s prog.icn

testlib.c: testlib.r
    rtt -x testlib.r

testlib.dll : testlib.obj
    cl /LD testlib.obj

Loading the library

Here is a sample program to load the library and run the bits() function. Save this as prog.icn.

import io, lang

procedure main()
   local p
   p := Proc.load("./testlib" || Files.LIB_SUFFIX, "bits") | stop(&why)
   write("okay, loaded ",image(p))
   every write(p(12345))
end

When compiled and run, the following output should result :-

okay, loaded function bits
1
4
5
6
13
14

Note that the library name in the first parameter of loadfunc is passed unaltered to the underlying system’s dynamic load function (on Unix, dlopen). This means that the library must either be an absolute path, relative to the current directory, or in one of the places that the system normally looks for shared libraries.

There are some functions available to help search for files on paths. In particular see io.Files.path_find() and io.Files.find_native_lib(). This latter function is used to load the shared libraries included in the distribution, using the path given by the environment variable OI_NATIVE.

Class.load_library()

This function can be used to define several native methods at once in a class, using a shared library. It should be called from the class’s static init method, with the shared library as a parameter. It goes through each native method in the class, looking for a matching function defined in the shared library. Any not found are just ignored. This means a class with numerous native methods can resolve them all at once with a single convenient call.

For example, if we replace testlib.r with the following :-

function Test_one()
    body {
       printf("test one\n");
       fail;
    }
end

function Test_two()
    body {
       printf("test two\n");
       fail;
    }
end

and prog.icn with :-

import lang

class Test()
   public native one()
   public native two()

   private static init()
      Class.load_library("./testlib" || Files.LIB_SUFFIX) | stop(&why)
   end
end

procedure main()
   local o
   o := Test()
   o.one()
   o.two()
end

and recompile using the Makefile already given above, then the output from prog will be :-

test one
test two

External headers and symbols

It is possible that the RTL code in a dynamic library will need to refer to another existing library and its associated C header files. For example, the mysql library needs to reference "mysql.h" and link to the mysqlclient library. These external header files are normally not processed by RTT - rather they are passed through RTT to be processed by the C compiler. For example

#passthru #include "mysql.h"

This raises a problem of how rtt will recognise symbol and types defined in the header concerned, which are then used in the RTL source. For example mysql.h defines the type MYSQL_RES, and this is used several times in mysql.r. For code in the interpreter, a separate header file, grttin.h, is processed specially by rtt in order to allow such symbols to be declared with dummy types, typically as typedef int. This allows them to pass through RTT without error. For an dynamic library we naturally don’t want to edit grttin.h, so a new option, -h specifies an extra header file to process immediately after grttin.h. Here we can provide the necessary dummy definitions of symbols like MYSQL_RES. The header for the mysql library looks like this :-

typedef int MYSQL, MYSQL_FIELD, my_bool, MYSQL_RES,
    MYSQL_FIELD_OFFSET, MYSQL_ROW, MYSQL_ROW_OFFSET, my_ulonglong;

and is processed (from a Makefile like the one above) with the command :-

$(RTT) -h gmysql.h mysql.r

Another way of passing symbols like MYSQL_RES through rtt unchanged is via the -t option to rtt :-

$(RTT) -t MYSQL_RES -t MYSQL_FIELD .... mysql.r

The indicated symbols then pass through to the C code unchanged.

Accessing symbols from the library

The above simple bits function used the conversion cnv:C_integer to check and convert the type of its argument. This syntax is translated by rtt into the C call cnv_c_int, a function defined in cnv.r and part of the main oix executable. On modern Unix systems (including Linux), this doesn’t present a problem, because the loader will be be able to link a shared library’s undefined symbols at load time to the parent program.

However, this doesn’t apply to all systems, and in particular a Windows DLL file cannot access its parent’s symbols in this way. Therefore another method has to be used to enable the library to call into the main program.

The method used is to pass a structure containing pointers to variables, constants and functions of the main program, into the library when it is first loaded. This structure can then be used to conveniently access the main program. There should be no need to access the pointer structure directly, since a file of macro definitions is included by the code generated by rtt in order to map the symbols to their pointer equivalents. For example cnv_c_int mentioned above is defined as :-

#define cnv_c_int (*(imported->cnv_c_int))

The full list of macro definitions is in the file base/h/imported.h.

The variable imported points to the structure passed from the main program.

To tell rtt to generate a C file which uses the above scheme for accessing the main program’s symbols, just pass the -x option to rtt. This will define the imported variable, together with an exported function to set it.

Symbol clashes

Do be aware that occasionally preprocessor definitions of this sort can cause trouble. For example, like cnv_c_int, the symbol why is also defined in imported.h :-

#define why (*(imported->why))

Imagine that a library wishes to include some third-party header as follows :-

/* trouble.h */
struct bother {
    int why;
};

As described above, this would be included in a .r file with the line :-

#passthru #include "trouble.h"

The problem is that the C preprocessor will replace the why in the structure definition with “(*(imported->why))”, causing a syntax error.

If we don’t need to use the imported why in our .r file, then we can just undefine it :-

#passthru #undef why
#passthru #include "trouble.h"

However, if we do wish to use why, then we must use something more verbose :-

#passthru #undef why
#passthru #define why _why
#passthru #include "trouble.h"
#passthru #undef why
#passthru #define why (*(imported->why))

and then refer to the field structure as follows :-

struct bother b;
b._why = 100;

Another option is to put the code using the problem library (and its include file) into a separate C source file, which is then called from the .r file.

Multiple source files

If you wish to create a single library file from several source files, then translate all but one of the files using the -y option with rtt. This ensures that certain variables generated by rtt (such as imported described above), are defined in just one file (the one translated without -y).