Dynamic loading question (somewhat long winded :)

[chrislee@lavash.ius.cs.cmu.edu]

First of all --- thanks to all the people working on Guile!  I find it
extremely useful in my work.  I use it for both scripting applications
for which most people would typically use perl (e.g. responding to CGI
queries and generating html forms for a web-based interactive database
application), and for tasks more typically associated with Matlab
(e.g. as an interface to blas/lapack routines for numerical work).

The one thing I still find frustrating about Guile is trying to
figure-out how to dynamically load modules based on
C-code---e.g. using the module system to dynamically link shared
libraries to the interpreter as needed.  I was able to do this for a
while (in Debian GNU/Linux 2.0.33 and Irix 6.3), but my ability to do
it disappeared as I upgraded Guile and Libtool over the past few
months.  I really liked doing this dynamic linking, as it allowed me
to avoid recompiling an interpreter executable every time I wrote a
new module or decided to use a new combination of modules (e.g. "maybe
I'll just use the guile-gtk module to get input from the user here on
how to train my neural networks --- whoops I have to build a new
executable linking the set of all necessary modules").

I'll write my best guesses (as a Guile user) on the situation below,
and I hope someone in the know can confirm the problem or set me
straight, and suggest anything I can do to help....  I'll admit now
that I'm a dynamic-linking newbie and don't understand all the issues
involved.


My best guess for the situation based on personal experience and what
I read on the Libtool web page (this mailing list has been pretty
quiet on the subject for a while), is that dynamic loading of modules
is problematic at the moment from a portability standpoint, and that
the Guile developers are waiting for Libtool to better deal with the
problem before worrying about it themselves.

The problems seem to have to do with library dependencies.  For
instance, I have a module which exports smob-types for
Fortran-compatible matrix and vector representations, and an interface
for calling blas functions.  This module is linked to the appropriate
shared library on the system containing the blas implementation.  My
understanding is that not all systems are smart enough to automatically
dynamically link the blas library to the process when the module
requiring is loaded (or that 2nd level dependencies are handled
incorrectly on some systems).

Another problem for some systems seems to be (at least as far as I
guess based on difficulties I have had) interdependencies between
modules.  For instance, I have a module for neural-network code which
exports functions that take vectors and matrices as inputs.  The code
which checks that these functions are called with the correct
parameter types and converts them from SCM/smob-form to the internal,
C-based representations requires referring to the smob-type which is
in the matrix/vector module/shared-library.  Thus, the neural-network
module/library must be linked against the matrix/vector library.  When
loading a set of modules with a web of interdependencies of this type,
I sometimes found that what looked like a matrix smob to one module
didn't look like one to another module.  My best guess about the
problem is that either there may have been more than one copy of the
matrix module in memory (due to multiple dependency paths wanting the
module?) or that the matrix smob type had been initialized more than
once during the run, creating different internal representations for
the same kind of object.


At some point, the dynamic module-loading code of Guile was changed so
that Libtool lib*.la files were checked-for first, and if found, the
dlname='*' line was parsed to get the name of the file to dlopen.
Once I figured-this out, I was happy to realize that a copy of the
lib*.la file could be installed in a module-centric path location just
like a Scheme module file.  That way, the actual shared lib*.so* files
could be installed in a single directory in the LD_LIBRARY_PATH
without having to put them in the same module-path (i.e. before, I had
named all my library/modules named as '(primitives <lib-name>) so that
I wouldn't have to add a bunch of new directories to my
LD_LIBRARY_PATH environment variable).

Is this Libtool module file planned as a basis for supporting modules
in a portable way?  Is guile going to parse the dependency_libs='*'
line to manually dlopen any unlinked shared libraries in the dependency
list?

Thanks,
 -Chris