Lab scripting language?

Thanks for the responses, it's been interesting learning a little about
scripting languages with external C interfaces. We were able to get
a tentative solution to our problem, but not instantly, here are some
notes which might save someone some time. Our answer for the
moment is Lua, Tolua, and Swig.

As a post mentioned, there are dozens of scripting languages and
interpreters, many of which have some kind of external C facility.
There are endless theological discussions about which is best, and
now I have a set of brand-new biases. Based on the advice of another
post ("run, don't walk"!), we looked into Lua. It's an elegantly slim
download, with a remarkable design, and with some hope we looked
into the external C call facility. However it requires C calls of a
specific form, to enable communication with Lua's stack. This is
infeasible for us, as we have a lot of pre-existing routines, graphics,
altivec modules, etc. that we don't want to change, as well as
libraries from third-party vendors for camera control, etc. that we
can't change.

Every language we looked at had some rigidity like that. Maybe we were
asking for too much, no portable scripting language can exist which can
directly call out to raw C routines, due to differences in C calling
conventions between different hardware and platforms. But we thought
something must be possible, as gdb almost does what we want. In gdb
you can "print" some subroutine, and it will be executed in an
interpreted fashion. But there is no built-in looping facility for
example.

Pursuing the C interface issue with lua, we finally found Tolua, and
Swig, which solve the problem in the following way.
Say you have a library of routines, you don't have the source, but you
do have the header files describing what the routines expect for
arguments and return values. You submit the header files as input to
a program which produces, as its output, source code for another
C routine. This code is in the correct format required by Lua,
and it deals with the stack, and does variable translations, and then
calls your library routine. You compile and link in this machine-
generated code, Lua talks to it, and both Lua and the library routine
are happy. There is a little per-call overhead, but probably
negligible in most cases.

As a test I was able to link Lua into a mixed graphics-intensive
program, which writes directly to the frame buffer with SDL,
and links with the carbon, cocoa, and quicktime frameworks.
It worked great, I could stop and start stuff at will, and there
was no perceptible speed penalty. So I guess I'm a convert to
the Cult of Lua, avoid all languages whose names begin with
the letter "p". Incidently the speed penalty for performing
numerics in the interpreter itself isn't too bad, straight C is
maybe 10 times faster for doing typical multiply/accumulate stuff.

I used Tolua to do the interface to C, as it is again slim, you can
read the machine-generated code, and I got it to work. But a friend
of mine also got Swig to work, it has a large user base, and can
produce glue code for a number of scripting languages. He was
able to dynamically link C routines into Lua, even while the
interpreter was running, using code generated by Swig. This,
and Tolua, worked so easily in Lua that we weren't motivated
to try the other interpreters that Swig can service.

We looked a bit at other options, such as C interpreters, as it would
be nice to have C syntax, but without the semi-colons, and we
thought a C interpreter might interact more naturally with C libraries.
But both Cint and ch apparently have their own nontrivial interface
protocols for linking in external C code. Also, they have their own
header files, and we weren't able to get ioctl calls to work, which
we need to run serial ports. Yes, plenty of brand-new equipment
uses good old RS232.

LabView is a possibility, apparently you can link in external C
modules, and there is claimed OSX support. Someone has
managed to call from LabView out to a Lua routine, which then
calls back into LabView. Why, I don't know, but if you can do that,
you probably can do anything. But LabView is expensive, proprietary,
and closed-source, with a top-heavy GUI and an uncertain future.
We already have one of those, it's called a Mac (sorry)!

Thanks for any further suggestions, and sorry for the length of this
post.

rob

ps: about the Tcl post. If I'd seen Tcl first, I'd probably be saying
it was the greatest thing since cheese slices individually wrapped
in plastic, instead of Lua. But Lua has a two-day head start here,
and that may be insurmountable. I like that Lua gave us a prompt
immediately, which is how we'll be using it in the lab. I liked that
Tcl gives you pwd and cd right out of the box. I'm sure each can do
the other in a few lines of code.
 >> Stay informed about: Lab scripting language? 

Rob Shaw wrote:

>LabView is a possibility, apparently you can link in external C
>modules, and there is claimed OSX support. Someone has
>managed to call from LabView out to a Lua routine, which then
>calls back into LabView. Why, I don't know, but if you can do that,
>you probably can do anything. But LabView is expensive, proprietary,
>and closed-source, with a top-heavy GUI and an uncertain future.
>We already have one of those, it's called a Mac (sorry)!

Although it is expensive, LabVIEW pays for itself with greatly reduced
development times, including freely available instrument libraries for
a large number of GPIB and serial devices. These libraries alone can
save you tons of development time. LabVIEW's support for OSX isn't
claimed, it is superb and has been available since 10.1.x. LabVIEW was
originally developed on the Macintosh back in the days when the only
way to do data acquisition with PCs was via command lines. The Windows
version didn't appear until Windows 95. The virtual instrument files
are extremely portable between Windows and OSX, and the Full
Development System can compile standalone applications. As their
flagship software product, National Instruments' support for LabVIEW is
quite good.

I'm not sure what you mean by "top-heavy GUI", but the front panel
objects are linked with the graphical programming language in an
intimate way. This intimacy is part of what makes LabVIEW so easy to
program with, and for measurement or automation operators to use.

As for external code, these are done with what are called Code
Interface Nodes (CIN). If the object code follows C linkage, it can be
used in LabVIEW. There are full development headers supplied for CINs.
 >> Stay informed about: Lab scripting language? 

I apologize about my flip comments on LabView, I was speaking out of
ignorance, yet again. They were based on an uninformed emotional
reaction to the marketing web pages, by someone who disapproves
of anything fancier than a command line.
Maybe we should take another look, but let me explain why large
LabView-type applications are probably not the best for us.

We are going to look through a microscope, do image processing,
and depending on what we see, move the stage, or adjust other
parameters, in real time. One route would be to buy packages to do
image tracking, which exist, at least for windows, and perhaps
bolt them onto LabView. We would spend lots of money on software,
but probably be up and running shortly. But the first time we try
and do something which isn't in the pull-down menu, things will get
a lot harder, if not impossible. We expect to be pushing limits of
image processing and device control, and we will have to know
in detail what the code is doing. This might require re-inventing
a few wheels, but we'll be able to quickly change tires.

Then there's the whole Mac issue. You might ask, and indeed I ask
myself often, if we're writing much of our own code, why aren't we
on a linux box, where the source is open right down to the driver
level, and you can get lots of help? Or why not a windows box,
where at least you have the option of chickening out and getting
a software package that might work? One answer is habit,
and an irrational attachment to Macs.

But another is that altivec PPC's are the fastest in their class,
for doing single-precision floating point, by a wide margin, despite
apple's current marketing claims. Just check the FFT benchmarks.
Trying to get close to the hardware for ultimate video performance
can be very frustrating on a mac, things are undocumented, and
subject to change. But I've fought some of those battles, and the
altivec battle, and the speed is what we need, so that argues for mac.

So an extreme statement of the options would be, use canned software
on a windows box which will run immediately, or, write loads of
custom software, orchestrated by a Brazilian dynamic language,
running on a truly fast box, which has been declared obsolete by its
manufacturer.
The choice is clear!

rob
 >> Stay informed about: Lab scripting language? 

In article ,
"Rob Shaw" wrote:

> Then there's the whole Mac issue. You might ask, and indeed I ask
> myself often, if we're writing much of our own code, why aren't we
> on a linux box, where the source is open right down to the driver
> level, and you can get lots of help?

I have found that you can get quite a bit of help on the various Apple
mailing lists for Apple-specific technology, and quite a bit on mailing
lists for open source tech. (For me, most of my open source code comes
from Apache.)

LabView, though, does do a lot of things, and can greatly simplify your
life. The drivers for various bits of hardware are very keen, IME. A
number of our biotech clients swear by it for equipment automation.

> But another is that altivec PPC's are the fastest in their class,
> for doing single-precision floating point, by a wide margin, despite
> apple's current marketing claims. Just check the FFT benchmarks.
> Trying to get close to the hardware for ultimate video performance
> can be very frustrating on a mac, things are undocumented, and
> subject to change. But I've fought some of those battles, and the
> altivec battle, and the speed is what we need, so that argues for mac.

Apple's marketing claims do match your experience. Now that they are on
Intel, you will note that they are discussing spec, esp. SPECint, which
Intel has classically been good at. If Intel gets the speed up enough,
they will win the fp battle too, but it is not clear how well they will
do at that.

For me, having dual core laptops is just plain keen.

> So an extreme statement of the options would be, use canned software
> on a windows box which will run immediately, or, write loads of
> custom software, orchestrated by a Brazilian dynamic language,
> running on a truly fast box, which has been declared obsolete by its
> manufacturer.

There is option three. Use a mix of canned and open source software as
needed to solve your problem, and use the platform that produces the
most happiness and productivity.

For me, that is a Mac, as I really want configure/make to work, but
Linux does not have the productivity apps I want. It may not _stay_
true, but it is certainly true for now.

Scott

--
Scott Ellsworth
scott.DeleteThis@alodar.nospam.com
Java and database consulting for the life sciences
 >> Stay informed about: Lab scripting language? 

Rob Shaw wrote:

> We are going to look through a microscope, do image processing,
> and depending on what we see, move the stage, or adjust other
> parameters, in real time.

Sounds like you want to do automated cytology, as in PAP smears.
Good luck! Been there, got the T-shirt. Technology can't keep up with
falling market barriers and influx of cheap labor (or does your PPC +
software + peripherals cover its costs for little more than the
equivalent of a bowl of rice/day?)

--
madiba
 >> Stay informed about: Lab scripting language? 

On 20/08/2006, madiba wrote in message
:

> Rob Shaw wrote:
>
> > We are going to look through a microscope, do image processing,
> > and depending on what we see, move the stage, or adjust other
> > parameters, in real time.

What you want is probably MatLab. It can do all the image-processing you
want, and it can be used to control an IO card which could make the
adjustments to the microscope you need. You will need at least one
electronics guy to hook up the microscope and one programmer to program
MatLab.

> Sounds like you want to do automated cytology, as in PAP smears.
> Good luck! Been there, got the T-shirt. Technology can't keep up with
> falling market barriers and influx of cheap labor (or does your PPC +
> software + peripherals cover its costs for little more than the
> equivalent of a bowl of rice/day?)

I think madiba is probably right for a large-scale project with many
performances of the same task. You'd need a real microscope+computer
setup to do this. It's cheaper to offshore it. However, if you can
develop your idea to the point where the hardware becomes cheaper, it
might be a player.

Simon.
--
http://www.hearsay.demon.co.uk
 >> Stay informed about: Lab scripting language? 

Thanks for the advice. It's certainly true that it's tough to beat the
human eye in image processing. And as long as there are more
people than jobs in the third world, robotics in general is not going
to be cost-effective.

But we're doing experiments that generate dynamical patterns,
it's not a production thing, at least not yet. I'm the electronics
guy, and help with the image processing software, we have a
more-or-less working system.

We ended up using lua as a scripting language, though still we mostly
write in C, it's maybe just as convenient with today's fast compilers.
If I were to do it again, I might look at something called "ipython",
which supposedly has a very nice shell interface, logging facilities,
etc. Lua doesn't have this, but it's so small that you can add stuff
in easily, and the user group is very helpful. It's been a time sink,
though, the jury is still out.

A lot of people swear by Matlab, but we're more in the
reinvent-the-wheel school of thought. One thing we've done is
interface a wireless gaming joystick to our setup, so we can
focus the scope, run the stage, change filters, take pictures,
and run other equipment, all from one unit. The mac
we use works well for us, it's a 1.4 GHz dual MMD, one of the last
with a standard PCI slot. I believe the new ones exclusively use
"PCI express", cards for that standard are vvvery expensive,
if they exist at all. Maybe a linux box makes more sense if
you're starting a setup these days, but we had the mac hardware.

Of course, the real test if is we can do any good science.

Thanks again,

rob

Simon Slavin wrote:
> On 20/08/2006, madiba wrote in message
> :
>
> > Rob Shaw wrote:
> >
> > > We are going to look through a microscope, do image processing,
> > > and depending on what we see, move the stage, or adjust other
> > > parameters, in real time.
>
> What you want is probably MatLab. It can do all the image-processing you
> want, and it can be used to control an IO card which could make the
> adjustments to the microscope you need. You will need at least one
> electronics guy to hook up the microscope and one programmer to program
> MatLab.
>
> > Sounds like you want to do automated cytology, as in PAP smears.
> > Good luck! Been there, got the T-shirt. Technology can't keep up with
> > falling market barriers and influx of cheap labor (or does your PPC +
> > software + peripherals cover its costs for little more than the
> > equivalent of a bowl of rice/day?)
>
> I think madiba is probably right for a large-scale project with many
> performances of the same task. You'd need a real microscope+computer
> setup to do this. It's cheaper to offshore it. However, if you can
> develop your idea to the point where the hardware becomes cheaper, it
> might be a player.
>
> Simon.
> --
> http://www.hearsay.demon.co.uk
 >> Stay informed about: Lab scripting language? 

Give this a shot:

http://homepages.cwi.nl/~jack/macpython/

You may also want to think about Pooch in order to take advantage of
distributed computing and OS X's built in xgrid.


In article ,
"Rob Shaw" wrote:

> Thanks for the advice. It's certainly true that it's tough to beat the
> human eye in image processing. And as long as there are more
> people than jobs in the third world, robotics in general is not going
> to be cost-effective.
>
> But we're doing experiments that generate dynamical patterns,
> it's not a production thing, at least not yet. I'm the electronics
> guy, and help with the image processing software, we have a
> more-or-less working system.
>
> We ended up using lua as a scripting language, though still we mostly
> write in C, it's maybe just as convenient with today's fast compilers.
> If I were to do it again, I might look at something called "ipython",
> which supposedly has a very nice shell interface, logging facilities,
> etc. Lua doesn't have this, but it's so small that you can add stuff
> in easily, and the user group is very helpful. It's been a time sink,
> though, the jury is still out.
>
> A lot of people swear by Matlab, but we're more in the
> reinvent-the-wheel school of thought. One thing we've done is
> interface a wireless gaming joystick to our setup, so we can
> focus the scope, run the stage, change filters, take pictures,
> and run other equipment, all from one unit. The mac
> we use works well for us, it's a 1.4 GHz dual MMD, one of the last
> with a standard PCI slot. I believe the new ones exclusively use
> "PCI express", cards for that standard are vvvery expensive,
> if they exist at all. Maybe a linux box makes more sense if
> you're starting a setup these days, but we had the mac hardware.
>
> Of course, the real test if is we can do any good science.
>
> Thanks again,
>
> rob
>
> Simon Slavin wrote:
> > On 20/08/2006, madiba wrote in message
> > :
> >
> > > Rob Shaw wrote:
> > >
> > > > We are going to look through a microscope, do image processing,
> > > > and depending on what we see, move the stage, or adjust other
> > > > parameters, in real time.
> >
> > What you want is probably MatLab. It can do all the image-processing you
> > want, and it can be used to control an IO card which could make the
> > adjustments to the microscope you need. You will need at least one
> > electronics guy to hook up the microscope and one programmer to program
> > MatLab.
> >
> > > Sounds like you want to do automated cytology, as in PAP smears.
> > > Good luck! Been there, got the T-shirt. Technology can't keep up with
> > > falling market barriers and influx of cheap labor (or does your PPC +
> > > software + peripherals cover its costs for little more than the
> > > equivalent of a bowl of rice/day?)
> >
> > I think madiba is probably right for a large-scale project with many
> > performances of the same task. You'd need a real microscope+computer
> > setup to do this. It's cheaper to offshore it. However, if you can
> > develop your idea to the point where the hardware becomes cheaper, it
> > might be a player.
> >
> > Simon.
> > --
> > http://www.hearsay.demon.co.uk
 >> Stay informed about: Lab scripting language? 

Rob Shaw wrote:
> A lot of people swear by Matlab...

You might consider the OCaml programming language. It is fast, free and
natively available on OSX.

--
Dr Jon D Harrop, Flying Frog Consultancy
OCaml for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists/?usenet
 >> Stay informed about: Lab scripting language? 

On 03/06/2007, Jon Harrop wrote in message <4662f383$0$8738$ed2619ec@ptn-
nntp-reader02.plus.net>:

> Rob Shaw wrote:
> > A lot of people swear by Matlab...
>
> You might consider the OCaml programming language. It is fast, free and
> natively available on OSX.

Not to mention 'R'.

There are lots of Lab languages for the Mac, for lots of different types
of lab. The question is what you want done: psychics ? lots of
complicated maths ? controlling lab equipment ? Tell us what you're
doing and we'll suggest something.

Simon.
--
http://www.hearsay.demon.co.uk
 >> Stay informed about: Lab scripting language? 

In article ,
Simon Slavin
wrote:

> On 03/06/2007, Jon Harrop wrote in message <4662f383$0$8738$ed2619ec@ptn-
> nntp-reader02.plus.net>:
>
> > Rob Shaw wrote:
> > > A lot of people swear by Matlab...
> >
> > You might consider the OCaml programming language. It is fast, free and
> > natively available on OSX.
>
> Not to mention 'R'.
>
> There are lots of Lab languages for the Mac, for lots of different types
> of lab. The question is what you want done: psychics ? lots of
^^^^^^^^
- is this a new use for a random number generator? "I see in your
future the need for a software upgrade..."
 >> Stay informed about: Lab scripting language?