Hi,

I have recently wrapped solr inside jetty with JCC (we need to access
very big result sets quickly, via JNI, but also keep solr running as
normal) and was wondering what strategies do you guys use to speak
*from inside* Java towards the Python end.

So far, I was able to think about these:

- raise exceptions in java and catch in python (I think I have seen
this in some posts from Bill Jansen)
- communicate via sockets
- wait passively - call some java method and wait for its return
- monitor actively - in python check in loop some java object

Is there something else?

Cheers,

  roman

  Hi Roman,

On Tue, 11 Jan 2011, Roman Chyla wrote:

> I have recently wrapped solr inside jetty with JCC (we need to access
> very big result sets quickly, via JNI, but also keep solr running as
> normal) and was wondering what strategies do you guys use to speak
> *from inside* Java towards the Python end.
>
> So far, I was able to think about these:
>
> - raise exceptions in java and catch in python (I think I have seen
> this in some posts from Bill Jansen)
> - communicate via sockets
> - wait passively - call some java method and wait for its return
> - monitor actively - in python check in loop some java object
>
> Is there something else?

I'm not sure I completely understand your questions but if what you're
asking is how to run Python code from inside a Java servlet container, that
I've done with Tomcat and Lucene.

Basically, instead of embedding a JVM inside a Python VM - as is done for
PyLucene - you do the opposite, you embed a Python VM inside a JVM.

For that purpose, see the org.apache.jcc.PythonVM class available in JCC's
java tree. This class must be instantiated from the main thread at Java
servlet engine startup time. In Tomcat, I patched some startup code, in
BootStrap.java (see patches below) for this purpose.

Then, to make some Python code accessible from Java, use the usual way of
writing "extensions", the so-called JCC in reverse trick. Define a Java class
with some native methods implemented in Python; define a Python class that
"extends" it; build the Java class into a JAR; include it into a JCC-built
egg; install the egg into Python's env (site-packages, PYTHONPATH, whatever);
Then, write servlet code in Java that imports your Java class and calls it.

As you can see, this sounds simple but the devil is in the details. Of course,
bending Jetty for this may have different requirements but the code snippets
below should give you a good idea about what's required.

This approach has been in production running the freebase.com's search server
for over two years now.

If you have questions, of course, please ask.
Good luck !

Andi..

----------------------
Patch to Bootstrap.java to use JCC's PythonVM (which initializes the embedded
Python VM)

--- apache-tomcat-6.0.29-src/java/org/apache/catalina/startup/Bootstrap.java 2010-07-19 06:02:32.000000000 -0700
+++ apache-tomcat-6.0.29-src/java/org/apache/catalina/startup/Bootstrap.java.patched 2010-08-04 08:49:05.000000000 -0700
@@ -30,16 +30,18 @@
  import javax.management.MBeanServer;
  import javax.management.MBeanServerFactory;
  import javax.management.ObjectName;

  import org.apache.catalina.security.SecurityClassLoad;
  import org.apache.juli.logging.Log;
  import org.apache.juli.logging.LogFactory;

+import org.apache.jcc.PythonVM;
+

  /**
   * Boostrap loader for Catalina.  This application constructs a class loader
   * for use in loading the Catalina internal classes (by accumulating all of the
   * JAR files found in the "server" directory under "catalina.home"), and
   * starts the regular execution of the container.  The purpose of this
   * roundabout approach is to keep the Catalina internal classes (and any
   * other classes they depend on, such as an XML parser) out of the system
@@ -398,22 +400,24 @@
          try {
              String command = "start";
              if (args.length > 0) {
                  command = args[args.length - 1];
              }

              if (command.equals("startd")) {
                  args[args.length - 1] = "start";
+                PythonVM.start("mql");
                  daemon.load(args);
                  daemon.start();
              } else if (command.equals("stopd")) {
                  args[args.length - 1] = "stop";
                  daemon.stop();
              } else if (command.equals("start")) {
+                PythonVM.start("mql");
                  daemon.setAwait(true);
                  daemon.load(args);
                  daemon.start();
              } else if (command.equals("stop")) {
                  daemon.stopServer(args);
              } else {
                  log.warn("Bootstrap: command \"" + command + "\" does not exist.");
              }

Define a Java class:

package ....

public class EMQL {

     private long pythonObject;

     public EMQL()
     {
     }

     public void pythonExtension(long pythonObject)
     {
         this.pythonObject = pythonObject;
     }
     public long pythonExtension()
     {
         return this.pythonObject;
     }

     public void finalize()
         throws Throwable
     {
         pythonDecRef();
     }

     public native void pythonDecRef();

     // the methods implemented in python
     public native String init(ME me);
     public native String emql_refresh(String tid, String type);
     public native String emql_status();

     etc .......... etc

The corresponding Python class

import ......

from jemql import initVM, CLASSPATH, EMQL

initVM(CLASSPATH)

class emql(EMQL):

     def __init__(self):
         super(emql, self).__init__()

     def init(self, me):
      ...........
     def emql_refresh(self, tid, type):
      ...........
     def emql_status(self):
      ...........
        return "some status"

     etc ...... etc

Makefile rules to build this via JCC (the jemql.egg file is just an empty
target file for Makefile, it's not used for anything else):

default: jemql.egg

jemql.jar: java/org/blah/blah/EMQL.java
  mkdir -p classes
  javac -classpath $(CLASSPATH):$(MORE_CLASSPATH):$(etc..etc) -d classes $(JAVAC_FLAGS) $<
  jar -cvf $@ -C classes .

jemql.egg: jemql.jar $(JMQL_JAR) emql.py
  $(JCC) --version 1.0 --jar $< \
                --classpath $(CLASSPATH):$(JME_JAR):$(JMQL_JAR) \
                org.blah.blah.me.ME \
                --package java.lang \
                --python jemql --build $(DBG_FLAGS) \
                --install \
                --module emql
  touch $@
Patch to Tomcat's build.xml ANT script to add JCC's classes (like PythonVM) to
the buil

Hi Andy,

This is much more than I could have hoped! Just yesterday, I was
looking for ways how to embed Python VM in Jetty, as that would be
more natural, but found only jepp.sourceforge.net and off-putting was
the necessity to compile it against the newly built python. I could
not want it from the guys who may need my extension. And I realize
only now, that embedding Python in Java is even documented on the
website, but honestly i would not know how to do it without your
detailed examples.

Now to the questions, I apologize, some of them or all must seem very
stupid to you

- pylucene is used on many platforms and with jcc always worked as
expected (i love it!), but is it as reliable in the opposite
direction? The PythonVM.java loads "jcc" library, so I wonder if in
principle there is any difference in the directionality - but I am not
sure. To rephrase my convoluted question: would you expect this
wrapping be as reliable as wrapping java inside python is now?

- in the past, i built jcc libraries on one host and distributed them
on various machines. As long the family OS and the python main version
were the same, it worked on Win/Lin/Mac just fine. As far as I can
tell, this does not change, or will it be dependent on the python
against which the egg was built?

- now a little tricky issue; when I wrap jetty inside python, I hoped
to build it in a shared mode with lucene to be able to do some
low-level lucene indexing tasks from inside Python. If I do the
opposite and wrap Python VM in Java, I would still like to access the
lucene (which is possible, as I see well from your examples) But on
the python side, you are calling initVM() - will the initVM() call
create a new Java VM or will it access the parent Java VM which
started it?

- you say that threads are not managed by the Python VM, does that
mean there is no Python GIL?

- I don't really know what is exactly in the python thread local
storage, could that somehow negatively affect the Python process if
acquireThreadState/releaseThreadState are not called?

Thank you.

Cheers,

  roman
On Tue, Jan 11, 2011 at 8:13 PM, Andi Vajda <[EMAIL PROTECTED]> wrote:
>
>  Hi Roman,
>
> On Tue, 11 Jan 2011, Roman Chyla wrote:
>
>> I have recently wrapped solr inside jetty with JCC (we need to access
>> very big result sets quickly, via JNI, but also keep solr running as
>> normal) and was wondering what strategies do you guys use to speak
>> *from inside* Java towards the Python end.
>>
>> So far, I was able to think about these:
>>
>> - raise exceptions in java and catch in python (I think I have seen
>> this in some posts from Bill Jansen)
>> - communicate via sockets
>> - wait passively - call some java method and wait for its return
>> - monitor actively - in python check in loop some java object
>>
>> Is there something else?
>
> I'm not sure I completely understand your questions but if what you're
> asking is how to run Python code from inside a Java servlet container, that
> I've done with Tomcat and Lucene.
>
> Basically, instead of embedding a JVM inside a Python VM - as is done for
> PyLucene - you do the opposite, you embed a Python VM inside a JVM.
>
> For that purpose, see the org.apache.jcc.PythonVM class available in JCC's
> java tree. This class must be instantiated from the main thread at Java
> servlet engine startup time. In Tomcat, I patched some startup code, in
> BootStrap.java (see patches below) for this purpose.
>
> Then, to make some Python code accessible from Java, use the usual way of
> writing "extensions", the so-called JCC in reverse trick. Define a Java
> class
> with some native methods implemented in Python; define a Python class that
> "extends" it; build the Java class into a JAR; include it into a JCC-built
> egg; install the egg into Python's env (site-packages, PYTHONPATH,
> whatever);
> Then, write servlet code in Java that imports your Java class and calls it.
>
> As you can see, this sounds simple but the devil is in the details. Of
> course,

On Tue, 11 Jan 2011, Roman Chyla wrote:

> Hi Andy,
>
> This is much more than I could have hoped! Just yesterday, I was
> looking for ways how to embed Python VM in Jetty, as that would be
> more natural, but found only jepp.sourceforge.net and off-putting was
> the necessity to compile it against the newly built python. I could
> not want it from the guys who may need my extension. And I realize
> only now, that embedding Python in Java is even documented on the
> website, but honestly i would not know how to do it without your
> detailed examples.
>
> Now to the questions, I apologize, some of them or all must seem very
> stupid to you
>
> - pylucene is used on many platforms and with jcc always worked as
> expected (i love it!), but is it as reliable in the opposite
> direction? The PythonVM.java loads "jcc" library, so I wonder if in
> principle there is any difference in the directionality - but I am not
> sure. To rephrase my convoluted question: would you expect this
> wrapping be as reliable as wrapping java inside python is now?

I've been using this for over two years, in production.
My main worry was memory leaks because a server process is expected to stay
up and running for weeks at a time and it's been very stable on that front
too. Of course, when there is a bug somewhere that causes your Python VM to
crash, the entire server crashes. Just like when the JVM crashes (which is
normally rare). In other words, this isn't any less reliable than a
standalone Python VM process. It can be tricky, but is possible, to run gdb,
pdb and jdb together to step through the three languages involved, python,
java and C++. I've had to do this a few times but not in a long time.

> - in the past, i built jcc libraries on one host and distributed them on
> various machines. As long the family OS and the python main version were
> the same, it worked on Win/Lin/Mac just fine. As far as I can tell, this
> does not change, or will it be dependent on the python against which the
> egg was built?

Distributing binaries is risky. The same caveats apply. I wouldn't do it,
even in the simple PyLucene case.

> - now a little tricky issue; when I wrap jetty inside python, I hoped
> to build it in a shared mode with lucene to be able to do some
> low-level lucene indexing tasks from inside Python. If I do the
> opposite and wrap Python VM in Java, I would still like to access the
> lucene (which is possible, as I see well from your examples) But on
> the python side, you are calling initVM() - will the initVM() call
> create a new Java VM or will it access the parent Java VM which
> started it?

No, initVM() in this case just initializes your egg and adds its stuff to
the CLASSPATH. No Java VM init is done. As with any shared-mode JCC-built
extension, all calls to initVM() but the first one just do that.
The first call to initVM() in the embedding Python case is like that too
because there already is a Java VM running when PythonVM is instantiated and
called.

> - you say that threads are not managed by the Python VM, does that
> mean there is no Python GIL?

No, there is a Pythonn GIL (and that is the Achille's Heel of this setup if
you expect high concurrent servlet performance from your server calling
Python). That Python GIL is connected to this thread state I was mentioning
earlier. Because the thread is not managed by Python, when Python is called
(by way of the code generated by JCC) it doesn't find a thread state for the
thread and creates one. When the call completes, the thread state is
destroyed because its refcount goes to zero. My TerminatingThread class
acquires a Python thread state and keeps it for the life of the thread,
thereby working this problem around.

> - I don't really know what is exactly in the python thread local
> storage, could that somehow negatively affect the Python process if
> acquireThreadState/releaseThreadState are not called?

Yes, if you depend on thread-local storage, it would get lost between calls
and cause confusion and bugs, defeating its purpose. Python's thread-local
storage support is documented here:
http://docs.python.org/library/threading.html, look for threading.local.

Andi..

Hi Andi,

I think I will give it a try, if only because I am curious. Please see
one remaining question below.
On Tue, Jan 11, 2011 at 10:37 PM, Andi Vajda <[EMAIL PROTECTED]> wrote:
>
>
> On Tue, 11 Jan 2011, Roman Chyla wrote:
>
>> Hi Andy,
>>
>> This is much more than I could have hoped! Just yesterday, I was
>> looking for ways how to embed Python VM in Jetty, as that would be
>> more natural, but found only jepp.sourceforge.net and off-putting was
>> the necessity to compile it against the newly built python. I could
>> not want it from the guys who may need my extension. And I realize
>> only now, that embedding Python in Java is even documented on the
>> website, but honestly i would not know how to do it without your
>> detailed examples.
>>
>> Now to the questions, I apologize, some of them or all must seem very
>> stupid to you
>>
>> - pylucene is used on many platforms and with jcc always worked as
>> expected (i love it!), but is it as reliable in the opposite
>> direction? The PythonVM.java loads "jcc" library, so I wonder if in
>> principle there is any difference in the directionality - but I am not
>> sure. To rephrase my convoluted question: would you expect this
>> wrapping be as reliable as wrapping java inside python is now?
>
> I've been using this for over two years, in production.
> My main worry was memory leaks because a server process is expected to stay
> up and running for weeks at a time and it's been very stable on that front
> too. Of course, when there is a bug somewhere that causes your Python VM to
> crash, the entire server crashes. Just like when the JVM crashes (which is
> normally rare). In other words, this isn't any less reliable than a
> standalone Python VM process. It can be tricky, but is possible, to run gdb,
> pdb and jdb together to step through the three languages involved, python,
> java and C++. I've had to do this a few times but not in a long time.
>
>> - in the past, i built jcc libraries on one host and distributed them on
>> various machines. As long the family OS and the python main version were the
>> same, it worked on Win/Lin/Mac just fine. As far as I can tell, this does
>> not change, or will it be dependent on the python against which the egg was
>> built?
>
> Distributing binaries is risky. The same caveats apply. I wouldn't do it,
> even in the simple PyLucene case.

unfortunately, I don't have that many choices left - this is not for
some client-software scenario, we are running the jobs on the grid,
and there I cannot compile the binaries. So, if previously the
location of the python interpreter or python minor version did not
cause problems, now perhaps it will be different. But that wasn't for
the Solr, wrapping Solr is not meant for the grid.

>
>> - now a little tricky issue; when I wrap jetty inside python, I hoped
>> to build it in a shared mode with lucene to be able to do some
>> low-level lucene indexing tasks from inside Python. If I do the
>> opposite and wrap Python VM in Java, I would still like to access the
>> lucene (which is possible, as I see well from your examples) But on
>> the python side, you are calling initVM() - will the initVM() call
>> create a new Java VM or will it access the parent Java VM which
>> started it?
>
> No, initVM() in this case just initializes your egg and adds its stuff to
> the CLASSPATH. No Java VM init is done. As with any shared-mode JCC-built
> extension, all calls to initVM() but the first one just do that.
> The first call to initVM() in the embedding Python case is like that too
> because there already is a Java VM running when PythonVM is instantiated and
> called.

And if in the python, I will do:

import lucene
import lucene.initVM(lucene.CLASSPATH)

Will it work in this case? Giving access to the java classes from
inside python. Or I will have to forget pylucene, and prepare some
extra java classes? (the jcc in reverse trick, as you put it)

>
>> - you say that threads are not managed by the Python VM, does that

OK, this then looks like a normal Python - which is somehow making me
less worried :) I wanted to use multiprocessing inside python to deal
with GIL, and I see no reason why it should not work in this case.

Thank you very much.
Cheers,

  roman

Hi Andi, all,

I tried to implement the PythonVM wrapping on Mac 10.6, with JDK
1.6.22, jcc is freshly built, in shared mode, v. 2.6. The python is
the standard Python distributed with MacOsX

When I try to run the java, it throws an error when it gets to:

static {
        System.loadLibrary("jcc");
    }

I am getting this error:

Exception in thread "main" java.lang.UnsatisfiedLinkError:
/Library/Python/2.6/site-packages/JCC-2.6-py2.6-macosx-10.6-universal.egg/libjcc.dylib:
 Symbol not found: _PyExc_RuntimeError   Referenced from:
/Library/Python/2.6/site-packages/JCC-2.6-py2.6-macosx-10.6-universal.egg/libjcc.dylib
  Expected in: flat namespace  in
/Library/Python/2.6/site-packages/JCC-2.6-py2.6-macosx-10.6-universal.egg/libjcc.dylib
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1823)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1746)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
at java.lang.System.loadLibrary(System.java:1045)
at org.apache.jcc.PythonVM.<clinit>(PythonVM.java:23)
at rca.solr.JettyRunnerPythonVM.start(JettyRunnerPythonVM.java:53)
at rca.solr.JettyRunnerPythonVM.main(JettyRunnerPythonVM.java:139)
MacBeth:JCC-2.6-py2.6-macosx-10.6-universal.egg rca$ nm libjcc.dylib | grep Exc
                 U _PyExc_RuntimeError
                 U _PyExc_TypeError
                 U _PyExc_ValueError
0000000000003442 T __ZNK6JCCEnv15reportExceptionEv
00000000000021f0 T __ZNK6JCCEnv23getPythonExceptionClassEv
Any pointers what I could do wrong? Note, I haven't built any emql.egg
yet, I just run my java program and try to start PythonVM() and see if
that works.

Thanks,

  roman

On Wed, Jan 12, 2011 at 11:05 AM, Roman Chyla <[EMAIL PROTECTED]> wrote:
> Hi Andi,
>
> I think I will give it a try, if only because I am curious. Please see
> one remaining question below.
>
>
> On Tue, Jan 11, 2011 at 10:37 PM, Andi Vajda <[EMAIL PROTECTED]> wrote:
>>
>>
>> On Tue, 11 Jan 2011, Roman Chyla wrote:
>>
>>> Hi Andy,
>>>
>>> This is much more than I could have hoped! Just yesterday, I was
>>> looking for ways how to embed Python VM in Jetty, as that would be
>>> more natural, but found only jepp.sourceforge.net and off-putting was
>>> the necessity to compile it against the newly built python. I could
>>> not want it from the guys who may need my extension. And I realize
>>> only now, that embedding Python in Java is even documented on the
>>> website, but honestly i would not know how to do it without your
>>> detailed examples.
>>>
>>> Now to the questions, I apologize, some of them or all must seem very
>>> stupid to you
>>>
>>> - pylucene is used on many platforms and with jcc always worked as
>>> expected (i love it!), but is it as reliable in the opposite
>>> direction? The PythonVM.java loads "jcc" library, so I wonder if in
>>> principle there is any difference in the directionality - but I am not
>>> sure. To rephrase my convoluted question: would you expect this
>>> wrapping be as reliable as wrapping java inside python is now?
>>
>> I've been using this for over two years, in production.
>> My main worry was memory leaks because a server process is expected to stay
>> up and running for weeks at a time and it's been very stable on that front
>> too. Of course, when there is a bug somewhere that causes your Python VM to
>> crash, the entire server crashes. Just like when the JVM crashes (which is
>> normally rare). In other words, this isn't any less reliable than a
>> standalone Python VM process. It can be tricky, but is possible, to run gdb,
>> pdb and jdb together to step through the three languages involved, python,
>> java and C++. I've had to do this a few times but not in a long time.
>>
>>> - in the past, i built jcc libraries on one host and distributed them on
>>> various machines. As long the family OS and the python main version were the
>>> same, it worked on Win/Lin/Mac just fine. As far as I can tell, this does

On Wed, 12 Jan 2011, Roman Chyla wrote:

> Hi Andi, all,
>
> I tried to implement the PythonVM wrapping on Mac 10.6, with JDK
> 1.6.22, jcc is freshly built, in shared mode, v. 2.6. The python is
> the standard Python distributed with MacOsX
>
> When I try to run the java, it throws an error when it gets to:
>
> static {
>        System.loadLibrary("jcc");
>    }
>
> I am getting this error:
>
> Exception in thread "main" java.lang.UnsatisfiedLinkError:
> /Library/Python/2.6/site-packages/JCC-2.6-py2.6-macosx-10.6-universal.egg/libjcc.dylib:
> Symbol not found: _PyExc_RuntimeError   Referenced from:

That's because Python's shared library wasn't found. The reason is that, by
default, Python's shared lib not on JCC's link line because normally JCC is
loaded into a Python process and the dynamic linker thus finds the symbols
needed inside the process.

Here, since you're not starting inside a Python process, you need to add
'-framework Python' to JCC's LFLAGS in setup.py so that the dynamic linker
can find the Python VM shared lib and load it.

Andi..

> /Library/Python/2.6/site-packages/JCC-2.6-py2.6-macosx-10.6-universal.egg/libjcc.dylib
>  Expected in: flat namespace  in
> /Library/Python/2.6/site-packages/JCC-2.6-py2.6-macosx-10.6-universal.egg/libjcc.dylib
> at java.lang.ClassLoader$NativeLibrary.load(Native Method)
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1823)
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1746)
> at java.lang.Runtime.loadLibrary0(Runtime.java:823)
> at java.lang.System.loadLibrary(System.java:1045)
> at org.apache.jcc.PythonVM.<clinit>(PythonVM.java:23)
> at rca.solr.JettyRunnerPythonVM.start(JettyRunnerPythonVM.java:53)
> at rca.solr.JettyRunnerPythonVM.main(JettyRunnerPythonVM.java:139)
>
>
> MacBeth:JCC-2.6-py2.6-macosx-10.6-universal.egg rca$ nm libjcc.dylib | grep Exc
>                 U _PyExc_RuntimeError
>                 U _PyExc_TypeError
>                 U _PyExc_ValueError
> 0000000000003442 T __ZNK6JCCEnv15reportExceptionEv
> 00000000000021f0 T __ZNK6JCCEnv23getPythonExceptionClassEv
>
>
> Any pointers what I could do wrong? Note, I haven't built any emql.egg
> yet, I just run my java program and try to start PythonVM() and see if
> that works.
>
> Thanks,
>
>  roman
>
>
>
> On Wed, Jan 12, 2011 at 11:05 AM, Roman Chyla <[EMAIL PROTECTED]> wrote:
>> Hi Andi,
>>
>> I think I will give it a try, if only because I am curious. Please see
>> one remaining question below.
>>
>>
>> On Tue, Jan 11, 2011 at 10:37 PM, Andi Vajda <[EMAIL PROTECTED]> wrote:
>>>
>>>
>>> On Tue, 11 Jan 2011, Roman Chyla wrote:
>>>
>>>> Hi Andy,
>>>>
>>>> This is much more than I could have hoped! Just yesterday, I was
>>>> looking for ways how to embed Python VM in Jetty, as that would be
>>>> more natural, but found only jepp.sourceforge.net and off-putting was
>>>> the necessity to compile it against the newly built python. I could
>>>> not want it from the guys who may need my extension. And I realize
>>>> only now, that embedding Python in Java is even documented on the
>>>> website, but honestly i would not know how to do it without your
>>>> detailed examples.
>>>>
>>>> Now to the questions, I apologize, some of them or all must seem very
>>>> stupid to you
>>>>
>>>> - pylucene is used on many platforms and with jcc always worked as
>>>> expected (i love it!), but is it as reliable in the opposite
>>>> direction? The PythonVM.java loads "jcc" library, so I wonder if in
>>>> principle there is any difference in the directionality - but I am not
>>>> sure. To rephrase my convoluted question: would you expect this
>>>> wrapping be as reliable as wrapping java inside python is now?
>>>
>>> I've been using this for over two years, in production.
>>> My main worry was memory leaks because a server process is expected to stay
>>> up and running for weeks at a time and it's been very stable on that front
>>> too. Of course, when there is a bug somewhere that causes your Python VM to

On Wed, 12 Jan 2011, Roman Chyla wrote:

> And if in the python, I will do:
>
> import lucene
> import lucene.initVM(lucene.CLASSPATH)
>
> Will it work in this case? Giving access to the java classes from
> inside python. Or I will have to forget pylucene, and prepare some
> extra java classes? (the jcc in reverse trick, as you put it)

Yes, just be sure to JCC-build your eggs with --import lucene so that you
don't wrap lucene multiple times.

>>> - you say that threads are not managed by the Python VM, does that
>>> mean there is no Python GIL?
>>
>> No, there is a Pythonn GIL (and that is the Achille's Heel of this setup if
>> you expect high concurrent servlet performance from your server calling
>> Python). That Python GIL is connected to this thread state I was mentioning
>> earlier. Because the thread is not managed by Python, when Python is called
>> (by way of the code generated by JCC) it doesn't find a thread state for the
>> thread and creates one. When the call completes, the thread state is
>> destroyed because its refcount goes to zero. My TerminatingThread class
>> acquires a Python thread state and keeps it for the life of the thread,
>> thereby working this problem around.
>
> OK, this then looks like a normal Python - which is somehow making me
> less worried :) I wanted to use multiprocessing inside python to deal
> with GIL, and I see no reason why it should not work in this case.

I tried that approach originally and gave up. There were too many strange
lock-ups talking to python subprocesses managed by multiprocessing. Now,
this was before it became part of Python's distribution so maybe bugs got
fixed since then.

Andi..

Hi Andi,

Thanks for the help, now I was able to run the java and loaded
PythonVM. I then built the python egg, after a bit of fiddling with
parameters, it seems ok. I can import the jcc wrapped python class and
call it:

In [1]: from solrpie_java import emql

In [2]: em = emql.Emql()

In [3]: em.javaTestPrint()
java is printing

In [4]: em.pythonTestPrint()
just a test

But I haven't found out how to call the same from java.

The egg is built fine, it is named solrpie_java and contains one python module:

=============
from solrpie_java import initVM, CLASSPATH, EMQL

initVM(CLASSPATH)
class Emql(EMQL):
    '''
    classdocs
    '''

    def __init__(self):
        super(Emql, self).__init__()
        print '__init__'
    def init(self, me):
        print self, me
        return 'init'
    def emql_refresh(self, tid, type):
        print self, tid, type
        return 'refresh'
    def emql_status(self):
        return "some status"

    def pythonTestPrint(self):
        print 'just a test'

=======The corresponding java class looks like this:
public class EMQL {

  private long pythonObject;

  public EMQL()
  {
  }

  public void pythonExtension(long pythonObject)
  {
      this.pythonObject = pythonObject;
  }
  public long pythonExtension()
  {
      return this.pythonObject;
  }

  public void finalize()
      throws Throwable
  {
      pythonDecRef();
  }

  public void javaTestPrint() {
  System.out.println("java is printing");
  }

  public native void pythonDecRef();

  // the methods implemented in python
  public native String init(EMQL me);
  public native String emql_refresh(String tid, String type);
  public native String emql_status();

  public native void pythonTestPrint();


}

======
I tried running it as:

                PythonVM vm = PythonVM.start("sorlpie_java");
EMQL em = new EMQL();
em.javaTestPrint();
em.pythonTestPrint();

I get this:

java is printing
Exception in thread "main" java.lang.UnsatisfiedLinkError:
rca.pythonvm.EMQL.pythonTestPrint()V
at rca.pythonvm.EMQL.pythonTestPrint(Native Method)
at rca.solr.JettyRunnerPythonVM.start(JettyRunnerPythonVM.java:60)
at rca.solr.JettyRunnerPythonVM.main(JettyRunnerPythonVM.java:148)

I understand that java cannot find the linked c++ method, but I don't
know how to fix that.
If i try:

PythonVM vm = PythonVM.start("sorlpie_java");
Object m = vm.instantiate("emql", "Emql");

I get:

org.apache.jcc.PythonException: No module named emql
ImportError: No module named emql

at org.apache.jcc.PythonVM.instantiate(Native Method)
at rca.solr.JettyRunnerPythonVM.start(JettyRunnerPythonVM.java:56)
at rca.solr.JettyRunnerPythonVM.main(JettyRunnerPythonVM.java:148)

I tried various combinations of instanatiation, and setting the
classpatt or -Djava.library.path
But no success. What am I doing wrong?

Thank you,

  roman

On Wed, Jan 12, 2011 at 7:55 PM, Andi Vajda <[EMAIL PROTECTED]> wrote:
>
> On Wed, 12 Jan 2011, Roman Chyla wrote:
>
>> Hi Andi, all,
>>
>> I tried to implement the PythonVM wrapping on Mac 10.6, with JDK
>> 1.6.22, jcc is freshly built, in shared mode, v. 2.6. The python is
>> the standard Python distributed with MacOsX
>>
>> When I try to run the java, it throws an error when it gets to:
>>
>> static {
>>       System.loadLibrary("jcc");
>>   }
>>
>> I am getting this error:
>>
>> Exception in thread "main" java.lang.UnsatisfiedLinkError:
>>
>> /Library/Python/2.6/site-packages/JCC-2.6-py2.6-macosx-10.6-universal.egg/libjcc.dylib:
>> Symbol not found: _PyExc_RuntimeError   Referenced from:
>
> That's because Python's shared library wasn't found. The reason is that, by
> default, Python's shared lib not on JCC's link line because normally JCC is
> loaded into a Python process and the dynamic linker thus finds the symbols
> needed inside the process.
>
> Here, since you're not starting inside a Python process, you need to add
> '-framework Python' to JCC's LFLAGS in setup.py so that the dynamic linker

Oh, nevermind I found it out:

It is loaded by python, so PYTHONPATH or other ways must be used.
Also i had to change exports:
inside sorlpie_java/__init__.py
i added line:

from emql import Emql

Then, in java i can do:

                PythonVM vm = PythonVM.start("sorlpie_java");
EMQL em = (EMQL)vm.instantiate("solrpie_java", "Emql");

em.javaTestPrint();
em.pythonTestPrint();
System.out.println(em.emql_status());

And I get:

java is printing
some status

funny the pythonTestPrint() never prints anything
Cheers,

   roman

On Wed, Jan 12, 2011 at 11:20 PM, Roman Chyla <[EMAIL PROTECTED]> wrote:
> Hi Andi,
>
> Thanks for the help, now I was able to run the java and loaded
> PythonVM. I then built the python egg, after a bit of fiddling with
> parameters, it seems ok. I can import the jcc wrapped python class and
> call it:
>
> In [1]: from solrpie_java import emql
>
> In [2]: em = emql.Emql()
>
> In [3]: em.javaTestPrint()
> java is printing
>
> In [4]: em.pythonTestPrint()
> just a test
>
> But I haven't found out how to call the same from java.
>
> The egg is built fine, it is named solrpie_java and contains one python module:
>
> =============>
> from solrpie_java import initVM, CLASSPATH, EMQL
>
> initVM(CLASSPATH)
>
>
> class Emql(EMQL):
>    '''
>    classdocs
>    '''
>
>    def __init__(self):
>        super(Emql, self).__init__()
>        print '__init__'
>
>
>    def init(self, me):
>        print self, me
>        return 'init'
>    def emql_refresh(self, tid, type):
>        print self, tid, type
>        return 'refresh'
>    def emql_status(self):
>        return "some status"
>
>    def pythonTestPrint(self):
>        print 'just a test'
>
> =======> The corresponding java class looks like this:
>
>
> public class EMQL {
>
>           private long pythonObject;
>
>           public EMQL()
>           {
>           }
>
>           public void pythonExtension(long pythonObject)
>           {
>               this.pythonObject = pythonObject;
>           }
>           public long pythonExtension()
>           {
>               return this.pythonObject;
>           }
>
>           public void finalize()
>               throws Throwable
>           {
>               pythonDecRef();
>           }
>
>           public void javaTestPrint() {
>                   System.out.println("java is printing");
>           }
>
>           public native void pythonDecRef();
>
>           // the methods implemented in python
>           public native String init(EMQL me);
>           public native String emql_refresh(String tid, String type);
>           public native String emql_status();
>
>           public native void pythonTestPrint();
>
>
> }
>
> ======>
> I tried running it as:
>
>                PythonVM vm = PythonVM.start("sorlpie_java");
>                EMQL em = new EMQL();
>                em.javaTestPrint();
>                em.pythonTestPrint();
>
> I get this:
>
> java is printing
> Exception in thread "main" java.lang.UnsatisfiedLinkError:
> rca.pythonvm.EMQL.pythonTestPrint()V
>        at rca.pythonvm.EMQL.pythonTestPrint(Native Method)
>        at rca.solr.JettyRunnerPythonVM.start(JettyRunnerPythonVM.java:60)
>        at rca.solr.JettyRunnerPythonVM.main(JettyRunnerPythonVM.java:148)
>
> I understand that java cannot find the linked c++ method, but I don't
> know how to fix that.
> If i try:
>
> PythonVM vm = PythonVM.start("sorlpie_java");
> Object m = vm.instantiate("emql", "Emql");
>
> I get:
>
> org.apache.jcc.PythonException: No module named emql
> ImportError: No module named emql
>
>        at org.apache.jcc.PythonVM.instantiate(Native Method)
>        at rca.solr.JettyRunnerPythonVM.start(JettyRunnerPythonVM.java:56)
>        at rca.solr.JettyRunnerPythonVM.main(JettyRunnerPythonVM.java:148)
>
> I tried various combinations of instanatiation, and setting the
> classpatt or -Djava.library.path
> But no success. What am I doing wrong?
>
> Thank you,
>
>  roman
>
>
>
> On Wed, Jan 12, 2011 at 7:55 PM, Andi Vajda <[EMAIL PROTECTED]> wrote:

  Hi Roman,

On Wed, 12 Jan 2011, Roman Chyla wrote:

> Thanks for the help, now I was able to run the java and loaded
> PythonVM. I then built the python egg, after a bit of fiddling with
> parameters, it seems ok. I can import the jcc wrapped python class and
> call it:
>
> In [1]: from solrpie_java import emql

Why are you calling your class EMQL ? (this name was just an example culled
from my code).

> In [2]: em = emql.Emql()
>
> In [3]: em.javaTestPrint()
> java is printing
>
> In [4]: em.pythonTestPrint()
> just a test
>
> But I haven't found out how to call the same from java.

Ah, yes, I forgot to tell you how to pull that in.
In Java, you import that 'EMQL' java class and instantiate it by way of the
PythonVM instance's instantiate() call:

             import org.blah.blah.EMQL;
             import org.apache.jcc.PythonVM;

             .............

             PythonVM vm = PythonVM.get();

             emql = (EMQL) vm.instantiate("jemql.emql", "emql");
             ... call method on emql instance just created ...

The instantiate("foo", "bar") method in effect asks Python to run
   "from foo import bar"
   "return bar()"

Andi..

On Wed, 12 Jan 2011, Roman Chyla wrote:

> Oh, nevermind I found it out:
>
> It is loaded by python, so PYTHONPATH or other ways must be used.
> Also i had to change exports:
> inside sorlpie_java/__init__.py
> i added line:
>
> from emql import Emql
>
> Then, in java i can do:
>
>                PythonVM vm = PythonVM.start("sorlpie_java");
> EMQL em = (EMQL)vm.instantiate("solrpie_java", "Emql");

Yep, that's the one !

Andi..

>
> em.javaTestPrint();
> em.pythonTestPrint();
> System.out.println(em.emql_status());
>
> And I get:
>
> java is printing
> some status
>
> funny the pythonTestPrint() never prints anything
>
>
> Cheers,
>
>   roman
>
>
>
> On Wed, Jan 12, 2011 at 11:20 PM, Roman Chyla <[EMAIL PROTECTED]> wrote:
>> Hi Andi,
>>
>> Thanks for the help, now I was able to run the java and loaded
>> PythonVM. I then built the python egg, after a bit of fiddling with
>> parameters, it seems ok. I can import the jcc wrapped python class and
>> call it:
>>
>> In [1]: from solrpie_java import emql
>>
>> In [2]: em = emql.Emql()
>>
>> In [3]: em.javaTestPrint()
>> java is printing
>>
>> In [4]: em.pythonTestPrint()
>> just a test
>>
>> But I haven't found out how to call the same from java.
>>
>> The egg is built fine, it is named solrpie_java and contains one python module:
>>
>> =============>>
>> from solrpie_java import initVM, CLASSPATH, EMQL
>>
>> initVM(CLASSPATH)
>>
>>
>> class Emql(EMQL):
>>    '''
>>    classdocs
>>    '''
>>
>>    def __init__(self):
>>        super(Emql, self).__init__()
>>        print '__init__'
>>
>>
>>    def init(self, me):
>>        print self, me
>>        return 'init'
>>    def emql_refresh(self, tid, type):
>>        print self, tid, type
>>        return 'refresh'
>>    def emql_status(self):
>>        return "some status"
>>
>>    def pythonTestPrint(self):
>>        print 'just a test'
>>
>> =======>> The corresponding java class looks like this:
>>
>>
>> public class EMQL {
>>
>>           private long pythonObject;
>>
>>           public EMQL()
>>           {
>>           }
>>
>>           public void pythonExtension(long pythonObject)
>>           {
>>               this.pythonObject = pythonObject;
>>           }
>>           public long pythonExtension()
>>           {
>>               return this.pythonObject;
>>           }
>>
>>           public void finalize()
>>               throws Throwable
>>           {
>>               pythonDecRef();
>>           }
>>
>>           public void javaTestPrint() {
>>                   System.out.println("java is printing");
>>           }
>>
>>           public native void pythonDecRef();
>>
>>           // the methods implemented in python
>>           public native String init(EMQL me);
>>           public native String emql_refresh(String tid, String type);
>>           public native String emql_status();
>>
>>           public native void pythonTestPrint();
>>
>>
>> }
>>
>> ======>>
>> I tried running it as:
>>
>>                PythonVM vm = PythonVM.start("sorlpie_java");
>>                EMQL em = new EMQL();
>>                em.javaTestPrint();
>>                em.pythonTestPrint();
>>
>> I get this:
>>
>> java is printing
>> Exception in thread "main" java.lang.UnsatisfiedLinkError:
>> rca.pythonvm.EMQL.pythonTestPrint()V
>>        at rca.pythonvm.EMQL.pythonTestPrint(Native Method)
>>        at rca.solr.JettyRunnerPythonVM.start(JettyRunnerPythonVM.java:60)
>>        at rca.solr.JettyRunnerPythonVM.main(JettyRunnerPythonVM.java:148)
>>
>> I understand that java cannot find the linked c++ method, but I don't
>> know how to fix that.
>> If i try:
>>
>> PythonVM vm = PythonVM.start("sorlpie_java");
>> Object m = vm.instantiate("emql", "Emql");
>>
>> I get:
>>
>> org.apache.jcc.PythonException: No module named emql
>> ImportError: No module named emql
>>
>>        at org.apache.jcc.PythonVM.instantiate(Native Method)
>>        at rca.solr.JettyRunnerPythonVM.start(JettyRunnerPythonVM.java:56)
>>        at rca.solr.JettyRunnerPythonVM.main(JettyRunnerPythonVM.java:148)

Hi Andi,

Your help is great, thanks a lot! Without your detailed instructions,
I would not be able to figure it out - and the last bit with the
python...I should have thought before writing :-)

I call the class EMQL just because I was lazy to change it. But I will
do now that I understand little bit more. What I find very cool is the
fact, that if I build this extension the way you showed me, I can run
java from inside python, but also python from inside Java - and with
one jar and one compiled egg. Very handy. But as you said, evil is in
details, so I expect some bumps.

And about the thing with LFLAGS 'platform Python', also other
platforms will need something similar like Mac? I assume this is a mac
dynamic discovery of the libraries, will anything bad happen if I
changed the path of the Python now when the extension was built?
Cheers!

  roman

On Wed, Jan 12, 2011 at 11:54 PM, Andi Vajda <[EMAIL PROTECTED]> wrote:
>
>  Hi Roman,
>
> On Wed, 12 Jan 2011, Roman Chyla wrote:
>
>> Thanks for the help, now I was able to run the java and loaded
>> PythonVM. I then built the python egg, after a bit of fiddling with
>> parameters, it seems ok. I can import the jcc wrapped python class and
>> call it:
>>
>> In [1]: from solrpie_java import emql
>
> Why are you calling your class EMQL ? (this name was just an example culled
> from my code).
>
>> In [2]: em = emql.Emql()
>>
>> In [3]: em.javaTestPrint()
>> java is printing
>>
>> In [4]: em.pythonTestPrint()
>> just a test
>>
>> But I haven't found out how to call the same from java.
>
> Ah, yes, I forgot to tell you how to pull that in.
> In Java, you import that 'EMQL' java class and instantiate it by way of the
> PythonVM instance's instantiate() call:
>
>            import org.blah.blah.EMQL;
>            import org.apache.jcc.PythonVM;
>
>            .............
>
>            PythonVM vm = PythonVM.get();
>
>            emql = (EMQL) vm.instantiate("jemql.emql", "emql");
>            ... call method on emql instance just created ...
>
> The instantiate("foo", "bar") method in effect asks Python to run
>  "from foo import bar"
>  "return bar()"
>
> Andi..
>

  Hi Roman,

On Thu, 13 Jan 2011, Roman Chyla wrote:

> Your help is great, thanks a lot! Without your detailed instructions,
> I would not be able to figure it out - and the last bit with the
> python...I should have thought before writing :-)
>
> I call the class EMQL just because I was lazy to change it. But I will
> do now that I understand little bit more. What I find very cool is the
> fact, that if I build this extension the way you showed me, I can run
> java from inside python, but also python from inside Java - and with
> one jar and one compiled egg. Very handy. But as you said, evil is in
> details, so I expect some bumps.
>
> And about the thing with LFLAGS 'platform Python', also other
> platforms will need something similar like Mac? I assume this is a mac
> dynamic discovery of the libraries, will anything bad happen if I
> changed the path of the Python now when the extension was built?

I don't know, I don't distribute binaries :-)

The 'otool -L' command on the mac will show you what libraries the linker is
going to be looking for for a given shared library. The DYLD_FRAMEWORK_PATH
(?) and DYLD_LIBRARY_PATH env vars may help you in overriding this but I'm
not so sure.

Andi..

>
>
> Cheers!
>
>  roman
>
> On Wed, Jan 12, 2011 at 11:54 PM, Andi Vajda <[EMAIL PROTECTED]> wrote:
>>
>> ï¿œHi Roman,
>>
>> On Wed, 12 Jan 2011, Roman Chyla wrote:
>>
>>> Thanks for the help, now I was able to run the java and loaded
>>> PythonVM. I then built the python egg, after a bit of fiddling with
>>> parameters, it seems ok. I can import the jcc wrapped python class and
>>> call it:
>>>
>>> In [1]: from solrpie_java import emql
>>
>> Why are you calling your class EMQL ? (this name was just an example culled
>> from my code).
>>
>>> In [2]: em = emql.Emql()
>>>
>>> In [3]: em.javaTestPrint()
>>> java is printing
>>>
>>> In [4]: em.pythonTestPrint()
>>> just a test
>>>
>>> But I haven't found out how to call the same from java.
>>
>> Ah, yes, I forgot to tell you how to pull that in.
>> In Java, you import that 'EMQL' java class and instantiate it by way of the
>> PythonVM instance's instantiate() call:
>>
>> ￜ ￜ ￜ ￜ ￜ ￜimport org.blah.blah.EMQL;
>> ￜ ￜ ￜ ￜ ￜ ￜimport org.apache.jcc.PythonVM;
>>
>> ￜ ￜ ￜ ￜ ￜ ￜ.............
>>
>> ￜ ￜ ￜ ￜ ￜ ￜPythonVM vm = PythonVM.get();
>>
>> ￜ ￜ ￜ ￜ ￜ ￜemql = (EMQL) vm.instantiate("jemql.emql", "emql");
>> ￜ ￜ ￜ ￜ ￜ ￜ... call method on emql instance just created ...
>>
>> The instantiate("foo", "bar") method in effect asks Python to run
>> ￜ"from foo import bar"
>> ￜ"return bar()"
>>
>> Andi..
>>
>