New topic.

I'd like to wrap my UpLib codebase, which is Python using PyLucene, in
Java using JEPP (http://jepp.sourceforge.net/), so that I can use it
with Tomcat.

Now, am I going to have to do some trickery to get a VM?  Or will
getVMEnv() just work with a previously initialized JVM?

Bill

On Mar 3, 2011, at 21:50, Bill Janssen <[EMAIL PROTECTED]> wrote:

> New topic.
>
> I'd like to wrap my UpLib codebase, which is Python using PyLucene, in
> Java using JEPP (http://jepp.sourceforge.net/), so that I can use it
> with Tomcat.
>
> Now, am I going to have to do some trickery to get a VM?  Or will
> getVMEnv() just work with a previously initialized JVM?

Not so long ago on this list someone asked about this, using python from java via jcc, something I've been doing with tomcat for a couple of years now.
I sent a long, detailed answer. I believe it was to Roman Chyla. A quick look in this mailing list archives should help you locate that thread and get answers to the above questions.

Andi..

>
> Bill

Yes, and I can say it is working extremely well so far - we have done
and are doing some extensive benchmarking and tests. I also use
multiprocessing inside (python2.6) and I hope I would be able to
publish the source code soon, it could be re-usable. If you are
interested before that happens, please send me an email.

Best,

  Roman

On Fri, Mar 4, 2011 at 7:27 AM, Andi Vajda <[EMAIL PROTECTED]> wrote:
>
> On Mar 3, 2011, at 21:50, Bill Janssen <[EMAIL PROTECTED]> wrote:
>
>> New topic.
>>
>> I'd like to wrap my UpLib codebase, which is Python using PyLucene, in
>> Java using JEPP (http://jepp.sourceforge.net/), so that I can use it
>> with Tomcat.
>>
>> Now, am I going to have to do some trickery to get a VM?  Or will
>> getVMEnv() just work with a previously initialized JVM?
>
> Not so long ago on this list someone asked about this, using python from java via jcc, something I've been doing with tomcat for a couple of years now.
> I sent a long, detailed answer. I believe it was to Roman Chyla. A quick look in this mailing list archives should help you locate that thread and get answers to the above questions.
>
> Andi..
>
>>
>> Bill
>
>

Andi Vajda <[EMAIL PROTECTED]> wrote:

> Not so long ago on this list someone asked about this, using python
> from java via jcc, something I've been doing with tomcat for a couple
> of years now.  I sent a long, detailed answer. I believe it was to
> Roman Chyla. A quick look in this mailing list archives should help
> you locate that thread and get answers to the above questions.

Ah, right.  I'd forgotten that you've added Java-side support to JCC.
Maybe I can do this without JEPP.

Bill

Andi Vajda <[EMAIL PROTECTED]> wrote:

> Not so long ago on this list someone asked about this, using python from java via jcc, something I've been doing with tomcat for a couple of years now.

See

http://mail-archives.apache.org/mod_mbox/lucene-pylucene-dev/201101.mbox/%[EMAIL PROTECTED]l%3e

Bill

The number of patches to Tomcat make me uneasy.  I was hoping to bundle
all this into a .war file containing a servlet which would use it with
an unmodified Tomcat.

Does that seem possible?

Also, what if I want to run a number of different Python programs?  Can
I instantiate PythonVM more than once?  Or if I need to run the same
Python program multiple times?

Bill

On Mar 4, 2011, at 9:40, Bill Janssen <[EMAIL PROTECTED]> wrote:

> The number of patches to Tomcat make me uneasy.  I was hoping to bundle
> all this into a .war file containing a servlet which would use it with
> an unmodified Tomcat.
>
> Does that seem possible?

No. The PythonVM must be initialized in the main thread and you have no control over that in Tomcat out of the box, apparently.

> Also, what if I want to run a number of different Python programs?  Can
> I instantiate PythonVM more than once?  

No. The PythonVM is loaded in process as a shared library and there can only be one per process. In other words, only one PythonVM can be instantiated from that library in a given process. See docs about Py_Initialize().

Andi..

> Or if I need to run the same
> Python program multiple times?

There is no notion of python program in this context (no __main__). You import a module, instantiate a class from it and invoke its methods. You can import and instantiate as many classes and modules as you like, of course.

Andi..

>
> Bill

Andi Vajda <[EMAIL PROTECTED]> wrote:

> > Or if I need to run the same Python program multiple times?
>
> There is no notion of python program in this context (no
> __main__). You import a module, instantiate a class from it and invoke
> its methods. You can import and instantiate as many classes and
> modules as you like, of course.

JEPP has an interface to Python's "exec".  This avoids the overhead of
writing a Java class which is then subclassed in Python which is then
instantiated and brought out to Java again so that you can call one of
its methods.  That would be a handy thing to have in PythonVM.  I
suppose I could add it, after I get PythonVM working (this loadLibrary
thing).

Alternatively, I could use JEPP, or JEPP tweaked for JCC, if there was
some way to pass the Java VM "into" the JEPP instantiation of Python, in
such a way that that initVM() could find it.

Bill

On Fri, 4 Mar 2011, Bill Janssen wrote:

> Andi Vajda <[EMAIL PROTECTED]> wrote:
>
>>> Or if I need to run the same Python program multiple times?
>>
>> There is no notion of python program in this context (no
>> __main__). You import a module, instantiate a class from it and invoke
>> its methods. You can import and instantiate as many classes and
>> modules as you like, of course.
>
> JEPP has an interface to Python's "exec".  This avoids the overhead of
> writing a Java class which is then subclassed in Python which is then
> instantiated and brought out to Java again so that you can call one of
> its methods.  That would be a handy thing to have in PythonVM.  I
> suppose I could add it, after I get PythonVM working (this loadLibrary
> thing).
>
> Alternatively, I could use JEPP, or JEPP tweaked for JCC, if there was
> some way to pass the Java VM "into" the JEPP instantiation of Python, in
> such a way that that initVM() could find it.

If JEPP execs python, then that Python VM runs in a subprocess of the Java
process. If you then call JCC's initVM() from that subprocess, I do not know
how the JNI api would behave in that situation. Will it "find" the parent
process' JVM, I don't know. I doubt that JNI has all the magic required for
transparent IPC to the parent process built-in out of the box.

This is somewhat similar in setup to what Roman is doing, using Python's
subprocessing module insteaf. I had done that too, originally, but gave up
because of inextricable deadlocks.

Andi..

Andi Vajda <[EMAIL PROTECTED]> wrote:

>
> On Fri, 4 Mar 2011, Bill Janssen wrote:
>
> > Andi Vajda <[EMAIL PROTECTED]> wrote:
> >
> >>> Or if I need to run the same Python program multiple times?
> >>
> >> There is no notion of python program in this context (no
> >> __main__). You import a module, instantiate a class from it and invoke
> >> its methods. You can import and instantiate as many classes and
> >> modules as you like, of course.
> >
> > JEPP has an interface to Python's "exec".  This avoids the overhead of
> > writing a Java class which is then subclassed in Python which is then
> > instantiated and brought out to Java again so that you can call one of
> > its methods.  That would be a handy thing to have in PythonVM.  I
> > suppose I could add it, after I get PythonVM working (this loadLibrary
> > thing).
> >
> > Alternatively, I could use JEPP, or JEPP tweaked for JCC, if there was
> > some way to pass the Java VM "into" the JEPP instantiation of Python, in
> > such a way that that initVM() could find it.
>
> If JEPP execs python, then that Python VM runs in a subprocess of the

It's not Java's Runtime.exec(), it's Python's built-in function, "exec".
So the Python VM is running in the same address space as Java.

> Java process. If you then call JCC's initVM() from that subprocess, I
> do not know how the JNI api would behave in that situation. Will it
> "find" the parent process' JVM, I don't know. I doubt that JNI has all
> the magic required for transparent IPC to the parent process built-in
> out of the box.

Right.  If I was going to do that, I'd just use java.lang.Runtime.exec().

Bill

On Mar 4, 2011, at 11:34, Bill Janssen <[EMAIL PROTECTED]> wrote:

> Andi Vajda <[EMAIL PROTECTED]> wrote:
>
>>
>> On Fri, 4 Mar 2011, Bill Janssen wrote:
>>
>>> Andi Vajda <[EMAIL PROTECTED]> wrote:
>>>
>>>>> Or if I need to run the same Python program multiple times?
>>>>
>>>> There is no notion of python program in this context (no
>>>> __main__). You import a module, instantiate a class from it and invoke
>>>> its methods. You can import and instantiate as many classes and
>>>> modules as you like, of course.
>>>
>>> JEPP has an interface to Python's "exec".  This avoids the overhead of
>>> writing a Java class which is then subclassed in Python which is then
>>> instantiated and brought out to Java again so that you can call one of
>>> its methods.  That would be a handy thing to have in PythonVM.  I
>>> suppose I could add it, after I get PythonVM working (this loadLibrary
>>> thing).
>>>
>>> Alternatively, I could use JEPP, or JEPP tweaked for JCC, if there was
>>> some way to pass the Java VM "into" the JEPP instantiation of Python, in
>>> such a way that that initVM() could find it.
>>
>> If JEPP execs python, then that Python VM runs in a subprocess of the
>
> It's not Java's Runtime.exec(), it's Python's built-in function, "exec".
> So the Python VM is running in the same address space as Java.

Ok, so how is JEPP doing this from Tomcat without patching the Tomcat startup code so that the PythonVM is initialized in the main thread ?

If what you say is true then how is that different from what Roman is doing with JCC and Python's subprocessing module ?

Andi..

>
>> Java process. If you then call JCC's initVM() from that subprocess, I
>> do not know how the JNI api would behave in that situation. Will it
>> "find" the parent process' JVM, I don't know. I doubt that JNI has all
>> the magic required for transparent IPC to the parent process built-in
>> out of the box.
>
> Right.  If I was going to do that, I'd just use java.lang.Runtime.exec().
>
> Bill

Andi Vajda <[EMAIL PROTECTED]> wrote:

> >> If JEPP execs python, then that Python VM runs in a subprocess of the
> >
> > It's not Java's Runtime.exec(), it's Python's built-in function, "exec".
> > So the Python VM is running in the same address space as Java.
>
> Ok, so how is JEPP doing this from Tomcat without patching the Tomcat
> startup code so that the PythonVM is initialized in the main thread ?

I don't know that it is.  What I'm interested in doing is starting a
Python interpreter in Java, then occasionally calling into it to exec
some Python code.  It would be great if it could also eval a Python
expression and return the unicode() of the result.  I'd like to see if
I can add that functionality to PythonVM.  Then I'll worry about Tomcat,
where I imagine I'll have to do just what you've already done.

> If what you say is true then how is that different from what Roman is
> doing with JCC and Python's subprocessing module ?

I don't know that it is.  I use the subprocess module a lot in UpLib, too.

Bill

On Fri, 4 Mar 2011, Bill Janssen wrote:

> Andi Vajda <[EMAIL PROTECTED]> wrote:
>
>>>> If JEPP execs python, then that Python VM runs in a subprocess of the
>>>
>>> It's not Java's Runtime.exec(), it's Python's built-in function, "exec".
>>> So the Python VM is running in the same address space as Java.
>>
>> Ok, so how is JEPP doing this from Tomcat without patching the Tomcat
>> startup code so that the PythonVM is initialized in the main thread ?
>
> I don't know that it is.  What I'm interested in doing is starting a
> Python interpreter in Java, then occasionally calling into it to exec
> some Python code.

I see. Look at the PythonVM.instantiate() method. It's the one that does the
python equivalent of "from module import blah; blah()".
You could add another method there that does an exec or eval instead.

Andi..