pyclass

[yakir12]

I'm trying to implement a Python class using PythonCall.pyclass.

The following works as expected:

using PythonCall

Person = pyclass("Person", (); print_random = print ∘ rand)
@pyexec (Person = Person) => """
p1 = Person()
p1.print_random() 
"""
# prints a random number

But any attempt to include methods that accept arguments referencing the instance of the class itself:

Person = pyclass("Person", (); __init__ = (self) -> (self.state = 0))
@pyexec (Person = Person) => """
p1 = Person()
"""

fails:

ERROR: Python: TypeError: Julia: MethodError: no method matching (::var"#5#6")()
Closest candidates are:
  (::var"#5#6")(!Matched::Any) at REPL[15]:1
Python stacktrace:
 [1] __call__
   @ /home/yakir/.julia/packages/PythonCall/Z6DIG/src/jlwrap/any.jl:167:30
 [2] <module>
   @ REPL[16]:1:1
Stacktrace:
 [1] pythrow()
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/err.jl:94
 [2] errcheck
   @ ~/.julia/packages/PythonCall/Z6DIG/src/err.jl:10 [inlined]
 [3] pycallargs(f::Py, args::Py)
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/abstract/object.jl:153
 [4] pycall(::Py, ::Py, ::Vararg{Py}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/abstract/object.jl:171
 [5] pycall
   @ ~/.julia/packages/PythonCall/Z6DIG/src/abstract/object.jl:161 [inlined]
 [6] #_#11
   @ ~/.julia/packages/PythonCall/Z6DIG/src/Py.jl:330 [inlined]
 [7] Py
   @ ~/.julia/packages/PythonCall/Z6DIG/src/Py.jl:330 [inlined]
 [8] pyexec(::Type{Nothing}, code::Py, globals::Module, locals::NamedTuple{(:Person,), Tuple{Py}})
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/concrete/code.jl:98
 [9] top-level scope
   @ ~/.julia/packages/PythonCall/Z6DIG/src/concrete/code.jl:146

More context

I'm trying to use the picamera Python module and process each frame in Julia. The following MWE however segfaults:

ENV["JULIA_PYTHONCALL_EXE"] = "/usr/bin/python3" # I have to include this due to https://github.com/cjdoris/PythonCall.jl/issues/120

using PythonCall

SideEffect = pyclass("SideEffect", (), 
                   __init__ = () -> nothing,
                   write = s -> println(rand()),
                   flush = () -> print("done")
                  )

@pyexec (SideEffect = SideEffect) => """
import picamera
with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264') # start_recording calls SideEffect.write for each frame and SideEffect.flush at the end
    camera.wait_recording(5)
    camera.stop_recording()
"""

[cjdoris]

__init__ = pymethod(self -> self.state=0) should work.

What's happening is that those methods you were defining are callable wrappers around Julia values, but are not proper Python functions, and only proper Python functions become class methods, everything else are just ordinary class members. The pymethod function converts any callable to a proper Python function.

[cjdoris]

Possibly pyclass could be changed to automatically do this wrapping for you on any Julia functions.

[yakir12]

Wrapping functions in pymethod fixed the toy example:

Person = pyclass("Person", (); 
                 __init__ = pymethod((self) -> (self.state = 0; nothing)), 
                 add = pymethod((self, n) -> (self.state += n)), 
                 print = pymethod((self) -> print(self.state))
                )
@pyexec (Person = Person) => """
p1 = Person()
p1.add(4)
p1.print() # prints 4
"""

pyclass could be changed to automatically do this wrapping for you on any Julia functions

Yeah, that would make it easier to use I think (could apply the wrapping automatically depending on the type of the arguments).

But my main use case still segfaults:

ENV["JULIA_PYTHONCALL_EXE"] = "/usr/bin/python3" # I have to include this due to https://github.com/cjdoris/PythonCall.jl/issues/120

using PythonCall

function finit(self)
    self.i = 0
    nothing # aparently, __init__ functions need to return "None"
end
function fwrite(self, s)
    self.i += 1
    println(self.i)
end
fflush(self) = print("Done")

SideEffect = pyclass("SideEffect", (), 
                     __init__ = pymethod(finit),
                     write = pymethod(fwrite),
                     flush = pymethod(fflush)
                  )

@pyexec (SideEffect = SideEffect) => """
import picamera
with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264')
    camera.wait_recording(5)
    camera.stop_recording()
"""

I realize this might be a bit hard to debug and outside this specific issue, but do you have any idea what might be going on...? The relevant documentation for picamera is here if that helps... Really appreciate any insight you might have.

[yakir12]

I imagine the following is near impossible to replicate without an RPI + Pi Camera, but here is a leaned down example:
This works:

@pyexec """
import picamera

class SideEffect(object):
    def write(self, s):
        print('a')

with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264')
    camera.wait_recording(5)
    camera.stop_recording()
"""

while the following, which is nearly identical, segfaults:

SideEffect = pyclass("SideEffect", (), write = pymethod((self, s) -> print("a")))

@pyexec (SideEffect = SideEffect) => """
import picamera

with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264')
    camera.wait_recording(5)
    camera.stop_recording()
"""

[cjdoris]

I can't really help without a MWE that works on Windows, Mac or Linux I'm afraid.

As a quick check, the following does work for me, so something non-trivial is going on:

SideEffect = pyclass(....)
@pyexec SideEffect => "SideEffect().write(12)"

[kdheepak]

If you want to hack it, you can dynamically add any function as a method to the class.

>>> from types import MethodType

>>> class Foo:
...     pass
... 

>>> f = Foo()
 
>>> def method(self):
...     print("hello world")
... 

>>> Foo.method = MethodType(method, Foo)

>>> f.method()
hello world

[yakir12]

I really appreciate the ideas (and I'll see if there is some mechanism to test this on systems that are not an RPI+PiCam), but unfortunately, the following also segfaults:

@pyexec """
global SideEffect
class SideEffect(object):
    pass
"""

fun = pymethod((self, s) -> println("frame"))

@pyexec (write = fun) => """
from types import MethodType
import picamera

SideEffect.write = MethodType(write, SideEffect)

with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264')
    camera.wait_recording(5)
    camera.stop_recording()
"""

Let me know @kdheepak if this implementation wasn't what you meant.

[kdheepak]

I'm not particularly familiar with the Python C API but maybe you can try this instead:

@pyexec """
global SideEffect
class SideEffect(object):
    pass
"""

fun = pymethod((self, s) -> println("frame"))

@pyexec (write = fun) => """
from types import MethodType
import picamera

SideEffect.write = write

with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264')
    camera.wait_recording(5)
    camera.stop_recording()
"""

pymethod seems to be calling PyInstanceMethod_New (CPython documentation):

https://github.com/cjdoris/PythonCall.jl/blob/781d7fc7954d8fbc5f68db244c1640e5bb412589/src/concrete/method.jl#L8

But my suggestion was for a Python function. I'm not seeing a pyfunc equivalent to pymethod in the documentation of this package though.

Does it segfault when you try running the code without using the picamera package?

[cjdoris]

Try the version with MethodType but without wrapping the Julia function with pymethod? I believe MethodType can take any callable.

[yakir12]

you can try this instead

Yeah, it segfaulted as well.

without wrapping the Julia function with pymethod

Segfault too.

I'm not seeing a pyfunc equivalent to pymethod

Indeed, might a pyfunc implementation fix this you think?

Does it segfault when you try running the code without using the picamera package?

Nope...

[yakir12]

The one constant is if the class was defined outside Python then it segfaults.

[cjdoris]

How about if you add a pure python method to the class using MethodType?

[yakir12]

Specifically

@pyexec """
from types import MethodType
import picamera

class SideEffect(object):
    pass

def write(self, s):
    print("frame")

SideEffect.write = MethodType(write, SideEffect)

with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264')
    camera.wait_recording(5)
    camera.stop_recording()
"""

Works.

[yakir12]

I can't really help without a MWE that works on Windows, Mac or Linux I'm afraid.

I found https://pypi.org/project/fake-rpi/ and while it does fake picamera it's really basic (with just 3 of the library's methods, neither of which are start_recording), so I doubt it'll do much good...

[cjdoris]

I meant to define the class from Julia but the method from Python.