Interoperability subclassing Java classes with Python classes is poor

[mwsealey]

Tested version: 25.0.0 ea.04 (CE and Oracle JVMs)

(apologies for the length in advance)

We are running an experiment to investigate whether we can replace or augment our embedded Python technology in products with Graal. Our product has significant amounts of legacy Jython code -- a good amount is not written by us and we must maintain some customer compatibility with minimal changes -- and obviously both our customers and our internal teams therefore expect some Jython semantics to be in place. We've accepted there may be significant differences but we are confident that we can work around the vast majority of them with decorators or modules. However the code relies on some behavior that is not present. These all might be the same issue underneath.

Java exception handling

On previous versions (~24.1) this was simply not working - the objects you got back from:

try:
    raise JavaException
except JavaException as e:
    print(type(e).mro())

.. or thrown by internal Java classes were not valid objects and had strange behavior in that it was a ForeignObject etc. that had none of the expected BaseException semantics, nor as it would have been in Jython, been a carbon copy of the Java Exception (up to and including any custom methods).

We are very pleased with the changes moiving from 24.1.1 to 25.0.0 ea.04 in that now we do not need to supply --python.EmulateJython=true to get anything to work, and that we get back as 'e' a very Pythonic BaseException-based object.

Unfortunately we have exceptions thrown by our own internal classes which have custom methods (e.g. OurException.getErrorCode() returns a number as encoded in the message string, since we have a library that acts a little bit like strerror() and comes up with a much more productive string. Don't judge.. it is an old bit of code). There seems to be no way to fetch the original Java exception object or access it as we would in Jython here.

Note that the documentation here doesn't seem correct:

https://github.com/oracle/graalpython/blob/c208cfb8984b4c650c86aea92beef7b2ceb36725/docs/user/Python-on-JVM.md#exceptions-from-java

.. in that

a) --python.EmulateJython=true is not necessary at all (we like this behavior, although either way b occurs)
b) e.getMessage() does not work in the example as e is not the original Java exception anymore.

The code in the example in the docs is exactly our litmus test for it.

Given that we want to call random unhealthy methods on Exception classes which is not very standard, Pythonic or Java-ic of us, this is something we would be happy to accept needs a little bit of porting. We did expect that e.super or something of that ilk would give us the original class but it does not.

Summary: Semantics are different from Jython in that the Exception object caught does not have the methods or members of the original Java Exception object. There is no way to walk the MRO or find the original object to access those methods.

Java subclassing

The documentation here (commit-ish is the master branch as of today):

https://github.com/oracle/graalpython/blob/c208cfb8984b4c650c86aea92beef7b2ceb36725/docs/user/Python-on-JVM.md#inheritance-from-java

Has a very confusing example. We have found that while this example works fine, there are two major issues (and one minor) with it.

a) (Minor) This is absolutely not compatible with Jython, although this is stated in the documentation. We also noted that --python.EmulateJython does not change the behavior at all implying this is not a Jython-emulation-specific feature in Graal. In this code:

from com.example.h2g2 import DeepThought

class MyDeepThought(DeepThought):
    def __init__(self):
        self.answer = 42

    def reveal_answer(self):
        print(f"the answer is... {self.answer}")

dt = MyDeepThought()
dt.reveal_answer()

Does not work. If you attempt to call a method on the Python side, you can't because it is actually

dt.this.reveal_answer()

This alteration does work, and in fact we think we could make a decorator to smooth this out a bit. It goes downhill from here.

b) Augmenting the class further we have to use our imagination and say that the original Java class has a simple member "question" and a method "guide_entry":

class MyDeepThought(DeepThought):
    def __init__(self):
        self.answer = 42
        # self.__super__.guide_entry("HARMLESS") # NoneType is not callable
        # self.__super__.__init__() # NoneType is not callable
        super().__init__() # this works, though
        self.guide_entry("HARMLESS") # this does too

    def guide_entry(self, arg): # this is overriding a method on Java class DeepThought
        print(f"{arg}")
        self.__super__.guide_entry(arg) # works ok?

    def reveal_answer(self):
        print(f"the answer is... {self.answer}")

    def reveal_question(self):
        # qu = self.this.question # wrong, AttributeError: 'PythonJavaExtenderClass' object has no attribute 'this'
        # qu = self.question # wrong, AttributeError: 'PythonJavaExtenderClass' object has no attribute 'answer'
        print(f"the question is... ") # should print: "how many roads must a man walk down?"

dt = MyDeepThought()
dt.guide_entry("MOSTLY HARMLESS") # prints "MOSTLY HARMLESS" and then calls the original Java method (yay!)
dt.this.reveal_answer()
dt.this.reveal_question() # will cause exception as above depending on impl.
print(f"{dt.question}") # will print the member value as intended

Essentially, we cannot access any Java members from inside the class, but methods are fine.

Those members are published outside the class definition on the object as constructed.

This feels as if the class is created, and then GraalPy is wrapping the class and adding a few members (graalpython.build_java_class() or .extend()?) but the user (whoever owns the 'dt' instance above) and the class (receiving 'self') are getting two different objects or different wrappings? Whatever the cause, it is inconsistent.

c) Additionally we can only call super methods which match the name of the subclassed function? Understandably having one function call a superclass method for a different method is a little strange but that is a programmer problem and we're not sure it should be enforced by the "language" in that you can do it in CPython.

If we try and run something from the __init__ method we also find that self.__super__ is None. Luckily, super() works. This is a deviation from the documentation (which is very good, we like that it works like Python).

I have not had time to express all the possibilities here in code to make sure.

Jython functionality might not be able to be fine-grained enabled/disabled

We are very pleased with the direction this is all taking except for the issues above. However we have to express a little fear of progress here, if features like "not having to EmulateJython=true for either of the above to be possible" are not intentional and this is all going to be hidden behind that feature, we really would like to have a more fine-grained enablement. As far as we're aware the Jython emulation has traditionally intended to bring in 3 features:

• Features like jarray (although this has always seemingly worked without emulation ✅ )
• Subclassing Java classes (although this only works properly in 25.0.0, and works without the emulation now ✅ )
• Providing Exceptions (although this only works in properly 25.0.0, and works without the emulation now ✅ )
• Exposing a renamed property when it detects a getter or setter in a class, e.g. if getMyInternalVariable() and setMyInternalVariable() exist then myclass.myInternalVariable will also exist (if get*() exists alone then it is read-only). ❌

We do not like the last feature as it confuses the namespace and in general makes our docs look incomplete, and we would like to disable such functionality while keeping others if at all possible. We considered patching Jython to disable this feature about a decade ago but it didn't seem worth the trouble to maintain it. In that sense, we would not like to see it back. We're looking for a very "clean Python-like" environment with implicit Java compatibility and no API changes.

I suppose this is a plea to see if we can find out if there is a goal for this or a JSR or JEP which covers what the API should look like, and what the behavior is. We are happy to work with the team here in figuring this out, so that we can plan accordingly and come back at a future version, or test more in the meantime.

[mwsealey]

from __future__ import print_function, division, absolute_import, unicode_literals

if __graalpython__:
    if __graalpython__.jython_emulation_enabled:
        from com.example.code import MiscUtils
    else:
        import java
        MiscUtils = java.type("com.example.code.MiscUtils")
    print(f"Emulating Jython: {__graalpython__.jython_emulation_enabled}")
else:
    from com.example.code import MiscUtils
    print("Is Jython!")


class SuperTest(object):
    def method_man(self):
        pass

    def madness_method(self):
        pass

class SuperSuperTest(SuperTest):
    def method_man(self):
        pass

print(f"=== Subclass testing (Python Pure)")

st = SuperSuperTest()
if not st.method_man and not st.madness_method:
    print(f" X  SuperSuperTest doesn't have it's methods (inconceivable!)")

print(f" i  SuperSuperTest() has mro: {type(st).mro()}")

sts = super(SuperSuperTest, st)
print(f"    super(SuperSuperTest, st) has mro: {type(sts).mro()}")
if sts.method_man:
    print(f" v  super(SuperSuperTest, st).method_man exists")

print(f" i  \\ sts subclass of SuperTest?: {issubclass(SuperSuperTest, SuperTest)}")

class Misc(MiscUtils):
    def __init__(self):
        # the Java class has no constructor so does this even run?
        self.misc_value = 42
        print(f"    Misc.__init__: self.__super__ is {type(self.__super__)}")

    def numIntsNeededToStoreBytes(self, byteCount):
        print(f"== Misc(MiscUtils).numIntsNeededToStoreBytes({byteCount=})")
        x = None
        try:
            # self.__super__ mro() seems to be [<class 'polyglot.ForeignObject'>, <class 'object'>]
            x = self.__super__.mask(width)
        except Exception as e:
            print(f" X  tried self.__super__.numIntsNeededToStoreBytes(): {e}")

        try:
            # super() mro() seems to be [<class 'super'>, <class 'object'>]
            x = super().numIntsNeededToStoreBytes(width)
        except Exception as e:
            print(f" X  tried super().numIntsNeededToStoreBytes() {e}")

        try:
            # 'PythonJavaExtenderClass' object has no attribute 'this'
            x = self.this.numIntsNeededToStoreBytes(width)
        except Exception as e:
            print(f" X  tried self.this.numIntsNeededToStoreBytes() {e}")

        try:
            # 'super' object has no attribute 'this'
            x = super().this.numIntsNeededToStoreBytes(width)
        except Exception as e:
            print(f" X  tried super().this.numIntsNeededToStoreBytes() {e}")

        if x is not None:
            return x

        print(f" X  calling all tests for super method failed :(")
        
        return (byteCount + 3) // 4

    def py_native_test_bit(self, val, bit):
        return val & (1 << bit)

    def py_call_numIntsNeededToStoreBytes(self, byteCount):
        print(f"  -->   inside py_call_numIntsNeededToStoreBytes")
        _ = self.misc_value   # inconsistent with outside the class
        y = self.numIntsNeededToStoreBytes(10)
        try:
            x = self.py_native_test_bit(y if y else 7, 1)
        except Exception as e:
            pass # I already know it works
        else:
            print(f"        self.py_native_test_bit() worked (inside py_call_numIntsNeededToStoreBytes)")

        print(f"  <--   inside py_call_numIntsNeededToStoreBytes -- {x=} {y=}")
        return y if x else None

misc = Misc()

print("=== By Method")

try:
    x = misc.this.mask(10)
    print(f"    misc.this.mask() worked")
except Exception as e:
    print(f" X  tried misc.this.mask(10): {e}")

try:
    x = misc.mask(10)
    print(f"    misc.mask() worked")
except Exception as e:
    print(f" X  tried misc.mask(10): {e}")

try:
    x = misc.this.py_call_numIntsNeededToStoreBytes(10)
    print(f"    misc.this.py_call_numIntsNeededToStoreBytes() worked")
except Exception as e:
    print(f" X  misc.this.py_call_numIntsNeededToStoreBytes(): {e}")

try:
    x = misc.py_call_numIntsNeededToStoreBytes(10)
    print(f"    misc.py_call_numIntsNeededToStoreBytes() worked")
except Exception as e:
    print(f" X  misc.py_call_numIntsNeededToStoreBytes(): {e}")

try:
    x = misc.this.py_native_test_bit(7, 1)
    print(f"    misc.this.py_native_test_bit() worked")
except Exception as e:
    print(f" X  misc.this.py_native_test_bit(): {e}")

try:
    x = misc.py_native_test_bit(7, 1)
    print(f"    misc.py_native_test_bit() worked")
except Exception as e:
    print(f" X  misc.py_native_test_bit(): {e}")

try:
    f = misc.extractBits(7, 1, 2)
    print(f"    misc.extractBits() worked")
except Exception as e:
    print(f" X  misc.extractBits(): {e}")

try:
    f = misc.this.extractBits(7, 1, 2)
    print(f"    misc.this.extractBits() worked")
except Exception as e:
    print(f" X  this.extractBits(): {e}")

try:
    f = misc.__super__.extractBits(7, 1, 2)
    print(f"    misc.__super__.extractBits() worked")
except Exception as e:
    print(f" X  misc.__super__.extractBits(): {e}")

try:
    f = super(Misc, misc).extractBits(7, 1, 2)
    print(f"    super(Misc, misc).extractBits() worked")
except Exception as e:
    print(f" X  super(Misc, misc).extractBits(): {e}")

try:
    f = misc.misc_value
    print(f"    misc.misc_value is {f}")
except Exception as e:
    print(f" X  misc.misc_value: {e}")

try:
    f = misc.this.misc_value
    print(f"    misc.this.misc_value is {f}")
except Exception as e:
    print(f" X  misc.this.misc_value: {e}")

print(f" i  misc has mro(): {type(misc).mro()}")
print(f" i  \\ misc subclass of MiscUtils?: {issubclass(MiscUtils, Misc)}")

exit()

The MiscUtils class is pretty simple in Java (this is production code):

public abstract class MiscUtils {

    public static int numIntsNeededToStoreBytes(int byteCount) {
        return (byteCount + 3) / 4;
    }

    public static long extractBits(long from, int lsb, int width) {
        return (from >>> lsb) & mask(width);
    }
}

So, we've got two functions in Java, numIntsNeededToStoreBytes() and extractBits()

in the Python we add py_native_test_bit() which is pure Python, py_call_numIntsNeededToStoreBytes() which is pure Python and attempts to call self.numIntsNeededToStoreBytes(), and obviously subclass numIntsNeededToStoreBytes() which tries to attain it's super but emulates the functionality of the super doesn't exist (because I know we can't resolve it).

This gives us the results which I think are surprising as it really means subclassing is VERY limited:

Emulating Jython: False
=== Subclass testing (Python Pure)
 i  SuperSuperTest() has mro: [<class '__main__.SuperSuperTest'>, <class '__main__.SuperTest'>, <class 'object'>]
    super(SuperSuperTest, st) has mro: [<class 'super'>, <class 'object'>]
 v  super(SuperSuperTest, st).method_man exists
 i  \ sts subclass of SuperTest?: True
    Misc.__init__: self.__super__ is <class 'NoneType'>
=== By Method
== Misc(MiscUtils).numIntsNeededToStoreBytes(byteCount=10)
 X  tried self.__super__.numIntsNeededToStoreBytes(): foreign object has no attribute 'numIntsNeededToStoreBytes'
 X  tried super().numIntsNeededToStoreBytes() 'super' object has no attribute 'numIntsNeededToStoreBytes'
 X  tried self.this.numIntsNeededToStoreBytes() 'PythonJavaExtenderClass' object has no attribute 'this'
 X  tried super().this.numIntsNeededToStoreBytes() 'super' object has no attribute 'this'
 X  calling all tests for super method failed :(
    misc.this.numIntsNeededToStoreBytes() worked
 X  tried misc.numIntsNeededToStoreBytes(10): foreign object has no attribute 'numIntsNeededToStoreBytes'
  -->   inside py_call_numIntsNeededToStoreBytes
== Misc(MiscUtils).numIntsNeededToStoreBytes(byteCount=10)
 X  tried self.__super__.numIntsNeededToStoreBytes(): foreign object has no attribute 'numIntsNeededToStoreBytes'
 X  tried super().numIntsNeededToStoreBytes() 'super' object has no attribute 'numIntsNeededToStoreBytes'
 X  tried self.this.numIntsNeededToStoreBytes() 'PythonJavaExtenderClass' object has no attribute 'this'
 X  tried super().this.numIntsNeededToStoreBytes() 'super' object has no attribute 'this'
 X  calling all tests for super method failed :(
        self.py_native_test_bit() worked (inside py_call_numIntsNeededToStoreBytes)
  <--   inside py_call_numIntsNeededToStoreBytes -- x=2 y=3
    misc.this.py_call_numIntsNeededToStoreBytes() worked
 X  misc.py_call_numIntsNeededToStoreBytes(): foreign object has no attribute 'py_call_numIntsNeededToStoreBytes'
    misc.this.py_native_test_bit() worked
 X  misc.py_native_test_bit(): foreign object has no attribute 'py_native_test_bit'
 X  misc.extractBits(): foreign object has no attribute 'extractBits'
 X  this.extractBits(): 'PythonJavaExtenderClass' object has no attribute 'extractBits'
 X  misc.__super__.extractBits(): foreign object has no attribute '__super__'
 X  super(Misc, misc).extractBits(): super(type, obj): obj must be an instance or subtype of type
 X  misc.misc_value: foreign object has no attribute 'misc_value'
    misc.this.misc_value is 42
 i  misc has mro(): [<class 'polyglot.ForeignObject'>, <class 'object'>]
 i  \ misc subclass of MiscUtils?: False
>>> dir(misc)
dir(misc)['equals', 'hashCode', 'super$equals', 'super$hashCode', 'super$toString', 'this', 'toString']
>>> dir(MiscUtils) # result redacted
dir(MiscUtils)[... 'extractBits', 'numIntsNeededToStoreBytes', ...]
>>> dir(Misc)
dir(Misc)['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__truffle_richcompare__', '__weakref__']
>>> dir(misc.this)
dir(misc.this)['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__super__', '__truffle_richcompare__', '__weakref__', 'misc_value', 'numIntsNeededToStoreBytes', 'py_call_numIntsNeededToStoreBytes', 'py_native_test_bit']

What I glean from this is:

• When defining the class, you can access instance members defined in init and methods, e.g. self.misc_value or self.py_native_test_bit(), as you would in Python which is contrary to the documentation
• When using the class instance, you can access instance members and methods from self.this.misc_value or self.this.py_native_test_bit() which is odd but documented
• Class methods cannot call the Java super methods any way I tried (e.g. super().numIntsNeededToStoreBytes() variations)
• -- I should note that I have another test where super().__init__() actually worked inside __init__ but this seems to be the only one it proxies
• -- this is contrary to the statement in the documentation that "The created object does not behave like a Python object but instead in the same way as a foreign Java object" (shouldn't it proxy to the Java object or have I misunderstood this? I am certain it did so in 24.1.1)
• Use of self.__super__ per the docs is confusing as it is None in __init__ (but super() does the right thing for __init__) which is inconsistent
• Users of the class instance cannot call the Java methods any way I tried (e.g. misc.extractBits() variations)
• super(X, A) does not work where X is a Java class since A is not a subclass of X, it's a ForeignObject, and obviously isinstance and issubclass do not operate correctly to confirm types.
• -- do I need to add interop behavior for every Java class I need to subclass to work around this?
• The behavior is the exact same regardless of the value of --python.EmulateJython

Would that be a fair assessment, and if so which are not intended and which are to be Graal behaviors for the 25.x milestone, if it is possible to make that judgment?

[msimacek]

Thank you for the feedback.

Re: Java exception handling. The problem with calling methods on a Java exception sounds to me like a bug, I would expect this to just work. @eregon could you please have a look (and also fix the docs)?

Re: Java subclassing. Subclassing Java classes is notoriously difficult, we'll need more time to evaluate what we can improve.

Re: Jython functionality might not be able to be fine-grained enabled/disabled.
Right now, EmulateJython seems to do just two things:

• Add the implicit getters/setters
• Enable import to directly import Java classes other than java.*

I think we can add two options that enable those things individually and then have EmulateJython as a shortcut to turn on both.

[timfel]

Unfortunately we have exceptions thrown by our own internal classes which have custom methods (e.g. OurException.getErrorCode() returns a number as encoded in the message string, since we have a library that acts a little bit like strerror() and comes up with a much more productive string. Don't judge.. it is an old bit of code). There seems to be no way to fetch the original Java exception object or access it as we would in Jython here.

As of 25.0.0-ea.05 this now works:

>>> import java.io.IOException
>>> try:
...   raise java.io.IOException("foobar")
... except BaseException as e:
...   x = e
...
>>> x
ForeignException('java.io.IOException: foobar')
>>> x.getMessage()
'foobar'

About subclassing: you are basically right, we create actually a compound from a Java object and a Python object, and they reference each other (using .this (Java->Python) and .__super__ (Python->Java))

When defining the class, you can access instance members defined in init and methods, e.g. self.misc_value or self.py_native_test_bit(), as you would in Python which is contrary to the documentation

You are defining a Python class, instances of which are used as a delegate from a Java object. You cannot access these members from Java, but from the Python defined code on the self you can.

When using the class instance, you can access instance members and methods from self.this.misc_value or self.this.py_native_test_bit() which is odd but documented

Yes, the instance is of a generated Java subclass of the Java superclass or interface you specify, and it automatically delegates to an instance of a Python class that you defined with your method. But to get back to Java, you use self.__super__

Class methods cannot call the Java super methods any way I tried (e.g. super().numIntsNeededToStoreBytes() variations)

Static members are not exposed on the instance. You can use Misc().getClass().static.numIntsNeededToStoreBytes() or (when already inside a Python method), self.__super__.getClass().static.numIntsNeededToStoreBytes()

-- I should note that I have another test where super().init() actually worked inside init but this seems to be the only one it proxies

That super().__init__() just calls the Python object.__init__ which is empty.

-- this is contrary to the statement in the documentation that "The created object does not behave like a Python object but instead in the same way as a foreign Java object" (shouldn't it proxy to the Java object or have I misunderstood this? I am certain it did so in 24.1.1)

Inside the Python-defined methods, you are in a Python object. From the outside, however, i.e., what the Misc() constructor returns, that's a Java object.

Use of self.super per the docs is confusing as it is None in init (but super() does the right thing for init) which is inconsistent

super() does not do the right thing. We will add to the docs that __init__ cannot access __super__.

Users of the class instance cannot call the Java methods any way I tried (e.g. misc.extractBits() variations)

You can call the Java methods, just not static methods. See above, you need to go via getClass().static.extractBits

super(X, A) does not work where X is a Java class since A is not a subclass of X, it's a ForeignObject, and obviously isinstance and issubclass do not operate correctly to confirm types.
-- do I need to add interop behavior for every Java class I need to subclass to work around this?

Yes, the super object does not work. We will document that you cannot use it in this context.

The behavior is the exact same regardless of the value of --python.EmulateJython

Yes, this is not governed by that flag.

---

If you want the self in the Python methods to behave more like you want, I would recommend you implement __getattr__ and check self.__super__ and the superclass for members as well.
Then this should work:

public abstract class MiscUtils {

    public String someInstanceMethod() {
        return "an instance method on the Java side";
    }

    public static int mask(int v) {
        return v;
    }

    public static int numIntsNeededToStoreBytes(int byteCount) {
        return (byteCount + 3) / 4;
    }

    public static long extractBits(long from, int lsb, int width) {
        return (from >>> lsb) & mask(width);
    }
}

import java
java.add_to_classpath(".")
MiscUtils = java.type("MiscUtils")


class Misc(MiscUtils):
    java_static_methods = MiscUtils

    def __init__(self):
        print("Misc.__init__ runs to initialize the Python object, still without a connection to the Java side")
        self.misc_value = 42

    def __getattr__(self, name):
        if hasattr(self.__super__, name):
            return getattr(self.__super__, name)
        if hasattr(self.java_static_methods, name):
            return getattr(self.java_static_methods, name)
        raise AttributeError(name)

    def numIntsNeededToStoreBytes(self, byteCount):
        print(f"== Misc(MiscUtils).numIntsNeededToStoreBytes({byteCount=}) overrides a Java method, so calls to the Java object go here")
        return self.java_static_methods.numIntsNeededToStoreBytes(byteCount)

    def py_native_test_bit(self, val, bit):
        return val & (1 << bit)

    def py_call_numIntsNeededToStoreBytes(self, byteCount):
        print(f"  -->   inside py_call_numIntsNeededToStoreBytes, which is not callable from Java at all")
        _ = self.misc_value
        y = self.numIntsNeededToStoreBytes(10)
        x = self.py_native_test_bit(y if y else 7, 1)
        return y if x else None

misc = Misc()

print(f"{misc.getClass().static.mask(10)=}")
print(f"{misc.this.mask(10)=}")
print(f"{misc.this.py_call_numIntsNeededToStoreBytes(10)=}")
print(f"{misc.this.py_native_test_bit(7, 1)=}")
print(f"{misc.getClass().static.extractBits(7, 1, 2)=}")
print(f"{misc.this.extractBits(7, 1, 2)=}")
print(f"{misc.this.misc_value=}")
print(f"{misc.someInstanceMethod()=}")
print(f"{type(misc).mro()=}")
print(f"{issubclass(Misc, MiscUtils)=}")
print(f"{isinstance(Misc(), MiscUtils)=}")

 > python misc.py
Misc.__init__ runs to initialize the Python object, still without a connection to the Java side
misc.getClass().static.mask(10)=10
misc.this.mask(10)=10
  -->   inside py_call_numIntsNeededToStoreBytes, which is not callable from Java at all
== Misc(MiscUtils).numIntsNeededToStoreBytes(byteCount=10) overrides a Java method, so calls to the Java object go here
misc.this.py_call_numIntsNeededToStoreBytes(10)=3
misc.this.py_native_test_bit(7, 1)=2
misc.getClass().static.extractBits(7, 1, 2)=2
misc.this.extractBits(7, 1, 2)=2
misc.this.misc_value=42
misc.someInstanceMethod()='an instance method on the Java side'
type(misc).mro()=[<class 'polyglot.ForeignObject'>, <class 'object'>]
issubclass(Misc, MiscUtils)=False
Misc.__init__ runs to initialize the Python object, still without a connection to the Java side
isinstance(Misc(), MiscUtils)=True

[timfel]

We will work to make this nicer for 25.0, but for 24.2 (which is due out next and feature freezes this week) the above may be the best we can do right now.

[mwsealey]

I should note that I have another test where super().init() actually worked inside init but this seems to be the only one it proxies

That super().init() just calls the Python object.init which is empty.

I meant I don't think it is broken in that that wouldn't be the case where the Java class had a constructor?

I have another class which has three constructors all with different numbers of parameters (there's a 0, 2 and 4 argument version), and calling super().__init__ with each of the three works perfectly - I get the class back I expect in that when I am in real code it does this. The class itself in Java is just storage like a Python dataclass, a bunch of public members and the constructor just plumbs them in. Lets assume that plumbing is stashing the parameters in the members and we add pairs of them together under other members (so the Java has three constructors and members p1, p2, p3, p4, one_plus_two, three_plus_four all assigned in the constructors):

public class JavaCP
{
    public JavaCP()
    {
    }

    public JavaCP(int p1, int p2)
    {
        this.p1 = p1;
        this.p2 = p2;
    }

    public JavaCP(int p1, int p2, int p3, int p4)
    {
        this.p1 = p1;
        this.p2 = p2;
        this.p3 = p3;
        this.p3 = p4;
        this.one_plus_two = p1 + p2;
        this.three_plus_four = p3 + p4;
    }

    public int p1;
    public int p2;
    public int p3;
    public int p4;
    public int one_plus_two;
    public int three_plus_four;
}

class CP(JavaCP):
    def __init__(self, p1, p2, p3, p4):
        if p1 is None or p2 is None:
            super().__init__()
        elif p3 is none or p4 is None:
            super().__init__(p1, p2)
        else:
            super().__init__(p1, p2, p3, p4)

w = CP()
print(w.p1, w.p2, w.one_plus_two) # prints ~0 0 0
print(w.p3, w.p4, w.three_plus_four) # prints ~0 0 0
x = CP(1, 2)
print(x.p1, x.p2, x.one_plus_two)  # prints 1 2 3
print(x.three_plus_four)
y = CP(1,2,3,4)
print(y.one_plus_two)
print(y.p3, y.p4, y.three_plus_four) # prints 3 4 7

It worked perfectly!

Where I found the differences here in the first place as the end code I had to write here was confusing in that to access the Java members I could just use {w,x,y}.name_of_attr but to add new members from Python they ended up in {w,x,y}.this.name_of_attr, even if they had the same name. This isn't how it usually works in Python in that you expect 'instance.anything' to "follow the mro and find the first one that responds favorably e.g. to __getattr__('anything')" (I know it is far more complicated than that but this is the illusion presented to the user).

Then I realized I couldn't actually access any of the Java members inside the class definition, e.g. I wanted to provide replacements for p3 and p4 while still calling the second constructor variant and e.g. create a Python function to modify one of the members

class CP(JavaCP):
    def __init__(self, p1, p2, p3, p4):
        self.newp4 = p4
        if p1 is None or p2 is None:
            super().__init__()
        elif p3 is none or p4 is None:
            super().__init__(p1, p2)
            self.p3 = 9   # exception, self has no attribute 'p3'
            self.p4 = 10 # exception, self has no attribute 'p4'
        else:
            super().__init__(p1, p2, p3, p4)

    def skibidi(self, newp4):
        self.newp4 = newp4 # exception, self has no attribute 'newp4'   

cp = CP(1, 2)

I figured okay, .this and .__super__ should be how I am doing this which is rather on the nose, but neither of them work in the class definition at all. To call skibidi:

cp.this.skibidi(10)

Having this over in cp.this made no sense to me in Python and it is even more confusing given that you just said "it isn't callable from Java". Having it cause an exception because self.newp4 didn't exist (and you can't access self.this.newp4 inside skibidi()) sent me for a coffee :D

For what it's worth, Jython isn't a lot better - first of all it uses the crusty Python 2.x super() semantics in that there is no no-args version of it, you have to be explicit. Historically it hasn't worked well, or been documented differently, so most people have code that looks like this[1]:

class CP(JavaCP):
    def __init__(self, p1, p2, p3, p4):
        JavaCP.__init__(self, p1, p2, p3, p4):

# or more modern...

class CP(JavaCP):
    # ...
    def skibidi_java(self, foo):
        super(JavaCP, self).skibidi_java(foo)

And this is fine, while in Python this is wildly inconvenient to hardcode your parent class, in Jython multiple inheritance is restricted (because it isn't possible in Java).

As far as I could tell in my code here, this doesn't work either - calling JavaCP.init is an invalid instantiation of foreign object.

I do confuse myself sometimes as I am always going back to "super() considered super!"[2] (there are two recipes[3][4] linked at the bottom with the two syntaxes, and then I cross-reference all of the Jython scripts I have to hand), since I can never remember whether I provide self or not to the named class vs. super(CP, self).init(p1...), what the parameter types need to be, but it doesn't work any which way as far as I tested.

Please don't let this come across as a disagreement, since I am struggling to keep in mind 3 implementations of a language (CPython 2, CPython 3 and Jython) and now a fourth I'm trying to get some clarity.

Your workaround via__getattr__ is great and I'll use it for the time being just to make progress, but are we saying it won't be needed or eventually be better? I had assumed that __super__ was literally the proxy object that would have the Java class in it at that point, and I am not sure I understand why it isn't or what it is. I think I understand that CP() returns a proxy object with CP().this pointing at the Python object and CP().__super__ at a proxy for the Java object. I ask because at some point I'll probably need to document the difference in behavior, at least internally, so it's best I grok it.

If you need to literally override __getattr__ forever, I am fine with THAT, too, so my question there is what is self.__super__ at that point if not a sort of proxy object of unbound methods wrapping a Java class (as assumed para above) and why would there need to be self.__super__ and self.java_static_methods to the same end? I suppose I assumed self.__super__ was the raw meat and super() would further wrap __super__ into the super() object so you'd get a double indirection.

What I would love to see is either one of rhettinger's examples working where e.g. Shape, Root were Java and the rest of the Python effectively unchanged. I am not sure I care if it looks like the Python 2 version or the CPython 3.x version, or the referenced dirt-old Jython docs, or the new Jython docs[5], but the resultant code is the same for all of them with no code changes besides that little bit of extra sugar for finding parent classes. As long as the raw meat of the interfacing stays inside the class definition and not in users of the instances?

Thank you in advance however it goes :)

[1] https://www.jython.org/jython-old-sites/archive/21/docs/subclassing.html
[2] https://rhettinger.wordpress.com/2011/05/26/super-considered-super/
[3] http://code.activestate.com/recipes/577720-how-to-use-super-effectively/
[4] https://code.activestate.com/recipes/577721-how-to-use-super-effectively-python-27-version/
[5] https://jython.readthedocs.io/en/latest/JythonAndJavaIntegration/#using-java-within-jython-applications