Issue1238

classification
Title: randomly select() exception on 2.5b1
Type: Severity: normal
Components: Versions: 2.5.1, 2.5b1, 2.5b0
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: amak Nosy List: amak, billiejoex
Priority: normal Keywords:

Created on 2009-01-14.15:30:48 by billiejoex, last changed 2009-04-03.16:49:50 by amak.

Files
File name Uploaded Description Edit Remove
ftpserver.py billiejoex, 2009-02-03.18:18:54 Modified ftpserver.py which partially works on Jython
test_ftpd.py billiejoex, 2009-02-03.18:19:48 Modified test suite which partially works on Jython
traceback.txt billiejoex, 2009-02-03.18:20:45 Test suite results
ftpserver.py billiejoex, 2009-02-04.17:20:00
test_ftpd.py billiejoex, 2009-02-04.17:20:18
Messages
msg4045 (view) Author: Giampaolo Rodola' (billiejoex) Date: 2009-01-14.15:30:48
By running pyftpdlib [1] test suite on Jython 2.5b1 I get this exception
from select().
I can't understand the reason since it happens randomly.


back (most recent call last):
  File "C:\jython2.5b0\Lib\threading.py", line 247, in _Thread__bootstrap
    self.run()
  File "test_ftpd.py", line 2003, in run
    self.server.serve_forever(timeout=self.__timeout, count=1,
  File "C:\jython2.5b0\bin\ftpserver.py", line 3126, in serve_forever
    poll_fun(timeout, map)
  File "C:\jython2.5b0\Lib\asyncore.py", line 108, in poll
    r, w, e = select.select(r, w, e, timeout)
  File "C:\jython2.5b0\Lib\select.py", line 171, in native_select
    pobj = poll()
  File "C:\jython2.5b0\Lib\select.py", line 63, in __init__
    self.selector = java.nio.channels.Selector.open()
IOException: java.io.IOException: Unable to establish loopback connection


[1] http://code.google.com/p/pyftpdlib
msg4120 (view) Author: Alan Kennedy (amak) Date: 2009-02-03.17:27:09
I downloaded pyftpdlib and tried to run the test suite.

It fails completely because pyftpdlib uses the cpython specific "buffer"
builtin, which is not available on jython.

The test suite then proceeds to hang, not sure why.

More on this later; I'm working on asyncore at the moment, and that
should be resolved before addressing this.

Have you contacted the pyftpdlib authors? Are they interested in having
their code run on jython?
msg4121 (view) Author: Alan Kennedy (amak) Date: 2009-02-03.17:28:27
See here for a good description of the buffer interface and its
problems, written by Guido himself.

http://mail.python.org/pipermail/python-dev/2000-October/009974.html
msg4122 (view) Author: Giampaolo Rodola' (billiejoex) Date: 2009-02-03.18:18:53
I'm the pyftpdlib author and even if it's not high priority I'd like to
see pyftpdlib work on Jython too.
pyftpdlib on Jython fails for many reasons.
By monkey patching such issues I managed to run the test suite despite a
lots of tests still keep failing.
Here's a list of some reasons why pyftpdlib test suite does not work on
Jython:


- Missing IPv6 support ( http://bugs.jython.org/issue1210 ) solved by
disabling the test

- asyncore.dispatcher._map attribute on Jython is deleted for no reason
( http://bugs.jython.org/issue1226 ) solved by monkey patching asyncore.py 

- socket.getservbyport not implemented (
http://bugs.jython.org/issue1225 ), solved by disabling the test

- missing buffer() function, solved by putting this piece of code
straight into ftpserver.py

try:
    buffer
except NameError:
    # hack for Jython which does not implement buffer() 
    def buffer(obj, start=None, stop=None):
        if not hasattr(obj, "__getitem__"):
            raise TypeError
        if start == None:
            start = 0
        if stop == None:
            stop = len(obj)
        return obj[start:stop]


Despite all these modifications I still get the following exception
(amongst others):
IOException: java.io.IOException: Unable to establish loopback connection

In case you want to try I attach the modified ftpserver.py and
test_ftpd.py scripts I modified to make them partially work with Jython,
and also the traceback message I get from the failing tests.

Note that I also get this error which is quite strange:


======================================================================
ERROR: test_retr_ascii (__main__.TestFtpRetrieveData)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_ftpd.py", line 1213, in test_retr_ascii
    expected = data.replace(os.linesep, '\r\n')
OutOfMemoryError: java.lang.OutOfMemoryError: Java heap space
msg4126 (view) Author: Giampaolo Rodola' (billiejoex) Date: 2009-02-04.13:16:08
I found this:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5073504
msg4131 (view) Author: Alan Kennedy (amak) Date: 2009-02-04.15:55:25
I have checked in a new version of asyncore at r6013.

Please, can you try your code again with the new version?

https://jython.svn.sourceforge.net/svnroot/jython/trunk/jython/Lib/asyncore.py
msg4132 (view) Author: Alan Kennedy (amak) Date: 2009-02-04.15:58:19
About this error:

======================================================================
ERROR: test_retr_ascii (__main__.TestFtpRetrieveData)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_ftpd.py", line 1213, in test_retr_ascii
    expected = data.replace(os.linesep, '\r\n')
OutOfMemoryError: java.lang.OutOfMemoryError: Java heap space

I notice that you're using the cpython "buffer" construct. Because
buffer is not available on jython, you're using a simple replacement,
shown above.

And now you're getting an "Out of memory" error. When you get this
error, how big is the content of data variable, i.e. what is len(data)?
msg4135 (view) Author: Giampaolo Rodola' (billiejoex) Date: 2009-02-04.16:09:39
> Please, can you try your code again with the new version?

Using 2.5 asyncore.py looks fine but I still get that select() related
error, which seems not related with asyncore itself.

> And now you're getting an "Out of memory" error. When you get this
> error, how big is the content of data variable, i.e. what is 
> len(data)?

>>> data = ('abcde12345' + os.linesep) * 100000
>>> len(data)
1200000

If I execute:

>>> expected = data.replace(os.linesep, '\r\n')

...from the Jython prompt instead of from the test suite I don't get any
exception.
msg4136 (view) Author: Alan Kennedy (amak) Date: 2009-02-04.16:23:02
Let's stay focussed on the select() issues in this bug; please consider
the out of memory error a separate issue.
msg4137 (view) Author: Alan Kennedy (amak) Date: 2009-02-04.16:29:02
In relation to the select bug, please can you provide the following?

1. What operating systems ad versions does it happen on?
2. What JVMs and versions does it happen on?
3. Are you running a firewall?
msg4138 (view) Author: Giampaolo Rodola' (billiejoex) Date: 2009-02-04.17:00:03
Windows XP Prof SP3, Java JRE 1.6.0_11, no firewall.
msg4139 (view) Author: Alan Kennedy (amak) Date: 2009-02-04.17:11:18
Please can you upload new versions of ftpserver.py and test_ftpd.py
which work with the new asyncore module?

I'll check it on windows and see if I can make any sense out of the bug,
which seems to happen in many different circumstances.

http://www.google.com/search?q=selector.open+%22java.io.IOException%3A+Unable+to+establish+loopback+connection%22
msg4140 (view) Author: Giampaolo Rodola' (billiejoex) Date: 2009-02-04.17:19:58
In attachment.
msg4183 (view) Author: Alan Kennedy (amak) Date: 2009-03-07.16:30:11
Any developments on this bug?

I will close the bug if there is no more activity on it.
msg4186 (view) Author: Giampaolo Rodola' (billiejoex) Date: 2009-03-07.17:00:52
I don't think that closing the report is a good idea when the problem is
still there.
Aside from asyncore/pyftpdlib it seems a problem related to select()
itself so any other application relying on it will most likely not work.
msg4368 (view) Author: Alan Kennedy (amak) Date: 2009-03-28.18:18:52
It looks like this is the same bug as 

http://bugs.jython.org/issue1291

Reported by Irmen de Jong.

All activity on this bug will continue on 1291, because Irmen has
isolated the bug with a small piece of code, thus making it far easier
reproduce and address.
msg4416 (view) Author: Alan Kennedy (amak) Date: 2009-04-03.16:34:07
I am fairly sure that this bug is a duplicate of bug 1291.
I intend to close this bug as a duplicate, but will give the reporter a
final chance to disagree.
msg4417 (view) Author: Giampaolo Rodola' (billiejoex) Date: 2009-04-03.16:42:34
It seems to be the same bug.
Please go on and close this one.
Thanks.
msg4418 (view) Author: Alan Kennedy (amak) Date: 2009-04-03.16:49:49
Closing this bug as a duplicate of 1291, by agreement with the reporter.

http://bugs.jython.org/issue1291
History
Date User Action Args
2009-04-03 16:49:50amaksetstatus: open -> closed
resolution: duplicate
messages: + msg4418
2009-04-03 16:42:34billiejoexsetmessages: + msg4417
2009-04-03 16:34:08amaksetmessages: + msg4416
2009-03-28 18:18:52amaksetmessages: + msg4368
2009-03-14 14:30:40fwierzbickisetpriority: normal
2009-03-07 17:00:52billiejoexsetmessages: + msg4186
2009-03-07 16:30:11amaksetmessages: + msg4183
2009-02-04 17:20:19billiejoexsetfiles: + test_ftpd.py
2009-02-04 17:20:04billiejoexsetfiles: + ftpserver.py
messages: + msg4140
2009-02-04 17:11:18amaksetmessages: + msg4139
2009-02-04 17:00:03billiejoexsetmessages: + msg4138
2009-02-04 16:29:04amaksetmessages: + msg4137
2009-02-04 16:23:02amaksetmessages: + msg4136
2009-02-04 16:09:40billiejoexsetmessages: + msg4135
2009-02-04 15:58:19amaksetmessages: + msg4132
2009-02-04 15:55:25amaksetmessages: + msg4131
2009-02-04 13:16:08billiejoexsetmessages: + msg4126
2009-02-03 18:20:45billiejoexsetfiles: + traceback.txt
2009-02-03 18:19:48billiejoexsetfiles: + test_ftpd.py
2009-02-03 18:18:59billiejoexsetfiles: + ftpserver.py
messages: + msg4122
2009-02-03 17:28:27amaksetmessages: + msg4121
2009-02-03 17:27:10amaksetmessages: + msg4120
2009-01-29 21:07:34amaksetassignee: amak
nosy: + amak
2009-01-14 15:30:49billiejoexcreate