Skip to content

gh-133982: Use implementation-specific open in test_fileio OtherFileTests #135364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 30, 2025

Conversation

cmaloney
Copy link
Contributor

@cmaloney cmaloney commented Jun 10, 2025

The tests around unicode filenames use both FileIO directly and open so kept using open for them / validates that unicode filenames passed through open() match behavior of FileIO called directly with an encoded name.

For the append tests it looks like FileIO directly is closer to what was intended, there is test_io that has append tests around self.open directly, this is test_fileio so intent I think was to test FileIO directly.

@ZeroIntensity ZeroIntensity added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Jun 16, 2025
@efimov-mikhail
Copy link
Contributor

efimov-mikhail commented Jun 16, 2025

Is it worth backporting on 3.13? cc @Yhg1s

P.S. Congratulations with becoming Core Dev, @ZeroIntensity !!!

@ZeroIntensity
Copy link
Member

We've been backporting these to 3.13 as far as I can tell.

@cmaloney
Copy link
Contributor Author

Not sure policy: Is a test-only change which enables more testing of _pyio, a _io implementation CPython doesn't use directly at the moment.

The work I'm doing on top of these patches is adding another _io.Buffered* implementation which is runtime switchable, starting in a copy/paste of _io, new import that swaps implementation to ease side by side performance testing/integrating while having a working/building/running CPython. Hoping to land the feature in the 3.15 feature window, but lots of steps to get there (implementation, creating I/O benchmarks in pyperformance, testing compatibility, validating performance on range of systems, PEP, ...)

@ZeroIntensity ZeroIntensity added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 30, 2025
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @ZeroIntensity for commit e69fba0 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F135364%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 30, 2025
Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ZeroIntensity ZeroIntensity merged commit 23caccf into python:main Jun 30, 2025
118 of 124 checks passed
@miss-islington-app
Copy link

Thanks @cmaloney for the PR, and @ZeroIntensity for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 30, 2025
…therFileTests` (pythonGH-135364)

(cherry picked from commit 23caccf)

Co-authored-by: Cody Maloney <[email protected]>
@bedevere-app
Copy link

bedevere-app bot commented Jun 30, 2025

GH-136148 is a backport of this pull request to the 3.14 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 30, 2025
…therFileTests` (pythonGH-135364)

(cherry picked from commit 23caccf)

Co-authored-by: Cody Maloney <[email protected]>
@bedevere-app bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Jun 30, 2025
@bedevere-app
Copy link

bedevere-app bot commented Jun 30, 2025

GH-136149 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Jun 30, 2025
ZeroIntensity pushed a commit that referenced this pull request Jun 30, 2025
…OtherFileTests` (GH-135364) (GH-136149)

gh-133982: Use implementation-specific `open` in `test_fileio.OtherFileTests` (GH-135364)
(cherry picked from commit 23caccf)

Co-authored-by: Cody Maloney <[email protected]>
@cmaloney cmaloney deleted the fileio_otherfiletests branch June 30, 2025 22:19
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.x (tier-1) has failed when building commit 23caccf.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/345/builds/11718) and take a look at the build logs.
  4. Check if the failure is related to this commit (23caccf) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/345/builds/11718

Failed tests:

  • test_statistics

Failed subtests:

  • test_large_pool - test.test_multiprocessing_forkserver.test_misc.MiscTestCase.test_large_pool
  • test_kde_random - test.test_statistics.TestKDE.test_kde_random
  • test_framing_many_objects - test.test_pickle.CDumpPickle_LoadPickle.test_framing_many_objects

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
                     "__main__", mod_spec)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 88, in _run_code
    exec(code, run_globals)
    ~~~~^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/worker.py", line 6, in <module>
    from test.support import os_helper, Py_DEBUG
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 10, in <module>
    import logging
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/logging/__init__.py", line 26, in <module>
    import sys, os, time, io, re, traceback, warnings, weakref, collections.abc
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/traceback.py", line 13, in <module>
    import _colorize
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/_colorize.py", line 158, in <module>
    @dataclass(frozen=True)
     ~~~~~~~~~^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/dataclasses.py", line 1362, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
                          frozen, match_args, kw_only, slots,
                          weakref_slot)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/dataclasses.py", line 1084, in _process_class
    _init_fn(all_init_fields,
    ~~~~~~~~^^^^^^^^^^^^^^^^^
             std_init_fields,
             ^^^^^^^^^^^^^^^^
    ...<9 lines>...
             slots,
             ^^^^^^
             )
             ^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/dataclasses.py", line 635, in _init_fn
    raise TypeError(f'non-default argument {f.name!r} '
                    f'follows default argument {seen_default.name!r}')
TypeError: non-default argument 'reset' follows default argument 'action'


Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_statistics.py", line 2510, in test_kde_random
    big_sample = sorted([rand() for i in range(n)])
                         ~~~~^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/statistics.py", line 1122, in rand
    return choice(data) + h * invcdf(random())
           ~~~~~~^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/random.py", line 352, in choice
    if not len(seq):
           ~~~^^^^^
TypeError: object of type 'Random' has no len()


Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/threading.py", line 1074, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/threading.py", line 1016, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_memoryview.py", line 757, in test
    obj.count(1)
    ~~~~~~~~~^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 532, in count
    return sum(value == entry for entry in self)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 532, in <genexpr>
    return sum(value == entry for entry in self)
                                           ^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 439, in __getitem__
    self._get_packing_format(position),
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 392, in _get_packing_format
    fmt = v.rstrip(b'\x00')
TypeError: a bytes-like object is required, not 'str'
Warning -- Uncaught thread exception: error
Exception in thread Thread-1 (test):
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/threading.py", line 1074, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/threading.py", line 1016, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_memoryview.py", line 757, in test
    obj.count(1)
    ~~~~~~~~~^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 532, in count
    return sum(value == entry for entry in self)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 532, in <genexpr>
    return sum(value == entry for entry in self)
                                           ^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 439, in __getitem__
    self._get_packing_format(position),
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 387, in _get_packing_format
    v = struct.unpack_from(
        ~~~~~~~~~~~~~~~~~~^
        "8s",
        ^^^^^
        self.shm.buf,
        ^^^^^^^^^^^^^
        self._offset_packing_formats + position * 8
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )[0]
    ^
struct.error: bad char in struct format
k


Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/pickletester.py", line 3623, in test_framing_many_objects
    self.check_frame_opcodes(pickled)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/pickletester.py", line 3572, in check_frame_opcodes
    for op, arg, pos in pickletools.genops(pickled):
                        ~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/pickletools.py", line 2291, in _genops
    arg = opcode.arg.reader(data)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/pickletools.py", line 243, in read_uint2
    raise ValueError("not enough data in stream to read uint2")
ValueError: not enough data in stream to read uint2


Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_concurrent_futures/util.py", line 54, in setUp
    self.manager = self.get_context().Manager()
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 57, in Manager
    m.start()
    ~~~~~~~^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/managers.py", line 570, in start
    self._address = reader.recv()
                    ~~~~~~~~~~~^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/connection.py", line 256, in recv
    buf = self._recv_bytes()
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/connection.py", line 447, in _recv_bytes
    buf = self._recv(4)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/connection.py", line 416, in _recv
    raise EOFError
EOFError


Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 6874, in test_large_pool
    self.assertFalse(err, msg=err.decode('utf-8'))
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'Fatal Python error: Segmentation fault\n\nCurrent thread 0x00007f8b54846740 [python] (most recent call first):\n  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/opcode.py", line 29 in <module>\n  File "<frozen importlib._bootstrap>", line 491 in _call_with_frames_removed\n  File "<frozen importlib._bootstrap_external>", line 762 in exec_module\n  File "<frozen importlib._bootstrap>", line 938 in _load_unlocked\n  File "<frozen importlib._bootstrap>", line 1342 in _find_and_load_unlocked\n  File "<frozen importlib._bootstrap>", line 1371 in _find_and_load\n  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/dis.py", line 8 in <module>\n  File "<frozen importlib._bootstrap>", line 491 in _call_with_frames_removed\n  File "<frozen importlib._bootstrap_external>", line 762 in exec_module\n  File "<frozen importlib._bootstrap>", line 938 in _load_unlocked\n  File "<frozen importlib._bootstrap>", line 1342 in _find_and_load_unlocked\n  File "<frozen importlib._bootstrap>", line 1371 in _find_and_load\n  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/inspect.py", line 149 in <module>\n  File "<frozen importlib._bootstrap>", line 491 in _call_with_frames_removed\n  File "<frozen importlib._bootstrap_external>", line 762 in exec_module\n  File "<frozen importlib._bootstrap>", line 938 in _load_unlocked\n  File "<frozen importlib._bootstrap>", line 1342 in _find_and_load_unlocked\n  File "<frozen importlib._bootstrap>", line 1371 in _find_and_load\n  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/dataclasses.py", line 5 in <module>\n  File "<frozen importlib._bootstrap>", line 491 in _call_with_frames_removed\n  File "<frozen importlib._bootstrap_external>", line 762 in exec_module\n  File "<frozen importlib._bootstrap>", line 938 in _load_unlocked\n  File "<frozen importlib._bootstrap>", line 1342 in _find_and_load_unlocked\n  File "<frozen importlib._bootstrap>", line 1371 in _find_and_load\n  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/_colorize.py", line 6 in <module>\n  File "<frozen importlib._bootstrap>", line 491 in _call_with_frames_removed\n  File "<frozen importlib._bootstrap_external>", line 762 in exec_module\n  File "<frozen importlib._bootstrap>", line 938 in _load_unlocked\n  File "<frozen importlib._bootstrap>", line 1342 in _find_and_load_unlocked\n  File "<frozen importlib._bootstrap>", line 1371 in _find_and_load\n  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/traceback.py", line 13 in <module>\n  File "<frozen importlib._bootstrap>", line 491 in _call_with_frames_removed\n  File "<frozen importlib._bootstrap_external>", line 762 in exec_module\n  File "<frozen importlib._bootstrap>", line 938 in _load_unlocked\n  File "<frozen importlib._bootstrap>", line 1342 in _find_and_load_unlocked\n  File "<frozen importlib._bootstrap>", line 1371 in _find_and_load\n  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", lin
, at +0x30a390 [0x5556d3167390]\n  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x3bfd0 [0x7f8b54884fd0]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyEval_EvalFrameDefault+0x6a2e [0x5556d30ac271]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x26b2f3 [0x5556d30c82f3]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x26b491 [0x5556d30c8491]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at PyEval_EvalCode+0xb6 [0x5556d30c8566]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x23fb7b [0x5556d309cb7b]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x23fc8a [0x5556d309cc8a]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x16d209 [0x5556d2fca209]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x11a69d [0x5556d2f7769d]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x11a98d [0x5556d2f7798d]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at PyObject_Call+0x20 [0x5556d2f779c5]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyEval_EvalFrameDefault+0x74f2 [0x5556d30acd35]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x26b2f3 [0x5556d30c82f3]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x26b491 [0x5556d30c8491]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x118dc1 [0x5556d2f75dc1]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x119110 [0x5556d2f76110]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x119d2c [0x5556d2f76d2c]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at PyObject_CallMethodObjArgs+0x101 [0x5556d2f76e53]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x2ae511 [0x5556d310b511]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at PyImport_ImportModuleLevelObject+0x1a3 [0x5556d310ee31]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyEval_ImportName+0xae [0x5556d30a488f]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyEval_EvalFrameDefault+0x11ae6 [0x5556d30b7329]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x26b2f3 [0x5556d30c82f3]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x26b491 [0x5556d30c8491]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at PyEval_EvalCode+0xb6 [0x5556d30c8566]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x23fb7b [0x5556d309cb7b]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x23fc8a [0x5556d309cc8a]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x16d209 [0x5556d2fca209]\n  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x11a69d [0x5556d2f7769d]\n  <truncated rest of calls>\n' is not false : Fatal Python error: Segmentation fault

ZeroIntensity pushed a commit that referenced this pull request Jun 30, 2025
…OtherFileTests` (GH-135364) (GH-136148)

gh-133982: Use implementation-specific `open` in `test_fileio.OtherFileTests` (GH-135364)
(cherry picked from commit 23caccf)

Co-authored-by: Cody Maloney <[email protected]>
@ZeroIntensity
Copy link
Member

I'm fairly sure that isn't related, especially considering I ran the buildbots beforehand and they were all green.

@cmaloney
Copy link
Contributor Author

Same for me. Lots of segmentation faults and failure to unpack / not enough data across many tests which feels like running out of disk space rather than anything specific around the test_fileio changes here.

@vstinner
Copy link
Member

vstinner commented Jul 1, 2025

https://buildbot.python.org/#/builders/345/builds/11718

The following build is green. That's strange :-(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
skip news tests Tests in the Lib/test dir
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants