py/objcode: Implement PEP626 to add co_lines to code objects on settrace builds. #16989

AJMansfield · 2025-03-21T18:59:48Z

Summary

This PR implements PEP 626 – Precise line numbers for debugging and other tools..

This adds one method, co_lines, that returns an iterator over (begin, end, line_no) tuples of line numbers for applicable bytecode ranges.

Testing

This PR includes automated tests of all of the currently-implemented code object attributes, including the existing attributes that previously had no test coverage.

The actual values in co_lines can't be compared directly to CPython's, but all of the invariants that the method is meant to observe are checked.

In addition to the github workflow checks, I've also manually verified that all tests pass when run against a unix MICROPY_PY_SYS_SETTRACE build. (In particular, I've checked to make sure they're not skipped).

Trade-offs and Alternatives

As in CPython, this is implemented as its own distinct iterator object. Potentially, this function could be functionally replicated by instead eagerly building a list of the tuples and returning an iterator over that list, at a slightly smaller code size; but the iterator approach allows for a tighter behavioral match with CPython (fun_code_full.py:25), and code size is not at nearly the same premium in settrace builds.

codecov · 2025-03-21T19:05:26Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.56%. Comparing base (f1018ee) to head (45623a4).
Report is 441 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #16989      +/-   ##
==========================================
+ Coverage   98.54%   98.56%   +0.02%     
==========================================
  Files         169      169              
  Lines       21877    21945      +68     
==========================================
+ Hits        21558    21631      +73     
+ Misses        319      314       -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-03-21T19:07:46Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +8 +0.001% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

Gadgetoid · 2025-03-24T10:59:48Z

Related to #16797?

dpgeorge

Thanks, this looks like a nice addition!

I think we should then deprecate co_lnotab following CPython.

And it would be good to add another test to this PR, to test the actual numeric values from co_lines(), to make sure they match the expected bytecode offsets for a simple test function (that test would need a corresponding .py.exp file because you can't use CPython to work out the right answer).

dpgeorge · 2025-06-16T02:36:31Z

py/objcode.c

+    iter->bc = 0;
+    iter->source_line = 1;
+    iter->ci = self->rc->prelude.line_info;
+    code_colines_next(MP_OBJ_FROM_PTR(iter));


This seems to discard the first line-no entry, is that intended?

This is intentional; this actually works around a bit of weirdness the test case helped me discover. For whatever reason, the first entry in this table always has a zero-length bytecode segment attributed to line 1.

This might be something to fix in the bytecode emitter, though that could also have knock-on effects for the other consumers of that table.

Skipping that first entry results in co_lines behavior behavior that matches CPython 3.10, where it only enumerates the body lines of a function --- though either way that's a diff with CPython 3.12, where co_lines includes the function's declaration line.

py/objcode.c

AJMansfield · 2025-06-26T17:02:19Z

Hmm, actually, looking at the compilation result, it looks like gcc -O3 isn't quite strong enough to make the whole result.info_increment part fully transparent --- at least in ARM, there's two extra opcodes with it, associated with moving around that 1 or 2 increment value and incrementing the line_info value with it.

https://godbolt.org/z/hqoYodEaE

AJMansfield · 2025-06-26T17:04:44Z

There we are, having it return the new line_info value instead of an increment value for it manages to get it fully elided: https://godbolt.org/z/bEY6ffTK8

Though, that makes it much less clear what's going on --- and there's similar differences in the other contexts with how incrementing bc and the like gets rolled in; doing the same thing for them makes it even harder and also ends up pessimizing the mp_bytecode_get_source_line to something even larger than without it.

(Also, clang gets the exact same compilation result regardless of whether or which way I refactor it.)

I'll leave the PR as-is unless we really feel like it's important to turn the screws as tight as possible just to win back 2 opcodes here.

AJMansfield · 2025-06-26T17:27:18Z

The thread/thread_gc1.py test failure in unix port / settrace_stackless (pull_request) seems to be spurious.

Was initially concerned if there was some sort of overflow making mp_obj_malloc(mp_obj_colines_iter_t, &mp_type_polymorph_iter); somehow a leak (with the fact that mp_obj_colines_iter_t is a lot larger than the mp_obj_iter_buf_t type used for this in mp_obj_tuple_getiter) --- but that test doesn't even interact with code objects, so idk.

py/bc.h

py/obj.h

py/objcode.h

Signed-off-by: Anson Mansfield <[email protected]>

AJMansfield · 2025-07-01T17:50:47Z

And it would be good to add another test to this PR, to test the actual numeric values from co_lines(), to make sure they match the expected bytecode offsets for a simple test function (that test would need a corresponding .py.exp file because you can't use CPython to work out the right answer).

I've been playing around with these tests; at the moment the test I added also covers the cases where a correct co_lines implementation might need to sum together multiple successive table entries to get the correct answer --- which it currently doesn't do.

The current outputs from co_lines are still meaningful and correct in some sense, though --- is that summing behavior actually something this needs to implement? Or can these extra coverage cases just be dropped (or made into cpydiffs or something)?

Also, should I add coverage for a case where the table needs to skip more than 2048 lines at once? (e.g. to overflow the 11-bit line number increment in the 2-byte format, and require a multi-entry encoding for it like bc+0, line+2047; bc+4, line+12)

Signed-off-by: Anson Mansfield <[email protected]>

AJMansfield · 2025-07-01T18:23:30Z

Ok, in the most recent variant of the test I've tried, I'm now really confused, and suspect I may have found a bug with the emitter for the underlying line info tables.

The function in question, which I've annotated with what seems to be the line-number encodings, is:

def f(x, y, obj, obj2, obj3):
    a = x + y # line 4: bc+4 line+4
    b = x - y # line 5: bc+4 line+1
    # line 6
    # line 7
    # line 8
    # line 9
    # line 10
    # line 11
    # line 12
    # line 13
    # line 14
    # line 15
    # line 16
    # line 17
    # line 18
    # line 19
    c = a * b # line 20: bc+4 line+15
    obj.a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z.fun() # line 21: bc+31 line+1; bc+27 line+0
    # line 22
    # line 23
    # line 24: bc+0 line+3
    # line 25
    # line 26
    # line 27: bc+0 line+3
    # line 28
    # line 29
    obj2.a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z.fun() # line 30: bc+31 line+3; bc+27 line+0
    # line 31
    # line 32
    # line 33: bc+0 line+3
    # line 34
    # line 35
    # line 36
    # line 37
    # line 38
    # line 39
    # line 40
    # line 41
    # line 42
    # line 43
    # line 44
    # line 45
    # line 46
    # line 47
    # line 48
    # line 49
    # line 50
    # line 51
    # line 52
    # line 53
    # line 54
    # line 55
    # line 56
    # line 57
    # line 58
    # line 59
    obj3.a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z.fun() # line 60: bc+31 line+27; bc+27 line+0
    return c # line 61: bc+2 line+1

I'd expect line 60 to get encoded as bc+15 line+27; bc+31 line+0; bc+11 line+0, but it seems to end up getting encoded in a much weirder way, with that bc+0 line+3 for line 33 and then bc+31 line+27 to jump from there to line 60 --- which doesn't seem like it should be a valid encoding in either the 1-byte or 2-byte formats.

I've not dug in to see if this is related to the SyntaxError this causes in the unix minimal and zephyr ports, but it doesn't seem far-fetched that this might also be part of that.

dpgeorge · 2025-07-02T00:34:39Z

tests/basics/fun_code_colines.py

+
+for start, end, line_no in f.__code__.co_lines():
+    print(f"line {line_no} start: {start}")
+    print(f"line {line_no} end: {end}")


Some ports (eg unix minimal, zephyr) don't enable f-strings, so we try to avoid using them in tests. Instead use "...".format(). Then the SyntaxError will go away on those CI build.s

dpgeorge · 2025-07-02T00:54:06Z

I'd expect line 60 to get encoded as bc+15 line+27; bc+31 line+0; bc+11 line+0, but it seems to end up getting encoded in a much weirder way, with that bc+0 line+3 for line 33 and then bc+31 line+27 to jump from there to line 60 --- which doesn't seem like it should be a valid encoding in either the 1-byte or 2-byte formats.

It's correct.

The line number encoding for f is:

60:24:84:0f:24:1f:7b:60:60:1f:7b:80:1b:1f:3b

And that decodes to the following (b,l) pairs:

60    = 0,3
24    = 4,1
80:0f = 4,15
24    = 4,1
1f    = 31,0
7b    = 27,3
60    = 0,3
60    = 0,3
1f    = 31,0
7b    = 27,3
80:1b = 0,27
1f    = 31,0
3b    = 27,1

dpgeorge · 2025-07-02T00:55:46Z

I've been playing around with these tests; at the moment the test I added also covers the cases where a correct co_lines implementation might need to sum together multiple successive table entries to get the correct answer --- which it currently doesn't do.

I'm not sure what you mean by "sum together multiple successive table entries"?

Also, should I add coverage for a case where the table needs to skip more than 2048 lines at once?

No, I think it's fine without that case.

AJMansfield changed the title ~~py/objcode: Implement PEP626 to add the co_lines to code objects.~~ py/objcode: Implement PEP626 to add co_lines to code objects on settrace builds. Mar 21, 2025

dpgeorge added the py-core Relates to py/ directory in source label Mar 27, 2025

dpgeorge reviewed Jun 16, 2025

View reviewed changes

dpgeorge reviewed Jul 1, 2025

View reviewed changes

py/bc.h Outdated Show resolved Hide resolved

py/obj.h Outdated Show resolved Hide resolved

py/objcode.h Outdated Show resolved Hide resolved

AJMansfield added 6 commits July 1, 2025 12:15

tests/basics/fun_code: Test for co_lines functionality.

bc6238d

Signed-off-by: Anson Mansfield <[email protected]>

py/objcode: Implement co_lines method.

19bc295

Signed-off-by: Anson Mansfield <[email protected]>

tests/basics/fun_code: Ignore deprecation warning about co_lnotab.

d5de371

Signed-off-by: Anson Mansfield <[email protected]>

py/bc: Factored out helper for line-no decoding.

3e704d7

Signed-off-by: Anson Mansfield <[email protected]>

py/objcode: Use line-no decoding helper.

373d06a

Signed-off-by: Anson Mansfield <[email protected]>

py/showbc: Use line-no decoding helper.

be3b303

Signed-off-by: Anson Mansfield <[email protected]>

AJMansfield force-pushed the code_co_lines branch 4 times, most recently from 5c31dc4 to c27a6ad Compare July 1, 2025 17:38

tests/basics/fun_code: Test decoded line-number values.

45623a4

Signed-off-by: Anson Mansfield <[email protected]>

AJMansfield force-pushed the code_co_lines branch from c27a6ad to 45623a4 Compare July 1, 2025 18:15

dpgeorge reviewed Jul 2, 2025

View reviewed changes

Uh oh!

py/objcode: Implement PEP626 to add co_lines to code objects on settrace builds. #16989

Are you sure you want to change the base?

py/objcode: Implement PEP626 to add co_lines to code objects on settrace builds. #16989

Conversation

AJMansfield commented Mar 21, 2025

Summary

Testing

Trade-offs and Alternatives

Uh oh!

codecov bot commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gadgetoid commented Mar 24, 2025

Uh oh!

dpgeorge left a comment

Choose a reason for hiding this comment

Uh oh!

dpgeorge Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

AJMansfield Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AJMansfield commented Jun 26, 2025

Uh oh!

AJMansfield commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AJMansfield commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AJMansfield commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AJMansfield commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpgeorge Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

dpgeorge commented Jul 2, 2025

Uh oh!

dpgeorge commented Jul 2, 2025

Uh oh!

Uh oh!

codecov bot commented Mar 21, 2025 •

edited

Loading

github-actions bot commented Mar 21, 2025 •

edited

Loading

AJMansfield commented Jun 26, 2025 •

edited

Loading

AJMansfield commented Jun 26, 2025 •

edited

Loading

AJMansfield commented Jul 1, 2025 •

edited

Loading

AJMansfield commented Jul 1, 2025 •

edited

Loading