py: Add PEP 750 template strings support #17557

koxudaxi · 2025-06-24T19:32:57Z

Summary

Implements PEP 750 template strings for MicroPython.

Started in discussion #17497. Template strings (t-strings) are new in Python 3.14 - they return Template objects instead of strings, so you can access the literal parts and expressions separately.

Changes:

t-string parsing in lexer/parser
Template and Interpolation objects
string.templatelib module
Conditional build with MICROPY_PY_TSTRINGS

Usage:

t = t"Hello {name}!"
str(t)  # "Hello World!"
t.args[1].expression  # "name" 
t.args[1].value  # "World"

Testing

Tested on unix port with both MICROPY_PY_TSTRINGS enabled and disabled.

Test coverage:

Basic template string functionality
Error handling and edge cases
Import system integration
Nested expression parsing

All existing tests continue to pass. New tests added in tests/basics/string_template*.py.

Ports tested: Unix (other ports need testing)

Trade-offs and Alternatives

Adds ~240 bytes when enabled on unix port. When disabled, no impact.

Worth it because:

Part of Python 3.14 standard
Optional feature (MICROPY_PY_TSTRINGS)
Useful for template processing

Config: Enabled by default when MICROPY_CONFIG_ROM_LEVEL >= EXTRA_FEATURES.
Needs MICROPY_PY_FSTRINGS=1.

To disable: make CFLAGS_EXTRA=-DMICROPY_PY_TSTRINGS=0

WebReflection · 2025-06-24T19:54:35Z

for what is worth it, we'd love to have this available at least for the PyScript WASM variant as this unlocks tons of UI related use cases we'd like to deliver to our users.

/cc @dpgeorge @ntoll

codecov · 2025-06-25T02:23:46Z

Codecov Report

Attention: Patch coverage is 97.73371% with 16 lines in your changes missing coverage. Please review.

Project coverage is 98.53%. Comparing base (e57aa7e) to head (50dd4d1).
Report is 27 commits behind head on master.

Files with missing lines	Patch %	Lines
py/parse.c	95.48%	8 Missing ⚠️
py/modtstring.c	99.10%	3 Missing ⚠️
py/tstring_expr_parser.c	95.31%	3 Missing ⚠️
py/objtype.c	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #17557      +/-   ##
==========================================
- Coverage   98.57%   98.53%   -0.04%     
==========================================
  Files         169      172       +3     
  Lines       21968    22637     +669     
==========================================
+ Hits        21654    22306     +652     
- Misses        314      331      +17

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-06-25T04:56:57Z

Code size report:

   bare-arm:   +28 +0.049% 
minimal x86:   +58 +0.031% [incl +32(data)]
   unix x64: +11712 +1.370% standard[incl +576(data)]
      stm32: +5884 +1.500% PYBV10
     mimxrt: +5856 +1.569% TEENSY40
        rp2: +5944 +0.647% RPI_PICO_W
       samd: +6104 +2.272% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32: +7003 +1.546% VIRT_RV32

dpgeorge

Thanks for the contribution! This will be nice to have for the webassembly port.

I didn't do a review yet, but there will need to be tests to get full coverage of the new code.

docs/library/btree.rst

py/modbuiltins.c

koxudaxi · 2025-06-25T06:33:07Z

@dpgeorge Thank you for the feedback and for taking time to look at this! I'm glad this will be useful for the webassembly port.

I'll add more tests to ensure full coverage when I find time.

ntoll · 2025-06-25T08:59:43Z

@koxudaxi slightly off topic - but I notice you'll be at EuroPython in Prague, as will I. We should look out for each other and have a coffee or lunch together! 🇪🇺 🐍

koxudaxi · 2025-06-25T18:05:58Z

@ntoll Sounds good! See you in Prague. ☕

frozen/string/templatelib.py

Implements template strings (t-strings) as specified in PEP 750. Includes Template and Interpolation objects, expression parser, string.templatelib module, tests and documentation. Enabled for Unix, Windows, and WebAssembly ports. Signed-off-by: Koudai Aono <[email protected]>

koxudaxi · 2025-07-02T16:04:39Z

@dpgeorge

I didn't do a review yet, but there will need to be tests to get full coverage of the new code.

I tried really hard to make 100% coverage but some code is never called and I cannot cover it. Do you know how to fix this?

koxudaxi · 2025-07-02T16:08:53Z

docs/library/btree.rst


-   A flag for `keys()`, `values()`, `items()` methods to specify that
+   A flag for :meth:`btree.keys`, :meth:`btree.values`, :meth:`btree.items` methods to specify that
   scanning should be inclusive of the end key.

 .. data:: DESC

-   A flag for `keys()`, `values()`, `items()` methods to specify that
+   A flag for :meth:`btree.keys`, :meth:`btree.values`, :meth:`btree.items` methods to specify that


I faced an error when building with Sphinx because it conflicted with string.templatelib.Template.values. To solve this problem, I renamed the values attribute in btree to be more explicit.

It seems like a useful change, then --- it's just not a change related to PEP-750 t-string support, so it needs to be a separate commit and separate PR.
Git cherry-pick and rebase -i are your friends!

koxudaxi · 2025-07-02T16:44:06Z

@dpgeorge
I'm not very familiar with the micropython codebase, so please let me know if you notice any issues with how I'm handling memory constraints. I think everything looks good, but I'd appreciate your review.

dpgeorge · 2025-07-03T01:35:13Z

py/objtype.c

@@ -542,7 +542,18 @@ const byte mp_binary_op_method_name[MP_BINARY_OP_NUM_RUNTIME] = {
 };

 static mp_obj_t instance_binary_op(mp_binary_op_t op, mp_obj_t lhs_in, mp_obj_t rhs_in) {
+    #if MICROPY_PY_TSTRINGS
+    if (op == MP_BINARY_OP_ADD || op == MP_BINARY_OP_INPLACE_ADD) {


This whole bit can be removed, it's not needed. There is already a check in template_binary_op().

dpgeorge · 2025-07-03T01:36:42Z

ports/unix/coverage.c

@@ -822,6 +847,356 @@ static mp_obj_t extra_coverage(void) {
            MICROPY_STACK_CHECK == 0 || old_stack_limit == new_stack_limit);
    }

+
+    #if MICROPY_PY_TSTRINGS


I'm surprised all of this coverage code is needed here, in C.

You should be able to move most (if not all) of this to pure Python tests. Did you have trouble doing that?

dpgeorge · 2025-07-03T01:39:07Z

py/modtstring.c

+                    mp_print_t repr_print;
+                    vstr_init_print(&repr_vstr, 16, &repr_print);
+                    mp_obj_print_helper(&repr_print, value, PRINT_REPR);
+                    conv_value = mp_obj_new_str_from_vstr(&repr_vstr);


These 3 cases for r/s/a all look the same. Can you factor the code to reduce code duplication?

dpgeorge · 2025-07-03T01:42:58Z

py/modtstring.c

+                }
+                dest[0] = mp_obj_new_tuple(interps->len, values);
+            } else {
+                mp_obj_t *values = m_new(mp_obj_t, interps->len);


You can simplify the code here and not have any temporary memory by the following:

mp_obj_tuple_t *tuple = MP_OBJ_TO_PTR(mp_obj_new_tuple(interps->len, NULL)); for (size_t i = 0; i < interps->len; i++) { tuple->items[i] = interp->value; } dest[0] = MP_OBJ_FROM_PTR(tuple);

dpgeorge · 2025-07-03T01:45:58Z

py/modtstring.c

+            return MP_OBJ_FROM_PTR(result);
+        }
+
+        case MP_BINARY_OP_REVERSE_ADD: {


You can just remove this case, it's not needed.

dpgeorge · 2025-07-03T01:48:13Z

py/parse.c

+        est_bytes += est_seg_cnt * sizeof(mp_parse_node_t) * 2; // Assume some growth
+        est_bytes += est_interp_cnt * sizeof(mp_parse_node_t) * 2;
+
+        if (est_bytes > MICROPY_PY_TSTRING_MAX_BYTES) {


I don't think this test is needed, what's wrong with the tstring growing large?

dpgeorge · 2025-07-03T01:51:11Z

I've made a few comments, mostly regarding coverage.

About the MICROPY_PY_TSTRING_MAX_xxx constants: are these really needed? What's wrong with just letting the t-strings grow as much as the user needs them to? (Note: if memory runs out then the runtime will already automatically raise an exception, that the memory allocation did not succeed.)

AJMansfield

This PR needs to be rebased into several distinct commits so that it can be properly reviewed. This is a very complicated set of changes that adds a lot of potential surface for errors, requiring careful scrutiny --- and the way it mixes concerns does not inspire confidence.

To make this easier to review, please split your commit into multiple commits. If you still have a non-squashed head you can submit instead, that'd be helpful; and especially if not, you're going to need to learn to love git cherry-pick and git rebase -i.

I'd consider several of the changes in this PR to still be untested despite appearing to have 100% test coverage. 100% coverage is just the bare minimum --- the standard to aspire to is 100% test sensitivity, i.e. tests don't just have to pass when the change is present, they also are also required to fail when the change in question is absent. (100% test sensitivity is, in general, not a possible goal; but it's still an essential tool for verifying things that are a matter of correctness rather than just style.)

Several of the changes seem to be fixes to string parsing generally, beyond just t-strings. It's not surprising that you might discover latent string-parsing bugs while implementing something like this, but those fixes need to be split into their own commit. I also want to see test sensitivity on those changes, or at least better explanations why.

Several of the changes are tooling changes: the MICROPY_STACK_CHECK_MARGIN and related changes that seem to be meant to accommodate running micropython with a sanitizer. Sanitizers are definitely helpful on features like this, and if those are changes that allow some sanitizer that previously didn't work on micropython at all to be used, that's potentially quite useful and should be submitted as its own PR. And if they're specific to bypassing errors introduced by this feature, this needs more discussion here.

I'd also appreciate separate commits for test code vs feature code, so that it's more straightforward to verify test sensitivity --- Doing it with separate commits makes it easy to just checkout the test commit on its own and run the suite from there.

All of the tests have .exp files associated with them --- it's understandable since this is a syntax feature not present in most of the CPython versions used for testing, either.
Having those as part of the same commit makes it less convenient for testing, though --- please separate out the .exp files that don't represent actual micropython-specific behaviors and put them in a separate commit from their tests.

AJMansfield · 2025-07-03T15:43:48Z

py/lexer.c

                    vstr_add_byte(&lex->vstr, '{');
                    next_char(lex);
-                } else {
+                } else if (is_fstring) {


This seems to be redundant with the while (is_fstring condition.

AJMansfield · 2025-07-03T15:52:06Z

docs/library/micropython.rst

+
+   Note: When building with sanitizers (ASan/UBSan), an 8 KiB stack margin is
+   automatically reserved unless the port overrides MICROPY_STACK_CHECK_MARGIN.


Documentation change not related to PEP-750.

AJMansfield · 2025-07-03T16:02:04Z

ports/unix/coverage.c

+extern mp_obj_t mp_builtin___template__(mp_obj_t strings, mp_obj_t interpolations);
+extern const mp_obj_fun_builtin_fixed_t mp_builtin___template___obj;
+extern const mp_obj_type_t mp_type_template;
+extern const mp_obj_type_t mp_type_interpolation;
+extern mp_obj_t mp_obj_new_interpolation(mp_obj_t value, mp_obj_t expression, mp_obj_t conversion, mp_obj_t format_spec);
+
+typedef struct _mp_obj_template_t {
+    mp_obj_base_t base;
+    mp_obj_t strings;
+    mp_obj_t interpolations;
+} mp_obj_template_t;


If you need all of these symbols just to test this, these symbols probably shouldn't be private.

AJMansfield · 2025-07-03T16:11:12Z

ports/unix/mpconfigport.h

+// Reserve extra C-stack headroom for overflow checks.
+// Sanitizer builds enlarge call frames; 8 KiB prevents
+// false positives without noticeably shrinking usable stack.
+#define MICROPY_STACK_CHECK_MARGIN  (8192)
+


This seems like a tooling change that's unrelated to PEP-750.

Though if I'm wrong about that, this needs more than just this comment to justify the claim that the sanitizer errors this bypasses are actually "false" positives.

AJMansfield · 2025-07-03T16:21:04Z

py/lexer.c

+#if MICROPY_PY_FSTRINGS
+static bool is_char_or4(mp_lexer_t *lex, byte c1, byte c2, byte c3, byte c4) {
+    return lex->chr0 == c1 || lex->chr0 == c2 || lex->chr0 == c3 || lex->chr0 == c4;
+}
+#endif
+


Move this up next to is_char_or and is_char_or3.
Also, this function doesn't need a macro guard -- it already has static linkage, so the compiler can already elide it if it's not used anywhere else in the file.

AJMansfield · 2025-07-03T16:54:56Z

py/parse.h

+typedef char _tstr_assert1[(TSTR_HDR_INT_SHIFT + 12 <= 32) ? 1 : -1];
+typedef char _tstr_assert2[(TSTR_MAX_SEG <= 0xFFF) ? 1 : -1];
+typedef char _tstr_assert3[(TSTR_MAX_INT <= 0xFFF) ? 1 : -1];


Interesting idiom, I wasn't aware you could do static asserts like that!
But you should use MP_STATIC_ASSERT (from misc.h) instead.

AJMansfield · 2025-07-03T16:59:59Z

py/qstrdefs.h

+
+#if MICROPY_PY_TSTRINGS
+Q(_templatelib)
+Q(__t)
+Q(Template)
+Q(Interpolation)
+Q(strings)
+Q(interpolations)
+Q(value)
+Q(expression)
+Q(conversion)
+Q(format_spec)
+Q(string_dot_templatelib)
+Q(string)
+Q(templatelib)
+Q(values)
+Q({)
+Q(})
+Q(__template__)
+Q(__lt_tstring_expr_gt_)
+Q(__iter__)
+Q(template_iterator)
+#endif


This is unnecessary, these should all already be picked up automatically from any MP_QSTR_* symbols, during the qstr preprocessing build step. (If you're getting IDE errors it's because you haven't set your include path to include the generated includes folder in the build output directory.)

AJMansfield · 2025-07-03T17:00:37Z

py/runtime.h

+    #if MICROPY_PY_SYS_SETTRACE
+    // Initialize profiling state
+    ts->prof_callback_is_executing = false;
+    ts->prof_trace_callback = MP_OBJ_NULL;
+    ts->current_code_state = NULL;
+    #endif
+


This is a tooling change unrelated to PEP-750.

AJMansfield · 2025-07-03T17:05:13Z

tests/thread/thread_stacksize1.py

+    # Use larger stack size for desktop platforms to accommodate sanitizers
+    if sys.platform in ("linux", "darwin", "emscripten", "win32"):
+        sz = 32 * 1024
+    else:
+        sz = 2 * 1024


This is a tooling change unrelated to PEP-750.

AJMansfield · 2025-07-03T18:21:53Z

py/lexer.c

+        // Ensure we have at least 3 characters before accessing
+        if (lex->fstring_args.len >= 3) {
+            lex->chr0 = lex->fstring_args.buf[0];
+            lex->chr1 = lex->fstring_args.buf[1];
+            lex->chr2 = lex->fstring_args.buf[2];
+        } else {
+            // This should not happen with well-formed f/t-strings
+            lex->chr0 = lex->fstring_args.len > 0 ? (unichar)lex->fstring_args.buf[0] : MP_LEXER_EOF;
+            lex->chr1 = lex->fstring_args.len > 1 ? (unichar)lex->fstring_args.buf[1] : MP_LEXER_EOF;
+            lex->chr2 = lex->fstring_args.len > 2 ? (unichar)lex->fstring_args.buf[2] : MP_LEXER_EOF;
+        }


This should be part of a separate commit, and needs a non-tstring test.

koxudaxi changed the title ~~py: Add PEP 750 template strings support.~~ py: Add PEP 750 template strings support Jun 24, 2025

dpgeorge added the py-core Relates to py/ directory in source label Jun 25, 2025

dpgeorge reviewed Jun 25, 2025

View reviewed changes

docs/library/btree.rst Outdated Show resolved Hide resolved

py/modbuiltins.c Outdated Show resolved Hide resolved

dpgeorge reviewed Jun 26, 2025

View reviewed changes

frozen/string/templatelib.py Outdated Show resolved Hide resolved

koxudaxi force-pushed the feature/pep750-template-strings branch 9 times, most recently from 330b05f to 7a1dc11 Compare July 2, 2025 15:26

koxudaxi force-pushed the feature/pep750-template-strings branch from 7a1dc11 to 50dd4d1 Compare July 2, 2025 15:42

koxudaxi requested a review from dpgeorge July 2, 2025 15:59

koxudaxi commented Jul 2, 2025

View reviewed changes

dpgeorge reviewed Jul 3, 2025

View reviewed changes

py/modtstring.c

return MP_OBJ_FROM_PTR(result);

}

case MP_BINARY_OP_REVERSE_ADD: {

Copy link

Member

dpgeorge Jul 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can just remove this case, it's not needed.

dpgeorge reviewed Jul 3, 2025

View reviewed changes

AJMansfield suggested changes Jul 3, 2025

View reviewed changes


		Note: When building with sanitizers (ASan/UBSan), an 8 KiB stack margin is
		automatically reserved unless the port overrides MICROPY_STACK_CHECK_MARGIN.

Uh oh!

py: Add PEP 750 template strings support #17557

Are you sure you want to change the base?

py: Add PEP 750 template strings support #17557

Conversation

koxudaxi commented Jun 24, 2025

Summary

Testing

Trade-offs and Alternatives

Uh oh!

WebReflection commented Jun 24, 2025

Uh oh!

codecov bot commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpgeorge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

koxudaxi commented Jun 25, 2025

Uh oh!

ntoll commented Jun 25, 2025

Uh oh!

koxudaxi commented Jun 25, 2025

Uh oh!

Uh oh!

koxudaxi commented Jul 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

koxudaxi commented Jul 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dpgeorge commented Jul 3, 2025

Uh oh!

AJMansfield left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Jun 25, 2025 •

edited

Loading

github-actions bot commented Jun 25, 2025 •

edited

Loading

AJMansfield left a comment •

edited

Loading