Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

builtins: make ltrimstr and rtrimstr error for non-string inputs #2969

Merged
merged 1 commit into from
Jan 16, 2024

Conversation

emanuele6
Copy link
Member

@emanuele6 emanuele6 commented Dec 9, 2023

I also refactored the code in rtrimstr to make it more similar to the
code used by ltrimstr.

Previously, ltrimstr/rtrimstr would just let the input pass through for
non-string inputs or arguments.

That was happening because, they were leaking the errors returned by
startswith/endswith treating them as if they were jv_false().

This patch also fixes that memory leak discovered by oss-fuzz

Ref: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64946

@emanuele6
Copy link
Member Author

emanuele6 commented Dec 9, 2023

gojq copies ltrimstr/rtrimstr behaving like . if the input or the argument is not a string; do we want to preserve that behaviour, and just free the error message instead of leaking it?

@itchyny
Copy link
Contributor

itchyny commented Dec 10, 2023

I thought the previous behavior is useful for recursive trimming (.. |= ltrimstr). But I also understand the inconsistency in the builtins, and using strings filter allows to trim recursively, so I'm okay with changing the behavior.

@emanuele6
Copy link
Member Author

emanuele6 commented Dec 10, 2023

Since f_startswith / f_endswith are basically

def ltrimstr($left): if startswith($left) then .[$left | length:] end;
def rtrimstr($right): if endswith($right) then .[:$right | -length] end;

defined in C, should we just rewrite them in jq in src/builtin.jq?

The memory "optimisation" that is mentioned in the FIXME in f_ltrimstr could be handled by string slices, if we even want it...

tests/jq.test Outdated Show resolved Hide resolved
@wader
Copy link
Member

wader commented Dec 10, 2023

Agree on making them consistent with other string functions and it does not look like we have documented the passthru behaviour anywhere.

@emanuele6
Copy link
Member Author

What do you think about rewriting those functions in jq instead of keeping the C implemenations?

@wader
Copy link
Member

wader commented Dec 10, 2023

What do you think about rewriting those functions in jq instead of keeping the C implemenations?

If no big performance difference i think it makes sense

Copy link
Member

@wader wader left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait to see if @itchyny is happy too?

Copy link
Contributor

@itchyny itchyny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@itchyny
Copy link
Contributor

itchyny commented Dec 11, 2023

If you could it's better to fix leaks in the patch release and change the behavior in 1.8.

@emanuele6
Copy link
Member Author

Should I just create a new PR and leave this open for 1.8 then?

@emanuele6 emanuele6 added this to the 1.8 release milestone Dec 11, 2023
emanuele6 added a commit to emanuele6/jq-1 that referenced this pull request Dec 11, 2023
ltrimstr/rtrimstr was ignoring and leaking the error returned by
f_startswith()/f_endswith().

This also means that they just let the input pass through for non-string
inputs or arguments.

Only fix the leak for now; in the next release, jqlang#2969 will make them
rethrow the error returned by startswith/endswith.

Ref: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64946
emanuele6 added a commit to emanuele6/jq-1 that referenced this pull request Dec 11, 2023
ltrimstr/rtrimstr was ignoring and leaking the error returned by
f_startswith()/f_endswith().

This also means that they just let the input pass through for non-string
inputs or arguments.

Only fix the leak for now; in the next release, jqlang#2969 will make them
rethrow the error returned by startswith/endswith.

Ref: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64946
@emanuele6 emanuele6 marked this pull request as draft December 11, 2023 10:17
@emanuele6 emanuele6 marked this pull request as ready for review December 11, 2023 10:42
@emanuele6
Copy link
Member Author

I think the next release should be 1.8 since we already merged non-bug fix changes like the one that allows using # \ everywhere

@itchyny
Copy link
Contributor

itchyny commented Dec 12, 2023

we already merged non-bug fix changes like the one that allows using # \ everywhere

That's the only feature addition in 1.7? I think we should still discuss on the patch release for security issues, reverting the comment changes.

@emanuele6
Copy link
Member Author

If you want to release a patch release before 1.8, reverting the comment change, only keeping the bug fix that allows CR in comments is fine I guess

emanuele6 added a commit that referenced this pull request Dec 13, 2023
ltrimstr/rtrimstr was ignoring and leaking the error returned by
f_startswith()/f_endswith().

This also means that they just let the input pass through for non-string
inputs or arguments.

Only fix the leak for now; in the next release, #2969 will make them
rethrow the error returned by startswith/endswith.

Ref: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64946
Previously, ltrimstr/rtrimstr would just let the input pass through for
non-string inputs or arguments.

That was happening because, they were leaking the errors returned by
startswith/endswith treating them as if they were jv_false().
The leak was resolved by jqlang#2977 for 1.7.1

This patch rewrites ltrimstr and rtrimstr in jq, and makes them not
ignore startswith and endswith errors anymore.
@nicowilliams
Copy link
Contributor

Any reason not to merge this?

@wader
Copy link
Member

wader commented Jan 16, 2024

LGTM unless the change of pass through is a backwards compatibility issue?

@nicowilliams nicowilliams merged commit 1f1e619 into jqlang:master Jan 16, 2024
28 checks passed
@nicowilliams
Copy link
Contributor

Yeah, I'm not concerned about the passthrough change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants