sys/linux: add descriptions to test bpf_lsm module #1971

chmnchiang · 2020-07-25T02:58:20Z

Add a new pseudo-syscall syz_btf_id_by_name to translate a hook name into btf_id.
Add syz_btf_id_by_name to supported syscalls list.
Add descriptions to load and attach bpf programs onto lsm hooks.
Add descriptions to construct a simple test bpf program.

Most of the relevant APIs are in /include/uapi/linux/bpf.c (git, bootlin) and /include/uapi/linux/btf.c (git, bootlin).

Some context:

BPF programs are programs in a special byte-code format that can be accept by Linux kernel and run on various hook.
The user can use bpf(BPF_PROG_LOAD, ...) to load a bpf program and bpf(BPF_RAW_TRACEPOINT_OPEN, ...) to attach the program onto a lsm hook.
The attaching lsm look is determine by attach_btf_id, and one is able to translate a hook name into ID by looking up in /sys/kernel/btf/vmlinux.
The format of sys/kernel/btf/vmlinux is documented in /include/uapi/linux/btf.c.

chmnchiang · 2020-07-25T03:01:49Z

Currently syz-env make test failed due to the following error:

--- FAIL: TestGenerate (7.42s)
    --- FAIL: TestGenerate/linux/386 (1.06s)
        csource_test.go:66: seed=1595642834780028916
        --- FAIL: TestGenerate/linux/386/23 (0.25s)

...
                <stdin>: In function 'syz_btf_id_by_name':
                <stdin>:330:8: error: 'BTF_KIND_FUNC_PROTO' undeclared (first use in this function); did you mean 'BTF_KIND_UNION'?
                <stdin>:330:8: note: each undeclared identifier is reported only once for each function it appears in
                <stdin>:331:18: error: invalid application of 'sizeof' to incomplete type 'struct btf_param'
                <stdin>:333:8: error: 'BTF_KIND_VAR' undeclared (first use in this function); did you mean 'BTF_KIND_PTR'?
                <stdin>:334:18: error: invalid application of 'sizeof' to incomplete type 'struct btf_var'
                <stdin>:336:8: error: 'BTF_KIND_DATASEC' undeclared (first use in this function); did you mean 'BTF_KIND_PTR'?
                <stdin>:337:18: error: invalid application of 'sizeof' to incomplete type 'struct btf_var_secinfo'

I suspect the reason is that syz-env is using an older linux version (so only a portion of BTF_* is not found). How should I fix this problem?

executor/common_linux.h

dvyukov · 2020-07-25T05:23:48Z

Currently syz-env make test failed due to the following error:

--- FAIL: TestGenerate (7.42s)
    --- FAIL: TestGenerate/linux/386 (1.06s)
        csource_test.go:66: seed=1595642834780028916
        --- FAIL: TestGenerate/linux/386/23 (0.25s)

...
                <stdin>: In function 'syz_btf_id_by_name':
                <stdin>:330:8: error: 'BTF_KIND_FUNC_PROTO' undeclared (first use in this function); did you mean 'BTF_KIND_UNION'?
                <stdin>:330:8: note: each undeclared identifier is reported only once for each function it appears in
                <stdin>:331:18: error: invalid application of 'sizeof' to incomplete type 'struct btf_param'
                <stdin>:333:8: error: 'BTF_KIND_VAR' undeclared (first use in this function); did you mean 'BTF_KIND_PTR'?
                <stdin>:334:18: error: invalid application of 'sizeof' to incomplete type 'struct btf_var'
                <stdin>:336:8: error: 'BTF_KIND_DATASEC' undeclared (first use in this function); did you mean 'BTF_KIND_PTR'?
                <stdin>:337:18: error: invalid application of 'sizeof' to incomplete type 'struct btf_var_secinfo'

I suspect the reason is that syz-env is using an older linux version (so only a portion of BTF_* is not found). How should I fix this problem?

https://github.com/google/syzkaller/blob/master/docs/pseudo_syscalls.md
should answer your question (search for "headers for recently added kernel subsystems").

executor/common_linux.h

dvyukov · 2020-07-25T09:00:20Z

Yes, malloc is not OK and don't use FILE*.

mmap'ing something at a fixed address is minimal disturbance to the kernel behavior. We could mmap the file as well rather than read it in, should also make the code simpler.
I think we could choose some fixed unused address, mmap the file there with MAP_FIXED, if the mmap fails because something is already mapped, assume another syscall has already mapped it, so just use it.

chmnchiang · 2020-07-25T21:49:35Z

I think we could choose some fixed unused address, mmap the file there with MAP_FIXED, if the mmap fails because something is already mapped, assume another syscall has already mapped it, so just use it.

It seems like it is not possible to mmap the file /sys/kernel/btf/vmlinux (see here). The last time I tried to mmap it, it failed with ENODEV.

Is using mmap with MAP_FIXED | MAP_ANONYMOUS to dynamically obtain memory allowed?

dvyukov · 2020-07-26T05:33:39Z

I think we could choose some fixed unused address, mmap the file there with MAP_FIXED, if the mmap fails because something is already mapped, assume another syscall has already mapped it, so just use it.

It seems like it is not possible to mmap the file /sys/kernel/btf/vmlinux (see here). The last time I tried to mmap it, it failed with ENODEV.

Yikes!
It's even stored in a continuous region of memory in the kernel already :(

Is using mmap with MAP_FIXED | MAP_ANONYMOUS to dynamically obtain memory allowed?

Let's say, yes. There are no strict rules. We are just trying to disturb execution as less as possible and make things as deterministic as possible.

morehouse · 2020-07-27T18:03:45Z

Is using mmap with MAP_FIXED | MAP_ANONYMOUS to dynamically obtain memory allowed?

Let's say, yes. There are no strict rules. We are just trying to disturb execution as less as possible and make things as deterministic as possible.

Should we be careful to not do a MAP_FIXED over an existing mapping? Maybe use MAP_FIXED_NOREPLACE instead?

chmnchiang · 2020-07-27T22:11:27Z

What will be the best way to obtain a fixed address that is portable to different architecture? I am looking at virtual memory mappings on different architecture, but not sure if I am on the right direction.

codecov · 2020-07-28T02:49:46Z

Codecov Report

Merging #1971 into master will increase coverage by 0.1%.
The diff coverage is 60.0%.

Impacted Files	Coverage Δ
pkg/runtest/run.go	`66.5% <0.0%> (ø)`
pkg/host/syscalls_linux.go	`73.0% <75.0%> (+<0.1%)`	⬆️
prog/hints.go	`88.0% <0.0%> (-1.6%)`	⬇️
prog/any.go	`84.7% <0.0%> (-0.9%)`	⬇️
prog/mutation.go	`91.7% <0.0%> (+1.3%)`	⬆️
prog/prog.go	`82.5% <0.0%> (+3.3%)`	⬆️
prog/target.go	`68.2% <0.0%> (+5.8%)`	⬆️

dvyukov · 2020-07-28T05:42:21Z

What will be the best way to obtain a fixed address that is portable to different architecture? I am looking at virtual memory mappings on different architecture, but not sure if I am on the right direction.

Check that it does not overlap with any existing mapping. Works for 32-bits. Not too low, not too high (kernel tends to map something there). I would probably take address close to our data region (mapped in os_init), but not adjacent to it.

executor/common_linux.h

sys/linux/bpf_lsm.txt

executor/common_linux.h

pkg/host/syscalls_linux.go

chmnchiang · 2020-07-29T22:57:23Z

Fix the problems above and convert the draft into a normal pull request.
@jedav @morehouse @dvyukov
I will squash the commits once it is approved.

chmnchiang · 2020-07-29T23:11:16Z

The only problem I am facing is that when I run the test syz-runtest -vv 5 -debug -config config.json -tests btf_id, it shows the following error:

btf_id /thr C                         : FAIL: run 0: wrong call 1 result 22, want 0
        ### start
        ### call=1 errno=22
        ### call=2 errno=9
        ### call=3 errno=9
        ### call=4 errno=9
        ### call=6 errno=22
        ### call=7 errno=9
        ### call=8 errno=9
        ### call=9 errno=9
        ### call=0 errno=0
        ### call=5 errno=0

where the content of btf_id is

r0 = syz_btf_id_by_name(&AUTO='bpf_lsm_path_mkdir\x00')
r1 = bpf$BPF_LSM_PROG_LOAD(0x5, &AUTO=@test0={0x1d, 0x2, &AUTO={{0xb7, 0x0, 0x0, 0x0, 0x0}, {0x95, 0x0, 0x0, 0x0, 0x0}}, &AUTO='GPL\x00', 0x0, 0x0, 0x0, 0x0, 0x0, 'test\x00', 0x0, 0x1b, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, r0, 0x0}, 0x78)
r2 = bpf$BPF_LSM_RAW_TRACEPOINT_OPEN(0x11, &AUTO={0x0, r1}, 0x10)
close(r2)
close(r1)
r3 = syz_btf_id_by_name(&AUTO='bpf_lsm_path_mkdir\x00')
r4 = bpf$BPF_LSM_PROG_LOAD(0x5, &AUTO=@test0={0x1d, 0x2, &AUTO={{0xb7, 0x0, 0x0, 0x0, 0x0}, {0x95, 0x0, 0x0, 0x0, 0x0}}, &AUTO='GPL\x00', 0x0, 0x0, 0x0, 0x0, 0x0, 'test\x00', 0x0, 0x1b, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, r3, 0x0}, 0x78)
r5 = bpf$BPF_LSM_RAW_TRACEPOINT_OPEN(0x11, &AUTO={0x0, r4}, 0x10)
close(r5)
close(r4)

So it seems like call 1 (r1 = bpf$BPF_LSM_PROG_LOAD) is completed before call 0 (r0 = syz_btf_id_by_name). I don't quite understand how this can happen because r0 is used as an argument in call `.

dvyukov · 2020-07-30T06:19:19Z

The only problem I am facing is that when I run the test syz-runtest -vv 5 -debug -config config.json -tests btf_id, it shows the following error:

btf_id /thr C                         : FAIL: run 0: wrong call 1 result 22, want 0
        ### start
        ### call=1 errno=22
        ### call=2 errno=9
        ### call=3 errno=9
        ### call=4 errno=9
        ### call=6 errno=22
        ### call=7 errno=9
        ### call=8 errno=9
        ### call=9 errno=9
        ### call=0 errno=0
        ### call=5 errno=0

where the content of btf_id is

r0 = syz_btf_id_by_name(&AUTO='bpf_lsm_path_mkdir\x00')
r1 = bpf$BPF_LSM_PROG_LOAD(0x5, &AUTO=@test0={0x1d, 0x2, &AUTO={{0xb7, 0x0, 0x0, 0x0, 0x0}, {0x95, 0x0, 0x0, 0x0, 0x0}}, &AUTO='GPL\x00', 0x0, 0x0, 0x0, 0x0, 0x0, 'test\x00', 0x0, 0x1b, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, r0, 0x0}, 0x78)
r2 = bpf$BPF_LSM_RAW_TRACEPOINT_OPEN(0x11, &AUTO={0x0, r1}, 0x10)
close(r2)
close(r1)
r3 = syz_btf_id_by_name(&AUTO='bpf_lsm_path_mkdir\x00')
r4 = bpf$BPF_LSM_PROG_LOAD(0x5, &AUTO=@test0={0x1d, 0x2, &AUTO={{0xb7, 0x0, 0x0, 0x0, 0x0}, {0x95, 0x0, 0x0, 0x0, 0x0}}, &AUTO='GPL\x00', 0x0, 0x0, 0x0, 0x0, 0x0, 'test\x00', 0x0, 0x1b, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, r3, 0x0}, 0x78)
r5 = bpf$BPF_LSM_RAW_TRACEPOINT_OPEN(0x11, &AUTO={0x0, r4}, 0x10)
close(r5)
close(r4)

So it seems like call 1 (r1 = bpf$BPF_LSM_PROG_LOAD) is completed before call 0 (r0 = syz_btf_id_by_name). I don't quite understand how this can happen because r0 is used as an argument in call `.

syz_btf_id_by_name is probably too slow.
executor has 45 ms timeout for execution of syscalls:
https://github.com/google/syzkaller/blob/master/executor/executor.cc#L731
If a syscall does not finish within that timeout the next syscalls is kicked off. That's probably what you are seeing.
Check out the timeout syscall attribute:

$ grep "(timeout" sys/linux/*.txt

executor/common_linux.h

sys/linux/bpf_lsm.txt

chmnchiang · 2020-07-31T04:03:46Z

After increasing the time limit, the problem with syz-runtest is now:

btf_id namespace                      : FAIL: run 0: wrong call 1 result 1, want 0
btf_id namespace/cover                : FAIL: run 0: call 0 is not executed
btf_id namespace C                    : FAIL: run 0: wrong call 1 result 1, want 0

I think I need to change pkg/host/syscalls_linux.go by allowing syz_btf_id_by_name only when onlySandboxNone(). Is this correct?

dvyukov · 2020-07-31T06:09:11Z

After increasing the time limit, the problem with syz-runtest is now:
btf_id namespace                      : FAIL: run 0: wrong call 1 result 1, want 0
btf_id namespace/cover                : FAIL: run 0: call 0 is not executed
btf_id namespace C                    : FAIL: run 0: wrong call 1 result 1, want 0
I think I need to change pkg/host/syscalls_linux.go by allowing syz_btf_id_by_name only when onlySandboxNone(). Is this correct?

If it requires CAP_ADMIN, then yes.

chmnchiang · 2020-08-05T01:49:54Z

I am waiting for #2005 to be merged so I can run make presubmit. I also need to change that part of the file.

dvyukov · 2020-08-05T13:08:03Z

#2005 is merged now. Please rebase.

sys/linux/test/btf_id

sys/linux/bpf_lsm.txt

chmnchiang · 2020-08-06T00:56:21Z

When I run syz-runtest -config config.json -tests btf_id the test failed with:

btf_id none/cover                     : FAIL: run 0: call 3: no cover
btf_id none/repeat/cover              : FAIL: run 1: call 3: no cover
btf_id none/thr/repeat/cover          : FAIL: run 1: call 3: no cover

Call 3 is the second call to the psuedo-syscall syz_btf_id_by_name. It is not getting coverage because syz_btf_id_by_name does not read vmlinux in second and later calls. Thus, it should be normal for this call to not get any coverage. Wonder what I should do here.

chmnchiang · 2020-08-06T01:01:24Z

Also, is it important to pass all the test in syz-runtest? I ask @necipfazil before and he advised me to use syz-execprog to do the testing instead.

By the way, syz-runtest do not work for now:

2020/08/05 17:59:25 io_uring: unknown comment "Create an io_uring instance"

It seems like there need to be an extra blank line before each syscall, or the parser will assume the comment is the expected errno.
I could submit a quick fix if this should be done.

chmnchiang · 2020-08-06T01:03:45Z

And also, I can not pass the make presubmit check now because there are a lot of commit messages that does not follow the format:

##[error]Wrong commit subject format: 'Revert "dashboard/config: select KCSAN_VERBOSE in KCSAN config"'. Please use 'main/affected/package: short change description'. See docs/contributing.md for details.
##[error]Wrong commit subject format: 'Revert "dashboard/config: disable PARAVIRT_DEBUG with KCSAN"'. Please use 'main/affected/package: short change description'. See docs/contributing.md for details.
##[error]Wrong commit subject format: 'Revert "executor: enable extra coverage on OpenBSD"'. Please use 'main/affected/package: short change description'. See docs/contributing.md for details.
##[error]Wrong commit subject format: 'dashboard/app: Receive Recipients from API'. Please use 'main/affected/package: short change description'. See docs/contributing.md for details.
...

I think this will be resolved once #2015 is merged. This commit passed other tests now. I will squash my commits once they are approved.

dvyukov · 2020-08-06T10:11:30Z

And also, I can not pass the make presubmit check now because there are a lot of commit messages that does not follow the format:

Fixes for this merged. Should work now.

dvyukov · 2020-08-06T10:15:47Z

Also, is it important to pass all the test in syz-runtest? I ask @necipfazil before and he advised me to use syz-execprog to do the testing instead.

Yes, syz-runtest should pass.
syz-execprog is handy for initial development b/c you can do iterations faster, while syz-runtest takes some time.
But syz-execprog tests literally nothing, it is happy with any outcome, it also does not test multiple execution modes, sandboxes, C reproducers, etc. While syz-runtest tests all that.
Eventually we will enable syz-runtest on CI, it's just tricky b/c it needs working kernel+vm.

Re syz_btf_id_by_name and "no coverage", I think we should just special-case syz_btf_id_by_name in pkg/runtest for now. It's intentional that it does not get coverage. And since it's first such syscall, I don't think it makes sense to implement something more fundamental right now.

dvyukov · 2020-08-06T10:17:09Z

The change itself looks good to me.
Please rebase to HEAD so that CI passes, squash commits and fix pkg/runtest, and we are ready for merge.

This commit includes the following changes: * executor: add a new syz_btf_id_by_name psuedo-syscall * sys/linux: add descriptions for BPF LSM subsystem * sys/linux: add instructions on how to dump vmlinux and install bpftool * sys/linux/test: add tests for the new psuedo-syscall * pkg/host: add support detection for the new psuedo-syscall * pkg/runtest: skip the coverage test when invoking the new psuedo-syscall Update google#533.

chmnchiang · 2020-08-07T01:54:55Z

I fixed the problem and now it passes CI checks.
Please let me know if there are any problem. Thank you very much.

dvyukov · 2020-08-07T07:28:22Z

Perfect!
I am eager to see syzbot reaction :)

Pull request google#1971 add the resource bpf_lsm_btf_id and make that a required resource for bpf$BPF_LSM_PROG_LOAD. However, we need google#2035 merged to get a bpf_lsm_btf_id, and the pull request is currently blocked by a pahole issue. Thus, bpf$BPF_LSM_PROG_LOAD will be disabled for now. This pull request makes bpf_lsm_btf_id optional for bpf$BPF_LSM_PROG_LOAD, so we can test this syscall before the issue is resolved.

Pull request #1971 add the resource bpf_lsm_btf_id and make that a required resource for bpf$BPF_LSM_PROG_LOAD. However, we need #2035 merged to get a bpf_lsm_btf_id, and the pull request is currently blocked by a pahole issue. Thus, bpf$BPF_LSM_PROG_LOAD will be disabled for now. This pull request makes bpf_lsm_btf_id optional for bpf$BPF_LSM_PROG_LOAD, so we can test this syscall before the issue is resolved.

chmnchiang commented Jul 25, 2020

View reviewed changes

executor/common_linux.h Show resolved Hide resolved

chmnchiang commented Jul 25, 2020

View reviewed changes

executor/common_linux.h Show resolved Hide resolved

chmnchiang commented Jul 25, 2020

View reviewed changes

executor/common_linux.h Outdated Show resolved Hide resolved

chmnchiang commented Jul 25, 2020

View reviewed changes

executor/common_linux.h Show resolved Hide resolved

dvyukov mentioned this pull request Jul 25, 2020

docs: add section on descriptions testing #1973

Merged

dvyukov reviewed Jul 25, 2020

View reviewed changes

executor/common_linux.h Show resolved Hide resolved

executor/common_linux.h Outdated Show resolved Hide resolved

jedav reviewed Jul 28, 2020

View reviewed changes

chmnchiang commented Jul 29, 2020

View reviewed changes

executor/common_linux.h Outdated Show resolved Hide resolved

chmnchiang commented Jul 29, 2020

View reviewed changes

pkg/host/syscalls_linux.go Outdated Show resolved Hide resolved

chmnchiang force-pushed the bpf_lsm branch from fe1169f to 08ac03f Compare July 29, 2020 22:54

chmnchiang marked this pull request as ready for review July 29, 2020 22:55

chmnchiang force-pushed the bpf_lsm branch from 08ac03f to d8e3379 Compare July 29, 2020 23:07

dvyukov reviewed Jul 30, 2020

View reviewed changes

dvyukov reviewed Aug 5, 2020

View reviewed changes

sys/linux/test/btf_id Outdated Show resolved Hide resolved

sys/linux/bpf_lsm.txt Outdated Show resolved Hide resolved

sys/linux/bpf_lsm.txt Outdated Show resolved Hide resolved

sys/linux/bpf_lsm.txt Outdated Show resolved Hide resolved

chmnchiang force-pushed the bpf_lsm branch from 754826b to 8152b3b Compare August 6, 2020 00:49

chmnchiang force-pushed the bpf_lsm branch from 8152b3b to faa3b0c Compare August 7, 2020 00:12

morehouse approved these changes Aug 7, 2020

View reviewed changes

chmnchiang force-pushed the bpf_lsm branch from faa3b0c to cce65b0 Compare August 7, 2020 00:17

chmnchiang requested review from jedav and dvyukov August 7, 2020 00:18

dvyukov mentioned this pull request Aug 7, 2020

dashboard/config: enable BPF LSM #2025

Merged

dvyukov merged commit 20a3465 into google:master Aug 7, 2020

chmnchiang mentioned this pull request Aug 15, 2020

sys/linux: make bpf_lsm_btf_id optional #2054

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sys/linux: add descriptions to test bpf_lsm module #1971

sys/linux: add descriptions to test bpf_lsm module #1971

chmnchiang commented Jul 25, 2020

chmnchiang commented Jul 25, 2020

dvyukov commented Jul 25, 2020

dvyukov commented Jul 25, 2020

chmnchiang commented Jul 25, 2020 •

edited

dvyukov commented Jul 26, 2020

morehouse commented Jul 27, 2020

chmnchiang commented Jul 27, 2020

codecov bot commented Jul 28, 2020 •

edited

dvyukov commented Jul 28, 2020

chmnchiang commented Jul 29, 2020

chmnchiang commented Jul 29, 2020

dvyukov commented Jul 30, 2020

chmnchiang commented Jul 31, 2020

dvyukov commented Jul 31, 2020

chmnchiang commented Aug 5, 2020

dvyukov commented Aug 5, 2020

chmnchiang commented Aug 6, 2020

chmnchiang commented Aug 6, 2020

chmnchiang commented Aug 6, 2020 •

edited

dvyukov commented Aug 6, 2020

dvyukov commented Aug 6, 2020

dvyukov commented Aug 6, 2020

chmnchiang commented Aug 7, 2020

dvyukov commented Aug 7, 2020

sys/linux: add descriptions to test bpf_lsm module #1971

sys/linux: add descriptions to test bpf_lsm module #1971

Conversation

chmnchiang commented Jul 25, 2020

chmnchiang commented Jul 25, 2020

dvyukov commented Jul 25, 2020

dvyukov commented Jul 25, 2020

chmnchiang commented Jul 25, 2020 • edited

dvyukov commented Jul 26, 2020

morehouse commented Jul 27, 2020

chmnchiang commented Jul 27, 2020

codecov bot commented Jul 28, 2020 • edited

Codecov Report

dvyukov commented Jul 28, 2020

chmnchiang commented Jul 29, 2020

chmnchiang commented Jul 29, 2020

dvyukov commented Jul 30, 2020

chmnchiang commented Jul 31, 2020

dvyukov commented Jul 31, 2020

chmnchiang commented Aug 5, 2020

dvyukov commented Aug 5, 2020

chmnchiang commented Aug 6, 2020

chmnchiang commented Aug 6, 2020

chmnchiang commented Aug 6, 2020 • edited

dvyukov commented Aug 6, 2020

dvyukov commented Aug 6, 2020

dvyukov commented Aug 6, 2020

chmnchiang commented Aug 7, 2020

dvyukov commented Aug 7, 2020

chmnchiang commented Jul 25, 2020 •

edited

codecov bot commented Jul 28, 2020 •

edited

chmnchiang commented Aug 6, 2020 •

edited