Tyson Smith [:tsmith] Reporter
	Description • 6 years ago

This was found with m-c 20181126-6c10213a8924.

The test case does not reproduce the issue and is unreduced. I will attach a useful test case when it is available.

==13128==ERROR: AddressSanitizer: heap-use-after-free on address 0x61d000c3dc88 at pc 0x556ea5ac5e46 bp 0x7fc6c68417f0 sp 0x7fc6c6840fa0
READ of size 1775 at 0x61d000c3dc88 thread T104 (MediaCache)
    #0 0x556ea5ac5e45 in __asan_memcpy /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:23:3
    #1 0x7fc701f95b88 in mozilla::MediaCacheStream::NotifyDataReceived(unsigned int, unsigned int, unsigned char const*) src/dom/media/MediaCache.cpp:1952:7
    #2 0x7fc701f35978 in mozilla::ChannelMediaResource::CopySegmentToCache(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*) src/dom/media/ChannelMediaResource.cpp:428:18
    #3 0x7fc6f9b69539 in nsStringInputStream::ReadSegments(nsresult (*)(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*), void*, unsigned int, unsigned int*) src/xpcom/io/nsStringStream.cpp:273:17
    #4 0x7fc6f9afb953 in mozilla::NonBlockingAsyncInputStream::ReadSegments(nsresult (*)(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*), void*, unsigned int, unsigned int*) src/xpcom/io/NonBlockingAsyncInputStream.cpp:242:24
    #5 0x7fc701f322d3 in OnDataAvailable src/dom/media/ChannelMediaResource.cpp:456:18
    #6 0x7fc701f322d3 in mozilla::ChannelMediaResource::Listener::OnDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) src/dom/media/ChannelMediaResource.cpp:80
    #7 0x7fc6f9e0e6cf in nsBaseChannel::OnDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) src/netwerk/base/nsBaseChannel.cpp:902:28
    #8 0x7fc6f9e7caca in nsInputStreamPump::OnStateTransfer() src/netwerk/base/nsInputStreamPump.cpp:597:29
    #9 0x7fc6f9e7b36f in nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) src/netwerk/base/nsInputStreamPump.cpp:432:25
    #10 0x7fc6f9afb4d7 in RunAsyncWaitCallback src/xpcom/io/NonBlockingAsyncInputStream.cpp:406:13
    #11 0x7fc6f9afb4d7 in mozilla::NonBlockingAsyncInputStream::AsyncWaitRunnable::Run() src/xpcom/io/NonBlockingAsyncInputStream.cpp:31
    #12 0x7fc6f9bfda88 in nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1244:14
    #13 0x7fc6f9c0683d in NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:530:10
    #14 0x7fc6fae82f64 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:364:5
    #15 0x7fc6fad7c7be in RunInternal src/ipc/chromium/src/base/message_loop.cc:325:10
    #16 0x7fc6fad7c7be in RunHandler src/ipc/chromium/src/base/message_loop.cc:318
    #17 0x7fc6fad7c7be in MessageLoop::Run() src/ipc/chromium/src/base/message_loop.cc:298
    #18 0x7fc6f9bf5ef3 in nsThread::ThreadFunc(void*) src/xpcom/threads/nsThread.cpp:503:11
    #19 0x7fc71ea9b676 in _pt_root src/nsprpub/pr/src/pthreads/ptthread.c:201:5
    #20 0x7fc71e6e36b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
    #21 0x7fc71d76041c in clone /build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109

0x61d000c3dc88 is located 8 bytes inside of 2048-byte region [0x61d000c3dc80,0x61d000c3e480)
freed by thread T0 here:
    #0 0x556ea5ac6a12 in __interceptor_free /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:124:3
    #1 0x7fc6f99921d5 in Truncate src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.4/../../../../include/c++/4.9.4/bits/atomic_base.h
    #2 0x7fc6f99921d5 in nsTSubstring<char>::SetIsVoid(bool) src/xpcom/string/nsTSubstring.cpp:1070
    #3 0x7fc6f9b690e9 in Clear src/xpcom/io/nsStringStream.cpp:78:11
    #4 0x7fc6f9b690e9 in nsStringInputStream::Close() src/xpcom/io/nsStringStream.cpp:228
    #5 0x7fc6f9afa969 in mozilla::NonBlockingAsyncInputStream::Close() src/xpcom/io/NonBlockingAsyncInputStream.cpp:146:33
    #6 0x7fc6f9e79962 in nsInputStreamPump::Cancel(nsresult) src/netwerk/base/nsInputStreamPump.cpp:219:23
    #7 0x7fc6f9e08124 in Cancel src/netwerk/base/nsBaseChannel.cpp:412:15
    #8 0x7fc6f9e08124 in non-virtual thunk to nsBaseChannel::Cancel(nsresult) src/netwerk/base/nsBaseChannel.cpp
    #9 0x7fc701f32c60 in mozilla::ChannelMediaResource::CloseChannel() src/dom/media/ChannelMediaResource.cpp:645:15
    #10 0x7fc701f37981 in mozilla::ChannelMediaResource::Close() src/dom/media/ChannelMediaResource.cpp:594:5
    #11 0x7fc701f2b4fc in mozilla::ChannelMediaDecoder::Shutdown() src/dom/media/ChannelMediaDecoder.cpp:229:16
    #12 0x7fc701c48b13 in mozilla::dom::HTMLMediaElement::ShutdownDecoder() src/dom/html/HTMLMediaElement.cpp:1851:13
    #13 0x7fc701c74a00 in mozilla::dom::HTMLMediaElement::~HTMLMediaElement() src/dom/html/HTMLMediaElement.cpp:3787:5
    #14 0x7fc701d5d769 in ~HTMLVideoElement src/dom/html/HTMLVideoElement.cpp:57:1
    #15 0x7fc701d5d769 in mozilla::dom::HTMLVideoElement::~HTMLVideoElement() src/dom/html/HTMLVideoElement.cpp:55
    #16 0x7fc6f99fda31 in SnowWhiteKiller::~SnowWhiteKiller() src/xpcom/base/nsCycleCollector.cpp:2740:7
    #17 0x7fc6f99fc0e3 in nsCycleCollector::FreeSnowWhite(bool) src/xpcom/base/nsCycleCollector.cpp:2966:3
    #18 0x7fc6f9a089b2 in nsCycleCollector::BeginCollection(ccType, nsICycleCollectorListener*) src/xpcom/base/nsCycleCollector.cpp:3999:3
    #19 0x7fc6f9a07c45 in nsCycleCollector::Collect(ccType, js::SliceBudget&, nsICycleCollectorListener*, bool) src/xpcom/base/nsCycleCollector.cpp:3820:9
    #20 0x7fc6f9a0cbc6 in nsCycleCollector_collect(nsICycleCollectorListener*) src/xpcom/base/nsCycleCollector.cpp:4411:21
    #21 0x7fc6fe031f7a in nsJSContext::CycleCollectNow(nsICycleCollectorListener*) src/dom/base/nsJSEnvironment.cpp:1524:3
    #22 0x7fc7009692f8 in mozilla::dom::FuzzingFunctions_Binding::cycleCollect(JSContext*, unsigned int, JS::Value*) src/obj-firefox/dom/bindings/FuzzingFunctionsBinding.cpp:66:3
    #23 0x7fc70a4693bd in CallJSNative src/js/src/vm/Interpreter.cpp:468:15
    #24 0x7fc70a4693bd in js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct) src/js/src/vm/Interpreter.cpp:560

previously allocated by thread T0 here:
    #0 0x556ea5ac6d93 in malloc /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7fc6f9980217 in Alloc src/xpcom/string/nsSubstring.cpp:260:22
    #2 0x7fc6f9980217 in nsTSubstring<char>::StartBulkWriteImpl(unsigned int, unsigned int, bool, unsigned int, unsigned int, unsigned int) src/xpcom/string/nsTSubstring.cpp:216
    #3 0x7fc6f9996a46 in nsTSubstring<char>::BulkWrite(unsigned int, unsigned int, bool, nsresult&) src/xpcom/string/nsTSubstring.cpp:75:12
    #4 0x7fc6f9ae3eba in Base64DecodeString<nsTSubstring<char> > src/xpcom/io/Base64.cpp:578:25
    #5 0x7fc6f9ae3eba in mozilla::Base64Decode(nsTSubstring<char> const&, nsTSubstring<char>&) src/xpcom/io/Base64.cpp:601
    #6 0x7fc6fa6e26cf in nsDataChannel::OpenContentStream(bool, nsIInputStream**, nsIChannel**) src/netwerk/protocol/data/nsDataChannel.cpp:90:14
    #7 0x7fc6f9e06064 in nsBaseChannel::BeginPumpingData() src/netwerk/base/nsBaseChannel.cpp:261:8
    #8 0x7fc6f9dedc5b in nsBaseChannel::AsyncOpen(nsIStreamListener*, nsISupports*) src/netwerk/base/nsBaseChannel.cpp:720:8
    #9 0x7fc6f9e0bea7 in AsyncOpen2 src/netwerk/base/nsBaseChannel.cpp:753:10
    #10 0x7fc6f9e0bea7 in non-virtual thunk to nsBaseChannel::AsyncOpen2(nsIStreamListener*) src/netwerk/base/nsBaseChannel.cpp
    #11 0x7fc701ccf628 in mozilla::dom::HTMLMediaElement::ChannelLoader::LoadInternal(mozilla::dom::HTMLMediaElement*) src/dom/html/HTMLMediaElement.cpp:1434:19
    #12 0x7fc701cd02bc in applyImpl<mozilla::dom::HTMLMediaElement::ChannelLoader, void (mozilla::dom::HTMLMediaElement::ChannelLoader::*)(mozilla::dom::HTMLMediaElement *), StoreRefPtrPassByPtr<mozilla::dom::HTMLMediaElement> , 0> src/obj-firefox/dist/include/nsThreadUtils.h:1191:12
    #13 0x7fc701cd02bc in apply<mozilla::dom::HTMLMediaElement::ChannelLoader, void (mozilla::dom::HTMLMediaElement::ChannelLoader::*)(mozilla::dom::HTMLMediaElement *)> src/obj-firefox/dist/include/nsThreadUtils.h:1197
    #14 0x7fc701cd02bc in mozilla::detail::RunnableMethodImpl<mozilla::dom::HTMLMediaElement::ChannelLoader*, void (mozilla::dom::HTMLMediaElement::ChannelLoader::*)(mozilla::dom::HTMLMediaElement*), true, (mozilla::RunnableKind)0, mozilla::dom::HTMLMediaElement*>::Run() src/obj-firefox/dist/include/nsThreadUtils.h:1242
    #15 0x7fc6f9bfda88 in nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1244:14
    #16 0x7fc6f9c0683d in NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:530:10
    #17 0x7fc6fae8159f in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:97:21
    #18 0x7fc6fad7c7be in RunInternal src/ipc/chromium/src/base/message_loop.cc:325:10
    #19 0x7fc6fad7c7be in RunHandler src/ipc/chromium/src/base/message_loop.cc:318
    #20 0x7fc6fad7c7be in MessageLoop::Run() src/ipc/chromium/src/base/message_loop.cc:298
    #21 0x7fc703d5dfa3 in nsBaseAppShell::Run() src/widget/nsBaseAppShell.cpp:158:27
    #22 0x7fc708374e70 in nsAppStartup::Run() src/toolkit/components/startup/nsAppStartup.cpp:290:30
    #23 0x7fc70864e28e in XREMain::XRE_mainRun() src/toolkit/xre/nsAppRunner.cpp:4791:22
    #24 0x7fc708650b60 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:4936:8
    #25 0x7fc7086524e3 in XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:5028:21
    #26 0x556ea5af967c in do_main src/browser/app/nsBrowserApp.cpp:233:22
    #27 0x556ea5af967c in main src/browser/app/nsBrowserApp.cpp:315

Thread T104 (MediaCache) created by T0 here:
    #0 0x556ea5aaf6ad in __interceptor_pthread_create /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:210:3
    #1 0x7fc71ea983a5 in _PR_CreateThread src/nsprpub/pr/src/pthreads/ptthread.c:433:14
    #2 0x7fc71ea97f8e in PR_CreateThread src/nsprpub/pr/src/pthreads/ptthread.c:518:12
    #3 0x7fc6f9bf8e69 in nsThread::Init(nsTSubstring<char> const&) src/xpcom/threads/nsThread.cpp:717:8
    #4 0x7fc6f9c054fe in nsThreadManager::NewNamedThread(nsTSubstring<char> const&, unsigned int, nsIThread**) src/xpcom/threads/nsThreadManager.cpp:485:22
    #5 0x7fc6f9c09e54 in NS_NewNamedThread(nsTSubstring<char> const&, nsIThread**, nsIRunnable*, unsigned int) src/xpcom/threads/nsThreadUtils.cpp:143:45
    #6 0x7fc701f853c3 in NS_NewNamedThread<11> src/obj-firefox/dist/include/nsThreadUtils.h:75:10
    #7 0x7fc701f853c3 in mozilla::MediaCache::GetMediaCache(long) src/dom/media/MediaCache.cpp:716
    #8 0x7fc701f9dca9 in mozilla::MediaCacheStream::Init(long) src/dom/media/MediaCache.cpp:2606:17
    #9 0x7fc701f36845 in mozilla::ChannelMediaResource::Open(nsIStreamListener**) src/dom/media/ChannelMediaResource.cpp:518:30
    #10 0x7fc701f2b9c6 in mozilla::ChannelMediaDecoder::Load(nsIChannel*, bool, nsIStreamListener**) src/dom/media/ChannelMediaDecoder.cpp:253:19
    #11 0x7fc701c86336 in nsresult mozilla::dom::HTMLMediaElement::SetupDecoder<mozilla::ChannelMediaDecoder, nsIChannel*&, bool&, nsIStreamListener**&>(mozilla::ChannelMediaDecoder*, nsIChannel*&, bool&, nsIStreamListener**&) src/dom/html/HTMLMediaElement.cpp:4833:27
    #12 0x7fc701c3cf11 in mozilla::dom::HTMLMediaElement::InitializeDecoderForChannel(nsIChannel*, nsIStreamListener**) src/dom/html/HTMLMediaElement.cpp:4919:10
    #13 0x7fc701c3a855 in mozilla::dom::HTMLMediaElement::MediaLoadListener::OnStartRequest(nsIRequest*, nsISupports*) src/dom/html/HTMLMediaElement.cpp:728:7
    #14 0x7fc6f9e0d499 in nsBaseChannel::OnStartRequest(nsIRequest*, nsISupports*) src/netwerk/base/nsBaseChannel.cpp:857:25
    #15 0x7fc6f9e7bf83 in nsInputStreamPump::OnStateStart() src/netwerk/base/nsInputStreamPump.cpp:524:25
    #16 0x7fc6f9e7b353 in nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) src/netwerk/base/nsInputStreamPump.cpp:429:25
    #17 0x7fc6f9afb4d7 in RunAsyncWaitCallback src/xpcom/io/NonBlockingAsyncInputStream.cpp:406:13
    #18 0x7fc6f9afb4d7 in mozilla::NonBlockingAsyncInputStream::AsyncWaitRunnable::Run() src/xpcom/io/NonBlockingAsyncInputStream.cpp:31
    #19 0x7fc6f9bfda88 in nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1244:14
    #20 0x7fc6f9c0683d in NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:530:10
    #21 0x7fc6fae8159f in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:97:21
    #22 0x7fc6fad7c7be in RunInternal src/ipc/chromium/src/base/message_loop.cc:325:10
    #23 0x7fc6fad7c7be in RunHandler src/ipc/chromium/src/base/message_loop.cc:318
    #24 0x7fc6fad7c7be in MessageLoop::Run() src/ipc/chromium/src/base/message_loop.cc:298
    #25 0x7fc703d5dfa3 in nsBaseAppShell::Run() src/widget/nsBaseAppShell.cpp:158:27
    #26 0x7fc708374e70 in nsAppStartup::Run() src/toolkit/components/startup/nsAppStartup.cpp:290:30
    #27 0x7fc70864e28e in XREMain::XRE_mainRun() src/toolkit/xre/nsAppRunner.cpp:4791:22
    #28 0x7fc708650b60 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:4936:8
    #29 0x7fc7086524e3 in XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:5028:21
    #30 0x556ea5af967c in do_main src/browser/app/nsBrowserApp.cpp:233:22
    #31 0x556ea5af967c in main src/browser/app/nsBrowserApp.cpp:315
    #32 0x7fc71d67982f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291

	Tyson Smith [:tsmith] Reporter
	Comment 1 • 6 years ago

No reproducible test case has be reported yet but fuzzers have hit this 4x in the last 12h so maybe this is due to a recent change?

	Daniel Veditz [:dveditz]
	Comment 2 • 6 years ago

Nils: any recent media changes that might account for this?

	Nils Ohlmeier [:drno]
	Comment 3 • 6 years ago

Andreas could this be related to the changes you landed recently (it doesn't look like to me)?

The other big change which comes to my mind is the code reformating with clang-format which went in for /dom/media before it was applied to the rest of the tree.

	Andreas Pehrson [:pehrsons]
	Comment 4 • 6 years ago

I doubt my changes caused this, they didn't really touch the decoder stack. There were timing changes though, so I won't rule it out completely either.

	Nils Ohlmeier [:drno]
	Comment 5 • 6 years ago

Rating as P1 on the available information so far.

Jean-Yves do you have any idea what recent change could cause this apparently new behavior?

	Tyson Smith [:tsmith] Reporter
	Comment 6 • 6 years ago

For reference m-c 20181126-6c10213a8924 was the product version fuzzers first found this issue in

	Jean-Yves Avenard [:jya] Assignee
	Comment 7 • 6 years ago

The only change to the MediaCacheStream is in bug 1492365, that went in on September 19th 2018

There's been no change to the code since, and I can't think of anything to the underlying MediaResource/nsHttp code that has changed either.

How long has this fuzzer existed?

	Jean-Yves Avenard [:jya] Assignee
	Comment 8 • 6 years ago

Tyson, do you reset the media cache whenever the fuzzer runs?

For this code to be triggered, the data must not be present in the mediacache

	Tyson Smith [:tsmith] Reporter
	Comment 9 • 6 years ago

(In reply to Jean-Yves Avenard [:jya] from comment #8)
> Tyson, do you reset the media cache whenever the fuzzer runs?

I have an example of this crash (unreproducible) that happened on the first iteration. Is it possible to flush manually?

> How long has this fuzzer existed?

It's been running for about a year and half. However I did check and it turns out I pushed a commit to force call GC and CC in a different way just before this crash appeared. So I'll play around with that previously mentioned testcase and see if I can make it trigger the crash.

	Jean-Yves Avenard [:jya] Assignee
	Comment 10 • 6 years ago

You could use a new profile, or delete the cache files stored in the suer profile only.
There's a way to clear the cache from the UI, there's probably a way to do that with SpecialPowers

	Tyson Smith [:tsmith] Reporter
	Comment 11 • 6 years ago

Got one!