Testcase found while fuzzing mozilla-central rev e8a29c8f1e09 (built with: --enable-debug --enable-fuzzing).

Testcase can be reproduced using the following commands:

$ pip install fuzzfetch grizzly-framework
$ python -m fuzzfetch --build e8a29c8f1e09 --debug --fuzzing -n firefox
$ python -m grizzly.replay ./firefox/firefox testcase.html

Assertion failure: mMainMaxSize >= aNewMinSize (Should only use this function for resolving min-size:auto, and main max-size should be an upper-bound for resolved val), at /layout/generic/nsFlexContainerFrame.cpp:627

    ==3948503==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f22a5e61036 bp 0x7ffd73e67d20 sp 0x7ffd73e67cf0 T3948503)
    ==3948503==The signal is caused by a WRITE memory access.
    ==3948503==Hint: address points to the zero page.
        #0 0x7f22a5e61036 in nsFlexContainerFrame::FlexItem::UpdateMainMinSize(int) /layout/generic/nsFlexContainerFrame.cpp:625:5
        #1 0x7f22a5e60526 in nsFlexContainerFrame::ResolveAutoFlexBasisAndMinSize(nsFlexContainerFrame::FlexItem&, mozilla::ReflowInput const&, nsFlexContainerFrame::FlexboxAxisTracker const&, bool) /layout/generic/nsFlexContainerFrame.cpp:1743:15
        #2 0x7f22a5e5f3c7 in nsFlexContainerFrame::GenerateFlexItemForChild(nsFlexContainerFrame::FlexLine&, nsIFrame*, mozilla::ReflowInput const&, nsFlexContainerFrame::FlexboxAxisTracker const&, bool) /layout/generic/nsFlexContainerFrame.cpp:1525:3
        #3 0x7f22a5e697b3 in nsFlexContainerFrame::GenerateFlexLines(mozilla::ReflowInput const&, int, nsTArray<nsFlexContainerFrame::StrutInfo> const&, nsFlexContainerFrame::FlexboxAxisTracker const&, int, bool, nsTArray<nsIFrame*>&, nsTArray<nsFlexContainerFrame::FlexLine>&) /layout/generic/nsFlexContainerFrame.cpp:4121:7
        #4 0x7f22a5e6d20e in nsFlexContainerFrame::DoFlexLayout(mozilla::ReflowInput const&, int&, int&, int&, nsTArray<nsFlexContainerFrame::FlexLine>&, nsTArray<nsFlexContainerFrame::StrutInfo>&, nsTArray<nsIFrame*>&, nsFlexContainerFrame::FlexboxAxisTracker const&, int, int, int, bool, ComputedFlexContainerInfo*) /layout/generic/nsFlexContainerFrame.cpp:5045:3
        #5 0x7f22a5e6b7be in nsFlexContainerFrame::Reflow(nsPresContext*, mozilla::ReflowOutput&, mozilla::ReflowInput const&, nsReflowStatus&) /layout/generic/nsFlexContainerFrame.cpp:4545:5
        #6 0x7f22a5f4841c in nsLineLayout::ReflowFrame(nsIFrame*, nsReflowStatus&, mozilla::ReflowOutput*, bool&) /layout/generic/nsLineLayout.cpp:875:13
        #7 0x7f22a5e3740f in nsBlockFrame::ReflowInlineFrame(mozilla::BlockReflowInput&, nsLineLayout&, nsLineList_iterator, nsIFrame*, LineReflowStatus*) /layout/generic/nsBlockFrame.cpp:4553:15
        #8 0x7f22a5e36926 in nsBlockFrame::DoReflowInlineFrames(mozilla::BlockReflowInput&, nsLineLayout&, nsLineList_iterator, nsFlowAreaRect&, int&, nsFloatManager::SavedState*, bool*, LineReflowStatus*, bool) /layout/generic/nsBlockFrame.cpp:4355:5
        #9 0x7f22a5e329b0 in nsBlockFrame::ReflowInlineFrames(mozilla::BlockReflowInput&, nsLineList_iterator, bool*) /layout/generic/nsBlockFrame.cpp:4240:9
        #10 0x7f22a5e2efb0 in nsBlockFrame::ReflowLine(mozilla::BlockReflowInput&, nsLineList_iterator, bool*) /layout/generic/nsBlockFrame.cpp:3217:5
        #11 0x7f22a5e2970b in nsBlockFrame::ReflowDirtyLines(mozilla::BlockReflowInput&) /layout/generic/nsBlockFrame.cpp:2751:7
        #12 0x7f22a5e250ab in nsBlockFrame::Reflow(nsPresContext*, mozilla::ReflowOutput&, mozilla::ReflowInput const&, nsReflowStatus&) /layout/generic/nsBlockFrame.cpp:1382:3
        #13 0x7f22a5e3567c in nsBlockReflowContext::ReflowBlock(mozilla::LogicalRect const&, bool, nsCollapsingMargin&, int, bool, nsLineBox*, mozilla::ReflowInput&, nsReflowStatus&, mozilla::BlockReflowInput&) /layout/generic/nsBlockReflowContext.cpp:288:11
        #14 0x7f22a5e3140c in nsBlockFrame::ReflowBlockFrame(mozilla::BlockReflowInput&, nsLineList_iterator, bool*) /layout/generic/nsBlockFrame.cpp:3874:11
        #15 0x7f22a5e2f056 in nsBlockFrame::ReflowLine(mozilla::BlockReflowInput&, nsLineList_iterator, bool*) /layout/generic/nsBlockFrame.cpp:3214:5
        #16 0x7f22a5e2970b in nsBlockFrame::ReflowDirtyLines(mozilla::BlockReflowInput&) /layout/generic/nsBlockFrame.cpp:2751:7
        #17 0x7f22a5e250ab in nsBlockFrame::Reflow(nsPresContext*, mozilla::ReflowOutput&, mozilla::ReflowInput const&, nsReflowStatus&) /layout/generic/nsBlockFrame.cpp:1382:3
        #18 0x7f22a5e3567c in nsBlockReflowContext::ReflowBlock(mozilla::LogicalRect const&, bool, nsCollapsingMargin&, int, bool, nsLineBox*, mozilla::ReflowInput&, nsReflowStatus&, mozilla::BlockReflowInput&) /layout/generic/nsBlockReflowContext.cpp:288:11
        #19 0x7f22a5e3140c in nsBlockFrame::ReflowBlockFrame(mozilla::BlockReflowInput&, nsLineList_iterator, bool*) /layout/generic/nsBlockFrame.cpp:3874:11
        #20 0x7f22a5e2f056 in nsBlockFrame::ReflowLine(mozilla::BlockReflowInput&, nsLineList_iterator, bool*) /layout/generic/nsBlockFrame.cpp:3214:5
        #21 0x7f22a5e2970b in nsBlockFrame::ReflowDirtyLines(mozilla::BlockReflowInput&) /layout/generic/nsBlockFrame.cpp:2751:7
        #22 0x7f22a5e250ab in nsBlockFrame::Reflow(nsPresContext*, mozilla::ReflowOutput&, mozilla::ReflowInput const&, nsReflowStatus&) /layout/generic/nsBlockFrame.cpp:1382:3
        #23 0x7f22a5e48fa0 in nsContainerFrame::ReflowChild(nsIFrame*, nsPresContext*, mozilla::ReflowOutput&, mozilla::ReflowInput const&, mozilla::WritingMode const&, mozilla::LogicalPoint const&, nsSize const&, nsIFrame::ReflowChildFlags, nsReflowStatus&, nsOverflowContinuationTracker*) /layout/generic/nsContainerFrame.cpp:1001:14
        #24 0x7f22a5e483aa in nsCanvasFrame::Reflow(nsPresContext*, mozilla::ReflowOutput&, mozilla::ReflowInput const&, nsReflowStatus&) /layout/generic/nsCanvasFrame.cpp:787:7
        #25 0x7f22a5e48fa0 in nsContainerFrame::ReflowChild(nsIFrame*, nsPresContext*, mozilla::ReflowOutput&, mozilla::ReflowInput const&, mozilla::WritingMode const&, mozilla::LogicalPoint const&, nsSize const&, nsIFrame::ReflowChildFlags, nsReflowStatus&, nsOverflowContinuationTracker*) /layout/generic/nsContainerFrame.cpp:1001:14
        #26 0x7f22a5e94b85 in nsHTMLScrollFrame::ReflowScrolledFrame(mozilla::ScrollReflowInput*, bool, bool, mozilla::ReflowOutput*) /layout/generic/nsGfxScrollFrame.cpp:762:3
        #27 0x7f22a5e95659 in nsHTMLScrollFrame::ReflowContents(mozilla::ScrollReflowInput*, mozilla::ReflowOutput const&) /layout/generic/nsGfxScrollFrame.cpp:883:3
        #28 0x7f22a5e99af9 in nsHTMLScrollFrame::Reflow(nsPresContext*, mozilla::ReflowOutput&, mozilla::ReflowInput const&, nsReflowStatus&) /layout/generic/nsGfxScrollFrame.cpp:1302:3
        #29 0x7f22a5e19b98 in nsContainerFrame::ReflowChild(nsIFrame*, nsPresContext*, mozilla::ReflowOutput&, mozilla::ReflowInput const&, int, int, nsIFrame::ReflowChildFlags, nsReflowStatus&, nsOverflowContinuationTracker*) /layout/generic/nsContainerFrame.cpp:1041:14
        #30 0x7f22a5e1943c in mozilla::ViewportFrame::Reflow(nsPresContext*, mozilla::ReflowOutput&, mozilla::ReflowInput const&, nsReflowStatus&) /layout/generic/ViewportFrame.cpp:374:7
        #31 0x7f22a5d1cfcb in mozilla::PresShell::DoReflow(nsIFrame*, bool, mozilla::OverflowChangedTracker*) /layout/base/PresShell.cpp:9576:11
        #32 0x7f22a5d270be in mozilla::PresShell::ProcessReflowCommands(bool) /layout/base/PresShell.cpp:9747:24
        #33 0x7f22a5d265b9 in mozilla::PresShell::DoFlushPendingNotifications(mozilla::ChangesToFlush) /layout/base/PresShell.cpp:4235:11
        #34 0x7f22a5ced72f in FlushPendingNotifications /builds/worker/workspace/obj-build/dist/include/mozilla/PresShell.h:1423:5
        #35 0x7f22a5ced72f in nsRefreshDriver::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp, nsRefreshDriver::IsExtraTick) /layout/base/nsRefreshDriver.cpp:2398:20
        #36 0x7f22a5cf592a in TickDriver /layout/base/nsRefreshDriver.cpp:353:13
        #37 0x7f22a5cf592a in mozilla::RefreshDriverTimer::TickRefreshDrivers(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp, nsTArray<RefPtr<nsRefreshDriver> >&) /layout/base/nsRefreshDriver.cpp:331:7
        #38 0x7f22a5cf5843 in mozilla::RefreshDriverTimer::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) /layout/base/nsRefreshDriver.cpp:347:5
        #39 0x7f22a5cf5710 in mozilla::VsyncRefreshDriverTimer::RunRefreshDrivers(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) /layout/base/nsRefreshDriver.cpp:782:5
        #40 0x7f22a5cf4daa in mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::TickRefreshDriver(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) /layout/base/nsRefreshDriver.cpp:705:16
        #41 0x7f22a5cf46b9 in mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::NotifyParentProcessVsync() /layout/base/nsRefreshDriver.cpp:622:7
        #42 0x7f22a5cf4129 in mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::NotifyVsync(mozilla::VsyncEvent const&) /layout/base/nsRefreshDriver.cpp:543:9
        #43 0x7f22a54b79c6 in mozilla::dom::VsyncChild::RecvNotify(mozilla::VsyncEvent const&, float const&) /dom/ipc/VsyncChild.cpp:68:15
        #44 0x7f22a20d8524 in mozilla::dom::PVsyncChild::OnMessageReceived(IPC::Message const&) /builds/worker/workspace/obj-build/ipc/ipdl/PVsyncChild.cpp:178:54
        #45 0x7f22a1eac22c in mozilla::ipc::PBackgroundChild::OnMessageReceived(IPC::Message const&) /builds/worker/workspace/obj-build/ipc/ipdl/PBackgroundChild.cpp:6207:32
        #46 0x7f22a1b368df in mozilla::ipc::MessageChannel::DispatchAsyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&) /ipc/glue/MessageChannel.cpp:2039:25
        #47 0x7f22a1b331c1 in mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) /ipc/glue/MessageChannel.cpp:1964:9
        #48 0x7f22a1b34645 in mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&) /ipc/glue/MessageChannel.cpp:1823:3
        #49 0x7f22a1b3528d in mozilla::ipc::MessageChannel::MessageTask::Run() /ipc/glue/MessageChannel.cpp:1851:14
        #50 0x7f22a10eb44e in mozilla::RunnableTask::Run() /xpcom/threads/TaskController.cpp:478:16
        #51 0x7f22a10c671f in mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /xpcom/threads/TaskController.cpp:781:26
        #52 0x7f22a10c5388 in mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /xpcom/threads/TaskController.cpp:617:15
        #53 0x7f22a10c5603 in mozilla::TaskController::ProcessPendingMTTask(bool) /xpcom/threads/TaskController.cpp:401:36
        #54 0x7f22a10ee996 in operator() /xpcom/threads/TaskController.cpp:126:37
        #55 0x7f22a10ee996 in mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_0>::Run() /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:531:5
        #56 0x7f22a10d9eff in nsThread::ProcessNextEvent(bool, bool*) /xpcom/threads/nsThread.cpp:1148:16
        #57 0x7f22a10e0c4a in NS_ProcessNextEvent(nsIThread*, bool) /xpcom/threads/nsThreadUtils.cpp:466:10
        #58 0x7f22a1b3c756 in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /ipc/glue/MessagePump.cpp:85:21
        #59 0x7f22a1a5cc87 in MessageLoop::RunInternal() /ipc/chromium/src/base/message_loop.cc:331:10
        #60 0x7f22a1a5cb92 in RunHandler /ipc/chromium/src/base/message_loop.cc:324:3
        #61 0x7f22a1a5cb92 in MessageLoop::Run() /ipc/chromium/src/base/message_loop.cc:306:3
        #62 0x7f22a59f1138 in nsBaseAppShell::Run() /widget/nsBaseAppShell.cpp:137:27
        #63 0x7f22a78764c3 in XRE_RunAppShell() /toolkit/xre/nsEmbedFunctions.cpp:917:20
        #64 0x7f22a1b3d64a in mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) /ipc/glue/MessagePump.cpp:235:9
        #65 0x7f22a1a5cc87 in MessageLoop::RunInternal() /ipc/chromium/src/base/message_loop.cc:331:10
        #66 0x7f22a1a5cb92 in RunHandler /ipc/chromium/src/base/message_loop.cc:324:3
        #67 0x7f22a1a5cb92 in MessageLoop::Run() /ipc/chromium/src/base/message_loop.cc:306:3
        #68 0x7f22a7875afe in XRE_InitChildProcess(int, char**, XREChildData const*) /toolkit/xre/nsEmbedFunctions.cpp:749:34
        #69 0x55cda905cb46 in content_process_main /browser/app/../../ipc/contentproc/plugin-container.cpp:57:28
        #70 0x55cda905cb46 in main /browser/app/nsBrowserApp.cpp:327:18
        #71 0x7f22b880f0b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308:16
        #72 0x55cda903994c in _start (/home/jkratzer/builds/mc-debug/firefox-bin+0x1594c)
    
    UndefinedBehaviorSanitizer can not provide additional info.
    SUMMARY: UndefinedBehaviorSanitizer: SEGV /layout/generic/nsFlexContainerFrame.cpp:625:5 in nsFlexContainerFrame::FlexItem::UpdateMainMinSize(int)
    ==3948503==ABORTING

Bugmon Analysis
Verified bug as reproducible on mozilla-central 20210913213224-b50ef8e31c4c.
The bug appears to have been introduced in the following build range:

Start: 855ec176a3c2239fe2f166431c96ee7710204b30 (20210106093742)
End: 50fb5b9343f6ddec6ec41d4e648fd8a4807ef133 (20210106090852)
Pushlog: https://hg.mozilla.org/mozilla-unified/pushloghtml?fromchange=855ec176a3c2239fe2f166431c96ee7710204b30&tochange=50fb5b9343f6ddec6ec41d4e648fd8a4807ef133

The severity field is not set for this bug.
:dholbert, could you have a look please?

For more information, please visit auto_nag documentation.

I checked to be sure this is still reproducible (it is, using mozilla-central changeset cc37b1400a58).

TYLin recently did some work in the vicinity of this assertion - maybe he could take a look, if he's got cycles?

(If not, feel free to kick this back over to me.)

I set a break point in SizeComputationInput::InitOffsets at https://searchfox.org/mozilla-central/rev/b847c844adf49013067281d3237e7ada24325a34/layout/generic/ReflowInput.cpp#2517, and see the following.

(rr) p mComputedPadding
$23 = {mWritingMode = {mWritingMode = {bits = 39 '\''}}, mMargin = {<mozilla::gfx::BaseMargin<int, nsMargin>> = {top = 1073741823, right = 0, bottom = 1073741823, left = 0}, <No data fields>}}
(rr) p mComputedBorderPadding          
$24 = {mWritingMode = {mWritingMode = {bits = 39 '\''}}, mMargin = {<mozilla::gfx::BaseMargin<int, nsMargin>> = {top = 1073742003, right = 120, bottom = 1073741943, left = 120}, <No data fields>}}
(rr) p mComputedBorderPadding.BStartEnd(wm)
$25 = -2147483350

This is a integer overflow problem. A flex item has a very huge top & bottom padding values, which accidentally equal to our unconstrained sentinel value. As a result, the border & padding value in block axis overflows and becomes negative. Any computation afterwards can go wrong ...

Daniel, any idea to deal with this gracefully?

We probably should:
(1) relax the MOZ_ASSERT to be a NS_ASSERTION (because fuzzers can trip it via huge sizes and/or sizes that accidentally match sentinel values). We'd still like to be able to notice if it fires for regular content, but it's not strictly enforceable and we don't want it to crash debug builds.

(2) Depending on why we're failing here and how badly things proceed from this point, we should consider adding a MOZ_UNLIKELY-guarded check to clean up slightly. e.g. if aNewMinSize is ending up huge, then we might consider setting aNewMinSize to equal mMainMaxSize in this case where it's exceeded it. Alternately, if mMainMaxSize is the bogus value (e.g. if it's hugely negative), then this would probably be the wrong spot to correct for that; and hopefully that's not something we have to correct for, and we can proceed and just end up with broken layout...

This patch clamps the resolvedMinSize before the size is passed into
UpdateMainMinSize, which is where the assertion is triggered.

Also, tweak the MOZ_ASSERT a bit because it's best to test an nscoord is
not NS_UNCONSTRAINEDSIZE before invoking comparison operator.

:TYLin, since this bug contains a bisection range, could you fill (if possible) the regressed_by field?
For more information, please visit auto_nag documentation.

The pushlog range on mozilla-unified in comment 2 is empty. Here is the range with s/mozilla-unified/mozilla-central.

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=855ec176a3c2239fe2f166431c96ee7710204b30&tochange=50fb5b9343f6ddec6ec41d4e648fd8a4807ef133

Set release status flags based on info from the regressing bug 1683976

https://hg.mozilla.org/mozilla-central/rev/a0192a2906a6

Bugmon Analysis
Verified bug as fixed on rev mozilla-central 20211026042228-ecc71584f003.
Removing bugmon keyword as no further action possible. Please review the bug and re-add the keyword for further analysis.