Open Bug 1898490 Opened 27 days ago Updated 21 hours ago

DevTools stop responsing

Categories

(DevTools :: General, defect, P2)

Firefox 126
defect

Tracking

(firefox-esr115 unaffected, firefox126 wontfix, firefox127 fixed, firefox128 fixed)

REOPENED
128 Branch
Tracking Status
firefox-esr115 --- unaffected
firefox126 --- wontfix
firefox127 --- fixed
firefox128 --- fixed

People

(Reporter: matic.mercina, Assigned: ochameau)

References

(Regression)

Details

(Keywords: regression)

Attachments

(9 files)

This has been happening after I updated to Firefox 126 on my Windows 11 laptop.

When running a locally hosted web application, the DevTools will often not open. The first time this happens, a blank window will open instead. After I close the blank window, neither F12 or "Inspect element" will do anything in the affected tab anymore. Opening a new tab will temporarily allow the use of DevTools again, but each time I close them, there is a chance they wont open anymore.

What I have already tried

My first thought was that my Firefox profile was broken. I therefore deleted the profile and created a new one (but I did login with my Firefox account, it's not a completely fresh profile). On day 1 after doing this I was certain I have solved the issue, since I had no problems the entire day. The day after that, the bug started re-appearing.

What can I do?

Is there any way for me to get more information about the issue? Does Firefox store logs somewhere on the hard drive, or can I configure it to do so?

Attached image Devtools.png

A screenshot of the blank DevTools window.

Thanks for the report Matic(In reply to Matic Mercina from comment #0)

What can I do?

Is there any way for me to get more information about the issue?

If you can share the local application code you're seeing this on, that would be great.
Also, are you experiencing that consistently?
Do you had a lot of other opened tabs when this happens? We know there's an issue with slow to initialize DevTools (Bug 1898463), so maybe you're hitting this, but it never settles?
You may also check that it's not a performance issue, recording a profile with https://profiler.firefox.com/ and sharing it here.

Does Firefox store logs somewhere on the hard drive, or can I configure it to do so?

You can open the Browser Console https://firefox-source-docs.mozilla.org/devtools-user/browser_console/index.html and copy all messages here when this happens.

Flags: needinfo?(matic.mercina)

(In reply to Nicolas Chevobbe [:nchevobbe] from comment #2)

If you can share the local application code you're seeing this on, that would be great.

Unfortunately it is part of a large legacy application, which is company code I am not allowed to share.

Also, are you experiencing that consistently?

The error usually happens at most an hour into a debug sesssion.

Do you had a lot of other opened tabs when this happens? We know there's an issue with slow to initialize DevTools (Bug 1898463), so maybe you're hitting this, but it never settles?

I usually have 2-5 tabs open.

You can open the Browser Console https://firefox-source-docs.mozilla.org/devtools-user/browser_console/index.html and copy all messages here when this happens.

I've attached two screenshots.

The first (Error log - blank DevTools.png) is when I pressed F12 and a blank window opened. Unfortunately, I am not entirely certain which message is the one that showed the moment I pressed F12, since having the Browser Console open seemingly makes the bug impossible (or at least way lesss likely) to trigger. Consequently, I had to trigger the bug with the Browser Console closed, and then open the console to see the output.

The second (Error log - F12 unresponsive.png ) is after I closed the blank window and tried to open the DevTools again with F12 (typeError: this.topWindow is null) and "Inspect element" (Error: can't select tool....).

Flags: needinfo?(matic.mercina)

When you reproduce the issue can you open about:processes and share the information there?
Maybe you can also give a try to Firefox ESR (https://www.mozilla.org/en-US/firefox/enterprise/) which is slightly older, it would help verify this is a Firefox 126 regression.

Flags: needinfo?(matic.mercina)
Attached image about:process
Flags: needinfo?(matic.mercina)

I've attached a screenshot of the about:process page after reproducing the issue on tab 3 (the one with 202MB memory usage on the screenshot).

It may be of note that at least on this occasion, it was only the "Network" tab of the DevTools that was blank at first. Closing and reopening the tools then first caused the entire window to be blank, after which the tools no longer opened, as previously observed.

Unfortunately, there is no actionable data out of about:processes.
So far the most useful data is attachment 9404054 [details] with the failure around watchTargets request.

Do you reproduce DevTools issues only against your company application?
Does that reproduce against a fresh instance of firefox with only one tab for that application?

Recording and sharing a performance profile recorded while trying to open DevTools the first time could help.
https://profiler.firefox.com/ Helps enabling the profiler in Firefox (nothing to install, this page will just help you enable it)

I tried to break watchTargets with some extreme scenarios.

  • Using a page with a worker having an infinite loop doesn't break DevTools:
    http://techno-barje.fr/fission/worker-busy/
    You are able to spawn DevTools and evaluate JS against the blocked workers (DevTools interrupt the infinite loop to execute the console input).

  • Having a page paused doesn't prevent the other tabs from opening DevTools.

  • Using a page doing an infinite loop... does prevent opening DevTools in any other page:
    data:text/html,<script>let i = 0; while(true) {i++; " ".repeat(100000);};</script>
    With such extreme scenario. DevTools would be left blank, and you would get similar exception as in comment 3:

JavaScript error: resource://devtools/server/connectors/js-process-actor/ContentProcessWatcherRegistry.sys.mjs, line 394: NotFoundError: No such JSProcessActor 'DevToolsProcess'
JavaScript error: resource://devtools/server/connectors/js-process-actor/ContentProcessWatcherRegistry.sys.mjs, line 394: NotFoundError: No such JSProcessActor 'DevToolsProcess'
JavaScript error: resource://devtools/server/connectors/js-process-actor/ContentProcessWatcherRegistry.sys.mjs, line 394: NotFoundError: No such JSProcessActor 'DevToolsProcess'
console.error: "Failed to sendQuery in DevToolsProcessParent" "DevToolsProcessParent:watchTargets"
console.error: "AbortError: Actor 'DevToolsProcess' destroyed before query 'DevToolsProcessParent:watchTargets' was resolved"
console.error: "Failed to sendQuery in DevToolsProcessParent" "DevToolsProcessParent:addOrSetSessionDataEntry"
console.error: "AbortError: Actor 'DevToolsProcess' destroyed before query 'DevToolsProcessParent:addOrSetSessionDataEntry' was resolved"
console.error: "Error while calling actor 'watcher's method 'watchTargets'" "Actor 'DevToolsProcess' destroyed before query 'DevToolsProcessParent:watchTargets' was resolved"

But looking at comment 7 data with about:processes, it doesn't seem related to an infinite loop.
It isn't clear how a Content process could possibly be stuck and not respond to JSProcess actor messages.

(In reply to Alexandre Poirot [:ochameau] from comment #9)

Do you reproduce DevTools issues only against your company application?

So far I have only ever reproduced the error against our company application, but I almost never use the DevTools anywhere else.

Does that reproduce against a fresh instance of firefox with only one tab for that application?

So far I haven't been able to immediately reproduce the error on a fresh instance with a single tab open. I have been able to reproduce the issue with a single tab open, but it was in an old instance of firefox in which i had opened and closed multiple tabs/DevTools before that.

Recording and sharing a performance profile recorded while trying to open DevTools the first time could help.

I am currently running firefox ESR, and will continue to do so for a few days (or until I manage to reproduce the error), just to confirm the issue is related to the current release. I will try getting more data after that.

Here is a snippet to execute in the browser console, just after reproducing a blank devtools, without trying to close the tab or doing anything else:
ChromeUtils.getAllDOMProcesses().map(p=>p.getActor("DevToolsProcess")).map(async a=>{try{await a.sendQuery("DevToolsProcessParent:packet");}catch(e){console.error(a.manager.name, a.manager.childID, a.manager.remoteType, a.manager.canSend, "exception", e)}})

It should print something like the attached screenshot.
It will help understand which particular process if failing and causing this issue.

I managed to reproduce with a script like this:

const ProcessTools = Cc["@mozilla.org/processtools-service;1"].getService(
  Ci.nsIProcessToolsService
);

setInterval(()=>{
  ProcessTools.kill(ChromeUtils.getAllDOMProcesses().filter(r=>r.remoteType=="privilegedabout")[0].osPid);
  gBrowser.addTab("about:home", {triggeringPrincipal:Services.scriptSecurityManager.getSystemPrincipal()})
});

When opening devtools on another tab.
This force creating and destroying content processes for about:home, which is privilegedabout.
So this confirms that if you are opening DevTools while a content process is being destroyed, it will break DevTools opening.

It is a bit surprising as the failing code looks like this:
https://searchfox.org/mozilla-central/rev/23efe2c8c5b3a3182d449211ff9036fb34fe0219/devtools/server/actors/watcher.js#251-267

    const domProcesses = ChromeUtils.getAllDOMProcesses();
    const promises = [];
    for (const domProcess of domProcesses) {
      if (domProcess == topLevelTargetProcess) {
        continue;
      }
      promises.push(
        domProcess.getActor(this._jsActorName).watchTargets({
          watcherActorID: this.actorID,
          targetType,
        })

In theory getAllDOMProcesses would only return valid processes, but somehow it ends up being destroyed which calling watchTargets.
It is being destroyed while processing this request in that content process.

All watcher code dispatching to all the content processes should probably avoid throwing when one content process throws while being destroyed.

The issue is that some calls to JSProcess Actor's would throw when trying to reach
content processes that get destroyed while DevTools are opening.
We have to reach out all the content processes as navigations may spawn
document in a process other than the currently's debugged tab's process.

We should ignore these failures. There isn't any explicit flag on DOM Process object
to identify destroyed processes, but canSend is a good fallback.

Assignee: nobody → poirot.alex
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Severity: -- → S3
Priority: -- → P2
Pushed by apoirot@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/bd636e92feb0
[devtools] Ignore closing content processes when trying to open DevTools. r=devtools-reviewers,nchevobbe

It looks like the killing API I'm using is easily throwing on Windows.
It sounds fine running this only on linux given that race condition is OS-agnostic.
https://treeherder.mozilla.org/jobs?repo=try&revision=c7b03e32f722a40b819b2b851ba9a2452c1b268c

Flags: needinfo?(poirot.alex)
Pushed by apoirot@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9b314490c4bf
[devtools] Ignore closing content processes when trying to open DevTools. r=devtools-reviewers,nchevobbe
Status: ASSIGNED → RESOLVED
Closed: 18 days ago
Resolution: --- → FIXED
Target Milestone: --- → 128 Branch

Matic, Thanks a lot for your report. I think I found the culprit behind your issue and fixed it in Firefox Nightly.
Would you be able to test your broken scenario against Firefox Nightly (https://www.mozilla.org/firefox/all/#product-desktop-nightly)
and confirm that DevTools now work fine for you?

Flags: needinfo?(matic.mercina)
Keywords: regression
Regressed by: 1866814

Set release status flags based on info from the regressing bug 1866814

Comment on attachment 9404791 [details]
Bug 1898490 - [devtools] Ignore closing content processes when trying to open DevTools.

Beta/Release Uplift Approval Request

  • User impact if declined: Some users may have broken devtools when opening them. The only way to open them is to close the tab and reopen the same URL in a new tab.
    This breakage reproduces when opening devtools while one content process is being destroyed while opening devtools. It is a race condition, but not so hard to reproduce.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: No
  • If yes, steps to reproduce: No reliable steps to reproduce without a browser console script (comment 13).
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Low risk as this only add try/catch block to ignore this race condition.
  • String changes made/needed:
  • Is Android affected?: No
Attachment #9404791 - Flags: approval-mozilla-beta?

Comment on attachment 9404791 [details]
Bug 1898490 - [devtools] Ignore closing content processes when trying to open DevTools.

127 is in RC now, moving the approval request accordingly.

Attachment #9404791 - Flags: approval-mozilla-beta? → approval-mozilla-release?

Comment on attachment 9404791 [details]
Bug 1898490 - [devtools] Ignore closing content processes when trying to open DevTools.

We have an RC2, taking as a ride-along.

Attachment #9404791 - Flags: approval-mozilla-release? → approval-mozilla-release+

(In reply to Alexandre Poirot [:ochameau] from comment #20)

Matic, Thanks a lot for your report. I think I found the culprit behind your issue and fixed it in Firefox Nightly.
Would you be able to test your broken scenario against Firefox Nightly (https://www.mozilla.org/firefox/all/#product-desktop-nightly)
and confirm that DevTools now work fine for you?

Sorry for the late reply, I am currently on vacation so I cannot test at this time. I will be sure to install Firefox Nightly (or beta, if it already contains the fix?) build when I get back.

Flags: needinfo?(matic.mercina)

Unfortunately the bug still appears in the latest nightly version.

As instructed, I tried to execute this snippet:

ChromeUtils.getAllDOMProcesses().map(p=>p.getActor("DevToolsProcess")).map(async a=>{try{await a.sendQuery("DevToolsProcessParent:packet");}catch(e){console.error(a.manager.name, a.manager.childID, a.manager.remoteType, a.manager.canSend, "exception", e)}})

But I don't know how to execute a code snippet in the browser console, since, unlike the regular DevTools console, there does not appear to be a text entry field anywhere.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

(In reply to Matic Mercina from comment #28)

Unfortunately the bug still appears in the latest nightly version.

As instructed, I tried to execute this snippet:

ChromeUtils.getAllDOMProcesses().map(p=>p.getActor("DevToolsProcess")).map(async a=>{try{await a.sendQuery("DevToolsProcessParent:packet");}catch(e){console.error(a.manager.name, a.manager.childID, a.manager.remoteType, a.manager.canSend, "exception", e)}})

But I don't know how to execute a code snippet in the browser console, since, unlike the regular DevTools console, there does not appear to be a text entry field anywhere.

Thanks for the feedback Matic!

For the Browser Console input, in the main Firefox window, go to about:config, search for devtools.chrome.enabled and set it to true. Then, next time you open the Browser Console, you should have the input visible and be able to execute snippet (see https://firefox-source-docs.mozilla.org/devtools-user/browser_console/index.html#browser-console-command-line)

Attached image Script result.png

I've added a screenshot of the browser console output of the snippet.

It does not seem to be failing in the way you'd probably expected - all of the promises are fulfilled. There is however an error in the console, but I'm not sure it's related, as I am sometimes getting that same error when running the snippet even when the DevTools are seemingly working.

Thanks Matic for testing, I think I found something in bug 1892411 which may relate to similar issue again.
I'll try to get the fix landed soon and ask you to test again on Nightly once it is fixed.

See Also: → 1892411
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: