https://bugzilla.mozilla.org/show_bug.cgi?id=1799332 moved channel-prefs to Framework on macOS. One of the implications of this is that we can longer change the update channel by simply adjusting a plain text file - we now need a properly built and signed ChannelPrefs framework to do this.

For the cases where we need to do this (mainly prerelease testing), it's important that we have these readily available. We should have the build system produce N ChannelPrefs frameworks alongside a Firefox build. (Only one - the real update channel - would end up embedded into the build, however.)

We'll want to produce:

• nightlytest alongside nightly
• -cdntest and -localtest variants alongside other channels (beta, aurora, release, esr). Eg: release-cdntest

I do not think we need to do the same for the UpdateSettings framework, because the accepted mar channel ids are the same between test channels and live channels.

We will need the ability to do this before ChannelPrefs hits Beta on March 18.

bytesized - just to echo what we talked about last week. You pointed out that we can use Enterprise Policies to override the update URL as an alternative. Assuming QA is fine with this it seems like an easier (and more flexible) alternative.

While enterprise policies could be used to override the update URL, we would lose test coverage of the new macOS Frameworks. While not all update testing necessarily has to occur using these test Frameworks, I would strongly advocate for at least some testing to occur with them. This would allow us to detect any problems along the lines of Frameworks failing to load, or updates accidentally breaking code signatures due to the Frameworks, before they occur.

I will be out next week, but one more thing I wanted to drop here is that in addition to building these, we'll also need them signed. Something like the following should do the trick, but it's not testable until we have the artifacts being produced:

diff --git a/taskcluster/ci/build-mac-notarization/kind.yml b/taskcluster/ci/build-mac-notarization/kind.yml
index 8d049b36954ae..ba007fa7be619 100644
--- a/taskcluster/ci/build-mac-notarization/kind.yml
+++ b/taskcluster/ci/build-mac-notarization/kind.yml
@@ -24,11 +24,13 @@ jobs:
             copy-attributes: true
         treeherder:
             symbol: BMN
         upstream-artifacts:
             - taskType: signing
               paths:
                   - public/build/target.tar.gz
                   - public/build/target.pkg
+                  - public/build/ChannelPrefsFrameworks.tar.gz
+                  - public/build/UpdateSettingsFrameworks.tar.gz
               formats: ["apple_notarization"]
               taskId:
                   task-reference: <build-mac-signing>
diff --git a/taskcluster/gecko_taskgraph/util/signed_artifacts.py b/taskcluster/gecko_taskgraph/util/signed_artifacts.py
index 2467ff80463cc..64d0b00c5d150 100644
--- a/taskcluster/gecko_taskgraph/util/signed_artifacts.py
+++ b/taskcluster/gecko_taskgraph/util/signed_artifacts.py
@@ -76,16 +76,28 @@ def generate_specifications_of_artifacts_to_sign(
             artifacts_specifications = [
                 {
                     "artifacts": [
                         get_artifact_path(job, f"{{locale}}/target.{extension}")
                     ],
                     "formats": ["macapp", "autograph_widevine", "autograph_omnija"],
                 }
             ]
+            # en-US only, we may have extra ChannelPrefs and UpdateSettings
+            # frameworks to sign
+            if not locales:
+                artifacts_specifications.extend([
+                    {
+                        "artifacts": [
+                            get_artifact_path(job, "ChannelPrefsFrameworks.tar.gz"),
+                            get_artifact_path(job, "UpdateSettingsFrameworks.tar.gz"),
+                        ],
+                        "formats": ["macapp"],
+                    }
+                ])
             langpack_formats = ["autograph_langpack"]
 
         if "ja-JP-mac" in locales and build_platform in LANGPACK_SIGN_PLATFORMS:
             artifacts_specifications += [
                 {
                     "artifacts": [
                         get_artifact_path(job, "ja-JP-mac/target.langpack.xpi")
                     ],

(Note that the artifact names need to line up with whatever new artifacts come out of the builds.)

This has turned out to be more complicated than I hoped. Or, at least, it involves a lot of build system and taskgraph stuff that I don't understand very well. I'm working on it as quickly as I can, but I think it's likely that it will not be ready by the March 18th deadline that we were hoping for. I'm not sure what to do about this. I can uplift it to beta if it doesn't make it in by then. But QA may still be a bit confused.

I'm going to ask my manager about talking to them about this, but I thought maybe I should alert you too, @jcristau.

Is reverting bug 1799332 et al from Beta post-merge (but before we build 125.0b1) a feasible option? I'm a bit nervous about that too, TBH, but maybe it's the least-disruptive option?

(In reply to Robin Steuber (they/them) [:bytesized] from comment #4)

This has turned out to be more complicated than I hoped. Or, at least, it involves a lot of build system and taskgraph stuff that I don't understand very well. I'm working on it as quickly as I can, but I think it's likely that it will not be ready by the March 18th deadline that we were hoping for. I'm not sure what to do about this. I can uplift it to beta if it doesn't make it in by then. But QA may still be a bit confused.

I'm not sure what this task is referring to, but I have to assume that this is to build these test-frameworks automatically with every nightly? At a minimum we have enterprise policies to test downloads, and it was my understanding that building a one-off Framework for ChannelPrefs and UpdateSettings shouldn't be too difficult to test downloads using the new Frameworks. It really comes down to signing two Frameworks with a channel set to nightlytest. I would vote against reverting bug 1799332 if at all possible.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #5)

Is reverting bug 1799332 et al from Beta post-merge (but before we build 125.0b1) a feasible option? I'm a bit nervous about that too, TBH, but maybe it's the least-disruptive option?

I agree with spohl that reverting Bug 1799332 would be undesirable. If it comes to it, there are stop gap measures that we can use to enable QA testing. For example, they could use an enterprise policy to change the update URL to one that has %CHANNEL% pre-substituted with the desired channel. Ideally, we would use a mechanism that would test a bit more completely (i.e. what we are trying to do here). But there are other options that are only slightly less comprehensive.

Originally, I wanted to include some testing in this patch. But given the time constraints, I'm spinning that work off into Bug 1885240. In order to try to get this bug done faster.

(In reply to Robin Steuber (they/them) [:bytesized] from comment #4)

This has turned out to be more complicated than I hoped. Or, at least, it involves a lot of build system and taskgraph stuff that I don't understand very well. I'm working on it as quickly as I can, but I think it's likely that it will not be ready by the March 18th deadline that we were hoping for. I'm not sure what to do about this. I can uplift it to beta if it doesn't make it in by then. But QA may still be a bit confused.

Would it be feasible, in the short term, to just build/sign the beta-cdntest ChannelPrefs framework once, e.g. like we did for nightlytest with https://hg.mozilla.org/projects/pine/rev/b478ecf6cfe27859a9203ea474a2db104c054bbb? (And then repeat for release-cdntest, although that won't be necessary until after the 125 release.) Then we can hand it off to QA for 125 beta testing, and that can be unblocked, without waiting for a cleaner solution where it's produced alongside each firefox build?

(In reply to Julien Cristau [:jcristau] from comment #9)

Would it be feasible, in the short term, to just build/sign the beta-cdntest ChannelPrefs framework once, e.g. like we did for nightlytest with https://hg.mozilla.org/projects/pine/rev/b478ecf6cfe27859a9203ea474a2db104c054bbb? (And then repeat for release-cdntest, although that won't be necessary until after the 125 release.) Then we can hand it off to QA for 125 beta testing, and that can be unblocked, without waiting for a cleaner solution where it's produced alongside each firefox build?

That seems possible. I'll give it a try today.

Does this commit look right to you for beta-cdntest? I assume that we want it signed with the same certificate that beta is signed with?

I believe that after this one, I'll also want to make frameworks for the beta-localtest, aurora-cdntest and aurora-localtest channels?

(In reply to Robin Steuber (they/them) [:bytesized] from comment #11)

Does this commit look right to you for beta-cdntest? I assume that we want it signed with the same certificate that beta is signed with?

That commit looks plausible, yes. I /think/ we're using the same apple certificate for all channels, so the signed ChannelPrefs framework from that pine push should work; Heitor, can you confirm?

I believe that after this one, I'll also want to make frameworks for the beta-localtest, aurora-cdntest and aurora-localtest channels?

Right.

(In reply to Julien Cristau [:jcristau] from comment #13)

(In reply to Robin Steuber (they/them) [:bytesized] from comment #11)

Does this commit look right to you for beta-cdntest? I assume that we want it signed with the same certificate that beta is signed with?

That commit looks plausible, yes. I /think/ we're using the same apple certificate for all channels, so the signed ChannelPrefs framework from that pine push should work; Heitor, can you confirm?

That's correct. All production certs are the same across all "channels" (nightly, beta, release) since they use the same scriptworkers.

As I mentioned in Comment 10, I made one-offs of the test frameworks we need. I'm attaching those here.

I'd still like for this work to be done at some point. But now that the one-offs are ready, this has lost a bit of its urgency. So I'm going to unassign myself for now. It's quite likely that the work that we are doing for Bug 1885985 is going to make this fairly trivial, so maybe we'll do this after we finish that work.

The bug is marked as tracked for firefox125 (beta). We have limited time to fix this, the soft freeze is in 6 days. However, the bug still isn't assigned and has low priority.

:Amir, could you please find an assignee and increase the priority for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Removing the tracking for 125. This work is no longer urgent as described in Comment 15.

One additional framework added