Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename "CII Best Practices Badge" to "OpenSSF Best Practices Badge" #1549

Open
david-a-wheeler opened this issue Jan 27, 2022 · 19 comments
Open
Assignees
Labels
check/CII-Best-Practices kind/docs Improvements or additions to documentation
Projects

Comments

@david-a-wheeler
Copy link
Contributor

Describe the bug
Rename "CII Best Practices Badge" to "OpenSSF Best Practices Badge"; the project recently changed its hame.

@david-a-wheeler david-a-wheeler added the kind/bug Something isn't working label Jan 27, 2022
@azeemshaikh38
Copy link
Contributor

@naveensrinivasan would you able to tackle this? Can be a P1/P2 item in the current list we have.

@naveensrinivasan
Copy link
Member

I can.

@naveensrinivasan naveensrinivasan self-assigned this Jan 28, 2022
@justaugustus justaugustus added this to Backlog in Scorecard via automation Feb 23, 2022
@david-a-wheeler
Copy link
Contributor Author

There are 2 related issues, the name of the metric includes CII, and the description includes CII. I expect the metric rename is a bigger deal, so we could change the description first & then do the actual name change later.

@justaugustus
Copy link
Member

Suggestion:
Copy/paste the existing check/use the same code path for the new one.
Mark the existing as deprecated in one release.
Remove in the next release.

@gabibguti
Copy link
Contributor

gabibguti commented Apr 25, 2023

May I suggest that we also change the check description to make it clear we are verifying if the project completed the OpenSSF Best Practices form?
Because, with the current description, it sounds like the check wants you to add a badge to your project. In fact, if you score a 0/10 in this check, Scorecard reports "no badge found", but deep down it looks for the form, not the badge. You might not want to display such badge, so it refrain users from completing this check.

@david-a-wheeler
Copy link
Contributor Author

david-a-wheeler commented Apr 25, 2023

May I suggest that we also change the check description to make it clear we are verifying if the project completed the OpenSSF Best Practices form?

It shouldn't. It should determine how fully it meets a badge. You can "fill in a form" and not earn a BP badge. In fact, you can "fill in a form" to show you meet 0 requirements today :-).

Because, with the current description, it sounds like the check wants you to add a badge to your project.

Are you assuming that "badge" is "a graphical image"? That's not what "Best Practices Badge" means. In particular, the current text asks if you have earned a badge, not if you display a badge.

In fact, if you score a 0/10 in this check, Scorecard reports "no badge found", but deep down it looks for the form, not the badge. You might not want to display such badge, so it refrain users from completing this check.

If I understand you correctly, that sounds like a bug, let's not enshrine that. I may not be understanding you correctly. Where's the code that does the evaluation?

Here's the current description: https://github.com/ossf/scorecard/blob/main/docs/checks.md#cii-best-practices

The OpenSSF Best Practices badge has 3 tiers: passing, silver, and gold. We give full credit to projects that meet the gold criteria, which is a significant achievement for many projects. Lower scores represent a project that is at least working to achieve a badge, with increasingly more points awarded as more criteria are met.
gold badge: 10
silver badge: 7
passing badge: 5
in progress badge: 2

I'd like to revisit this grading. "Gold" is really hard to achieve. Earning passing is a big deal. I'd suggest this kind of ranking instead:

  • Gold: 10
  • Silver: 9
  • Passing: 8
  • In progress: Rate 1..5 depending on progress (1=started but 0 progress, 5=99%, convert to closest integer).
  • Not in progress: 0

@david-a-wheeler
Copy link
Contributor Author

I think you're using the term "badge" to mean a graphical image, but in the "best practices badge" we mean the English word, that is, something like an indication of meeting certain criteria. It doesn't matter if a graphical image is displayed, the issue is, "has this project met certain criteria"?

@david-a-wheeler
Copy link
Contributor Author

I think the solution is to change the message "no badge found" to "no effort to earn a best practices badge found". It doesn't matter if a graphical badge image is displayed - what matters is if a project is trying to earn a best practices badge.

@david-a-wheeler
Copy link
Contributor Author

Also: I would delete the text & list beginning with "To earn the passing badge, the project MUST:". That isn't the full list of criteria; the full list is linked-to above.

@gabibguti
Copy link
Contributor

I think you're using the term "badge" to mean a graphical image, but in the "best practices badge" we mean the English word, that is, something like an indication of meeting certain criteria. It doesn't matter if a graphical image is displayed, the issue is, "has this project met certain criteria"?

Yes, I meant "badge" as a graphical image. I do understand it has a different meaning in English now, thanks for the explanation. Still, I believe "badge" in the GitHub context maps to README badges such as workflow status badges, not "meeting certain criteria".

@gabibguti
Copy link
Contributor

If I understand you correctly, that sounds like a bug, let's not enshrine that. I may not be understanding you correctly. Where's the code that does the evaluation?

results = checker.CreateMinScoreResult(name, "no badge detected")

@david-a-wheeler
Copy link
Contributor Author

david-a-wheeler commented Apr 25, 2023

Yes, I meant "badge" as a graphical image. I do understand it has a different meaning in English now, thanks for the explanation. ...

The project is literally named the "OpenSSF Best Practices badge", and I think no one is interested in a rename.

This is not a big deal, I think the Scorecard text could be tweaked to make this very clear. Here's one proposal.

FIRST: The Scorecard README https://github.com/ossf/scorecard says:

Does the project have an [OpenSSF (formerly CII) Best Practices Badge](https://bestpractices.coreinfrastructure.org/en)?

I would change that to:

Has the project earned a [OpenSSF (formerly CII) Best Practices Badge](https://bestpractices.coreinfrastructure.org/) at the passing, silver, or gold level?

Notice the switch from "have" to "earned" (which is what matters), and the express mention of passing / silver / gold level; both make it clear that this isn't just about an image. Also, I removed "/en" from the URL; the "/en" forces English display. Don't force the language unless you know you want the user to only see the English display. In this case, I think you want the user to see the browser's preferred locale.

SECOND:

In the details at https://github.com/ossf/scorecard/blob/main/docs/checks.md#cii-best-practices

I would change this text:

This check determines whether the project has earned an OpenSSF (formerly CII) Best Practices Badge, which indicates that the project uses a set of security-focused best development practices for open source software. The check uses the URL for the Git repo and the OpenSSF Best Practices badge API.

The OpenSSF Best Practices badge has 3 tiers: passing, silver, and gold. We give full credit to projects that meet the [gold criteria](https://bestpractices.coreinfrastructure.org/criteria/2), which is a significant achievement for many projects. Lower scores represent a project that is at least working to achieve a badge, with increasingly more points awarded as more criteria are met.

Into this:

This check determines whether the project has earned an OpenSSF (formerly CII) Best Practices Badge at the passing, silver, or gold level. The OpenSSF Best Practices badge indicates whether or not the project uses a set of security-focused best development practices for open source software. The check uses the URL for the Git repo and the OpenSSF Best Practices badge API.

The OpenSSF Best Practices badge has 3 tiers: passing, silver, and gold. We give full credit to projects that meet the [gold criteria](https://bestpractices.coreinfrastructure.org/criteria/2), which is a very significant achievement for projects and requires multiple developers. Lower scores represent a project that has met the silver criteria, passing criteria, or trying achieve the passing badge, is at least working to achieve a badge, with increasingly more points awarded as more criteria are met. Note that even meeting the passing criteria is a significant achievement.

Delete the text "To earn the passing badge, the project MUST:" and the list that follows it. The page already links to the full criteria; listing the criteria here suggests it's the complete list (and it's not).

THIRD:

Regarding:

scorecard/checks/evaluation/cii_best_practices.go

Line 44 in 2bde7ca

Change:

 results = checker.CreateMinScoreResult(name, "no badge detected"

to:

 results = checker.CreateMinScoreResult(name, "no effort to earn a best practices badge found"

FOURTH:

As a separate action, I suggest using a different value ranking (you're renaming it anyway):

  • Gold: 10
  • Silver: 9
  • Passing: 8
  • In progress: Rate 1..5 depending on progress (1=started but 0 progress, 5=99%, convert to closest integer).
  • Not in progress: 0

Passing is really good, only about 20% of current projects pursuing a badge achieve a passing badge. Gold is really hard; it requires multiple developers, and that fact by itself excludes the majority of OSS projects.

@gabibguti
Copy link
Contributor

I agree with the suggestions

@david-a-wheeler
Copy link
Contributor Author

Okay, I've create PR #2907 to clarify things.

That will give us a better starting point, as it's clearer, but it does NOT resolve this issue. It does not rename the Scorecard criterion, nor does it change the scoring system as suggested above. It simply makes the existing system easier to understand.

So hopefully that PR will be accepted, and that will make it easier to implement this one :-).

@david-a-wheeler
Copy link
Contributor Author

Okay, back to the main discussion, presuming that PR #2907 or something like it will be accepted.

The main problem now is there's a metric named "CII-Best-Practices" that should be renamed to "OpenSSF-Best-Practices". If we're going to rename it, I suggest also updating to a different scoring system, it's an ideal time to do it. My recommendation, as noted above:

  • Gold: 10
  • Silver: 9
  • Passing: 8
  • In progress: Rate 1..5 depending on progress (1=started but 0 progress, 5=99%, convert to closest integer).
  • Not in progress: 0

I personally think the metric should have more weight in Scorecard, but keeping the existing weight is okay if others prefer it as-is. I think it's more important to have a better scoring regardless of its weight.

If possible, I'd love for the Scorecard JSON file to refer to the URL of the Best Practices badge entry. That way, readers of this metric could quickly jump to the badge entry to learn more information. E.g., here's an example of a current result:

{"name":"CII-Best-Practices","score":10,"reason":"badge detected: gold","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/e1d4f3726920cc49615c1cae8d920a75975500d3/docs/checks.md#cii-best-practices"}}

Somewhere in there it'd be great to have a link to https://bestpractices.coreinfrastructure.org/en/projects/1 (e.g., inside "documentation" add "additional-information": "https://bestpractices.coreinfrastructure.org/en/projects/1"), so that it's easy to start from the Scorecard data and go to the Best Practices Badge data.

I don't know if it's important to continue to support the older metric "CII-Best-Practices". If you do, you could keep the old score with a weight of "0". But unless some user needs it, I'd just drop it, it's confusing & it's not clear it's helpful for backwards compatibility.

@jsoref
Copy link
Contributor

jsoref commented Jan 28, 2024

@david-a-wheeler #1549 (comment) is really hard to follow because GitHub forces users to independently horizontally scroll each blob of text.

diff tagged markdown

Here's the laziest improvement to reading the changes (just grouping everything together into ```diff tagged markdown:

-Does the project have an [OpenSSF (formerly CII) Best Practices Badge](https://bestpractices.coreinfrastructure.org/en)?
+Has the project earned a [OpenSSF (formerly CII) Best Practices Badge](https://bestpractices.coreinfrastructure.org/) at the passing, silver, or gold level?
-This check determines whether the project has earned an OpenSSF (formerly CII) Best Practices Badge, which indicates that the project uses a set of security-focused best development practices for open source software. The check uses the URL for the Git repo and the OpenSSF Best Practices badge API.
-
-The OpenSSF Best Practices badge has 3 tiers: passing, silver, and gold. We give full credit to projects that meet the [gold criteria](https://bestpractices.coreinfrastructure.org/criteria/2), which is a significant achievement for many projects. Lower scores represent a project that is at least working to achieve a badge, with increasingly more points awarded as more criteria are met.
+This check determines whether the project has earned an OpenSSF (formerly CII) Best Practices Badge at the passing, silver, or gold level. The OpenSSF Best Practices badge indicates whether or not the project uses a set of security-focused best development practices for open source software. The check uses the URL for the Git repo and the OpenSSF Best Practices badge API.
+
+The OpenSSF Best Practices badge has 3 tiers: passing, silver, and gold. We give full credit to projects that meet the [gold criteria](https://bestpractices.coreinfrastructure.org/criteria/2), which is a very significant achievement for projects and requires multiple developers. Lower scores represent a project that has met the silver criteria, passing criteria, or trying achieve the passing badge, is at least working to achieve a badge, with increasingly more points awarded as more criteria are met. Note that even meeting the passing criteria is a significant achievement.
- results = checker.CreateMinScoreResult(name, "no badge detected"
+ results = checker.CreateMinScoreResult(name, "no effort to earn a best practices badge found"

diff tagged markdown with text wrapped for readability

-Does the project have an
+Has the project earned a
 [OpenSSF (formerly CII) Best Practices Badge](
-https://bestpractices.coreinfrastructure.org/en
-)?
+https://bestpractices.coreinfrastructure.org/
+) at the passing, silver, or gold level?
 This check determines whether the project has earned an OpenSSF (formerly CII) Best Practices
-Badge, which indicates that
+Badge at the passing, silver, or gold level.
+The OpenSSF Best Practices badge indicates whether or not
 the project uses a set of security-focused best development practices for open source software.
 The check uses the URL for the Git repo and the OpenSSF Best Practices badge API.
 
 The OpenSSF Best Practices badge has 3 tiers: passing, silver, and gold.
 We give full credit to projects that meet the [gold criteria](https://bestpractices.coreinfrastructure.org/criteria/2),
-which is a significant achievement for many projects.
+which is a very significant achievement for projects and requires multiple developers.
 Lower scores represent a project that
-is at least working to achieve a badge,
+has met the silver criteria, passing criteria, or trying achieve the passing badge,
+is at least working to achieve a badge,
 with increasingly more points awarded as more criteria are met.
+Note that even meeting the passing criteria is a significant achievement.
 results = checker.CreateMinScoreResult(name,
-"no badge detected"
+"no effort to earn a best practices badge found"

--- fwiw, the text changes proposed seem quite reasonable (once I'm able to actually compare before/after).

@david-a-wheeler
Copy link
Contributor Author

jsoref - I think the text changes were already merged in #2907 - I think what's undone is renaming the criterion (which changes the name of a key!).

@afmarcum
Copy link
Contributor

afmarcum commented Feb 5, 2024

Newer structure probe referred to as OpenSSF Best Practices, but older references remain CII Best Practices.

Review old references in code and documentation and update. ~1 hr effort.

@afmarcum afmarcum added the good first issue Good for newcomers label Feb 5, 2024
@spencerschrock
Copy link
Contributor

but older references remain CII Best Practices

I believe we intentionally didn't rename the check in the code, for both backwards compatibility reasons, as well as BigQuery reasons.

@afmarcum afmarcum added kind/docs Improvements or additions to documentation and removed kind/bug Something isn't working labels Mar 7, 2024
@spencerschrock spencerschrock removed the good first issue Good for newcomers label Apr 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
check/CII-Best-Practices kind/docs Improvements or additions to documentation
Projects
Scorecard
Backlog
Status: Todo
Development

No branches or pull requests

9 participants