Proposing a community maintained database of PyPI package vulnerabilities

nhatkhai · 2021-04-25T01:17:30.602Z

It would also make legacy eggs arbitrary code execution during pip install or import a no longer a problem in term of security

ammaraskar · 2021-04-25T01:44:36.570Z

Just for inspiration/precedent, the Rust ecosystem has a similar system for packages on crates.io: Advisories › RustSec Advisory Database and GitHub - RustSec/cargo-audit: Audit Cargo.lock files for crates with security vulnerabilities acting as an auditing tool for dependencies.

oliverchang · 2021-04-26T01:43:55.494Z

Indeed!

Ruby also has a community maintained database of vulnerabilities: GitHub - rubysec/ruby-advisory-db: A database of vulnerable Ruby Gems

And there is a Go proposal/prototype to do the same: vulndb - Git at Google

tiran · 2021-04-26T15:42:41.132Z

Thanks Oliver! I like the general idea. I’ll try to find some time to read your proposal and give feed after I have dealt with 3.10 feature freeze frenzy.

@brettcannon I think @oliverchang’s proposal belongs in the packaging category to draw in more attention from packaging folks.

There is already a commercial security database for Python packages at https://pyup.io/. They release their database once a month at GitHub - pyupio/safety-db: A curated database of insecure Python packages

brettcannon · 2021-04-26T22:10:52.643Z

@tiran depends if this is more for the project on GitHub or coming to a general conclusion. I read:

oliverchang:

Would you be open to having this database live under Python Software Foundation · GitHub as a community owned database of vulnerabilities?

as the key question. If it’s more for the idea then I can move it back.

oliverchang · 2021-04-27T01:13:59.453Z

@brettcannon Yes that’s the key question from me

For more general packaging/warehouse discussion, I opened an issue at github/pypa/warehouse/issues/9407 (I’m not able to post the full link), or should there be another discussion topic on here for that?

uranusjr · 2021-04-27T02:57:17.480Z

Just a note, the psf and pypa orgs are managed by different people and have different criteria to join. If joining psf does not end up making sense (I don’t personally have an opinion but don’t have a say anyway), it may make sense for you to join pypa instead if you are inclined to. See also: PyPA Members, And How To Join — PyPA documentation

Blackward · 2021-05-01T14:41:40.935Z

Hi Oliver,

I love the idea of such a database! I hope either the PSF or the PyPI guys will love it too…

Thumbs up!

Cheers, Dominik

dustin · 2021-05-12T19:41:15.305Z

uranusjr:

Just a note, the psf and pypa orgs are managed by different people and have different criteria to join. If joining psf does not end up making sense (I don’t personally have an opinion but don’t have a say anyway), it may make sense for you to join pypa instead if you are inclined to. See also: PyPA Members, And How To Join — PyPA documentation

I talked to @EWDurbin about this offline and we both agreed that this probably would be more appropriate under the pypa org instead.

Can a mod move this back to the Packaging category so we can give this a little more time for discussion with the members there?

Assuming this doesn’t get any significant opposition there, I’ll probably kick that off sometime next week.

brettcannon · 2021-05-13T22:16:26.050Z

dustin:

Can a mod move this back to the Packaging category so we can give this a little more time for discussion with the members there?

Moved over to Packaging

dustin · 2021-05-20T16:27:25.998Z

Here’s an example of the repo @oliverchang is proposing to create: GitHub - oliverchang/python-vuln-examples

Here’s a sample of what an advisory would look like: python-vuln-examples/PYSEC-2021-63.yaml at main · oliverchang/python-vuln-examples · GitHub

I’ll be starting the pypa-committers vote to create this as a new repo, pypa/advisory-db on Monday!

Blackward · 2021-05-20T18:07:37.791Z

Hi Dustin,

cool! I in particular like " The goal is to have the pip install (and an additional pip audit ) command automatically report vulnerabilities out of the box." - that would be supercool

Cheers, Dominik

nhatkhai · 2021-05-21T02:02:47.850Z

And flexible for some don’t care about it at some point in time too.

dustin · 2021-05-24T16:28:30.228Z

Per PEP 609 I’ve started the PyPA vote to create https://github.com/pypa/advisory-db (borrowing the same name as https://github.com/RustSec/advisory-db)

ehashman · 2021-05-24T22:43:48.017Z

The advisory repo looks like it’s being kept up to date by osv-robot · GitHub which as far as I can tell looks like a Google-run bot. Is the source for that bot available somewhere? Will a non-Googler be able to continue to maintain this repo if Google chooses to stop sponsoring the work?

alex_Gaynor · 2021-05-24T22:59:45.099Z

python friends. I help maintain the advisory-db for RustSec, and also contributed a smidge to the OSV spec. If there’s any stuff I can share around our experience running a vulnerability DB, please just ask!

dustin · 2021-05-24T23:02:12.795Z

@ehashman Great question. The source for the robot (and the rest of the OSV tooling) is here: https://github.com/google/osv (the robot specifically is here: https://github.com/google/osv/blob/master/docker/worker/worker.py)

The goal of the repo is that while it can be backfilled/bootstrapped by this robot, individuals can also submit advisories directly to it as well.

The project is in its infancy but Google is making an effort to ensure it is not entirely Google-driven, e.g. there is a proposal that helped define the vulnerability format that had many community contributors: https://tinyurl.com/vuln-json

oliverchang · 2021-06-02T03:36:21.201Z

Thanks for the positive feedback everybody!

This is now public at pypa/advisory-db on GitHub (I’m not allowed to post the full link here for some reason) and currently contains 550~ advisories dating back to 2019 (more will be backfilled soon).

dustin · 2021-06-02T12:20:10.673Z

The repo is here: GitHub - pypa/advisory-db: Security advisory database for Python packages published on pypi.org

fridex · 2021-06-14T11:27:39.008Z

If you would like to use the CVE database, Thoth’s resolver can automatically remove packages with vulnerabilities stated in the database - see the demo:

Currently, we are ingesting dependency data for other operating systems than Red Hat Enterprise Linux 8 and UBI 8 (with Python 3.8).

If you would like to consume recommendations for other operating systems, feel free to let us know to eventually prioritize data ingestion based on the user base.

And last but not least, thanks for the database!