- Why popularity is the wrong proxy
Stars are cheap to accumulate and easy to inflate, and they measure attention rather than the things that determine whether a dependency will survive contact with production: whether fixes land, whether changes are reviewed, whether more than one organization can keep the project alive, and whether the build can be trusted. A project can be widely used and still concentrated in the hands of a single overloaded maintainer, which is the condition that turns a routine dependency into a continuity and security problem.
The scale of the underlying exposure is not in dispute. A vendor analysis cited by the OpenSSF Scorecard project reports that 84% of codebases contain at least one vulnerability, with an average of 158 per codebase, most of them documented and older than two years [1]. That figure comes from a commercial report rather than an independent census, so treat the exact number as an estimate, but it makes the point that adoption is a risk decision worth doing deliberately.
- Use a framework instead of a gut reaction
Two foundation projects give you machine-checkable structure instead of vibes. OpenSSF Scorecard runs automated checks across the supply chain — source code, build, dependencies, testing, and maintenance — where each check returns a score from zero to ten with a risk level that rolls into an aggregate score. At the time of writing it runs 18 checks across three themes, and the public data covers more than a million projects [1]. CHAOSS approaches the same question from the community side, organizing viability into four categories — compliance and security, governance, community, and strategy — each populated by metrics that describe contributor and maintainer sustainability [2]. A peer-reviewed analysis of the CHAOSS system notes on the order of 89 distinct metrics, which is a reminder that the framework is a menu to select from, not a list every team must compute [2].
The OpenSSF Best Practices Badge sits alongside these as a self-certification against published criteria, including concrete requirements such as a documented security contact and vulnerability disclosure path [3]. A badge is evidence the project went through a checklist; it is not an audit.
- The case that exposes what popularity misses
The xz-utils incident is the clearest recent demonstration that a popular, apparently active project can carry a planted backdoor. The official CVE record states that malicious code entered the upstream tarballs beginning with version 5.6.0, where an obfuscated build step extracted a prebuilt object that modified liblzma and could intercept authentication in sshd; it was assigned a CVSS base score of 10.0 and affected 5.6.0 and 5.6.1 [4]. What makes it relevant to adoption rather than incident response is how the access was gained. An analysis of the attack, circulated as a preprint and not yet peer-reviewed, describes the "Jia Tan" identity building trust through routine contributions starting in 2021 before implanting the code in 2024 [5]. The contribution history looked healthy. The governance and review thickness underneath it was not.
The framework below maps the dimensions an adoption decision actually rests on.
 Figure by the author (illustrative).
- A score is a signal, not a verdict
The temptation is to treat a high aggregate score as a certificate. The empirical evidence does not support that reading. A study of 2,268 npm and PyPI packages built five models relating Scorecard practice scores to externally reported vulnerability counts. It identified four practices — maintained, code review, branch protection, and security policy — as the most important predictors, and the individual features were statistically significant at p < 0.01. But the models explained very little variance, with R-squared between 9% and 12%, and the result that should chasten any automated threshold went the wrong way: packages with higher aggregate security scores had more, not fewer, reported vulnerabilities, a statistically significant positive relationship (p < 0.001, R-squared of 0.04) worth roughly half an additional reported vulnerability per unit of score [6].
The authors explain the inversion through confounding rather than the practices causing harm. Popular packages are attacked and scrutinized more, so more of their vulnerabilities get reported, and externally reported counts miss flaws found internally before release [6]. This is a preprint and not peer-reviewed, but the methodological caution is sound: a score cannot be interpreted without controlling for how widely the package is used, and no current dataset cleanly proves that the practices causally reduce vulnerabilities.
- An adoption checklist you can actually run
Dimension | What to check | Why it matters |
|---|
Maintenance | Recent commits, responsive issue handling, maintained-status check | A project that has stalled cannot absorb fixes |
Contributor spread | Contributors from multiple organizations, bus factor | One person leaving should not orphan the dependency |
Governance | Branch protection, code review before merge, documented decisions | Catches the thin-review condition behind xz |
Security practice | Security policy and contact, SAST, fuzzing, no open OSV findings | Establishes a disclosure path and basic hygiene |
Build integrity | Pinned dependencies, least-privilege tokens, signed releases | Protects the path from source to published package |
License | Declared, compatible license | Prevents a downstream legal blocker |
No single row is decisive, and a small but well-governed library can be a better choice than a famous one with a single maintainer.
These frameworks were largely built around GitHub-hosted, foundation-backed projects, so their checks generalize poorly to ecosystems that lock dependencies elsewhere — the same study notes that the pinned-dependencies check did not inspect the lockfiles npm and PyPI developers actually use, which limits that specific score [6]. Automated tools also cannot judge social-engineering risk directly; they infer it from review and governance thickness, and a determined attacker who plays the long game can clear those signals. For low-traffic or internal dependencies the data may simply be too thin to score, which is itself a reason to prefer a more established alternative or to vendor and maintain the code yourself. What you should not do is convert a weak evidence base into a confident "certified safe" label, or fall back to star count because the rigorous answer is probabilistic.
Next steps: before adding a new dependency, run Scorecard and pull the relevant CHAOSS metrics rather than reading the star count. Work through the table above, and where a dimension cannot be assessed, record that as an explicit risk rather than skipping it. Re-check critical dependencies on a schedule, since maintenance and governance can decay after adoption.
References
- OpenSSF Scorecard, project documentation and check catalogue (18 checks, scored 0–10) — https://scorecard.dev/
- "On the Critical Path to Implant Backdoors and the Effectiveness of Potential Mitigation Techniques: Early Learnings from XZ," arXiv:2404.08987 (preprint, not peer-reviewed) — https://arxiv.org/html/2404.08987
- "Do Software Security Practices Yield Fewer Vulnerabilities?" arXiv:2210.14884 (preprint, not peer-reviewed; 2,268 packages, R-squared 9–12%, positive score–vulnerability relationship) — https://arxiv.org/html/2210.14884
Loading article