AWS Builder Center

Stop Judging a Dependency by Its Stars: Assess Project Health Before You Adopt It

A high star count and a large download number tell you a package is popular, not that it is maintained, governed, or safe to build on. This article argues that adopting an open source dependency should be a structured viability assessment across maintenance, governance, and security practice, and that the automated scores used to do it are risk signals a human still has to interpret — not guarantees.

Series: agentic-ai (4 articles)

  1. …
  2. 3
    Stop Judging a Dependency by Its Stars: Assess Project Health Before You Adopt It This article

  1. Why popularity is the wrong proxy
Stars are cheap to accumulate and easy to inflate, and they measure attention rather than the things that determine whether a dependency will survive contact with production: whether fixes land, whether changes are reviewed, whether more than one organization can keep the project alive, and whether the build can be trusted. A project can be widely used and still concentrated in the hands of a single overloaded maintainer, which is the condition that turns a routine dependency into a continuity and security problem.
The scale of the underlying exposure is not in dispute. A vendor analysis cited by the OpenSSF Scorecard project reports that 84% of codebases contain at least one vulnerability, with an average of 158 per codebase, most of them documented and older than two years [1]. That figure comes from a commercial report rather than an independent census, so treat the exact number as an estimate, but it makes the point that adoption is a risk decision worth doing deliberately.
  1. Use a framework instead of a gut reaction
Two foundation projects give you machine-checkable structure instead of vibes. OpenSSF Scorecard runs automated checks across the supply chain — source code, build, dependencies, testing, and maintenance — where each check returns a score from zero to ten with a risk level that rolls into an aggregate score. At the time of writing it runs 18 checks across three themes, and the public data covers more than a million projects [1]. CHAOSS approaches the same question from the community side, organizing viability into four categories — compliance and security, governance, community, and strategy — each populated by metrics that describe contributor and maintainer sustainability [2]. A peer-reviewed analysis of the CHAOSS system notes on the order of 89 distinct metrics, which is a reminder that the framework is a menu to select from, not a list every team must compute [2].
The OpenSSF Best Practices Badge sits alongside these as a self-certification against published criteria, including concrete requirements such as a documented security contact and vulnerability disclosure path [3]. A badge is evidence the project went through a checklist; it is not an audit.
  1. The case that exposes what popularity misses
The xz-utils incident is the clearest recent demonstration that a popular, apparently active project can carry a planted backdoor. The official CVE record states that malicious code entered the upstream tarballs beginning with version 5.6.0, where an obfuscated build step extracted a prebuilt object that modified liblzma and could intercept authentication in sshd; it was assigned a CVSS base score of 10.0 and affected 5.6.0 and 5.6.1 [4]. What makes it relevant to adoption rather than incident response is how the access was gained. An analysis of the attack, circulated as a preprint and not yet peer-reviewed, describes the "Jia Tan" identity building trust through routine contributions starting in 2021 before implanting the code in 2024 [5]. The contribution history looked healthy. The governance and review thickness underneath it was not.
The framework below maps the dimensions an adoption decision actually rests on.
![Dependency adoption framework: popularity alone is insufficient; assess maintenance, governance, and security and build, then treat scores as risk signals rather than guarantees.]( https://aka.doubaocdn.com/s/qQpQhpNEPj )
Figure by the author (illustrative).
  1. A score is a signal, not a verdict
The temptation is to treat a high aggregate score as a certificate. The empirical evidence does not support that reading. A study of 2,268 npm and PyPI packages built five models relating Scorecard practice scores to externally reported vulnerability counts. It identified four practices — maintained, code review, branch protection, and security policy — as the most important predictors, and the individual features were statistically significant at p < 0.01. But the models explained very little variance, with R-squared between 9% and 12%, and the result that should chasten any automated threshold went the wrong way: packages with higher aggregate security scores had more, not fewer, reported vulnerabilities, a statistically significant positive relationship (p < 0.001, R-squared of 0.04) worth roughly half an additional reported vulnerability per unit of score [6].
The authors explain the inversion through confounding rather than the practices causing harm. Popular packages are attacked and scrutinized more, so more of their vulnerabilities get reported, and externally reported counts miss flaws found internally before release [6]. This is a preprint and not peer-reviewed, but the methodological caution is sound: a score cannot be interpreted without controlling for how widely the package is used, and no current dataset cleanly proves that the practices causally reduce vulnerabilities.
  1. An adoption checklist you can actually run
Dimension
What to check
Why it matters
Maintenance
Recent commits, responsive issue handling, maintained-status check
A project that has stalled cannot absorb fixes
Contributor spread
Contributors from multiple organizations, bus factor
One person leaving should not orphan the dependency
Governance
Branch protection, code review before merge, documented decisions
Catches the thin-review condition behind xz
Security practice
Security policy and contact, SAST, fuzzing, no open OSV findings
Establishes a disclosure path and basic hygiene
Build integrity
Pinned dependencies, least-privilege tokens, signed releases
Protects the path from source to published package
License
Declared, compatible license
Prevents a downstream legal blocker
No single row is decisive, and a small but well-governed library can be a better choice than a famous one with a single maintainer.
  1. Honest limits
These frameworks were largely built around GitHub-hosted, foundation-backed projects, so their checks generalize poorly to ecosystems that lock dependencies elsewhere — the same study notes that the pinned-dependencies check did not inspect the lockfiles npm and PyPI developers actually use, which limits that specific score [6]. Automated tools also cannot judge social-engineering risk directly; they infer it from review and governance thickness, and a determined attacker who plays the long game can clear those signals. For low-traffic or internal dependencies the data may simply be too thin to score, which is itself a reason to prefer a more established alternative or to vendor and maintain the code yourself. What you should not do is convert a weak evidence base into a confident "certified safe" label, or fall back to star count because the rigorous answer is probabilistic.


Next steps: before adding a new dependency, run Scorecard and pull the relevant CHAOSS metrics rather than reading the star count. Work through the table above, and where a dimension cannot be assessed, record that as an explicit risk rather than skipping it. Re-check critical dependencies on a schedule, since maintenance and governance can decay after adoption.

References

  1. OpenSSF Scorecard, project documentation and check catalogue (18 checks, scored 0–10) —  https://scorecard.dev/ 
  1. CHAOSS, "Practitioner Guide: Assessing Viability" (viability across compliance and security, governance, community, and strategy) —  https://chaoss.community/practitioner-guide-viability/ 
  1. OpenSSF Best Practices Badge, criteria and security-contact requirements —  https://www.bestpractices.dev/ 
  1. CVE Record, CVE-2024-3094 (xz/liblzma backdoor, CVSS 10.0) —  https://www.cve.org/CVERecord?id=CVE-2024-3094 
  1. "On the Critical Path to Implant Backdoors and the Effectiveness of Potential Mitigation Techniques: Early Learnings from XZ," arXiv:2404.08987 (preprint, not peer-reviewed) —  https://arxiv.org/html/2404.08987 
  1. "Do Software Security Practices Yield Fewer Vulnerabilities?" arXiv:2210.14884 (preprint, not peer-reviewed; 2,268 packages, R-squared 9–12%, positive score–vulnerability relationship) —  https://arxiv.org/html/2210.14884 

Series: agentic-ai (4 articles)

  1. …
  2. 3
    Stop Judging a Dependency by Its Stars: Assess Project Health Before You Adopt It This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article