feat: Use lockfile scalibr interface

[another-rex]

This PR contains all the code required to move to osv-scalibr while making the existing code compile and pass all tests (container tests not passing because of a bug in the scalibr alpine extractor).

Changes not mentioned in the following list will be split off in separate PRs which should land before this PR.

Those are:

• chore: Replace package level ignores with entire directory ignores in our osv-scanner.toml in fixtures #1337
• feat: Migrate pomxml extractor which also performs transitive dependency resolution #1331
• feat: Add osvscanner.json extractor #1338
• feat: Use custom pathtree library instead of third party trie #1341
• feat: migrate node-modules to a osv-scalibr extractor #1345

Changes in this PR:

• Fixture changes:
  • Scalibr Python requirements.txt extractor currently doesn't support packages without versions, so added some version strings to the test files
• Image package required quite a bit of reworking to successfully update.
  • Add the ability to iterate through a directory via the pathtree library
  • Support scalibr FS interface for Layers
  • Add conversion code to convert inventories from osv-scalibr back to v1's lockfile and Inventory
    • This is done to minimize snapshot changes. Followup PRs should remove this conversion
• Add internal/lockfilescalibr package:
  • errors.go adds common extraction errors we want to translate.
  • translation.go adds helper functions and translation logic between osv-scanner v1 extractor names, and osv-scalibr extractor names.

Changes in followup PRs:

• Delete lockfiles package and migrate everything to use osv-scalibr extractors
• Remove conversion code in image

[another-rex]

This PR is ready for review, but the snapshots tests will be updated to the updated version to make the test pass once google/osv-scalibr#241 is merged.

We also currently use a pseudo version of osv-scalibr until a release is made with all the latest changes.

[oliverchang]

why skip scanning if /usr/local/ ?

/usr/local generally has stuff not installed by OS package managers right? it's everything else under /usr that's problematic wrt double scanning?

[another-rex]

I think we found a lot of OS packages get installed here. @hogo6002 IIRC you found a lot of go compiler binaries here?

[hogo6002]

Yes, we found a lot of go compiler binaries there
Update: not usr/local, it was usr/lib

[hogo6002]

Example:
GO-2023-2375 has been found in the actual project binary.

But it also shows 20 times in /usr/lib

[another-rex]

Ah gotcha, I'll update this path. I think I found some npm packages in /usr/local though. I'll double check

[another-rex]

Checking https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

It looks like we should ignore the entire /usr directory. I think everything in there is written by os package managers, since it is suppose to be read only.

[G-Rath]

awesome! I won't pretend that I understand all of it, but it's exciting to see 😄

Scalibr Python requirements.txt extractor currently doesn't support packages without versions, so added some version strings to the test files

that defeats the point of those test fixtures, and makes this a breaking change (oh wait, we're already doing one of those 🙈)

I'm guessing it's not an easy fix, but it'd nice if we could note somewhere that ideally these should be removed once support is added

---

I'm not sure if there's anything else like that - if there is, maybe would it be useful if we created like a "post v2 follow-up" issue where we could just chuck this stuff into so we don't forget, and then we can come back after the majors out to properly split/link things with proper issues? (probably not worth it though if this is the only one)

[G-Rath]

bro 🤜

[cuixq]

I think this bug number should be removed?

[another-rex]

I think this is just reference for us to keep track.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 77.60252% with 71 lines in your changes missing coverage. Please review.

Project coverage is 68.68%. Comparing base (1638434) to head (d1e9eb6).

Files with missing lines	Patch %	Lines
internal/image/layer.go	42.85%	29 Missing and 3 partials ⚠️
internal/lockfilescalibr/translation.go	82.35%	8 Missing and 4 partials ⚠️
internal/image/extractor.go	82.50%	6 Missing and 1 partial ⚠️
pkg/osvscanner/osvscanner.go	90.00%	4 Missing and 3 partials ⚠️
internal/image/scan.go	94.52%	2 Missing and 2 partials ⚠️
...lescalibr/language/osv/osvscannerjson/extractor.go	0.00%	4 Missing ⚠️
...ckfilescalibr/language/java/pomxmlnet/extractor.go	0.00%	3 Missing ⚠️
...alibr/language/javascript/nodemodules/extractor.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               v2    #1330      +/-   ##
==========================================
- Coverage   68.72%   68.68%   -0.05%     
==========================================
  Files         187      188       +1     
  Lines       17884    18075     +191     
==========================================
+ Hits        12290    12414     +124     
- Misses       4917     4977      +60     
- Partials      677      684       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.