Conformance Tests

[puerco]

We should write a couple of conformance tests to ensure the parsers are following the design we want to apply to all of them. I will start dumping some thoughts about what I think should be important to capture in the conformance suite:

Tests Across All Ecosystems:

• Ensure Uniform Package Representation
  We should create test repositories consisting of a simple project with a fixed set of dependencies. Maybe two direct dependencies one of them with a transient one. Once we replicate
• Ensure Uniform License and Copyright Detection
  License data is often found in code comments. We need to make sure all ecosystems are extracting the same data from their own language.
• Ensure Consistent Hashing
  One of the common problems in SBOMs is the wrong hashing of files. We should ensure all ecosystems produce the same hashes when looking at the same file while expressing their own ecosystem hashes correctly.
• Common Errors for Repeatable Failures
  While I'm not a fan of predefining errors, I think it is useful when dealing with plugin-like projects to factor out common errors. Things like emitting errors when the build environment is not ready or complete, execution errors when shelling out, etc are good candidates to unify.