write tests

[bluegenes]

• what is the smallest test data we can use?
• download large testdata where needed
• include small intermediate files in tests/test-data wherever possible
• see https://github.com/dib-lab/genome-grist/blob/latest/tests/test_snakemake.py for an example of testing a snakemake workflow. The @pytest.mark.dependency() allows running a specific step, and using the output as input into the next step
• automate with github actions, see https://github.com/dib-lab/genome-grist/blob/latest/.github/workflows/test.yml

[bluegenes]

@taylorreiter

Proposed road map:

1. using your existing repo, add input/output specification capability into the config file
2. Try small or local test data using ^ framework / testing config file
   • merge this version intomain
3. Build testing framework using test data
   • test_snakemake file / pytest stuff
   • data download as needed
   • gh-actions, etc etc
4. Integrate protein changes, which mostly involve:
   • sketch instead of compute
   • alpha-ksize wildcard / folder, to enable add'l alphas
   • add testing for protein
5. Continue with other enhancements/changes...

If you are up for doing 1/2, I can work on 3/4 (or we can work on 3 jointly). Thoughts?

[taylorreiter]

• what is the smallest test data we can use?

So for the variable selection code to work/produce results, the files need to be kind of big. I subsetted to what I think is the smallest set, and it's still 5gb. Spacegraphcats gets around large test data by having the Makefile download large files before executing tests.

The signatures that the majority of the pipeline actually runs on will be small, which is good.

I'm working on uploading this data somewhere (probably osf), and will update when it's uploaded.

using your existing repo, add input/output specification capability into the config file

I don't totally know what you mean by this...can you link to some code that does what you're talking about? I saw you changed the input metadata to read from conf, do you mean do that for other things as well?

[bluegenes]

ok!

I don't totally know what you mean by this...can you link to some code that does what you're talking about? I saw you changed the input metadata to read from conf, do you mean do that for other things as well?

yes! Just would be good to change all inputs to read from a config, so you wouldn't need to worry about input data being in the right location (e.g. inputs/raw).