Add Fairy #6647

SantaMcCloud · 2024-12-20T00:50:19Z

FOR CONTRIBUTOR:

I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
License permits unrestricted use (educational + commercial)
This PR adds a new tool or tool collection
This PR updates an existing tool or tool collection
This PR does something else (explain below)

SantaMcCloud · 2024-12-20T00:52:46Z

I know that there will be an error because of the format bcsp. Is it possible to add this format into the galaxy datatype?

I can do it but i have to read and maybe ask question about it. I tried to use the tabular format but with this the tool had always erros possible because the bcsp is a binary file.

SantaMcCloud · 2024-12-20T01:08:25Z

i also get this error when i try to test fairy_cov.xml

(planemo) (fairy_env) sf373@LAPTOP-7RMLPR2D:~/sf373$ planemo test tools-iuc/tools/fairy/fairy_cov.xml 
Traceback (most recent call last):
  File "/home/sf373/sf373/planemo/bin/planemo", line 8, in <module>
    sys.exit(planemo())
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/planemo/cli.py", line 103, in handle_blended_options
    return f(*args, **kwds)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/planemo/commands/cmd_test.py", line 76, in cli
    return_value = test_runnables(ctx, runnables, original_paths=uris, **kwds)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/planemo/engine/test.py", line 13, in test_runnables
    test_data = engine.test(runnables, test_timeout=kwds.get("test_timeout"))
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/planemo/engine/interface.py", line 79, in test
    test_cases = [t for tl in map(cases, runnables) for t in tl]
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/planemo/engine/interface.py", line 79, in <listcomp>
    test_cases = [t for tl in map(cases, runnables) for t in tl]
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/planemo/runnable.py", line 261, in cases
    test_dicts = tool_source.parse_tests_to_dict()
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/galaxy/tool_util/parser/xml.py", line 649, in parse_tests_to_dict
    tests.append(_test_elem_to_dict(test_elem, i, profile))
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/galaxy/tool_util/parser/xml.py", line 693, in _test_elem_to_dict
    outputs=__parse_output_elems(test_elem),
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/galaxy/tool_util/parser/xml.py", line 718, in __parse_output_elems
    name, file, attributes = __parse_output_elem(output_elem)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/galaxy/tool_util/parser/xml.py", line 728, in __parse_output_elem
    file, attributes = __parse_test_attributes(output_elem, attrib, parse_discovered_datasets=True)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/galaxy/tool_util/parser/xml.py", line 781, in __parse_test_attributes
    attributes["object"] = json.loads(attrib.pop("value_json"))
  File "/home/sf373/anaconda3/envs/comebin_env/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/sf373/anaconda3/envs/comebin_env/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/sf373/anaconda3/envs/comebin_env/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I dont know how this error happen but i will have a look at this the next couple of days. Maybe i can fix it somehow

bernt-matthias · 2024-12-20T07:45:31Z

I know that there will be an error because of the format bcsp. Is it possible to add this format into the galaxy datatype?

Yes. Is they documentation available?

bernt-matthias · 2024-12-20T07:50:37Z

I dont know how this error happen but i will have a look at this the next couple of days. Maybe i can fix it somehow

I think value_json="normal_test.tsv" should be value="normal_test.tsv".

SantaMcCloud · 2024-12-20T13:07:26Z

I know that there will be an error because of the format bcsp. Is it possible to add this format into the galaxy datatype?

Yes. Is they documentation available?

Yes and i found the right tab to it. I will read it and try it out!

SantaMcCloud · 2024-12-20T13:07:50Z

I dont know how this error happen but i will have a look at this the next couple of days. Maybe i can fix it somehow

I think value_json="normal_test.tsv" should be value="normal_test.tsv".

yes i didnt saw this typo yesterday thank you for the hint :)

SantaMcCloud · 2024-12-20T15:23:39Z

@bernt-matthias i have a question about adding a datatype. Do i have to add a sniffer or is this an optional function?

SantaMcCloud · 2024-12-20T18:56:18Z

galaxyproject/galaxy#19365

bernt-matthias · 2024-12-22T10:20:19Z

@bernt-matthias i have a question about adding a datatype. Do i have to add a sniffer or is this an optional function?

Optional, I would say.

bernt-matthias

Added a few more comments.

tools/fairy/fairy_cov.xml

bernt-matthias · 2024-12-22T10:30:04Z

tools/fairy/fairy_cov.xml

+        <param name="contig" type="data" format="fasta,fasta.gz" label="Input fasta contig file" help="Input the raw fasta contig file. It can be gzip!"/>
+        <param name="bcsp_file" type="data" format="bcsp" label="Input the pre-sketched file (.bcsp file)" help="This file will be generated with the fairy sketch tool."/>
+        <param argument="--minimum-ani" type="integer" optional="true" min="0" max="100" value="95" label="Set minimum ANI" help="Set the minimum adjusted ANI for the coverage calculation. CARE: only adjust it when you know what it does!"/>
+        <param argument="--min-number-kmers" type="integer" value="8" optional="true" label="Genome filter" help="Set the number for exclude genomes with less than this number of sampled k-mers."/>


Don't undersand this parameter. For the default sequences with less than 8 kmers are removed?

I did rewrote this help, maybe it is better now to understand but yes this paramtere should be a filter to filter out genomes with less k-mers then this paramter

I still do not understand this parameter. What does it mean if a genome has x k-mers?

It means that a genome have less then x (the value of the parameter) k-mers it will execlude from the algorithm. It should not appear in the coverage file generated by Fairy. It this more cleare? And sorry for this missunderstanding

Unfortunately no. I just do not understand what it means that a genome has a kmer. A genome of length n has n-k+1 kmers. There must be something special about the x kmers the help text is talking about.

This x stands for a lower border to filter out genomes with less then x k-mers to not appear in the coverage file.

For an example there are 3 genome in a contig. The first hast 5 k-mers, the second has 10 k-mers and the last one has 8 k-mers. We set this paramter to a 7 which means that the first genome with 5 k-mers will not be included since it hast less then 7 k-mers. When we set this to 2 then there will be no excluding of a genome since all 3 has more then 2 k-mers.

Is this explenation better?

tools/fairy/fairy_sketch.xml

SantaMcCloud · 2024-12-23T00:52:05Z

Will still fail since the datatype is not in galaxy yet

SantaMcCloud · 2025-01-08T21:47:38Z

Hm okay i dont know why this test is still failing here. I did test it localy with cloning the 24.1 branch and use this as root for planemo and this worked fine

bgruening · 2025-01-08T22:09:13Z

The test is fine, the linting fails.

bgruening · 2025-01-08T22:09:39Z

.. WARNING: Repository name [Fairy] invalid. Repository names must contain only lower-case letters, numbers and underscore.

SantaMcCloud · 2025-01-08T22:23:26Z

The test is fine, the linting fails.

oh mb then thought it was the test thank you for looking at it!

bgruening · 2025-01-09T08:01:51Z

tools/fairy/fairy_cov.xml

+        <param argument="--minimum-ani" type="integer" optional="true" min="0" max="100" value="95" label="Set minimum ANI" help="Set the minimum adjusted ANI for the coverage calculation. CARE: only adjust it when you know what it does!"/>
+        <param argument="--min-number-kmers" type="integer" value="8" optional="true" label="Genome filter" help="Set a filter to filter out genomes with less then x k-mer sampled."/>
+        <param argument="-c" type="integer" value="50" optional="true" label="Set subsampling rate" help="This value does not interact with the .bcsp file which was used as input."/>
+        <param argument="-k" type="select" label="Select k-mer size" help="This value does not interact with the .bcsp file which was used as input.">


I do not understand the help text and the above.

the inputed contig file will be sketch aswell from in this part of the tool which also run the sketching algo. This value is one setting of this algo which means that the presketch file, which is the .bcsp file, will not be changed by this value since it was sketch by the other part of the tool befor hand by the same algo.

tools/fairy/fairy_cov.xml

bgruening · 2025-01-09T08:04:32Z

tools/fairy/fairy_cov.xml

+            <option value="21">21</option>
+        </param>
+        <param argument="--min-spacing" type="integer" value="30" label="Set spacing between k-mers" help=" Minimum spacing between selected k-mers on the contigs."/>
+        <param argument="--full-contig-name" type="boolean" falsevalue="" truevalue="--full-contig-name" label="Full contig name"/>


what does this option do?

When a contig has a space in then when this option is not used the tool only take evreything befor this space as name and not using the hole name. When set this option on true evreything will be used. I add a help text to it!

bgruening · 2025-01-09T08:07:28Z

tools/fairy/fairy_sketch.xml

+            #end if
+            &&
+
+            cp './res/${filename}' $output


Can you please add a comment, why the filename is chosen by the program? Its a bit confusing that only file_1 is used.

Suggested change

cp './res/${filename}' $output

cp './res/${filename}' '$output'

I add a comment to it and this is because the tool used the first inputed file to name outpout file. There is an option to change the name but galaxy this option is not needed. To copy the outputfile then i used file_1 plus the new fromat, which is .bcsp to find the output and copy this to the output placeholder.

tools/fairy/fairy_sketch.xml

Co-authored-by: Björn Grüning <[email protected]>

…o fairy

SantaMcCloud added 2 commits December 20, 2024 01:44

Add Fairy

2617364

remove uneeded file

f93520a

fix typo but test will still error since new format

9071a35

fix test. Now evreything is done just wait till datatype is up

6a9629b

SantaMcCloud mentioned this pull request Dec 20, 2024

Add new datatype bcsp galaxyproject/galaxy#19363

Closed

4 tasks

bernt-matthias reviewed Dec 22, 2024

View reviewed changes

add review changes

fecf9b3

Commit for restart tests

c1fe391

Update .shed.yml (fix typo)

6a0f306

bgruening reviewed Jan 9, 2025

View reviewed changes

SantaMcCloud and others added 5 commits January 10, 2025 15:53

Update tools/fairy/fairy_cov.xml

23e2e3f

Co-authored-by: Björn Grüning <[email protected]>

Update tools/fairy/fairy_cov.xml

eab609b

Co-authored-by: Björn Grüning <[email protected]>

Update tools/fairy/fairy_cov.xml

5cd463f

Co-authored-by: Björn Grüning <[email protected]>

reviews changes

3bbd5cf

Merge branch 'fairy' of https://github.com/SantaMcCloud/tools-iuc int…

72923cb

…o fairy

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Fairy #6647

Add Fairy #6647

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024 •

edited

Loading

SantaMcCloud commented Dec 20, 2024 •

edited

Loading

bernt-matthias commented Dec 20, 2024

bernt-matthias commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024 •

edited

Loading

bernt-matthias commented Dec 22, 2024

bernt-matthias left a comment

bernt-matthias Dec 22, 2024

SantaMcCloud Dec 23, 2024

bernt-matthias Dec 23, 2024

SantaMcCloud Dec 23, 2024

bernt-matthias Dec 24, 2024

SantaMcCloud Dec 27, 2024 •

edited

Loading

SantaMcCloud commented Dec 23, 2024

SantaMcCloud commented Jan 8, 2025

bgruening commented Jan 8, 2025

bgruening commented Jan 8, 2025

SantaMcCloud commented Jan 8, 2025

bgruening Jan 9, 2025

SantaMcCloud Jan 10, 2025

bgruening Jan 9, 2025

SantaMcCloud Jan 10, 2025

SantaMcCloud Jan 10, 2025

bgruening Jan 9, 2025

SantaMcCloud Jan 10, 2025

SantaMcCloud Jan 10, 2025

	cp './res/${filename}' $output
	cp './res/${filename}' '$output'

Add Fairy #6647

Are you sure you want to change the base?

Add Fairy #6647

Conversation

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024 • edited Loading

SantaMcCloud commented Dec 20, 2024 • edited Loading

bernt-matthias commented Dec 20, 2024

bernt-matthias commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024 • edited Loading

bernt-matthias commented Dec 22, 2024

bernt-matthias left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SantaMcCloud Dec 27, 2024 • edited Loading

Choose a reason for hiding this comment

SantaMcCloud commented Dec 23, 2024

SantaMcCloud commented Jan 8, 2025

bgruening commented Jan 8, 2025

bgruening commented Jan 8, 2025

SantaMcCloud commented Jan 8, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SantaMcCloud commented Dec 20, 2024 •

edited

Loading

SantaMcCloud commented Dec 20, 2024 •

edited

Loading

SantaMcCloud commented Dec 20, 2024 •

edited

Loading

SantaMcCloud Dec 27, 2024 •

edited

Loading