Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pynml-archive: add -sedml #335

Merged
merged 6 commits into from
Apr 18, 2024
Merged

pynml-archive: add -sedml #335

merged 6 commits into from
Apr 18, 2024

Conversation

sanjayankur31
Copy link
Member

No description provided.

@sanjayankur31
feat(sedml): also get file list from a sedml file
e22259c
@sanjayankur31
feat(archive): implement -sedml
c57755b 
This generates a `sedml` file from the main `lems` file and then creates
an archive as the sedml file as "master".
@sanjayankur31 sanjayankur31 added T: enhancement Type: enhancement S: ready for review Status: ready for review labels Apr 17, 2024
@sanjayankur31 sanjayankur31 requested a review from pgleeson April 17, 2024 14:15
@sanjayankur31
Copy link
Member Author

@pgleeson : do we have a different format string for LEMS? I can't find one. This is what the manifest current looks like:

<?xml version="1.0" encoding="utf-8"?>
<omexManifest xmlns="http://identifiers.org/combine.specifications/omex-manifest">
        <content location="." format="http://identifiers.org/combine.specifications/omex"/>
        <content location="LEMS_NML2_Ex5_DetCell.sedml" master="true" format="http://identifiers.org/combine.specifications/sed-ml"/>
        <content location="LEMS_NML2_Ex5_DetCell.xml"  format="http://identifiers.org/combine.specifications/neuroml"/>
        <content location="NaConductance.channel.nml"  format="http://identifiers.org/combine.specifications/neuroml"/>
        <content location="KConductance.channel.nml"  format="http://identifiers.org/combine.specifications/neuroml"/>
        <content location="LeakConductance.channel.nml"  format="http://identifiers.org/combine.specifications/neuroml"/>
        <content location="NML2_SingleCompHHCell.nml"  format="http://identifiers.org/combine.specifications/neuroml"/>
</omexManifest>

@sanjayankur31 sanjayankur31 linked an issue Apr 17, 2024 that may be closed by this pull request
Add option to generate SEDML file from LEMS and use this as master in exported OMEX archive #329
Open
@pgleeson
Copy link
Member

Not sure... might be best to run an example model on the biosim website and then try to down load the omex file...
What I noticed was it didn't like *neux files, and also seemed to want a metadata file in there too, with rdf. Not sure what goes in that that's not in the manifest...

@sanjayankur31
Copy link
Member Author

I'll try downloading an example to see what they do.

In the meantime, I've tweaked it so that if the master file is sedml, the archive will be *.omex. Will check what the rdf file includes---that isn't implemented at the moment.

@sanjayankur31
Copy link
Member Author

I downloaded this, extracted it, created an archive and re-uploaded it:

https://biosimulations.org/projects/Hodgkin-Huxley-cell-Gleeson

It errored with (formatted):

"`/root/archive.omex` is not a valid COMBINE/OMEX archive.
  - The SED-ML file at location `LEMS_model.sedml` is invalid.
    - The value of a 'sedml:id' must conform to the syntax of the <sbml> data type 'SId'
      Reference: L3V1 Sedml V1 Section
       The id on the <report> is 'results/ex5_v', which does not conform to the syntax.
      
    - The value of a 'sedml:id' must conform to the syntax of the <sbml> data type 'SId'
      Reference: L3V1 Sedml V1 Section
       The id on the <report> is 'results/ex5_vars', which does not conform to the syntax.

So, the id attribute cannot have / in it from the looks of it. We'll need to replace it with _ or something in org.neuroml.export. Full output (formatted):

  "output": "\u001b[0;36mThank you for using runBioSimulations!\u001b[0m

\u001b[0;36m================================================ Loading Singularity ================================================\u001b[0m

\u001b[0;36m=================================================== Set up storage ==================================================\u001b[0m

\u001b[0;36m========================================== Downloading COMBINE/OMEX archive =========================================\u001b[0m
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100   139  100   139    0     0    346      0 --:--:-- --:--:-- --:--:--   345
\r100 23446  100 23446    0     0  43446      0 --:--:-- --:--:-- --:--:-- 43446

\u001b[0;36m=========================================== Executing COMBINE/OMEX archive ==========================================\u001b[0m
INFO:    Using cached SIF image
\u001b[31m`/root/archive.omex` is not a valid COMBINE/OMEX archive.
  - The SED-ML file at location `LEMS_model.sedml` is invalid.
    - The value of a 'sedml:id' must conform to the syntax of the <sbml> data type 'SId'
      Reference: L3V1 Sedml V1 Section
       The id on the <report> is 'results/ex5_v', which does not conform to the syntax.
      
    - The value of a 'sedml:id' must conform to the syntax of the <sbml> data type 'SId'
      Reference: L3V1 Sedml V1 Section
       The id on the <report> is 'results/ex5_vars', which does not conform to the syntax.
      \u001b[0m
srun: error: xanadu-69: task 0: Exited with exit code 1
srun: Terminating job step 7915253.1

\u001b[0;36m===================================================== Saving log ====================================================\u001b[0m
Completed 1.8 KiB/1.8 KiB (11.7 KiB/s) with 1 file(s) remaining\rupload: ./rawLog.txt to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/rawLog.txt

\u001b[0;36m===================================================== Saving Structured log ==========================================\u001b[0m
Completed 750 Bytes/750 Bytes (6.0 KiB/s) with 1 file(s) remaining\rupload: outputs/log.yml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/outputs/log.yml

\u001b[0;36m================================================== Zipping outputs ==================================================\u001b[0m
  adding: outputs/ (stored 0%)
  adding: outputs/log.yml (deflated 52%)
  adding: rawLog.txt (deflated 70%)

\u001b[0;36m=================================================== Saving HDF5 outputs ==================================================\u001b[0m

\u001b[0;36m=================================================== Saving non-HDF5 outputs ==================================================\u001b[0m
Completed 1.6 KiB/4.4 KiB (15.3 KiB/s) with 2 file(s) remaining\rupload: ./output.zip to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/output.zip
Completed 1.6 KiB/4.4 KiB (15.3 KiB/s) with 1 file(s) remaining\rCompleted 4.4 KiB/4.4 KiB (28.2 KiB/s) with 1 file(s) remaining\rupload: ./rawLog.txt to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/rawLog.txt

\u001b[0;36m====================================== Unzip contents of COMBINE/OMEX archive ======================================\u001b[0m
Archive:  archive.omex
 extracting: contents/LEMS_model.sedml  
 extracting: contents/LEMS_model.xml  
 extracting: contents/NaConductance.channel.nml  
 extracting: contents/KConductance.channel.nml  
 extracting: contents/LeakConductance.channel.nml  
 extracting: contents/NML2_SingleCompHHCell.nml  
 extracting: contents/manifest.xml   

\u001b[0;36m====================================== Save contents of COMBINE/OMEX archive ======================================\u001b[0m
Completed 10.4 KiB/22.1 KiB (88.6 KiB/s) with 7 file(s) remaining\rupload: contents/LEMS_model.sedml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/LEMS_model.sedml
Completed 10.4 KiB/22.1 KiB (88.6 KiB/s) with 6 file(s) remaining\rCompleted 12.5 KiB/22.1 KiB (82.1 KiB/s) with 6 file(s) remaining\rupload: contents/KConductance.channel.nml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/KConductance.channel.nml
Completed 12.5 KiB/22.1 KiB (82.1 KiB/s) with 5 file(s) remaining\rCompleted 13.1 KiB/22.1 KiB (84.4 KiB/s) with 5 file(s) remaining\rupload: contents/LeakConductance.channel.nml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/LeakConductance.channel.nml
Completed 13.1 KiB/22.1 KiB (84.4 KiB/s) with 4 file(s) remaining\rCompleted 16.5 KiB/22.1 KiB (103.0 KiB/s) with 4 file(s) remaining\rupload: contents/LEMS_model.xml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/LEMS_model.xml
Completed 16.5 KiB/22.1 KiB (103.0 KiB/s) with 3 file(s) remaining\rCompleted 18.8 KiB/22.1 KiB (115.9 KiB/s) with 3 file(s) remaining\rupload: contents/NML2_SingleCompHHCell.nml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/NML2_SingleCompHHCell.nml
Completed 18.8 KiB/22.1 KiB (115.9 KiB/s) with 2 file(s) remaining\rCompleted 19.7 KiB/22.1 KiB (102.1 KiB/s) with 2 file(s) remaining\rupload: contents/manifest.xml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/manifest.xml
Completed 19.7 KiB/22.1 KiB (102.1 KiB/s) with 1 file(s) remaining\rCompleted 22.1 KiB/22.1 KiB (112.0 KiB/s) with 1 file(s) remaining\rupload: contents/NaConductance.channel.nml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/NaConductance.channel.nml

\u001b[0;36m==================================================== Updating log (2) ===============================================\u001b[0m
Completed 5.7 KiB/5.7 KiB (43.8 KiB/s) with 1 file(s) remaining\rupload: ./rawLog.txt to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/rawLog.txt

\u001b[0;36m================================================== Saving final log =================================================\u001b[0m

\u001b[0;36m=========================================== Post-processing simulation run ==========================================\u001b[0m
\u001b[0;32mProcess the manifest file in the COMBINE archive ... succeeded.
Read the manifest and post the files to the API ... succeeded.
Create thumbnails from the images specified in the manifest ... succeeded.
Post thumbnails to the API ... succeeded.
Process the SED-ML file in the COMBINE archive ... succeeded.
Post the SED-ML file to the API ... succeeded.
Retrieve the logs of the simulation run ... succeeded.
Post the logs to the API ... succeeded.
Process and upload the outputs of the simulation run ... succeeded.\u001b[0m
\u001b[1;33mProcess the metadata in the COMBINE archive ... failed due to: The metadata of the file could not be found:
There was an error in processing the metadata file. More information is available at https://docs.biosimulations.org/concepts/conventions/simulation-project-metadata/. A validation tool is available at https://run.biosimulations.org/utils/validate-metadata.
Post the metadata to the API ... failed due to: Metadata not found due to a dependent step failing.\u001b[0m
\u001b[0;36m================================ Run complete. Thank you for using runBioSimulations! ===============================\u001b[0m",

So it does require some metadata also. Will have to figure out how that works.

@sanjayankur31
Copy link
Member Author

Also verified with pynml -validate-sedml now:

$ pynml -validate-sedm LEMS_model.sedml
line 181: (10302 [Error]) The value of a 'sedml:id' must conform to the syntax of the <sbml> data type 'SId'
Reference: L3V1 Sedml V1 Section
 The id on the <report> is 'results/ex5_v', which does not conform to the syntax.

line 187: (10302 [Error]) The value of a 'sedml:id' must conform to the syntax of the <sbml> data type 'SId'
Reference: L3V1 Sedml V1 Section
 The id on the <report> is 'results/ex5_vars', which does not conform to the syntax.


pyNeuroML >>> ERROR - one or more SEDML files failed to validate

@sanjayankur31 sanjayankur31 mentioned this pull request Apr 17, 2024
Closed
@pgleeson pgleeson merged commit efda4bd into development Apr 18, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pgleeson pgleeson pgleeson approved these changes

Assignees
No one assigned
Labels
S: ready for review Status: ready for review T: enhancement Type: enhancement
Projects
NeuroML backlog
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add option to generate SEDML file from LEMS and use this as master in exported OMEX archive
2 participants
@sanjayankur31 @pgleeson