How to parse system dependencies

[maelle]

No description provided.

[maelle]

Currently the package does not parse SystemRequirements at all. How should it be done @cboettig ?

Especially since there's no standard on its format.

Or one could create some sort of table from all the possibilities seen on METACRAN 🤔

[cboettig]

yeah, exactly. MetaCRAN is definitely the best shot, I believe Daniel @nuest has implemented some parsing for this at https://github.com/o2r-project/containerit already. Still sounds like it will be dodgy at best, so would need a way to fail gracefully.

[maelle]

We'll need this for rOpenSci registry so I'm tempted to make this a feature for v0.1.6 (otherwise in the registry that field will be a copy of the DESCRIPTION one, fine but not optimal)... will see what I can do.

[maelle]

• I looked at containerit code and saw it used sysreqs that I had managed to star and forget. We'd need it to be on CRAN cf CRAN bound? r-hub/sysreqs#5

• containerit code made me open Should dependencies be listed iteratively #148

[maelle]

That said sysreqs help downloading, maybe not exactly what we're after. 🤔

Maybe what would make sense would be to have pre-prepared codemeta fragments for all possible system requirements listed in https://sysreqs.r-hub.io/list? And then use code similar to the one used in sysreqs to select them? Which doesn't eliminate the need for graceful failure.

[maelle]

Actually sysreqs doesn't only download, I was wrong, but it doesn't seem to work yet, e.g.

sysreqs::sysreqs(file.path(find.package("magick"), "DESCRIPTION"))

fails but I wouldn't expect to.

[maelle]

https://github.com/r-hub/sysreqsdb

[jeroen]

Below an example function to find links to system dependencies based on the r-hub sysreqdb api.

You can either find links using only the package name, or also using the SystemRequirements line from the package DESCRIPTION file, if there is one.

getSysLinks <- function(pkg, description = ""){
  out1 <- jsonlite::fromJSON(sprintf("https://sysreqs.r-hub.io/pkg/%s", pkg), simplifyVector = FALSE)
  out2 <- jsonlite::fromJSON(sprintf("https://sysreqs.r-hub.io/map/%s", curl::curl_escape(description)), simplifyVector = FALSE)
  sysreqs <- unique(c(sapply(out1, names), sapply(out2, names)))
  sprintf("https://sysreqs.r-hub.io/get/%s", sysreqs)
}

## EXAMPLES ##

# Resolve only by package name
getSysLinks("xml2")
getSysLinks("magick")
getSysLinks("curl")

# Resolve by package name or description string
getSysLinks("blabla", description = "This package requires OpenSSL on the system!")
getSysLinks("something", description = "We like GDAL")

# Test with no dependencies
getSysLinks("something", description = "Nothing interesting")

We can use this database as the provider for system dependencies.

[maelle]

Very cool, and now reprexed 😉

getSysLinks <- function(pkg, description = ""){
  out1 <- jsonlite::fromJSON(sprintf("https://sysreqs.r-hub.io/pkg/%s", pkg), simplifyVector = FALSE)
  out2 <- jsonlite::fromJSON(sprintf("https://sysreqs.r-hub.io/map/%s", curl::curl_escape(description)), simplifyVector = FALSE)
  sysreqs <- unique(c(sapply(out1, names), sapply(out2, names)))
  sprintf("https://sysreqs.r-hub.io/get/%s", sysreqs)
}

## EXAMPLES ##

# Resolve only by package name
getSysLinks("xml2")
#> [1] "https://sysreqs.r-hub.io/get/libxml2"
getSysLinks("magick")
#> [1] "https://sysreqs.r-hub.io/get/libmagic"   
#> [2] "https://sysreqs.r-hub.io/get/magick++"   
#> [3] "https://sysreqs.r-hub.io/get/imagemagick"
getSysLinks("curl")
#> [1] "https://sysreqs.r-hub.io/get/libcurl"
#> [2] "https://sysreqs.r-hub.io/get/openssl"

# Resolve by package name or description string
getSysLinks("blabla", description = "This package requires OpenSSL on the system!")
#> [1] "https://sysreqs.r-hub.io/get/openssl"
getSysLinks("something", description = "We like GDAL")
#> [1] "https://sysreqs.r-hub.io/get/gdal"

# Test with no dependencies
getSysLinks("something", description = "Nothing interesting")
#> character(0)

Created on 2018-05-02 by the reprex package (v0.2.0).

[maelle]

@cboettig compared to the info in https://sysreqs.r-hub.io/get/libmagic CodeMeta would need to know provider right? I mean is content missing from sysreqs compared to what we'd need to include in codemeta.json?

[cboettig]

There's no 'required' fields in schema.org, so we don't need a provider. Goal is just to get as detailed a description as possible; though I'd have to think if we have a good way to capture the platform names and disambiguate the platform-specific binary names from the 'generic' sysreq name...

[maelle]

So maybe in the meantime we can "just" use the sysreqs URL?

If provider is sthg nice to have we could somehow suggest and help to add that to sysreqs DB?

[maelle]

Minimal parsing of system dependencies now implemented

magick_cm <- codemetar::create_codemeta("magick")
magick_cm$softwareRequirements
#> [[1]]
#> [[1]]$`@type`
#> [1] "SoftwareApplication"
#> 
#> [[1]]$identifier
#> [1] "Rcpp"
#> 
#> [[1]]$name
#> [1] "Rcpp"
#> 
#> [[1]]$version
#> [1] ">= 0.12.12"
#> 
#> [[1]]$provider
#> [[1]]$provider$`@id`
#> [1] "https://cran.r-project.org"
#> 
#> [[1]]$provider$`@type`
#> [1] "Organization"
#> 
#> [[1]]$provider$name
#> [1] "Comprehensive R Archive Network (CRAN)"
#> 
#> [[1]]$provider$url
#> [1] "https://cran.r-project.org"
#> 
#> 
#> [[1]]$sameAs
#> [1] "https://CRAN.R-project.org/package=Rcpp"
#> 
#> 
#> [[2]]
#> [[2]]$`@type`
#> [1] "SoftwareApplication"
#> 
#> [[2]]$identifier
#> [1] "magrittr"
#> 
#> [[2]]$name
#> [1] "magrittr"
#> 
#> [[2]]$provider
#> [[2]]$provider$`@id`
#> [1] "https://cran.r-project.org"
#> 
#> [[2]]$provider$`@type`
#> [1] "Organization"
#> 
#> [[2]]$provider$name
#> [1] "Comprehensive R Archive Network (CRAN)"
#> 
#> [[2]]$provider$url
#> [1] "https://cran.r-project.org"
#> 
#> 
#> [[2]]$sameAs
#> [1] "https://CRAN.R-project.org/package=magrittr"
#> 
#> 
#> [[3]]
#> [[3]]$`@type`
#> [1] "SoftwareApplication"
#> 
#> [[3]]$identifier
#> [1] "knitr"
#> 
#> [[3]]$name
#> [1] "knitr"
#> 
#> [[3]]$provider
#> [[3]]$provider$`@id`
#> [1] "https://cran.r-project.org"
#> 
#> [[3]]$provider$`@type`
#> [1] "Organization"
#> 
#> [[3]]$provider$name
#> [1] "Comprehensive R Archive Network (CRAN)"
#> 
#> [[3]]$provider$url
#> [1] "https://cran.r-project.org"
#> 
#> 
#> [[3]]$sameAs
#> [1] "https://CRAN.R-project.org/package=knitr"
#> 
#> 
#> [[4]]
#> [[4]]$`@type`
#> [1] "SoftwareApplication"
#> 
#> [[4]]$identifier
#> [1] "curl"
#> 
#> [[4]]$name
#> [1] "curl"
#> 
#> [[4]]$provider
#> [[4]]$provider$`@id`
#> [1] "https://cran.r-project.org"
#> 
#> [[4]]$provider$`@type`
#> [1] "Organization"
#> 
#> [[4]]$provider$name
#> [1] "Comprehensive R Archive Network (CRAN)"
#> 
#> [[4]]$provider$url
#> [1] "https://cran.r-project.org"
#> 
#> 
#> [[4]]$sameAs
#> [1] "https://CRAN.R-project.org/package=curl"
#> 
#> 
#> [[5]]
#> [[5]]$`@type`
#> [1] "SoftwareApplication"
#> 
#> [[5]]$identifier
#> [1] "https://sysreqs.r-hub.io/get/libmagic"
#> 
#> 
#> [[6]]
#> [[6]]$`@type`
#> [1] "SoftwareApplication"
#> 
#> [[6]]$identifier
#> [1] "https://sysreqs.r-hub.io/get/magick++"
#> 
#> 
#> [[7]]
#> [[7]]$`@type`
#> [1] "SoftwareApplication"
#> 
#> [[7]]$identifier
#> [1] "https://sysreqs.r-hub.io/get/imagemagick"

Created on 2018-05-03 by the reprex package (v0.2.0).

@cboettig Now it remains to be decided whether

• we want to parse that URL

• we should ask sysreqs DB maintainers like @jeroen (how) to add information about providers?