Allow absubmit to run high-level extraction locally

[jackwilsdon]

It'd be nice if absubmit could run high-level extraction locally (like AcousticBrainz does when we submit them the low-level data extracted by streaming_extractor_music).

The main goal I'd have for this is the ability to tag tracks without MBIDs, as that currently isn't possible (to upload to AcousticBrainz all tracks must have MBIDs).

Another benefit is that we wouldn't need to round-trip the data up to AcousticBrainz (via absubmit) and then download it again (via acousticbrainz) to tag music with the extracted data, as it could all be done locally instead.

The user would need a copy of the SHM high-level extractor and accompanying models, but these seem readily available online: https://github.com/metabrainz/acousticbrainz-server/blob/b1722e4791b7767a0d9f7e4e2f3d5843c3ad1d9e/Dockerfile#L62-L80

For reference, here's the code that AcousticBrainz uses at the moment to extract the high-level data from the low-level data we currently send them: https://github.com/metabrainz/acousticbrainz-server/blob/24494db50fe45f1610ee47bd6f49769519f80d79/hl_extractor/hl_calc.py

Would this be a welcome feature in the absubmit plugin? It's something we'd obviously leave disabled by default, but it seems like a useful option to have.

[sampsyo]

Ah, neat!! I actually didn't know that the software for high-level feature extraction was readily available; I thought the only expedient way was server-side. Seems like a great idea to add this to enable fully-local feature extraction.

[RollingStar]

For anyone lost: https://beets.readthedocs.io/en/stable/plugins/absubmit.html

[jackwilsdon]

Sounds like this will soon become the only way to get acoustic data: https://blog.metabrainz.org/2022/02/16/acousticbrainz-making-a-hard-decision-to-end-the-project/

[martinkirch]

Hello,
Is there any other project/model ? streaming_extractor_music is a fairly old static binary - it will be harder to run as it gets older, and it's 7 years old already.
I'm much interested in such local analysis, too.

[jackwilsdon]

It looks like BeetsPluginXtractor already implements this high-level extraction.

[JOJ0]

Yes BeetsPluginXtractor does that already. It would be wonderful to see some contribution there. @adamjakab gave me permission a couple of months ago. I can merge stuff. The only drawback is that the Essentia extractors are required to be built manually by the user. There are pre-built binaries by @doctorfree for several Linux derivates though, which he provides since he requires them as a dependency for his own project https://github.com/doctorfree/MusicPlayerPlus

Download a release here to get the pre-built extractors: https://github.com/doctorfree/mpplus-essentia

They are provided for Arch, Debian, Fedora and CentOS distros.

[JOJ0]

It'd be nice if absubmit could run high-level extraction locally (like AcousticBrainz does when we submit them the low-level data extracted by streaming_extractor_music).

As for your original idea in this thread @jackwilsdon: I had that thought a couple of months ago when I tried to get BeetsPluginXtractor running on my Mac. The high-level extractors (SVM models) need to fit with the low-level streaming_extractor_music binary. Unfortunately the Essentia project does not provide the fitting ones. AcousticBrainz used extractor binary 2.1 beta2, which we can still download here: https://essentia.upf.edu/extractors/

But unfortunately the SVM models for the high level extractions are not available in that version.

What we can download though, is SVM models version 2.1 beta5 which we could get here: https://essentia.upf.edu/svm_models/ but as tried to explain above, we are missing an extractor binary for that version.

I have not managed to build the extractor on macOS though, I think I hit a bug back then.... If anyone's interested I can try to find post my progress and details about that here...

I hope I remember all this correctly, it's been some time.... The main message is though: I'm very happy @doctorfree ships a pre-built package, at least for Linux, which I'm very thankful for and use every day.

HTH

[JOJ0]

I've just added a chapter to the https://github.com/adamjakab/BeetsPluginXtractor README explaining how the precompiled mpplus-essentia packages can be used for the beets configuration: https://github.com/adamjakab/BeetsPluginXtractor/blob/master/README.md#precompiled-packages

BTW as of the time of writing, these packages are available for Arch-like, Debian-like and RPMish Linux distros! 👍

[doctorfree]

This looks good. If the mpplus-essentia package is used to install the extractor binary and SVM models then the following paths need to be set in the xtractor section of the Beets configuration (as an example):

xtractor:
    auto: no
    ...
    essentia_extractor: /usr/bin/essentia_streaming_extractor_music
    extractor_profile:
        outputFormat: json
        ...
        highlevel:
            compute: 1
            svm_models:
                - /usr/share/mpplus-essentia/svm_models/danceability.history
                - /usr/share/mpplus-essentia/svm_models/gender.history
#               - /usr/share/mpplus-essentia/svm_models/genre_dortmund.history
#               - /usr/share/mpplus-essentia/svm_models/genre_electronic.history
#               - /usr/share/mpplus-essentia/svm_models/genre_tzanetakis.history
                - /usr/share/mpplus-essentia/svm_models/genre_rosamerica.history
                - /usr/share/mpplus-essentia/svm_models/ismir04_rhythm.history
                - /usr/share/mpplus-essentia/svm_models/mood_acoustic.history
                - /usr/share/mpplus-essentia/svm_models/mood_aggressive.history
                - /usr/share/mpplus-essentia/svm_models/mood_electronic.history
                - /usr/share/mpplus-essentia/svm_models/mood_happy.history
                - /usr/share/mpplus-essentia/svm_models/mood_party.history
                - /usr/share/mpplus-essentia/svm_models/mood_relaxed.history
                - /usr/share/mpplus-essentia/svm_models/mood_sad.history
                - /usr/share/mpplus-essentia/svm_models/moods_mirex.history
                - /usr/share/mpplus-essentia/svm_models/timbre.history
                - /usr/share/mpplus-essentia/svm_models/tonal_atonal.history
                - /usr/share/mpplus-essentia/svm_models/voice_instrumental.history

I've found that bumping the threads value up helps quite a bit on my systems. The sample config sets threads to 1. Determine how many threads your system will support with a command like:

lscpu | egrep 'Model name|Socket|Thread|NUMA|CPU\(s\)'

and bump the threads setting up to improve performance.

This Beets xtractor plugin configuration info may not be suitable for inclusion in the Xtractor README as it is specific to mpplus-essentia but portions may be suitable. Perhaps I need to add something like this to the mpplus-essentia README.

[JOJ0]

Hi @doctorfree thanks a lot for the details, appreciated! I tried to give some hints regarding paths in the new chapter over at the xtractor repo: #4249 (comment)

[JOJ0]

Also @doctorfree I suppose you're using the latest version from PyPI right? I've added a couple of features to the latest git master version of xtractor (eg improving the handling of cpu count - configure 0 for auto-use all available threads). Those are not on on PyPI yet because I don't have access to it.

I would like to kindly ask @adamjakab if he could bump out a release with the latest git master version. I'm certainly offering to write the release notes for it :-) Or if it's desired I would even offer to be added as a collaborator on PyPi as well :-) Either way is fine and I'm happy to help. And sorry for weirdly posting this request not on the xtractor repo itself. I even think it might be fun for you to instantly get some context and see your project is still of use and talked about in a current beets discussion :-))

[doctorfree]

@JOJ0 yes, I am using the latest version from PyPi and I do not foresee deviating from that even though I would like to pickup the more recent changes. I don't want to venture into the "whatever has been committed" territory. So, for my packaging, your enhancements will need to be incorporated into PyPi before they can be used by MusicPlayerPlus. But, good work!

[JOJ0]

That's totally understandable for a packaged software like MusicPlayerPlus. I've just reached out to Adam via email. I'll report back with news here.

In the meantime, this tag would be the newest and stable version of the xtractor plugin: https://github.com/adamjakab/BeetsPluginXtractor/tree/v0.4 and I've also just prepared release notes for that version. I've set it to draft, so they are still hidden though.

[JOJ0]

I've just released a new version of BeetsPluginXtractor. It's available via pip install beets-xtractor.

A couple of fixes and some handy new features. Read the release notes here: https://github.com/adamjakab/BeetsPluginXtractor/releases/tag/v0.4