Indexer: permanently high CPU load

[cbieser]

The indexer still has a constant CPU load of over 300% (docker stats) after more than 24 hours after the last update with a newly built index, activated live index and following manually triggered replication.

MusicBrainz Docker: v-2021-05-19
Docker Compose: 1.21.0
Docker Client/Server: 19.03.15/19.03.15

CONTAINER ID        NAME                               CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
279a677402e2        musicbrainz-docker_musicbrainz_1   0.00%               25.21MiB / 15.64GiB   0.16%               87.9MB / 121MB      599MB / 24.6kB      28
c75b6239808d        musicbrainz-docker_indexer_1       343.69%             4.126GiB / 15.64GiB   26.37%              24.8GB / 425GB      4.23GB / 8.19kB     36
062ff55b5da3        musicbrainz-docker_db_1            0.00%               204.5MiB / 15.64GiB   1.28%               3.51GB / 24.9GB     149GB / 28.4GB      15
b5e64dbc3c4d        musicbrainz-docker_search_1        46.90%              2.731GiB / 15.64GiB   17.46%              809GB / 1.1GB       1.3TB / 423MB       90
1249c229c6a3        musicbrainz-docker_mq_1            0.18%               518.3MiB / 15.64GiB   3.24%               518MB / 203MB       10.3GB / 115kB      77
c582492a20a5        musicbrainz-docker_redis_1         0.08%               2.422MiB / 15.64GiB   0.02%               435kB / 371kB       11.9MB / 1.74MB     3

# admin/check-search-indexes all
CORE           STATUS  INDEX     DB
editor         OK      0         /0
instrument     OK      1007      /1007
series         --      13314     /13331
place          --      47304     /47364
event          --      51236     /51403
tag            --      108350    /108435
area           --      118519    /118522
label          --      202304    /202568
cdstub         --      289486    /289473
annotation     --      458964    /424361
work           --      1505035   /1506421
artist         --      1825377   /1827190
release-group  --      2272704   /2274749
release        --      2885503   /2888409
url            --      7771029   /7780195
recording      --      24834328  /24854241

[cbieser]

I have rebooted the entire server and then started the replication manually. Afterwards, the CPU load remains high as described above.

[yvanzo]

Hi! Did you try allocating more memory to Solr and PostgreSQL? See https://github.com/metabrainz/musicbrainz-docker#modify-memory-settings

[cbieser]

This is my memory-settings.yml

version: '3.1'

# Description: Customize memory settings

services:
  db:
    command: postgres -c "shared_buffers=4096MB" -c "shared_preload_libraries=pg_amqp.so"
    shm_size: "4GB"

  search:
    environment:
      - SOLR_HEAP=4g

[JoshDi]

this should be fixed in one of the last few releases of musicbrainz-docker. Replication packets for a slave server only take a few minutes now to process the index update via SIR