Document new usage stats implementation

[bozana]

Document the new usage stats implementation:

• https://github.com/pkp/ojs/blob/main/docs/dev/swagger-source.json
• https://docs.pkp.sfu.ca/dev/documentation/en/statistics-reader:
• Reader statistics: pkp/pkp-lib#8014 Dev documentation for reader statistics  pkp-docs#1029
• Usage Statistics in 3.4 technical documentation (maybe as an extra document, because interested for developers and admins)
• UsageStatsFlowChart.pdf: Improve usage statistics handling in the background/code #6782 (comment)
• Usage Statistics Framework: https://docs.google.com/document/d/13VGLv4aCaEb0yyG3Nw9ur12sGx1cJenlvJay8FMfQM8/edit)
• https://docs.pkp.sfu.ca/learning-ojs/en/statistics:
• Statistics: https://docs.google.com/document/d/1SLJsybCB4XBOO3qnyOXk07m04NVRlAZ0ZxgkcQlUsHc/edit#
• Institutions: https://docs.google.com/document/d/1b_gf4zvf1alwo4qNT0AfCZUk95X9_Ka-otStbkDj7U8/edit
• https://docs.pkp.sfu.ca/learning-ops/en/statistics + institutions (similar/same as the documentation for OJS)
• add/create https://docs.pkp.sfu.ca/learning-omp/en/statistics +institutions (similar/same as the documentation for OJS) ?
• https://docs.pkp.sfu.ca/admin-guide/en/statistics:
• https://docs.google.com/document/d/1D05BEyzM9bvsUzejSSG0xL9v--SQJ5zStaz2qc2-T8o/edit#
• https://github.com/pkp/pkp-docs/tree/main/dev/release-notebooks/en
  Document what has been removed with usage stats improvements in this Github issue Improve usage statistics handling in the background/code #6782, and how someone might replace e.g. calls to the removed methods with the new service classes.

[bozana]

PR swagger docu:
pkp/ojs#3610
pkp/omp#1243

[bozana]

@NateWr, could you please take a look at the PRs?
For OMP and OPS we do not have API documentation, there is only the OMP swagger-source.json file, right? Shell I create the swagger file for OPS?

[NateWr]

I didn't even remember that we had a swagger-source.json file for OMP. It must be really out of date now because I've never kept it updated. I don't think we should keep duplicate copies of this documentation, so I'd recommend we remove it from OMP and don't add it to OPS.

[bozana]

@NateWr, thanks a lot for the review!!! I have considered all your comments, also removed the file from OMP. Would you like to double check it? Else, I would merge, once the tests are passed... Thanks a lot!!!

[NateWr]

I'm happy for you to merge when you're ready. 👍

[bozana]

@NateWr, this is the issue that contains the links to the documentation I've done so far -- for you if you start working on it and I am not there...

[NateWr]

@bozana I've put some time into the documentation for stats. I looked through all of your documentation (it's great!) and incorporated much of it into drafts of three documents:

Draft	Description
Admin Guide > Statistics	Everything a system administrator needs to know to configure stats and reprocess log files
Documentation > Statistics	Everything a contributor or plugin developer needs to know about how stats are logged, processed, and compiled into metrics.
Release Notebook 3.4	A section in the release notebook about what to know for upgrades.

I still need to update the API documentation and include the database descriptions in the DB schema. But the bulk of what you've put together should be in these three drafts. Let me know if I have missed anything important.

I had a couple of questions:

1. Where are institutional stats stored? And how are they accessed? I don't seem to have a metrics table for them, and I can't find an API endpoint to download stats.
2. What exactly does an upgrader need to do before they run an upgrade to 3.4? At one point, it says that stats for the current day should be processed before the upgrade. But I believe the processor only runs up to (but not including) the current day. Can we include step-by-step instructions for someone upgrading? (I'd like to roll this into the how to upgrade guide, as well as the release notebook.)
3. In the dev documentation, there is a brief section on the stats service classes. I'm not very familiar with the new methods. Can you share some simple examples like in the existing code that will illustrate how to use the different methods (like getTotal, getTimeline, etc)?
4. Finally, not related to documentation, but should the IP geographic DB be downloaded if geographic stats are turned off?

Thanks again, all of this documentation was really helpful!

[bozana]

Hi @NateWr, thanks a lot!!!

Institutional stats are stored in metrics_counter_submission_institution_daily and metrics_counter_submission_institution_monthly. Currently they are only used in COUNTER reports accessed/provided via COUNTER SUSHI API. Later we will implement the possibility to download those reports. But, we currently do not plan any other, e.g. internal reporting for institutions.

Before running the upgrade:

• ensure all (except the current) log files are successfully processed
• if needed download reports that are removed in 3.4: PKP Usage statistics report, View Report, Custom Report Generator, and R3 reports in COUNTER plugin

After the upgrade:

• if Geo data was collected earlier, fix the old region codes, i.e. run the following two commands/migration scripts from your OJS, OMP and OPS folder:
  php lib/pkp/migration.php "PKP\migration\upgrade\v3_4_0\I6782_RegionMappings" up
  php lib/pkp/migration.php "PKP\migration\upgrade\v3_4_0\I6782_FixRegionCodes" up

See this PR: pkp/pkp-docs#1029

Hmmm... Currently it is apparently always downloaded...

And there is a mistake regarding the chapter COUNTER R5 SUSHI in the admin guide:
Going to Statistics > Reports > COUNTER Report the user can access only R4.1 reports.
R5 reports are currently only available via SUSHI API in JSON format.

[NateWr]

Thanks @bozana!

ensure all (except the current) log files are successfully processed

Can you explain what this means? What exact CLI commands or point-and-click steps should someone take before they upgrade? Is the "current log file" today's log file? What happens to that file that doesn't happen to other files?

[bozana]

In normal case it means that the scheduled task UsageStatsLoader has been already run that day, so that the log file from the previous day is processed. This can also extra be done by:

• removing that row in the DB table scheduled_tasks: delete from scheduled task where class_name = 'plugins.generic.usageStats.UsageStatsLoader'
• accessing the journal home page (which will then run that scheduled task)

In the case when there were some (older) processing errors, i.e. when the folder usageStats/processing, usageStats/stage or usageStats/reject contain some log files, the problems should be solved, e.g. by:

• taking a look in the scheduledTaskLogs/Usagestatisticsfileloadertask... log (to see what was the problem)
• fixing the problem
• copying the files to the stage folder
• running the scheduled task again.

Yes, with 'current log file' I mean the today's log file.

The upgrade takes care of the today's log file -- the entries in that log file are converted to the new format, so that usage can be continued to be logged in that file.

[NateWr]

Ok, then a couple questions. First, let's assume that the journal is running correctly -- meaning that logs are being processed daily without any errors. In that case:

1. Is this a step that should be done with every upgrade? If so, should we add it to the How to Upgrade document?
2. If this is not something that should be done every time, what would be the extra cost in upgrade time if we converted today's and yesterday's log files during upgrade? If we already have the code to convert one log file, it'd be great if we just converted both so that upgraders didn't have to do anything. Here's what I'm thinking:

• a) Add a pre-flight check that will fail if it finds log files for days other than today or yesterday. The pre-flight check can say "hey it looks like your stats aren't being processed daily, you need to handle this before you upgrade."
• b) During upgrade it converts today's and yesterday's logs to the new format.

This way, 95% of system administrators have to do nothing. And it's only when there's a pre-existing problem of some kind that they need to change things up.

[bozana]

Hi @NateWr, yes that sounds perfect :-) 👍
This is not something that should be done every time.

[NateWr]

@bozana with #8844 completed, I think this is done. I'll close this for now, but let me know if I've missed anything.