Integration with Extra Keys

[helicos6]

Since Logitech no longer updates Logitech Gaming Software for current macOS and therefore no more support for G13 Gameboard, there is a 3rd party Mac app called Extra Keys that allows for the G13 to be used on the latest macOS systems. It can be downloaded from the MacAppStore here.

It would be excellent if this Touch Portal plugin integrates with Extra Keys since Logitech Gaming Software is not an option on the current version of macOS.

[mpaperno]

Hi, thanks for posting.

Very interesting, and I didn't know the old LGS was end-of-life on Mac... what a waste. (Full disclosure: My only Mac is a virtual machine I use for testing software, but I do like to support the platform when I can.)

Do you know if Extra Keys supports Lua scripting in profiles, like LGS does/did? Or have any of its own API-type features for integration? (Maybe we can ping @MarkJerde here and see what happens (-: )

Since the profiles are exactly the same format as LGS (from what I can tell?), then LGKeys should be able to parse them as-is. There is a plugin option to tell it the current directory to check for profiles... it could be as simple as configuring that?

To change the profiles path from default, you need to run LGKeys "manually" once from a command line/terminal with a -p option (while TouchPortal is running but with the LGKeys plugin stopped).
For example lgkeys -p ~/Library/Application Support/Logitech (that is the default profile path for LGS, I'd assume the one for EK may be different)

Once you do this, the new profile directory should be stored in the plugin settings and used next time the plugin runs (either manually or via TouchPortal). (There are some more details on this procedure in the Troubleshooting Wiki Page under item 3.)

Another quick way to check would be to try the list_devices script included with this plugin, which also accepts the -p option in the same way. The utility scans the profiles to pick out unique device names. It's not really a full test though.

The more "advanced" features of detecting M-state change or physical button presses didn't work on Mac anyway, so if the EK profiles can be parsed properly, I think that would cover it. If EK can provide extra features I couldn't get our of LGS, that would be even cooler :)

Anyway if you could try it with the current plugin/tools version to see if those EK profiles can be parsed that would be great. Do let me know what you find and if you think anything can be done to improve the experience. I guess for one thing I could update the docs. But if there is some default folder location for EK profiles, it may make sense to make that a default on Mac for example.

Thanks again,
-Max

[helicos6]

Hi Max,

Yes, the profiles used by Extra Keys are the same format as LGS. I’m not sure if Extra Keys supports Lua scripting in profiles, although if LGS did, then I would guess it’s likely Extra Keys does also, but need to confirm with @MarkJerde

The profiles path for Extra Keys is ~/Library/Containers/com.jerdeapps.Extra-Keys/Data/Profiles

I navigated to this directory in Terminal
~/Documents/TouchPortal/plugins/LGKeys/lgkeys

and

I typed lgkeys -p ~/Library/Containers/com.jerdeapps.Extra-Keys/Data/Profiles

But getting this error:

zsh: command not found: lgkeys

So I tried to manually edit the entry.tp file to set the profiles directory but it’s not clear where the path needs to be entered. Can you please help with this?

Thanks,
Praveen

[mpaperno]

Hi Praveen, thanks for following up.

Actually if you're comfortable just editing the entry.tp file, that's the best way to go. I probably should have suggested that in the first place...

You can just add the -p option into the "plugin_start_cmd" value, probably something like this:

"plugin_start_cmd": "%TP_PLUGIN_FOLDER%LGKeys/lgkeys -l %TP_PLUGIN_FOLDER%LGKeys/lgkeys.log -p ~/Library/Containers/com.jerdeapps.Extra-Keys/Data/Profiles --tpstart",

The default profile path is actually set in the plugin's code at the very first start and then saved to TP's settings as a read-only value... after which it can only be changed with that -p switch. Awkward for sure, but workaround for a TP limitation with Settings.

For running from terminal... maybe need to add ./ before lgkeys? ./lgkeys .... I assume it's already marked executable (since TP can run it?) but maybe it doesn't have exec permissions for your user from a terminal?

Anyway there should be useful messages in the plugin's log after it starts. Oh and you can also add a -d flag to the startup command to enable debug-level logging.

Let me know how that goes... Thanks!
-Max

[mpaperno]

PS. You will need to restart Touch Portal after editing the entry.tp file!

[helicos6]

I'm getting this error in the Log:

17:04:23.327 - [ERROR] (Plugin System) Trying to execute the plugin service but it failed: /Users/praveen/Documents/TouchPortal/plugins/LGKeys/lgkeys -l -d /Users/praveen/Documents/TouchPortal/plugins/LGKeys/lgkeys.log -p /Users/praveen/Library/Containers/com.jerdeapps.Extra-Keys/Data/Profiles --tpstart(Cannot run program "/Users/praveen/Documents/TouchPortal/plugins/LGKeys/lgkeys" (in directory "/Users/praveen/Documents/TouchPortal/plugins/LGKeys"): error=13, Permission denied)

Permissions to Documents folder is given, so I'm not sure what's going on.

When I click on Touch Portal in the menu bar, I see "JarRsrcLoader". Maybe I need to give this the folder access permissions?

[mpaperno]

Strange... And not entirely clear from "permissions denied" whether TP is denied permissions to start the plugin, or the plugin program itself lacks permissions to run.

Can I see the output of
ls -l lgkeys
from the plugin's folder?

Maybe the "execute" permissions aren't set...? If so, chmod +x lgkeys may fix the issue. You should be able to run lgkeys from the terminal, that earlier error you were getting could have been indicative of the permissions issue (perhaps).

PS. Touch Portal actually runs as a Java application, so the strange name in "About" probably has something to do with that. I'm afraid though that's pretty much where my familiarity with TP on Mac ends... I haven't tried it in over a year.

[helicos6]

I tried those terminal commands but now getting new errors, so progress? 😂

19:25:39.586 - [LOG] (Plugin Instance) Incoming data: {"type": "settingUpdate", "name": "Profiles Directory", "value": "/Users/praveen/Library/Containers/com.jerdeapps.Extra-Keys/Data/Profiles"}
19:25:39.586 - [ERROR] (Plugin Instance) Exception in thread Thread-1:
19:25:39.587 - [ERROR] (Plugin Instance) Traceback (most recent call last):
19:25:39.587 - [LOG] (Plugin Instance) Incoming data: {"type": "stateUpdate", "id": "us.wdg.max.tpp.lgk.state.autoSwitch", "value": "Enabled"}
19:25:39.587 - [ERROR] (Plugin Instance) File "threading.py", line 954, in _bootstrap_inner
19:25:39.587 - [LOG] (Plugin Instance) Incoming data: {"type": "stateUpdate", "id": "us.wdg.max.tpp.lgk.state.message", "value": "Automatic profile switching Enabled"}
19:25:39.587 - [ERROR] (Plugin Instance) File "modules/profile_watcher.py", line 113, in run
19:25:39.587 - [LOG] (State System) Changed state us.wdg.max.tpp.lgk.state.autoSwitch from to value Enabled
19:25:39.588 - [LOG] (Plugin Instance) Incoming data: {"type": "stateUpdate", "id": "us.wdg.max.tpp.lgk.state.kb.memorySlot", "value": "1"}
19:25:39.587 - [ERROR] (Plugin Instance) File "modules/profile_watcher.py", line 59, in watch_profiles
19:25:39.588 - [LOG] (State System) Changed state us.wdg.max.tpp.lgk.state.message from Default Profile to value Automatic profile switching Enabled
19:25:39.588 - [LOG] (Plugin Instance) Incoming data: {"type": "stateUpdate", "id": "us.wdg.max.tpp.lgk.state.lhc.memorySlot", "value": "1"}
19:25:39.588 - [ERROR] (Plugin Instance) File "modules/profile_watcher.py", line 36, in files_to_timestamp
19:25:39.588 - [LOG] (Plugin Instance) Sending update for settings for plugin with id TouchPortal_Keyboard_Maestro
19:25:39.588 - [LOG] (Plugin Instance) Incoming data: {"type": "stateUpdate", "id": "us.wdg.max.tpp.lgk.state.message", "value": "Connected to LGKeys TouchPortal Plugin v1.0"}
19:25:39.588 - [ERROR] (Plugin Instance) FileNotFoundError: [Errno 2] No such file or directory: '/Users/praveen/Library/Containers/com.jerdeapps.Extra-Keys/Data/Profiles/profiles'
19:25:39.589 - [LOG] (Plugin Instance) Sending update for settings for plugin with id us.wdg.max.tpp.lgk
19:25:39.589 - [LOG] (State System) Changed state us.wdg.max.tpp.lgk.state.message from Automatic profile switching Enabled to value Connected to LGKeys TouchPortal Plugin v1.0
19:25:39.692 - [LOG] (Plugin Instance) Incoming data: {"type": "choiceUpdate", "id": "us.wdg.max.tpp.lgk.act.switchProfile.profileName", "value": []}
19:25:39.693 - [LOG] (State System) Choice list with us.wdg.max.tpp.lgk.act.switchProfile.profileName has changed to value []

[mpaperno]

Oh hey, it runs, yay! 🤣

I could have sworn the x execute bit was set on that file when it was packaged up... I'll have to check but I wonder if something (MacOS?) changed it. Anyway at least that's solved.

The ugly error is because the game profile reader can't find (or get to) the folder you specified. Could they be at some other path? Alternately it's possible the plugin doesn't have permissions to read the directory (although I'd expect the error to be different in that case).

Now that lgkeys runs OK it may be quicker to troubleshoot by running it from the terminal command line as I suggested originally. Otherwise you'll have to be restarting TP for each test...

And if you can run it from the terminal, try running as "root" user to see if it can get to the profiles folder now...
sudo ./lgkeys -p ~/Library/Containers/com.jerdeapps.Extra-Keys/Data/Profiles/profiles -d

This way you will also see the plugins logging output right in the terminal window. CMD-C to exit the plugin.

-Max

[helicos6]

This is the correct path:

'/Users/praveen/Library/Containers/com.jerdeapps.Extra-Keys/Data/Profiles'

For some reason, an extra 'profiles' is added to the end. I tried removing the 'Profiles' in entry.tp, saved, closed and restarted Touch Portal but the changes are not picked up, it still shows the old path:

'/Users/praveen/Library/Containers/com.jerdeapps.Extra-Keys/Data/Profiles/profiles'

I even tried deleting the plugin and reinstalling it, it still keeps the old location. 😲

Same thing in the Terminal.

[helicos6]

CMD-C to exit the plugin.

That didn't work. But strangely CTRL-C did.

[MarkJerde]

Yeah. CTRL-C because the Mac Terminal is a normal UNIX terminal which all use CTRL-C as “break”. 🙂

I’ll be too busy to give a full reply for a couple days probably, but no Lau support in Extra Keys at this point because that would be complicated and the integration Logitech defined for that seems odd to me.

That profile path sounds right. Extra Keys supports both Logitech profile formats (lgp and xml) but will define a new format (JSON, I think, and easy to read because Swift can import and export it directly and I don’t want to create something to write Logitech’s format. 😄

Hope this helps,
Mark

[mpaperno]

Hi Mark, thanks for chiming in.

Sorry about the "off topic" stuff about getting the plugin running... we moved the discussion to Discord

We got it running and it looks good so far parsing the XML files, with no code changes on my side. Which makes sense if they're the same format :)

I'm not familiar with the .lgp format at all.... where did that come from? The LGS I have on my Mac uses the XML format, I'm pretty sure (it's been a while).

GHUB uses a JSON format, though they stuff all the profiles into a Sqlite3 .db which is pretty annoying... the whole DB is one data field with all the profiles serialized as JSON... yes all in one huge blob. And the JSON itself is only mildly annoying to parse... :)

I have a version of this plugin which does parse the GHUB profiles, but couldn't get any real testers and don't have any GHUB devices myself.

Anyway, on Windows we can use a hack with the Lua scripting to detect when a user switches profiles or M slots. Actually have to hook a debugger to the LGS process to do that... silly but it works. W/out that (and on Mac), this plugin has to rely on detecting when the profile XML files are updated with the lastplayeddate timestamp. And there's no way to detect M slot changes at all.

Does Extra Keys do anything now when switching profiles which an external process (like this plugin) could detect? I'm guessing you don't bother with lastplayeddate but perhaps I'm mistaken?

Thanks again,
-Max

[MarkJerde]

Hi Max,
No problem, really. It sounded like I could contribute some Mac Terminal experience, so I was glad to help.

That SQL-wrapper-around-a-single-JSON-blob "format" sounds hilarious. Based on digging into a Logitech app or two, I think they contract out all of their software development, so it's not surprising that each new software they release is different and weird in its own new way. The .lgs format was from G-series Key Profiler, which came before LGS. The format is somewhat similar, but somewhat similar. The G-series Key Profiler editor never really worked for me, but the device handling at least worked. LGS wouldn't even run. It's possible that they all shared a common theme of being too poorly written to run on a case-sensitive filesystem.

Extra Keys doesn't update the lastplayeddate or do anything else externally detectable when switching profiles, but I could easily put another file in the container, such as ~/Library/Containers/com.jerdeapps.Extra-Keys/Data/currentProfile which is rewritten each time the profile changes. How does that sound?

Thanks,
Mark

[mpaperno]

Hi Mark,

Ah yes, the "Key Profiler," how could I forget that one. Guess I never looked at those files back in the day. Also I forgot Macs come with CTRL keys these days... :-)

I could easily put another file in the container, such as ~/Library/Containers/com.jerdeapps.Extra-Keys/Data/currentProfile which is rewritten each time the profile changes.

That would work. Or even just a timestamp update "touch" on the last used profile (which is essentially what changing the "lastplayeddate" also does). Though if we could also detect M slot changes, that would be the "ultimate" solution... maybe currentProfile could include the profile GUID and current M "shift state?" I know... I'm really pushing it now... :)

Thanks,
-Max

[MarkJerde]

Hi Max,
How does this content look?

{
  "description" : "XPlane",
  "file" : "file:\/\/\/Users\/atrus\/Library\/Containers\/com.jerdeapps.Extra-Keys\/Data\/Profiles\/XPlane.lgp",
  "guid" : "BAB6631B-6E2A-49A9-A564-96C8F3297F85",
  "mKey" : 2
}

Located in ~/Library/Containers/com.jerdeapps.Extra-Keys/Data/Profiles/activeProfile.json.

[mpaperno]

Hi Mark,

Looks good to me! Don't need the description or file for my purposes, but no harm either (maybe useful later, who knows).

I'll need to add a "change handler" for EK specifically, but that shouldn't be a big deal. If nothing else I should be able to detect that EK is being used based on the profiles path.

Now the question is if I should make EK the default G keys handler for MacOS. Maybe I can base it on MacOS version... either of you folks know the OS version at which LGS quit working?

Thanks!
-Max

[MarkJerde]

Hi Max,
From my notes, it looks like LGS stopped working on Catalina (10.15). So if your users are on 10.14 or lower they likely are using LGS, and on 10.15 or higher are (hopefully 😄) using Extra Keys. You could also, depending on how things are able to work on your end, possibly base it on the presence of ~/Library/Containers/com.jerdeapps.Extra-Keys. Extra Keys supports down to 10.14, because I started development before updating out of support from Logitech.

Thanks!
Mark

[MarkJerde]

Extra Keys 1.1.2 with the new feature which the LGKeys plugin will use has been released. Thanks for integrating with Extra Keys! https://apps.apple.com/us/app/extra-keys/id1541973430?mt=12

Mark

[mpaperno]

Hi Mark, wow, that was quick, thank you!

Now my turn... and I'm about 3 projects behind right now... plus a couple IRL... LOL. Will get to it ASAP!

Thanks again Mark, a pleasure,
-Max