Accessing Jupyter Kernels programmatically from API

[aminelajmi]

Hi,

We are developing a hybrid notebook that is able to execute cells of 2 types: Python and custom DSL. For this to work, we'll need to grab/wrap a reference to the default python kernel programmatically, then based on the type/content of cell we'll redirect to the appropriate executor.

After a lot of research on the topic, I've came across many outdated content and broken links, ex:

• https://github.com/microsoft/vscode-jupyter/blob/main/src/client/api/apiAccessService.ts
• https://github.com/microsoft/vscode-jupyter/blob/main/src/client/api/extension.d.ts
• https://github.com/microsoft/vscode-jupyter/wiki/Kernel-Execution
• https://github.com/microsoft/vscode-simple-jupyter-notebook

It seems to me the API has been reworked in a way it doesn't allow to expose kernel connections anymore, and many code examples I find here and there don't help unfortunately.

I see also answers in this discussion stating this is not impossible, but there is no explanation on how to do it.

Finally I found a similar discussion on the topic here stating it is possible to have this feature, provided that we give more info about the usage/context and the product we are building. Additionally, this package has no public releases available for install.

What's the status of this feature on May 2023 ? How can one get access to the Python ipykernel using API and requestExecute ?

Thanks in advance for your help

[DonJayamanne]

This is now supported via the api here

const api = vscode.extensions.get<JupyterAPI>('ms-toolsai.jupyter').exports
const kernelApi = await api.getKernelService();
const jupyterKernel = kernelApi.getKernel(<Notebook URI>).connection.connection;
jupyterKernel.sendControlMessage(....)
jupyterKernel.sendControlMessage(...)

As this is a proposed API, we would like to keep track of th extensions using this (so we can notify them of any changes to teh API).
Hence a message will be displayed if you attempt to use this API.
To get rid of this message we request you file an issue with the details of your extension (extension id) so we can allow it to use this in release mode.

[aminelajmi]

Hi @DonJayamanne, thank you for your answer, and for the good news actually. I will report the issue shortly, cheers.

[nadavy]

@DonJayamanne I think that I need this API. I want to play with that but I keep getting undefined as a value. Is that due to the need to register the extension id?
If so, by To get rid of this message we request you file an issue with the details of your extension (extension id) so we can allow it to use this in release mode, you mean that I need to open an issue in this repo?

[DonJayamanne]

solution using stable api

please use the following stable API for executing of coffee against a kernel

https://github.com/microsoft/vscode-extension-samples/tree/main/jupyter-kernel-execution-sample

[nadavy]

@DonJayamanne thanks for the quick response, much appreciated!
Basically, I see that this allows to query the list of existing kernels and use them. What I'm searching is for a way to automatically detect remote kernels (by server URL), and then configure that without any user interference. This requires editing the notebooks to use the kernel, but I'm not sure what's the best way.

I've thought about querying the api/kernelspec of the remote server and somehow use this info to configure the kernel and connect to. Is that doable?

[DonJayamanne]

Unfortunately no, we do not expose this information.

• Out of curiosity, what do you mean by configure the kernel
• Is your use case to connect to the kernel from outside the context of notebooks?

[nadavy]

My extension provisions a dedicated instance of Jupyter for each developer. As a result, I have a specific server with a single kernel (by default). If a certain notebook isn't configured with a kernel, I want to automatically add this server and select the kernel without asking my user to do so.

I want my user to have as few clicks as possible.

[nadavy]

@DonJayamanne, I hope this answers your question. Feel free to get more info if needed :)
And, thanks for your help. It is much appreciated.

[SebSIK]

My extension provisions a dedicated instance of Jupyter for each developer. As a result, I have a specific server with a single kernel (by default). If a certain notebook isn't configured with a kernel, I want to automatically add this server and select the kernel without asking my user to do so.

I want my user to have as few clicks as possible.

@nadavy Did you ever figure out how to do this? I have the same problem.

[nadavy]

I can’t remember all the details. I think that I edited the files of Jupyter in the disk. There’s a json file that contains all the kernels and then had to reset the kernel. I think that’s inside the folder of the Jupyter plugin. Let’s say that it wasn’t pretty 😅 If that’s important for you, I can search for the code and probably give more info about the solution

…

On Sun, 17 Nov 2024 at 20:49 SebSIK ***@***.***> wrote: My extension provisions a dedicated instance of Jupyter for each developer. As a result, I have a specific server with a single kernel (by default). If a certain notebook isn't configured with a kernel, I want to automatically add this server and select the kernel without asking my user to do so. I want my user to have as few clicks as possible. @nadavy <https://github.com/nadavy> Did you ever figure out how to do this? I have the same problem. — Reply to this email directly, view it on GitHub <#13548 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7MUZR7LC4D24Y7CTUS37T2BDQLDAVCNFSM6AAAAABR6GZDZGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMRYGQ4DCOI> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[SebSIK]

Thanks for the follow up. I looked around and could not find a file like you mentioned. If it's not too difficult let me know the file path.

[nadavy]

//Taken from https://github.com/microsoft/vscode-jupyter/blob/36562246a326ef7fdd93a9faaf4b241ffb496cb7/src/kernels/jupyter/constants.ts#L4
const RemoteKernelSpecCacheFileName = 'remoteKernelSpecCache.json';

So the full path is something like:

const remoteKernelSpecCachePath = path.join(
            context.globalStorageUri.fsPath,   //VsCode context
            '../',
            this.jupyterExtensionId,
            RemoteKernelSpecCacheFileName,
        );

Basically I've patched the cache. I can't put the full code, but something like:

private updateKernelsCache() {
        try {
            const buffer = fs.readFileSync(this.remoteKernelSpecCachePath);
            const kernelsCache = JSON.parse(buffer.toString());
            if (!kernelsCache?.data) {
                return;
            }

            const keys = Object.keys(kernelsCache.data);
            keys.forEach((key) => {
            
                if (key.includes(this.context.extension.id)) {
                    //We found an entry of our extension. Let's edit that with the new url.
                    const entries = kernelsCache.data[key];
                    entries.forEach((entry: unknown) => {
                        if (this.isRemoteKernelEntry(entry)) {
                            this.log(
                                `Updating entry with id ${entry.id}. Old baseUrl: ${entry.baseUrl} to ${this.getUrl().toString()}`,
                            );
                            entry.baseUrl = this.getUrl().toString();
                        }
                    });
                }
            });

            fs.writeFileSync(this.remoteKernelSpecCachePath, JSON.stringify(kernelsCache));
        } catch (e) {
            this.log(`Error updating kernels cache: '${e}'`);
        }
    }

    private isRemoteKernelEntry(o: unknown): o is RemoteKernelEntry {
        return typeof o === 'object' && o !== null && 'id' in o && 'baseUrl' in o;
    }

After you edit the file, you'll have to restart the kernel. In our use case that was ok, but not in every use case.

   ` private async restartKernel() {
    this.log('about to restart kernel');
    await vscode.commands.executeCommand('jupyter.restartkernel');
    this.log('done restarting kernel');
}`

[nadavy]

Note that assumes you have the jupyter extension installed