ConnectionRefusedError when running Marimo in kubernetes deployed Jupyterhub

[ZazenX]

@akshayka @marimo-team, tagging you in this as dicussed on Discord.

I am having issues deploying Marimo to a k3s cluster. I am able to open the notebook from the launcher, and even pick an existing *.py notebook from the workspace explorer.

Deploying and testing locally on minikube works just fine, I can open marimo from the jupyterlab launcher and write and execute notebooks. When deployed on our k3s cluster on our servers though we run into errors like the ones attached. When the marimo interface opens, I get ConnectionRefusedErrors from the singleuser pod.

Opening files from the jupyter-server-proxy seems to refuse connections from the marimo api.

[W 2024-08-08 11:34:56.195 ServerApp] Skipping XSRF check for insecure request GET /user/shaggy/marimo/ws
[I 2024-08-08 11:34:56.196 ServerApp] 101 GET /user/shaggy/marimo/ws?file=gantt.py&session_id=s_wyr78q ([email protected]) 0.87ms
[I 2024-08-08 11:34:56.196 ServerApp] Trying to establish websocket connection to ws://localhost:42191/user/shaggy/marimo/ws?file=gantt.py&session_id=s_wyr78q
[E 2024-08-08 11:34:56.197 ServerApp] Uncaught exception GET /user/shaggy/marimo/ws?file=gantt.py&session_id=s_wyr78q (10.212.134.202)
    HTTPServerRequest(protocol='http', host='jupyterhub.example.com', method='GET', uri='/user/shaggy/marimo/ws?file=gantt.py&session_id=s_wyr78q', version='HTTP/1.1', remote_ip='10.212.134.202')
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.11/site-packages/tornado/websocket.py", line 940, in _accept_connection
        await open_result
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server_proxy/handlers.py", line 904, in open
        return await super().open(path)
               ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server_proxy/handlers.py", line 730, in open
        return await super().open(self.port, path)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server_proxy/handlers.py", line 612, in open
        return await self.proxy_open("localhost", port, proxied_path)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server_proxy/handlers.py", line 540, in proxy_open
        await start_websocket_connection()
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server_proxy/handlers.py", line 520, in start_websocket_connection
        self.ws = await pingable_ws_connect(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2024-08-08 11:34:56.283 ServerApp] Uncaught exception GET /user/shaggy/marimo/api/usage (10.212.134.202)
    HTTPServerRequest(protocol='http', host='jupyterhub.example.com', method='GET', uri='/user/shaggy/marimo/api/usage', version='HTTP/1.1', remote_ip='10.212.134.202')
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.11/site-packages/tornado/web.py", line 1790, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server_proxy/websocket.py", line 101, in get
        return await self.http_get(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server_proxy/handlers.py", line 727, in http_get
        return await ensure_async(self.proxy(self.port, path))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/jupyter_core/utils/__init__.py", line 198, in ensure_async
        result = await obj
                 ^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server_proxy/handlers.py", line 900, in proxy
        return await ensure_async(super().proxy(port, path))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/jupyter_core/utils/__init__.py", line 198, in ensure_async
        result = await obj
                 ^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server_proxy/handlers.py", line 724, in proxy
        return await ensure_async(super().proxy(port, path))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/jupyter_core/utils/__init__.py", line 198, in ensure_async
        result = await obj
                 ^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server_proxy/handlers.py", line 385, in proxy
        response = await client.fetch(req, raise_error=False)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2024-08-08 11:34:56.284 ServerApp] {
      "X-Forwarded-Host": "jupyterhub.example.com",
      "X-Forwarded-Port": "80",
      "Cookie": "jupyterhub-user-shaggy=[secret]; _xsrf=[secret]; session_50541=[secret]; jupyterhub-session-id=[secret]; session_57921=[secret]; session_42191=[secret]",
      "Priority": "u=1, i",
      "Accept-Language": "nb-NO,nb;q=0.9,no;q=0.8,nn;q=0.7,en-US;q=0.6,en;q=0.5",
      "Accept-Encoding": "gzip, deflate, br, zstd",
      "Referer": "https://jupyterhub.example.com/user/shaggy/marimo/?file=gantt.py",
      "Sec-Fetch-Dest": "empty",
      "Sec-Fetch-Mode": "cors",
      "Sec-Fetch-Site": "same-origin",
      "Accept": "*/*",
      "Marimo-Server-Token": "KM4QHuVLKc72hDcw52_Wtw",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
      "Sec-Ch-Ua-Mobile": "?0",
      "Marimo-Session-Id": "s_wyr78q",
      "Sec-Ch-Ua-Platform": "\"Windows\"",
      "Content-Type": "application/json",
      "Sec-Ch-Ua": "\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\"",
      "X-Real-Ip": "10.212.134.202",
      "X-Forwarded-For": "10.212.134.202,::ffff:10.42.0.1",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Scheme": "https",
      "Host": "jupyterhub.example.com"
    }

[dmadisetti]

@ZazenX Can you locally patch this line?

jupyter-marimo-proxy/jupyter_marimo_proxy/__init__.py

Line 20 in 014e721

'command': ['marimo', 'edit', '--port', '{port}', '--base-url', os.environ['JUPYTERHUB_SERVICE_PREFIX'] + 'marimo', '--token', '--token-password', token, '--headless'],

Maybe

'command': ['marimo', 'edit', '--port', '{port}', '--base-url', os.environ['JUPYTERHUB_SERVICE_PREFIX'] + 'marimo', '--token', '--token-password', token, '--headless', '--host', '0.0.0.0'],

To see if that works? Otherwise you can put a proxy on top and forward traffic

[jyio]

Thank you for investigating this issue. I don't have the capability to work on it myself (not familiar with Kubernetes) but would be happy to incorporate a fix.

[ZazenX]

@ZazenX Can you locally patch this line?

jupyter-marimo-proxy/jupyter_marimo_proxy/__init__.py

Line 20 in 014e721

'command': ['marimo', 'edit', '--port', '{port}', '--base-url', os.environ['JUPYTERHUB_SERVICE_PREFIX'] + 'marimo', '--token', '--token-password', token, '--headless'],

Maybe

'command': ['marimo', 'edit', '--port', '{port}', '--base-url', os.environ['JUPYTERHUB_SERVICE_PREFIX'] + 'marimo', '--token', '--token-password', token, '--headless', '--host', '0.0.0.0'],

To see if that works? Otherwise you can put a proxy on top and forward traffic

That seemed to do the trick! Cloned the repo and pip install -e on the singleuser notebook container. My pod logs show all requests to the marimo api work as expected now. I can run code and open files in the jupyter workspace

[benz0li]

@ZazenX Could you also test '--host', '127.0.0.1'?