Frequent Exceptions when deploying services in the cluster

Frequent exceptions

This page contains information about frequent exceptions and how to fix them

Exception registry configuration file incomplete

This problem most likely stems from the fact that you have not provided a valid registry configuration in your docker container. The information which is required has the following form:

endpoint =
gasPrice =
gasLimit =
communityTagIndexAddress =
userRegistryAddress =
reputationRegistryAddress =
serviceRegistryAddress =

Most namespaces provide a Config Map storing the configuration. You can provide it in your deployment file. The volume should be mounted to the /src/etc/i5.las2peer.registry.data.RegistryConfiguration.properties location.

Example:

# ...
spec:
  volumes:
    - name: registry-config-volume
      configMap:
        name: registry-config
  containers:
    - name: your-service-name
      # ...
      volumeMounts:
        - name: registry-config-volume
          mountPath: >-
            /src/etc/i5.las2peer.registry.data.RegistryConfiguration.properties
          subPath: registry-config.properties
# ...

Alternatively, the main las2peer node which is running as an ethereum node provides this configuration file, which can be accessed trough the default port 8001. So by providing the global env variable LAS2PEER_ETH_HOST hinting at the main las2peer node which is running as an ethereum and inserting this into your docker entrypoint if not already there:

ETH_PROPS_DIR=/app/las2peer/etc/
ETH_PROPS=i5.las2peer.registry.data.RegistryConfiguration.properties

if [ -n "$LAS2PEER_ETH_HOST" ]; then
    echo Replacing Ethereum client host in config files ...
    ETH_HOST_SUB=$(host $LAS2PEER_ETH_HOST)
    sed -i "s|^endpoint.*$|endpoint = http://${LAS2PEER_ETH_HOST}|" "${ETH_PROPS_DIR}${ETH_PROPS}"
    echo done.
fi

The configuration file can be downloaded from the node itself. Pay attention of the ETH_PROPS_DIR as it may vary depending on your Dockerfile.

Exception: i5.las2peer.security.InternalSecurityException: deserializiation problems with decryption

problematic message:
<las2peer:message>
 <sendingNode encoding="base64">...</sendingNode>
 <content encoding="base64">...</content>
 <signature encoding="base64" method="SHA256withRSA">...</signature>
</las2peer:message>
Exception: i5.las2peer.security.InternalSecurityException: deserializiation problems with decryption!
i5.las2peer.security.InternalSecurityException: deserializiation problems with decryption!
 at i5.las2peer.communication.Message.open(Message.java:708)
 at i5.las2peer.communication.Message.open(Message.java:610)
 at i5.las2peer.security.ServiceAgentImpl.receiveMessage(ServiceAgentImpl.java:125)
 at i5.las2peer.p2p.Node.receiveMessage(Node.java:929)
 at i5.las2peer.p2p.pastry.NodeApplication$2.run(NodeApplication.java:485)
 at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: i5.las2peer.serialization.SerializationException: IO problems
 at i5.las2peer.serialization.SerializeTools.deserialize(SerializeTools.java:91)
 at i5.las2peer.serialization.SerializeTools.deserializeBase64(SerializeTools.java:117)
 at i5.las2peer.communication.Message.open(Message.java:701)
 ... 5 more
Caused by: java.io.InvalidClassException: i5.las2peer.api.p2p.ServiceVersion; local class incompatible: stream classdesc serialVersionUID = {{id}}, local class serialVersionUID = 1
 ...

The origin of the issue is still unknown.

To fix it try to update to the latest las2peer core version (right now 1.1.2)

Also, note that the lib folder might contain an old las2peer bundle. Make sure to delete that one also when building the docker container locally

Starting Service failed

This problem mostly occurs when the service had been published before and the node tries to recover the information by fetching the related (so-called) envelopes from another node which is responsible for that part of the DHT. It can occur that the lookup fails, which means that the service also can't start.

Reason

The reason for this error is not yet known. The error might happen if some nodes in the network are not running, which are responsible for the part of the storage space, where the envelope resides. Information about where this envelope lies is stored in a metadata envelope. We are not aware of a way to read the content of this envelope to determine where the missing envelope is stored. So we just delete the metadata envelope, which will result in new envelopes being created on the next service start. It might be also be that some deployments are configured wrong. For example if they don't have a persistent volume claim or the claim is configured wrong. In that case all data which the node stores will be gone if the pod shuts down or is restarted.

Fix: deleting the envelope

The services in the cluster all share a node storage that is residing in a persistent volume per namespace. Thus you can look for the metadata in any container that is running (even on the one which is failing to start).

1. Open a shell in the container of the service that is failing to start. (Make sure that you are in the /src directory. You should be in it by default after opening a shell there.)

2. Run cat log/las2peer.log.0

3. Find the line where the service start fails.

4. Slightly above it will be a line like the following

   FINE i5.las2peer.persistency.SharedStorage: Fetching part (0) of envelope 'agent-<some_id>' with id <ENVELOPE_ID>

5. Copy the ENVELOPE_ID, but remove the first 4 characters (Those are part of the directory structure and not contained in the filename which we are looking for) We call this FILENAME from now on. Save the following command somewhere for later:

   find /src/node-storage/ -name *<FILENAME> -delete

   Don't forget to replace <FILENAME> with your value. This command looks up all files in the node-storage directory that contain FILENAME and delete them.

   Important: the command will delete all files matching FILENAME. So make sure to select the right one. you cant also remove the delete flag to only list all files.

6. Now open a shell in any container that has a node that is connected to the las2peer network. Run the command that you saved in that container. This might take a while depending on the size of the node storage.

Repeat the last step for all namespaces that contain nodes which are connected to your las2peer network.

Check the deployment files

A correct deployment contains the following:

spec:
  ...
  template:
    spec:
      volumes:
        - name: las2peer-nodestorage-persistent
          persistentVolumeClaim:
            claimName: las2peer-nodestorage-pv-claim
        ...    
      containers:
        - name: name-of-your-las2peer-service
          ...
          volumeMounts:
            - name: las2peer-nodestorage-persistent
              mountPath: /src/node-storage
          ...

It might also be that the NODE_ID_SEED environment variable is not set. In this case a random value is selected on each restart which also means that the past storage is lost. This value should be set to something static and unique. We recommend using the same number as the PORT of your service since it will be unique in the network.