diff --git a/doc/install_chainer_gpu.rst b/doc/install_chainer_gpu.md similarity index 67% rename from doc/install_chainer_gpu.rst rename to doc/install_chainer_gpu.md index ce5fc43147..cd4debf0b4 100644 --- a/doc/install_chainer_gpu.rst +++ b/doc/install_chainer_gpu.md @@ -15,37 +15,37 @@ Requirements Version Compatibilities for 18.04 --------------------------------- -(Recommended) Use CUDA 9.1 from Official ubuntu repository (https://packages.ubuntu.com/bionic/nvidia-cuda-dev) +(Recommended) Use CUDA 9.1 from Official ubuntu repository [https://launchpad.net/ubuntu/bionic/+package/nvidia-cuda-dev](https://launchpad.net/ubuntu/bionic/+package/nvidia-cuda-dev) - Chainer - - chainer == 6.7.0 (last version supoprting python2. See https://github.com/chainer/chainer/releases/tag/v6.7.0) - - cupy-cuda91 == 6.7.0 (chainer v6.7.0 requires cupy/cudnn for hardware acceleration support https://docs.chainer.org/en/v6.7.0/install.html) + - chainer == 6.7.0 (last version supoprting python2. See [https://github.com/chainer/chainer/releases/tag/v6.7.0](https://github.com/chainer/chainer/releases/tag/v6.7.0) + - cupy-cuda91 == 6.7.0 (chainer v6.7.0 requires cupy/cudnn for hardware acceleration support [https://docs.chainer.org/en/v6.7.0/install.html](https://docs.chainer.org/en/v6.7.0/install.html) - PyTorch - - pytorch == 1.1.0 (Latest pytorch version supporting CUDA 9.1 https://download.pytorch.org/whl/cu90/torch_stable.html) - - CUDA >= 9.0 (Minimum required version for PyTorch 1.1.0 https://pytorch.org/get-started/previous-versions/#v110) + - pytorch == 1.1.0 (Latest pytorch version supporting CUDA 9.1 [https://download.pytorch.org/whl/cu90/torch_stable.html](https://download.pytorch.org/whl/cu90/torch_stable.html) + - CUDA >= 9.0 (Minimum required version for PyTorch 1.1.0 [https://pytorch.org/get-started/previous-versions/#v110](https://pytorch.org/get-started/previous-versions/#v110) -(Experimental) Use CUDA 10.2 from Nvidia Developer's site (https://developer.nvidia.com/cuda-10.2-download-archive) +(Experimental) Use CUDA 10.2 from Nvidia Developer's site [https://developer.nvidia.com/cuda-10.2-download-archive](https://developer.nvidia.com/cuda-10.2-download-archive) - Chainer - - chainer == 6.7.0 (last version supoprting python2. See https://github.com/chainer/chainer/releases/tag/v6.7.0) - - cupy >=6.7.0,<7.0.0 (chainer v6.7.0 requires cupy/cudnn for hardware acceleration support https://docs.chainer.org/en/v6.7.0/install.html) + - chainer == 6.7.0 (last version supoprting python2. See [https://github.com/chainer/chainer/releases/tag/v6.7.0](https://github.com/chainer/chainer/releases/tag/v6.7.0) + - cupy >=6.7.0,<7.0.0 (chainer v6.7.0 requires cupy/cudnn for hardware acceleration support [https://docs.chainer.org/en/v6.7.0/install.html](https://docs.chainer.org/en/v6.7.0/install.html) - cuDNN < 8 (cupy 6.7.0 requires cuDNN v5000= and <=v7999) - - CUDA 10.2 (cuDNN v7.6.5 requires CUDA 10.2 https://developer.nvidia.com/rdp/cudnn-archive) + - CUDA 10.2 (cuDNN v7.6.5 requires CUDA 10.2 [https://developer.nvidia.com/rdp/cudnn-archive](https://developer.nvidia.com/rdp/cudnn-archive)) - PyTorch - pytorch >= 1.4.0 - - CUDA >= 9.2 (Minimum required version for PyTorch https://pytorch.org/get-started/previous-versions/#v140) - - Driver Version >= 396.26 (From CUDA Toolkit and Corresponding Driver Versions in https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) + - CUDA >= 9.2 (Minimum required version for PyTorch [https://pytorch.org/get-started/previous-versions/#v140](https://pytorch.org/get-started/previous-versions/#v140) + - Driver Version >= 396.26 (From CUDA Toolkit and Corresponding Driver Versions in [https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) Install CUDA ------------ -- Ubuntu 14.04 : Download deb file from https://developer.nvidia.com/cuda-downloads?target_os=Linux:: +- Ubuntu 14.04 : Download deb file from [https://developer.nvidia.com/cuda-downloads?target_os=Linux](https://developer.nvidia.com/cuda-downloads?target_os=Linux): ```bash # If you'd like to use CUDA8.0 on Ubuntu 14.04. @@ -56,7 +56,7 @@ Install CUDA sudo apt-get install cuda ``` - - Add below to your `~/.bashrc`:: + - Add below to your `~/.bashrc`: ```bash # setup cuda & cudnn @@ -76,7 +76,7 @@ Install CUDA ``` -- Ubuntu 16.04 : Download deb file from https://developer.nvidia.com/cuda-downloads?target_os=Linux:: +- Ubuntu 16.04 : Download deb file from [https://developer.nvidia.com/cuda-downloads?target_os=Linux](https://developer.nvidia.com/cuda-downloads?target_os=Linux): ```bash # If you'd like to use CUDA9.2 on Ubuntu 16.04. @@ -91,7 +91,7 @@ Install CUDA sudo apt install nvidia-cuda-toolkit sudo apt install nvidia-cuda-dev -- (Experimental) Ubuntu 18.04 : CUDA 10.2 is the latest version which supports `jsk_perception`. Download deb file from https://developer.nvidia.com/cuda-downloads?target_os=Linux:: +- (Experimental) Ubuntu 18.04 : CUDA 10.2 is the latest version which supports `jsk_perception`. Download deb file from https://developer.nvidia.com/cuda-downloads?target_os=Linux: ```bash # If you'd like to use CUDA10.2 on Ubuntu 18.04. @@ -124,11 +124,11 @@ Install CUDA Install CUDNN ------------- -- If you install `pip install cupy-cuda91`, you do not need to install CUDNN manually. (c.f. https://github.com/jsk-ros-pkg/jsk_visualization/issues/809). Thus, default 18.04 user can use CUDA 9.1 and `cupy-cuda91==6.7.0` for `chainer==6.7.0` and you can SKIP this section. +- If you install `pip install cupy-cuda91`, you do not need to install CUDNN manually. (c.f. [https://github.com/jsk-ros-pkg/jsk_visualization/issues/809](https://github.com/jsk-ros-pkg/jsk_visualization/issues/809)). Thus, default 18.04 user can use CUDA 9.1 and `cupy-cuda91==6.7.0` for `chainer==6.7.0` and you can SKIP this section. Installing CUDNN manually only requires for experimental user who install CUDA 10.2 manually. -- You need to login at https://developer.nvidia.com/cudnn +- You need to login at [https://developer.nvidia.com/cudnn](https://developer.nvidia.com/cudnn) - Go to cuDNN Download and choose version - Download deb files of cuDNN Runtime Library and cuDNN Developer Library @@ -156,7 +156,7 @@ Install Cupy - (Default) Chainer 6.7.0 requires CuPy 6.7.0 and if you have CUDA 9.1, you can use CuPy pre-compiled binary package. - - Pre-compiled Install Cupy for CUDA 9.1 :: + - Pre-compiled Install Cupy for CUDA 9.1 : ```bash sudo pip install cupy-cuda91==6.7.0 @@ -164,7 +164,7 @@ Install Cupy - (Experimental) If you have newer CUDA version. You need to install CuPy with source distribution. This requires CUDNN before you run `pip install cupy` . - - Source Install Cupy for CUDA 10.2 :: + - Source Install Cupy for CUDA 10.2 : ```bash sudo pip install -vvv cupy --no-cache-dir @@ -174,7 +174,7 @@ Install Cupy Install PyTorch --------------- -- 18.04 provides CUDA 9.1 by defualt. To install PyTorch compatible with this version, download following wheel from https://download.pytorch.org/whl/cu90/torch_stable.html, and install manually. +- 18.04 provides CUDA 9.1 by defualt. To install PyTorch compatible with this version, download following wheel from [https://download.pytorch.org/whl/cu90/torch_stable.html](https://download.pytorch.org/whl/cu90/torch_stable.html), and install manually. ```bash sudo pip install torch-1.1.0-cp27-cp27mu-linux_x86_64.whl @@ -187,12 +187,12 @@ Install PyTorch sudo pip install torch==1.4.0 ``` -- See https://github.com/jsk-ros-pkg/jsk_recognition/pull/2601#issuecomment-876948260 for more info. +- See [https://github.com/jsk-ros-pkg/jsk_recognition/pull/2601#issuecomment-876948260](https://github.com/jsk-ros-pkg/jsk_recognition/pull/2601#issuecomment-876948260) for more info. Try Chainer Samples ----------- -You can try to run samples to check if the installation succeeded:: +You can try to run samples to check if the installation succeeded: roslaunch jsk_perception sample_fcn_object_segmentation.launch gpu:=0 roslaunch jsk_perception sample_people_pose_estimation_2d.launch GPU:=0 @@ -201,7 +201,7 @@ You can try to run samples to check if the installation succeeded:: Try PyTorch Samples ----------- -You can try to run samples to check if the installation succeeded:: +You can try to run samples to check if the installation succeeded: roslaunch jsk_perception sample_hand_pose_estimation_2d.launch gpu:=0 diff --git a/doc/jsk_perception/nodes/classification_node.md b/doc/jsk_perception/nodes/classification_node.md index 457a566b91..f038526fb1 100644 --- a/doc/jsk_perception/nodes/classification_node.md +++ b/doc/jsk_perception/nodes/classification_node.md @@ -60,7 +60,9 @@ make ## Dynamic Reconfigure Parameters * `~queries` (string, default: `human;kettle;cup;glass`) - Default categories used for subscribing image topic. + Default categories used for subscribing image topic. + + You can send multiple queries with separating semicolon. ### Run inference container on another host or another terminal Now you can use CLIP or ImageBind. diff --git a/doc/jsk_perception/nodes/vqa_node.md b/doc/jsk_perception/nodes/vqa_node.md index b2c8056586..f31f0c0498 100644 --- a/doc/jsk_perception/nodes/vqa_node.md +++ b/doc/jsk_perception/nodes/vqa_node.md @@ -62,7 +62,12 @@ make * `~questions` (string, default: `what does this image describe?`) Default questions used for subscribing image topic. - + + You can send multiple questions with separating semicolon like the below. + ``` + What does this image describe?;What kinds of objects exists? + ``` + ## Sample ### Run inference container on another host or another terminal diff --git a/jsk_perception/launch/classification.launch b/jsk_perception/launch/classification.launch index 548a03986d..3403c87249 100644 --- a/jsk_perception/launch/classification.launch +++ b/jsk_perception/launch/classification.launch @@ -6,6 +6,7 @@ + @@ -16,6 +17,7 @@ host: $(arg host) port: $(arg port) model: $(arg model) + image_transport: $(arg image_transport) diff --git a/jsk_perception/launch/vqa.launch b/jsk_perception/launch/vqa.launch index a1e1ccfc69..32b3988b4b 100644 --- a/jsk_perception/launch/vqa.launch +++ b/jsk_perception/launch/vqa.launch @@ -5,6 +5,7 @@ + @@ -14,6 +15,7 @@ host: $(arg host) port: $(arg port) + image_transport: $(arg image_transport) diff --git a/jsk_perception/src/jsk_perception/vil_inference_client.py b/jsk_perception/src/jsk_perception/vil_inference_client.py index ae2b252cf6..3f04f175b0 100644 --- a/jsk_perception/src/jsk_perception/vil_inference_client.py +++ b/jsk_perception/src/jsk_perception/vil_inference_client.py @@ -39,13 +39,27 @@ def __init__(self, action, # default inference image self.default_img = None # ROS - self.image_sub = rospy.Subscriber("~image", Image, - callback=self.topic_cb, - queue_size=1, - buff_size=2**26) + self.transport_hint = rospy.get_param('~image_transport', 'raw') + if self.transport_hint == 'compressed': + self.image_sub = rospy.Subscriber( + "{}/compressed".format(rospy.resolve_name('~image')), + CompressedImage, + callback=self.topic_cb, + queue_size=1, + buff_size=2**26 + ) + + else: + self.image_sub = rospy.Subscriber("~image", Image, + callback=self.topic_cb, + queue_size=1, + buff_size=2**26) self.result_topic_type = result_topic self.result_pub = rospy.Publisher("~result", result_topic, queue_size=1) - self.image_pub = rospy.Publisher("~result/image", Image, queue_size=1) + if self.transport_hint == 'compressed': + self.image_pub = rospy.Publisher("~result/image/compressed", CompressedImage, queue_size=1) + else: + self.image_pub = rospy.Publisher("~result/image", Image, queue_size=1) self.vis_pub = rospy.Publisher("~visualize", String, queue_size=1) self.action_server = actionlib.SimpleActionServer("~inference_server", action, @@ -156,9 +170,9 @@ def topic_cb(self, data): vis_msg = "" for i, label in enumerate(msg.label_names): vis_msg += "{}: {:.2f}% ".format(label, msg.probabilities[i]*100) - vis_msg += "\n" + vis_msg += "\n\nCosine Similarity\n" for i, label in enumerate(msg.label_names): - vis_msg += "{}: {:.2f}% ".format(label, msg.label_proba[i]*100) + vis_msg += "{}: {:.4f} ".format(label, msg.label_proba[i]) self.vis_pub.publish(vis_msg) def create_queries(self, goal):