v2.8.1

Release Artifacts

• 🐋 Docker container: tag v2.8.0-dgpu and v2.8.0-igpu
• 🐍 Python wheel: pip install holoscan==2.8.0
• 📦️ Debian packages: 2.8.0.1-1
• 📕 Documentation

See supported platforms for compatibility.

Release Notes

Issues Fixed

• Issue with incorrect url for GXF packaged fixed in Dockerfile

Known Issues

This section details issues discovered during development and QA but unresolved in this release.

Issue	Description
4062979	When Operators connected in a Directed Acyclic Graph (DAG) are executed in a multithreaded scheduler, it is not ensured that their execution order in the graph is adhered.
4267272	AJA drivers cannot be built with RDMA on IGX SW 1.0 DP iGPU due to missing nv-p2p.h. Expected to be addressed in IGX SW 1.0 GA.
4384768	No RDMA support on JetPack 6.0 DP and IGX SW 1.0 DP iGPU due to missing nv-p2p kernel module. Expected to be addressed in JP 6.0 GA and IGX SW 1.0 GA respectively.
4190019	Holoviz segfaults on multi-gpu setup when specifying device using the --gpus flag with docker run. Current workaround is to use CUDA_VISIBLE_DEVICES in the container instead.
4210082	v4l2 applications seg faults at exit or crashes at start with '_PyGILState_NoteThreadState: Couldn't create autoTSSkey maping'
4339399	High CPU usage observed with video_replayer_distributed application. While the high CPU usage associated with the GXF UCX extension has been fixed since v1.0, distributed applications using the MultiThreadScheduler (with the check_recession_period_ms parameter set to 0 by default) may still experience high CPU usage. Setting the HOLOSCAN_CHECK_RECESSION_PERIOD_MS environment variable to a value greater than 0 (e.g. 1.5) can help reduce CPU usage. However, this may result in increased latency for the application until the MultiThreadScheduler switches to an event-based multithreaded scheduler.
4318442	UCX cuda_ipc protocol doesn't work in Docker containers on x86_64. As a workaround, we are currently disabling the UCX cuda_ipc protocol on all platforms via the UCX_TLS environment variable.
4325468	The V4L2VideoCapture operator only supports YUYV and AB24 source pixel formats, and only outputs the RGBA GXF video format. Other source pixel formats compatible with V4L2 can be manually defined by the user, but they're assumed to be equivalent to RGBA8888.
4325585	Applications using MultiThreadScheduler may exit early due to timeouts. This occurs when the stop_on_deadlock_timeout parameter is improperly set to a value equal to or less than check_recession_period_ms, particularly if check_recession_period_ms is greater than zero.
4301203	HDMI IN fails in v4l2_camera on IGX Orin Devkit for some resolution or formats. Try the latest firmware as a partial fix. Driver-level fixes expected in IGX SW 1.0 GA.
4384348	UCX termination (either ctrl+c , press 'Esc' or clicking close button) is not smooth and can show multiple error messages.
4481171	Running the driver for a distributed applications on IGX Orin devkits fails when connected to other systems through eth1. A workaround is to use eth0 port to connect to other systems for distributed workloads.
4458192	In scenarios where distributed applications have both the driver and workers running on the same host, either within a Docker container or directly on the host, there's a possibility of encountering "Address already in use" errors. A potential solution is to assign a different port number to the HOLOSCAN_HEALTH_CHECK_PORT environment variable (default: 8777), for example, by using export HOLOSCAN_HEALTH_CHECK_PORT=8780.
4782662	Installing Holoscan wheel 2.0.0 or later as root causes error.
4768945	Distributed applications crash when the engine file is unavailable/generating engine file.
4753994	Debugging Python application may lead to segfault when expanding an operator variable.
	Wayland: holoscan::viz::Init() with existing GLFW window fails.
4394306	When Python bindings are created for a C++ Operator, it is not always guaranteed that the destructor will be called prior to termination of the Python application. As a workaround to this issue, it is recommended that any resource cleanup should happen in an operator's stop() method rather than in the destructor.
4902749	V4L2 applications segfault at start if using underlying NVV4L2
4909073	V4L2 and AJA applications in x86 container report Wayland XDG_RUNTIME_DIR not set error
4909088	CPP video_replayer_distributed example throws UCX errors and segfaults on close
4911129	HoloHub Endoscopy Tool Tracking application latency exceeds 50ms on Jetson devices

v2.8.0

Release Artifacts

• 🐋 Docker container: tag v2.8.0-dgpu and v2.8.0-igpu
• 🐍 Python wheel: pip install holoscan==2.8.0
• 📦️ Debian packages: 2.8.0.1-1
• 📕 Documentation

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Core

• There is a change to how the Arg class handles const char* type. Both the constructor and assignment operators will now automatically convert a value provided as const char* to a std::string. This change is made to make it easier to pass arguments corresponding to string-valued parameters to Fragment::make_operator and related methods. As an example, the "input_tensor_name" parameer of BayerDemosaicOp could now be specified via Arg("input_tensor_name", "input_frame") without having to explicitly use a std::string or string literal for the second argument to the Arg constructor. Previously, pass such a const char* value would have compiled, but lead to an error in setting the parameter at application run time.

• It is now also possible to add a message-based condition that takes a "receiver" or "transmitter" argument as a positional argument to Fragment::make_operator (C++) or the operator's constructor (Python). Any "receiver" or "transmitter" parameter of the condition should be specified via a string-valued argument that takes the name of the port to which the condition would apply. The SDK will then take care of automatically swapping in the actual underlying Receiver or Transmitter object used by the named port when the application is run.

• The gRPC Health Checking Service is now disabled by default, regardless of whether the --driver or --worker option is used. To enable the service, set the HOLOSCAN_ENABLE_HEALTH_CHECK environment variable.

• The CLI Packager no longer enables the health check service by default. Use the --health-check option to enable it with the CLI Runner.

• New stream-handling APIs for use from an operator's compute method have been added. Note that previously, several of the built-in operators used a provided CudaStreamHandler utility class to implement this type of functionality, but using it required adding this utility class as a data member on the operator and then using lower-level GXF Entity APIs to work with the streams. The CudaStreamHandler also could not be used from native Python operators. The new built-in stream handling functionality is now available from both C++ and Python via public APIs that accept or return the CUDA C Runtime API's standard cudaStream_t type and do not require using any underlying GXF Entity or CudaStream classes. The new methods on InputContext are receive_cuda_stream and receive_cuda_streams. ExecutionContext now provides allocate_cuda_stream, synchronize_streams and device_from_stream methods and OutputContext provides a new set_cuda_stream method. A new user guide section covers this new stream handling capability in additional detail. The prior CudaStreamHandler utility class continues to be provided for backwards compatibility and operators currently using it will continue to interoperate seamlessly with operators using the new built-in stream handling mechanism.

Operators/Resources/Conditions

• The InferenceOp now has a new parameter enable_cuda_graphs which defaults to true. Usage of CUDA Graphs had been unconditionally enabled with version 2.6 for the TensorRT backend. However, models including loops or conditions are not supported with CUDA Graphs. For these models usage of CUDA Graphs needs to be disabled.

• When using the new stream handling APIs, the Operator will be able to find a provided CudaStreamPool resource even if a dedicated parameter has not been added for it. To do this for a C++ operator, just pass an Arg of type std::shared_ptr<CudaStreamPool> to make_operator. For Python, the CudaStreamPool class can be passed as a positional argument to the operator's constructor.

• The C++ API documentation for all provided Condition classes (CountCondition, PeriodicCondition, etc.) now includes detailed descriptions of the available parameters.

• Condition classes that need to specify the "receiver" or "transmitter" object to which they apply were previously not able to be passed as an Arg to make_operator (C++) or as a positional argument to the operator's constructor (Python). This is because the actual receiver or transmitter objects get created automatically behind the scenes by Holoscan and the application author does not have access to them. To resolve this, starting from release v2.8, any Condition class that takes a "receiver" argument can now specify a const char* or std::string (in C++) or str (in Python) with the name of the input port to which the condition should apply. The actual Receiver object created for that port will be automatically used instead by Holoscan during initialization of the condition. Similarly, any "transmitter" argument can be specified via the name of the output port corresponding to the Transmitter the condition will apply to. As a concrete example, this would allow specifying use of a CudaStreamCondition on an input port like in the following example:

In C++, assuming we have defined a class named MyOperator with an input port named "in" we could require stream synchronization for that port using:

auto my_op = make_operator<MyOperator>(
	"my_op",
	make_condition<CudaStreamCondition>("stream_sync", Arg("receiver", "in")),
	// any additional Arg or ArgList here
	);

or from Python using

my_op = MyOperator(
	self,
	CudaStreamCondition(self, receiver="in", name="stream_sync"),
	name="my_op",
	# any additional kwargs here
)

Holoviz module

Holoinfer module

Utils

HoloHub

Documentation

Breaking Changes

Bug fixes

Issue	Description
4903377	The newly introduced RMMAllocator and StreamOrderedAllocator incorrectly parse the Bytes suffix ("B") as Megabytes ("MB") for all parameters related to memory size. Please specify any memory size parameters using the "KB", "MB", "GB" or "TB" suffixes instead.
	A bug in the number of channels returned by FormatConverterOp in the specific case of a RGB VideoBuffer input with RGB output was fixed. The bug resulted in 4 channels instead of 3 being allocated if no resize or channel conversion was being performed. The output should now always have the correct number of channels.

Known Issues

This section details issues discovered during development and QA but unresolved in this release.

Issue	Description
4062979	When Operators connected in a Directed Acyclic Graph (DAG) are executed in a multithreaded scheduler, it is not ensured that their execution order in the graph is adhered.
4267272	AJA drivers cannot be built with RDMA on IGX SW 1.0 DP iGPU due to missing nv-p2p.h. Expected to be addressed in IGX SW 1.0 GA.
4384768	No RDMA support on JetPack 6.0 DP and IGX SW 1.0 DP iGPU due to missing nv-p2p kernel module. Expected to be addressed in JP 6.0 GA and IGX SW 1.0 GA respectively.
4190019	Holoviz segfaults on multi-gpu setup when specifying device using the --gpus flag with docker run. Current workaround is to use CUDA_VISIBLE_DEVICES in the container instead.
4210082	v4l2 applications seg faults at exit or crashes at start with '_PyGILState_NoteThreadState: Couldn't create autoTSSkey maping'
4339399	High CPU usage observed with video_replayer_distributed application. While the high CPU usage associated with the GXF UCX extension has been fixed since v1.0, distributed applications using the MultiThreadScheduler (with the check_recession_period_ms parameter set to 0 by default) may still experience high CPU usage. Setting the HOLOSCAN_CHECK_RECESSION_PERIOD_MS environment variable to a value greater than 0 (e.g. 1.5) can help reduce CPU usage. However, this may result in increased latency for the application until the MultiThreadScheduler switches to an event-based multithreaded scheduler.
4318442	UCX cuda_ipc protocol doesn't work in Docker containers on x86_64. As a workaround, we are currently disabling the UCX cuda_ipc protocol on all platforms via the UCX_TLS environment variable.
4325468	The V4L2VideoCapture operator only supports YUYV and AB24 source pixel formats, and only outputs the RGBA GXF video format. Other source pixel formats compatible with V4L2 can be manually defined by the user, but they're assumed to be equivalent to RGBA8888.
4325585	Applications using MultiThreadScheduler may exit early due to timeouts. This occurs when the stop_on_deadlock_timeout parameter is improperly set to a value equal to or less than check_recession_period_ms, particularly if check_recession_period_ms is greater than zero.
4301203	HDMI IN fails in v4l2_camera on IGX Orin Devkit for some resolution or formats. Try the latest firmware as a partial fix. Driver-level fixes expected in IGX SW 1.0 GA.
4384348	UCX termination (either ctrl+c , press 'Esc' or clicking close button) is not smooth and can show mult...

v2.7.0

Release Artifacts

• 🐋 Docker container: tag v2.7.0-dgpu and v2.7.0-igpu
• 🐍 Python wheel: pip install holoscan==2.7.0
• 📦️ Debian packages: 2.7.0.0-1
• 📕 Documentation

See supported platforms for compatibility.

Release Notes

New Features and Improvements

• Flow Tracking API now has an option to have limited tracking only between root and leaf operators.
  Intermediate operators are not timestamped if this feature is enabled with is_limited_tracking
  parameter to the track() method. This feature is useful when the user wants to track only the
  end-to-end latency of the whole application and does not want fine-grained tracking data of each operator.

• The requirements.txt files for Python are no longer pinned to specific versions of each dependency. Instead a range of versions is allowed to make it easier to install the Python wheels alongside other third-party Python packages.

Core

• Support for adding additional files and dependencies have been added to the Holoscan CLI Packager.

• Flow Tracking in distributed applications are now supported on multiple machines.

• It is now possible to use user-defined thread pools with applications using the EventBasedScheduler or MultiThreadScheduler. Operators can be assigned to thread pools in order to pin operators to a specific thread. This new functionality is described in the user guide section on creating an application and has basic C++ and Python examples.

• Added the HOLOSCAN_REGISTER_GXF_EXTENSIONS option to register GXF extensions to the local Graph Composer
  extension registry at build time (x86 build environment only).

Operators/Resources

• The HolovizOp parameter window_close_scheduling_term has been renamed to window_close_condition. For now, the old parameter name will also continue to work, but please migrate to the new name as the old one will be removed in a future release.

• The HolovizOp parameter window_close_condition is also now available from the Python API. In most cases the user does not need to provide this parameter explicitly, but it can be used if there is a desire to also share the same BooleanCondition
  with other operators.

• There are two new Condition types which can be applied to control operator execution based on message arrival across multiple input ports of an operator. For example, only execute when some specific number of messages has arrived across multiple input ports (or alternatively, execute after a timeout interval even if not all messages have arrived). See the descriptions for MultiMessageAvailableCondition and MultiMessageAvailableTimeoutCondition in the docs section on conditions. Examples of use from both C++ and Python are given in https://github.com/nvidia-holoscan/holoscan-sdk/tree/main/examples/conditions/multi_message.

Holoviz module

• The Holoviz module now has a new API function 'PrimitiveCudaDevice()` which supports geometry data in CUDA memory.

Holoinfer module

• The ONNX runtime backend now
  • supports end-to-end CUDA-based data buffers
  • supports CUDA based inference on ARM
  • is using the TensorRT provider additionally to the CUDA provider
  • supports fp16 inference provided by the TensorRT provider
  • writes engine cache files

Utils

HoloHub

Documentation

Breaking Changes

Bug fixes

Issue	Description
4930333	It's possible that both AppDriver and AppWorker are launching a gRPC server on the same port, causing a conflict without any error message. This is fixed by ensuring that AppDriver and AppWorker use different ports for the gRPC server and display an error message if the port is already in use.
4938973	Holoscan log level settings are not applied to GXF log messages in Python applications. When using Pybind11 to create a Python extension module (e.g., _core.cpython-310-x86_64-linux-gnu.so), it links against the same C++ libraries as the main application (such as libholoscan_core.so). However, without proper configuration, the dynamic linker may load separate copies of these shared libraries. This results in separate instances of global symbols, meaning each loaded library maintains its own set of global variables and functions. Consequently, nvidia::LoggingFunction exists in both the main application and the Python extension module, but at different memory addresses. The solution is to use the RTLD_GLOBAL flag when loading the holoscan.core._core module in Python.
4824619	iGPU: Rendering YUV images with HolovizOp fails on first run
4956212	Image is sometimes corrupted when using Holoviz to display YUV images which are in CUDA device memory

Known Issues

This section details issues discovered during development and QA but unresolved in this release.

Issue	Description
4062979	When Operators connected in a Directed Acyclic Graph (DAG) are executed in a multithreaded scheduler, it is not ensured that their execution order in the graph is adhered.
4267272	AJA drivers cannot be built with RDMA on IGX SW 1.0 DP iGPU due to missing nv-p2p.h. Expected to be addressed in IGX SW 1.0 GA.
4384768	No RDMA support on JetPack 6.0 DP and IGX SW 1.0 DP iGPU due to missing nv-p2p kernel module. Expected to be addressed in JP 6.0 GA and IGX SW 1.0 GA respectively.
4190019	Holoviz segfaults on multi-gpu setup when specifying device using the --gpus flag with docker run. Current workaround is to use CUDA_VISIBLE_DEVICES in the container instead.
4210082	v4l2 applications seg faults at exit or crashes at start with '_PyGILState_NoteThreadState: Couldn't create autoTSSkey maping'
4339399	High CPU usage observed with video_replayer_distributed application. While the high CPU usage associated with the GXF UCX extension has been fixed since v1.0, distributed applications using the MultiThreadScheduler (with the check_recession_period_ms parameter set to 0 by default) may still experience high CPU usage. Setting the HOLOSCAN_CHECK_RECESSION_PERIOD_MS environment variable to a value greater than 0 (e.g. 1.5) can help reduce CPU usage. However, this may result in increased latency for the application until the MultiThreadScheduler switches to an event-based multithreaded scheduler.
4318442	UCX cuda_ipc protocol doesn't work in Docker containers on x86_64. As a workaround, we are currently disabling the UCX cuda_ipc protocol on all platforms via the UCX_TLS environment variable.
4325468	The V4L2VideoCapture operator only supports YUYV and AB24 source pixel formats, and only outputs the RGBA GXF video format. Other source pixel formats compatible with V4L2 can be manually defined by the user, but they're assumed to be equivalent to RGBA8888.
4325585	Applications using MultiThreadScheduler may exit early due to timeouts. This occurs when the stop_on_deadlock_timeout parameter is improperly set to a value equal to or less than check_recession_period_ms, particularly if check_recession_period_ms is greater than zero.
4301203	HDMI IN fails in v4l2_camera on IGX Orin Devkit for some resolution or formats. Try the latest firmware as a partial fix. Driver-level fixes expected in IGX SW 1.0 GA.
4384348	UCX termination (either ctrl+c , press 'Esc' or clicking close button) is not smooth and can show multiple error messages.
4481171	Running the driver for a distributed applications on IGX Orin devkits fails when connected to other systems through eth1. A workaround is to use eth0 port to connect to other systems for distributed workloads.
4458192	In scenarios where distributed applications have both the driver and workers running on the same host, either within a Docker container or directly on the host, there's a possibility of encountering "Address already in use" errors. A potential solution is to assign a different port number to the HOLOSCAN_HEALTH_CHECK_PORT environment variable (default: 8777), for example, by using export HOLOSCAN_HEALTH_CHECK_PORT=8780.
4782662	Installing Holoscan wheel 2.0.0 or later as root causes error.
4768945	Distributed applications crash when the engine file is unavailable/generating engine file.
4753994	Debugging Python application may lead to segfault when expanding an operator variable.
	Wayland: holoscan::viz::Init() with existing GLFW window fails.
4394306	When Python bindings are created for a C++ Operator, it is not always guaranteed that the destructor will be called prior to termination of the Python application. As a workaround to this issue, it is recommended that any resource cleanup should happen in an operator's stop() method rather than in the destructor.
4902749	V4L2 applications segfault at start if using underlying NVV4L2
4909073	V4L2 and AJA applications in x86 container report Wayland XDG_RUNTIME_DIR not set error
4909088	CPP video_replayer_distributed example throws UCX errors and segfaults on close
4911129	HoloHub Endoscopy Tool Tracking application latency exceeds 50ms on Jetson devices
4903377	The newly introduced RMMAllocator and StreamOrderedAllocator incorrectly parse the Bytes suffix ("B") as Megabytes ("MB") for all parameters re...

v2.6.0

Release Artifacts

• 🐋 Docker container: tag v2.6.0-dgpu and v2.6.0-igpu
• 🐍 Python wheel: pip install holoscan==2.6.0
• 📦️ Debian packages: 2.6.0.1-1
• 📕 Documentation

See supported platforms for compatibility.

Release Notes

New Features and Improvements

• Support for Python 3.12 has been added and support for Python 3.8 was dropped. Python wheels are now provided for Python 3.9, 3.10, 3.11 and 3.12.

• Data Flow Tracking is available for multi-fragment distributed applications. It is currently
  supported on a single machine. Later releases will support multiple machine scenarios.

Core

• Dependencies are updated to align with the IGX OS 1.1 compute stack and DLFW 24.08 containers.
  Refer to the project Dockerfile and to DEVELOP.md documentation for a complete list of
  package versions.
• Distributed application fragments not using GPU data can now run on nodes that do not have a GPU. Any CPU-only nodes used by the application must have CUDA installed.
• Distributed applications can be run in a legacy, synchronous data transmit mode by setting HOLOSCAN_UCX_ASYNCHRONOUS=False (this will match the distributed app behavior from Holoscan releases v0.6 and v1.x). The synchronous mode blocks on sending a message from an output port until the send is complete. The default, asynchronous mode that has been the default since 2.0 may show a performance benefit as it does non-blocking sends, but this can make it harder to predict the required minimum memory buffer space when using an allocator such as BlockMemoryPool or RMMAllocator as the transmitter may call compute again before the prior message has been received. The user can now control which mode is preferred.

Operators/Resources

• The GPU performance of the Holoinfer module and the Inference operator had been improved by

  • eliminating CPU to GPU synchronizations
  • avoiding copies from CPU to GPU buffers and back
  • using CUDA events to synchronize between CUDA streams used by input and output data transfer and also preparing data for multi-GPU usage
  • using CUDA graphs in the TensoRT backend
  • providing a GPU accelerated version of the max_per_channel_scaled post processing operation

• A new RMMAllocator class that uses RMM (Rapids Memory Manager) to support simultaneous host and device memory pools. This is an improvement over the existing BlockMemoryPool allocator which could only have a memory pool on a single device. RMMAllocator can be used with operators such as VideoStreamReplayerOp that need to allocate both host and device memory buffers.

• A new StreamOrderedAllocator has been added that supports asynchronous memory allocation from a device memory pool associated with a given CUDA stream. This is based on CUDA's stream-ordered allocator APIs (cudaMallocFromPoolAsync, cudaFreeAsync). This allocator is not yet used in Holoscan v2.6, but is part of enabling improved CUDA stream handling support for Holoscan in the near future.

Holoviz module

Utils

HoloHub

Documentation

Breaking Changes

• Support for Python 3.8 has been dropped. To continue using Python 3.8, please use Holoscan SDK <= 2.5.
• Holoscan SDK 2.6 requires an upgraded underlying compute stack with CUDA Toolkit 12.6 and TensorRT 10.3. We recommend that customers using Holoscan SDK on the IGX 1.0 platform install Holoscan SDK 2.6 via the NGC container. Customers using Holoscan SDK via Debian or Python wheel packages on IGX 1.0 should use Holoscan <=SDK 2.5.

Bug fixes

Issue	Description
4792457	Fixed various memory issues, including memory leaks and memory corruption. An issue with the validation of IPv6 addresses when resolving IP addresses (hostname) from the CLI has been fixed. An issue with heap-use-after-free when traversing the graph has been fixed.

Known Issues

This section supplies details about issues discovered during development and QA but not resolved in this release.

Issue	Description
4062979	When Operators connected in a Directed Acyclic Graph (DAG) are executed in a multithreaded scheduler, it is not ensured that their execution order in the graph is adhered.
4267272	AJA drivers cannot be built with RDMA on IGX SW 1.0 DP iGPU due to missing nv-p2p.h. Expected to be addressed in IGX SW 1.0 GA.
4384768	No RDMA support on JetPack 6.0 DP and IGX SW 1.0 DP iGPU due to missing nv-p2p kernel module. Expected to be addressed in JP 6.0 GA and IGX SW 1.0 GA respectively.
4190019	Holoviz segfaults on multi-gpu setup when specifying device using the --gpus flag with docker run. Current workaround is to use CUDA_VISIBLE_DEVICES in the container instead.
4210082	v4l2 applications seg faults at exit or crashes at start with '_PyGILState_NoteThreadState: Couldn't create autoTSSkey maping'
4339399	High CPU usage observed with video_replayer_distributed application. While the high CPU usage associated with the GXF UCX extension has been fixed since v1.0, distributed applications using the MultiThreadScheduler (with the check_recession_period_ms parameter set to 0 by default) may still experience high CPU usage. Setting the HOLOSCAN_CHECK_RECESSION_PERIOD_MS environment variable to a value greater than 0 (e.g. 1.5) can help reduce CPU usage. However, this may result in increased latency for the application until the MultiThreadScheduler switches to an event-based multithreaded scheduler.
4318442	UCX cuda_ipc protocol doesn't work in Docker containers on x86_64. As a workaround, we are currently disabling the UCX cuda_ipc protocol on all platforms via the UCX_TLS environment variable.
4325468	The V4L2VideoCapture operator only supports YUYV and AB24 source pixel formats, and only outputs the RGBA GXF video format. Other source pixel formats compatible with V4L2 can be manually defined by the user, but they're assumed to be equivalent to RGBA8888.
4325585	Applications using MultiThreadScheduler may exit early due to timeouts. This occurs when the stop_on_deadlock_timeout parameter is improperly set to a value equal to or less than check_recession_period_ms, particularly if check_recession_period_ms is greater than zero.
4301203	HDMI IN fails in v4l2_camera on IGX Orin Devkit for some resolution or formats. Try the latest firmware as a partial fix. Driver-level fixes expected in IGX SW 1.0 GA.
4384348	UCX termination (either ctrl+c , press 'Esc' or clicking close button) is not smooth and can show multiple error messages.
4481171	Running the driver for a distributed applications on IGX Orin devkits fails when connected to other systems through eth1. A workaround is to use eth0 port to connect to other systems for distributed workloads.
4458192	In scenarios where distributed applications have both the driver and workers running on the same host, either within a Docker container or directly on the host, there's a possibility of encountering "Address already in use" errors. A potential solution is to assign a different port number to the HOLOSCAN_HEALTH_CHECK_PORT environment variable (default: 8777), for example, by using export HOLOSCAN_HEALTH_CHECK_PORT=8780.
4782662	Installing Holoscan wheel 2.0.0 or later as root causes error.
4768945	Distributed applications crash when the engine file is unavailable/generating engine file.
4753994	Debugging Python application may lead to segfault when expanding an operator variable.
	Wayland: holoscan::viz::Init() with existing GLFW window fails.
4394306	When Python bindings are created for a C++ Operator, it is not always guaranteed that the destructor will be called prior to termination of the Python application. As a workaround to this issue, it is recommended that any resource cleanup should happen in an operator's stop() method rather than in the destructor.
4824619	iGPU: Rendering YUV images with HolovizOp fails on first run
4902749	V4L2 applications segfault at start if using underlying NVV4L2
4909073	V4L2 and AJA applications in x86 container report Wayland XDG_RUNTIME_DIR not set error
4909088	CPP video_replayer_distributed example throws UCX errors and segfaults on close
4911129	HoloHub Endoscopy Tool Tracking application latency exceeds 50ms on Jetson devices
4903377	The newly introduced RMMAllocator and StreamOrderedAllocator incorrectly parse the Bytes suffix ("B") as Megabytes ("MB") for all parameters related to memory size. Please specify any memory size parameters using the "KB", "MB", "GB" or "TB" suffixes instead.

v2.5.0

Release Artifacts

• 🐋 Docker container: tag v2.5.0-dgpu and v2.5.0-igpu
• 🐍 Python wheel: pip install holoscan==2.5.0
• 📦️ Debian packages: 2.5.0.1-1
• 📕 Documentation

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Core

• Updates to Python decorator API to simplify syntax for conversion of functions to Holoscan operators. Now, a function returning a tuple of arrays can be easily decorated to emit these arrays from separate output ports.
• Added support for application profiling using NSight Systems and NVTX using the HOLOSCAN_ENABLE_PROFILE environment variable.

Operators/Resources

• The Holoviz operator now has a new parameter display_color_space. This allows HDR output on Linux distributions and displays supporting that feature.
• A bug in the Holoviz module that prevented using multiple HolovizOp operators with the event-based or multi-thread schedulers was fixed. To demonstrate this, the existing video_replayer example has a new dual_window parameter in its YAML config that can be set to true in order to run a variant of that example which opens a pair of display windows that run in parallel.

Holoviz module

• The Holoviz operator and module now has a new parameter display_color_space. This allows HDR output on Linux distributions and displays supporting that feature.
• The Holoviz operator and module now support callbacks. These callbacks can be used to receive updates on key presses, mouse position and buttons, and window size. Additionally, there is a callback executed when the HolovizOp finishes drawing layers, allowing apps to

Breaking Changes

• Compile time type checking of arguments passed to logging functions may be more strict than before. If you see compilation errors related to passing enum types to the HOLOSCAN_LOG_* functions, this can be resolved either by casting the enum to an integer type or defining a fmt::formatter for the enum type (please see https://fmt.dev/latest/api/#formatting-user-defined-types).

Bug fixes

Issue	Description
4728118	Holoviz window's initialization is not thread-safe so cannot launch multiple Holoviz windows at once.
4866740	The run vscode_remote command is failing with the message docker cp /home/xxx/.ssh/id_ed25519 xxxxxx:/home/holoscan-sdk/.ssh/ no such directory. The run vscode_remote command is fixed to reflect the home directory change (from /home/holoscan-sdk to /home/holoscan) in the container.

Known Issues

This section supplies details about issues discovered during development and QA but not resolved in this release.

Issue	Description
4062979	When Operators connected in a Directed Acyclic Graph (DAG) are executed in a multithreaded scheduler, it is not ensured that their execution order in the graph is adhered.
4267272	AJA drivers cannot be built with RDMA on IGX SW 1.0 DP iGPU due to missing nv-p2p.h. Expected to be addressed in IGX SW 1.0 GA.
4384768	No RDMA support on JetPack 6.0 DP and IGX SW 1.0 DP iGPU due to missing nv-p2p kernel module. Expected to be addressed in JP 6.0 GA and IGX SW 1.0 GA respectively.
4190019	Holoviz segfaults on multi-gpu setup when specifying device using the --gpus flag with docker run. Current workaround is to use CUDA_VISIBLE_DEVICES in the container instead.
4210082	v4l2 applications seg faults at exit or crashes at start with '_PyGILState_NoteThreadState: Couldn't create autoTSSkey maping'
4339399	High CPU usage observed with video_replayer_distributed application. While the high CPU usage associated with the GXF UCX extension has been fixed since v1.0, distributed applications using the MultiThreadScheduler (with the check_recession_period_ms parameter set to 0 by default) may still experience high CPU usage. Setting the HOLOSCAN_CHECK_RECESSION_PERIOD_MS environment variable to a value greater than 0 (e.g. 1.5) can help reduce CPU usage. However, this may result in increased latency for the application until the MultiThreadScheduler switches to an event-based multithreaded scheduler.
4318442	UCX cuda_ipc protocol doesn't work in Docker containers on x86_64. As a workaround, we are currently disabling the UCX cuda_ipc protocol on all platforms via the UCX_TLS environment variable.
4325468	The V4L2VideoCapture operator only supports YUYV and AB24 source pixel formats, and only outputs the RGBA GXF video format. Other source pixel formats compatible with V4L2 can be manually defined by the user, but they're assumed to be equivalent to RGBA8888.
4325585	Applications using MultiThreadScheduler may exit early due to timeouts. This occurs when the stop_on_deadlock_timeout parameter is improperly set to a value equal to or less than check_recession_period_ms, particularly if check_recession_period_ms is greater than zero.
4301203	HDMI IN fails in v4l2_camera on IGX Orin Devkit for some resolution or formats. Try the latest firmware as a partial fix. Driver-level fixes expected in IGX SW 1.0 GA.
4384348	UCX termination (either ctrl+c , press 'Esc' or clicking close button) is not smooth and can show multiple error messages.
4481171	Running the driver for a distributed applications on IGX Orin devkits fails when connected to other systems through eth1. A workaround is to use eth0 port to connect to other systems for distributed workloads.
4458192	In scenarios where distributed applications have both the driver and workers running on the same host, either within a Docker container or directly on the host, there's a possibility of encountering "Address already in use" errors. A potential solution is to assign a different port number to the HOLOSCAN_HEALTH_CHECK_PORT environment variable (default: 8777), for example, by using export HOLOSCAN_HEALTH_CHECK_PORT=8780.
4782662	Installing Holoscan wheel 2.0.0 or later as root causes error.
4768945	Distributed applications crash when the engine file is unavailable/generating engine file.
4753994	Debugging Python application may lead to segfault when expanding an operator variable.
	Wayland: holoscan::viz::Init() with existing GLFW window fails.
4394306	When Python bindings are created for a C++ Operator, it is not always guaranteed that the destructor will be called prior to termination of the Python application. As a workaround to this issue, it is recommended that any resource cleanup should happen in an operator's stop() method rather than in the destructor.
4638505	A distributed application bug was fixed which failed to properly allow tensors to be sent from one fragment when the destination operator in another fragment was a GXFOperator. Connections were previously working correctly only when the destination operator was a native Operators. It now works as expected for GXFOperators as well.
4824619	iGPU: Rendering YUV images with HolovizOp fails on first run

v2.4.0

Release Artifacts

• 🐋 Docker container: tag v2.4.0-dgpu and v2.4.0-igpu
• 🐍 Python wheel: pip install holoscan==2.4.0
• 📦️ Debian packages: 2.4.0.1-1
• 📕 Documentation

See supported platforms for compatibility.

Release Notes

New Features and Improvements

• The Holoscan CLI packager has been updated to create application containers that are up to 78% smaller than their previous standard container based size. The new --includes option allows the packager to include only runtime dependencies relevant to the application. Refer to the documentation for more information.

• The Holoscan pipeline metadata feature introduced for the C++ API in release v2.3 is now also available from the Python API. Interaction with metadata can be done with an API very similar to Python's built-in dictionaries. Please see the dynamic application metadata section of the user guide for details.

• The V4L example now supports a YUV configuration to display YUV input directly without RGB conversion.

• Added support to build Holoscan SDK without docker cache in run script e.g., ./run build --no-cache

Core

• It is now possible to set a HOLOSCAN_QUEUE_POLICY environment variable to override the default queue policy that is used by the input and output ports of the SDK. Valid options (case insensitive) are:

  • "pop": a new item that arrives when the queue is full replaces the oldest item
  • "reject": a new item that arrives when the queue is discarded
  • "fail": terminate the application if a new item arrives when the queue is full

  The default behavior remains "fail" if the environment variable is not specified. If an operator's setup method explicitly sets a receiver or transmitter via the IOSpec::connector method, this default value does not override the policy of that connector.

• Improve the handling of extra arguments passed to the application via CLI.

  • Now calls the allow_extras() method to permit extra arguments instead of ignoring them on an ExtrasError in the CLI11 parser.

• The SDK no longer spawns a new process to check for unused network ports for UCX communication when running a distributed application. Previously, this process caused issues such as redundant system resource consumption when the Holoscan distributed application was run as part of a larger application (e.g., importing Holoscan as a Python module after importing other modules) because a new process was created solely for checking unused network ports. This issue has been resolved by performing the network port check in in-process mode.

Operators/Resources

• Two new operators useful for examples and testing were added. PingTensorTxOp will emit a TensorMap containing a single tensor with user-specified name, shape, data type and storage type (e.g. host vs. device). PingTensorRxOp will receive a message containing a TensorMap and print some attributes of any tensors contained within it. Versions of these previously existed in examples and test code, but have now been moved to a common public location (holoscan::ops namespace for C++ and under holoscan.operators for Python).
• The V4L2VideoCaptureOp now supports passing the input buffer unmodified to the output. This can be enabled by using the parameter pass_through, by default this is disabled.
• Handle/enhance various cases of multi-receiver input ports (holoscan::InputContext::receive<std::vector<T>>())
  • Support receiving an array of TensorMap items from the input port.
  • Improve handling of cases where no data or null pointers are received from the input port.
  • Throw an invalid argument exception if the operator attempts to receive non-vector data (op_input.receive<T>()) from an input port with a queue size of IOSpec::kAnySize.
  • Avoid using nvidia::TypenameAsString for the type name in error messages, as it may include characters that are not permitted in the message (e.g., {anonymous}), which could be interpreted as a format specifier. This can result in an exception being thrown during message formatting.
• The HolovizOp now supports YUV (aka YCbCr) images as input. Various 420 and 422 formats are supported.
  • New image formats:
    • y8u8y8v8_422_unorm
    • u8y8v8y8_422_unorm
    • y8_u8v8_2plane_420_unorm
    • y8_u8v8_2plane_422_unorm
    • y8_u8_v8_3plane_420_unorm
    • y8_u8_v8_3plane_422_unorm
    • y16_u16v16_2plane_420_unorm
    • y16_u16v16_2plane_422_unorm
    • y16_u16_v16_3plane_420_unorm
    • y16_u16_v16_3plane_422_unorm
  • YUV color model conversions:
    • yuv_601
    • yuv_709
    • yuv_2020
  • YUV ranges:
    • itu_full
    • itu_narrow
  • Chroma locations in x and y:
    • cosited_even
    • midpoint
• The AJASourceOp now supports the following video formats:
  • 720p @ 50, 59.94, 60Hz
  • 1080i @ 50, 59.94, 60Hz
  • 1080p @ 23.98, 24, 25, 29.97, 30, 50, 59.94, 60Hz
  • 3840x2160 (UHD) @ 23.98, 24, 25, 29.97, 30, 50, 59.94, 60Hz
  • 4096x2160 (4K) @ 23.98, 24, 25, 29.97, 30, 50, 59.94, 60Hz

Holoviz module

• Now supports YUV (aka YCbCr) images and YUV conversion parameters. The functions to specify image layer data have been extended to support planar formats.
  • New entry point ImageYuvModelConversion() to specify the YUV model conversion (BT.601, BT.709, BT.2020)
  • New entry point ImageYuvRange() to specify the YUV range (ITU full and ITU narrow)
  • New entry point ImageChromaLocation() to specify the chroma location (cosited even and midpoint)

Utils

• An aja_build.sh script was added to automate the download, build, and loading of the AJA NTV2 drivers and SDK.

HoloHub

Documentation

Breaking Changes

Bug fixes

Issue	Description
-	Holoviz operator fails with Surface format '29, 0' not supported when enabling sRGB framebuffer in headless mode.
-	Fixed a bug where the run vscode --parallel <num_workers> command was not working as expected, displaying the message arg: unbound variable.
4791938	v4l_camera doesn't work with the USB camera when 800x600 is set, and there are multiple sizes available for width 800.
4792457	Heap memory error was found in GXFParameterAdaptor with AddressSanitizer (ASAN) during dynamic analysis.
4752615	In Python, the operator's parameter values are not available in the initialize() method. This bug was introduced in version 2.1.0.
4510522	V4L2VideoCaptureOp does not work with RGB.

Known Issues

This section supplies details about issues discovered during development and QA but not resolved in this release.

Issue	Description
4062979	When Operators connected in a Directed Acyclic Graph (DAG) are executed in a multithreaded scheduler, it is not ensured that their execution order in the graph is adhered.
4267272	AJA drivers cannot be built with RDMA on IGX SW 1.0 DP iGPU due to missing nv-p2p.h. Expected to be addressed in IGX SW 1.0 GA.
4384768	No RDMA support on JetPack 6.0 DP and IGX SW 1.0 DP iGPU due to missing nv-p2p kernel module. Expected to be addressed in JP 6.0 GA and IGX SW 1.0 GA respectively.
4190019	Holoviz segfaults on multi-gpu setup when specifying device using the --gpus flag with docker run. Current workaround is to use CUDA_VISIBLE_DEVICES in the container instead.
4210082	v4l_camera example seg faults at exit.
4339399	High CPU usage observed with video_replayer_distributed application. While the high CPU usage associated with the GXF UCX extension has been fixed since v1.0, distributed applications using the MultiThreadScheduler (with the check_recession_period_ms parameter set to 0 by default) may still experience high CPU usage. Setting the HOLOSCAN_CHECK_RECESSION_PERIOD_MS environment variable to a value greater than 0 (e.g. 1.5) can help reduce CPU usage. However, this may result in increased latency for the application until the MultiThreadScheduler switches to an event-based multithreaded scheduler.
4318442	UCX cuda_ipc protocol doesn't work in Docker containers on x86_64. As a workaround, we are currently disabling the UCX cuda_ipc protocol on all platforms via the UCX_TLS environment variable.
4325468	The V4L2VideoCapture operator only supports YUYV and AB24 source pixel formats, and only outputs the RGBA GXF video format. Other source pixel formats compatible with V4L2 can be manually defined by the user, but they're assumed to be equivalent to RGBA8888.
4325585	Applications using MultiThreadScheduler may exit early due to timeouts. This occurs when the stop_on_deadlock_timeout parameter is improperly set to a value equal to or less than check_recession_period_ms, particularly if check_recession_period_ms is greater than zero.
4301203	HDMI IN fails in v4l2_camera on IGX Orin Devkit for some resolution or formats. Try the latest firmware as a partial fix. Driver-level fixes expected in IGX SW 1.0 GA.
4384348	UCX termination (either ctrl+c , press 'Esc' or clicking close button) is not smooth and can show multiple error messages.
4481171	Running the driver for a distributed applications on IGX Orin devkits fails when connected to other systems through eth1. A workar...

v2.3.0

Release Artifacts

• 🐋 Docker container: tag v2.3.0-dgpu and v2.3.0-igpu
• 🐍 Python wheel: pip install holoscan==2.3.0
• 📦️ Debian packages: 2.3.0.1-1
• 📕 Documentation

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Core

• Explicitly delete the copy constructor and assignment operator of Config/Executor/Graph classes private to prevent copying of these objects. This change ensures that the Config/Executor/Graph classes are not copied, as they are not intended to be copied, which is inefficient. This change is backward compatible, as the classes are still movable.
  • [Internal] Application::fragment_graph_ is now a std::shared_ptr to FragmentGraph to prevent copying of FragmentGraph objects in Python bindings.
  • [Internal] Fragment::graph_ is now a std::shared_ptr to OperatorGraph to prevent copying of OperatorGraph objects in Python bindings.
  • [Internal] Added Fragment::config_shared()/Fragment::executor_shared()/Fragment::graph_shared() to return a shared pointer to the Config/Executor/OperatorGraph objects respectively.

Operators/Resources

• A new decorator, holoscan.decorator.create_op is provided that can wrap an existing function or generator as a native Python Operator. This new API is still considered experimental and may be updated in a subsequent release based on initial feedback. An example of using this decorator is provided under examples/python_decorator/video_replayer.py as well as in the test applications within python/tests/system/test_decorator_apps.py.

Utils

HoloHub

Documentation

Breaking Changes

Bug fixes

Issue	Description
4687735	Fixed a bug where the CountCondition in Python doesn't accept a negative value for the count parameter, even though it is allowed in C++. Note that using a negative value for count is not recommended as it would lead to an infinite loop until it reaches the minimum value of the data type (int64_t) and then starts counting down from the maximum value up to zero.
4689604	Fixed a bug where GXF extensions listed in the config file (YAML) are not loaded when GXFCodeletOp or GXFComponentResource is used. Test cases have been added to verify the fix, and the documentation has been updated to reflect the changes. The HOLOSCAN_WRAP_GXF_COMPONENT_AS_RESOURCE macro has also been updated to support the constructor with no arguments.
4706559	Fixed a bug where nullptr was returned even when no message was available in calls to holoscan::IOContext::receive<T*>() or holoscan::IOContext::receive<std::shared_ptr<T>> in the C++ API. Now, the receive method correctly returns a holoscan::unexpected<holoscan::RuntimeError> value when no message is available.

Known Issues

This section supplies details about issues discovered during development and QA but not resolved in this release.

Issue	Description
4062979	When Operators connected in a Directed Acyclic Graph (DAG) are executed in a multithreaded scheduler, it is not ensured that their execution order in the graph is adhered.
4267272	AJA drivers cannot be built with RDMA on IGX SW 1.0 DP iGPU due to missing nv-p2p.h. Expected to be addressed in IGX SW 1.0 GA.
4384768	No RDMA support on JetPack 6.0 DP and IGX SW 1.0 DP iGPU due to missing nv-p2p kernel module. Expected to be addressed in JP 6.0 GA and IGX SW 1.0 GA respectively.
4190019	Holoviz segfaults on multi-gpu setup when specifying device using the --gpus flag with docker run. Current workaround is to use CUDA_VISIBLE_DEVICES in the container instead.
4210082	v4l_camera example seg faults at exit.
4339399	High CPU usage observed with video_replayer_distributed application. While the high CPU usage associated with the GXF UCX extension has been fixed since v1.0, distributed applications using the MultiThreadScheduler (with the check_recession_period_ms parameter set to 0 by default) may still experience high CPU usage. Setting the HOLOSCAN_CHECK_RECESSION_PERIOD_MS environment variable to a value greater than 0 (e.g. 1.5) can help reduce CPU usage. However, this may result in increased latency for the application until the MultiThreadScheduler switches to an event-based multithreaded scheduler.
4318442	UCX cuda_ipc protocol doesn't work in Docker containers on x86_64. As a workaround, we are currently disabling the UCX cuda_ipc protocol on all platforms via the UCX_TLS environment variable.
4325468	The V4L2VideoCapture operator only supports YUYV and AB24 source pixel formats, and only outputs the RGBA GXF video format. Other source pixel formats compatible with V4L2 can be manually defined by the user, but they're assumed to be equivalent to RGBA8888.
4325585	Applications using MultiThreadScheduler may exit early due to timeouts. This occurs when the stop_on_deadlock_timeout parameter is improperly set to a value equal to or less than check_recession_period_ms, particularly if check_recession_period_ms is greater than zero.
4301203	HDMI IN fails in v4l2_camera on IGX Orin Devkit for some resolution or formats. Try the latest firmware as a partial fix. Driver-level fixes expected in IGX SW 1.0 GA.
4384348	UCX termination (either ctrl+c , press 'Esc' or clicking close button) is not smooth and can show multiple error messages.
4481171	Running the driver for a distributed applications on IGX Orin devkits fails when connected to other systems through eth1. A workaround is to use eth0 port to connect to other systems for distributed workloads.
4458192	In scenarios where distributed applications have both the driver and workers running on the same host, either within a Docker container or directly on the host, there's a possibility of encountering "Address already in use" errors. A potential solution is to assign a different port number to the HOLOSCAN_HEALTH_CHECK_PORT environment variable (default: 8777), for example, by using export HOLOSCAN_HEALTH_CHECK_PORT=8780.
	Wayland: holoscan::viz::Init() with existing GLFW window fails.

Holoscan SDK v2.2.0 Release

Release Artifacts

• 🐋 Docker container: tag v2.2.0-dgpu and v2.2.0-igpu
• 🐍 Python wheel: pip install holoscan==2.2.0
• 📦️ Debian packages: 2.2.0.0-1
• 📕 Documentation

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Core

• Explicitly delete the copy constructor and assignment operator of Config/Executor/Graph classes private to prevent copying of these objects. This change ensures that the Config/Executor/Graph classes are not copied, as they are not intended to be copied, which is inefficient. This change is backward compatible, as the classes are still movable.
  • [Internal] Application::fragment_graph_ is now a std::shared_ptr to FragmentGraph to prevent copying of FragmentGraph objects in Python bindings.
  • [Internal] Fragment::graph_ is now a std::shared_ptr to OperatorGraph to prevent copying of OperatorGraph objects in Python bindings.
  • [Internal] Added Fragment::config_shared()/Fragment::executor_shared()/Fragment::graph_shared() to return a shared pointer to the Config/Executor/OperatorGraph objects respectively.

Operators/Resources

• A new decorator, holoscan.decorator.create_op is provided that can wrap an existing function or generator as a native Python Operator. This new API is still considered experimental and may be updated in a subsequent release based on initial feedback. An example of using this decorator is provided under examples/python_decorator/video_replayer.py as well as in the test applications within python/tests/system/test_decorator_apps.py.

Utils

HoloHub

Documentation

Breaking Changes

Bug fixes

Issue	Description
4687735	Fixed a bug where the CountCondition in Python doesn't accept a negative value for the count parameter, even though it is allowed in C++. Note that using a negative value for count is not recommended as it would lead to an infinite loop until it reaches the minimum value of the data type (int64_t) and then starts counting down from the maximum value up to zero.
4689604	Fixed a bug where GXF extensions listed in the config file (YAML) are not loaded when GXFCodeletOp or GXFComponentResource is used. Test cases have been added to verify the fix, and the documentation has been updated to reflect the changes. The HOLOSCAN_WRAP_GXF_COMPONENT_AS_RESOURCE macro has also been updated to support the constructor with no arguments.
4706559	Fixed a bug where nullptr was returned even when no message was available in calls to holoscan::IOContext::receive<T*>() or holoscan::IOContext::receive<std::shared_ptr<T>> in the C++ API. Now, the receive method correctly returns a holoscan::unexpected<holoscan::RuntimeError> value when no message is available.

Known Issues

This section supplies details about issues discovered during development and QA but not resolved in this release.

Issue	Description
4062979	When Operators connected in a Directed Acyclic Graph (DAG) are executed in a multithreaded scheduler, it is not ensured that their execution order in the graph is adhered.
4267272	AJA drivers cannot be built with RDMA on IGX SW 1.0 DP iGPU due to missing nv-p2p.h. Expected to be addressed in IGX SW 1.0 GA.
4384768	No RDMA support on JetPack 6.0 DP and IGX SW 1.0 DP iGPU due to missing nv-p2p kernel module. Expected to be addressed in JP 6.0 GA and IGX SW 1.0 GA respectively.
4190019	Holoviz segfaults on multi-gpu setup when specifying device using the --gpus flag with docker run. Current workaround is to use CUDA_VISIBLE_DEVICES in the container instead.
4210082	v4l_camera example seg faults at exit.
4339399	High CPU usage observed with video_replayer_distributed application. While the high CPU usage associated with the GXF UCX extension has been fixed since v1.0, distributed applications using the MultiThreadScheduler (with the check_recession_period_ms parameter set to 0 by default) may still experience high CPU usage. Setting the HOLOSCAN_CHECK_RECESSION_PERIOD_MS environment variable to a value greater than 0 (e.g. 1.5) can help reduce CPU usage. However, this may result in increased latency for the application until the MultiThreadScheduler switches to an event-based multithreaded scheduler.
4318442	UCX cuda_ipc protocol doesn't work in Docker containers on x86_64. As a workaround, we are currently disabling the UCX cuda_ipc protocol on all platforms via the UCX_TLS environment variable.
4325468	The V4L2VideoCapture operator only supports YUYV and AB24 source pixel formats, and only outputs the RGBA GXF video format. Other source pixel formats compatible with V4L2 can be manually defined by the user, but they're assumed to be equivalent to RGBA8888.
4325585	Applications using MultiThreadScheduler may exit early due to timeouts. This occurs when the stop_on_deadlock_timeout parameter is improperly set to a value equal to or less than check_recession_period_ms, particularly if check_recession_period_ms is greater than zero.
4301203	HDMI IN fails in v4l2_camera on IGX Orin Devkit for some resolution or formats. Try the latest firmware as a partial fix. Driver-level fixes expected in IGX SW 1.0 GA.
4384348	UCX termination (either ctrl+c , press 'Esc' or clicking close button) is not smooth and can show multiple error messages.
4481171	Running the driver for a distributed applications on IGX Orin devkits fails when connected to other systems through eth1. A workaround is to use eth0 port to connect to other systems for distributed workloads.
4458192	In scenarios where distributed applications have both the driver and workers running on the same host, either within a Docker container or directly on the host, there's a possibility of encountering "Address already in use" errors. A potential solution is to assign a different port number to the HOLOSCAN_HEALTH_CHECK_PORT environment variable (default: 8777), for example, by using export HOLOSCAN_HEALTH_CHECK_PORT=8780.
	Wayland: holoscan::viz::Init() with existing GLFW window fails.

Holoscan SDK v2.1.0

Release Artifacts

• 🐋 Docker container: tag v2.1.0-dgpu and v2.1.0-igpu
• 🐍 Python wheel: pip install holoscan==2.1.0
• 📦️ Debian packages: 2.1.0.1-1
• 📕 Documentation

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Core

• A report with execution time statistics for individual operators can now be enabled. This report will contain information like median, 90th percentile and maximum times for operator execution. Setting environment variable HOLOSCAN_ENABLE_GXF_JOB_STATISTICS=true enables this report (it is disabled by default as statistics collection may introduce a minor performance overhead). For more details see the documentation on the feature](https://docs.nvidia.com/holoscan/sdk-user-guide/gxf_job_statistics.html).

• The holoscan.Tensor object's data property in the Python API now returns an integer (pointer address) instead of a NULL PyCapsule object, potentially avoiding confusion about data availability. Users can confirm the presence of data via the __array_interface__ or __cuda_array_interface__ properties. This change allows for direct access to the data pointer, facilitating debugging and performance optimization.

• The string representation of the IOSpec object, generated by IOSpec::to_yaml_node(), includes ConditionType information in the type field. It correctly displays kNone when no condition (ConditionType::kNone in C++ and ConditionType.NONE in Python) is explicitly set.

  name: receiver
  io_type: kInput
  typeinfo_name: N8holoscan3gxf6EntityE
  connector_type: kDefault
  conditions:
    - type: kNone
  

• Enhanced the macros (HOLOSCAN_CONDITION_FORWARD_TEMPLATE, HOLOSCAN_RESOURCE_FORWARD_TEMPLATE, HOLOSCAN_OPERATOR_FORWARD_TEMPLATE, etc.) by using the full namespace of the classes, improving their robustness and adaptability across different namespace scopes.

• Updated the holoscan.core.py_object_to_arg() method to allow conversion of Python objects to Arg objects using YAML::Node. This resolves type mismatches, such as when the underlying C++ parameter expects an int32_t type but Python uses int64_t.

• The Python OperatorSpec/ComponentSpec class exposes the inputs and outputs properties, providing direct access to the input and output IO specs. This enhancement simplifies the process of setting conditions on inputs and outputs.

  def setup(self, spec: OperatorSpec):
    spec.input("data")
  
    # Set the NONE condition to the input port named `data`.
    spec.inputs["data"].condition(ConditionType.NONE)
  
    print(spec.inputs["data"])

• Workflows where an operator connects to multiple downstream operators within the same fragment may see a minor performance boost. This is because of an internal refactoring in how connections between operators are made. Previously a GXF broadcast codelet was automatically inserted into the graph behind the scenes to broadcast the output to multiple receivers. As of this release, direct 1:N connection from an output port is made without the framework needing to insert this extra codelet to enable this.

• fmt::format support for printing the Parameter class has been added (there is no longer a need to call the get() method to print out the contained value). This allows parameter values to be directly printed in HOLOSCAN_LOG_* statements. For example :

  MetaParameter p = MetaParameter<int>(5);
  HOLOSCAN_LOG_INFO("Formatted parameter value: {}", p);

  // can also pass parameter to fmt::format
  std::string format_message = fmt::format("{}", p);

Operators/Resources

• Most built-in operators now do additional validation of input tensors and will raise more helpful messages if the dimensions, data type or memory layout of the provided tensors is not as expected. Remaining operators (InferenceOp, InferenceProcessorOp) will updated in the next release.

• BayerDemosaicOp and FormatConverterOp will now automatically perform host->device copies if needed for either nvidia::gxf::VideoBuffer or Tensor inputs. Previously these operators only did the transfer automatically for nvidia::gxf::VideoBuffer, but not for Tensor and in the case of FormatConverterOp that transfer was only automatically done for pinned host memory. As of this release both operators will only copy unpinned system memory, leaving pinned host memory as-is.

• When creating Python bindings for C++ operators, it is now possible to register custom type conversion functions for user defined C++ types. These handle conversion to and from a corresponding Python type. See the newly expanded section on creating Python bindings for C++ operators for details.

• As of this release, all provided Python operators support passing conditions such as CountCondition or PeriodicCondition as positional arguments. In previous releases, there was a limitation that Python operators that wrapped an underlying C++ operator did not support this. As a concrete example, one could now pass a CountCondition to limit the number of frames the visualization operator will run for.

    holoviz = HolovizOp(
        self,
        # add count condition to stop the application after short duration (i.e. for testing)
        CountCondition(self, count),
        name="holoviz",
        **self.kwargs("holoviz"),
    )

• The AJA NTV2 dependency, and the corresponding AJA Source Operator, have been updated to use the latest official AJA NTV2 17.0.1 release. This new NTV2 version also introduces support for the KONA XM hardware.

• The Holoviz operator now supports setting the camera for layers rendered in 3d (geometry layer with 3d primitives and depth map layer).
  The camera eye, look at and up vectors can be initialized using parameters or dynamically changed at runtime by providing data at the respective input channels.
  More information can be found in the documentation.
  There is also a new C++ example holoviz_camera.cpp.

• The Holoviz operator now supports different types of camera pose outputs. In additional to the 4x4 row major projection matrix, a camera extrinsics model of type nvidia::gxf::Pose3D can now also be output. The output type is selected by setting the camera_pose_output_type parameter.

• The Holoviz operator now supports Wayland. Also the run launch command has been updated to support Wayland.

• The inference operator (InferenceOp) now supports a new optional parameter, temporal_map, which can be used to specify a frame interval at which inference will be run. For example, setting a value of 10 for a given model will result in inference only being run on every 10th frame. Intermediate frames will output the result from the most recent frame at which inference was run. The interval value is specified per-model, allowing different inference models to be run at different rates.

• The existing asynchronous scheduling condition is now also available from Python (via holoscan.conditions.AsynchronousCondition). For an example of usage, see the new asynchronous ping example.

• We introduce the GXFCodeletOp and GXFComponentResource classes, streamlining the import of GXF Codelets and Components into Holoscan applications. These additions simplify the setup process, allowing users to utilize custom GXF components more intuitively and efficiently.

  auto tx = make_operator<ops::GXFCodeletOp>(
      "tx",
      "nvidia::gxf::test::SendTensor",
      make_condition<CountCondition>(15),
      Arg("pool") = make_resource<GXFComponentResource>(
          "pool",
          "nvidia::gxf::BlockMemoryPool",
          Arg("storage_type") = static_cast<int32_t>(1),
          Arg("block_size") = 1024UL,
          Arg("num_blocks") = 2UL));

  tx = GXFCodeletOp(
      self,
      "nvidia::gxf::test::SendTensor",
      CountCondition(self, 15),
      name="tx",
      pool=GXFComponentResource(
          self,
          "nvidia::gxf::BlockMemoryPool",
          name="pool",
          storage_type=1,
          block_size=1024,
          num_blocks=2,
      ),
  )

  Please check out the examples in the examples/import_gxf_components directory for more information on how to use these new classes.

• When calling op_output.emit from the compute method of a Python operator, it is now possible to provide a third argument that overrides the default choice of the type of object type emitted. This is sometimes needed to emit a certain C++ type from a native Python operator when connecting it to a different Python operator that wraps an underlying C++ operator. For example, one could emit a Python string as a C++ std::string instead of the Python string object via op_output.emit(py_str, "port_name", "std::string"). See additional examples and a table of the C++ types registered by default [here](https://docs.nvidia.com/holoscan/sdk-user-guide/holoscan_create_operator_...

Holoscan SDK v2.0.0

Release Artifacts

• 🐋 Docker container: tag v2.0.0-dgpu and v2.0.0-igpu
• 🐍 Python wheel: pip install holoscan==2.0.0
• 📦️ Debian packages: 2.0.0.2-1
• 📕 Documentation

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Core

• make_condition, make_fragment, make_network_context, make_operator, make_resource, and
  make_scheduler now accept a non-const string or character array for the name parameter.
• A new event-based mult-thread scheduler (EventBasedScheduler) is available. It is an alternative to the existing, polling-based MultiThreadScheduler and can be used as a drop-in replacement. The only difference in parameters is that it does not take check_recession_period_ms parameter, as there is no such polling interval for this scheduler. It should give similar performance to the MultiThreadScheduler with a very short polling interval, but without the high CPU usage seen for the multi-thread scheduler in that case (due to constant polling for work by one thread).
• When an exception is raised from the Operator methods start, stop or compute, that exception will first trigger the underlying GXF scheduler to terminate the application graph and then the exception will be raised by Holoscan SDK. This resolves an issue with inconsistent behavior from Python and C++ apps on how exceptions were handled and fixes a crash in C++ apps when an operator raised an exception from the start or stop methods.
• Now, when an exception occurs during the execution of a Holoscan application, it is propagated to
  the application's run method, allowing users to catch and manage exceptions within their
  application.
  Previously, the Holoscan runtime would catch and log exceptions, with the application continuing
  to run (in Python) or exit (in C++) without a clear indication of the exception's origin.
  Users can catch and manage exceptions by enclosing the run method in a try block.
• In the case of the holoscan::Fragment::run_async and holoscan.Application.run_async methods
  for C++ and Python, they return std::future and concurrent.futures.Future respectively.
  The revised documentation advises using future.get() in C++ and future.result() in Python to
  wait until the application has completed execution and to address any exceptions that occurred.

Operators

• V4L2 Video Capture: added support to set manual exposure and gain values for cameras that support it.
• Inference: one can now run multiple instances of the Inference operator in a single application without ressource conflicts.

Utils

• Can now build from source for iGPU (IGX iGPU, Jetpack) from a non-iGPU system (IGX dGPU, x86_64)
• The NGC container now supports packaging and running Holoscan Application Packages using the Holoscan CLI.
• CLI runner - better handling of the use of available GPUs by reading the package manifest file and check the system for available GPUs. New --gpus argument to override the default values.

Breaking Changes

• The VideoStreamRecorderOp and VideoStreamReplayerOp now work without requiring the libgxf_stream_playback.so extension. Now that the extension is unused, it has been removed from the SDK and should no longer be listed under the extensions section of application YAML files using these operators.

• As of version 2.0, we have removed certain Python bindings to align with the unified logger interface:

  • Removed APIs:
    • holoscan.logger.enable_backtrace()
    • holoscan.logger.disable_backtrace()
    • holoscan.logger.dump_backtrace()
    • holoscan.logger.should_backtrace()
    • holoscan.logger.flush()
    • holoscan.logger.flush_level()
    • holoscan.logger.flush_on()
  • However, the following APIs remain accessible for Python. These are intended for logging in Holoscan's core or for C++ operators (e.g., using the HOLOSCAN_LOG_INFO macro), and are not designed for Python's logging framework. Python API users are advised to utilize the standard logging module for their logging needs:
    • holoscan.logger.LogLevel
    • holoscan.logger.log_level()
    • holoscan.logger.set_log_level()
    • holoscan.logger.set_log_pattern()

• Several GXF headers have moved from gxf/std to gxf/core:

  • parameter_parser.hpp
  • parameter_parser_std.hpp
  • parameter_registrar.hpp
  • parameter_storage.hpp
  • parameter_wrapper.hpp
  • resource_manager.hpp
  • resource_registrar.hpp
  • type_registry.hpp

• Some C++ code for tensor interoperability has been upstreamed from Holoscan SDK into GXF. The public holoscan::Tensor class will remain, but there have been a small number of backward incompatible changes in related C++ classes and methods in this release. Most of these were used internally and are unlikely to affect existing applications.

  • supporting classes holoscan::gxf::GXFTensor and holoscan::gxf::GXFMemoryBuffer have been removed. The DLPack functionality that was formerly in holoscan::gxf::GXFTensor is now available upstream in GXF's nvidia::gxf::Tensor.
  • The struct holoscan::gxf::DLManagedTensorCtx has been renamed to holoscan::gxf::DLManagedTensorContext and is now just an alias for nvidia::gxf::DLManagedTensorContext. It also has two additional fields (dl_shape and dl_strides to hold shape/stride information used by DLPack).
  • holoscan::gxf::DLManagedMemoryBuffer is now an alias to nvidia::gxf::DLManagedMemoryBuffer

• The GXF UCX extension, used in distributed applications, now sends data asynchronously by default, which can lead to issues such as insufficient memory on the transmitter side when a memory pool is used. Specifically, the concern is only for operators that have a memory pool and connect to an operator in a separate fragment of the distributed application. As a workaround, users can increase the num_blocks parameter to a higher value in the BlockMemoryPool or use the UnboundedAllocator to avoid the problem. This issue will be addressed in a future release by providing a more robust solution to handle the asynchronous data transmission feature of the UCX extension, eliminating the need for manual intervention (see Known Issue 4601414).

  • For fragments using a BlockMemoryPool, the num_blocks parameter can be increased to a higher value to avoid the issue. For example, the following code snippet shows the existing BlockMemoryPool resource being created with a higher number of blocks:

    recorder_format_converter = make_operator<ops::FormatConverterOp>(
      "recorder_format_converter",
      from_config("recorder_format_converter"),
      Arg("pool") =
        //make_resource<BlockMemoryPool>("pool", 1, source_block_size, source_num_blocks));
        make_resource<BlockMemoryPool>("pool", 1, source_block_size, source_num_blocks * 2));

    source_pool_kwargs = dict(
        storage_type=MemoryStorageType.DEVICE,
        block_size=source_block_size,
        #num_blocks=source_num_blocks,
        num_blocks=source_num_blocks * 2,
    )
    recorder_format_converter = FormatConverterOp(
            self,
            name="recorder_format_converter",
            pool=BlockMemoryPool(self, name="pool", **source_pool_kwargs),
            **self.kwargs("recorder_format_converter"),
        )
    )

  • Since the underlying UCXTransmitter attempts to send the emitted data regardless of the status of the downstream Operator input port's message queue, simply doubling the num_blocks may not suffice in cases where the receiver operator's processing time is slower than that of the sender operator.

  • If you encounter the issue, consider using the UnboundedAllocator instead of the BlockMemoryPool to avoid the problem. The UnboundedAllocator does not have a fixed number of blocks and can allocate memory as needed, though it can cause some overhead due to the lack of a fixed memory pool size and may lead to memory exhaustion if the memory is not released in a timely manner.
    The following code snippet shows how to use the UnboundedAllocator:

    ...
      Arg("pool") = make_resource<UnboundedAllocator>("pool");

    from holoscan.resources import UnboundedAllocator
    ...
            pool=UnboundedAllocator(self, name="pool"),
    ...

Bug fixes

Issue	Description
4381269	Fixed a bug that caused memory exhaustion when compiling the SDK in the VSCode Dev Container (using 'Tasks: Run Build Task') due to the missing CMAKE_BUILD_PARALLEL_LEVEL environment variable. Users can specify the number of jobs with the --parallel option (e.g., ./run vscode --parallel 16).
4569102	Fixed an issue where the log level was not updated from the environment variable when multiple Application classes were created during the session. Now, the log level setting in Application class allows for a reset from the environment variable if overridden.
4578099	Fixed a segfault in FormatConverterOp if used with a BlockMemoryPool with insufficient capacity to create the output tensor.
4571581	Fixed an issue where the documentation for the built-in operators was either missing or incorrectly rendered.
4591763	Application crashes if an exception is thrown from Operator::start or Operator::stop
4595680	Fixed an issue that caused the Inference operator to fail when multiple instances were composed in a...