-
Notifications
You must be signed in to change notification settings - Fork 193
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Impossible to build for Ubuntu Jammy 22.04 / Humble (probably because of OpenVDB / TBB) #232
Comments
Thanks for bringing this up / investigating, this is very frustrating. I think you've taken all of the reasonable steps, let me know if there's anything I can do to help! This seems like a crazy oversight and I really hope its handled quickly. Who's deciding on these openVDB versions? Is the problem that it's being released by someone else hap-haphazardly? I think there's nothing wrong with multiple released versions, so if 8.2 handles, we can target that and update the rosdistro keys to use that one explicitly. How is 8.1 built / released without TBB? I have not followed the nanoVDB work. I'm happy to see some of that Nvidia GVDB work getting back into the main repository vs trying to consume openVDB with their own project. I would agree that sounds indeed interesting - would that contain some of these same issues as we see with OpenVDB / TBB on versioning? I haven't found a good list of "where" the GPU is used most other than raycasting, which we don't use here, but it claims random access is faster, which we use alot of. So either way, its a good bet. I'd love to use NanoVDB as the basis of a new nav2 environmental model, but all these versioning issues are killing us and if this was in the core nav2 stack, that could be critical to release and put us back months on each OS release. The docs say NanoVDB has few external dependencies, is that true that it would solve this issue for us? If that is, and could stop future versioning issues, it might be a good idea to do it here as a trial and use our experience there to inform our future decisions. Having GPU enablable raycasting could really help save me a ton of time down the road on a voxel-grid replacement in Nav2. |
Hello there, just throwing in my 2 cents. In case you are interested, I managed to build OpenVDB from the source within the project build. The build script is here. Feel free to copy-paste it, just acknowledge the source :) With this "superbuild" approach I managed to also build the application on different Ubuntu, Fedora, CentOS distributions, macOS, and so on. I quickly tried to build I hope this helps, if you need extra support, or if you would like me to help migrate the build system just let me know. NOTE: The drawback of this approach is that you need to build OpenVDB from source(slow), but using Best wishes! |
I am not familiar with the Ubuntu release process so I don't really now, but it is certainly not triggered by the OpenVdb team.
No idea... maybe I miss something obvious
Well, I believe the versioning is properly done by the OpenVDB team: https://github.com/AcademySoftwareFoundation/openvdb/releases. Then it is just a matter of proper deb packaging. If we fear that the Ubuntu releases are going to be an issue, we can decide on doing our own ros openvdb release, for instance And I would love to see the result in nav2! |
Good to know that it is easy to build and link to from source, at least for trying it out. |
Not really, since I'm always building OpenVDB from source... to avoid exactly this sort of problems. I can't rely on Ubuntu and their extremely slow and complicated packaging process (I don't blame them though, it's a big jungle anyways). That's why I've decided to include it in my own build, I can control it and I know it's working (almost) everywhere. As an example, I'm using some docker images built on top of CentOS (manylinux) where I can't control which OpenVDB version will be packaged for that distro, therefore It made much more sense to just build it :) If you think it will be valuable for the community I can simple open a PR here with the updated build system (shouldn't take long) Let me know! |
All right, that leave us the following options: A. Wait for a package fix in Ubuntu.
B. Do an OpenVDB ros release.
C. OpenVDB source build as part of the package that needs it (here STVL), i.e @nachovizzo solution
I am personally in favor of B.
Thx, I ll take it and it will be helpful for testing STVL with Humble beta though I am not sure it is ideal in the long term |
I think B is the most reasonable, but I would give A a few weeks / a month to see if we can't get an updated release out and have that just be part of the package manager without our doing. If we add our own buildfarm setup, we'll have to be owning that process and version controlling for ROS and having to explain to folks why we need it. The issue we bring up is in the same theme as the other openVDB issues we've had in the past that did eventually get resolved. Want to start that process? It might be easier than we think if we have this versioning information / issues documented like above, seems like a no brainer to get v8.2 out -- maybe in addition to v8.1 if we can't replace |
I'm going to preface this by saying that this, unfortunately, is going to be tricky to fix. (as a side note, this is one of the reasons that we aggressively change Ubuntu distributions to the latest LTS as soon as packages are available; if you find these issues before the release, they are easier to fix in the distribution itself)
This is, by far, the best option. If we can convince the Ubuntu maintainers to upgrade, or at least take a patch to fix this particular issue, that will fix it for this package and for the rest of the distribution. It will also make it so that binaries that link both against code here and against anything else that uses OpenVDB will continue to work without ABI issues.
This can be done, but it has to be done extremely carefully. In particular, you need to make sure that nothing else in the system will accidentally find this package instead of the system one (as that can lead to the ABI issues I mentioned above). For instance, when we vendor OGRE for Rviz, we install it to /opt/ros/rolling/opt/rviz_ogre_vendor , which is outside of the normal hierarchy, and won't be found by CMake or pkg-config by default. Then you need to install an "extras" file so that that path is exposed to things looking for the vendor package. And finally, you need to be careful to do That makes it work, but it is still not great. If a third-party package wants to depend both on something that uses libogre-dev (from the system), and something that uses rviz_ogre_vendor, they can't. Actually, it is worse than that; it will all compile just fine, and randomly crash because of ABI issues. So in short, I'd say this is a last-ditch effort, because it is tricky to setup properly and tricky for downstream users to use properly.
If you can't do A, I'll suggest this one. However, you also need to be careful here. The absolute best way to do this is to build OpenVDB as a static library, and then link it into your main executable with private symbols. If you do that, then anything downstream that wants to depend on both your library and something else that uses OpenVDB should work. |
Thanks for the clarification Chris! Well then I guess we should push for A. I investigated a bit more and managed to build STVL (with few code adaptation #233) by uninstalling Launchpad bug report is here: https://bugs.launchpad.net/ubuntu/+source/openvdb/+bug/1970108 |
In the meantime waiting for a proper integration by Ubuntu packaging team, I've ported debian packaging on the v8.2.0 tag of source and built .deb. They are delivered without any warranty but we are currently using it without issue on our dev branches. You can find them here: https://github.com/wyca-robotics/openvdb/releases/tag/v8.2.0-debian |
@SteveMacenski @clalancette is there an official update on this? |
The resolution is in Ubuntu's hand. You can put pressure by confirmed the issue on this ticket: https://bugs.launchpad.net/ubuntu/+source/openvdb/+bug/1970108 |
Thank you @xouillet for the current workaround, I can confirm that STVL builds and works after installing these debians. |
An alternative idea: replacing OpenVDB with Bonxai https://github.com/facontidavide/Bonxai . But for this we would need a bit of visibility on Bonxai maintenance and maturity. What do you think @facontidavide ? |
facontidavide/Bonxai#1 We've had this discussion - it doesn't seem to be suitable given Davide's interest in it as more an exercise than something to support long-term. |
Said that... Bonxai just works. Even without further development, it might be sufficient for your application |
I really think you should try bonxai, if I remember what you do, you already have everything you need |
Definitely not a judgement on valuable proposition 😄 Its more a matter of risk, if you're yourself not a user of it and not going to maintain it long term, that introduces a bit more there. If you were using it actively in another project (or another major project was using it and could give the thumbs up) or committing to maintaining it for the foreseeable future, that'd be a different story. Since STVL is a relatively "historical" package, I'm a bit risk averse to major backbone changes since I'm not currently in a position to do extensive hardware testing to make sure this still works for the spanning set of user applications. I simply don't have the time or the hardware to make that kind of change comfortably or to contribute to Bonxai should there be issues or future features needed. This is very much a hardware-first driven idea and I don't have robots right now with a satisfactory number of sensors for me to test and feel totally comfortable that I'm not going to break people processing 10+ RGBD sensors (e.g. scales) for long-duration uses (e.g. verify no memory leaks or growth). OpenVDB works well enough (when you can install the damn thing). Its definitely not ideal, but its the beast we know after you get the install complete. Its my plan in the next ~2 years to fully revamp costmaps and this work so in the medium-term future this project may be deprecated by being ingested into the main navigation stack itself in a new form. With that said, I wouldn't block the use of it if someone opened a PR and was happy with it after some relatively extensive testing with a number of sensors. I don't know how I would deal with it at that time off the top of my head, but I'd find a compromise that works for everyone (maybe a bonxai branch of stvl released under a slightly different name & removing OpenVDB's version after enough users give the +1). |
Perfectly valid reasoning 😊 |
If I had 2 months I could make this my primary project, 10/10 I would rip out OpenVDB from this + vet Bonxai so we can be forever done with it. However even if I was time rich, I'm still 10+ sensor robot poor. The systems that motivated this work I left when I left Simbe. But, temporal-ness is a key requirement I'm establishing from the on-set when I do my environmental modeling kick. Its unclear if I'll get to that in 2023 or 2024, but its high up on the list next to localization and my current project in trajectory planning. Never a boring year... |
Thank you both for your comments. I am not using any RGB-D camera at the moment and hence I am only following from far the maintenance of STVL. But I may have to jump back early 2023, and if by that time OpenVDB is still not fixed in Ubuntu 22.04, I may just do that: create a Bonxai based branch of STVL and test its robustness on a continuously running robot. If I have good results I will happily share them and then we can debate if this is a desirable direction. I feel committed to help maintaining this beautiful package with which I had so good results in the past. |
Agreed, building OpenVDB isn't too bad but yes, it is rather annoying that we cannot release STVL binaries for users to install more naturally. Your release though makes that easier |
It seems that there hasn't been any progress on this. Anything that I can do to move it along? |
@doisyg I believe was making some headway but I know he's busy right now so it might be some time before he responds |
Sorry for adding noise once more, but I still can provide support for managing OpenVDB as a static library. I think it was option C while ago. While this will impact build times + binary size... It can help getting rid of annoying packaging problems from Debian/Ubuntu. Your call @SteveMacenski. Even if you need a draft I'm happy to do it. Best! |
Can you clarify what you mean / intend to do? A static |
Yes, to be clear, my option C above wasn't "commit a binary |
In my Dockerfile, I'm currently doing this: FROM ros:humble
# Build and install patched version of OpenVDB (see https://github.com/SteveMacenski/spatio_temporal_voxel_layer/issues/232).
RUN apt-get remove libopenvdb*; apt-get update && apt-get install -y libboost-system-dev libboost-iostreams-dev libtbb-dev libblosc-dev; \
git clone --recurse --branch v8.2.0-debian https://github.com/wyca-robotics/openvdb.git /opt/openvdb && \
mkdir /opt/openvdb/build && cd /opt/openvdb/build && \
cmake .. && \
make -j1 && make install && \
cd ..; rm -rf /opt/openvdb/build
# Perform ROS dependency installation for our workspace.
ADD ./src /opt/robot/src
RUN rosdep update && rosdep install --from-paths /opt/robot/src --ignore-src -r -y
# Build workspace.
ENV ROBOT_WS /opt/robot
RUN cd "$ROBOT_WS" && . /opt/ros/humble/setup.sh && colcon build Obviously, this takes forever. |
Good question - @clalancette I know git submodules don't clone in rosdistro for packaging, but is there another way we can ingest a desire to build OpenVDB next to STVL in the buildfarm to enable binary install? For instance, invoking the download of the source / build via cmake? I know that's probably not ideal, but I think we're currently picking the least sticky option 💩 That's basically the same as releasing to rosdistro OpenVDB in terms of compute time used, but at least isolates it if we statically link to STVL. The git submodule or bash script doing what @agoeckner shows in docker would work, but not allow us to release binaries. Which of course is still better than the status quo. |
You should be able to use the FetchContent functionality of cmake to do something like this. You can see how we use it a bit in the core in e.g. https://github.com/ros2/rosbag2/blob/rolling/mcap_vendor/CMakeLists.txt |
@agoeckner that seems like a good option with a reference 😄 |
It certainly is enough for me to build for our lab. How would we make this work with the ROS build farm though? |
I believe fetch content should work in the build farm, it has a network connection |
@SteveMacenski @doisyg Sorry for being late to the party. I've created a repo where I'm testing a fork where I conditionally build OpenVDB from the source. It takes a bit more time, but at least succeeds in building. The repo is here the fork is there If the builds suceed on the CI I will open a PR soon. |
Hey @nachovizzo, did your build succeed? edit: Nevermind, I see PR #267. |
Cool, I was able to build successfully, does this also mean that we can have apt-installable binaries for humble and iron? (sorry if it was mentioned somewhere, didn't see it) |
It does open that door now! |
Nice, who's going to do us the honor and save us 5 mins of building time in our CI? 😬 (I'm not sure what needs to be done otherwise would have volunteered) |
Tomorrow's release day :-) |
Actually, I'm blocked by the rolling transient issues right now so I can't run a release. But, Humble sync was today, so its at least a month before another one anyway so its no different than if I were to do it today vs 2 weeks from now. I've rescheduled it for 2 weeks from now to hopefully have all the 24.04 and rosdep stuff ironed out |
Where is the next Humble sync date listed? |
An iron release would also be appreciated (just so it's not forgotten 🙂 ) |
Anything I can do to help get these packages out the door? |
I'll do it today, syncs are in a weird state due to Rolling on 24.04 and some issue there and I just haven't been able to get to everything I would have liked to & many things generally blocked. I had tried to release it but I wanted to build locally in a clean container first and it was crashing on me from OpenVDB so it got shelved since I didn't have time to look into it. I can try today however If I run into any major issues again with the build farm on the vendor package, I could use some help from those that want to use it (as I'm not really using this on a day to day at the moment & my main priority is Nav2) |
I'm trying humble and seeing if the vendorization works, then I'll follow up with Iron ros/rosdistro#40797 |
@Timple @agoeckner @nachovizzo we fail: https://build.ros2.org/job/Hbin_ujv8_uJv8__openvdb_vendor__ubuntu_jammy_arm64__binary/1/console
Missing dependency! But, a good sign is that the source builds turn over https://build.ros2.org/job/Hsrc_uJ__spatio_temporal_voxel_layer__ubuntu_jammy__source/ so its just the details of getting the missing |
Strangely enough it works fine in a dockerfile: FROM ros:humble
SHELL [ "/bin/bash", "-c" ]
RUN apt-get update -qq && apt-get install -qqy python3-colcon-core
RUN mkdir -p /ws/src
WORKDIR /ws
RUN git clone https://github.com/SteveMacenski/spatio_temporal_voxel_layer /ws/src/spatio_temporal_voxel_layer
RUN rosdep update && rosdep install --from-paths src --ignore-src -y
RUN source /opt/ros/iron/setup.bash && colcon build But it seems it is required indeed. I'm a bit at the edge of my cmake knowledge here, but from why I can reproduce from this missing dependency, it seems the only missing dependency? So added it now. |
Ah, hold on. I need to isolate the openvdb_vendor package from the stvl. Because the dependencies come in indirectly. |
Because the STVL layer is being built in the same workspace, so you're pulling in deps from the layer that might satisfy this. The vendor/layer are built completely isolated in the build farm |
Ho lord! what a mesh... Getting closer anyways guys. I'm sorry I didn't see this coming, I do have this also as a 3rdparty dependency in vdbfusion, and for whatever reason, it didn't make it to my original PR . Sorry! |
Greetings @nachovizzo,
and also
are these the errors you were able to get rid of using compiler flags? Thanks! |
@grusel-opi Sorry, but after two years of course, I forgot about it :) So I don't know. In any case, the build should be working since #281 If you can't build it is likely due a system-wide installed TBB that mismatches with the requirements of the openvdb_vendor |
@nachovizzo Understandable :D Thanks for the quick reply anyways. Guess then i only have to find out what version of TBB is required and how to include it without breaking the system version. |
Hello,
FYI and to track the issue:
In preparation for the Humble release I tried to build STVL under Humble beta with Ubuntu 22.04 and the blocking point seems to be, surprise, the version of the OpenVDB package released in Ubuntu.
My quick analysis of the cause:
Ubuntu 20.04/focal
to v8.1 inUbuntu 22.04/jammy
Ubuntu 20.04/focal
to 2021.5 inUbuntu 22.04/jammy
#include <tbb/task_scheduler_init.h>
https://github.com/AcademySoftwareFoundation/openvdb/blob/ea786c46b7a1b5158789293d9b148b379fc9914c/openvdb/openvdb/tools/VolumeToMesh.h#L21See https://www.intel.com/content/www/us/en/developer/articles/release-notes/intel-oneapi-threading-building-blocks-release-notes.html and Usage of deprecated and removed tbb::task_scheduler_init and task API (Intel oneTBB) DrTimothyAldenDavis/SuiteSparse#72
OpenVDB seems to already have a version (v8.2) compatible with TBB 2021.5 since november : https://github.com/AcademySoftwareFoundation/openvdb/releases/tag/v8.2.0. My guess is that releasing v8.2 instead of v8.1 to
Ubuntu 22.04/jammy
would solve the issue. I will try to file a bug report on launchpad hoping that it will be faster than https://bugs.launchpad.net/ubuntu/+source/openvdb/+bug/1882998 (remember #167).(Side note, looking at OpenVDB release page: https://github.com/AcademySoftwareFoundation/openvdb/releases, I noticed the release of v9.0 with
It is GPU agnotic, more info: https://academysoftwarefoundation.github.io/openvdb/NanoVDB_FAQ.html
@SteveMacenski, did you follow ? Maybe that's interesting to look at for STVL and voxels in nav2)
The text was updated successfully, but these errors were encountered: