Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error after updating to convolve the entire 3D array #9

Open
shriram-jagan opened this issue Mar 19, 2024 · 0 comments
Open

Error after updating to convolve the entire 3D array #9

shriram-jagan opened this issue Mar 19, 2024 · 0 comments
Assignees

Comments

@shriram-jagan
Copy link
Owner

shriram-jagan commented Mar 19, 2024

3D convolutions at large resolutions use cuFFT internally and gives the following error. Reverting this PR fixes it since in that PR we loop over each variable and convolve instead of using convolutions for the entire 3D array. Creating this issue so that I'll remember to look into this in a couple of weeks.

Start: total_mass, total_energy: 152577474.45441937, 28337000790599.57
Signal 8 received by node 0, process 3158851 (thread 7fd8e8a20000) - obtaining backtrace
Signal 8 received by process 3158851 (thread 7fd8e8a20000) at: stack trace: 16 frames
  [0] = void cunumeric::cufft_convolution<double, 3>(Legion::FieldAccessor<(legion_privilege_mode_t)268435463, double, 3, long long, Realm::AffineAccessor<double, 3, long long>, false>, Legion::FieldAccessor<(legion_privilege_mode_t)1, double, 3, long long, Realm::AffineAccessor<double, 3, long long>, false>, Legion::FieldAccessor<(legion_privilege_mode_t)1, double, 3, long long, Realm::AffineAccessor<double, 3, long long>, false>, Realm::Rect<3, long long> const&, Realm::Rect<3, long long> const&, Realm::Rect<3, long long> const&) at unknown file:0 [00007fd8945c5324]
  [1] = void cunumeric::ConvolveImpl<(cunumeric::VariantKind)2>::operator()<(legate::Type::Code)11, 3, (void*)0>(cunumeric::ConvolveArgs&) const at unknown file:0 [00007fd8945c5b66]
  [2] = void cunumeric::convolve_template<(cunumeric::VariantKind)2>(legate::TaskContext&) at unknown file:0 [00007fd894565dba]
  [3] = cunumeric::ConvolveTask::gpu_variant(legate::TaskContext) at unknown file:0 [00007fd894565fd6]
  [4] = legate::detail::task_wrapper(void (*)(legate::TaskContext), legate_core_variant_t, std::optional<std::basic_string_view<char, std::char_traits<char> > >, void const*, unsigned long, void const*, unsigned long, Realm::Processor) at unknown file:0 [00007ffdcc9d77fb]
  [5] = void legate::LegateTask<cunumeric::ConvolveTask>::task_wrapper_<&cunumeric::ConvolveTask::gpu_variant, (legate_core_variant_t)2>(void const*, unsigned long, void const*, unsigned long, Realm::Processor) at unknown file:0 [00007fd8927b6d1b]
  [6] = Realm::Cuda::GPUProcessor::execute_task(unsigned int, Realm::ByteArrayRef const&) at unknown file:0 [00007ffff53d8a0d]
  [7] = Realm::Task::execute_on_processor(Realm::Processor) at unknown file:0 [00007ffff53c5872]
  [8] = Realm::KernelThreadTaskScheduler::execute_task(Realm::Task*) at unknown file:0 [00007ffff53c58f9]
  [9] = Realm::Cuda::GPUTaskScheduler<Realm::KernelThreadTaskScheduler>::execute_task(Realm::Task*) at unknown file:0 [00007ffff541e2d7]
  [10] = Realm::ThreadedTaskScheduler::scheduler_loop() at unknown file:0 [00007ffff53c3ec3]
  [11] = Realm::ThreadedTaskScheduler::scheduler_loop_wlock() at unknown file:0 [00007ffff53c4430]
  [12] = Realm::KernelThread::pthread_entry(void*) at unknown file:0 [00007ffff53c8325]
  [13] = start_thread at unknown file:0 [00007ffff2999608]
  [14] = __clone at unknown file:0 [00007ffff4b5a132]
  [15] = unknown symbol at unknown file:0 [ffffffffffffffff]
@shriram-jagan shriram-jagan self-assigned this Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant