Spill framework refactor for better performance and extensibility [databricks] #11747

abellina · 2024-11-21T19:55:27Z

This is a very large PR that I'd like some 👀 on. Marked it as draft as I still have some TODOs around more tests. The PR is NOT going to go to 24.12, it's just that we don't have a 25.02 available.

The main file I think one should focus on is SpillFramework.scala (yeap one file, let me know if you want me to break that into multiple files). SpillFramework.scala has a comment describing how things should work, please take a look at that.

The main contribution here is a simplification of the framework where we remove the idea of a RapidsBuffer that has to be acquired and unacquired, for the idea of a handle that just knows how to materialize. There isn't a concept of acquisition in the new framework.

There is a SpillableColumnarBatch api and a lazy-spillable api for Join that I did not touch and left there on purpose, but we can start to remove that API and create spillable handles that replicate the lazy behavior we wanted in lazy spillable, or the recomputing behavior we want for broadcasts. This is the second contribution of the PR: handles decide how to spill, not the framework.

There is one easily fixable shortcoming today in the multiple-spiller case, that I will fix in a follow on PR. While we are spilling a handle, the handle holds a lock. The same lock is used to figure out if the handle is spillable. A second thread that is trying to spill may need to wait for this lock (and spill) to finish, to figure out if it needs to spill that handle or not. We can make this more straightforward by handling the spill state separate from the materialization/data state, but I'd like to submit that work as an improvement.

I have run this against NDS @ 3TB in our perf cluster and I don't see regressions, and have run it against spill prone cases and I am able to see multiple threads in the "spill path", and no deadlocks. I'll post more results when I can run them.

Signed-off-by: Alessandro Bellina <[email protected]>

abellina · 2024-11-21T20:03:06Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsSerializerManager.scala

@@ -44,22 +44,20 @@ class RapidsSerializerManager (conf: SparkConf) {

  private lazy val compressionCodec: CompressionCodec = TrampolineUtil.createCodec(conf)


I'd like to remove this class, or make it much simpler.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/DeviceMemoryEventHandler.scala

sql-plugin/src/main/scala/com/nvidia/spark/rapids/SpillFramework.scala

…the cuda stream

revans2

didn't finish, but things keep changing under me so I thought I would publish what I have so far

revans2 · 2024-11-22T15:22:51Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/DeviceMemoryEventHandler.scala

        true
      }
+
+      shouldRetry


nit: why is this shouldRetry change in there? I assume it was for debugging at some point.

revans2 · 2024-11-22T15:26:02Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuShuffledSizedHashJoinExec.scala

@@ -1110,7 +1110,7 @@ class CudfSpillableHostConcatResult(
    val hmb: HostMemoryBuffer) extends SpillableHostConcatResult {

  override def toBatch: ColumnarBatch = {
-    closeOnExcept(buffer.getHostBuffer()) { hostBuf =>
+    closeOnExcept(buffer.getHostBuffer) { hostBuf =>


nit: why change this?

revans2 · 2024-11-22T15:27:15Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/JoinGatherer.scala

@@ -307,7 +307,6 @@ class LazySpillableColumnarBatchImpl(
          spill = Some(SpillableColumnarBatch(cached.get,
            SpillPriorities.ACTIVE_ON_DECK_PRIORITY))
        } finally {
-          // Putting data in a SpillableColumnarBatch takes ownership of it.


Is this no longer true?

I was debugging something here, and I must have forgotten to undo this change. coming up

but to answer your question, the spill framework takes ownership, always

abellina · 2024-11-22T15:35:33Z

Sorry for the rapid movement @revans2. I'll pause for a bit, and come back to address the ChunkedPacker comments and other comments I get.

zpuller

likewise leaving partial review

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

revans2

I still have a lot more to go through, but I thought I would at least get some of my comments in.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/ShuffleBufferCatalog.scala

revans2 · 2024-11-25T21:28:58Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/SpillableColumnarBatch.scala

+   * Set a new spill priority.
+   */
+  override def setSpillPriority(priority: Long): Unit = {
+    // TODO: handle.setSpillPriority(priority)


Or remove this entirely.

Other code in the plugin calls this. I kept it for now, but can clean up if you really want me to.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/SpillableColumnarBatch.scala

revans2 · 2024-11-25T21:33:18Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/SpillableColumnarBatch.scala

@@ -245,18 +338,29 @@ object SpillableColumnarBatch {
   */
  def apply(batch: ColumnarBatch,
      priority: Long): SpillableColumnarBatch = {
+    Cuda.DEFAULT_STREAM.sync()


I'll add comments in the code around why we have the sync.

The reason is that if you hand off an object to the framework, we could turn around and spill them immediately. We need to make sure the object is immutable when it hits the store.

We can move the sync back to the factory methods in the spill framework if that is desired?

revans2 · 2024-11-25T21:34:06Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/SpillableColumnarBatch.scala

-        RapidsBufferCatalog.addBatch(batch, initialSpillPriority)
-      }
-    }
+    Cuda.DEFAULT_STREAM.sync()


revans2 · 2024-11-25T21:34:38Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/SpillableColumnarBatch.scala

-      catalog: RapidsBufferCatalog): RapidsBufferHandle = {
-    withResource(batch) { batch =>
-      catalog.addBatch(batch, initialSpillPriority)
+      val handle = SpillableHostColumnarBatchHandle(batch)


Why no stream sync if it is needed for the other APIs?

this is a host batch, so we didn't need a device sync on this :)

revans2 · 2024-11-25T21:34:58Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/SpillableColumnarBatch.scala

-    val handle = withResource(buffer) { _ => 
-      RapidsBufferCatalog.addBuffer(buffer, meta, priority)
-    }
+    Cuda.DEFAULT_STREAM.sync()


revans2

Still have a lot more to look at, but I am making progress

revans2 · 2024-11-26T17:26:22Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/shuffle/BufferSendState.scala

            buffOffset += blockRange.rangeSize()
          }
          needsCleanup = false
        } catch {
-          case ioe: IOException =>
+          case ex: Throwable =>


Why do we need to catch Errors?

Let me fix to catch Exception

tests/src/test/scala/com/nvidia/spark/rapids/ShuffleBufferCatalogSuite.scala

tests/src/test/scala/com/nvidia/spark/rapids/spill/SpillFrameworkSuite.scala

revans2

Still not done, but I need to switch to some other things so I am going to comment on what I have looked at so far.

revans2 · 2024-11-26T17:40:00Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

+ *
+ * CUDA/Host synchronization:
+ *
+ * We assume all device backed handles are completely materialized on the device (before adding


Why do we need this? Please add that to the comments so that it is clear what is happening.

I added info in this comment. We could, at least for the GPU handles, add an event per handle and we could record on that event, instead of synchronizing. I think this means I should move the synchronization to the framework, right now (as you spotted), I let callers synchronize before they call the factory methods.

revans2 · 2024-11-26T17:43:04Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

+ * with extra locking is the `SpillableHostStore`, to maintain a `totalSize` number that is
+ * used to figure out cheaply when it is full.
+ *
+ * Handles hold a lock to protect the user against when it is either in the middle of


This is a bit confusing. So essentially the lock prevents race conditions when the object is in the middle of spilling or being closed?

Yeah, this is a confusing comment. It seems important to someone designing a new spillable handle, not to a end user. I have reworded it.

revans2 · 2024-11-26T17:43:49Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

+   * - If sizeInBytes is 0, the object is tracked by the stores so it can be
+   *   removed on shutdown, or by handle.close, but 0-byte handles are not spillable.
+   */
+  val sizeInBytes: Long


If this is approximate we should name it such. So there is no confusion about how that can be used.

Also this is a val, but if something spills wouldn't it change to 0 from whatever it was before? Shouldn't it be a def?

So it is, and it isn't. That's the issue with sizeInBytes. I am going to rework that and make it clearer. I think something like approxSizeInBytes makes sense, or trackedSizeInBytes, and it should be private[spill] going with the other things that are marked that way. But some handles do have a size that is not approximate, and IS used for creating buffers, so I kind of want that size reported with a different API, specific to each handle.

It is a val because I didn't see a point in changing it. If the object is spilled that is determined by dev, or host being empty.

It is also a val, because I wanted to set spillable at construction for 0-byte handles. Any other handle is spillable by definition at the level of the trait SpillableHandle, but if it's not a val then there's a chance that approxSizeInBytes is not ready at construction time, so I don't know if I can call it reliably.

agree this is confusing. The val part is because I want to mark objects that are 0 sized as not spillable, and I'd like to do that at construction time. It removes some locking that otherwise I have to do.

I have renamed this approxSizeInBytes and left sizeInBytes public in handles that do support this non-approximate value. I default this so in most cases approxSizeInBytes = sizeInBytes, but I do think this makes it clearer. Please take a look at d9490ee

revans2 · 2024-11-26T20:12:19Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

+   * or directly against the handle.
+   * @return sizeInBytes if spilled, 0 for any other reason (not spillable, closed)
+   */
+  def spill: Long


nit: I thought the convention in Scala was that a method with no parameters should have parens if it did work (like side effects). Should this be def spill(): Long then?

fixing all of these

Should be fixed with d9dbf36

revans2 · 2024-11-26T20:13:45Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

+  /**
+   * Method called to spill this handle. It can be triggered from the spill store,
+   * or directly against the handle.
+   * @return sizeInBytes if spilled, 0 for any other reason (not spillable, closed)


Just to clarify do we want them to return the result of calling sizeInBytes? Especially if it is just an approximate size? Can we clarify a little bit here about what is expected to be returned by this API?

Hopefully this is clearer with a comment I added.

revans2 · 2024-11-26T20:17:40Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

+   *       is a `val`.
+   * @return true if currently spillable, false otherwise
+   */
+  private[spill] def spillable: Boolean = sizeInBytes > 0


Why is this private to spill? Just curious, because if we want others to add spillable things in does that mean they have to put the handles in com.nvidia.spark.rapids.spill to make them work?

I think so. We can try to open that up if we want spillables in other packages, but right now it's all in here so I made this change in response to a comment by @jlowe. I did want to access this from unit test, and that's why it is private[spill] specifically.

revans2 · 2024-11-26T20:21:28Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

+    // do we care if:
+    // handle didn't fit in the store as it is too large.
+    // we made memory for this so we are going to hold our noses and keep going
+    // we could spill `handle` at this point.


I thought we spoke about this already. There are currently two cases.

We are running with no limit on the host memory, but a spill store limit

We have a host memory limit.

If we have a host memory limit, then the spill store is constrained by the host memory limit so we can ignore it here. we will not have allocated a HostMemoryBuffer that is too large to fit.

If we have unlimited host memory, then making a SpillableHostBufferHandle is there for code compatibility, but it should not be added to the host store at all. It should never spill. We have unlimited host memory.

We should also document this somehere.

…ring

…which is different than before

pxLi · 2024-12-09T00:11:07Z

build - re-deploy jenkins instance for an internal mandatory ops. rekicked the blossom-ci

pxLi · 2024-12-09T00:26:11Z

build

sql-plugin/src/main/scala/com/nvidia/spark/rapids/ShuffleBufferCatalog.scala

jlowe · 2024-12-06T22:21:40Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/ShuffleReceivedBufferCatalog.scala

-  private val MAX_TABLE_ID = Integer.MAX_VALUE
-  private val TABLE_ID_UPDATER = new IntUnaryOperator {
-    override def applyAsInt(i: Int): Int = if (i < MAX_TABLE_ID) i + 1 else 0
+  def getColumnarBatchAndRemove(handle: RapidsShuffleHandle,


I'm wondering why this method exists on the catalog rather than a getColumnarBatch method and a sizeInBytes method on the RapidsShuffleHandle. Then the caller can just use withResource on the shuffle handle directly and call those methods within the withResource block.

tableMeta is not part of the spill framework for UCX, it is stored in the shuffle catalogs instead using a RapidsShuffleHandle (which doesn't live in the spill framework... yet).

I think we can make these first class spill handles.

tests/src/test/scala/com/nvidia/spark/rapids/timezone/TimeZonePerfSuite.scala

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

zpuller

I just did a pass on the SpillFramework tests and had some questions

zpuller · 2024-12-04T19:42:04Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

+    }
+  }
+
+  def initialize(rapidsConf: RapidsConf): Unit = synchronized {


Should we check if it was already initialized, and if so, throw IllegalStateException ?

zpuller · 2024-12-09T18:00:39Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

+ *
+ * We handle aliasing of objects, either in the spill framework or outside, by looking at the
+ * reference count. All objects added to the store should support a ref count.
+ * If the ref count is greater than the expected value, we assume it is being aliased,


How does this work? What is the "expected value" derived from?

each handle overrides def spillable and does things differently. A buffer is super easy, it's just MemoryBuffer.getRefCount==1, a ColumnarBatch needs to figure out if it has repetition: A CB of [col0, col0, col0] will have three columns, but they will have a refcount of 3, so 3 is the spillable ref count, not 1, and the spillable method checks this for every column of the batch.

zpuller · 2024-12-09T18:01:52Z

tests/src/test/scala/com/nvidia/spark/rapids/spill/SpillFrameworkSuite.scala

+    }
+  }
+
+  test("an aliased contiguous table is not spillable (until closing the original) ") {


Shouldn't this be "until closing the alias?"

zpuller · 2024-12-09T18:03:28Z

tests/src/test/scala/com/nvidia/spark/rapids/spill/SpillFrameworkSuite.scala

+        assertResult(2)(SpillFramework.stores.deviceStore.numHandles)
+        assert(!handle.spillable)
+        assert(!aliasHandle.spillable)
+      } // we now have two copies in the store


Is this comment supposed to apply to the code inside the curly brackets, or just after closing? It's a bit confusing

zpuller · 2024-12-09T18:05:14Z

tests/src/test/scala/com/nvidia/spark/rapids/spill/SpillFrameworkSuite.scala

+
+  test("a buffer is not spillable until the owner closes columns referencing it") {
+    val (ct, _) = buildContiguousTable()
+    // the contract for spillable handles is that they take ownership


naive question - why can't the call to getBuffer contain the incRefCount? Are there instances where we want/need a non-owning reference (despite the contract)?

It should. We should file a follow on to fix that.

rapidsai/cudf#17566

zpuller · 2024-12-09T20:47:12Z

tests/src/test/scala/com/nvidia/spark/rapids/spill/SpillFrameworkSuite.scala

+      assertResult(0)(SpillFramework.stores.hostStore.numHandles)
+      assertResult(1)(SpillFramework.stores.diskStore.numHandles)
+      assert(handle.dev.isEmpty)
+      assert(handle.host.isDefined)


can you explain/add a comment as to why handle.host is defined but handle.host.get.host is not? I'm sure there's a good reason but it's not obvious

The way we have architected the spill handles is that they can cascade device->host->disk. But, the device handles don't have a host AND disk handles, they just have a host handle. If the host handle itself spilled, or could not fit on host to begin with, its host component is empty, and its disk component is set. So the view of the world from the handle's perspective is either set in the place where they are supposed to be, or they have a handle to something that can help them find the object later.

Ah I see, it's kind of like how the spill stores had a next store reference

zpuller · 2024-12-09T20:50:16Z

tests/src/test/scala/com/nvidia/spark/rapids/spill/SpillFrameworkSuite.scala

+  }
+
+  test("host originated: get host memory buffer") {
+    val spillPriority = -10


We had this value in the test before, I just copied it. I didn't want to remove spill priorities because we still want to implement them, and was hoping for tests to fail on me when I change the interface (or I could easily find/replace things)

zpuller · 2024-12-09T20:53:40Z

tests/src/test/scala/com/nvidia/spark/rapids/spill/SpillFrameworkSuite.scala

+  test("host originated: a host batch supports aliasing and duplicated columns") {
+    SpillFramework.shutdown()
+    val sc = new SparkConf
+    // disables the host store limit


Is this just a confusing config name where setting it (enabled) to true disables it (because it enables some alternative mechanism or something), or is the comment wrong?

The way we have defined host limits is that if they are set, the host spill's store own limits are ignored.

The idea is that host spills will be triggered only by a host OOM, and not via a host store limit. So yes, enabling the off heap limit, disables the host store limit. I'll try to change the comment.

zpuller · 2024-12-09T20:55:51Z

tests/src/test/scala/com/nvidia/spark/rapids/spill/SpillFrameworkSuite.scala

+  // this is a key behavior that we wanted to keep during the spill refactor
+  // where host objects that are added directly to the store do not cause a
+  // host->disk spill on their own, instead they will get spilled later
+  // due to device->host spills.


Is this because we are assuming the memory is already allocated/accounted for? Are we still adding the new buffer size to the total store allocated size?

yes and yes. The host memory was allocated, and we track it as such. But we are not going to actively spill right now. We'll spill later when device wants to spill.

This may change later. And when host limits are enabled (not host store limits) then it fits that model better.

zpuller · 2024-12-09T20:56:54Z

tests/src/test/scala/com/nvidia/spark/rapids/spill/SpillFrameworkSuite.scala

+    }
+  }
+
+  val hostSpillStorageSizes = Seq("-1", "1MB", "16MB")


what does size -1 do?

-1 means no host store limit. Will add comment.

Signed-off-by: Alessandro Bellina <[email protected]>

abellina · 2024-12-10T15:33:20Z

build

abellina · 2024-12-10T16:35:52Z

build

abellina · 2024-12-10T19:00:19Z

build

abellina · 2024-12-10T23:22:24Z

build

abellina · 2024-12-11T14:13:42Z

@zpuller @jlowe @revans2 Thank you for your review so far. Please let me know if there is anything else.

zpuller · 2024-12-11T19:07:47Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

+    if (!spillable) {
+      0L
+    } else {
+      synchronized {


Is there a race condition here? If Thread t1 calls spill(), and they see that spillable is true, but then thread t2 jumps in and calls materialize() and gets the lock and materializes. Say the buffer is on the device only at the moment, so t2 gets a ref to the DeviceMemoryBuffer, and ref count goes to 2. Now t2 drops the lock and goes on to actually use the buffer, and t1 proceeds and starts spilling even though it's no longer spillable.

Yes, this is an acceptable race.

In this case we copy to host but we can't free (we call close but we don't actually free). From the perspective of the spill framework, we are going to synchronize up to the point where we call .close on the DeviceMemoryBuffer. The rest is left to the caller. The caller needs to synchronize w.r.t. apis that it calls, and I think that part is understood (it's part of the original spill framework as well.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala

Signed-off-by: Alessandro Bellina <[email protected]>

abellina · 2024-12-11T22:10:33Z

build

abellina · 2024-12-12T14:32:42Z

running it again with db enabled.

abellina · 2024-12-12T14:32:47Z

build

abellina · 2024-12-13T14:29:47Z

build

Spill framework refactor for better performance and extensibility

ed170dc

Signed-off-by: Alessandro Bellina <[email protected]>

abellina commented Nov 21, 2024

View reviewed changes

jlowe reviewed Nov 21, 2024

View reviewed changes

code review comment changes

4cde905

revans2 mentioned this pull request Nov 22, 2024

[FEA] remove OOM retry disable config #11751

Open

abellina added 2 commits November 22, 2024 07:26

private[spill]

2619cab

remove extra sync, and make sure copyNext is always synchronous with …

b48b801

…the cuda stream

revans2 reviewed Nov 22, 2024

View reviewed changes

abellina added 2 commits November 22, 2024 07:32

private getHostBuffer

8c42a01

bring back comment on ownership

a53f10b

remove comment, as metrics are being fixed in a different pr

433746b

zpuller reviewed Nov 25, 2024

View reviewed changes

abellina changed the base branch from branch-24.12 to branch-25.02 November 25, 2024 19:59

revans2 reviewed Nov 25, 2024

View reviewed changes

revans2 reviewed Nov 26, 2024

View reviewed changes

abellina added 13 commits December 2, 2024 09:13

Merge remote-tracking branch 'nvidia/branch-25.02' into spill_refacto…

fd863f1

…ring

Upmerge

116478f

ChunkedPacker as a true iterator using a pool

513d084

approxSizeInBytes

d9490ee

add note to spill method

50e6b21

bring parens back

d9dbf36

make sure we return from spill non zero only when we actually spill

d1117e8

make ShuffleInfo a type alias

91006c5

catch Exception not all of Throwable

455e99c

rework comments around locking and cuda synchronization

16a1685

add a comment around how host objects are behaving with this change, …

6b692df

…which is different than before

Add columnar batch spill to host/disk and reconstitute tests

301a163

Add more tests in shuffle catalog suite

a882fb2

abellina marked this pull request as ready for review December 9, 2024 14:31

jlowe reviewed Dec 9, 2024

View reviewed changes

zpuller reviewed Dec 9, 2024

View reviewed changes

abellina added 3 commits December 9, 2024 14:00

Remove unintened change in TimeZonePerfSuite

a7db69b

Fix comment in ShuffleBufferCatalog

03b75d5

Code review comments

2919030

Signed-off-by: Alessandro Bellina <[email protected]>

abellina mentioned this pull request Dec 10, 2024

[FEA] Break up SpillFramework into multiple files #11852

Open

abellina added 2 commits December 10, 2024 07:20

SpillFramework.remove should be private[spill]

754a2f9

Remove whitespace changes

5bea81b

storesInternal cant be private[spill] due to HostAllocSuite

0702bd4

fix require condition

5aacedb

sameerz added the performance A performance related task/issue label Dec 11, 2024

zpuller reviewed Dec 11, 2024

View reviewed changes

jlowe reviewed Dec 11, 2024

View reviewed changes

sql-plugin/src/main/scala/com/nvidia/spark/rapids/spill/SpillFramework.scala Outdated Show resolved Hide resolved

Split SpillFramework.remove

88c18b7

Signed-off-by: Alessandro Bellina <[email protected]>

jlowe approved these changes Dec 11, 2024

View reviewed changes

abellina changed the title ~~Spill framework refactor for better performance and extensibility~~ Spill framework refactor for better performance and extensibility [databricks] Dec 12, 2024

abellina merged commit e3798d2 into NVIDIA:branch-25.02 Dec 13, 2024
53 checks passed

abellina mentioned this pull request Dec 18, 2024

[BUG] data corruption with spill framework changes #11885

Closed

		@@ -44,22 +44,20 @@ class RapidsSerializerManager (conf: SparkConf) {

		private lazy val compressionCodec: CompressionCodec = TrampolineUtil.createCodec(conf)

Spill framework refactor for better performance and extensibility [databricks] #11747

Spill framework refactor for better performance and extensibility [databricks] #11747

Conversation

abellina commented Nov 21, 2024

Choose a reason for hiding this comment

revans2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abellina commented Nov 22, 2024

zpuller left a comment

Choose a reason for hiding this comment

revans2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

revans2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

revans2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abellina Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pxLi commented Dec 9, 2024 • edited Loading

pxLi commented Dec 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zpuller left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abellina commented Dec 10, 2024

abellina commented Dec 10, 2024

abellina commented Dec 10, 2024

abellina commented Dec 10, 2024

abellina Dec 2, 2024 •

edited

Loading

pxLi commented Dec 9, 2024 •

edited

Loading

zpuller Dec 11, 2024 •

edited

Loading