Merge byods

.github/workflows/rust.yml

@@ -2,9 +2,11 @@ name: Rust
 
 on:
   push:
-    branches: [ master, ci, lattices, wasm-support, macro-in-macro, rel-index-trait, par ]
+    branches: [ master, ci, par, byods ]
   pull_request:
-    branches: [ master ]
+    branches: [ master, byods ]
+    paths-ignore:
+      - '**/*.MD'
 
 env:
   CARGO_TERM_COLOR: always

BYODS.MD

@@ -0,0 +1,40 @@
+BYODS is an extension of Ascent that allows custom data structures backing relations. The OOPSLA paper [Bring Your Own Data Structures to Datalog](https://dl.acm.org/doi/pdf/10.1145/3622840) describes the semantics and high-level design of BYODS, and presents a number of experiments demonstrating the capabilities of BYODS.
+
+With BYODS, any relation in an Ascent program can be tagged with a data structure provider:
+
+```Rust
+ascent! {
+    #[ds(rels_ascent::eqrel)]
+    relation rel(u32, u32);
+    // ...
+}
+```
+
+In the above example, relation `rel` is backed by the `eqrel` data structure provider, which makes `rel` an equivalence relation.
+
+In the above example, `rels_ascent::eqrel` is the path of the module containing the macros required of a data structure provider. You can define your own data structure providers. To see what macros are required of a data structure provider, see `ascent::rel`, which is the default data structure provider.
+
+The most important macros of a data structure privider are `rel_ind` and `rel_ind_common`. 
+
+Macro invocations for `rel_ind_common` and other macros look like this:
+
+```Rust
+my_provider::rel_ind_common!(
+  rel,          // rel name 
+  (u32, u32),   // column types
+  [[1]],        // logical indices
+  ser,          // parallel (par) or serial (ser)
+  (),           // user-specified params
+)
+```
+
+These macro invocations evaluate to types that implement certain traits. `rel_ind!` returned types must implement `ToRelIndex`. This trait provides an indirection, allowing relation indices to share data (the type returned by `rel_ind_common!`). The type returned by the `to_rel_index` function of this trait is used to read the relation data, and must implement `RelIndexRead` and `RelIndexReadAll` traits. The type returned by `to_rel_index_write` is used to write new values to an index, and must implement `RelIndexWrite` and `RelIndexMerge`. 
+
+These traits have `Key` and `Value` associated types. `Key` is the tuple of the indexed-on columns, and `Value` is the tuple of the remaining columns. For example, for an index [1, 2] of a relation `relation rel(Col0, Col1, Col2)`, the `Key` type would be `(Col1, Col2)` and the `Value` type would be `(Col0,)`. For `RelIndexReadAll`, the `Key` and `Value` types are references to tuples, or tuples of references (see `TupleOfBorrowed`). The same is true for the `Value` type of `RelIndexRead`.  
+
+`rel_ind_common!` provides sharing of data between indices of a relation. Types returned by this macro invocation only need to implement `RelIndexMerge`.
+
+
+To support parallel Ascent, relation indices need to implement parallel versions of the above traits. For example, `CRelIndexRead`, `CRelIndexReadAll`, and `CRelIndexWrite`.
+
+**TODO**: finish the guide

README.MD

@@ -5,6 +5,8 @@ Ascent is a logic programming language (similar to Datalog) embedded in Rust via
 
 For more information, check out [the CC paper](https://s-arash.github.io/ascent/cc22main-p95-seamless-deductive-inference-via-macros.pdf) on Ascent.
 
+In addition, [the OOPSLA paper](https://dl.acm.org/doi/pdf/10.1145/3622840) describes the "Bring Your Own Data Structures to Datalog" aspect of Ascent.
+
 ## Examples
 
 ### Computing all the connected nodes in a graph
@@ -147,6 +149,24 @@ In the above example, we initialize the relation `r` directly to shorten the pro
 It may be useful to define macros that expand to either body items or head items. Ascent allows you to do this.
 
 You can find more about macros in Ascent macros [here](MACROS.MD).
+
+### BYODS
+BYODS (short for Bring Your Own Data Structures to Datalog) is an extension of Ascent that enables relations to be backed by custom data structures. This feature allows improving the algorithmic complexity of Ascent programs by optimizing the data structures used to back relations. For example, a program that requires transitive relation computation of a large graph could improve its performance by choosing a union-find based data structure for the transitive closure relation:
+
+```Rust
+ascent! {
+   #[ds(trrel_uf)]
+   relation path(Node, Node);
+
+   path(x, y) <-- edge(x, y);
+}
+```
+
+The `#[ds(trrel_uf)]` attibute directs the Ascent compiler to use the data structure provider defined in the module `trrrel_uf` for the `path` relation. See [BYODS.MD](BYODS.MD) for more information on BYODS.
+
+(Note: custom data structure providers like `trrel_uf` have not been merged into the master branch yet.)
+
+
 ### Misc
 - **`#![measure_rule_times]`** causes execution times of individual rules to be measured. Example: 
    ```Rust

ascent/Cargo.toml

@@ -32,6 +32,7 @@ boxcar = "0.1.0"
 sync-unsafe-cell = "0.1.0"
 
 once_cell = "1.13.1"
+paste = "1.0"
 
 [dev_dependencies]
 

ascent/src/c_lat_index.rs

@@ -6,7 +6,7 @@ use std::collections::HashSet;
 use std::hash::{Hash, BuildHasherDefault};
 
 use crate::c_rel_index::{shards_count, DashMapViewParIter};
-use crate::internal::{RelIndexWrite, CRelIndexWrite};
+use crate::internal::{RelIndexWrite, CRelIndexWrite, RelIndexMerge, Freezable};
 use crate::rel_index_read::{RelIndexRead, RelIndexReadAll, CRelIndexRead, CRelIndexReadAll};
 
 type SetType<T> = HashSet<T>;
@@ -15,20 +15,23 @@ pub enum CLatIndex<K, V> {
    Frozen(dashmap::ReadOnlyView<K, SetType<V>, BuildHasherDefault<FxHasher>>)
 }
 
-impl<K: Clone + Hash + Eq, V: Clone + Hash + Eq> CLatIndex<K, V> {
-   pub fn freeze(&mut self) {
+impl<K: Clone + Hash + Eq, V: Clone + Hash + Eq> Freezable for CLatIndex<K, V> {
+   fn freeze(&mut self) {
       update(self, |_self| match _self {
          CLatIndex::Unfrozen(dm) => Self::Frozen(dm.into_read_only()),
          CLatIndex::Frozen(_) => _self,
       })
    }
 
-   pub fn unfreeze(&mut self) {
+   fn unfreeze(&mut self) {
       update(self, |_self| match _self {
          CLatIndex::Frozen(v) => Self::Unfrozen(v.into_inner()),
          CLatIndex::Unfrozen(_) => _self,
       })
    }
+}
+
+impl<K: Clone + Hash + Eq, V: Clone + Hash + Eq> CLatIndex<K, V> {
 
    #[inline]
    pub fn unwrap_frozen(&self) -> &dashmap::ReadOnlyView<K, SetType<V>, BuildHasherDefault<FxHasher>> {
@@ -122,6 +125,22 @@ impl<'a, K: 'a + Clone + Hash + Eq + Send + Sync, V: 'a + Clone + Hash + Eq + Se
    type Key = K;
    type Value = V;
 
+   fn index_insert(&mut self, key: Self::Key, value: Self::Value) {
+      let dm = self.unwrap_mut_unfrozen();
+      // let shard = dm.determine_map(&key);
+      // let entry = dm.shards_mut()[shard].get_mut().entry(key).or_insert(SharedValue::new(Default::default()));
+      // entry.get_mut().push(value);
+
+      let hash = dm.hash_usize(&key);
+      let shard = dm.determine_shard(hash);
+      let entry = dm.shards_mut()[shard].get_mut().raw_entry_mut()
+         .from_key_hashed_nocheck(hash as u64, &key)
+         .or_insert(key, SharedValue::new(Default::default()));
+      entry.1.get_mut().insert(value);
+   }
+}
+
+impl<'a, K: 'a + Clone + Hash + Eq + Send + Sync, V: 'a + Clone + Hash + Eq + Send + Sync> RelIndexMerge for CLatIndex<K, V> {
    fn move_index_contents(from: &mut Self, to: &mut Self) {
       let before = Instant::now();
       let from = from.unwrap_mut_unfrozen();
@@ -159,24 +178,10 @@ impl<'a, K: 'a + Clone + Hash + Eq + Send + Sync, V: 'a + Clone + Hash + Eq + Se
       }
 
    }
-
-   fn index_insert(ind: &mut Self, key: Self::Key, value: Self::Value) {
-      let dm = ind.unwrap_mut_unfrozen();
-      // let shard = dm.determine_map(&key);
-      // let entry = dm.shards_mut()[shard].get_mut().entry(key).or_insert(SharedValue::new(Default::default()));
-      // entry.get_mut().push(value);
-
-      let hash = dm.hash_usize(&key);
-      let shard = dm.determine_shard(hash);
-      let entry = dm.shards_mut()[shard].get_mut().raw_entry_mut()
-         .from_key_hashed_nocheck(hash as u64, &key)
-         .or_insert(key, SharedValue::new(Default::default()));
-      entry.1.get_mut().insert(value);
-   }
 }
 
 impl<'a, K: 'a + Clone + Hash + Eq, V: 'a + Clone + Hash + Eq> RelIndexReadAll<'a> for CLatIndex<K, V> {
-   type Key = K;
+   type Key = &'a K;
    type Value = V;
 
    type ValueIteratorType = std::iter::Cloned<std::collections::hash_set::Iter<'a, V>>;
@@ -190,7 +195,7 @@ impl<'a, K: 'a + Clone + Hash + Eq, V: 'a + Clone + Hash + Eq> RelIndexReadAll<'
 }
 
 impl<'a, K: 'a + Clone + Hash + Eq + Sync + Send, V: 'a + Clone + Hash + Eq + Sync + Send> CRelIndexReadAll<'a> for CLatIndex<K, V> {
-   type Key = K;
+   type Key = &'a K;
    type Value = &'a V;
 
    type ValueIteratorType = rayon::collections::hash_set::Iter<'a, V>;
@@ -212,9 +217,9 @@ impl<'a, K: 'a + Clone + Hash + Eq, V: 'a + Clone + Hash + Eq> CRelIndexWrite fo
    type Value = V;
 
    #[inline(always)]
-   fn index_insert(ind: & Self, key: Self::Key, value: Self::Value) {
+   fn index_insert(&self, key: Self::Key, value: Self::Value) {
       // let before = Instant::now();
-      ind.insert(key, value);
+      self.insert(key, value);
       // unsafe {
       //    crate::internal::INDEX_INSERT_TOTAL_TIME += before.elapsed();
       // }

ascent/src/c_rel_full_index.rs

@@ -5,7 +5,7 @@ use rustc_hash::FxHasher;
 use std::hash::{Hash, BuildHasherDefault};
 
 use crate::c_rel_index::{DashMapViewParIter, shards_count};
-use crate::internal::{RelIndexWrite, CRelIndexWrite, RelFullIndexRead, RelFullIndexWrite, CRelFullIndexWrite};
+use crate::internal::{RelIndexWrite, CRelIndexWrite, RelFullIndexRead, RelFullIndexWrite, CRelFullIndexWrite, RelIndexMerge, Freezable};
 use crate::rel_index_read::{RelIndexRead, RelIndexReadAll, CRelIndexRead, CRelIndexReadAll};
 
 
@@ -14,20 +14,31 @@ pub enum CRelFullIndex<K, V> {
    Frozen(dashmap::ReadOnlyView<K, V, BuildHasherDefault<FxHasher>>)
 }
 
-impl<K: Clone + Hash + Eq, V> CRelFullIndex<K, V> {
-   pub fn freeze(&mut self) {
+impl<K: Clone + Hash + Eq, V> Freezable for CRelFullIndex<K, V> {
+   fn freeze(&mut self) {
       update(self, |_self| match _self {
          CRelFullIndex::Unfrozen(dm) => Self::Frozen(dm.into_read_only()),
          CRelFullIndex::Frozen(_) => _self,
       })
    }
 
-   pub fn unfreeze(&mut self) {
+   fn unfreeze(&mut self) {
       update(self, |_self| match _self {
          CRelFullIndex::Frozen(v) => Self::Unfrozen(v.into_inner()),
          CRelFullIndex::Unfrozen(dm) => CRelFullIndex::Unfrozen(dm),
       })
    }
+}
+
+impl<K: Clone + Hash + Eq, V> CRelFullIndex<K, V> {
+
+
+   pub fn exact_len(&self) -> usize {
+      match self {
+        CRelFullIndex::Unfrozen(uf) => uf.len(),
+         CRelFullIndex::Frozen(f) => f.len(),
+      }
+   }
 
    #[inline]
    pub fn unwrap_frozen(&self) -> &dashmap::ReadOnlyView<K, V, BuildHasherDefault<FxHasher>> {
@@ -135,7 +146,7 @@ impl<'a, K: 'a + Clone + Hash + Eq, V: 'a + Sync> CRelIndexRead<'a> for CRelFull
 
 }
 
-impl<'a, K: 'a + Clone + Hash + Eq, V: 'a> RelFullIndexRead for CRelFullIndex<K, V> {
+impl<'a, K: 'a + Clone + Hash + Eq, V: 'a> RelFullIndexRead<'a> for CRelFullIndex<K, V> {
    type Key = K;
 
    #[inline(always)]
@@ -186,7 +197,7 @@ impl<'a, K: 'a + Clone + Hash + Eq, V: 'a> CRelFullIndexWrite for CRelFullIndex<
 }
 
 impl<'a, K: 'a + Clone + Hash + Eq, V: 'a + Clone> RelIndexReadAll<'a> for CRelFullIndex<K, V> {
-   type Key = K;
+   type Key = &'a K;
    type Value = V;
 
    type ValueIteratorType = std::iter::Once<V>;
@@ -199,7 +210,7 @@ impl<'a, K: 'a + Clone + Hash + Eq, V: 'a + Clone> RelIndexReadAll<'a> for CRelF
 }
 
 impl<'a, K: 'a + Clone + Hash + Eq + Send + Sync, V: 'a + Clone + Send + Sync> CRelIndexReadAll<'a> for CRelFullIndex<K, V> {
-   type Key = K;
+   type Key = &'a K;
    type Value = &'a V;
 
    type ValueIteratorType = rayon::iter::Once<&'a V>;
@@ -221,6 +232,21 @@ impl<'a, K: 'a + Clone + Hash + Eq + Send + Sync, V: 'a + Send + Sync> RelIndexW
    type Key = K;
    type Value = V;
 
+   fn index_insert(&mut self, key: Self::Key, value: Self::Value) {
+      let dm = self.unwrap_mut_unfrozen();
+
+      // let shard = dm.determine_map(&key);
+      // dm.shards_mut()[shard].get_mut().insert(key, SharedValue::new(value));
+
+      let hash = dm.hash_usize(&key);
+      let shard = dm.determine_shard(hash);
+      dm.shards_mut()[shard].get_mut().raw_entry_mut()
+         .from_key_hashed_nocheck(hash as u64, &key)
+         .insert(key, SharedValue::new(value));
+   }
+}
+
+impl<'a, K: 'a + Clone + Hash + Eq + Send + Sync, V: 'a + Send + Sync> RelIndexMerge for CRelFullIndex<K, V> {
    fn move_index_contents(from: &mut Self, to: &mut Self) {
       let before = Instant::now();
 
@@ -247,29 +273,16 @@ impl<'a, K: 'a + Clone + Hash + Eq + Send + Sync, V: 'a + Send + Sync> RelIndexW
       }
 
    }
-
-   fn index_insert(ind: &mut Self, key: Self::Key, value: Self::Value) {
-      let dm = ind.unwrap_mut_unfrozen();
-
-      // let shard = dm.determine_map(&key);
-      // dm.shards_mut()[shard].get_mut().insert(key, SharedValue::new(value));
-
-      let hash = dm.hash_usize(&key);
-      let shard = dm.determine_shard(hash);
-      dm.shards_mut()[shard].get_mut().raw_entry_mut()
-         .from_key_hashed_nocheck(hash as u64, &key)
-         .insert(key, SharedValue::new(value));
-   }
 }
 
 impl<'a, K: 'a + Clone + Hash + Eq, V: 'a> CRelIndexWrite for CRelFullIndex<K, V> {
    type Key = K;
    type Value = V;
 
    #[inline(always)]
-   fn index_insert(ind: &Self, key: Self::Key, value: Self::Value) {
+   fn index_insert(&self, key: Self::Key, value: Self::Value) {
       // let before = Instant::now();
-      ind.insert(key, value);
+      self.insert(key, value);
       // unsafe {
       //    crate::internal::INDEX_INSERT_TOTAL_TIME += before.elapsed();
       // }