📦 NEW: add filters support

[saqibameen]

Description

Add support for optional array of filters to narrow down the search results.

type FilterOperator = 'Eq' | 'NotEq' | 'In' | 'NotIn' | 'And' | 'Or';
type FilterConnective = 'And' | 'Or';
type FilterValue = string | string[];
type MemoryFilters = [FilterOperator | FilterConnective, FilterValue | MemoryFilters][];

Filters can be either:

• A single condition: ["field", "operator", "value"]
• A nested structure: ["And"|"Or", MemoryFilters]

Available operators: "Eq", "NotEq", "In", "NotIn", "And", "Or"

Testing

Basic Testing

1- Create a memory

const memory = await langbase.memory.create({ name: 'test-memory' })

2- Upload docs w metadata

Keep the content same for each document.

// Document 1
...
    meta: { author: 'AuthorA', category: 'Category1' },
...

// Document 1 with different name.  (i.e., content should be same)
...
    meta: { author: 'AuthorB', category: 'Category1' },
...

3- Retrieve:

 query: 'Relevant query',
  memory: [{ name: 'test-memory',  filters: ['Eq', ['author', 'AuthorA']] }],
,

This should correctly return the chunks from document 1.

Further Testing

1- Test a complex filter:

 ['And', [
      ['Eq', ['author', 'AuthorA']],
      ['Eq', ['category', 'Category1']],
    ]],

2- Test without setting any filter
3- Test with retrieving on two memories

  memory: [
  { name: 'test-memory',  filters: ['Eq', ['author', 'AuthorA']] }
  { name: 'test-memory2',  filters: ['Eq', ['author', 'AuthorB']] }
  ],

4- Test e2e RAG
5- Test UI (upload/retrieval test)

Docs

Docs: https://github.com/LangbaseInc/docs/pull/166
SDK: #78
API: https://github.com/LangbaseInc/langbase-api/pull/357

Deployment

1- Run test locally
2- Deploy workers
3- Run prod tests