Support for input validation with micro-batching

[fernandocamargoai]

I'm using JsonInput and I'm planning to validate the input using Pydantic. But the problem is, if I'm using micro-batching, I could end up refusing multiple requests because one of them contains some error. I think it'd be useful to have a callback where we could register a custom validator. Such callback would be called before the micro batching.

[parano]

Great question @fernandocamargoti!

Adding a validation callback seems like a really good approach, it can be a parameter to BaseHandler:

def input_validator(json):
      if  'required_filed' not in json:
          return False, "'requried_filed' missing in input"

class MyService(BentoService):

    @api(input=JsonInput(validation=input_validator), batch=True)
    def predict(json_lists):
           ...

and requests that failed the validation callback will not be sent to the user-defined API callback function similar to how we currently handle the default validation logic in each handler's implementation. cc @bojiang and @akainth015, you both have recently looked into this issue, thoughts?

btw here are some semi-related discussions #924

[parano]

Update: discussed with @bojiang earlier this week and he is now working on a refactoring that will clean up the input/output adapter code path a lot and allow users to do validation more easily. Here's the draft PR #961