Concrete examples are needed.

[Volkmire]

Disclaimer: i might be a potato.

Hi, @hassanhabib !

After reading a good chunk of the Standard, I sense that it has a lot of valuable knowledge, but also I have a feeling that maybe it's a little bit too... abstract? I caught myself being unsure on where would I put even the most common everyday concerns within the model.

I suspect the answers already are in some form in The Standard and my lizard brain is just unable to see or understand them, but I also have a feeling that concrete implementation of the abstraction would help a ton.

To be more specific, below I will provide examples of what I'm confused about, if someone could comment on how to use them within the Standard, it would help me a lot.

Thanks in advance. :)

Examples:

Area Of Confusion: Exceptions
Project: storage API in front of a database.
Scenario: Client tries to save an identity that already exists (e.g. violated some unique constraint, like a book's ISBN). Naturally, I want to return an error to the user with 409 (Conflict).
Ambiguity: In the section about foundation services, it is expected for service to throw an AlreadyExistsException, wraped in DependencyValidationException.
Does this mean that I'm to wrap exceptions like a Russian doll all the way to my controller?
If so, then my controller MUST KNOW the depth of a higher level logic (how many layers of inner exceptions it must traverse before finding that AlreadyExistsException).
Or am I supposed to unwrap the inner exception before rethrowing in higher level services? But what is AlreadyExistsException then? Part of my business logic? If not, Controller will be tightly coupled to foundation, because it MUST KNOW about what exceptions it produces to emit 409.
Side Note: Won't wrapping and re-throwing produce a metric ton of logs that would be equivalent to a stack trace from the 1st thrown exception?

Area Of Confusion: Reliability and Control-flow
Project: External Integration Service
Scenario: We must submit a StudentDto to some external service and it fails or external endpoint challenges us randomly to repeat a request with a computed signature.
Ambiguity: Say I do a request and get HttpRequestException (DNS failed to resolve a host or got a connection reset). Where do I handle it and retry it? I would assume that broker is supposed to handle it, because only it know about networking, but no flow control is allowed. Foundation service? It is not supposed to do anything, but map and validate. Processing service? Wouldn't that expose it to lower level implementation details that were supposed to be abstracted by FS and Broker?
Next level: I get a 500 error from the other side and naturally would like to retry the request. Same questions apply.
Boss level: I get a challenge from the server and need to supply an answer by computing some hash over the initial request model AND a response header. Where should I put this logic and how to I expose necessary details?
Final boss level: Same as boss level, but I am also redirected to another endpoint via 307 and the solution is expected to be retried for this URL, before I am able to call the 1st one.
Side Note: while last 2 cases may seem like I pulled them out of my ass, they are real, albeit much less frequent than first 2.

Area Of Confusion: Scalability
Project: ex-monolith, now distributed system - SchoolControl: student management service, student storage api, library card storage api.
Scenario: I am trying to add a new student and expect it to have a library card. I also expect that library cards exist only for existing students.
Ambiguity: Standard seems like a scalable model, as in you can apply it to a higher order objects (processes, micro services, etc). But it's hard to imagine how that logic would split when we split the app into multiple smaller apps. Where do we put the student validation logic? It sounds like it certainly must be in the student storage api, since it should probably be saving only valid students to DB, but then again, why should it know what a valid student is, when certainly the concept of valid student should be contained in the higher level logic (eg. I'm being told that students must only have mustaches by a school director, not by a storage provider).
Same goes to library card thing. How would a storage know if the student with this id exists? It can't reasonably enforce that constraint, so something else must. But if it can't, what's to prevent a direct call to that api that would make a bogus library card? Do I not have validation in the storage services at all? Do I trust my clients to create proper library cards and only store them?

[hassanhabib]

@Volkmire let me try to address some of these questions.

Area Of Confusion: Exceptions
Project: storage API in front of a database.
Scenario: Client tries to save an identity that already exists (e.g. violated some unique constraint, like a book's ISBN). Naturally, I want to return an error to the user with 409 (Conflict).
Ambiguity: In the section about foundation services, it is expected for service to throw an AlreadyExistsException, wraped in DependencyValidationException.
Does this mean that I'm to wrap exceptions like a Russian doll all the way to my controller?
If so, then my controller MUST KNOW the depth of a higher level logic (how many layers of inner exceptions it must traverse before finding that AlreadyExistsException).
Or am I supposed to unwrap the inner exception before rethrowing in higher level services? But what is AlreadyExistsException then? Part of my business logic? If not, Controller will be tightly coupled to foundation, because it MUST KNOW about what exceptions it produces to emit 409.
Side Note: Won't wrapping and re-throwing produce a metric ton of logs that would be equivalent to a stack trace from the 1st thrown exception?

When your foundation service throws an AlreadyExistsStudentException it is doing something called "Localization" for the exception. You are taking an external model DuplicateKeyException and localizing it to describe the exact problem that your system is dealing with.

If you decide to make one generic AlreadyExistsException you lose the context in your final error message that you want to communicate to your controller.

Now, with that being said, if you are building an orchestration level service or maybe even something higher like a coordination service. You don't really continue to wrap every exception coming from underneath or lower stream services with a new exception. You unwrap the categorization exception like StudentDependencyValidationException and keep the inner localized exception in this case it would be AlreadyExistStudentException and then wrap it again up with something like StudentProcessingDependencyValidationException or StudentOrchestrationDependencyValidationException.

The unwrapping of the existing categorization exception will help your controller remove the extra layer of categorization (if needed) and handle the specific exception you are working with.

So it's not really a Russian doll more like a Shawarma wrap :-) Read more here about unwraping and rewraping for localized exceptions.

There are situations then where your StudentCoordinationDependencyValidationException could have anything in it? this is where we said your API controller section will handle all of them like this:

    catch (StudentCoordinationDependencyValidationException studentCoordinationDependencyValidationException)
        (when studentValidationException.InnerException 
            is NotFoundStudentException
            or NotFoundLibraryCardException
            or NotFoundStudentContactException)
    {
        return NotFound(studentValidationException.InnerException)
    }

Now, the other point you made was about the controller being tied to your Foundation service. This is partially true. Your controller doesn't really know about your Foundation service contract, but it knows about the exception model coming from downstream services. controllers (just like brokers) are not fully abstract. They know a tiny bit about your lower dependency and it's immediate dependencies so they can communicate the right message.

Controllers will carry a bit of details from your local models and then convert them back into external model just how they started (being external). And that's okay because controllers aren't meant to be fully abstract, but rather disposable exposers to your business logic. They should know combine some knowledge about inner logic and outer protocols just like your Foundation services know a little about your external models and inner logic.

Now, if we rely only on the stack trace that was thrown from the original exception, we lose the ability to handle these exceptions in upstream services. Imagine a situation where you have a Coordination level service. This coordination service may need to apply some retry logic or any form of exception handling at it's level - in this case here your coordination service can't deal with just orchestration services (it's dependencies) exceptions - now it has to know about all the possible 9 Foundation services that live under a maximum of 3 orchestration services underneath. Although that the Coordination service only cares about just handling any dependency exception that may occur and retry with a different route/logic to resolve an issue.

I will try to address the rest of your questions as time allows. I hope this helps.

[Volkmire]

@hassanhabib , thanks for clarifications and commitment. :) When writing code, I'm trying to be meaningful in what I do, but the land of software design is very confusing for the inexperienced. So thanks for helping out. :)

Anyway, back the point.

Your answer clarified most of my concerns on that subject, but I still have some concerns left, if you wouldn't mind.

1. Abstraction. One can argue that having layered exceptions (Orchestration, Processing, etc) subtly breaks Dependency Inversion principle, because each higher level service now responsible for knowing what kind of concrete services it depends on (because you must know exceptions of which layer you are handling).

2. Quantity. There are a lot of exceptions, wrapping and unwrapping operations:
   Worst case scenario: 4 * (2 * n) + 3 * (o + c + m), where n - number of Foundation+Processing leafs, o - number of orchestration services, c - number of coordination services, m - # of management services. Even a simple app with 2 leafs and 1 orchestration will get us to 19 (!) exception wrapper classes.

3. Use of. I assume this one is related to simplicity of examples, but they never show any use of wrapper exceptions themselves outside of mapping to a higher level services exceptions.
   Now, you mentioned that you could use exception category (Dependency, DependencyValidation, Validation, Self) to trigger a, say, retries. But due to the aggregating nature of wrapper exceptions, you actually don't want to retry some operations. In the FS exceptions mapping, I can argue that I should retry on StudentDependencyException category. But! I also can argue, that I should retry when StudentDependencyValidationException is thrown, but only when it's LockedStudentException.
   This means that I have a desired behavior (retries), but categories aren't actually helping me ease the use of it.

While writing this, I had a think on how would one solve this issues and had a naive idea that most of them could be solved by marker interfaces. Reason being is that categories and wrappers don't seem to have innate value (at least I'm not yet onto it) outside of categorizing exception groups for recovery behavior. But behavior is better contained by interfaces, so why not have something like this:

public class StudentAlreadyExistsException : Exception, IRetriable { /*..*/ }

Then we can ditch all of the wrappers and rewraping and use pattern matching like so:

try { /*...*/ }
catch (Exception ex) when (ex is IRetriable) {
  // do retry logic
}

(although, I'm not sure if Polly would agree with this approach)

Now, I'm probably missing some other benefit that outweighs mentioned downsides (or maybe they are not downsides?), would be glad to hear your thoughts.

[hassanhabib]

Hi @Volkmire
These are great points. Let's see if I can answer some of them.

Abstraction. One can argue that having layered exceptions (Orchestration, Processing, etc) subtly breaks Dependency Inversion principle, because each higher level service now responsible for knowing what kind of concrete services it depends on (because you must know exceptions of which layer you are handling).

This is a strong point. you are saying that upstream services are now tightly coupled with concrete implementations for the exceptions from downstream services. However, because of the strict naming conventions for these dependencies, whatever is implementing IStudentProcessingService must throw a StudentProcessingDependencyException at all times. The naming convention here should mandate the type of exception that will be thrown from downstream dependencies.

If I decide today to change the implementation of some StudentProcessingService - I should treat the exceptions contract (which are categorical - Validation, Dependency and Service) the same way I treat the interface itself IStudentProcessingService. Anything that doesn't stay true to these contracts won't be able to plug into the system.

The trick here is in the localized exceptions. You have the freedom to change the inner exception into whatever fits your new implementation. because the inner exceptions are case-specific. things like InvalidStudentProcessingException or InactiveStudentProcessingException or whatever the case may be. This goes back to our earlier discussion around localized versus categorized exceptions.

The bottom line here, is that the categorical exceptions at every layer are just as contractual as the interface itself. whatever comes underneath that is your specific implementation that is subject to change at any time without breaking your upstream services except in very rare occasions.

You have to also consider that the code here is just a reflection of a thought process. If an orchestration service can call Foundations or Processings services, I can tell which dependency it's calling from the naming of the methods on these dependencies. For instance, if an orchestration service is calling UpsertStudentAsync I don't need to know anything else about the dependency because now I know automatically that it's a Processing service for the Student entity. It's like DNA, you take one part of the code and it tells you the whole story about where it came from and what it's targeting and in which part of the system it is.

Quantity. There are a lot of exceptions, wrapping and unwrapping operations:
Worst case scenario: 4 * (2 * n) + 3 * (o + c + m), where n - number of Foundation+Processing leafs, o - number of orchestration services, c - number of coordination services, m - # of management services. Even a simple app with 2 leafs and 1 orchestration will get us to 19 (!) exception wrapper classes.

That's true. you have to implement a lot of exceptions for each service. But it's necessary for three reasons:

• First: we treat every service as an independent node that is ready to be separated from the monolith into it's own infrastructure. Today, if I decided to take a StudentOrchestrationService and put it in an Azure Function or a Lambda all I have to do is to pluck out the Students folder in the models, and the Students folder under Services/Orchestrations and maybe the StudentsController. I always say: every service for itself meaning that these services are only aware of the contracts they require irregardless of who is implementing these contracts. This approach is valid whether you are building a monolith or a microservice because you are building both with a distributed system mindset.
• Second: I don't want the exceptions from downstream services to pass through another layer of services without being recognized, categorized and logged. If I just do something like catch (Exception exception) throw; like this - that would be a huge problem, because I'm basically opening the door for anything to go through my service without it being recognized. It could be showstopper that I need to react to or just simply an exception that needs to be unwrapped and wrapped back into a bigger category. Every service will log what kind of exceptions went through it and how we treated it for observation and monitoring services which is much, much cleaner and easier to query than the chaotic stack trace.

The unwrapping then wrapping process allows your controller to deal only with 3 types of exceptions in addition to the hybrid. Look at this example here:

• Third: More work == Less effort. how's that possible? well, If you put more work in development-time into how you handle your exceptions, you will spend much lesser effort trying to understand what happened to your system during debugging-time. This is very crucial especially in larger, enterprise-level systems. If you end up with some 120 entities in some monolithic system built according to The Standard. It will take any engineer regardless of their level of experience less than 10 mins to know exactly what happened, where it happened, why it happened and when it happened. In a clean, organized manner that allows you to draw graphs about the timeline of any request/event going through the system.

I had a couple of more thoughts but I need to step out. will get back to you on the rest later.

[hassanhabib]

Use of. I assume this one is related to simplicity of examples, but they never show any use of wrapper exceptions themselves outside of mapping to a higher level services exceptions.
Now, you mentioned that you could use exception category (Dependency, DependencyValidation, Validation, Self) to trigger a, say, retries. But due to the aggregating nature of wrapper exceptions, you actually don't want to retry some operations. In the FS exceptions mapping, I can argue that I should retry on StudentDependencyException category. But! I also can argue, that I should retry when StudentDependencyValidationException is thrown, but only when it's LockedStudentException.
This means that I have a desired behavior (retries), but categories aren't actually helping me ease the use of it.

@Volkmire good point.
Categorical exceptions help you operate on an error at the categorical level. You can choose to retry all exceptions at the categorical level of DependencyException or you could choose to operate on a more fine-grained level such as LockedException.

Categorical exceptions are just carriers of multiple localized more issue-focused exceptions. They make things easy if and only if you choose to operate at the categorical level. But they don't deny you access to information if you choose to operate on a more issue-based level.

More often than ever, you will end up retrying on a categorical level exception that an issue-focused localized exception. But that certainly isn't a measure nor a reason for driving this level of implementation.

Categorical exceptions also help Orchestration services aggregate common issues with common categories. Imagine if you have StudentValidationException and LibraryCardValidationException and CourseValidationException all incoming from dependency of some StudentOrchestrationException - I wouldn't need to worry about the 100 possible inner exceptions that each one of these categories may have. I can just aggregate and merge all of their errors together (in the case of a non-circuit breaking process for non-critical path) without having any issues at all.

It even makes it easier to handle all of this at the controller (exposure) layer because now your controller has to deal only with one exception instead of 20.

I shall come back to this thread again as time allows.

[Volkmire]

Categorical exceptions are just carriers of multiple localized more issue-focused exceptions.

I'm thinking DRY here. What would happen if I forgo creating 4 wrappers per type, but instead use 4 types in total? What do I lose?

Example. On a higher level (say Orchestration), if i want to retry any particular dependency call - I will be forced to wrap EXACTLY that call in a retry policy. Consider this:

class Orchestrator
{
  public OrchestrationResult Orchestrate()
  {
    return Do();
  }

  private OrchestrationResult Do()
  {
    var one = _studentService.Work();
    var two = _libraryService.Work();
    return new OrchestrationResult(one, two);
  }
}

If i apply a retry policy around the Do method call, regardless of the actual exception type thrown (that I want to retry on), I will make a retry a call of both dependencies:

  public OrchestrationResult Orchestrate()
  {
    return RetryOnException(Do);
  }

If, in this case, I would want to retry only on LibraryDependencyValidationException, I will be forced to call a _studentService again, leading to potentially unpredictable result (think adding a student once more and getting UniqueIndexViolationException or something like that).

If i want to retry any particular call, I need to wrap EXACTLY that call in a retry policy, meaning - I don't need to know what exception name being thrown, all i care is it's category/interface:

  private OrchestrationResult  Do()
  {
    var one = RetryOnException(_studentService.Work);
    var two = RetryOnException(_libraryService.Work);
    return new OrchestrationResult(one, two);
  }

I can't think of an example, where I would care that it's StudentSomethingException (accent on the name, not the category), but would not handle it around the _studentService call. Meaning, I can actually have 4 wrapper exceptions total (Self, Validation, DependencyValidation, Dependency) and achieve the same result. No?

If we consider TryCatch (for purposes of remapping), as mentioned, you can just decorate it. In fact, one can argue that this behavior can be generalized with some static generic method.

Am I wrong? Could you, perhaps, provide a counter example?

[hassanhabib]

Example. On a higher level (say Orchestration), if i want to retry any particular dependency call - I will be forced to wrap EXACTLY that call in a retry policy. Consider this:

@Volkmire what if you need to retry on a certain occasion of the dependency call?
What if you only want to retry on StudentDependencyException versus propagating the error if it's a StudentDependencyValidationException?

Your example here implies that I retry on all errors that come from a dependency.

[Volkmire]

@hassanhabib , not quite got you.

If I retry in an orchestration service, I don't know what dependencies does my _studentService have, because any named exception thrown from dependency of _studentService would be unwrapped and inner exception would be re-wrapped to a StudentService<Category>Exception.

Or do you mean a case, where I have _student AND _library calls, but I only need to retry on _student AND I wrap around both calls in a retry policy that only tries to find a student? Why won't I just wrap _student service call in a retry then? Anything above Orchestration service won't know what underlying service caused an exception anyway, due to rewrapping (and it shouldn't), so I can handle them only when I already know what calls I make and which of them I need to retry.
Like this:

private OrchestrationResult  Do()
{
  var one = RetryOnException(_studentService.Work);
  var two = _libraryService.Work();
  return new OrchestrationResult(one, two);
}

Just in case, here's an example of RetryOnException method for a particular call, if that caused a misunderstanding:

private Student RetryOnException(Func<Student> f, int retryCount = 3)
{
  try
  {
     return f();
  }
  catch(StudentDepencyValidationException ex)
  {
     return retryCount > 0 ? RetryOnException(f, --retryCount) : throw ex;
  }  
}

[hassanhabib]

Or do you mean a case, where I have _student AND _library calls, but I only need to retry on _student AND I wrap around both calls in a retry policy that only tries to find a student? Why won't I just wrap _student service call in a retry then?

Because you won't need to retry with every exception coming from the studentService you will need to retry based on the scenario and the criticality of the data flow. Especially in online scenarios where the user is waiting on the other side.

I've seen a pattern in our industry that is very binary-influenced in terms of work flow. You will see this a lot, a request either fails completely or succeeds completely - but there's nothing in the middle.

And then some folks proposed the eventual consistency pattern, and I thought to myself, omg that's it! We are going to partialize our requests and deliver a report of errors and help end-users to make the next move. But then it just turned out to be (for most of the implementations I've seen except for mine) to be all or none again.

I want you to think a little about hybrid situations, where a student registration process have some critical paths, they may require a retry for twice or thrice, and a less critical path that is still needed in a registration process where say a student library card couldn't be processed for whatever reason.

Imagine this visualization here for a second:

This visualization here tells you two online processes with different priority levels have been processed through some orchestration service. There's this misconception in the engineering community that a every exception is a circuit-breaker. That's not true. Some exceptions are less of a circuit breaker than others. Although that are all forms of flow control.

I wouldn't personally implement the retry logic the way you did. It's too complicated for a rubber duckie to understand that flow. There's a different simpler way we can do retries which I plan to detail in The Standard soon.

I will come back to you soon with more insights and answers.

[Volkmire]

I wouldn't personally implement the retry logic the way you did. It's too complicated for a rubber duckie to understand that flow.

Consider the implementations a pseudo-code. It's just a general representation of what is happening and where it is happening.

There's a different simpler way we can do retries which I plan to detail in The Standard soon.

Is there, perhaps, an example of The Standard implementation with all the bells and whistles?
A minimal enough app that is able to illustrate all the concepts and their usefulness? I've tried to check some of your other repos, notably OtripleS, Cofee and some others, but never found anything beyond copying the project structure, message mapping and rewraping (even though wrappers themselves are never used for anything else).

[hassanhabib]

Is there, perhaps, an example of The Standard implementation with all the bells and whistles?
A minimal enough app that is able to illustrate all the concepts and their usefulness? I've tried to check some of your other repos, notably OtripleS, Cofee and some others, but never found anything beyond copying the project structure, message mapping and rewraping (even though wrappers themselves are never used for anything else).

I will try to put something together as soon as possible. might take a bit of time because I'm trying to make sure that any public representation of these practices is properly documented because it's not just you and me on the internet haha.

[Volkmire]

Sure thing. To be honest, considering the amount of content you are pumping out, I'm a little afraid for your sanity. :>