Improve interaction with Unquote

[cloudRoutine]

FsCheck and Unquote are by far my favorite testing libraries and it would be great if it were easier to use the two of them together.

It seems like it'd be possible to pass the args returned by a failing case to the property inside a quotation and return the unquoted evaluation as part of the failure message.

An ?unquote:bool optional parameter could be added to Check's methods, or a new UnquoteCheck class could be added with the same methods as Check

[kurtschelfthout]

Great idea. This would probably work even better with the automatic quoting of arguments introduced in F# 4.

On 9 Aug 2015, at 06:25, Jared Hester [email protected] wrote:

FsCheck and Unquote are by far my favorite testing libraries and it would be great if it were easier to use the two of them together.

It seems like it'd be possible to pass the args returned by a failing case to the property inside a quotation and return the unquoted evaluation as part of the failure message.

An ?unquote:bool optional parameter could be added to Check's methods, or a new UnquoteCheck class could be added with the same methods as Check

—
Reply to this email directly or view it on GitHub.

[ploeh]

The combination of FsCheck and Unquote is fantastic! I use it all the time.

What do you need that isn't already possible?

Here's a test that combines the two:

[<Property>]
let ``run returns StoppedState when it occurs`` (states : State list) =
    states |> List.exists isStopped ==> lazy

    let actual : State = run states

    let expected = states |> List.find isStopped
    expected =! actual

This test uses the FsCheck.Xunit Glue Library, and Unquote, which you can see in action because of the =! operator.

If I change the function being tested so that the test fails, I get output like this when I run the property:

Test 'Ploeh.Samples.PollingConsumerProperties.run returns StoppedState when it occurs' failed: 
Falsifiable, after 2 tests (18 shrinks) (StdGen (2140451515,296042823)):
[StoppedState []; ReadyState {Started = 08.01.2069 00:00:00 +00:00;
                              Stopped = 04.07.1977 00:00:00 +00:00;
                              Result = [];}]

---- 

StoppedState [] = ReadyState {Started = 08.01.2069 00:00:00 +00:00;
            Stopped = 04.07.1977 00:00:00 +00:00;
            Result = [];}
false


    ----- Inner Stack Trace -----
    c:\Users\Mark\Documents\Pluralsight\Type-Driven Development with FSharp\Src\PollingConsumer\PollingConsumer.Tests\PollingConsumerProperties.fs(130,0): at Ploeh.Samples.PollingConsumerProperties.run returns StoppedState when it [email protected](Unit unitVar)
    at System.Lazy`1.CreateValue()
    at System.Lazy`1.LazyInitValue()
    at System.Lazy`1.get_Value()
    at FsCheck.Testable.Prop.safeForce[a](Lazy`1 body)

(This particular test is taken from my Type-Driven Development with F# Pluralsight course, which contains more than an hour of footage showcasing the combination of FsCheck and Unquote.)

What more do you need?

I'd consider it a poor idea to do anything in FsCheck that explicitly couples it to any other library (even one as excellent as Unquote), but if it's possible to do it in a way where the FsCheck library (think: the NuGet package) is still independent of Unquote, it'd be fine. Adding a hypothetical FsCheck.Unquote Glue Library wouldn't hurt, then, but I wonder what it would add...

[kurtschelfthout]

Wouldn't you somehow want the whole property (i.e. in your example, everything inside your outer let) passed into Unquote so it could analyze as a whole?

@ploeh why would you dislike taking a dependency on Unquote - do you feel like it's shoving it down the user's throat?

Must admit I have a soft spot for this, before Unquote existed I started on something similar on my own but never got nearly as far as Unquote.

[ploeh]

If you add Unquote as a dependency to FsCheck, kittens will die 🙀

With a high-quality library, this can be a difficult point to make, but consider this:

• Why is NUnit integration a Glue Library for FsCheck?
• Why is xUnit.net integration a Glue Library for FsCheck?

Why not add both NUnit and xUnit to FsCheck itself?

I hope the answer to this question is obvious.

Some users of FsCheck may not want Unquote. What if people for some unconceivable reason want to use something else?

What if development of Unquote stops? This may seem like a moot point at the moment, because with its minimal API, Unquote already feels quite done. Still, things may happen in the future that requires further development. As an example, does Unquote work on .NET Core? Who knows? It's not done yet.

This is only an example of the sort of trouble taking a dependency can get you into.

What if someone wants to use FsCheck, but together with some other library that indirectly depends on Unquote? This could lead to dependency hell if the version of Unquote that library depends on is incompatible with the version of Unquote FsCheck depends on.

As I commented above, I wouldn't mind adding a new FsCheck.Unquote Glue Library, but I think it would be a grave mistake adding Unquote (or any other non-essential dependency) to FsCheck itself. It's basic library dependency hygiene.

That said, I also don't understand what this particular suggestion adds that not already possible, but that just me being dim, I suppose.

[kurtschelfthout]

Yes, point(s) taken. I don't think a separate glue library is really an option either though, because now you get into combinatorial dependency hell, e.g. I want to use FsCheck with both xunit and Unquote, so now I need to install FsCheck.Xunit and FsCheck.Unquote. And maybe some of the glue libraries have integration amongst themselves, so then should we create FsCheck.Xunit.Unquote? Madness.

Incidentally, Unquote itself afaik detects dynamically whether xunit et al are loaded and then uses the appropriate assert. I.e. without actually taking a static dependency. Sounds pretty brittle though.

I've also thought a bit more about using the auto-quoting feature in F# 4, but it can only be used on methods, so presumably an overload method on the Check type could be added that captures the quoted property. Not sure that would help much though, but something to consider if there is a use case.

Also another suggestion in this vein I've heard before is to make the PropertyAttribute for xunit and NUnit inherit from ReflectedDefinition, but this is not possible because they already need to inherit from xUnit or NUnit-specific attributes.

So with all that I am now inclined to close this as "nothing much to do", unless @cloudRoutine would like to provide more specific use cases? Again on the whole I really like Unquote and would be happy to make changes to make it easier to use with FsCheck. I'm just having trouble finding something to do :)

[ploeh]

e.g. I want to use FsCheck with both xunit and Unquote, so now I need to install FsCheck.Xunit and FsCheck.Unquote

That precisely describes the situation we have with AutoFixture. You can for example combine AutoFixture.Xunit with AutoFixture.AutoMoq. This works extremely well; I've never had any problems with this. AutoFixture offers a lot of packages you can choose and combine as you like it: https://github.com/AutoFixture/AutoFixture#downloads

Some of those packages are more than four years old. My experience with this design has been overwhelmingly positive.

[ploeh]

Sorry if I'm being a pest about this, but it's only because I like FsCheck so much that I care so much 😳

[cloudRoutine]

I've been trying out some of Unquote's operators like @ploeh suggested, but the results are mixed.

e.g. -

let complex (m: int) (n: int) =
  let res = n + m
  (res >=! m)    |@ "result > #1" .&.
  (res >=! n)    |@ "result > #2" .&. 
  (res <! m + n) |@ "result not sum"
----------------------------
> Check.Quick complex;;

...

Test failed:
-59 < -59
false

Test failed:
-2 >= 52
false

Test failed:
-2 < -2
false

Ok, passed 100 tests.

I'm really not sure how to approach this but I'll take a shot in the dark.

It might be possible to do it as a kind of glue library/ plugin if FsCheck was tweaked a bit. Instead of having Check overloads with an Unquote option like a suggested before, perhaps there could be overloads or an addition type (CheckOut 😸 ) that function as a plug. Instead of returning unit on failure it returns a data structure containing the shrunk arguments from TestResult cast to their proper types. Then if you had a reflected definition of the property you could Expr.TryGetReflectedDefinition and maybe substitute the failure values for variables inside the quotation and then unquote that expression?

It might be "useful" in general to have FsCheck output data structures containing the values involved in its tests (generated arguments, labels, classifications, collected data values, observations, etc.) so that it could function as a part of a pipeline, sending stuff into a logging system, a charting/visualization system, a fitness function for a genetic algorithm, storing results of complex test with a low probability of failure in a database like Hypothesis , or whatever other esoteric uses people conjure up.

[kurtschelfthout]

Interesting, not sure what happened there. It occurs to me I probably haven't actually used vanilla FsCheck with just Unquote (at least not that I can remember), so it looks like something in the interaction between FsCheck and Unquote gets confused. I am definitely interested in an exploration of what is going wrong there.

I also like your idea of having Check overloads that return the raw test results. Although most of this is probably already possible using implementations of IRunner, but it all feels a bit clunky.

[moodmosaic]

I also see FsCheck as the main/core library exposing the basic building blocks for other Glue Libraries (e.g. FsCheck.Xunit, a hypothetical FsCheck.Unquote, and so on).

In fact, it's awesome that xUnit.net¹, FsCheck, and Unquote can all work autonomously:

• xUnit.net as test-discovery mechanism
• Unquote for assertions
• FsCheck for generating test cases for properties

Does FsCheck needs Unquote to work? – Not really, as far as I can tell...

¹And also NUnit

[kurtschelfthout]

I can't reproduce what you are getting with the property there. When I run it as is, the output is:

Falsifiable, after 1 test (3 shrinks) (StdGen (1864336695,296051884)):
Original:
-1
1
Shrunk:
0
0
with exception:
Swensen.Unquote.AssertionFailedException: Test failed:

0 < 0
false

   at Examples.complexUq(Int32 m, Int32 n) in <snip> Examples.fs:line 18 <snip>

Also, the operators are applied after quoting, in other words, Unquote doesn't see the quoted arguments. It's easy to improve things at the expense of some speed though:

let complexUq (m: int) (n: int) =
  let res = n + m
  (res >=! m)    |@ "result > #1" .&.
  (res >=! n)    |@ "result > #2" .&. 
  test <@ (res < m + n) @> |@ "result not sum"

Check.Quick complexUq

And now you get the reduction that makes Unquote so useful (although a bit silly in this example):

0 < 0 + 0
0 < 0
false

So that all plays pretty well with FsCheck. If you wanted something like being able to see the whole body of complexUq reduced by Unquote, I think that would be somewhat unreasonably hard to do for Unquote, because it would have to know about the custom operators FsCheck defines (besides, Unquote's test expects an argument that returns a boolean, while FsCheck returns a Property). I'm also not sure if it would buy you all that much.

[kurtschelfthout]

I am going to close this; note that the aspect of having FsCheck output data structures as a result of test (effectively, re-think the way the runner works) is captured in the FsCheck 3.0 issue #198.

I'm unclear if this contains anything else that is unresolved. Upon re-reading, I couldn't remember why I kept it open, seems to me like the interaction with Unquote is as expected.

Feel free to re-open and clarify if I missed anything.