Proposal: Make neil command automatically run clojure -X:alias or clojure -M:alias

[didibus]

Proposal

I was thinking it could be nice if:

neil command

Would be smart enough to look at the deps.edn, lookup an alias for command, and figure out if it needs to be called with -X or -M or -T, and than basically run it with either:

clojure -X:command
or
clojure -M:command
or
clojure -T:command

If command clashed with neil's own commands, I would put a message like:

conflicts with neil command and your deps.edn command, defaulting to running neil's command, if you want to run deps's command explicitly call: neil clojure command

So for example, one could do:

deps.edn

{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}}
 :aliases
 {:run {:main-opts ["-m" "myapp.myapp"]}
  :run-x {:ns-default myapp.myapp
          :exec-fn greet
          :exec-args {:name "Clojure"}}
  :build {:deps {io.github.clojure/tools.build {:git/tag "v0.8.3" :git/sha "0d20256"}}
          :ns-default build
          :neil/run-as :tool}
  :test {:extra-paths ["test"]
         :extra-deps {org.clojure/test.check {:mvn/version "1.1.1"}
                      io.github.cognitect-labs/test-runner
                      {:git/tag "v0.5.1" :git/sha "dfb30dd"}}
         :main-opts ["-m" "cognitect.test-runner"]
         :exec-fn cognitect.test-runner.api/test
         :neil/run-as :main}}}

And now from command line:
$ neil run - runs clojure -M:run
$ neil run-x - runs clojure -X:run-x
$ neil build uber - runs clojure -T:build uber
$ neil test - runs clojure -M:test

When it is ambiguous what the run-as should be, such as here:

:build {:deps {io.github.clojure/tools.build {:git/tag "v0.8.3" :git/sha "0d20256"}}
          :ns-default build
          :neil/run-as :tool}

I think neil could prompt:

1. Run as Tool
2. Run as Main
3. Run as Exec

Or if the alias contains a :neil/run-as than that is used instead.

Maybe we can also have a neil command to set the :neil/run-as or after it prompts the first time we can have it set it automatically for future invocation.

This would basically allow any alias to become a neil command.

Alternative

Alternatively, we can put this behavior behind a command, so maybe:

neil run alias

It would behave the same, but you'd have to do:
$ neil run run - runs clojure -M:run
$ neil run run-x - runs clojure -X:run-x
$ neil run build uber - runs clojure -T:build uber
$ neil run test - runs clojure -M:test

And maybe in this case we could also have an explicit run-as command:
$ neil run-as main run - runs clojure -M:run
$ neil run-as exec run-x - runs clojure -X:run-x
$ neil run-as tool build uber - runs clojure -T:build uber
$ neil run-as main test - runs clojure -M:test

[didibus]

It could also be cool to add a run-as option of exec-cli which wraps an exec using: https://github.com/babashka/cli so you can then pass it normal CLI args, even though the alias is an exec-fn.

[borkdude]

I'm not sure if we should re-invent the clojure CLI, but I think it would be powerful if neil offered some short-hands for things you can install with neil. E.g.:

neil add test

and then

neil test :only foo.bar

would translate into

clojure -M:test -n foo.bar

And similar for neil uberjar -> clojure -T:build uber

[borkdude]

I agree that neil -x ... could be a babashka.cli invocation of clojure -X .... What about aliases?

neil --aliases foo bar -x ...

? Or just neil -x:foo:bar (this would not work very well with babashka CLI right now) ?

[borkdude]

Implemented neil test now in 0.1.41.

[rads]

@borkdude @didibus: The neil test command makes sense to me because it's a higher-level abstraction than what tools.deps provides. Also, something like bb -x makes sense to me because it's not a wrapper around tools.deps.

When we add things like neil -x, neil run-as tool build uber, etc..., these are basically just another syntax for the clj CLI, similar but potentially different enough to cause confusion. In my opinion, we should focus on providing higher-level abstractions on top of the API rather than providing syntax sugar for clj -X/-T/-M, and encourage people to use clj for running aliases.

Consider someone who is learning Clojure today. They can't avoid the clj tool, but they also now need to be aware of projects that use neil (and likely lein, but that's a different issue) which will have a different way of invoking the same commands compared to clj. I think they should just use clj instead and neil when they don't want to think about what clj command to run or alias to add.

[borkdude]

I agree. So something like neil test makes sense, maybe also neil uberjar (assuming neil add build was ran, similar to lein uberjar).

[rads]

Another suggestion: neil could support running Babashka tasks in bb.edn like bb test or bb uberjar. Then it's the job of bb.edn to figure out what options to pass into the clj command. That way the user could still rely on things like neil test or neil uberjar but it would ultimately be under their control.

To expand on this, maybe neil test would provide the current default behavior, but could be overridden as a bb task if needed. Just an idea though, need to think about this some more.

[borkdude]

I think typing neil test vs bb test doesn't have much benefit.

Btw, I do see a benefit in having arguments run through babashka.cli first before they are passed to the clojure CLI, since the clojure CLI demands you to quote things as EDN first which is problematic for some people (on Windows, but also when you use $(...) in the middle of an argument).

E.g. neil tool install --git/url .. git/tag .. --as foo could work instead of clj -Ttool install '{:git/url "... "}' :as foo which is very hard to figure out in Powershell for example.

[rads]

@borkdude:

I think typing neil test vs bb test doesn't have much benefit.

Yeah, I think you're right. It even takes more characters! So with that in mind, my stance is that people should run their tasks directly with bb and clj when possible. That way we encourage people to learn the base tools and not hide them with neil.

Btw, I do see a benefit in having arguments run through babashka.cli first before they are passed to the clojure CLI, since the clojure CLI demands you to quote things as EDN first which is problematic for some people (on Windows, but also when you use $(...) in the middle of an argument).

That makes sense. I think if there was one command that did this for all possible invocations of clj/clojure, basically a "transpiler" from the babashka.cli to Clojure CLI, that would be simpler than trying to create specific commands for generic operations like run-as. That said, even if we don't end up doing that (which would be fine with me), adding commands for specific aliases like neil tool seems like a reasonable compromise.

[borkdude]

I think it would even make more sense if we supported:

clojure -Ttools install io.github.seancorfield/deps-new '{:git/tag "v0.4.13"}' :as new

vs

neil tool install io.github.seancorfield/deps-new :as new

and neil picked the most recent version by default. So, not only transpiling, but also offering some convenience.

[borkdude]

and once that's supported, perhaps:

neil run new app --foo 1 --bar 2

where run is short for clojure -Tnew or something.

So: not only a simple transpilation, but convenience which justifies this feature.

[rads]

Another idea:

# Equivalent commands
# Note: `neil new` doesn't use the `clj` command, but the output should be the same
$ neil new app my-app --foo 1 --bar 2
$ neil clj -Tnew app my-app --foo 1 --bar 2
$ clj -Tnew app my-app :foo 1 :bar 2

# Equivalent commands
$ neil tool install io.github.seancorfield/deps-new --as new
$ neil tool install io.github.seancorfield/deps-new --git/tag v0.4.13 --as new
$ neil clojure -Ttools install io.github.seancorfield/deps-new '{:git/tag "v0.4.13"}' --as new
$ clojure -Ttools install io.github.seancorfield/deps-new '{:git/tag "v0.4.13"}' :as new

In this example, the sole purpose of the neil clj and neil clojure commands is to provide babashka.cli integration.

For abstractions on top of clj/clojure, the specific top-level commands (neil tool, neil new) can include additional functionality such as inferring the deps info from the lib name.

[didibus]

One issue I see with Clojure CLI is that running an alias is kind of complicated. You have to figure out if it's supposed to be used with -X or -T or -T: or -M, which means you almost always have to inspect the deps.edn to remember how to call the alias. And then even if you know how to call it, the invokation is extra long and cryptic and escaping options is more difficult than standard Unix.

This is why you added neil new and neil test no?

But instead of eventually adding a special command for every common alias that comes into existence so that beginners have an easier way to remember and call those aliases. Why not make it generic that all aliases automatically becomes neil commands?

Say I want to add antq for checking outdated deps:

neil tools install-latest --lib com.github.liquidz/antq --as antq

neil antq outdated

Alternatively I create an issue in neil and ask to add neil outdated as a command.

I still think it makes sense to have the most common aliases lifted into neil itself so that it auto-injects the alias and so standardizes things in a way, but having a generic "alias" is automatically a "command" of neil seems to fill the gap, and I think you could then replace all usage of Clojure CLI related to build with neil.

[borkdude]

An alias can support both -X and -M. In fact, the test alias added by neil can be run in both ways. So there is no automatic way of detecting that. I think standardizing the way neil behaves is the best way forward.

[didibus]

An alias can support both -X and -M. In fact, the test alias added by neil can be run in both ways. So there is no automatic way of detecting that.

That's true, but like I mentioned, it could either prompt or we can use an extra key on the alias to specify which one neil should use like so:

:build {:deps {io.github.clojure/tools.build {:git/tag "v0.8.3" :git/sha "0d20256"}}
          :ns-default build
          :neil/run-as :tool}

I think standardizing the way neil behaves is the best way forward.

It's possible as well. I was hoping more for a Clojure CLI but with better command line UX. But also possibly if neil just covers 99% of all common commands just in a standard way it brings most of the same benefits with less effort and complexity, because then I only need to reach for Clojure CLI in exceptional cases. As always, I trust your judgement.

[borkdude]

From 0.1.43 on neil saves data in {:aliases {:neil {:project ...}}}. Using that data, we could have some more possibilities here (not sure what precisely).
E.g. the project name is saved under :name myorg/mylib.
I'm planning to put :version ... there too and have a neil version patch command which increments the patch version, etc. Similar to what npm does. Npm also adds a tag for the version but I think we could have that as a separate neil tag command which creates a git tag for the {:project {:version ...}}.