Vaex, a python library to explore big tabular datasets

[NikosAlexandris]

Just wanted to bring your attention to https://vaex.io/ (see also https://towardsdatascience.com/vaex-a-dataframe-with-super-strings-789b92e8d861). I see (partial) "functionality overlap" relevance to Miller when doing statistics or working with large/massive datasets. @johnkerl What do you think?

[trantor]

Mmmm.
Speaking for myself, the Apache Arrow format underneath seems interesting. As for the rest, it superficially looks like the usual collection of Python glue code on top of a C/C++ library doing the actual work. And there are a ton of them of every possible flavour around.
Not to mention that miller shines by being an independent CLI tool with a builtin set of verbs operating on tabular data, not a library performing statistics or data-science stuff on data. So I am not even sure whether one could even compare the two, aside from the obvious fact that both operate on structured data to some degree or end.

[NikosAlexandris]

Thanks @trantor for sharing your view. I am interested in all about processing large/massive structured datasets. I don't know if you had a look at the performance graphs at Beyond Pandas: Spark, Dask, Vaex and other big data technologies battling head to head -- they are impressive.

[trantor]

@NikosAlexandris from what I can understand (unfortunately it's not my field) they certainly appear to be. It just seems to be a rather different game and space (and a crowded one, for good reason) compared to miller's, aside from the fact that those libraries and the software built upon them can mostly ingest CSV data. Also, those libraries seem more geared towards data processing than data exploration per se. Tools such as datasette or visidata or OpenRefine seem more exploratory in purpose and closer to the scope miller operates in.
One thing I appreciate about miller is precisely that it's not geared towards the massive data processing that is all the rage nowadays (again, the latter for a mountain of valid reasons). Problem is that all attention goes there without that much effort going into manual data exploration and data wrangling tools, which is something, at least for my concerns (and far from just mine), far more significant. Especially talking about open-source and CLI-oriented tools.
Just to be clear, I do not mean to appear hostile to the kind of data-processing software/libraries you are talking about @NikosAlexandris (although I am a tiny bit hostile to Python as a language, but that's another story 🤣 ). Mine is a rather biased opinion of course, since I do not quite enjoy the imbalance I see in the areas attracting investement and software development, with certain areas (shall we talk about ML?) seeing a positive glut of libraries/software/companies/solutions and others, arguably also significant, seeing a veritable drought when it comes to options and interest by developers.
Btw I believe I derailed the conversation in quite another direction, sorry about that 🙏.