Testing

[aiverson]

What testing system should we use for terra-stdlib? Busted is a nice and well developed framework for using with Lua, and is applicable to Terra-stdlib testing requirements, though it would be slightly more difficult to use since it couldn't directly interact with terra variables in the middle of terra functions. I'm also looking at the possibility of writing a new testing system designed like a cross of Spock and Busted to be able to interact with both Terra and Lua directly, and anything that Terra can abstract across through the C-ABI with ease. That is a bit further off though.

Should I just start writing tests in Busted for now and work on other things? or prioritize getting a testing platform with native support for Terra before proceding to other implementations?

[elliottslaughter]

I don't have a strong preference. Busted looks ok. The main limitation (of any Lua framework, not just Busted) is that you can't assert inside a Terra function, right? So you have to break up your tests and pass results back out of Terra whenever you want to check them. (Or I guess wrap the asserts in a cast so you can call from Terra.) I agree it's not pretty, but it doesn't seem that terrible either. I'd be inclined to move forward with something rather that works rather than rebuild the test suite from scratch.

I am slightly concerned about installation. How many dependencies does Busted have? Do we have luarocks integration or is that something we can do easily?

I'm happy to get other opinions on this too.

[aiverson]

I'd much rather live in the lit or nix ecosystems than the luarocks ecosystem. The ability to make local and self-contained package dependencies is important and a massive headache-remover. But there isn't as well developed of a testing framework in lit as busted.

Yes, the main drawback of using a lua testing framework is not having knowledge of Terra things. Being able to assert inside a terra function such that it gets the types from the callsite to be polymorphic, the ability to break down the expression into components and display an annotated readout of exactly what failed, etc. I'd also like for it to integrate with my TerraWatch tool to be able to automatically rerun only the tests affected by a given source change based on what parts of the stdlib they import.

Using busted as the testing framework wouldn't create a dependency on the luarocks ecosystem at runtime. It would only be needed to run the tests during development, which is less bad. There is no need for luarocks integration in terra, developers and CI machines would just need to have luarocks to fetch busted and dependencies. No part of terra or the terra-stdlib code would need to know about busted, only the tests being written in busted formats.

Busted only has a small handful of dependencies.

It appears that someone has already added support for loading Terra files in busted so it should nominally just work to point the busted test root at a directory full of terra files. (in practice, it is appearing to require some nonzero effort.)

[aiverson]

The existing terra support in busted was four years out of date and no longer functioning. I made a PR to update it. I have busted with Terra working on my machine now, but my setup is fragile. On more conventional systems, it should be easier.

[elliottslaughter]

Are there instructions for setting up Busted with Terra? It's probably worth adding to our CI as an external application so we don't break it in the future.

[aiverson]

There are not yet, but it's pretty simple. Busted provides a script that just runs the line require 'busted.runner'({ standalone = false }) in a lua/luajit interpreter with the luarocks loaders available. To make it work in Terra, all that needs to be done is make a script that runs that line in Terra with the luajit version of the luarocks loaders available.

[Qix-]

I would recommend using CircleCI + Dockerfile for testing. Make a .circleci folder with config.yml and Dockerfile and the config becomes quite simplified. What's more is that tests are reproducible locally by running docker build -f .circleci/Dockerfile ..

[aiverson]

That looks nice, but isn't the problem under discussion right now.

[aiverson]

But just shoving the testing into a Dockerfile to make configuration easily reproducible does seem like a good way to accomplish it. It shouldn't be the only way though because we do need to run the tests on many different systems because Terra actually touches some of the architectural details that nodejs is designed to hide. Especially the stdlib which is specifically for providing a consistent and uniform interface on multiple platforms. Maybe it just needs more dockerfiles, one for each target system under test.

[elliottslaughter]

I agree with @Qix- that this is what we should be doing, but this should really be split into a separate issue on the Terra repo.