Testing ArkScript

This page guides you through enabling and running ArkScript test suites, which is important when developping features for the language, to ensure it still works as intended.

Running the test suites#

When builing ArkScript, you can build the tests using the CMake configuration flag -DARK_TESTS=On. It will generate an additional target unittests.

Run the unittests target to:

• run various C++ unit tests,
• run error messages testing,
• run AST & parsing tests,
• run formatting tests,
• run language tests (tests/unittests/resources/LangSuite/)

Understanding the different test suites#

The easiest tests to add are snapshot tests, in tests/unittests/resources/...Suite/. They expect an input .ark file and a desired output file (.expected or .json depending on the suite).

• ASTSuite tests the AST parsing, which is then dumped to JSON by the JsonCompiler
• BytecodeReaderSuite tests the decoding of ArkScript bytecode files
• CompilerSuite tests the IR generation and optimization
• DiagnosticsSuite tests the generation of error messages
  • compileTime for errors generated by the compiler
  • runtime for errors generated by the virtual machine
• FormatterSuite tests the code formatter. Input is badly formatted, output is the expected formatted code
• LangSuite are unit tests written in ArkScript itself, testing the builtins and the language
• NameResolutionSuite tests the resolution of names and packages with import directives
• OptimizerSuite tests the AST optimizer. Nodes should be removed by this pass, we ensure they are not present in the final AST
• ParserSuite tests the parser, along with parsing specific errors
  • failure for the parsing failures
  • success for the successful parsing, output is the pretty printed AST
• RosettaSuite tests the Rosetta Code solutions submitted for ArkScript
• TypeCheckerSuite tests the runtime typechecker and how it reports errors

Coverage reports#

It is recommended to add -DARK_COVERAGE=On to generate coverage reports, this way you can check if your feature has sufficient test coverage instead of waiting on the CI.

Generate coverage reports with cmake --build build --target coverage, it will generate HTML files under build/coverage/ (main file is index.html).