Pfthb

I want to make a tool that test different exercises. One exercise is unit-testing. Therefore I need to test whether the tests that are made by the student are good tests. So for example the student has the following code:

export class HelloWorld 
 public static showHello(): string 
 return 'Hello World!';
 

With the following jest test:

import HelloWorld from '..';

describe(Hello World exercise, () => 
 test('Check function is defined', () => 
 expect(HelloWorld.showHello()).toBeDefined();
 );

 test('Empty input results in Hello World!', () => 
 expect(HelloWorld.showHello()).toBe('Hello World!');
 );
);

How can I test that the student did indeed test these two tests? I thought about

export const firstTest = test()...

And then test whether firstTest test the right thing. But the disadvantage is that you have to export every test for this solution.

Testing tests requires black box testing, which means you either need to know which tests are written before hand (and how they should be written) or you need review each test and write unique tests for each one. Given you are trying to test students' understanding of tests and do this in an automated way, I doubt either option is plausible.

However, there are several heuristics you can use. I suggest using a combination of the first 3 of these methods.

1. Define a public interface your students must implement and write tests to this public interface. This doesn't directly test your students' tests, but if any of your tests fail, they did not write sufficient (quality) tests. These tests should be easy and obvious for your students should focus on.


2. Use a code coverage tool such as istanbul. This doesn't test the quality of the tests directly, but it does give you confidence that the tests actually touch the code.


3. This method is inspired by (not an implementation of) a technique I recently discovered called "Adversarial Gamedays". For this method, you would random break the student's code in several places, let's just say 10, and let the code run. Chances are the tests won't catch all of the bugs you made, but they should catch x%, where x is the acceptable code coverage you determined. The bugs should be sufficiently spread across the program so as not to take advantage of a single flaw. This tests both quantity and quality.


4. Visual inspection and/or manual testing. This method is error-prone and time consuming. If you don't have TAs, it's probably not a viable option. However, if you spend sufficient time on this method, it is the most thorough and least biased.

As I stated, a combination of the first 3 options would be optimal. For example, provide a public interface for the system, run code coverage tools, and randomly break y areas of each student's private implementation.