Is This A Proper Way To Test Stdout With Python 3 Unittest?

Suppose I have a submission file, fileFromStudent.py, and the only thing in it is: print('hello world') I would like to test the stdout to see if the student has properly written

Solution 1:

I've spent a couple weeks working on this with moderate success, headaches, and Google. One of the reasons I did not go the Popen route was that I wanted to capture if students submitted bad code that instantly crashes. Believe or not, the first few weeks of an Intro course are like that. Since everything I've found was from 2011-2012, I figured I'd post this so future Google'ers can find it.

Expanding out what I wrote above, let's assume the next assignment was to get an input and say "Hi"

name = input("What's your name? ")
print("Hi " + name)

Now, I want to automate the test to see if I can type in "Adam" and get back "Hi Adam". To do this, I chose to use StringIO as my stdin (sys.stdin = StringIO("Adam")). This allows me to have control of where text streams are coming from and going. In addition, I don't want to see all the errors a student might have happen (sys.stderr = StringIO()).

As I mentioned, I chose to use importlib instead of Popen. I wanted to ensure that if the Student submitted bogus code, instead of breaking everything, just fail whatever test I was running. I experimented with subprocess and py.test and while they might be a better, cleaner fit, I couldn't find anything that made sense to me on how to get it moving properly.

Below is a copy of my latest version of the test:

from io import StringIO
from unittest.mock import patch
import unittest, importlib, sys, os
from time import sleep

# setup the environment
backup = sys.stderr

classTestTypingExercise(unittest.TestCase):
    def__init__(self, test_name, filename, inputs):
        super(TestTypingExercise, self).__init__(test_name)
        self.library = filename.split('.')[0]
        self.inputs = inputs

    defsetUp(self):
        sys.stdin = StringIO(self.inputs[0])
        try:
            ## Stores output from print() in fakeOutputwith patch('sys.stdout', new=StringIO()) as self.fakeOutput:
                ## Loads submission on first test, reloads on subsequent testsif self.library in sys.modules:
                    importlib.reload(sys.modules[ self.library ] )
                else:
                    importlib.import_module( self.library )
        except Exception as e:
            self.fail("Failed to Load - {0}".format(str(e)))

    ## Test Casesdeftest_code_runs(self):
        test_case = "Checking to See if code can run"
        self.assertTrue(True, msg=test_case)

    deftest_says_hello(self):
        test_case = "Checking to See if code said 'Hi Adam'"# Regex might be cleaner, but this typically solves most cases
        self.output = self.fakeOutput.getvalue().strip().lower()
        self.assertTrue('hi adam'in self.output, msg=test_case)

if __name__ == '__main__':
    ignore_list = ["grader.py"]

    # Run Through Each Submitted File
    directory = os.listdir('.')
    for filename insorted(directory):
        if (filename.split('.')[-1] != 'py') or (filename in ignore_list):
            continue#print("*"*15, filename, "*"*15)# 'Disables' stderr, so I don't have to see all their errors
        sys.stderr = StringIO()     # capture output# Run Tests Across Student's Submission
        suite = unittest.TestSuite()
        suite.addTest(TestTypingExercise('test_code_runs', filename, 'Adam'))
        suite.addTest(TestTypingExercise('test_says_hello', filename, 'Adam'))
        results = unittest.TextTestRunner().run(suite)

        # Reset stderr
        out = sys.stderr.getvalue() # release output
        sys.stderr.close()  # close the stream 
        sys.stderr = backup # restore original stderr# Display Test Resultsprint(filename,"Test Results - ", end='')
        ifnot results.wasSuccessful():
            print("Failed (test cases that failed):")
            for error in results.failures:
                print('\t',error[1].split('\n')[-2])
        else:
            print("Pass!")
        sleep(0.05)

Here is the end result:

StudentSubmission01.py Test Results - Failed (test cases that failed):
     AssertionError: Failed to Load - EOL while scanning string literal (StudentSubmission01.py, line 23)
     AssertionError: Failed to Load - EOL while scanning string literal (StudentSubmission01.py, line 23)
StudentSubmission02.py Test Results - Pass!
StudentSubmission03.py Test Results - Pass!
StudentSubmission04.py Test Results - Pass!
StudentSubmission05.py Test Results - Pass!
StudentSubmission06.py Test Results - Pass!
StudentSubmission07.py Test Results - Pass!
StudentSubmission08.py Test Results - Pass!
StudentSubmission09.py Test Results - Pass!
StudentSubmission10.py Test Results - Pass!
StudentSubmission11.py Test Results - Pass!
StudentSubmission12.py Test Results - Pass!
StudentSubmission13.py Test Results - Pass!
[Finished in 0.9s]

I might need to move things around if I want to test multiple different inputs, but for now this works.