FinalPrototype-Group:BigBrother
From CS160 User Interfaces Fa06
Contents
|
Deliverables
Team Members and Roles
- Charles Leung: worked on the grading interface, made changes on the proctor interface, wiki writeup
- Jonathan Chang: worked on the grading interface, made changes on the proctor interface, powerpoint presentation, wiki writeup
- Randy Hilarbo: worked on the grading interface, made changes on the desktop interface, wiki writeup
Problem and Solution Overview
While the main function of university exams is not to entertain and delight the student, we of Team BigBrother believe that there are aspects of the process that can be improved, resulting in improved academic performance and more efficient distribution of resources post-exam.
What’s Wrong?
For a test-minded student, getting an exam-related question answered in the middle of a large undergraduate midterm proctored by a single GSI can be a daunting process. Such an environment is also rife with "gray area" cheating like starting ahead of time, or having an extra five minutes to work from sitting in the back of the testing room. After the exam, students often do not have all the resources necessary directly at hand in order to make an informed decision of whether or not to request a regrade. This can lead to either frivolous requests, which waste the graders' time and frustrates students who mistakenly believe they can earn more points, or a decision not to submit a valid regrade request.
How Do We Fix It?
We deploy a testing application that records everything written on an exam, and processes it to solve the stated problems. Students can reliably ask questions with a paper messaging system, and timestamps allow the system to flag pens that are working before the test begins or after it ends. The same application deploys what it recorded to a web application where students on their own time can view their graded tests, consisting of everything the wrote and everything the grader wrote, and based on a perceptually close grading rubric submit regrade requests.
Target User Group
While this application necessarily must be used by more than one user group, we have primarily designed it with students in mind. We are targeting undergraduate students who have large class sizes, and thus large exam populations. Our application does not depend on any degree of pre-existing technical proficiency; we would like students to be able to transition from their old method of test-taking to ours with little to no training or other adaptation. Our target user group is also generally honest and trustworthy in test-taking situations, but can benefit from a little bit of administrative and peer pressure to remain honest.
Tasks
Easy: Asking Questions
In the current exam setting, students only have to raise their hands in order to let the monitoring figures know of their inquiries about the exam. However, this current method does not always guarantee that a student’s question is immediately attended if it is attended at all. The reasons why this is included, based on the results of our contextual inquiries, the inattentiveness of those who are monitoring the exam, the invisibility of the raised hands to the monitoring figures due to the large size of the class as well as the distant position of the GSIs/professor relative to the student with the raised hand.
Our interactive prototype provides a better alternative to raising hands guarantees that a student’s inquiry will be attended by immediately sending notification to the test proctor of what the question is. In addition, if the proctor sees that many students have similar questions, he can simply make an announcement to the whole class and save himself, and his students time.
Medium: Taking the Test
Taking the Anoto-enabled pen is categorized as a medium task because of some expressed discomfort during the Lo-Fi prototyping with taking the test in our altered format. While we did spend some time attempting to address this issue by brainstorming ways of making the format more approachable, we eventually decided that not only was the dissenting voice in the monitor, but each designer could remember a similarly designed exam they'd taken before. Therefore, we decided to keep our previous format of a sheet of questions, and blank Anoto-enabled answer sheets for the students work and answer on. With respect to the potential discomfort, we kept the rating of this task of Medium, and chose exam implementations that already used the separate question and answer format.
Hard: Reviewing and Requesting Regrade
In some classes, the GSI only gives students a limited time to decide whether to ask for a re-grade. These comprise the reasons of why we identify this as a hard task in the current exam setting. The students are required to follow certain methods (i.e. prove necessity of a re-grade through writing, etc.) just to challenge the consistency of the graders in marking exams. In terms of this assignment, our interactive prototype for performing this task may require the most effort from our users. Our users need to interact with a computer in order to complete these tasks by visiting a website that we built for students to lookup their tests and request re-grades.
Design Evolution
Our system underwent major changes during the course of the design cycle. We narrowed our focus from general users to a more specific target users. We also found ourselves changing our interfaces to reflect features that we want to incorporate to our evolving design. Deploying our interfaces for testing has greatly influenced these changes.
Initial Sketches
We started with a design that addresses the needs of three different roles found in an ongoing examination: a student taking the exam, the GSI proctoring the exam, and a professor viewing statistics of an exam. For each, we designed an interface to meet their needs.
Student Interface
We proposed an exam booklet consisting of an ‘answer’ section, a ‘question’ section and an ‘attention’ section to be used by students who are taking an exam.
- The ‘question’ section mainly contains the problem sets for the exam.
- The ‘answer’ section is where the student would place his/her answers. An answer sheet contains a box at the upper-left corner of the page which when checked, allows the student to indicate the problem he is answering. This sheet also contains a big space where the student can work out his answers and a solution box, which should contain the student’s solution to the problem. A set of boxes for the grader to mark the student’s score is also reserved on each sheet of this ‘answer’ section.
- The ‘attention’ section serves as the student’s means of communicating to those proctoring the examination. This is consists of boxes where the student can write his question on an exam problem or report any suspicious behavior. This is consist of rectangular regions with two small boxes; the box on the upper-left corner needs to be checked to activate the region and a ‘send’ box, which will send whatever the student wrote on the rectangular region to the proctors.
GSI/TA Interface
The initial proctor interface is a GUI consisting of a matrix of circles. Each circle represents a student who is taking the exam; that is, a circle at row 2, column 3 maps to the student sitting at row 2, column 3. To enforce cheating detection, a circle corresponding to a suspected cheating student is colored with red. Cheating is detected from analyzing the timestamp of when students write their answers and their locations relative to the students answering the same questions at the same time. The same interface also communicates with the student’s attention page. The student’s sent question/message (a question about an exam or a message informing the proctor that someone near the student maybe cheating) is displayed in the proctor interface. The proctor will then know which student to attend to based on the position of the activated circle in the GUI.
Professor Interface
The professor interface is a graphical interface, which allows an instructor to view information about the exam s/he has given out. One important aspect of this interface is to provide the professor with more than the current available exam statistics; that is, this interface is supposed to compute the performance of the class based on the time they spent answering each question and so on. Sketched features of this interface are as follows: viewing the student’s answer sheet side by side with the answer key, emailing the student his answer sheet and easily updating the student’s score.
Change of Focus
Before deploying our design for the initial user testing, we managed to narrow our target users so that they only include students. As a result, we abandoned the professor interface and came out with a third student’s interface. This third interface is a web application that the students can use to view their exams, the statistics of the exam as well as to request a regrade of their exams. The initial design of this interface was greatly influenced by the professor interface; this interface adapted the professor’s interface’s feature of viewing the student’s answers along with the answer key. The regrade request feature is also adapted to that of the previous interface where the professor can email the student his answer sheet. In short, the professor interface helped shape the design decision we made for this student’s interface. In addition, we divided the student’s interface into two interfaces: the ‘answer’ section and the ‘attention’ section. We kept the proctor interface only to respond to these students’ interfaces.
Student Interface. |
GSI/TA Interface. |
Professor Interface. |
Low-Fidelity Testing
Exam Interface
For our lo-fi prototype, the exam we gave our subjects consisted of a test booklet with spaces after each question for the student to write answers, and a page for questions at the back of the packet.
- While we had originally thought to have a separate answer booklet, we wanted to keep the test as close as possible to a familiar format. As such, we decided against the answer pages with question boxes for a simple question-answer format test.
- The question page had been divided into four sections for four questions. It consisted of simple instructions for using the page, a large amount of space for writing the message, and a box to check in order to send the message to the test proctor.
GSI/TA Interface
Since we had decided to focus primarily on students as a user group, this interface did not see any progress during this phase of our application development. We assumed that eventually, something would be developed in the way of a proctor interface in order to have a complete application, but this was not the time.
Review/Regrade Interface
We designed four rough web pages for our students to use to access their old tests, view statistics, and request regrades. The web application starts with a login page, after which it goes to a page with the graded test and "Regrade", "Stats", and "Main" buttons. Clicking "Regrade" brings up a page consisting mostly of a textbox and instructions to fill out the form and submit the regrade. Clicking "Stats" brings up the main test statistics and a histogram of the class distribution. "Main" returns the student to the first page. We took this basic functionality from some sketched storyboards during brainstorming, and printed and cut out the major parts of the web application to move around for the benefit of subjects during user prototyping.
Answer Sheet. |
Question Page. |
Web Interface 1. |
Web Interface 2. |
Web Interface 3. |
Feedback and Evolution
Most feedback to our prototypes was positive, with the exception of the exam itself. While subjects like the question page, they were not sure how well the test would work, and how their work would be recognized. Some subjects even preferred the use of a completely separate answer packet, since that went better with the idea of using the pen freely to do the test. They wanted the flexibility of variable space for each answer, instead of being confined to the area beneath each question. We integrated this into our design for the next wave of prototyping.
Pilot Usability Study
The Pilot Usability Study mostly confirmed the correct decisions that we made on the Paper Interfaces. Our users see the purpose of the Questions Paper Interface as a better alternative to raising their hands to ask a question during an examination. The simplicity of the Answer Sheet Paper Interface is seen as comparable to ‘blue-book’ type exams. The reviewing/requesting a regrade interface is recognized as a faster and a more accessible way for students to examine their answers as well as petition for a regrade of their test.
Exam Interface
Answer Sheet Paper Interface
Our study indicates no need to change our design of the Answer Sheet Paper UI. Having an answer sheet separated from the exam questions allows a variety of exam types to be supported by our application. (This also lowers the burden for instructors to prepare their exams.) Moreover, this interface provides a lot of space for students to write their answers.
Questions Paper UI
After our study, we see an opportunity to extend this interface so that it supports our initial goal of preventing exams. We changed the instructions on this paper interface so that this interface not only permits the students to ask questions about the exam but also send messages to the proctors. That is, the students can also use this interface if they want to report someone who they think is cheating. This additional functionality echoes the students we interviewed for this study about their need to remain anonymous if they are to report a cheater.
Proctor Interface
The following are changes made for the proctor interface after the pilot usability study.
Increased Visibility
We depended on our usability test results to better the front-end look of this part of the system.
- We used contrasting colors to highlight different parts of the interface. For example, we made the background color black and dark gray while the message lists and panes as white. This choice increases the visibility if there are new messages for the proctors. We also used colors to alert the proctors if the exam is about to reach the allotted time for it; that is, we implemented a timer where the color of the time changes if there is less than a minute left for the exam.
- In the usability study, we have observed a lot of user errors because they did not immediately recognized a message popup. To resolve these errors, we changed the font-sizes used in this interface.
- To minimize the errors made in our usability study, we also added the ‘ok/cancel’ button on the popup messages.
- To eliminate the errors produced during the usability study, we also grayed out the buttons that are not supposed to be selected yet. That is, we only make the button ‘Cease Button’ active if the exam is actually started. This makes sure that the user will not be pushing buttons that they are nit suppose to press.
Consistency
To address the consistency issues we received from the feedback of our testers, we changed the following to give our application a better look.
- We made the popup confirmation messages the same throughout this interface.
- We made the color choices for this interface consistent with the review/regrade interface.
Added Functionality
- We added a timer to the proctor interface. Before starting the exam, the proctor is asked to set the time that will be allotted for the exam. The timer decrements as the test begin.
- We added popup messages, which would confirm the user’s action. This makes sure that the user’s locus of attention is on the task that s/he is performing, a common problem we observed on the usability study.
- We see an opportunity to add grading as another functionality of this interface. When grading, the proctor would push the ‘Start Grading’ button. This allows the application to distinguish between the student’s and the grader’s ink strokes.
Review/Regrade Interface
In our usability study, we have our participants examine two versions of the review/regrade interface: a web interface and a desktop interface both performing the same functions. We did this to help us decide whether it is necessary to replace our web interface with the new swing-implemented desktop interface. We implemented the desktop version of our previous web interface in order to get rid of the ‘wizard of oz’ implementation of transferring exam files to a server database (which we did not have). In this new desktop interface, flat files acted the role of the database storing the exam data as files. Our usability study results led us to switch to the desktop version of the review/regrade interface. Having our usability study results as a guide, we bettered this interface as follows:
- We consistently used a small set of colors throughout the interface so that it will match the colors used in our proctor interface.
- We removed unnecessary popup messages (for popup messages with only the ‘ok’ button, we had it display the message on the page instead). This decreases the required actions that the user may do.
- We changed the ‘Help’ and ‘About Us’ page so that they only contain text. Before, we rendered url pages to display documentation of the interface. But in our usability study, we noticed that the participants attempt to interact with these pages since the pages appear to be active.
- We removed all the hardcoding in the previous implementation; that is, we worked on the backend part of this interface (will be described in Final Prototype).
Exam Interface. |
Message Interface. |
Proctor Interface. |
Review/Regrade Interface. |
Technique Evaluation
We found the low-fidelity testing to be the most valuable to our prototypes usability. This technique has the most influence in shaping our early decisions on how to go with our design. If we had not have users test the paper versions of our prototypes first, we probably found ourselves facing a lot of design decisions to make at the implementation stage. Having done the lo-fi testing, we had a very good idea of how our design would suit our users. We are able to make important decisions that affected how we implemented our prototypes. For example, we were not sure how acceptable our design of the Question Sheet Paper UI at the beginning of the design cycle. We would have gone to a different design of this prototype. But our low-fidelity test results show that our initial idea of this interface is close enough to what our users wanted. Therefore, we stick with our initial design, which turns out to be the perfect decision; it made the proctor interface, which communicates with the Question UI easier to implement. In short, this technique geared us toward implementing our design, which we did not second-guess at all. We also observed that our low-fidelity testing results addresse wider aspects of our initial design. We received comments on the usability, design choices as well as the functionality of our prototypes. The results of this low-fidelity testing lead us to consider all aspects of our design before we actually implemented it. In our pilot usability study, our results mostly addresses the usability issues of our design: color choices, errors made, etc.
Final Interface
Design
Functionality
Asking Questions / Making Comments
Students can ask questions or make comments without having to raise their hands or leave their seats. By using provided AskQuestions sheets and an Anoto pen, students can ask questions and make comments without having to waste time waiting for a GSI.
Taking a Test
Students can take an exam and answer the questions by filling out the provided AnswerQuestion sheet. The sheet is a normal piece of paper that has an Anoto pen region so that the information can be sent real-time to the proctor.
Reviewing and Requesting a Re-grade
Students can look at their exams and look at how the graders graded their tests. In addition, they can also look at the answer key that the graders used. If they want, they can also look at statistics to see how they compared with the rest of the class.
While we mainly focus on the tasks performed by the students, we also implemented the following functionalities in order to make the system complete.
Proctoring the Exam
This interface works in correspondence to the Questions Paper UI. The students’ questions/messages written on their Questions UI are sent to this interface via the Bluetooth device. This also supports the following functionalities:
- Starting the timer for the exam
- Storing students questions/messages in history
- Cheating Detection
- Grading
Grading the Test
The GSI’s can collect the student’s tests and grade with an Anoto pen so that the computer will have a digital backup copy of the student’s answers with the grader’s corrections.
User Interface
Asking Questions / Making Comments
While taking an exam, students can easily send comments to the test proctor by filling out our AskQuestions sheet. If a student would like to ask or send a comment to the proctor, the student will write down their question on a highlighted region, and then check a box to signify which question his question pertains to, and then tap a send box to send the message to the proctor. The AskQuestions sheet has multiple regions so that a user can send multiple questions to the proctor.
Taking a Test
Students take tests in a similar manner in which they are normally accustomed to. For example, they will take an exam by filling out their answers within a specially marked region within our AnswerSheet sheet.
Reviewing and Requesting a Re-grade
- To access the exam page, the user needs to authenticate himself/herself. (Jon/Jon, Charles/Charles, and Randy/Randy as username/password are working accounts).
- To view an exam, the user selects the ‘exam’ tab to go to the exam page. In here, the user is presented with two lists: the course list, consisting of the courses that the student has taken and the exam list consisting of the exams that this application recorded for the student. To view an exam, the user can either select an exam and then click ‘View Exam’ or first select a course, which would dynamically list the associated exams for that course on the other list, select an exam and click the ‘View Exam’ button.
- The student will then be taken to a page where his exam is displayed along with the exam answer key. This allows the student to easily compare his answers with the answer key. The student can make both his answer sheet and the exam answer key appear in new windows by pushing the ‘View My Exam’ and ‘View Answer Sheet’ buttons respectively.
- The bottom section of this exam page shows the statistics for the exam. Selecting the buttons ‘Request Regrade’ and ‘View Stats’ alternately display the Re-grade Request form and the exam statistics. The ‘Request Regrade’ form initially contains instructions on how the student can submit a re-grade request. To do this, the student would explain the reasons of the request and then select the ‘Submit Request’ button to submit his petition to his instructor.
- The student can click on the ‘About Us’ and the ‘Help’ buttons residing at the top of the application at anytime during the session. The ‘About Us’ page contains information about Team BigBrother and the ‘Help’ page provides help on how to use the application.
Proctoring the Exam
Our study results show that the proctor interface is easy to use. We further decreased the learning rate required to use this system as a result of our pilot usability study (graying out of buttons, contrasting color choices, etc as described above). Furthermore, we printed instructions of how to use this interface on the proctor interface display. The following describes the usage of the functionalities provided by this interface:
- To start the timer, the proctor needs to enter the amount of time allotted for the exam on the timer fields. If the proctor forgets to perform this task, the proctor will alert him to do so before he can start the exam. The timer counts downs until it reached zero or the proctor manually stops the timer; if the timer reaches zero, it will alarm the proctor to stop the exam. We used three different colors to display the time: ‘pink’ indicating that there is less than five minutes left for the exam, ‘red’ means that there is less than a minute left to finish the exam, and ‘black’ otherwise.
- To officially start the exam, the proctor needs only to push the ‘Start Testing’ button. To call out the test, the proctor would need to push the ‘Cease Testing’ button. The proctor keeps two list of messages received from the students’ Questions Paper UI; the message list contains questions/messages that are not viewed yet while the history list is populated my questions/messages that were viewed already. Selecting a message on each list brings out a window where the message corresponding to the selected entry is displayed.
- Cheating is automatically detected by the interface. The ‘Start/Cease Testing’ buttons are used to record the times when the exam officially starts and ends. If the student started marking his/her answer sheet before the ‘Start Testing’ or after the ‘Stop Testing’ is pushed, the proctor will red flag the student as a possible cheater. Another way to red flag a potential cheater is through another student’s report sent to the proctor interface using the Questions UI sheet.
Grading the Test
To grade an exam, the grader only needs to push the ‘Start Grading’ button. S/he can then start marking the student’s interface. To indicate that s/he has finished grading an exam, s/he only needs to push the ‘Stop Grading’ button. The ‘Start Grading’ button is only available after the proctor has cease the testing; that is, this button is grayed out when the exam is currently being given out. The ‘Start Grading’ button becomes the ‘Stop Grading’ button after it is hit once. This decreases the number of buttons displayed on the proctor interface.
Implementation
Paper UIs
- We made our Answer Sheet Paper UI so that it contains one big active region, which will contain the student’s answers for the exam. We added labels on the upper right corner of this interface where the student can write his name, and information about the exam. The answer sheet will be automatically registered to the student’s pen ID. Therefore, the label at the upper-right corner of the page only serves as the identification for the paper copy of the answer sheet, which serves as the backup copy of the student’s exam. We chose to design this interface this way to mirror the ‘blue-book,’ a typical answer sheet used by Berkeley undergraduates. We figured that this design is flexible enough to support different types of exam (multiple-choice, math exams, essay exam).
- We designed our Question Sheet Paper UI so that it contains three types of active regions: a question box region indicating the question number that the question is referring to, a question box, which should contain the student’s question/or message to the proctor, and a send box region acting as a trigger to communicate the student’s question/message to the proctor interface. The number of question boxes is adjustable depending on the number of questions on the exam (this is adjusted by manually changing the value of the variable ‘numQuestions’ in AskQuestionsPaperUI class). We also outlined an instruction on how to use this sheet on the page.
- The hardest parts of implementing this part of the system are as follows:
- Figuring out the dimensions of the regions of both the paper UI was a bit of a challenge.
- Since we have two paper UIs, which serve different purposes, we need to make the region from one UI different from the other. We had earlier problems of making the regions of the Question UI different from that of the Answer Sheet Paper UI. We resolved this by rendering the two sheets as one pdf file. But, we ran into troubles because we cannot print the pdf as two pages. After seeking help from Ron, we created a new class ‘BigBrotherPaperUI’ which resolved our problems (This class allows us to render multiple pdf pages, each with different active regions).
Answer Sheet Paper UI. |
Question Sheet Paper UI. |
Review/Request Regrade
- We implemented this interface using the JAVA Swing API. We have a total of four pages that a user can browse in this interface. In each page, we used a Jframe. We then add necessary components to the Jframe. Most of the components have listeners in them where we add methods to change the display to correspond to the user’s actions.
- The login page reads the ‘data/users/accounts.txt’ file and check if the provided login matches an entry in the file. If so, the user is granted access to the rest of the interface. Otherwise, a Jbutton listener triggers an error message to be displayed on the login page.
- The exam page is initially a Jframe containing two Jlist, each containing either a list of courses that the student took or a list of exams for those courses. We populated these lists by reading from the user’s file (data/users/user.txt).
- The second exam page is divided into three sections; the first two sections contain the student’s answer sheet and the exam answer key while the third section displays the exam statistics and the form for requesting a regrade. To display the student’s answer sheet, we looked for the answer sheet associated to the student, which should have been created after running the proctor interface. The exam answer key is grabbed from the folder ‘data/exams/Exams’. We added listeners to the buttons ‘View My Exam’ and ‘View Answer Sheet’ so that when clicked a window would pop out containing the student’s answer sheet or the exam answer key respectively. We used ScrollDemo.java, a shareable code we found online, to implement this pop ups.
- The hardest part in implementing this interface is getting all the Swing components to work together. We spent a lot of time debugging so that the layout of each page is what we wanted it to be. We also spent a great deal in making the back-end part of this interface to work. It was a challenge to decide how we would format each user file to contain information we would need for this interface. Making authentication to work as well as populating the course and exam list were a bit complicated.
Failed Login |
Course/Exam Lists |
View Exam Button |
Exam Page |
Maximize Exam |
Regrade |
About Us |
Help |
Proctor Interface
- The proctor interface is implemented using the JAVA Swing API. We use the R3 toolkit’s InkCollector to store the ink strokes received from the Anoto pen. We use a lot of Swing components to make this interface. We added listeners to these components to respond the actions trigger by both the user and the pen.
- One of the hardest parts in implementing this interface is dealing with the Swing components. We were just using too many of them that we found ourselves spending a lot of time debugging the code. Finally, we decided to rewrite our code so that the Swing components are organized into modules to separate the action listeners, components, and message handling from each other.
- Another hard part is listening to the events generated by the Anoto pen. We had problems distinguishing which active regions did the pen event came from. We did resolve this after careful debugging of our code.
Grading the Test
- To implement the grading, we only had to make sure that we distinguish between the student’s ink strokes and the grader’s ink strokes. We accomplished this by rendering the grader’s stroke using a ‘red’ color while using a ‘black’ color to display the student’s strokes.
- The hard part about implementing this additional feature is the rendering of strokes to a jpeg file using two different colors. We had to modify the R3 toolkit in order to achieve this requirement. In the InkRenderer class, we defined a public variable ‘timestamp’ which is set when the ‘Start Grading’ button is hit in the proctor interface. Inside the class RenderingTechniqueCatMullRom class, which does the actual rendering to jpeg, we check if the stroke.timestamp is greater than the timestamp of when the ‘Start Grading’ button was hit. If so, we use a ‘red’ color (g2d.setColor(Color.RED)) to render the stroke.
Main |
Error Starting Exam |
Cheating Detection |
Message |
Start Timer |
Timer Change Color |
Stop Test |
Start Grading |
Unimplemented Sections
Database
While we have come much further in terms of implementing the backend of the review/regrade application, it is still a matter of outputting the ink to a file on the local harddrive, then retrieving it with the review application from the same place. We did not believe it necessary to actually implement the database, since this demonstrates our ability to use a database in the same capacity as the local disk. The testing application outputs an image file, which after being saved to a temporary location on the local disk, could be inserted into a database located on a convenient server. In order for the review application to work elsewhere, it would ask the server to retrieve the image file from the database and place it in a temporary location on the harddrive of the machine running the review application. As such, our application could remain unchanged and still plug into a database with proven implementations.
Multiple Pens
Since this is a testing application, we designed our project to be extendable to a whole classroom of students using distinct pens to record their exam and communicate with proctors. There are mechanisms in place that check for the pen's ID, and messages are hashed distinctly with regards to the specific pen. We believe that our application would support more pens to the extent of our original mission plan, though we did not at this time try specifically to implement on a large scale. Further testing may reveal bandwidth limitations, when many pens attempt to maintain a constant stream of information through the Bluetooth connection. There may also be problems with range that could also lead to dropped packets or other application failures. Instead of wrestling with these issues, we forced ourselves to be satisfied with refining a small scale application, while leaving theoretical room for it to be extended in the future.
