The Curriculum Script

Douglas Hacker, Myles Bogner, Holly Yetman, and Bianca Klettke

Research on tutoring has suggested that learning is enhanced to the extent to which the tutor develops and utilizes a well-defined model of the tutee (e.g., Anderson, Corbett, Koedinger, & Pelletier, 1995; Merrill, Reiser, Ranney, & Trafton, 1992; Ohlsson, 1986; Winne, 1989). Such a model has been presumed to serve as a mechanism through which a tutor can better diagnose and remediate the problems a tutee encounters. More accurate diagnoses make possible better decisions concerning whether new material should be given or old material clarified, or whether the tutee needs additional prompting, questioning, or practice (Putnam, 1987). However, other tutoring research has shown that tutors rarely develop a well-defined model of their tutees (Graesser, Person, & Magliano, 1995; McArthur, Stasz, & Zmuidzinas, 1990; Putnam, 1987). Putnam (1987) has shown that only about 7% of the tutee problems are ever followed up by the tutor for the purposes of diagnosing knowledge deficits. Thus, in some tutoring sessions, little time and effort are devoted to constructing a diagnostic/remedial model of the tutee during instruction.

Instead of a well-defined model of their tutees, tutors appear to follow a predetermined script, focusing greater attention only on those elements of the script the tutee has missed. Adherence to a tutor-driven script may be nearly absolute. Research by Graesser and his associates on normal tutors (Graesser & Person, 1994; Graesser et al., 1995) indicates that the tutor sets 100% of the agenda in a tutoring session, introduces 95% of the topics, presents 80% of the examples, and asks 80% of the questions. There are differences among researchers concerning how rigidly tutors adhere to a predetermined script (cf. McArthur et al., 1990); however, there is agreement that the overall macrostructure of a tutoring session is driven by a curriculum script, and deviations from the macrostructure occur only when there are errors or difficulties encountered within the microstructure of the script (McArthur et al., 1990).

According to Putnam (1987), a curriculum script is "a loosely ordered but well-defined set of skills and concepts students are expected to learn, along with the activities and strategies for teaching this material" (p. 17). The curriculum script is formulated by the tutor (Graesser & Person, 1994; Graesser et al., 1995) and is the "major determinant of the agenda for the tutoring lesson" (Putnam, 1987, p. 17). Contained within the curriculum script are predetermined sequences of problem types, subtopics, examples, questions, and lessons that are used to instruct a discrete topic area (McArthur, Stasz, & Zmuidzinas, 1990; Putnam, 1987). Within the curriculum script macrostructure is the agenda, or microstructure, that is to be followed during a tutoring session. The microstructure is a "dynamic plan or set of goals and actions for a particular lesson" (Putnam, 1987, p. 39). In addition, the microstructure contains policies and microplans that may be used when difficulties or misconceptions arise (McArthur et al., 1990). These policies and microplans can serve as procedures that determine when a specific line of questioning is to be terminated, what new subtopic is to be presented, or the number of examples to be presented.

The macrostructure of AutoTutor’s curriculum script is arranged hierarchically. At the broadest level is the knowledge domain, which is a global body of knowledge that the tutoring addresses. Within the knowledge domain are topics, natural chunks of knowledge characterized by common themes. At the next level are subtopics, which are subchunks of knowledge characterized by more discrete themes within each topic. Each subtopic is a structured database that is further divided into smaller levels, or slots, each focused on a specific component of the tutor-tutee dialogue. Further details of each level of the curriculum script are described next.

The knowledge domain selected for AutoTutor is computer literacy. In selecting a topic, we considered several possibilities, including introductory medicine, biology, and chemistry. The only restriction placed on our selection was from Latent Semantic Analysis (LSA), which requires that the knowledge domain be representative of a less precisely defined domain in contrast to a well-defined domain, such as mathematics or computer programming. From among the several possibilities, computer literacy was selected for several reasons. Because computer literacy is a required course for all undergraduates, we were afforded a large subject pool for preliminary tests of AutoTutor and for gathering information relevant to the curriculum script. Most of the personnel working on the AutoTutor project have expert knowledge in this domain. Also, the creation of a computerized tutoring system capable of instructing a required course would be of potential benefit to the University.

Topics

Three topics were selected from the computer literacy course: computer hardware, the operating system, and the Internet. These three were selected because each represents a distinct knowledge structure within computer literacy, and each contains sufficient content to be addressed by AutoTutor. The three topics also provide a logical sequencing that parallels the computer literacy course: Knowledge of computer hardware is essential to knowledge of operating systems, which is essential to knowledge of the Internet. In the course textbook (Beekman, 1997), computer hardware is addressed in chapters 2 and 3, operating systems in chapter 4, and the Internet in chapter 10.

We have developed 37 subtopics (i.e., subchunks of knowledge) in the curriculum script for the knowledge domain of computer literacy. The first subtopic, or "seed" subtopic, contains a question that is used to assess tutees’ knowledge of computer literacy. The other 36 subtopics are divided into 3 sets of 12, with each set representing one of the topics (i.e., computer hardware, operating system, and Internet). Within each set of 12 subtopics, 4 subtopic formats are crossed with 3 levels of difficulty. The four subtopic formats are: Didactic-information+Question+Answer, Question+Answer, Graphic-display+Question+Answer, and Problem+Solution. The three levels of difficulty (easy, medium, difficult) map onto Bloom’s (1956) taxonomy of cognitive objectives.

Seed question. The first subtopic, which is given to all tutees, provides an initial estimate of the ability of the tutee. This subtopic asks tutees, "What are the parts and uses of a computer?" Using the tutee’s answer to the question, LSA calculates the tutee’s computer literacy ability. Based on this calculation, the tutee will be placed at the most appropriate subtopic/question in the curriculum script (e.g., low ability learners should get easy subtopics)

As part of a larger pilot study, we provided the seed question to about 50 students enrolled in a computer literacy course. The answers provided by these students were ranked ordered on quality and were combined with our own ideal answers to the seed question. This collection of answers serves as the basis on which AutoTutor calculates tutee’s computer literacy ability. As more tutees interact with AutoTutor, their answers to this question will be collected, ranked on quality, and added to the database. Consequently, with greater experience, AutoTutor should become better at assessing tutees’ computer literacy ability.

Subtopic questions. Each subtopic format defines a specific kind of question that will be given to the tutees: The Didactic-information format provides tutees with short didactic content and then asks a question related to that content; the Question format simply asks a question with no content; the Graphic-display format provides tutees with pictorial information and then asks a question based on the pictorial information; and the Problem+Solution format provides tutees with a short problem scenario that students are asked to solve. There are 12 questions within a topic, three for each of the four subtopic formats, with the three questions for each format scaled on difficulty (easy, medium, and difficult).

Once tutees have been placed at the most appropriate subtopic/question in the curriculum script, the subsequent selection of subtopics is partly scripted and partly sensitive to each tutee’s ability. The scripted aspect specifies that subtopics within the computer hardware topic will be covered first, followed by the operating system, followed by the Internet. Also, constraints in the curriculum script specify that one subtopic is a prerequisite for another subtopic. For example, the Didactic-information format is presented first, followed by Question+Answer, followed by Graphic-display, followed by Problem+Solution. The learner-sensitive aspect specifies that the selection of subtopics within a topic is determined by tutees’ abilities as calculated by LSA on all previous subtopic answers and by a set of fuzzy production rules in a production system architecture (Cox, 1995). Thus, the selection of subtopics is strongly influenced by the dialogue between the tutee and AutoTutor who collaboratively generate answers or solutions through several turns. The LSA ratings of those answers or solutions determine whether on the next turn the tutee is given a more difficult or easy question or whether the tutee has demonstrated proficiency with the topic and a new one will be presented

Following guidelines established in Graesser & Person (1994) and Graesser et al., (1995), we scaled the three difficulty levels of our questions to map onto Bloom’s (1956) taxonomy of cognitive objectives. Bloom’s taxonomy classifies cognitive activity into six categories on the basis of complexity. From lower to higher complexity, these six categories are: knowledge, comprehension, application, analysis, synthesis, and evaluation. With the exception of the last two, higher-ordered cognitive activities incorporate lower-ordered ones (Wakefield, 1996). Thus, although evaluation does not necessarily depend on synthesis, synthesis and evaluation build on analysis, analysis builds on application, application builds on comprehension, and comprehension builds on knowledge (Wakefield, 1996).

Easy questions were constructed to map onto the lowest levels of Bloom’s taxonomy. These questions require only recall of specific facts, terminology, or explicit information that is presented in a textbook (Graesser et al., 1995). In answering these questions, tutees do not need to relate their understanding to other information or knowledge, nor do they need to evaluate their understanding using specific criteria (Wakefield, 1996). Some of our easy questions ask for sensory information (e.g., What does X look like?), or sets of components (e.g., What are the components of X?), and others ask for properties of components (What are the properties of X?) (Graesser, Langston, & Baggett, 1993). Our 12 easy questions are shown in Table 1.

Table 1

Table 2

Table 3

Contents of each subtopic. Each subtopic is a structured database with several "slots" of information. Each slot contains a list of one or more words, sentences, or paragraphs which are entered in English (as opposed to predicate calculus, LISP, or other formal languages). The slots are:

(1) Topic description (e.g., computer hardware)

(2) Subtopic format and difficulty level (e.g., Question+Answer, Medium)

(3) Delivery of Context Information (e.g., a verbal or pictorial delivery of content information)

(4) Focal question (e.g., the main question being asked in the subtopic)

(5) An ideal answer to the question or solution to the problem.

(6) A long list of different good answers or pieces of relevant information

(7) A long list of different bad answers, bugs, blind alleys, and misconceptions

(8) A long list of hints, scaled on difficulty

(9) A long list of prompts, scaled on directness and difficulty

(10) A succinct summary of the answer

(11) A list of anticipated student questions and answers to these questions

(12) A list of good keywords (i.e., terms likely to be found in the ideal answer #5 and in list #6)

(13) A list of bad keywords (i.e., terms likely to be found in list #7)

(14) A verbal description of the graphic display (if there is a display). The picture description language follows the representational system described in Baggettt and Graesser (1995).

Slots 1-5, 8-10, and 14 represent the scripted portion of the curriculum script, and as such, were generated entirely by us based on our goals for instruction, the content of the course textbook, and the structural constraints of our curriculum script. Slot 5, the ideal answer for each question, is information from the textbook that is in direct response to the question. Each ideal answer is intended to be complete and comprehensive, with each element of the answer fully elaborated. Slot 10, the summary, is a reduced version of the ideal answer. Only the key unelaborated elements of the ideal answer are contained in the summary. The information contained in slot 14 was generated by us to serve as verbal descriptions of the graphics used in the nine Graphic-display questions. These verbal descriptions will be used as a way to resolve referential ambiguity when tutees make deictic references to specific portions of the graphics. We also intend to test whether the semantic content of the pictures as represented in the descriptions can be used by LSA.

Research on naturalistic tutors (Graesser et al., 1995) and expert tutors (Hume, Rovick, Rovick, & Evens, 1996) served as the bases for generating the hints and prompts in slots 8 and 9, respectively. Prompts are rhetorical devices initiated by the tutor for the purpose of eliciting from the tutee a more elaborate response. A prompt is used when the tutee has provided very little information in response to a question and therefore needs to be prompted to give more (Graesser et al., 1995). Our prompts take the form of a short declarative statement with a blank at the end that is to be filled by the tutee (e.g., Utility programs serve as tools for doing ). The number of prompts we have generated so far is limited to about three or four, but as we gain more experience through actual tutoring sessions, this list will grow.

When a sufficient number of prompts has been generated, we intend to scale the prompts on two dimensions, directness and difficulty. Directness is a measure of the number of key words contained in a prompt. For example, to elicit more information from tutees who are rated by LSA as low ability, the prompts must contain a greater number of key words in comparison to the prompts used for tutees who are rated as high ability (Hume et al, 1996). The more direct the prompt, therefore, the more scaffolding is provided for the tutee to give a response. Difficulty is a measure of the level of cognitive complexity required by a prompt. We again will use Bloom’s taxonomy to guide our scaling of difficulty such that low difficulty will map onto lower levels of the taxonomy, and high difficulty will map onto high levels of the taxonomy.

A hint is a rhetorical device initiated by the tutor for the purpose of bringing a verbose tutee back to the main topic (Graesser et al., 1995). Hints provide information that stimulates the tutee’s recall of specific knowledge or facilitates the tutee’s inferential processing to arrive at a desired answer (Hume et al., 1996). In this case, the tutee has given many contributions, however, the contributions are not germane to the asked question. A hint is then provided to redirect the tutee back to the subtopic addressed in the question. Hints are shorter than prompts and can take three forms: an interrogative (e.g., What about system maintenance?), an imperative (e.g., Tell me more about RAM), or a declarative statement (e.g., The application must interact with the hardware). Like prompts, hints will be scaled on difficulty using Bloom’s taxonomy.

To hints and prompts we intend to add splices. A splice is a rhetorical device that is initiated when the tutee gives an error-ridden contribution (Graesser et al., 1995). Immediately after the error-ridden contribution, AutoTutor will give corrective feedback providing the tutee with the correct answer through a series of dialogue moves. The mechanism for generating splices is still being planned, but two possibilities are currently being considered. When a tutee’s contribution is given a high LSA match with stored bad answers (i.e., the contribution is a bad answer), the tutee’s contribution will be matched with the highest relevant good answer, which will serve as the splice. Alternatively, each bad answer stored in the database will be paired with a canned response that will serve as the splice. When the tutee’s contribution is given a high match with a specific bad answer, the canned response paired with the bad answer will be spliced automatically into the dialogue.

The information contained in slots 6-7 and 11-13 has been generated in three ways. First, after reading the textbook from the computer literacy course (Beekman, 1997), we generated the ideal answers contained in slot 5. We then generated various permutations of the ideal answers to serve as the initial collections of good answers in slot 6. Each of our good answers contains some of the key elements contained in the ideal answers, and bad answers in slot 7 contain none of the key elements. From the good and bad answers, we extracted our collections of good and bad keywords in slots 12 and 13, respectively. Finally, the list of anticipated student questions and answers to those questions contained in slot 11 was generated for each subtopic by selecting all terms contained in the subtopic that we believed would be problematic for tutees.

Second, we conducted a pilot study in which approximately 100 students enrolled in the computer literacy course were asked the 37 questions contained in the curriculum script. Each participating student provided answers or solutions at two times throughout the semester, about mid-way through the course and at the end. Each time they were asked to respond to 12 questions, which were randomly selected from the 37 questions, but at the end of the course 6 of the 12 were questions they had previously answered. We asked them to reanswer the same 6 questions to measure growth in knowledge specific to the questioned subtopic. The students’ answers or solutions have been rated by two independent raters on a 1-5 Likert scale (1 = very bad, 5 = very good). Answers or solutions on which the two raters gave either a 4 or 5 will be inserted in the good answers, and answers or solutions on which the two raters gave either a 1 or 2 will be inserted in the bad answers. Also, these additional good and bad answers or solutions will be the source of additional good and bad keywords.

In addition to the questions, we also asked students to rate the difficulty of each question on a 1-5 Likert scale (1 = very easy, 5 = very hard) and to describe what they found difficult about answering the question. The difficulty ratings along with tutees’ answers will be used to judge the difficulty scaling of our questions. We also intend to use these ratings to examine whether students can metacognitively monitor the appropriateness of responses to open-ended questions. Tutees’ responses to what they found difficult have been used to generate additional question-answer pairs (i.e., information contained in slot 11). The question portions of these question-answer pairs (e.g., What is ROM?), augmented with additional questions from experts, are being used in a corpus for AutoTutor in the detection of tutee questions. We have developed a program to detect whether a tutee’s question is of the form found in the question corpus. If a match is made between the tutee’s question and the corpus, the correct answer, which is part of the question corpus, is returned to the tutee.

The third way in which additional data will be generated for these slots is through actual tutoring sessions with tutees. The lists of good and bad answers, good and bad keywords, and question-answer pairs will grow as AutoTutor gains more tutoring experience. That is, tutee contributions that are very high or low in quality will be added to the lists of good or bad answers, and from these additions, new good or bad keywords will be extracted. Finally, new question-answer pairs will be generated from actual dialogue between AutoTutor and tutees. This is how AutoTutor improves with experience.

Currently, a curriculum script authoring tool is being developed to facilitate the creation of new scripts. The authoring tool will help instructors or researchers dissect the content of a knowledge domain into each of the 14 slots described. The initial design phase of the authoring tool has been completed. Currently, a prototype is being developed.

Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of Learning Sciences, 4, 167-207.

Baggettt, W. B., & Graesser, A. C. (1995). Question answering in the context of illustrated expository text. Proceedings of the 17th Annual Conference of the Cognitive Science Society (pp. 334-339). Hillsdale, NJ: Erlbaum.

Beekman, G. (1997). Computer confluence. New York: Benjamin/Cummings.

Bloom, B. S. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New York: McKay.

Cox, E. (1995). The fuzzy systems handbook. AP Professional.

Graesser, A. C., Langston, M. C., & Baggettt, W. B. (1993). Exploring information about concepts by asking questions. In G. v. Nakamura, R. M. Taraban and D. Medin (Eds.), Acquisition, representation, and processing of categories and concepts (pp. 411-436). Orlando, FL: Academic Press.

Graesser, A.C., & Person, N.K. (1994). Question asking during tutoring. American Educational Research Journal, 31, 104-137.

Graesser, A.C., Person, N.K., & Magliano, J.P. (1995). Collaborative dialogue patterns in naturalistic one-on-one tutoring. Applied Cognitive Psychology, 9, 359-387.

Hume, G., Rovick, J. M., Rovick, A., & Evens, M. (1996). Hinting as a tactic in one-on-one tutoring. The Journal of Learning Sciences, 5, 213-47.

Merrill, D. C., Reiser, B. J., Ranney, M., & Trafton, J. G. (1992). Effective tutoring techniques: A comparison of human tutors and intelligent tutoring systems. The Journal of Learning Sciences, 2, 277-305.

McArthur, D., Stasz, C., & Zmuidzinas, M. (1990). Tutoring techniques in algebra. Cognition and Instruction, 7, 197-244.

Ohlsson, S. (1986). Some principles of intelligent tutoring. Instructional Science, 14, 293-326.

Putnam, R. T. (1987). Structuring and adjusting content for students: A study of live and simulated tutoring of addition. American Educational Research Journal, 24, 13-48.

Wakefield, J. F. (1996). Educational psychology: Learning to be a problem solver. Boston: Houghton Mifflin.

Winne, P. H. (1989). Theories of instruction and of intelligence for designing artificially intelligent tutoring systems. Educational Psychologist, 24, 229-259.