1. Getting Started

The latest version of the CHREST software is included in the software folder of the tutorial CD, or may be downloaded from http://chrest.info/software.html

You will need Java installed on your computer to run the software. In most cases, you can run the software by double-clicking on the relevant "jar" file. (If this does not work, you need to ensure Java is available as an application for your operating system.) Different instructions apply in two cases:

First, you need to run large models, and so must tell the Java Virtual Machine to use more memory. For this, you must run CHREST using a start-up script: on windows the script extension is .bat, on linux and macintosh the extension is .sh. You may edit the script and change the value of the flag -XmxNNNNm to your computer’s memory size: NNNN is the number of megabytes, e.g. -Xmx1000m represents a gigabyte. After editing, double click on the script to start CHREST.

Second, you want to write a script in your favourite JVM language, and need to include CHREST as a library. In most cases, all you need to do is make the file jchrest.jar available as a library: for example, in Netbeans, you add the jar file as a library.

2. Introducing the shell and basic learning operations

Start the CHREST program. You will be presented with an empty central window, and a menu at the top. The menu offers three choices.

Shell

provides information on the version of CHREST and allows you to change the theme (the visual appearance) of the program.

Data

currently offers one option, to open a dataset. Opening a dataset will result in a different interface for running experiments, depending on the type of data.

Model

provides options to view and query the current model for information. We go through the model options as we go through the experiments.

Learning about the learning operations

The simplest type of interface to the model allows you to present individual patterns to the model, and see what information it retrieves when asked to sort a pattern:

  1. Use the Data/Open menu to locate and open the sample data file "demo-1.txt".

  2. Use the Model/View menu to open a view onto the model (rearrange the windows so you can see both).

  3. Highlight a pattern on the display.

  4. Click on Learn for CHREST to perform one cycle of recognition and learning.

  5. Observe the change in the discrimination network.

  6. Click on Recognise, which shows the image of the node reached after sorting the highlighted pattern through the network.

  7. Repeat from (3)

You should explore the pattern learning in this way, perhaps following the example in the slides, until you understand and can predict the operations of familiarisation, discrimination and chunking.

Timing of learning
After exploring the basic operations, observe the 'clock' at the top-left of the
model view.  You can alter the timing parameters by using the 'Model/Properties'
dialog box.
Effects of learning

Open the file "demo-2.txt" and clear the model.

  1. Train completely on the pattern <A B C D $>

  2. Check that the pattern is recalled when you click Recognise

  3. Now train one step on the pattern <A B C E $>

  4. Check that the pattern <> is recalled when you click Recognise. Why?

  5. Now, try to recognise the pattern <A B C D $>

    1. What result do you get?

    2. How does this compare with what happened in step 2?

    3. Can you say the model has "forgotten" the original pattern?

Exercises
  1. Compare the network created when you learn each pattern just once, and repeat in a cycle until every pattern is complete (e.g. by using the Learn All button), with the network created when you learn each pattern completely (e.g. by repeatedly clicking Learn on each pattern) before moving onto the next one.

  2. Look at the files "demo-1.txt" and "demo-2.txt" in a text editor. Now, create your own file containing a list of five words. Load into CHREST, and see how long it takes CHREST to learn and what the network looks like.

  3. Create a file containing five patterns with at least four symbols each which will result in the shallowest possible network. Create another set of five patterns to result in the deepest possible network. Explain what characterises the patterns in each case.

(Note: you can save the network from CHREST using the View menu on the view frame.)

3. Verbal-learning experiments

The verbal-learning paradigm was once the dominant paradigm in psychology, and is still one of the most revealing ways of demonstrating the micro-level of human memory and learning. The experiment and simulations in this section enable you to gather your own data and develop models of the main results in this area.

3.1. Experiment: Protocol analysis

The software folder contains a folder called verbal-learning. This folder contains a program for running a verbal-learning experiment on a human participant. There are separate "jar" files for different computer platforms: double-click on the one for your computer to start.

The experiment involves learning a sequence of stimulus-response pairs, in the form of three-letter nonsense "words". You can select a dataset and the times for inter-trial and inter-item delay. To start, select "example 1" as the dataset, 2000ms for inter-trial time and 1000ms for inter-item delay. Finally, click "run".

The display will change, and show you each stimulus-response pair in turn, for the given amount of time. Attempt to memorise each pair! Once one trial has finished, you will be shown each stimulus in turn and prompted to enter the response. Use "tab" to move between each field, and press "enter" when you are ready for the next response. (Case does not matter.) You will then be shown the correct response, and then the next stimulus. Trials will repeat until you get all responses correct.

Once correct, the main display will return to the first screen. A results window will also open, showing your responses and the target stimulus-response pair. You can scroll across the responses looking for evidence of the typical error patterns, and can save the responses as a CSV file (which can be loaded into a spreadsheet).

When you are happy with how the software works, try the list "experiment 1" which is relatively short. If you want to try a more complex learning scenario, try the other "experiment N" lists.

3.2. Simulation: Protocol analysis

In this simulation, we look for evidence of patterns of errors in verbal-learning protocols. In particular, we are looking for evidence of stimulus generalisation, perseveration and oscillation.

Open the data file "anticipation-2.txt" in CHREST. The display will show, on the left, the list of pairs, ordered as stimulus and response. On the right is table, which will show the model’s subject protocol.

We will make the model a little more realistic: Under the menu option Model/Properties is a dialog to change CHREST’s parameters. The value for rho is the probability that CHREST will learn when it can. Change rho to 0.7, and then close the dialog.

To run the experiment:

  1. Select "random order" and give a 10-second inter-item time. Train CHREST on the data for a number of trials.

  2. Try to identify cases of stimulus generalisation, perseveration and oscillation.

  3. As the model is stochastic, you may need to repeat the simulation a few times before observing the results.

Exercises
  1. Construct a list with 6 very similar pairs of items. How long does CHREST take to learn the list? How many errors are made when CHREST responds with something other than NONE?

  2. Now construct a list with 6 very different pairs of items. Again, how long does CHREST take to learn the list? And how many non-NONE errors does it make?

3.3. Simulation: Pretraining

This simulation looks at the effects of prior knowledge of stimuli or responses on the time to learn a list.

To illustrate, given the following list of pairs:

B A J - J I D
C I G - K O L
D E J - L E V

We explore the following questions:

  1. Will learning the words BAJ, CIG, DEJ before learning the list affect the number of trials to learn the complete list of pairs?

  2. Will learning the words JID, KOL, LEV before learning the list affect the number of trials to learn the complete list of pairs?

  3. Is the effect of (1) better or worse than that of (2)?

Procedure
  1. Create a datafile containing a list of at least 10 stimulus-response pairs. All 10 responses should be different from the stimuli.

  2. Create a separate datafile containing just the stimuli, and another containing just the responses (labelled recognise-and-learn).

  3. Select values for rho and the times for presenting the list; these values should be constant throughout the experient.

  4. Train CHREST on the paired-associate list until it gets all responses correct. Record the number of trials. To avoid quirks of randomisation, you should repeat this experiment, perhaps 5 times, and calculate the average time. This is the U-U condition.

  5. Clear CHREST, and use the learn-and-recognise list of stimuli. Train CHREST on this list until it has learnt all the patterns. Now train CHREST on the paired-associate list again, until it gets all responses correct. Record the number of trials. Again, you should repeat this experiment to compute an average time. This is the F-U condition.

  6. Clear CHREST again, and use the learn-and-recognise list of responses. Train CHREST on this list until it has learnt all the patterns. Now train CHREST on the paired-associate list again, until it gets all responses correct. Record the number of trials. Again, you should repeat this experiment to compute an average time. This is the U-F condition.

  7. Clear CHREST again, and train CHREST on both the stimulus and response lists, until it is familiar with all the patterns. Now train CHREST on the paired-associate list again, until it gets all responses correct. Record the number of trials. Again, you should repeat this experiment to compute an average time. This is the F-F condition.

  8. Record the results in a table. You should compute the ratio of each condition with the number of trials for the F-F condition.

Human experiments

You can compare your results with those obtained from an experiment with people; Richman, Simon and Feigenbaum (2000) have a good summary of these experiments, and pointers to the results.

Condition People

F-F

1

U-F

1.2

F-U

1.6

U-U

1.8

3.4. Simulation: Interlist similarity

This simulation looks at the effect of interference between lists of learnt information. To illustrate, given the following two lists of paired-associate pairs:

B A J - J I D      J I L - B A K
C I G - K O L      G E D - P U W
D E J - L E V      L E M - W I J

First, we train the model on the first list until it gets every item correct. Second, we train on the second list, until it gets every item correct. finally, we see how long it takes to relearn the first list. To test the amount of interference, we make the elements of the second list more or less similar to elements of the first list.

Explore the following four cases:

  1. The stimuli of the second list are the same as the stimuli of the first list.

  2. The responses of the second list are the same as the responses of the first list.

  3. The stimuli of the second list agree in just the first letter with those of the first list.

  4. The responses of the second list agree in just the first letter with those of the first list.

Procedure
  1. Create datafiles for the first list and each of the above four conditions, with at least 5 pairs of words in each list.

  2. Select a value for rho and the times for presented the list: keep these constant through the experiment.

  3. For each of the four experiments:

    1. Train CHREST on the first list, until it gets all responses correct.

    2. Train CHREST on the second list, depending on the experiment, until it gets all responses correct.

    3. Train CHREST on the first list again, until it gets all responses correct. Record the required number of trials.

  4. To avoid quirks of randomisation, repeat the above steps 5 times and calculate the average number of required trials.

  5. Record the results in a table.

Summary

Which of the four cases was harder, in terms of needing more trials to learn? Can you explain in words why this should be the case, based on your knowledge of CHREST’s learning mechanisms? Compare your results with those obtained in experiments on humans; Richman, Simon and Feigenbaum (2000) provide a good summary and pointers to the literature.

3.5. Further Experiments

Bugelski’s experiment, about the inverse relation between item presentation time and required number of trials, can be performed by creating a reasonably sized dataset, and then training the model to complete success using varying values for the item presentation time.

Perhaps the best sources of information for further experiments in this paradigm and about CHREST’s predecessor, EPAM, are the papers:

  1. Feigenbaum, E. A., and Simon, H. A. 1984. EPAM-like models of recognition and learning. Cognitive Science 8:305–336.

  2. Richman, H. B.; Simon, H. A.; and Feigenbaum, E. A. 2002. Simulations of Paired Associate Learning using EPAM VI. Complex Information Processing, Working Paper #553. (see Papers/epam-vi.pdf)

The latter lists 20 different results within this paradigm, and demonstrates how EPAM (and consequently CHREST) provides a successful model for each.

4. Categorisation experiments

Open the data file "categorisation-five-four.txt". The display will show a list of pairs. The first pattern is the visual image, the set of features to learn from. These are a set of binary patterns coding the Brunsvik faces, which we met in the slides. The second pattern is a class, which will be treated as a verbal image. Finally, a tick box lets you choose which patterns to include in training or testing. Untick the patterns with class <X $>.

  1. Look at the binary patterns for the classes A and B. Can you think of a rule for deciding which are in A and which in B?

  2. Which of the patterns labelled X do you think should be in A and which in B?

  3. Using random order and a rho of 0.7, train the model on the data several times. Look at the pattern of responses for the patterns labelled X. The response patterns of A and B should vary.

  4. Does the model agree with your thoughts in part (2)?

  5. View the model - the light green numbers indicate cross-modal links, between the visual and verbal patterns. You will see nodes in both the visual and verbal short-term memories.

Exercise

Explore categorisation behaviour for the dataset "categorisation-harry-charlie.txt". (Harry/Charlie is another classic categorisation experiment - you can read about it through an online search.)

5. Perception and expertise in board games

There are some differences between the version of the Perceiver in the code and the description in the slides:

  1. The complete set of pieces in the field of view is retrieved and sorted, not just the piece in the centre.

  2. The only implemented heuristics are for random piece/item and LTM fixation.

    Recall

    the proportion of correct pieces placed on the board.

    Precision

    the proportion of pieces placed on the board which are correct.

    Errors of ommission

    the number of pieces which should have been placed but have not.

    Errors of commission

    pieces added to the recalled board, which should not be there.

Exercises

For these exercises, load the chess-positions.dat dataset.

  1. Create a network of 1000 nodes. Try recalling a position a few times. How do the precision/recall relate to each other?

  2. Look at the log. The chunks at the end are the ones used for the recall. Check that the chunks describe the items that are recalled by the model (on the recall tab).

  3. Also in the log is a list of the fixation types. How often is LTM used?

  4. Create a network of 50,000 nodes. Try recalling a position a few times. Are the precision/recall different to (1)?

  5. Again, look at the log. Is the proportion of LTM fixations different to (4)?

  6. Try to find an example of a template being used. (How many templates are in the larger network? You can find out under the model/information menu.)