MouseTracker Help

Jon Freeman
New York University



>> Running experiments

>> Creating experiments

>> Interfacing with an external environment

 Step 1: WAITING signal
 Step 2: START signal
 Step 3: START confirm
 Step 4: Trial input
 Step 5: Response output

 Step 6: END signal
 Viewing interface log
 MATLAB Example: Adaptive

>> About MouseTracker data

>> Analyzing data

>> Exporting data

>> Converting from MouseTracker 1.x


Interfacing experiments with an external environment

There are certain cases in which you may need to have more on-line control of what Runner presents to subjects. For example, you may want to determine on-line what the next stimulus presented to subjects is based on their current performance in the task, as is often the case in psychophysical experiments (and many other situations). To accomplish this, MouseTracker can interface on-line with an external application or programming environment (e.g., MATLAB, Python, Java, etc.) to receive information trial-by-trial on what stimulus (or stimulus sequence) and response options to present to subjects, and then also send back information on what the selected response was, whether it was correct/incorrect, and what the trajectory's initiation and response times were (so that the external environment can then generate information for the next trial, for MouseTracker to receive).

The interface is accomplished through the very simple barebones protocol, UDP (User Datagram Protocol). Thus, so long as your external environment (e.g., MATLAB) can communicate over UDP, it should be able to interface with MouseTracker.

UDP works over network socket ports. MouseTracker will receive messages from the external environment via an input port (e.g., 8000) and send messages to it via an output port (e.g., 8001). Socket ports go up to 65535; to stay clear of other applications' network use, it is advisable to select input and output ports above 1024. It is also important to make sure the ports are not blocked by a firewall. In this article and the sample files, we use ports 8000 and 8001.

The external environment could be on the local computer itself (e.g., MATLAB or Python running in the background) or it could be on a remote computer that connects to the local computer (running MouseTracker) via UDP. This allows for possibilities where an environment on a remote "server" computer (e.g., MATLAB) controls MouseTracker on two "client" computers, e.g., if two subjects are participating in a competitive or cooperative game. Thus, the UDP interface allows MouseTracker to be controlled on a trial-by-trial basis by an external environment running locally or remotely.

In the example later and in the sample MATLAB example, the ports are as follows:

MouseTracker input port: 8000
MouseTracker output port: 8001

MATLAB input port: 8001
MATLAB output port: 8000

Setup: Activating UDP Interface (udpinterface) and specifying trials to be interfaced

Add into your experiment .CSV file a new experimental parameter, udpinterface. It specifies information about the UDP interface in the following order:

ID number, input port, output port, [remote host] (optional)

The ID number is arbitrary, but just makes sure that MouseTracker and the external environment are on the same page. This becomes important, however, if an environment on a remote "server" computer is controlling MouseTracker on two "client" computers, in which case the ID number serves to identify which "client" computer's MouseTracker is communicating (e.g., Player 1 or Player 2).

When the external environment is simply running in the background of the local computer, the following is sufficient:

If the external environment is on a remote computer, the remote host (host name or IP address) is necessary:


In your stimuli/response list, you need to specify which trials you would like to be specified by the external environment on-line. To let MouseTracker know you'd like a trial to be "to be determined", use trial-type 7 and you can simply leave the rest of the information blank (as it will be determined later by the external environment). Using the randomize-block column ("rand" here) does not make a whole lot of sense using udpinterface, because the external environment is determining all the trials. Thus, randomization should be done by the external environment, not MouseTracker.

For example, if you'd like an instruction screen to first appear, followed by 10 trials to be determined by the external environment, then a break screen, and then another 10 trials to be determined by the external environment, you could specify the following:

Note that because the external environment can specify on-line a trial-type 0 as well, that you could in theory make all these trials trial-type 7 (i.e., change the instruction and break slides to trial-type 7), and then have the external environment specify those as well. Whichever way works.

MouseTracker and the external environment will now communicate by sending brief comma-separated messages via UDP in a specific sequence (the sequence is maintained by a message ID). When MouseTracker sends or receives information through UDP, the message should always start with:

[interface ID],[message ID]                      

[interface ID] = The ID number specified in the udpinterface parameter of the .CSV file

[message ID] = This is simply the current index of how many messages have been sent/received by MouseTracker. It is important so that MouseTracker and the external environment make sure they are at the same step of the experiment's sequence (e.g., to make sure that MouseTracker did not miss a message intended for it from the external environment). The very first message sent by MouseTracker has a message ID of 1, and then MouseTracker waits for an incoming message with a message ID of 2. It will then send out a message with a message ID of 3, and then wait for a message with a message ID of 4, and so on and so forth.

Step 1: Initial WAITING signal from MouseTracker

When MouseTracker has loaded the experiment successfully, it sends out an initial waiting signal:

(The arbitrary number of 123 will be used to denote the interface ID for the remainder of the article, for simplicity)


This now lets the external environment know that MouseTracker is loaded up and ready to begin.


Step 2: START signal from external environment

MouseTracker will now wait for the external environment to load up and be ready. When it is ready, the external environment should send the following UDP message:



Step 3: START confirmation from MouseTracker

MouseTracker will confirm that it has received the START signal from the external environment by sending out the following message to the external environment:


(Note how the message ID is continuously incrementing, to keep pace with one another.)

At this point, any time that MouseTracker encounters a trial-type 7 in the experiment .CSV stimuli/response list, it is going to wait and probe the input UDP port for the appropriate message ID. In the example above, the first trial is trial-type 0, the instructions screen. Once that finishes, the next trial is trial-type 7, at which point MouseTracker will wait for a message from the external environment. The first trial to be run through the interface should therefore have a message ID of 4 (as the previous message--the START confirmation--had an ID of 3).

Step 4: External environment sends trial information

To tell MouseTracker how to run the trial (with trial-type 7), the external environment should send a message with the typical header plus the trial's comma-separated-value information exactly as it would appear in the experiment .CSV file (see Creating Experiments | Stimuli and response list for information on how to format this). For example:


Here, "123" is the interface ID and "4" is the current message ID. The rest is the trial information.

Thus, MouseTracker is going to load the current trial (trial-type 7) as if the experiment .CSV had specified the following comma-separated values: 2,MANGO,less_common-word,interface,1,2,FRUIT,VEGGIE. And thus, MouseTracker will now treat this trial as a trial-type 2 (letter-string stimulus), will display the string "MANGO", will assign the trial to condition "less_common-word", will display "FRUIT" and "VEGGIE" response options, with the correct response being "FRUIT" and incorrect being "VEGGIE". Note that "interface" has been put in the randomize-block column. Typically, without using udpinterface, a number would be put here to tell MouseTracker to randomize the trial with a certain block. Randomization is pretty meaningless here because the external environment is the one deciding what each trial is. Thus, the randomize-block column is not used and "interface" has been put only because it is arbitrary (anything else could be used).

Step 5: MouseTracker sends external environment response information

Once the subject clicks on a response, MouseTracker sends a message to let the external environment know how the subject did. It sends it in the following format (using comma-separated values):

Interface Header            Selected Response #        Incorrect?        Initiation Time        Response Time

For example, when the subject completes the trial above, MouseTracker would send a message such as:


MouseTracker is letting the external environment know that on the last trial the subject selected response #1, which was correct (if incorrect, the value would be a 1, not 0), with an initiation time of 194 ms, and a response time of 1024 ms. And to be sure MouseTracker and the external environment are talking about the same trial, the current message ID (5) would mean that the last trial this is in reference to would have been specified by a message ID of 4 (as in Step 4 above).

Now, on the next trial-type 7 encountered by MouseTracker (e.g., the very next trial), it will wait again for more trial information (repeat Step 4 above). In such a way, the external environment can use previous trials' performance to guide what the next trials will be (and Step 4 and Step 5 repeat until there are no more trials with trial-type 7).

If you'd like to terminate the experiment adaptively, you can have the external environment send an "END" signal to MouseTracker. Just use the correct message ID and have the external environment send "END" (e.g., 123,119,END), and MouseTracker will send an "END" signal back to the external environment (see Step 6 below), save the data, and end. Otherwise, MouseTraker will terminate after the last trial of the experiment (see below). 

Step 6: END signal from MouseTracker

Once the last trial of the experiment runs, MouseTracker lets the external environment know that the experiment has finished by sending a final END signal. It then saves data and ends. Using an arbitrary message ID of 120 (assuming there are more messages sent/received across the experiment), MouseTracker sends:



Viewing the udpinterface log

While an experiment is running, you can view the current udpinterface log, which contains all the messages successfully sent and received through the interface. Just press F9. This is especially useful for debugging purposes.


Sending keystrokes directly to MouseTracker

There are some cases where you might need to send keystrokes directly to MouseTracker from the external environment, thereby mimicking the subject actually pressing the keys. This can help with certain situations of critical timing.

For example, a compound-trial type 4 may be used to synchronize MouseTracker's trial onset with an event known to the external environment. As one example, imagine you would like to present your word stimulus and have the subject start moving the mouse the moment that an fMRI scanner, eye-tracker, or EEG system begins recording or is ready for a trial's onset. One way to do it is to have an external environment (e.g., MATLAB, Python) communicate with that device. Using your own code, you can wait for the device (e.g., fMRI scanner) to signal to the environment (e.g., MATLAB) that the trial should begin (e.g., an MRI trigger pulse). When this occurs, the environment can send a keystroke signal directly to MouseTracker to present the subject with the critical stimulus and have him/her start moving at that moment. 

Imagine a compound trial-type 4: 2/+/-1;2/CRITICAL-STIMULUS!/0. A duration of -1 means that that step of the sequence requires the subject to hit the Enter key to move forward onto the next step. So, in this trial, the subject presses the "Start" button, at which point a fixation cross appears indefinitely until the Enter key is pressed. Once Enter is pressed, the critical stimulus is presented. Rather than asking the subject to hit Enter, you can have an external environment advance to the critical stimulus the exact moment you would like it to (e.g., when the fMRI scanner is ready). At the moment you would like to launch the critical stimulus, have the external environment send the Enter keystroke directly to MouseTracker. This can be done by using the UDP interface with a Send-Keys (SK) bypass. You need the interface ID but not a message ID---as this bypasses the standard back-and-forth of messaging and immediately mimics keystrokes in MouseTracker. Instead of a message ID, put in its place "SK" to indicate that this is a Send-Keys bypass rather than a standard message:


After the "SK" and a comma, you then can put the keystroke string to send to MouseTracker. In this case, {ENTER} will send an Enter keystroke. You can send any number of keystrokes. Here is a full list.


MATLAB Example: Adaptive Task

An example of an experiment interfaced with MATLAB is provided in the sample folder. The Runner experiment is at C:\MouseTracker\sample\adaptive_matlab.csv, and the corresponding MATLAB script file is at C:\MouseTracker\sample\adaptive_matlab.m. The MATLAB script is commented in detail, so that you can easily understand how to generate your own interface using MATLAB (or some other external environment).

This example task is silly and doesn't make a whole lot of sense, but it provides a good demonstration of how to create adaptive MouseTracker paradigms, where future trials are determined by subjects' past responses or performance. Here, subjects go through 10 trials. On each trial, a technology brand (e.g., Microsoft) or a fashion brand (e.g., Gucci) is presented as a word stimulus, and subjects need to decide how valuable the brand is. On the first trial in the experiment, the two response alternatives start as $100 and $200. If the subject selects the larger alternative (e.g., $200), then on the next trial the two response alternatives double in size ($200 and $400). If the subject selects the smaller alternative (e.g., $100), then on the next trial the two response alternatives are cut in half ($50 and $100).

Again, this experiment makes absolutely no sense and should never be run on real subjects. It just was a fast way off the top of my head to show how to create an adaptive MouseTracker paradigm, using subjects' prior data to determine future trials. For example, you could use past responses to determine future stimuli, a current measure of accuracy/performance to determine how easy/difficult future trials will be, or change future stimuli/responses based on subjects' current initiation times or response times, among other possibilities.

To understand the interface, it is best to check out the experiment .CSV file and thoroughly go over the MATLAB script file, looking at the code in tandem with the comments. You should be able to generate an interface of your own in no time.

Good luck!