Interfacing
experiments with an external environment
There are certain cases in which you may need to have more
on-line control of what Runner presents to subjects. For
example, you may want to determine on-line what the next
stimulus presented to subjects is based on their current
performance in the task, as is often the case in psychophysical
experiments (and many other situations). To accomplish this,
MouseTracker can interface on-line with an external application
or programming environment (e.g., MATLAB, Python, Java, etc.) to
receive information trial-by-trial on what stimulus (or stimulus
sequence) and response options to present to subjects, and then
also send back information on what the selected response was,
whether it was correct/incorrect, and what the trajectory's
initiation and response times were (so that the external
environment can then generate information for the next trial,
for MouseTracker to receive).
The interface is accomplished through the very simple
barebones protocol,
UDP (User Datagram Protocol). Thus, so long as
your external environment (e.g., MATLAB) can communicate over
UDP, it should be able to interface with MouseTracker.
UDP works over network socket ports. MouseTracker will
receive messages from the external environment via an input port
(e.g., 8000) and send messages to it via an output port (e.g.,
8001). Socket ports go up to 65535; to stay clear of other
applications' network use, it is advisable to select input and
output ports above 1024. It is also important to make sure the
ports are not blocked by a firewall. In this article and the
sample files, we use ports 8000 and 8001.
The external environment could be on the local computer
itself (e.g., MATLAB or Python running in the background) or it
could be on a remote computer that connects to the local
computer (running MouseTracker) via UDP. This allows for
possibilities where an environment on a remote "server" computer
(e.g., MATLAB) controls MouseTracker on two "client" computers,
e.g., if two subjects are participating in a competitive or
cooperative game. Thus, the UDP interface allows MouseTracker to
be controlled on a trial-by-trial basis by an external
environment running locally or remotely.
In the example later and in the sample MATLAB example, the
ports are as follows:
MouseTracker input port: 8000
MouseTracker output port: 8001
MATLAB input port: 8001
MATLAB output port: 8000
Setup: Activating UDP Interface (udpinterface) and
specifying trials to be interfaced
Add into your experiment .CSV file a new experimental
parameter, udpinterface. It specifies information about
the UDP interface in the following order:
ID number, input port, output port, [remote host] (optional)
The ID number is arbitrary, but just makes sure that MouseTracker and the external environment are on the same page.
This becomes important, however, if an environment on a remote
"server" computer is controlling MouseTracker on two "client"
computers, in which case the ID number serves to identify which
"client" computer's MouseTracker is communicating (e.g., Player
1 or Player 2).
When the external environment is simply running in the
background of the local computer, the following is sufficient:
If the external environment is on a remote computer, the
remote host (host name or IP address) is necessary:
In your stimuli/response list, you need to specify which
trials you would like to be specified by the external
environment on-line. To let MouseTracker know you'd like a trial
to be "to be determined", use trial-type 7 and you can simply
leave the rest of the information blank (as it will be
determined later by the external environment). Using the
randomize-block column ("rand" here) does not make a whole lot
of sense using udpinterface, because the external
environment is determining all the trials. Thus, randomization
should be done by the external environment, not MouseTracker.
For example, if you'd like an instruction screen to first
appear, followed by 10 trials to be determined by the external
environment, then a break screen, and then another 10 trials to
be determined by the external environment, you could specify the
following:
Note that because the external environment can specify
on-line a trial-type 0 as well, that you could in theory make
all these trials trial-type 7 (i.e., change the instruction and
break slides to trial-type 7), and then have the external
environment specify those as well. Whichever way works.
MouseTracker and the external environment will now
communicate by sending brief comma-separated messages via UDP
in a specific sequence (the sequence is maintained by a message
ID). When MouseTracker sends or receives information through UDP,
the message
should always start with:
[interface ID],[message
ID]
[interface ID] = The ID number
specified in the udpinterface parameter of the .CSV file
[message ID] = This is simply
the current index of how many messages have been sent/received
by MouseTracker. It is important so that MouseTracker and the
external environment make sure they are at the same step of the
experiment's sequence (e.g., to make sure that MouseTracker did
not miss a message intended for it from the external
environment). The very first message sent by MouseTracker has a
message ID of 1, and then MouseTracker waits for an incoming
message with a message ID of 2. It will then send out a message
with a message ID of 3, and then wait for a message with a
message ID of 4, and so on and so forth.
Step
1: Initial WAITING signal from MouseTracker
When MouseTracker has loaded the experiment successfully,
it sends out an initial waiting signal:
(The arbitrary number of 123
will be used to denote the interface ID for the remainder of the
article, for simplicity)
123,1,WAITING
This now lets the external environment know that
MouseTracker is loaded up and ready to begin.
Step 2:
START signal from external environment
MouseTracker will now wait for the external environment to load
up and be ready. When it is ready, the external environment
should send the following UDP message:
123,2,START
Step 3:
START confirmation from MouseTracker
MouseTracker will confirm that it has received the START signal
from the external environment by sending out the following
message to the external environment:
123,3,START
(Note how the message ID is continuously incrementing, to keep
pace with one another.)
At this point, any time that MouseTracker encounters a
trial-type 7 in the experiment .CSV stimuli/response list, it is
going to wait and probe the input UDP port for the appropriate
message ID. In the example above, the first trial is trial-type
0, the instructions screen. Once that finishes, the next trial
is trial-type 7, at which point MouseTracker will wait for a
message from the external environment. The first trial to be run
through the interface should therefore have a message ID of 4
(as the previous message--the START confirmation--had an ID of
3).
Step 4: External environment sends trial information
To tell MouseTracker how to run the trial (with trial-type 7),
the external environment should send a message with the
typical header plus the trial's comma-separated-value
information exactly as it would appear in the experiment .CSV
file (see
Creating Experiments | Stimuli and response list for
information on how to format this). For example:
123,4,2,MANGO,less_common-word,interface,1,2,FRUIT,VEGGIE
Here, "123" is the interface ID and "4" is the current message
ID. The rest is the trial information.
Thus, MouseTracker is going to load the current trial
(trial-type 7) as if the experiment .CSV had specified the
following comma-separated values:
2,MANGO,less_common-word,interface,1,2,FRUIT,VEGGIE. And
thus, MouseTracker will now treat this trial as a trial-type 2
(letter-string stimulus), will display the string "MANGO", will
assign the trial to condition "less_common-word", will display
"FRUIT" and "VEGGIE" response options, with the correct response
being "FRUIT" and incorrect being "VEGGIE". Note that
"interface" has been put in the randomize-block column.
Typically, without using udpinterface, a number would be
put here to tell MouseTracker to randomize the trial with a
certain block. Randomization is pretty meaningless here because
the external environment is the one deciding what each trial is.
Thus, the randomize-block column is not used and "interface" has
been put only because it is arbitrary (anything else could be
used).
Step 5: MouseTracker sends external environment response
information
Once the subject clicks on a response, MouseTracker sends a
message to let the external environment know
how the subject did. It sends it in the following format (using
comma-separated values):
Interface Header Selected Response #
Incorrect?
Initiation Time
Response Time
For example, when the subject completes the trial above,
MouseTracker would send a message such as:
123,5,1,0,194,1024
MouseTracker is letting the external environment know that on
the last trial the subject selected response #1, which was
correct (if incorrect, the value would be a 1, not 0), with an
initiation time of 194 ms, and a response time of 1024 ms. And
to be sure MouseTracker and the external environment are talking
about the same trial, the current message ID (5) would mean that
the last trial this is in reference to would have been specified
by a message ID of 4 (as in Step 4 above).
Now, on the next trial-type 7 encountered by MouseTracker
(e.g., the very next trial), it will wait again for more trial
information (repeat Step 4 above). In such a way, the external
environment can use previous trials' performance to guide what
the next trials will be (and Step 4 and Step 5 repeat until
there are no more trials with trial-type 7).
If you'd like to terminate the experiment adaptively, you
can have the external environment send an "END" signal to
MouseTracker. Just use the correct message ID and have the
external environment send "END" (e.g.,
123,119,END),
and MouseTracker will send an "END" signal back to the external
environment (see Step 6 below), save the data, and end.
Otherwise, MouseTraker will terminate after the last trial of
the experiment (see below).
Step 6: END
signal from MouseTracker
Once the last trial of the experiment runs, MouseTracker lets
the external environment know that the experiment has finished
by sending a final END signal. It then saves data and ends.
Using an arbitrary message ID of 120 (assuming there are more
messages sent/received across the experiment), MouseTracker
sends:
123,120,END
Viewing the udpinterface
log
While an experiment is running, you can view the current
udpinterface log, which contains all the messages
successfully sent and received through the interface. Just press
F9. This is especially useful for debugging purposes.
Sending
keystrokes directly to MouseTracker
There are some cases where you might need to send
keystrokes directly to MouseTracker from the external
environment, thereby mimicking the subject actually pressing the
keys. This can help with certain situations of critical timing.
For example, a compound-trial type 4 may be used to
synchronize MouseTracker's trial onset with an event known to
the external environment. As one example, imagine you would like
to present your word stimulus and have the subject start moving
the mouse the moment that an fMRI scanner, eye-tracker, or EEG
system begins recording or is ready for a trial's onset. One way
to do it is to have an external environment (e.g., MATLAB,
Python) communicate with that device. Using your own code, you
can wait for the device (e.g., fMRI scanner) to signal to the
environment (e.g., MATLAB) that the trial should begin (e.g., an
MRI trigger pulse). When this occurs, the environment can send a
keystroke signal directly to MouseTracker to present the subject
with the critical stimulus and have him/her start moving at that
moment.
Imagine a compound trial-type 4:
2/+/-1;2/CRITICAL-STIMULUS!/0. A duration of -1 means
that that step of the sequence requires the subject to hit the
Enter key to move forward onto the next step. So, in this trial,
the subject presses the "Start" button, at which point a
fixation cross appears indefinitely until the Enter key is
pressed. Once Enter is pressed, the critical stimulus is
presented. Rather than asking the subject to hit Enter, you can
have an external environment advance to the critical stimulus
the exact moment you would like it to (e.g., when the fMRI
scanner is ready). At the moment you would like to launch the
critical stimulus, have the external environment send the Enter
keystroke directly to MouseTracker. This can be done by using
the UDP interface with a Send-Keys (SK) bypass. You need the
interface ID but not a message ID---as this bypasses the
standard back-and-forth of messaging and immediately mimics
keystrokes in MouseTracker. Instead of a message ID, put in its
place "SK" to indicate that this is a Send-Keys bypass rather
than a standard message:
123,SK,{ENTER}
After the "SK" and a comma, you then can put the keystroke
string to send to MouseTracker. In this case, {ENTER} will send
an Enter keystroke. You can send any number of keystrokes. Here
is a full list.
MATLAB
Example: Adaptive Task
An example of an experiment interfaced with MATLAB is
provided in the sample folder. The Runner experiment is at
C:\MouseTracker\sample\adaptive_matlab.csv, and the
corresponding MATLAB script file is at C:\MouseTracker\sample\adaptive_matlab.m.
The MATLAB script is commented in detail, so that you can easily
understand how to generate your own interface using MATLAB (or
some other external environment).
This example task is silly and doesn't make a whole lot of
sense, but it provides a good demonstration of how to create
adaptive MouseTracker paradigms, where future trials are
determined by subjects' past responses or performance. Here,
subjects go through 10 trials. On each trial, a technology brand
(e.g., Microsoft) or a fashion brand (e.g., Gucci) is presented
as a word stimulus, and subjects need to decide how valuable the
brand is. On the first trial in the experiment, the two response
alternatives start as $100 and $200. If the subject selects the
larger alternative (e.g., $200), then on the next trial the two
response alternatives double in size ($200 and $400). If the
subject selects the smaller alternative (e.g., $100), then on
the next trial the two response alternatives are cut in half
($50 and $100).
Again, this experiment makes absolutely no sense and should
never be run on real subjects. It just was a fast way off the
top of my head to show how to create an adaptive MouseTracker
paradigm, using subjects' prior data to determine future trials.
For example, you could use past responses to determine future
stimuli, a current measure of accuracy/performance to determine
how easy/difficult future trials will be, or change future
stimuli/responses based on subjects' current initiation times or
response times, among other possibilities.
To understand the interface, it is best to check out the
experiment .CSV file and thoroughly go over the MATLAB script
file, looking at the code in tandem with the comments. You
should be able to generate an interface of your own in no time.
Good luck!