Interactive experiment with multiple objects on screen

[matt-erhart]

Amazing work here by the way, looking forward to using it in an up coming pilot! I've done very little js, but a lot of programming (of experiments), so jspsych is great for my current situation.

I'm going to be building an experiment that plays 9 different audio clips when subjects place the mouse in 9 different locations on screen: a 3x3 grid with a novel visual object in each cell. I'll need to trigger the sound by mouse x,y coordinates (doing eye tracking too), not by mouseover. Their task is to learn all the names of the novel objects. They will then go to a test screen with the objects in randomized order and will click on the object they hear. I plan to figure this out on my own, so no expectations here, but do you have any recommendations on how to best approach this in jspsych? Perhaps, a trial is created on each mouse over during a training screen? For the testing screen, they will need to make 9 clicks; what's the best way to store that kind of response? Happy to turn this into a plugin if I can get it working well.

thanks,
Matt

[jodeleeuw]

For the training part, you're probably going to need a new plugin since the way that people are interacting with the trial is pretty unique. My recommendation would be to try coding that screen without jsPsych first. Once you get it working, it's easy to take the code you've generated and convert it to a jsPsych plugin.

For the test trials, I'd recommend combining the single-audio plugin with the button-response plugin (currently on the /dev branch of the github repo; will be released in the next version). Use the single-audio plugin to play the sound, and then the button-response plugin to show the 9 objects and allow them to make a choice. You can use the prompt parameter of the single-audio plugin to display the objects while the sound is playing, but they won't be clickable until the button-response trial starts. Repeat the sequence 9 times for the different object names, treating each name as a new trial. The drawback of this approach is that they won't be able to click on the objects while the name is playing. If that's important for your design, you'll have to modify the button-response plugin to play audio.

[jodeleeuw]

Not yet, but it's not a difficult thing to add to an existing plugin. JavaScript has a pretty straightforward set of events for capturing mouse location data. Here's a useful article: https://developer.mozilla.org/en-US/docs/Web/API/Element/mousemove_event.

I'd love to incorporate mouse tracking plugins, so if you decide to develop one please keep me in the loop!

[GEJ1]

Thank you very much! I'll be working on that.

[GEJ1]

Hello again @jodeleeuw ,

I take your advice and was really useful. Now i'm trying to draw over an image in a trial of jsPsych but I can't do it. I write this code but have problems trying to put in a jsPsych trial: https://github.com/GEJ1/jsPsych_TMT/blob/master/mouse_tracking_v01.html

I'm using the jspsych-psychophysics plugin and now I can show an image and save the mouse coordinates.

I think that my problem it's with canvas and I found that you respond a similar question here: https://groups.google.com/forum/#!topic/jspsych/n_jeN3py0YA , so I try to use the The on_load callback but without success, haha.

My goal is to show an image, can draw over it and save mouse coordinates.

How do you think will be a good way to put this in a jsPsych trial?

Thank you very much in advance!!

Gustavo.

[vijaymarupudi]

Gustavo,

In order to do this, in a custom plugin (or a modification of an existing plugin), you will need to render a canvas element, insert an image into it using the drawImage() method. Then you'll need to use document.addEventListener (mousedown, mousemove, mouseup) to track the points the user is drawing. You can use the lineTo() method to provide visual feedback.

I've done this before in a clustering plugin, where participants had to draw clusters around points. Here's the code for that.

function getXY(event, canvas) {
  const x = event.clientX - canvas.offsetLeft;
  const y = event.clientY - canvas.offsetTop;
  return [x, y];
}

function clusterAttempt (displayElement, trial, onComplete, onSingleCluster) {
  const stimulus = trial.stimulus

  const canvas = document.createElement("canvas")
  canvas.setAttribute('width', 800)
  canvas.setAttribute('height', 500)
  canvas.classList.add("clustering-canvas") // for border
  displayElement.innerHTML = '' // clear whatever was in there
  displayElement.appendChild(canvas)
  const ctx = canvas.getContext("2d");

  // Visual display state
  const clusterPaths = []
  let currentlyDrawing = false
  let currentClusterPath = new Path2D()

  // Event listener state
  let removeAllEventListeners = null // will be set after listeners are created.

  // Data state
  const clustersData = [];
  let currentClusterPointsData = [];



  // Data
  const metadata = {
    startDateTime: Date.now(),
    startTimestamp: performance.now(),
    endTimestamp: null
  };


  function addClusterPoint(x, y) {
    currentClusterPointsData.push({
      x,
      y,
      timestamp: performance.now()
    });
  }

  function finishClusterPoint(x, y) {
    addClusterPoint(x, y);
    clustersData.push(currentClusterPointsData);
    currentClusterPointsData = [];
  }

  function recolorPoints () {
    for (const { x, y } of stimulus.points) {
      for (const path of clusterPaths) {
        if (ctx.isPointInPath(path, x, y)) {
          ctx.save();
          ctx.fillStyle = "#0000FF";
          ctx.beginPath();
          ctx.arc(x, y, 5, 0, Math.PI * 2);
          ctx.fill();
          ctx.restore();
        }
      }
    }
  }

  function completeTrial (data) {
    removeAllEventListeners()
    onComplete(data) 
  }

  function completeSingleClusterTrial() {
    removeAllEventListeners()
    onSingleCluster()
  }

  function completionCheck () {
    const everyPointInSomePath = stimulus.points.every(({ x, y }) => {
      return clusterPaths.some(path =>
        ctx.isPointInPath(path, x, y)
      );
    });

    if (everyPointInSomePath) {
      metadata.endTimestamp = performance.now();
      const completionData = metadata;
      completionData.stimulus = stimulus;
      completionData.clusters = [];
      for (let i = 0; i < clustersData.length; i++) {
        const clusterPath = clusterPaths[i];
        const clusterInfo = {
          edgePoints: clustersData[i],
          pointMembership: stimulus.points.map(point => ({
            point: point,
            member: ctx.isPointInPath(clusterPath, point.x, point.y)
          }))
        };
        completionData.clusters.push(clusterInfo);
      }

      // Prevent single clusters
      if (completionData.clusters.length === 1) {
        completeSingleClusterTrial()
      } else {
        completeTrial(completionData);
      }
    }
  }

  function mouseDownListener (e) {
    currentlyDrawing = true;
    const [x, y] = getXY(e, canvas);
    addClusterPoint(x, y);
    ctx.beginPath();
    ctx.moveTo(x, y);
    currentClusterPath.moveTo(x, y);
  }

  function mouseMoveListener (e) {
    if (currentlyDrawing) {
      const [x, y] = getXY(e, canvas);
      addClusterPoint(x, y);
      ctx.lineTo(x, y);
      ctx.stroke();
      currentClusterPath.lineTo(x, y);
    }
  }

  function mouseUpListener (e) {
    const [x, y] = getXY(e, canvas);
    finishClusterPoint(x, y);
    ctx.closePath();
    ctx.stroke();
    currentClusterPath.closePath(x, y);
    clusterPaths.push(currentClusterPath);
    currentClusterPath = new Path2D();
    currentlyDrawing = false;
    recolorPoints();
    completionCheck();
  }

  ctx.lineWidth = 5;
  for (const { x, y } of stimulus.points) {
    ctx.beginPath();
    ctx.arc(x, y, 5, 0, Math.PI * 2);
    ctx.fill();
  }
  ctx.beginPath();

  document.addEventListener("mousedown", mouseDownListener);
  document.addEventListener("mousemove", mouseMoveListener);
  document.addEventListener("mouseup", mouseUpListener);

  // to be used when ending the trial
  removeAllEventListeners = () => {
    document.removeEventListener("mousedown", mouseDownListener);
    document.removeEventListener("mousemove", mouseMoveListener);
    document.removeEventListener("mouseup", mouseUpListener);
  }

}

function trialFunction(displayElement, trial) {

  let numberOfTries = 0;

  const onComplete = data => {
    numberOfTries += 1
    data = { numberOfTries, ...data }
    displayElement.innerHTML = `<p>You have completed the trial</p>
      <div><button>Continue</button></div>`
    displayElement.querySelector('button').addEventListener('click', () => {
      // for csv output
      jsPsych.finishTrial({ clusteringData: JSON.stringify(data) })
    })
  }

  const onSingleCluster = () => {
    numberOfTries += 1
    const p = document.createElement('p')
    p.classList.add("single-cluster-warning")
    p.innerText = "WARNING: PLEASE USE MORE THAN ONE CLUSTER."
    const continueDiv = document.createElement('div')
    continueDiv.innerHTML = `<button>Retry</button>`
    continueDiv.addEventListener('click', () => {
      clusterAttempt(displayElement, trial, onComplete, onSingleCluster)
    })
    displayElement.innerHTML = ''
    displayElement.appendChild(p)
    displayElement.appendChild(continueDiv)
  }
  
  // Start trial

  displayElement.innerHTML = `<p>Click the button to start the trial</p>
    <button>Begin</button>`
  displayElement.querySelector('button').addEventListener('click', () => {
    clusterAttempt(displayElement, trial, onComplete, onSingleCluster)
  })
}

const plugin = {}

plugin.info = {
  name: "clustering",
  parameters: {}
}

plugin.trial = trialFunction

[GEJ1]

Thank you very much @vijaymarupudi , your answer was really useful!