Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metrics for curio-X series of projects #1

Open
yamins81 opened this issue Jan 14, 2021 · 1 comment
Open

metrics for curio-X series of projects #1

yamins81 opened this issue Jan 14, 2021 · 1 comment

Comments

@yamins81
Copy link

  1. define the metric Metric

  2. compute for pairs of conditios

m1 = Metric(set of trajectories for condition 1 [object1], also for set of alternatives)
m2 = Metric(set of trajectories for condition 2 [object1])

  1. compute linking function

linking_function(m1, m2) = probability of choice of m1

linking_function ~ softmax ??

  1. sample
    draw sample from the distrubiton of choices across a bunch of conditions to get trials

  2. correlate
    then compare model's trials to real huan trials and divide by squareroot of product of reliabilies for model and human

========but what specific set of metrics?

-- total variance in the velocity vector of object just after first collision

-- proportion of trials in which the dropped object comes to a steady collision state (e.g. suppporting)

-- some metric of the sharpness of transition in the response function of variability in velocity vector or positional dispersion as a function of initial state variability of dropped object

-- some metric of the sharpness of transition in the response function of variability in velocity vector or positional dispersion as a function of initial pose of the dropped object

-- maximum velocity of (dropped or target?) obtained during trajectory

-- net displacement of target

-- specificity of some feature above to pairing of objects e.g. for some pairs of obejcts, there will be especially more of a given feature relative to other pairs;

@mcfrank
Copy link

mcfrank commented Jan 15, 2021

  1. whether there is a big effect (summed velocity after collision)
  2. uncertainty in the size of effect (variance V after collison)
  3. interesting outcomes (proportion support/contain)
  4. skills-y-ness of the task (proposal: loss on linear function predicting final state from initial state)
  5. made the target move (summed velocity of target)
  6. uniqueness of outcome (can only get this effect for a particular pair)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants