Skip to content

Commit

Permalink
update defs, add yes/no
Browse files Browse the repository at this point in the history
  • Loading branch information
@AndrewRWilliams
AndrewRWilliams committed Jan 23, 2025
1 parent 62d77d4 commit 406284d
Show file tree
Hide file tree
Showing 3 changed files with 113 additions and 41 deletions.
46 changes: 34 additions & 12 deletions annotation/index.html
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
<!DOCTYPE html>
<html lang="en">

<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>CIK Capability Annotations</title>
<link rel="stylesheet" href="styles.css">
</head>

<body>
<div id="container">
<h1 style="padding-bottom: 20px;">Context is Key Benchmark - Annotator Platform</h1>
Expand All @@ -28,7 +30,10 @@ <h1 style="padding-bottom: 20px;">Context is Key Benchmark - Annotator Platform<

<!-- Annotation Instruction -->
<p class="progress-instruction">
Please inspect the task below and annotate it by selecting which capabilities are required to infer the ground truth time series based on the provided contextual information and history. Before proceeding, please read the <a href="#" class="definition-link" onclick="showDefinitions()">definition</a> of each capability carefully.
Please inspect the task below and annotate it by selecting which capabilities are required to infer the
ground truth time series based on the provided contextual information and history. Before proceeding, please
read the <a href="#" class="definition-link" onclick="showDefinitions()">definition</a> of each capability
carefully.
Once you are done, please download the results and send us the json file.
</p>

Expand All @@ -41,9 +46,6 @@ <h3>Capabilities</h3>
Please select all that apply.
</p>
<form id="capabilities-form">
<label class="capability-checkbox">
<input type="checkbox" value="instruction following"> Instruction Following
</label>
<label class="capability-checkbox">
<input type="checkbox" value="retrieval: context"> Retrieval: Context
</label>
Expand All @@ -62,8 +64,19 @@ <h3>Capabilities</h3>
<label class="capability-checkbox">
<input type="checkbox" value="reasoning: causal"> Reasoning: Causal
</label>
<!-- New Yes/No Question -->
<div class="yes-no-question">
<p>Useful Context</p>
<label class="yes-no-option">
<input type="radio" name="test-question" value="yes"> Yes
</label>
<label class="yes-no-option">
<input type="radio" name="test-question" value="no"> No
</label>
</div>
<div class="button-container">
<button type="button" class="next-button" onclick="saveAnnotationsAndNext()" disabled>Save</button>
<button type="button" class="next-button" onclick="saveAnnotationsAndNext()"
disabled>Save</button>
<button type="button" class="skip-button" onclick="skipTask()">Skip</button>
</div>
</form>
Expand All @@ -77,18 +90,27 @@ <h3>Capabilities</h3>
<span class="close-button" onclick="closeDefinitions()">&times;</span>
<h2>Capability Definitions</h2>
<ul>
<li><strong>Instruction Following:</strong> Ability to execute tasks based on explicit instructions.</li>
<li><strong>Retrieval: Context:</strong> Extracting relevant information from external sources to contextualize the task.</li>
<li><strong>Retrieval: Memory:</strong> Leveraging stored or previously learned information to solve the task.</li>
<li><strong>Reasoning: Analogy:</strong> Drawing parallels between similar situations to deduce solutions.</li>
<li><strong>Reasoning: Deduction:</strong> Applying logical rules to derive conclusions from given premises.</li>
<li><strong>Reasoning: Math:</strong> Solving problems involving numerical and mathematical reasoning.</li>
<li><strong>Reasoning: Causal:</strong> Identifying and reasoning about cause-and-effect relationships.</li>
<li><strong>Retrieval: Context:</strong> Forecasts would rely on a human's ability to retrieve relevant
information from the context and distinguish it from irrelevant information.</li>
<li><strong>Retrieval: Memory:</strong> Forecasts would rely on facts from memory or a knowledge base,
such as relevant physical constants or quantitative laws.</li>
<li><strong>Reasoning: Deduction:</strong>Forecasts would rely on a human's ability to draw valid
inferences from the context, e.g. inferring from the context that certain values are logically
impossible to occur.</li>
<li><strong>Reasoning: Causal:</strong> Forecasts would rely on a human's ability to reason about cause
and effect from the context, such as reasoning about the impacts of various actions described in the
context.</li>
<li><strong>Reasoning: Math:</strong>Forecasts would rely on a human's ability to perform mathematics
over the context, such as arithmetic or solving equations.</li>
<li><strong>Reasoning: Analogy:</strong>Forecasts would rely on a human's ability to forecast by way of
analogy between entities or events, for instance, applying knowledge from a past event that is
similar to an upcoming one.</li>
</ul>
</div>
</div>

<!-- Include Scripts -->
<script src="./scripts.js"></script>
</body>

</html>
29 changes: 21 additions & 8 deletions annotation/scripts.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,24 +82,31 @@ function saveAnnotationsAndNext() {
return;
}

// Get selected capabilities
// Get the form
const form = document.getElementById("capabilities-form");

// Get selected capabilities (checkboxes)
const selectedCapabilities = Array.from(form.elements)
.filter(input => input.checked)
.filter(input => input.checked && input.type === "checkbox")
.map(input => input.value);

// Save annotations to the dictionary
// Get selected yes/no radio
const selectedRadio = Array.from(form.elements)
.find(input => input.checked && input.type === "radio")?.value;

// Save both sets of data into the dictionary
dictionary[currentTask].annotations = selectedCapabilities;
dictionary[currentTask].yesNoResponse = selectedRadio;

// Update localStorage
localStorage.setItem("cik_capability_annotations", JSON.stringify(dictionary));

// Reset checkboxes
// Reset form and re-check button state
form.reset();
toggleNextButtonState();

// Update the progress bar
// Update progress bar and move on
updateProgressBar();

// Proceed to the next task
openRandomUnannotatedTask();
}

Expand Down Expand Up @@ -221,21 +228,27 @@ function downloadJSON() {
function toggleNextButtonState() {
const form = document.getElementById("capabilities-form");
const checkboxes = Array.from(form.elements).filter(input => input.type === "checkbox");
const yesNoButtons = Array.from(form.elements).filter(input => input.type === "radio");
const nextButton = form.querySelector(".next-button");

// Enable the Next button if at least one checkbox is checked
const isAnyChecked = checkboxes.some(checkbox => checkbox.checked);
nextButton.disabled = !isAnyChecked; // Disable if none are checked
const isYesNoSelected = yesNoButtons.some(button => button.checked);
nextButton.disabled = !isAnyChecked || !isYesNoSelected; // Disable if no checkboxes are checked and no radio buttons are selected
}

// Attach event listeners to checkboxes to update the button state
function initializeCheckboxListeners() {
const form = document.getElementById("capabilities-form");
const checkboxes = Array.from(form.elements).filter(input => input.type === "checkbox");
const yesNoButtons = Array.from(form.elements).filter(input => input.type === "radio");

checkboxes.forEach(checkbox => {
checkbox.addEventListener("change", toggleNextButtonState);
});
yesNoButtons.forEach(button => {
button.addEventListener("change", toggleNextButtonState);
});

// Ensure the button state is updated on page load
toggleNextButtonState();
Expand Down
79 changes: 58 additions & 21 deletions annotation/styles.css
View file
Original file line number Diff line number Diff line change
Expand Up @@ -101,15 +101,24 @@ h2 {
.progress-instruction {
font-size: 18px;
font-weight: 500;
color: #555; /* Neutral dark gray for text */
background-color: #fff3cd; /* Light yellow background */
border: 1px solid #ffeeba; /* Slightly darker yellow border */
padding: 15px 20px; /* Add padding inside the box */
border-radius: 8px; /* Rounded corners */
box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1); /* Subtle shadow */
margin: 25px auto; /* Add spacing around and center the box */
max-width: 800px; /* Limit the width for a clean look */
text-align: left; /* Align text to the left */
color: #555;
/* Neutral dark gray for text */
background-color: #fff3cd;
/* Light yellow background */
border: 1px solid #ffeeba;
/* Slightly darker yellow border */
padding: 15px 20px;
/* Add padding inside the box */
border-radius: 8px;
/* Rounded corners */
box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
/* Subtle shadow */
margin: 25px auto;
/* Add spacing around and center the box */
max-width: 800px;
/* Limit the width for a clean look */
text-align: left;
/* Align text to the left */
}

.content-wrapper {
Expand Down Expand Up @@ -158,6 +167,22 @@ h2 {
font-weight: 500;
}

.capability-switch {
display: flex;
align-items: center;
margin: 10px 0;
}

.capability-switch input[type="checkbox"] {
margin-right: 10px;
}

.capability-switch label {
font-size: 16px;
font-weight: 500;
color: #333;
}

.next-button {
margin-top: 20px;
padding: 10px 20px;
Expand All @@ -175,10 +200,14 @@ h2 {
}

.next-button:disabled {
background-color: #d6d6d6; /* Light gray to show it's disabled */
color: #999; /* Muted text color */
cursor: not-allowed; /* Indicate the button cannot be clicked */
opacity: 0.6; /* Reduce opacity for a clearer disabled effect */
background-color: #d6d6d6;
/* Light gray to show it's disabled */
color: #999;
/* Muted text color */
cursor: not-allowed;
/* Indicate the button cannot be clicked */
opacity: 0.6;
/* Reduce opacity for a clearer disabled effect */
}

.skip-button {
Expand All @@ -199,15 +228,20 @@ h2 {

.button-container {
display: flex;
justify-content: center; /* Center both buttons horizontally */
gap: 10px; /* Add space between buttons */
margin-top: 20px; /* Add some spacing above the buttons */
justify-content: center;
/* Center both buttons horizontally */
gap: 10px;
/* Add space between buttons */
margin-top: 20px;
/* Add some spacing above the buttons */
}

.next-button,
.skip-button {
flex: 1; /* Ensure both buttons have equal width */
max-width: 150px; /* Optional: Limit maximum button width */
flex: 1;
/* Ensure both buttons have equal width */
max-width: 150px;
/* Optional: Limit maximum button width */
}

.modal {
Expand Down Expand Up @@ -265,7 +299,10 @@ h2 {
}

#container {
position: relative; /* Ensure proper positioning for scrolling */
overflow-y: auto; /* Enable vertical scrolling */
max-height: 100vh; /* Prevent overflow beyond the viewport */
position: relative;
/* Ensure proper positioning for scrolling */
overflow-y: auto;
/* Enable vertical scrolling */
max-height: 100vh;
/* Prevent overflow beyond the viewport */
}

0 comments on commit 406284d

Please sign in to comment.