Expand source code -Browse git +Browse git
"""
.. include:: ../docs/about/annotate.md
diff --git a/tf/about/annotateBrowser.html b/tf/about/annotateBrowser.html
index 3f7ea99e6..33ee0370e 100644
--- a/tf/about/annotateBrowser.html
+++ b/tf/about/annotateBrowser.html
@@ -267,7 +267,7 @@ Programming
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/annotateBrowser.md
diff --git a/tf/about/apps.html b/tf/about/apps.html
index 24100294b..2944bb6fb 100644
--- a/tf/about/apps.html
+++ b/tf/about/apps.html
@@ -115,7 +115,7 @@ Two contexts
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/apps.md
diff --git a/tf/about/background.html b/tf/about/background.html
index 3b63c1713..28c4341e2 100644
--- a/tf/about/background.html
+++ b/tf/about/background.html
@@ -155,7 +155,7 @@ History
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/background.md
diff --git a/tf/about/browser.html b/tf/about/browser.html
index 63f59bd26..142d489c9 100644
--- a/tf/about/browser.html
+++ b/tf/about/browser.html
@@ -174,7 +174,7 @@ UNICODE in Excel CSVs
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/browser.md
diff --git a/tf/about/clientmanual.html b/tf/about/clientmanual.html
index 2657a5551..cbfa4bef7 100644
--- a/tf/about/clientmanual.html
+++ b/tf/about/clientmanual.html
@@ -565,7 +565,7 @@ Credits
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/clientmanual.md
diff --git a/tf/about/code.html b/tf/about/code.html
index 81bebdf8f..d1f41ca1f 100644
--- a/tf/about/code.html
+++ b/tf/about/code.html
@@ -98,7 +98,7 @@ Writing
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/code.md
diff --git a/tf/about/corpora.html b/tf/about/corpora.html
index e4574a7e5..8ca3f514e 100644
--- a/tf/about/corpora.html
+++ b/tf/about/corpora.html
@@ -355,7 +355,7 @@ Extra data
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/corpora.md
diff --git a/tf/about/datamodel.html b/tf/about/datamodel.html
index 83253e3af..5961c6ad6 100644
--- a/tf/about/datamodel.html
+++ b/tf/about/datamodel.html
@@ -265,7 +265,7 @@ Serializing and pre-computing
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/datamodel.md
diff --git a/tf/about/datasharing.html b/tf/about/datasharing.html
index 3e31c0f03..16a27216c 100644
--- a/tf/about/datasharing.html
+++ b/tf/about/datasharing.html
@@ -362,7 +362,7 @@ More modules at the same time
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/datasharing.md
diff --git a/tf/about/displaydesign.html b/tf/about/displaydesign.html
index 413854437..8e7eebe17 100644
--- a/tf/about/displaydesign.html
+++ b/tf/about/displaydesign.html
@@ -151,7 +151,7 @@ Output
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/displaydesign.md
diff --git a/tf/about/faq.html b/tf/about/faq.html
index d32e66656..195089740 100644
--- a/tf/about/faq.html
+++ b/tf/about/faq.html
@@ -161,7 +161,7 @@ GitHub Rate Limit Exceeded!
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/faq.md
diff --git a/tf/about/fileformats.html b/tf/about/fileformats.html
index d5621fe34..ec9a44740 100644
--- a/tf/about/fileformats.html
+++ b/tf/about/fileformats.html
@@ -158,7 +158,7 @@ Single values
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/fileformats.md
diff --git a/tf/about/index.html b/tf/about/index.html
index 0a131a4f1..996bd7db6 100644
--- a/tf/about/index.html
+++ b/tf/about/index.html
@@ -33,7 +33,7 @@ Documents
Expand source code
-Browse git
+Browse git
"""
# Documents
diff --git a/tf/about/install.html b/tf/about/install.html
index 766098dad..24fda616c 100644
--- a/tf/about/install.html
+++ b/tf/about/install.html
@@ -108,7 +108,7 @@ Note for Linux users
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/install.md
diff --git a/tf/about/manual.html b/tf/about/manual.html
index 7ca8542f3..6cfd017a3 100644
--- a/tf/about/manual.html
+++ b/tf/about/manual.html
@@ -390,7 +390,7 @@ Keyboard shortcuts
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/manual.md
diff --git a/tf/about/optimizations.html b/tf/about/optimizations.html
index 599c25653..b192b7305 100644
--- a/tf/about/optimizations.html
+++ b/tf/about/optimizations.html
@@ -187,7 +187,7 @@ Edge features
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/optimizations.md
diff --git a/tf/about/releases.html b/tf/about/releases.html
index c73158cbd..920010187 100644
--- a/tf/about/releases.html
+++ b/tf/about/releases.html
@@ -43,6 +43,16 @@ Release notes
12
12.5
+12.5.4
+2024-09-16
+Many changes in NER, TEI conversion, WATM generation, IIIF manifest generation.
+Documentation on these points may lag behind.
+Also: when the TF browser starts, it selects a port to listen to.
+For each datasource, a different port number is chosen.
+However, different versions of the same datasource were assigned the same version.
+Now the version number is taken into account when selecting a port number, so
+that you can have a TF brtowser running on different versions of the same datasource
+at the same time.
12.5.3
2024-07-05
NER
@@ -528,7 +538,7 @@ Older releases
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/releases.md
@@ -602,6 +612,7 @@ Index
Release notes
- 12
- 12.5
+- 12.5.4
- 12.5.3
- 12.5.2
- 12.5.1
diff --git a/tf/about/releasesold.html b/tf/about/releasesold.html
index cd349d33a..127899e7e 100644
--- a/tf/about/releasesold.html
+++ b/tf/about/releasesold.html
@@ -3430,7 +3430,7 @@ Changed
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/releasesold.md
diff --git a/tf/about/searchdesign.html b/tf/about/searchdesign.html
index 4a5dd3a3b..0a3f66a11 100644
--- a/tf/about/searchdesign.html
+++ b/tf/about/searchdesign.html
@@ -477,7 +477,7 @@ Small-first strategy
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/searchdesign.md
diff --git a/tf/about/searchusage.html b/tf/about/searchusage.html
index c51aaa20d..bbae07777 100644
--- a/tf/about/searchusage.html
+++ b/tf/about/searchusage.html
@@ -776,7 +776,7 @@ Implementation
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/searchusage.md
diff --git a/tf/about/tests.html b/tf/about/tests.html
index 3d7ea6f23..42bd67cf6 100644
--- a/tf/about/tests.html
+++ b/tf/about/tests.html
@@ -71,7 +71,7 @@ Relations
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/tests.md
diff --git a/tf/about/use.html b/tf/about/use.html
index afdd6e6d3..57fd34739 100644
--- a/tf/about/use.html
+++ b/tf/about/use.html
@@ -90,7 +90,7 @@ TF API
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/use.md
diff --git a/tf/about/usefunc.html b/tf/about/usefunc.html
index 526ee6786..141f4cdf1 100644
--- a/tf/about/usefunc.html
+++ b/tf/about/usefunc.html
@@ -419,7 +419,7 @@ Prevent data loading
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/usefunc.md
diff --git a/tf/about/variants.html b/tf/about/variants.html
index e9722efa3..bad54cd6a 100644
--- a/tf/about/variants.html
+++ b/tf/about/variants.html
@@ -438,7 +438,7 @@ The stack
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/variants.md
diff --git a/tf/about/volumes.html b/tf/about/volumes.html
index 861d83f27..7e7f7f168 100644
--- a/tf/about/volumes.html
+++ b/tf/about/volumes.html
@@ -323,7 +323,7 @@ Reflection
Expand source code
-Browse git
+Browse git
"""
.. include:: ../docs/about/volumes.md
diff --git a/tf/advanced/annotate.html b/tf/advanced/annotate.html
index 99911b2f7..c149f796c 100644
--- a/tf/advanced/annotate.html
+++ b/tf/advanced/annotate.html
@@ -34,7 +34,7 @@ Module tf.advanced.annotate
Expand source code
-Browse git
+Browse git
"""
Enable manual annotation APIs.
diff --git a/tf/advanced/app.html b/tf/advanced/app.html
index dc456f455..4f3aeebc0 100644
--- a/tf/advanced/app.html
+++ b/tf/advanced/app.html
@@ -31,7 +31,7 @@ Module tf.advanced.app
Expand source code
-Browse git
+Browse git
import types
import traceback
@@ -990,7 +990,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class App:
def __init__(
diff --git a/tf/advanced/condense.html b/tf/advanced/condense.html
index bda22914e..a3521e982 100644
--- a/tf/advanced/condense.html
+++ b/tf/advanced/condense.html
@@ -31,7 +31,7 @@ Module tf.advanced.condense
Expand source code
-Browse git
+Browse git
def condense(api, tuples, condenseType, multiple=False):
F = api.F
diff --git a/tf/advanced/data.html b/tf/advanced/data.html
index fd94ed2f5..625b5bbb4 100644
--- a/tf/advanced/data.html
+++ b/tf/advanced/data.html
@@ -31,7 +31,7 @@ Module tf.advanced.data
Expand source code
-Browse git
+Browse git
from ..core.helpers import itemize
from ..core.files import backendRep, expandDir, prefixSlash, normpath
@@ -533,7 +533,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class AppData:
def __init__(
diff --git a/tf/advanced/display.html b/tf/advanced/display.html
index d75ce9f32..edeebccd9 100644
--- a/tf/advanced/display.html
+++ b/tf/advanced/display.html
@@ -69,7 +69,7 @@ See also
Expand source code
-Browse git
+Browse git
"""
# Display
diff --git a/tf/advanced/find.html b/tf/advanced/find.html
index 4f3c7584f..a2af8d469 100644
--- a/tf/advanced/find.html
+++ b/tf/advanced/find.html
@@ -31,7 +31,7 @@ Module tf.advanced.find
Expand source code
-Browse git
+Browse git
import sys
from importlib import util
diff --git a/tf/advanced/helpers.html b/tf/advanced/helpers.html
index 68b012453..463a406ef 100644
--- a/tf/advanced/helpers.html
+++ b/tf/advanced/helpers.html
@@ -31,7 +31,7 @@ Module tf.advanced.helpers
Expand source code
-Browse git
+Browse git
import collections
from textwrap import dedent
diff --git a/tf/advanced/highlight.html b/tf/advanced/highlight.html
index 020da8759..66b0765df 100644
--- a/tf/advanced/highlight.html
+++ b/tf/advanced/highlight.html
@@ -31,7 +31,7 @@ Module tf.advanced.highlight
Expand source code
-Browse git
+Browse git
from .search import runSearch
diff --git a/tf/advanced/index.html b/tf/advanced/index.html
index aa596c099..14677d4b0 100644
--- a/tf/advanced/index.html
+++ b/tf/advanced/index.html
@@ -66,7 +66,7 @@ Advanced API
Expand source code
-Browse git
+Browse git
"""
# Advanced API
diff --git a/tf/advanced/interchange.html b/tf/advanced/interchange.html
index 49b9040f3..435edf852 100644
--- a/tf/advanced/interchange.html
+++ b/tf/advanced/interchange.html
@@ -35,7 +35,7 @@ Module tf.advanced.interchange
Expand source code
-Browse git
+Browse git
"""
Produce exports of the whole dataset in different formats.
diff --git a/tf/advanced/links.html b/tf/advanced/links.html
index c74b69e9c..ea64dead3 100644
--- a/tf/advanced/links.html
+++ b/tf/advanced/links.html
@@ -32,7 +32,7 @@ Module tf.advanced.links
Expand source code
-Browse git
+Browse git
"""
Produce links to TF data and links from nodes to web resources.
diff --git a/tf/advanced/options.html b/tf/advanced/options.html
index f281f2a51..b77043b30 100644
--- a/tf/advanced/options.html
+++ b/tf/advanced/options.html
@@ -392,7 +392,7 @@ Parameters
Expand source code
-Browse git
+Browse git
"""
# Display Settings.
@@ -1144,7 +1144,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Options:
def __init__(self, app):
@@ -1366,7 +1366,7 @@ Methods
Expand source code
-Browse git
+Browse git
class OptionsCurrent:
def __init__(self, options):
diff --git a/tf/advanced/render.html b/tf/advanced/render.html
index e83f58d10..40fd186a1 100644
--- a/tf/advanced/render.html
+++ b/tf/advanced/render.html
@@ -45,7 +45,7 @@ Information shielding
Expand source code
-Browse git
+Browse git
"""
# Render
diff --git a/tf/advanced/repo.html b/tf/advanced/repo.html
index 4c705e09f..067a42fde 100644
--- a/tf/advanced/repo.html
+++ b/tf/advanced/repo.html
@@ -295,7 +295,7 @@ Updating a corpus that you alre
Expand source code
-Browse git
+Browse git
"""
# Auto downloading from a back-end repository
@@ -2791,7 +2791,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Checkout:
"""Auxiliary class for `checkoutRepo`"""
@@ -4062,7 +4062,7 @@ Methods
Expand source code
-Browse git
+Browse git
class Repo:
"""Auxiliary class for `releaseData`"""
diff --git a/tf/advanced/search.html b/tf/advanced/search.html
index 67d408d12..e35d05c6d 100644
--- a/tf/advanced/search.html
+++ b/tf/advanced/search.html
@@ -32,7 +32,7 @@ Module tf.advanced.search
Expand source code
-Browse git
+Browse git
"""
Calls from the advanced API to the Search API.
diff --git a/tf/advanced/sections.html b/tf/advanced/sections.html
index bda4573e4..0ab78dfc7 100644
--- a/tf/advanced/sections.html
+++ b/tf/advanced/sections.html
@@ -49,7 +49,7 @@ Structure
Expand source code
-Browse git
+Browse git
"""Section
diff --git a/tf/advanced/settings.html b/tf/advanced/settings.html
index a32e15785..f1db06370 100644
--- a/tf/advanced/settings.html
+++ b/tf/advanced/settings.html
@@ -844,7 +844,7 @@ writing
Expand source code
-Browse git
+Browse git
"""
# App settings
@@ -2936,7 +2936,7 @@ Classes
Expand source code
-Browse git
+Browse git
class AppCurrent:
def __init__(self, specs):
@@ -2987,7 +2987,7 @@ Methods
Expand source code
-Browse git
+Browse git
class Check:
def __init__(self, app, withApi):
diff --git a/tf/advanced/tables.html b/tf/advanced/tables.html
index a7fc0f590..7a6446708 100644
--- a/tf/advanced/tables.html
+++ b/tf/advanced/tables.html
@@ -31,7 +31,7 @@ Module tf.advanced.tables
Expand source code
-Browse git
+Browse git
from ..core.text import DEFAULT_FORMAT
from .helpers import getHeaders, RESULT
diff --git a/tf/advanced/text.html b/tf/advanced/text.html
index d279897f6..589701b8c 100644
--- a/tf/advanced/text.html
+++ b/tf/advanced/text.html
@@ -31,7 +31,7 @@ Module tf.advanced.text
Expand source code
-Browse git
+Browse git
import types
from textwrap import dedent
diff --git a/tf/advanced/unravel.html b/tf/advanced/unravel.html
index 930ef9359..0a20a706f 100644
--- a/tf/advanced/unravel.html
+++ b/tf/advanced/unravel.html
@@ -35,7 +35,7 @@ Unravel
Expand source code
-Browse git
+Browse git
"""
# Unravel
@@ -758,7 +758,7 @@ Classes
Expand source code
-Browse git
+Browse git
class NodeProps:
"""Node properties during plain() or pretty().
@@ -851,7 +851,7 @@ Instance variables
Expand source code
-Browse git
+Browse git
class OuterSettings:
"""Common properties during plain() and pretty().
@@ -933,7 +933,7 @@ Instance variables
Expand source code
-Browse git
+Browse git
class TreeInfo:
"""Tree properties during plain() or pretty().
diff --git a/tf/advanced/volumes.html b/tf/advanced/volumes.html
index dd9fa31a8..d2755b1e5 100644
--- a/tf/advanced/volumes.html
+++ b/tf/advanced/volumes.html
@@ -32,7 +32,7 @@ Module tf.advanced.volumes
Expand source code
-Browse git
+Browse git
"""
Produce links to TF data and links from nodes to web resources.
diff --git a/tf/advanced/zipdata.html b/tf/advanced/zipdata.html
index 480374e6e..3c8060838 100644
--- a/tf/advanced/zipdata.html
+++ b/tf/advanced/zipdata.html
@@ -31,7 +31,7 @@ Module tf.advanced.zipdata
Expand source code
-Browse git
+Browse git
import sys
import types
diff --git a/tf/app.html b/tf/app.html
index 813fc69fe..6f994d8aa 100644
--- a/tf/app.html
+++ b/tf/app.html
@@ -39,7 +39,7 @@ Module tf.app
Expand source code
-Browse git
+Browse git
"""
Make use of a corpus.
diff --git a/tf/browser/command.html b/tf/browser/command.html
index 7a56f4041..35753b2ac 100644
--- a/tf/browser/command.html
+++ b/tf/browser/command.html
@@ -32,7 +32,7 @@ Module tf.browser.command
Expand source code
-Browse git
+Browse git
"""
Command-line argument processing
@@ -55,6 +55,7 @@ Module tf.browser.command
(appName, checkoutApp, dataLoc) = argParam(cargs)
backend = _argCollect("backend", cargs)
checkout = _argCollect("checkout", cargs)
+ version = _argCollect("version", cargs)
dataRelative = _argCollect("relative", cargs)
if (
@@ -70,7 +71,7 @@ Module tf.browser.command
checkout = "clone"
if simple:
- return appName
+ return f"{appName}:{version}"
locations = _argCollect("locations", cargs)
modules = _argCollect("modules", cargs)
diff --git a/tf/browser/html.html b/tf/browser/html.html
index cac17ff80..31954f7d8 100644
--- a/tf/browser/html.html
+++ b/tf/browser/html.html
@@ -34,7 +34,7 @@ Module tf.browser.html
Expand source code
-Browse git
+Browse git
"""HTML generation done in the Pythonic way.
@@ -63,6 +63,9 @@ Module tf.browser.html
span
style
summary
+ table
+ td
+ tr
ul
""".strip().split()
@@ -283,7 +286,7 @@ Classes
Expand source code
-Browse git
+Browse git
class H:
"""Provider of HTML serializing functions per element type.
@@ -424,6 +427,24 @@ Returns
-
+
+def table(*content, **atts)
+-
+
+
+
+def td(*content, **atts)
+-
+
+
+
+def tr(*content, **atts)
+-
+
+
def ul(*content, **atts)
Hspan
stylesummarytabletdtrul
diff --git a/tf/browser/index.html b/tf/browser/index.html
index b2c4aa3ff..1a76cbe98 100644
--- a/tf/browser/index.html
+++ b/tf/browser/index.html
@@ -32,7 +32,7 @@ Local TF data and web server
Expand source code
-Browse git
+Browse git
"""
# Local TF data and web server
diff --git a/tf/browser/kernel.html b/tf/browser/kernel.html
index 470bdae9d..82f738758 100644
--- a/tf/browser/kernel.html
+++ b/tf/browser/kernel.html
@@ -38,7 +38,7 @@ Kernel API
Expand source code
-Browse git
+Browse git
"""
# TF kernel
diff --git a/tf/browser/ner/corpus.html b/tf/browser/ner/corpus.html
index 0764b4508..2cbbefd35 100644
--- a/tf/browser/ner/corpus.html
+++ b/tf/browser/ner/corpus.html
@@ -39,7 +39,7 @@ Module tf.browser.ner.corpus
Expand source code
-Browse git
+Browse git
"""Access to the corpus.
@@ -55,7 +55,6 @@ Module tf.browser.ner.corpus
control over how to customize the annotator for different corpora.
"""
-
import re
from .settings import Settings, TOOLKEY
@@ -111,6 +110,7 @@ Module tf.browser.ner.corpus
features = settings.features
keywordFeatures = settings.keywordFeatures
bucketType = settings.bucketType
+ lineType = settings.lineType
entityType = settings.entityType
strFeature = settings.strFeature
afterFeature = settings.afterFeature
@@ -166,8 +166,20 @@ Module tf.browser.ner.corpus
# text = WHITE_RE.sub(" ", text)
return text
- def getTokens(node):
- return [(t, strv(t)) or "" for t in L.d(node, otype=slotType)]
+ def getTokens(node, lineBoundaries=False):
+ if lineBoundaries and lineType is None:
+ lineBoundaries = False
+
+ ts = L.d(node, otype=slotType)
+ tokens = [(t, strv(t) or "") for t in ts]
+
+ if not lineBoundaries:
+ return tokens
+
+ lines = L.d(node, otype=lineType) or []
+ lineEnds = {L.d(ln, otype=slotType)[-1] for ln in lines}
+
+ return (lineEnds, tokens)
def getStrings(tokenStart, tokenEnd):
return tuple(
@@ -180,7 +192,7 @@ Module tf.browser.ner.corpus
return L.d(T.sectionTuple(node)[1], otype=bucketType)
def getHeadings(node):
- return tuple(str(x).lstrip("0") for x in T.sectionFromNode(node))
+ return tuple(int(str(x).lstrip("0")) for x in T.sectionFromNode(node))
def get0(slots):
text = getText(slots)
@@ -454,7 +466,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Corpus(Settings):
def __init__(self):
@@ -500,6 +512,7 @@ Classes
features = settings.features
keywordFeatures = settings.keywordFeatures
bucketType = settings.bucketType
+ lineType = settings.lineType
entityType = settings.entityType
strFeature = settings.strFeature
afterFeature = settings.afterFeature
@@ -555,8 +568,20 @@ Classes
# text = WHITE_RE.sub(" ", text)
return text
- def getTokens(node):
- return [(t, strv(t)) or "" for t in L.d(node, otype=slotType)]
+ def getTokens(node, lineBoundaries=False):
+ if lineBoundaries and lineType is None:
+ lineBoundaries = False
+
+ ts = L.d(node, otype=slotType)
+ tokens = [(t, strv(t) or "") for t in ts]
+
+ if not lineBoundaries:
+ return tokens
+
+ lines = L.d(node, otype=lineType) or []
+ lineEnds = {L.d(ln, otype=slotType)[-1] for ln in lines}
+
+ return (lineEnds, tokens)
def getStrings(tokenStart, tokenEnd):
return tuple(
@@ -569,7 +594,7 @@ Classes
return L.d(T.sectionTuple(node)[1], otype=bucketType)
def getHeadings(node):
- return tuple(str(x).lstrip("0") for x in T.sectionFromNode(node))
+ return tuple(int(str(x).lstrip("0")) for x in T.sectionFromNode(node))
def get0(slots):
text = getText(slots)
diff --git a/tf/browser/ner/data.html b/tf/browser/ner/data.html
index cb543f390..9734a1959 100644
--- a/tf/browser/ner/data.html
+++ b/tf/browser/ner/data.html
@@ -43,7 +43,7 @@ Module tf.browser.ner.data
Expand source code
-Browse git
+Browse git
"""Annotation data module.
@@ -752,6 +752,7 @@ Module tf.browser.ner.data
self.mergeEntities(addE, _lowlevel=_lowlevel)
self.loadSetData()
+
if returns:
return (present, added)
@@ -958,7 +959,7 @@ Module tf.browser.ner.data
dataFile = f"{annoDir}/{setName}/entities.tsv"
- with fileOpen(dataFile, mode="a") as fh:
+ with fileOpen(dataFile, mode="w" if _lowlevel else "a") as fh:
for fVals, slots in newEntities:
fh.write("\t".join(str(x) for x in (*fVals, *slots)) + "\n")
@@ -1024,7 +1025,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class Data(Corpus):
def __init__(self, sets=None):
@@ -1700,6 +1701,7 @@ Parameters
self.mergeEntities(addE, _lowlevel=_lowlevel)
self.loadSetData()
+
if returns:
return (present, added)
@@ -1906,7 +1908,7 @@ Parameters
dataFile = f"{annoDir}/{setName}/entities.tsv"
- with fileOpen(dataFile, mode="a") as fh:
+ with fileOpen(dataFile, mode="w" if _lowlevel else "a") as fh:
for fVals, slots in newEntities:
fh.write("\t".join(str(x) for x in (*fVals, *slots)) + "\n")
diff --git a/tf/browser/ner/form.html b/tf/browser/ner/form.html
index 88921ccd2..d49443bdb 100644
--- a/tf/browser/ner/form.html
+++ b/tf/browser/ner/form.html
@@ -37,7 +37,7 @@ Module tf.browser.ner.form
Expand source code
-Browse git
+Browse git
"""Machinery for request reading.
@@ -229,7 +229,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class Form:
def __init__(
diff --git a/tf/browser/ner/fragments.html b/tf/browser/ner/fragments.html
index 96f65e32e..9a66642e4 100644
--- a/tf/browser/ner/fragments.html
+++ b/tf/browser/ner/fragments.html
@@ -35,7 +35,7 @@ Module tf.browser.ner.fragments
Expand source code
-Browse git
+Browse git
"""Wraps various pieces into HTML.
@@ -59,29 +59,30 @@ Module tf.browser.ner.fragments
v.messages = H.p((H.span(text, cls=lev) + H.br() for (lev, text) in messageSrc))
def wrapSets(self):
- """HTML for the annotation set chooser.
+ """HTML for the annotation task chooser.
- It is a dropdown with options, each corresponding to an existing annotation set.
- There is also a control to delete the set.
+ It is a dropdown with options, each corresponding to an existing annotation task.
+ There is also a control to delete the task.
Apart from these buttons there is a button to switch to the entities that are
present in the TF dataset as nodes of the entity type specified
in the YAML file with corresponding
features.
- Finally, it is possible to create a new, empty annotation set.
+ Finally, it is possible to create a new, empty annotation task.
"""
ner = self.ner
- setNames = ner.setNames
+ taskNames = ner.getTasks()
v = self.v
- chosenSet = v.set
+ chosenTask = v.task
+ sheetCase = v.sheetcase
content1 = [
H.input(
type="hidden",
- name="set",
- value=chosenSet,
+ name="task",
+ value=chosenTask,
id="seth",
),
H.input(
@@ -100,7 +101,7 @@ Module tf.browser.ner.fragments
"+",
type="submit",
id="anew",
- title="create a new annotation set",
+ title="create a new annotation task",
cls="mono",
),
" ",
@@ -108,22 +109,36 @@ Module tf.browser.ner.fragments
"++",
type="submit",
id="adup",
- title="duplicate this annotation set",
+ title="duplicate this annotation task",
cls="mono",
),
" ",
H.select(
(
H.option(
- ner.setInfo(setName=setName)[0],
- value=setName,
- selected=setName == chosenSet,
+ ner.setInfo(taskName)[0],
+ value=taskName,
+ selected=taskName == chosenTask,
)
- for setName in [""] + sorted(setNames)
+ for taskName in [""] + sorted(taskNames)
),
cls="selinp",
id="achange",
),
+ " ",
+ H.input(
+ type="hidden",
+ id="sheetcase",
+ name="sheetcase",
+ value="v" if sheetCase else "x",
+ ),
+ H.button(
+ "C" if sheetCase else "¢",
+ type="submit",
+ id="sheetcasebutton",
+ title="toggle case-sensitivity",
+ cls="mono",
+ ),
]
content2 = (
@@ -138,7 +153,7 @@ Module tf.browser.ner.fragments
"→",
type="submit",
id="arename",
- title="rename current annotation set",
+ title="rename current annotation task",
cls="mono",
),
" ",
@@ -146,15 +161,15 @@ Module tf.browser.ner.fragments
"-",
type="submit",
id="adelete",
- title="delete current annotation set",
+ title="delete current annotation task",
cls="mono",
),
]
- if not (chosenSet == "" or chosenSet.startswith("."))
+ if not (chosenTask == "" or chosenTask.startswith("."))
else []
)
- v.sets = H.p(content1, content2)
+ v.tasks = H.p(content1, content2)
def wrapCaption(self):
v = self.v
@@ -462,11 +477,11 @@ Module tf.browser.ner.fragments
value="" if subtleFilter is None else "v" if subtleFilter else "x",
),
"" if not setIsX else H.button(
- "all sheets"
+ "all scopes"
if subtleFilter is None
- else "contexts"
+ else "scoped"
if subtleFilter
- else "main sheet",
+ else "unscoped",
type="submit",
id="subtlefilterbutton",
cls="mono",
@@ -1049,7 +1064,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Fragments:
def wrapMessages(self):
@@ -1060,29 +1075,30 @@ Classes
v.messages = H.p((H.span(text, cls=lev) + H.br() for (lev, text) in messageSrc))
def wrapSets(self):
- """HTML for the annotation set chooser.
+ """HTML for the annotation task chooser.
- It is a dropdown with options, each corresponding to an existing annotation set.
- There is also a control to delete the set.
+ It is a dropdown with options, each corresponding to an existing annotation task.
+ There is also a control to delete the task.
Apart from these buttons there is a button to switch to the entities that are
present in the TF dataset as nodes of the entity type specified
in the YAML file with corresponding
features.
- Finally, it is possible to create a new, empty annotation set.
+ Finally, it is possible to create a new, empty annotation task.
"""
ner = self.ner
- setNames = ner.setNames
+ taskNames = ner.getTasks()
v = self.v
- chosenSet = v.set
+ chosenTask = v.task
+ sheetCase = v.sheetcase
content1 = [
H.input(
type="hidden",
- name="set",
- value=chosenSet,
+ name="task",
+ value=chosenTask,
id="seth",
),
H.input(
@@ -1101,7 +1117,7 @@ Classes
"+",
type="submit",
id="anew",
- title="create a new annotation set",
+ title="create a new annotation task",
cls="mono",
),
" ",
@@ -1109,22 +1125,36 @@ Classes
"++",
type="submit",
id="adup",
- title="duplicate this annotation set",
+ title="duplicate this annotation task",
cls="mono",
),
" ",
H.select(
(
H.option(
- ner.setInfo(setName=setName)[0],
- value=setName,
- selected=setName == chosenSet,
+ ner.setInfo(taskName)[0],
+ value=taskName,
+ selected=taskName == chosenTask,
)
- for setName in [""] + sorted(setNames)
+ for taskName in [""] + sorted(taskNames)
),
cls="selinp",
id="achange",
),
+ " ",
+ H.input(
+ type="hidden",
+ id="sheetcase",
+ name="sheetcase",
+ value="v" if sheetCase else "x",
+ ),
+ H.button(
+ "C" if sheetCase else "¢",
+ type="submit",
+ id="sheetcasebutton",
+ title="toggle case-sensitivity",
+ cls="mono",
+ ),
]
content2 = (
@@ -1139,7 +1169,7 @@ Classes
"→",
type="submit",
id="arename",
- title="rename current annotation set",
+ title="rename current annotation task",
cls="mono",
),
" ",
@@ -1147,15 +1177,15 @@ Classes
"-",
type="submit",
id="adelete",
- title="delete current annotation set",
+ title="delete current annotation task",
cls="mono",
),
]
- if not (chosenSet == "" or chosenSet.startswith("."))
+ if not (chosenTask == "" or chosenTask.startswith("."))
else []
)
- v.sets = H.p(content1, content2)
+ v.tasks = H.p(content1, content2)
def wrapCaption(self):
v = self.v
@@ -1463,11 +1493,11 @@ Classes
value="" if subtleFilter is None else "v" if subtleFilter else "x",
),
"" if not setIsX else H.button(
- "all sheets"
+ "all scopes"
if subtleFilter is None
- else "contexts"
+ else "scoped"
if subtleFilter
- else "main sheet",
+ else "unscoped",
type="submit",
id="subtlefilterbutton",
cls="mono",
@@ -2175,14 +2205,14 @@ Methods
def wrapSets(self)
-
-
HTML for the annotation set chooser.
-It is a dropdown with options, each corresponding to an existing annotation set.
-There is also a control to delete the set.
+HTML for the annotation task chooser.
+It is a dropdown with options, each corresponding to an existing annotation task.
+There is also a control to delete the task.
Apart from these buttons there is a button to switch to the entities that are
present in the TF dataset as nodes of the entity type specified
in the YAML file with corresponding
features.
-Finally, it is possible to create a new, empty annotation set.
+Finally, it is possible to create a new, empty annotation task.
def wrapSubtle(self)
diff --git a/tf/browser/ner/helpers.html b/tf/browser/ner/helpers.html
index 66879dc49..d13b86f7c 100644
--- a/tf/browser/ner/helpers.html
+++ b/tf/browser/ner/helpers.html
@@ -35,7 +35,7 @@ Module tf.browser.ner.helpers
Expand source code
-Browse git
+Browse git
"""Auxiliary functions.
@@ -46,14 +46,25 @@ Module tf.browser.ner.helpers
import re
import unicodedata
+import functools
+from ...core.helpers import console
from ..html import H
from .settings import STYLES
+WHITE_STRIP_RE = re.compile(r"""(?:\s|[])+""")
+
+
+def xstrip(x):
+ return WHITE_STRIP_RE.sub("", x)
+
+
WHITE_RE = re.compile(r"""\s+""", re.S)
NON_WORD = re.compile(r"""\W+""", re.S)
+LOC_RE = re.compile(r"[.:@]")
+
PART_CUT_OFF = 8
"""Maximum length of parts of entity identifiers."""
@@ -86,7 +97,7 @@ Module tf.browser.ner.helpers
return WHITE_RE.sub(" ", text).strip()
-def toTokens(text, spaceEscaped=False):
+def toTokens(text, spaceEscaped=False, caseSensitive=False):
"""Split a text into tokens.
The text is split on white-space.
@@ -100,6 +111,10 @@ Module tf.browser.ner.helpers
"""
result = TOKEN_RE.findall(normalize(text))
result = tuple((t.replace("_", " ") for t in result) if spaceEscaped else result)
+
+ if not caseSensitive:
+ result = tuple(t.lower() for t in result)
+
return tuple(t for t in result if t != " ")
@@ -117,9 +132,10 @@ Module tf.browser.ner.helpers
)
-def tnorm(text, spaceEscaped=False):
+def tnorm(text, spaceEscaped=False, caseSensitive=False):
return fromTokens(
- toTokens(text, spaceEscaped=spaceEscaped), spaceEscaped=spaceEscaped
+ toTokens(text, spaceEscaped=spaceEscaped, caseSensitive=caseSensitive),
+ spaceEscaped=spaceEscaped,
)
@@ -156,7 +172,12 @@ Module tf.browser.ner.helpers
which ultimately is provided in the corpus-dependent
`ner/config.yaml` .
"""
- parts = [y for x in toId(text).split(".") if (y := transform.get(x, x))]
+ eid = toId(text)
+
+ if len(eid) <= CUT_OFF:
+ return eid
+
+ parts = [y for x in eid.split(".") if (y := transform.get(x, x))]
result = []
n = 0
@@ -307,18 +328,344 @@ Module tf.browser.ner.helpers
return allCss
-def getPath(heading, instructions):
- for n in range(len(heading), 0, -1):
- path = heading[0:n]
+def getIntvIndex(buckets, instructions, getHeadings):
+ intervals = sortScopes(x for x in instructions if x != ())
+
+ nIntervals = len(intervals)
+
+ if nIntervals == 0:
+ return {bucket: () for bucket in buckets}
+
+ intvIndex = {}
+ i = 0
+ intv = intervals[i]
+ (b, e) = intv
+
+ for bucket in buckets:
+ hd = getHeadings(bucket)
+
+ assigned = False
+
+ if _locCmp(b, hd) == 1:
+ intvIndex[bucket] = ()
+ continue
+
+ while i < nIntervals:
+ if _locCmp(e, hd) == -1:
+ i += 1
+
+ if i < nIntervals:
+ intv = intervals[i]
+ (b, e) = intv
+ else:
+ intvIndex[bucket] = ()
+ assigned = True
+ break
+ else:
+ intvIndex[bucket] = () if _locCmp(b, hd) == 1 else intv
+ assigned = True
+ break
- if path in instructions:
- return path
+ if not assigned:
+ intvIndex[bucket] = ()
- return ()
+ return intvIndex
def repSet(s):
- return "{" + ", ".join(str(x) for x in sorted(s)) + "}"
+ return "{" + ", ".join(str(x) for x in sorted(s)) + "}"
+
+
+# SCOPES
+
+
+def _prevLoc(x):
+ # only for locs that can start a scope
+
+ if x[2] != 0:
+ return (x[0], x[1], x[2] - 1)
+
+ if x[1] != 0:
+ return (x[0], x[1] - 1, -1)
+
+ if x[0] != 0:
+ return (x[0] - 1, -1, -1)
+
+ return (0, 0, 0)
+
+
+def _nextLoc(x):
+ # only for locs that can end a scope
+
+ if x[2] != -1:
+ return (x[0], x[1], x[2] + 1)
+
+ if x[1] != -1:
+ return (x[0], x[1] + 1, 0)
+
+ if x[0] != -1:
+ return (x[0] + 1, 0, 0)
+
+ return (-1, -1, -1)
+
+
+def _pointCmp(x, y):
+ return 0 if x == y else -1 if y == -1 or (x != -1 and x < y) else 1
+
+
+def _locCmp(xLoc, yLoc):
+ for x, y in zip(xLoc, yLoc):
+ if x != y:
+ eq = _pointCmp(x, y)
+
+ if eq != 0:
+ return eq
+
+ return 0
+
+
+def _sameLoc(xLoc, yLoc):
+ return all(
+ xc == yc or ((xc == -1 or xc == 0) and (yc == -1 or yc == 0))
+ for (xc, yc) in zip(xLoc, yLoc)
+ )
+
+
+def _scopeBefore(xScope, yScope):
+ (xB, xE) = xScope
+ (yB, yE) = yScope
+
+ eqx = _locCmp(xB, yB)
+
+ return _locCmp(yE, xE) if eqx == 0 else eqx
+
+
+_locSort = functools.cmp_to_key(_locCmp)
+_scopeSort = functools.cmp_to_key(_scopeBefore)
+
+
+def sortLocs(locs):
+ return tuple(sorted(locs, key=_locSort))
+
+
+def sortScopes(scopes):
+ return tuple(sorted(scopes, key=_scopeSort))
+
+
+def repLoc(loc):
+ return ".".join(
+ f"{x:02}" if i == 0 else f"{x:03}" if i == 1 else f"{x}"
+ for (i, x) in enumerate(loc)
+ if x != 0 and x != -1
+ )
+
+
+def parseLoc(locStr, plain=True):
+ locStr = locStr.strip()
+
+ if not locStr:
+ result = ()
+ return () if plain else dict(result=(), warning=None, normal=repLoc(result))
+
+ parts = LOC_RE.split(locStr, maxsplit=2)
+
+ if any(x != "-1" and not (x.lstrip("0") or "0").isdecimal() for x in parts):
+ result = None
+ normal = None
+ warning = f"not a valid location: {locStr}"
+ else:
+ result = tuple(int(x.lstrip("0") or "0") for x in parts)
+ normal = repLoc(result)
+ warning = None
+
+ return result if plain else dict(result=result, warning=warning, normal=normal)
+
+
+def repScope(scope):
+ if scope is None or len(scope) == 0:
+ return "()"
+
+ (b, e) = scope
+ return repLoc(b) if _sameLoc(b, e) else f"{repLoc(b)}-{repLoc(e)}"
+
+
+def repScopes(scopes):
+ return ", ".join(repScope(scope) for scope in scopes)
+
+
+def parseScope(scopeStr, plain=True):
+ result = None
+ warnings = []
+
+ scopeStr = scopeStr.strip()
+
+ if not scopeStr:
+ result = ((0, 0, 0), (-1, -1, -1))
+ return (
+ result
+ if plain
+ else dict(result=result, warning=warnings, normal=repScope(result))
+ )
+
+ parts = scopeStr.split("-", 1)
+
+ if len(parts) == 1:
+ info = parseLoc(scopeStr, plain=False)
+ w = info["warning"]
+
+ if w:
+ warnings.append(w)
+ else:
+ s = info["result"]
+ result = (
+ (s + (0, 0, 0))[0:3],
+ (s + (-1, -1, -1))[0:3],
+ )
+ else:
+ part = parts[0].strip()
+ info1 = parseLoc(part, plain=False)
+ w1 = info1["warning"]
+
+ part = parts[1].strip()
+ info2 = parseLoc(part, plain=False)
+ w2 = info2["warning"]
+
+ if w1 or w2:
+ if w1:
+ warnings.append(w1)
+ if w2:
+ warnings.append(w2)
+ else:
+ s1 = info1["result"]
+ s2 = info2["result"]
+ result = (
+ (s1 + (0, 0, 0))[0:3],
+ (s2 + (-1, -1, -1))[0:3],
+ )
+
+ normal = None if result is None else repScope(result)
+
+ return result if plain else dict(result=result, warning=warnings, normal=normal)
+
+
+def parseScopes(scopeStr, plain=True):
+ results = []
+ warnings = []
+
+ if not scopeStr:
+ return () if plain else dict(result=(), warning=warnings, normal="")
+
+ for scopeStrPart in scopeStr.split(","):
+
+ if not scopeStrPart:
+ continue
+
+ info = parseScope(scopeStrPart, plain=plain)
+ result = info if plain else info["result"]
+
+ if result is None:
+ if not plain:
+ warnings.extend(info["warning"])
+ continue
+
+ results.append(result)
+
+ results = tuple(sortScopes(results))
+
+ return (
+ results
+ if plain
+ else dict(result=results, warning=warnings, normal=repScopes(results))
+ )
+
+
+def partitionScopes(scopesDict):
+ scopeFromStr = {}
+ strFromScope = {}
+ boundaries = {}
+ intervals = []
+
+ for scopeStr, scopes in scopesDict.items():
+ for scope in scopes:
+ (b, e) = scope
+ boundaries.setdefault(b, {}).setdefault("b", set()).add(scope)
+ boundaries.setdefault(e, {}).setdefault("e", set()).add(scope)
+
+ scopeFromStr.setdefault(scopeStr, set()).add(scope)
+ strFromScope.setdefault(scope, set()).add(scopeStr)
+
+ curScopes = set()
+ inScope = False
+
+ for x in sortLocs(boundaries):
+ beginScopes = boundaries[x].get("b", set())
+ endScopes = boundaries[x].get("e", set())
+
+ inScope = len(curScopes) > 0
+
+ hasB = len(beginScopes) > 0
+ hasE = len(endScopes) > 0
+
+ if hasB and hasE:
+ intervals[-1][1] = _prevLoc(x)
+ if intervals[-1][1] < intervals[-1][0]:
+ intervals.pop()
+ curScopes |= beginScopes
+ intervals.append([x, x, sortScopes(curScopes)])
+ curScopes -= endScopes
+ inScope = len(curScopes) > 0
+ if inScope:
+ intervals.append([_nextLoc(x), None, sortScopes(curScopes)])
+ elif hasB:
+ if inScope:
+ intervals[-1][1] = _prevLoc(x)
+ if intervals[-1][1] < intervals[-1][0]:
+ intervals.pop()
+ curScopes |= beginScopes
+ intervals.append([x, None, sortScopes(curScopes)])
+ elif hasE:
+ intervals[-1][1] = x
+ curScopes -= endScopes
+ inScope = len(curScopes) > 0
+ if inScope:
+ intervals.append([_nextLoc(x), None, sortScopes(curScopes)])
+
+ if inScope:
+ intervals[-1][1] = x
+
+ for x in intervals:
+ newX = []
+ seen = set()
+
+ for scope in x[2]:
+ for scopeStr in strFromScope[scope]:
+ if scopeStr in seen:
+ continue
+
+ seen.add(scopeStr)
+ newX.append(scopeStr)
+
+ x[2] = tuple(newX)
+
+ return intervals
+
+
+def testScopes(scopeStrs):
+ scopeIndex = {}
+
+ for scopeStr in scopeStrs:
+ info = parseScopes(scopeStr, plain=False)
+ warning = info["warning"]
+
+ if len(warning):
+ console(f"Errors in {scopeStr}: {'; '.join(warning)}")
+ else:
+ scopes = info["result"]
+ normScopeStr = info["normal"]
+ console(f"{scopeStr} => {normScopeStr}\n\t{repScopes(sortScopes(scopes))}")
+ scopeIndex[normScopeStr] = scopes
+
+ partitionScopes(scopeIndex)
@@ -374,8 +721,8 @@ Returns
fromTokens first.
You can use tnorm() for that.
-
-def getPath(heading, instructions)
+
+def getIntvIndex(buckets, instructions, getHeadings)
-
@@ -401,6 +748,30 @@
Parameters
-
Normalize white-space in a text.
+
+def parseLoc(locStr, plain=True)
+-
+
+
+
+def parseScope(scopeStr, plain=True)
+-
+
+
+
+def parseScopes(scopeStr, plain=True)
+-
+
+
+
+def partitionScopes(scopesDict)
+-
+
+
def repIdent(features, vals, active='')
Parameters
Can be used to mark the entity as active.
+
+def repLoc(loc)
+-
+
+
+
+def repScope(scope)
+-
+
+
+
+def repScopes(scopes)
+-
+
+
def repSet(s)
Parameters
Can be used to mark the entity as active.
+
+def sortLocs(locs)
+-
+
+
+
+def sortScopes(scopes)
+-
+
+
+
+def testScopes(scopeStrs)
+-
+
+
-def tnorm(text, spaceEscaped=False)
+def tnorm(text, spaceEscaped=False, caseSensitive=False)
-
@@ -470,7 +877,7 @@
Parameters
ner/config.yaml .
-def toTokens(text, spaceEscaped=False)
+def toTokens(text, spaceEscaped=False, caseSensitive=False)
-
Split a text into tokens.
@@ -489,6 +896,12 @@ Parameters
-
HTML representation of an entity as a sequence of feat=val strings.
+
+def xstrip(x)
+-
+
+
@@ -567,18 +980,29 @@ Index
findCompilefromTokensgetPathgetIntvIndexmakeCssnormalizeparseLocparseScopeparseScopespartitionScopesrepIdentrepLocrepScoperepScopesrepSetrepSummarysortLocssortScopestestScopestnormtoAsciitoIdtoSmallIdtoTokensvalRepxstrip
diff --git a/tf/browser/ner/match.html b/tf/browser/ner/match.html
index 5164d59d7..573864855 100644
--- a/tf/browser/ner/match.html
+++ b/tf/browser/ner/match.html
@@ -34,7 +34,7 @@ Module tf.browser.ner.match
Expand source code
-Browse git
+Browse git
"""Match functions.
@@ -43,10 +43,17 @@ Module tf.browser.ner.match
"""
from .settings import NONE
-from .helpers import getPath, fromTokens
+from .helpers import fromTokens, getIntvIndex, xstrip
-def occMatch(getTokens, getHeadings, buckets, instructions, spaceEscaped):
+def occMatch(
+ getTokens,
+ getHeadings,
+ buckets,
+ instructions,
+ spaceEscaped,
+ caseSensitive=False,
+):
"""Finds the occurrences of multiple sequences of tokens in a single bucket.
Parameters
@@ -75,7 +82,7 @@ Module tf.browser.ner.match
-------
dict
A multiply nested dict, first keyed by tuples of entity identifier and kind,
- then by its triggers then by the sheets where such a trigger occurs, and then
+ then by its triggers then by the scope of such a trigger, and then
the value is a list of occurrences, where each occurrence is a tuple of slots.
"""
@@ -85,26 +92,38 @@ Module tf.browser.ner.match
# keys are tokens and whose values are the token sequences that have that token
# at that position
+ LONGEST_ONLY = True
+
+ # We can switch between two behaviours:
+ #
+ # 1. once a match is found, stop looking for smaller matches that start at the same place
+ # of that match, or somewhere inside that match
+ #
+ # 2. find all matches, irrespective of overlap
+
results = {}
- shown = False
+
+ intvIndex = getIntvIndex(buckets, instructions, getHeadings)
for b in buckets:
- heading = getHeadings(b)
- path = getPath(heading, instructions)
- data = instructions[path]
+ intv = intvIndex[b]
+ data = instructions[intv]
+
tPos = data["tPos"]
tMap = data["tMap"]
idMap = data["idMap"]
# compile the bucket into logical tokens
- bTokensAll = getTokens(b)
- bTokens = [x for x in bTokensAll if (x[1] or "").strip()]
+ (lineEnds, bTokensAll) = getTokens(b, lineBoundaries=True)
+ # bTokensAll = getTokens(b, lineBoundaries=False)
+ bTokens = [x for x in bTokensAll if xstrip(x[1] or "")]
bStrings = []
bStringFirst = {}
bStringLast = {}
for t, s in bTokens:
- if len(bStrings) > 1 and bStrings[-1] == "-":
+ # if len(bStrings) > 1 and bStrings[-1] == "-":
+ if t - 1 in lineEnds and len(bStrings) > 1 and bStrings[-1] == "-":
bStrings.pop()
bStrings[-1] += s
bStringLast[len(bStrings) - 1] = t
@@ -114,16 +133,18 @@ Module tf.browser.ner.match
bStringLast[len(bStrings) - 1] = t
bStrings = tuple(bStrings)
- nBStrings = len(bStrings)
- if not shown and heading[0] == "4" and heading[1] == "48":
- shown = True
+ if not caseSensitive:
+ bStrings = tuple(t.lower() for t in bStrings)
+
+ nBStrings = len(bStrings)
# perform the search
i = 0
while i < nBStrings:
j = 0
+
candidateMatches = None
matches = {}
@@ -148,19 +169,26 @@ Module tf.browser.ner.match
if len(resultMatches):
resultMatch = resultMatches[0]
+
trigger = fromTokens(resultMatch, spaceEscaped=spaceEscaped)
- tPath = tMap[trigger]
+ scope = tMap[trigger]
+
firstT = bStringFirst[i]
lastT = bStringLast[i + m]
slots = tuple(range(firstT, lastT + 1))
eidkind = idMap[trigger]
- dest = results.setdefault(eidkind, {}).setdefault(trigger, {})
- destHits = dest.setdefault(tPath, [])
- destHits.append(slots)
- break
- shift = m + 1
- i += shift
+ dest = (
+ results.setdefault(eidkind, {})
+ .setdefault(trigger, {})
+ .setdefault(scope, [])
+ )
+ dest.append(slots)
+
+ if LONGEST_ONLY:
+ break
+
+ i += m + 1 if LONGEST_ONLY else 1
return results
@@ -229,7 +257,7 @@ Module tf.browser.ner.match
fits = True
bTokensAll = getTokens(b)
- bTokens = [x for x in bTokensAll if (x[1] or "").strip()]
+ bTokens = [x for x in bTokensAll if xstrip(x[1] or "")]
if fits is None or fits:
if anyEnt is not None:
@@ -413,7 +441,7 @@ Returns
-def occMatch(getTokens, getHeadings, buckets, instructions, spaceEscaped)
+def occMatch(getTokens, getHeadings, buckets, instructions, spaceEscaped, caseSensitive=False)
-
Finds the occurrences of multiple sequences of tokens in a single bucket.
@@ -442,7 +470,7 @@ Returns
dict- A multiply nested dict, first keyed by tuples of entity identifier and kind,
-then by its triggers then by the sheets where such a trigger occurs, and then
+then by its triggers then by the scope of such a trigger, and then
the value is a list of occurrences, where each occurrence is a tuple of slots.
diff --git a/tf/browser/ner/ner.html b/tf/browser/ner/ner.html
index 8d8158fdf..e02039d05 100644
--- a/tf/browser/ner/ner.html
+++ b/tf/browser/ner/ner.html
@@ -244,7 +244,7 @@ Inspection
Expand source code
-Browse git
+Browse git
"""API for Named Entity marking.
@@ -527,7 +527,7 @@ Inspection
)
from ...core.timestamp import SILENT_D, DEEP
from .sheets import Sheets
-from .helpers import findCompile
+from .helpers import findCompile, toTokens
from .sets import Sets
from .show import Show
from .match import entityMatch, occMatch
@@ -607,10 +607,96 @@ Inspection
if not self.properlySetup:
return
- def setTask(self, task):
+ def setTask(self, task, force=False, caseSensitive=False):
+ (newSetNameRep, newSetRo, newSetSrc, newSetX) = self.setInfo(task)
self.setSet(task)
- setIsX = self.setIsX
- self.setSheet(task[1:] if setIsX else None)
+ self.setSheet(
+ task[1:] if newSetX else None,
+ force=force,
+ caseSensitive=caseSensitive,
+ )
+
+ def getTasks(self):
+ setNames = self.setNames
+
+ tasks = set()
+
+ for setName in setNames:
+ tasks.add(
+ setName.removesuffix("-no-case").removesuffix("-with-case")
+ if setName.startswith(".")
+ else setName
+ )
+
+ return tasks
+
+ def getMeta(self):
+ """Retrieves the metadata of the current sheet."""
+ sheetData = self.getSheetData()
+ return (sheetData.metaFields, sheetData.metaData)
+
+ def findTrigger(self, trigger, show=True):
+ if not self.properlySetup:
+ return []
+
+ app = self.app
+ L = app.api.L
+
+ settings = self.settings
+ spaceEscaped = settings.spaceEscaped
+
+ setData = self.getSetData()
+ getTokens = self.getTokens
+ getHeadings = self.getHeadings
+
+ buckets = setData.buckets or ()
+
+ sheetName = self.sheetName
+ sheetData = self.sheets[sheetName]
+
+ caseSensitive = sheetData.caseSensitive
+
+ triggerT = toTokens(
+ trigger, spaceEscaped=spaceEscaped, caseSensitive=caseSensitive
+ )
+ idMap = {trigger: trigger}
+ tMap = {trigger: ""}
+ tPos = {}
+
+ for i, token in enumerate(triggerT):
+ tPos.setdefault(i, {}).setdefault(token, set()).add(triggerT)
+
+ instructions = {(): dict(tPos=tPos, tMap=tMap, idMap=idMap)}
+
+ inventory = occMatch(
+ getTokens,
+ getHeadings,
+ buckets,
+ instructions,
+ spaceEscaped,
+ caseSensitive=caseSensitive,
+ )
+
+ occs = inventory.get(trigger, {}).get(trigger, {}).get("", [])
+
+ if show:
+ nOccs = len(occs)
+
+ if nOccs:
+ plural = "" if nOccs == 1 else "s"
+ app.dm(f"**{nOccs} occurrence{plural}**\n")
+
+ headings = set()
+ highlights = set()
+
+ for occ in occs:
+ headings.add(L.u(occ[0], otype="chunk")[0])
+
+ for slot in occ:
+ highlights.add(slot)
+
+ for hd in sorted(headings):
+ app.plain(hd, highlights=highlights)
def findOccs(self):
"""Finds the occurrences of multiple triggers.
@@ -646,10 +732,15 @@ Inspection
sheetName = self.sheetName
sheetData = self.sheets[sheetName]
-
instructions = sheetData.instructions
+ caseSensitive = sheetData.caseSensitive
sheetData.inventory = occMatch(
- getTokens, getHeadings, buckets, instructions, spaceEscaped
+ getTokens,
+ getHeadings,
+ buckets,
+ instructions,
+ spaceEscaped,
+ caseSensitive=caseSensitive,
)
def filterContent(
@@ -907,7 +998,7 @@ Inspection
self.console(f"{nResults} {bucketType}{pluralR}")
return results
- def reportHits(self, silent=None):
+ def reportHits(self, silent=None, showNoHits=False):
"""Reports the inventory."""
if not self.properlySetup:
return
@@ -928,26 +1019,29 @@ Inspection
allTriggers = set()
- for path, data in instructions.items():
+ for intv, data in instructions.items():
idMap = data["idMap"]
tMap = data["tMap"]
- for trigger, tPath in tMap.items():
- eidkind = idMap[trigger]
- name = nameMap[eidkind][0]
- allTriggers.add((name, eidkind, trigger, tPath))
+ for trigger, scope in tMap.items():
+ eidkind = idMap.get(trigger, None)
+
+ if eidkind is None:
+ continue
+
+ name = nameMap[eidkind]
+ allTriggers.add((name, eidkind, trigger, scope))
hitData = []
names = set()
- triggersSuccess = 0
+ noHits = set()
for e in sorted(allTriggers):
- (name, eidkind, trigger, tPath) = e
+ (name, eidkind, trigger, scope) = e
names.add(name)
- sheet = ".".join(tPath)
- entry = (name, trigger, sheet)
+ entry = (name, trigger, scope)
section = ""
hits = ""
@@ -955,30 +1049,46 @@ Inspection
if entInfo is None:
hitData.append(("!E", *entry, "", 0))
+ noHits.add(trigger)
continue
triggerInfo = entInfo.get(trigger, None)
if triggerInfo is None:
hitData.append(("!T", *entry, "", 0))
+ noHits.add(trigger)
continue
- occs = triggerInfo.get(tPath, None)
+ occs = triggerInfo.get(scope, None)
- if occs is None:
+ if occs is None or len(occs) == 0:
hitData.append(("!P", *entry, "", 0))
+ noHits.add(trigger)
continue
- triggersSuccess += 1
sectionInfo = collections.Counter()
for slots in occs:
- section = ".".join(getHeadings(slots[0]))
+ section = ".".join(str(x) for x in getHeadings(slots[0]))
sectionInfo[section] += 1
for section, hits in sorted(sectionInfo.items()):
hitData.append(("OK", *entry, section, hits))
+ trigWithout = len(noHits)
+
+ if showNoHits and (trigWithout > 0):
+ console(
+ "Triggers without hits: " f"{trigWithout}x:",
+ error=True,
+ )
+
+ if len(noHits):
+ console(" triggers without hits:")
+
+ for trigger in sorted(noHits):
+ console(f" {trigger}", error=True)
+
with fileOpen(reportFile, "w") as rh:
rh.write("label\tname\ttrigger\tsheet\tsection\thits\n")
@@ -995,11 +1105,11 @@ Inspection
if silent
else dedent(
f"""
- Entities targeted: {nEnt:>5}
- Triggers searched for: {nTriggers:>5}
- Triggers with hits: {triggersSuccess:>5}
- Triggers without hits: {nTriggers - triggersSuccess:>5}
- Total hits: {nHits:>5}
+ Entities targeted: {nEnt:>5}
+ Triggers searched for: {nTriggers:>5}
+ Triggers without hits: {trigWithout:>5}
+ Triggers with hits: {nTriggers - trigWithout:>5}
+ Total hits: {nHits:>5}
All hits in report file: {reportFile}
"""
@@ -1214,7 +1324,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class NER(Sheets, Sets, Show):
def __init__(self, app, normalizeChars=None, data=None, browse=False, silent=False):
@@ -1290,10 +1400,96 @@ Parameters
if not self.properlySetup:
return
- def setTask(self, task):
+ def setTask(self, task, force=False, caseSensitive=False):
+ (newSetNameRep, newSetRo, newSetSrc, newSetX) = self.setInfo(task)
self.setSet(task)
- setIsX = self.setIsX
- self.setSheet(task[1:] if setIsX else None)
+ self.setSheet(
+ task[1:] if newSetX else None,
+ force=force,
+ caseSensitive=caseSensitive,
+ )
+
+ def getTasks(self):
+ setNames = self.setNames
+
+ tasks = set()
+
+ for setName in setNames:
+ tasks.add(
+ setName.removesuffix("-no-case").removesuffix("-with-case")
+ if setName.startswith(".")
+ else setName
+ )
+
+ return tasks
+
+ def getMeta(self):
+ """Retrieves the metadata of the current sheet."""
+ sheetData = self.getSheetData()
+ return (sheetData.metaFields, sheetData.metaData)
+
+ def findTrigger(self, trigger, show=True):
+ if not self.properlySetup:
+ return []
+
+ app = self.app
+ L = app.api.L
+
+ settings = self.settings
+ spaceEscaped = settings.spaceEscaped
+
+ setData = self.getSetData()
+ getTokens = self.getTokens
+ getHeadings = self.getHeadings
+
+ buckets = setData.buckets or ()
+
+ sheetName = self.sheetName
+ sheetData = self.sheets[sheetName]
+
+ caseSensitive = sheetData.caseSensitive
+
+ triggerT = toTokens(
+ trigger, spaceEscaped=spaceEscaped, caseSensitive=caseSensitive
+ )
+ idMap = {trigger: trigger}
+ tMap = {trigger: ""}
+ tPos = {}
+
+ for i, token in enumerate(triggerT):
+ tPos.setdefault(i, {}).setdefault(token, set()).add(triggerT)
+
+ instructions = {(): dict(tPos=tPos, tMap=tMap, idMap=idMap)}
+
+ inventory = occMatch(
+ getTokens,
+ getHeadings,
+ buckets,
+ instructions,
+ spaceEscaped,
+ caseSensitive=caseSensitive,
+ )
+
+ occs = inventory.get(trigger, {}).get(trigger, {}).get("", [])
+
+ if show:
+ nOccs = len(occs)
+
+ if nOccs:
+ plural = "" if nOccs == 1 else "s"
+ app.dm(f"**{nOccs} occurrence{plural}**\n")
+
+ headings = set()
+ highlights = set()
+
+ for occ in occs:
+ headings.add(L.u(occ[0], otype="chunk")[0])
+
+ for slot in occ:
+ highlights.add(slot)
+
+ for hd in sorted(headings):
+ app.plain(hd, highlights=highlights)
def findOccs(self):
"""Finds the occurrences of multiple triggers.
@@ -1329,10 +1525,15 @@ Parameters
sheetName = self.sheetName
sheetData = self.sheets[sheetName]
-
instructions = sheetData.instructions
+ caseSensitive = sheetData.caseSensitive
sheetData.inventory = occMatch(
- getTokens, getHeadings, buckets, instructions, spaceEscaped
+ getTokens,
+ getHeadings,
+ buckets,
+ instructions,
+ spaceEscaped,
+ caseSensitive=caseSensitive,
)
def filterContent(
@@ -1590,7 +1791,7 @@ Parameters
self.console(f"{nResults} {bucketType}{pluralR}")
return results
- def reportHits(self, silent=None):
+ def reportHits(self, silent=None, showNoHits=False):
"""Reports the inventory."""
if not self.properlySetup:
return
@@ -1611,26 +1812,29 @@ Parameters
allTriggers = set()
- for path, data in instructions.items():
+ for intv, data in instructions.items():
idMap = data["idMap"]
tMap = data["tMap"]
- for trigger, tPath in tMap.items():
- eidkind = idMap[trigger]
- name = nameMap[eidkind][0]
- allTriggers.add((name, eidkind, trigger, tPath))
+ for trigger, scope in tMap.items():
+ eidkind = idMap.get(trigger, None)
+
+ if eidkind is None:
+ continue
+
+ name = nameMap[eidkind]
+ allTriggers.add((name, eidkind, trigger, scope))
hitData = []
names = set()
- triggersSuccess = 0
+ noHits = set()
for e in sorted(allTriggers):
- (name, eidkind, trigger, tPath) = e
+ (name, eidkind, trigger, scope) = e
names.add(name)
- sheet = ".".join(tPath)
- entry = (name, trigger, sheet)
+ entry = (name, trigger, scope)
section = ""
hits = ""
@@ -1638,30 +1842,46 @@ Parameters
if entInfo is None:
hitData.append(("!E", *entry, "", 0))
+ noHits.add(trigger)
continue
triggerInfo = entInfo.get(trigger, None)
if triggerInfo is None:
hitData.append(("!T", *entry, "", 0))
+ noHits.add(trigger)
continue
- occs = triggerInfo.get(tPath, None)
+ occs = triggerInfo.get(scope, None)
- if occs is None:
+ if occs is None or len(occs) == 0:
hitData.append(("!P", *entry, "", 0))
+ noHits.add(trigger)
continue
- triggersSuccess += 1
sectionInfo = collections.Counter()
for slots in occs:
- section = ".".join(getHeadings(slots[0]))
+ section = ".".join(str(x) for x in getHeadings(slots[0]))
sectionInfo[section] += 1
for section, hits in sorted(sectionInfo.items()):
hitData.append(("OK", *entry, section, hits))
+ trigWithout = len(noHits)
+
+ if showNoHits and (trigWithout > 0):
+ console(
+ "Triggers without hits: " f"{trigWithout}x:",
+ error=True,
+ )
+
+ if len(noHits):
+ console(" triggers without hits:")
+
+ for trigger in sorted(noHits):
+ console(f" {trigger}", error=True)
+
with fileOpen(reportFile, "w") as rh:
rh.write("label\tname\ttrigger\tsheet\tsection\thits\n")
@@ -1678,11 +1898,11 @@ Parameters
if silent
else dedent(
f"""
- Entities targeted: {nEnt:>5}
- Triggers searched for: {nTriggers:>5}
- Triggers with hits: {triggersSuccess:>5}
- Triggers without hits: {nTriggers - triggersSuccess:>5}
- Total hits: {nHits:>5}
+ Entities targeted: {nEnt:>5}
+ Triggers searched for: {nTriggers:>5}
+ Triggers without hits: {trigWithout:>5}
+ Triggers with hits: {nTriggers - trigWithout:>5}
+ Total hits: {nHits:>5}
All hits in report file: {reportFile}
"""
@@ -1975,14 +2195,32 @@ Returns
names: keyed by tuples (eid, kind) and then path,
the value is the name of that entity in the context indicated by path.
+
+def findTrigger(self, trigger, show=True)
+-
+
+
+
+def getMeta(self)
+-
+
Retrieves the metadata of the current sheet.
+
+
+def getTasks(self)
+-
+
+
-def reportHits(self, silent=None)
+def reportHits(self, silent=None, showNoHits=False)
-
Reports the inventory.
-def setTask(self, task)
+def setTask(self, task, force=False, caseSensitive=False)
-
@@ -2132,10 +2370,13 @@
Index
-
NER
-
+
diff --git a/tf/browser/ner/request.html b/tf/browser/ner/request.html
index 88edd9cbd..b445a9edd 100644
--- a/tf/browser/ner/request.html
+++ b/tf/browser/ner/request.html
@@ -34,7 +34,7 @@ Module tf.browser.ner.request
Expand source code
-Browse git
+Browse git
"""Auxiliary functions for managing request data.
@@ -89,7 +89,7 @@ Module tf.browser.ner.request
sec0
sec1
sec2
- set
+ task
duset
rset
dset
@@ -110,6 +110,7 @@ Module tf.browser.ner.request
keysBool = """
formattingdo
bfindc
+ sheetcase
""".strip().split()
keysTri = """
@@ -139,6 +140,7 @@ Module tf.browser.ner.request
defaults = dict(
sortkey=None,
sortdir=None,
+ sheetcase="x",
freestate="all",
scope=SC_ALL,
modwidgetstate="add",
@@ -384,7 +386,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Request(Form):
def __init__(self):
@@ -426,7 +428,7 @@ Classes
sec0
sec1
sec2
- set
+ task
duset
rset
dset
@@ -447,6 +449,7 @@ Classes
keysBool = """
formattingdo
bfindc
+ sheetcase
""".strip().split()
keysTri = """
@@ -476,6 +479,7 @@ Classes
defaults = dict(
sortkey=None,
sortdir=None,
+ sheetcase="x",
freestate="all",
scope=SC_ALL,
modwidgetstate="add",
diff --git a/tf/browser/ner/serve.html b/tf/browser/ner/serve.html
index 6252c21e3..6bdd98dad 100644
--- a/tf/browser/ner/serve.html
+++ b/tf/browser/ner/serve.html
@@ -36,7 +36,7 @@ Module tf.browser.ner.serve
Expand source code
-Browse git
+Browse git
"""Main controller for Flask
@@ -76,11 +76,6 @@ Module tf.browser.ner.serve
self.initVars()
- # v = self.v
- # setName = v.set
-
- # ner.setTask(setName)
-
def setupFull(self):
"""Prepares to serve a complete page.
@@ -283,7 +278,7 @@ Module tf.browser.ner.serve
ner.readSheets()
v = self.v
- chosenSet = v.set
+ chosenTask = v.task
dupSet = v.duset
renamedSet = v.rset
deleteSet = v.dset
@@ -293,22 +288,23 @@ Module tf.browser.ner.serve
if deleteSet:
messages.extend(ner.setDel(deleteSet))
v.dset = ""
- v.set = ""
+ v.task = ""
if dupSet:
messages.extend(ner.setDup(dupSet))
- v.set = dupSet
+ v.task = dupSet
v.duset = ""
- if renamedSet and chosenSet:
+ if renamedSet and chosenTask:
messages.extend(ner.setMove(renamedSet))
- v.set = renamedSet
+ v.task = renamedSet
v.rset = ""
v.messagesrc = messages
- chosenSet = v.set
- ner.setTask(chosenSet)
+ chosenTask = v.task
+ sheetCase = v.sheetcase
+ ner.setTask(chosenTask, caseSensitive=sheetCase)
ner.loadSetData()
self.wrapSets()
@@ -464,7 +460,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class Serve(Request, Fragments):
def __init__(self, web):
@@ -488,11 +484,6 @@ Parameters
self.initVars()
- # v = self.v
- # setName = v.set
-
- # ner.setTask(setName)
-
def setupFull(self):
"""Prepares to serve a complete page.
@@ -695,7 +686,7 @@ Parameters
ner.readSheets()
v = self.v
- chosenSet = v.set
+ chosenTask = v.task
dupSet = v.duset
renamedSet = v.rset
deleteSet = v.dset
@@ -705,22 +696,23 @@ Parameters
if deleteSet:
messages.extend(ner.setDel(deleteSet))
v.dset = ""
- v.set = ""
+ v.task = ""
if dupSet:
messages.extend(ner.setDup(dupSet))
- v.set = dupSet
+ v.task = dupSet
v.duset = ""
- if renamedSet and chosenSet:
+ if renamedSet and chosenTask:
messages.extend(ner.setMove(renamedSet))
- v.set = renamedSet
+ v.task = renamedSet
v.rset = ""
v.messagesrc = messages
- chosenSet = v.set
- ner.setTask(chosenSet)
+ chosenTask = v.task
+ sheetCase = v.sheetcase
+ ner.setTask(chosenTask, caseSensitive=sheetCase)
ner.loadSetData()
self.wrapSets()
diff --git a/tf/browser/ner/sets.html b/tf/browser/ner/sets.html
index fdb5e0801..627d82e3f 100644
--- a/tf/browser/ner/sets.html
+++ b/tf/browser/ner/sets.html
@@ -36,7 +36,7 @@ Module tf.browser.ner.sets
Expand source code
-Browse git
+Browse git
"""Annotation set management.
@@ -130,9 +130,11 @@ Module tf.browser.ner.sets
setNameRep = (
f"{SET_ENT} {entitySet}"
if setName == ""
- else f"{SET_SHEET} {setName[1:]}"
- if setName.startswith(".")
- else f"{SET_MAIN} {setName}"
+ else (
+ f"{SET_SHEET} " f"{setName[1:]}"
+ if setName.startswith(".")
+ else f"{SET_MAIN} {setName}"
+ )
)
if inObject:
self.setNameRep = setNameRep
@@ -149,6 +151,7 @@ Module tf.browser.ner.sets
by working with annotations in a Jupyter Notebook.
"""
annoDir = self.annoDir
+
self.setNames = set(dirContents(annoDir)[1])
def getSetData(self):
@@ -201,7 +204,7 @@ Module tf.browser.ner.sets
nEntities = len(entities)
plural = "" if nEntities == 1 else "s"
self.console(
- f"Annotation set {setNameRep} has {nEntities} annotation{plural}"
+ f"Annotation set {setNameRep} " f"has {nEntities} annotation{plural}"
)
def _addToSet(self, newEntities):
@@ -455,7 +458,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class Sets(Data):
def __init__(self, sets=None):
@@ -521,9 +524,11 @@ Parameters
setNameRep = (
f"{SET_ENT} {entitySet}"
if setName == ""
- else f"{SET_SHEET} {setName[1:]}"
- if setName.startswith(".")
- else f"{SET_MAIN} {setName}"
+ else (
+ f"{SET_SHEET} " f"{setName[1:]}"
+ if setName.startswith(".")
+ else f"{SET_MAIN} {setName}"
+ )
)
if inObject:
self.setNameRep = setNameRep
@@ -540,6 +545,7 @@ Parameters
by working with annotations in a Jupyter Notebook.
"""
annoDir = self.annoDir
+
self.setNames = set(dirContents(annoDir)[1])
def getSetData(self):
@@ -592,7 +598,7 @@ Parameters
nEntities = len(entities)
plural = "" if nEntities == 1 else "s"
self.console(
- f"Annotation set {setNameRep} has {nEntities} annotation{plural}"
+ f"Annotation set {setNameRep} " f"has {nEntities} annotation{plural}"
)
def _addToSet(self, newEntities):
diff --git a/tf/browser/ner/settings.html b/tf/browser/ner/settings.html
index d574f2ac6..003074f25 100644
--- a/tf/browser/ner/settings.html
+++ b/tf/browser/ner/settings.html
@@ -34,7 +34,7 @@ Module tf.browser.ner.settings
Expand source code
-Browse git
+Browse git
"""Corpus dependent setup of the annotation tool.
@@ -277,7 +277,7 @@ Module tf.browser.ner.settings
def consoleLine(self, isError, indent, msg):
silent = self.silent
- if not silent:
+ if silent and not isError:
return
tabs = " " * indent
@@ -416,7 +416,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Settings:
def __init__(self):
@@ -464,7 +464,7 @@ Classes
def consoleLine(self, isError, indent, msg):
silent = self.silent
- if not silent:
+ if silent and not isError:
return
tabs = " " * indent
diff --git a/tf/browser/ner/sheets.html b/tf/browser/ner/sheets.html
index 4cb1675f3..c6c5e8893 100644
--- a/tf/browser/ner/sheets.html
+++ b/tf/browser/ner/sheets.html
@@ -31,17 +31,24 @@ Module tf.browser.ner.sheets
Expand source code
-Browse git
+Browse git
-import collections
-import re
+import re
import time
import pickle
import gzip
from ...parameters import GZIP_LEVEL, PICKLE_PROTOCOL
from ...capable import CheckImport
-from .helpers import tnorm, normalize, toSmallId, toTokens, repSet
+from .helpers import (
+ tnorm,
+ normalize,
+ toSmallId,
+ toTokens,
+ repScope,
+ parseScopes,
+ partitionScopes,
+)
from ...core.generic import AttrDict
from ...core.helpers import console
from ...core.files import fileOpen, dirContents, extNm, fileExists, mTime
@@ -50,6 +57,25 @@ Module tf.browser.ner.sheets
DS_STORE = ".DS_Store"
SHEET_RE = re.compile(r"""^([0-9]+)((?:-[0-9]+)?)\.xlsx$""", re.I)
+SHEET_KEYS = """
+ caseSensitive
+ metaFields
+ metaData
+ logData
+ nameMap
+ scopeMap
+ rowMap
+ instructions
+ inventory
+ triggerFromMatch
+ hitData
+""".strip().split()
+
+CLEAR_KEYS = """
+ raw
+ compiled
+""".strip().split()
+
class Sheets:
def __init__(self, sheets=None):
@@ -57,6 +83,7 @@ Module tf.browser.ner.sheets
if CI.importOK(hint=True):
openpyxl = CI.importGet()
self.loadXls = openpyxl.load_workbook
+ self.Workbook = openpyxl.Workbook
else:
self.properlySetup = False
return None
@@ -82,9 +109,6 @@ Module tf.browser.ner.sheets
self.readSheets()
- # if not browse:
- # self.loadSheetData()
-
def readSheets(self):
"""Read the list current ner sheets (again).
@@ -103,79 +127,7 @@ Module tf.browser.ner.sheets
sheetNames.add(sheetName)
setNames.add(f".{sheetName}")
- def readData(self, sheetData, sheetUpdated):
- setName = self.setName
- annoDir = self.annoDir
- setDir = f"{annoDir}/{setName}"
- dataFile = f"{setDir}/data.gz"
- timeFile = f"{setDir}/time.txt"
-
- tm = 0
-
- uptodate = fileExists(timeFile)
-
- if uptodate:
- with fileOpen(timeFile) as fh:
- info = fh.read().strip()
-
- try:
- tm = float(info)
- uptodate = sheetUpdated < tm
- except Exception:
- uptodate = False
-
- if not uptodate:
- return False
-
- if fileExists(dataFile):
- with gzip.open(dataFile, mode="rb") as f:
- data = pickle.load(f)
-
- for k, v in data.items():
- sheetData[k] = v
-
- sheetData.time = tm
-
- return True
- else:
- return False
-
- def writeData(self, sheetData, tm):
- setName = self.setName
- annoDir = self.annoDir
- setDir = f"{annoDir}/{setName}"
- dataFile = f"{setDir}/data.gz"
- timeFile = f"{setDir}/time.txt"
-
- with gzip.open(dataFile, mode="wb", compresslevel=GZIP_LEVEL) as f:
- f.write(pickle.dumps(sheetData, protocol=PICKLE_PROTOCOL))
-
- with fileOpen(timeFile, "w") as fh:
- fh.write(f"{tm}\n")
-
- def log(self, isError, indent, msg):
- silent = self.silent
-
- if silent:
- return
-
- browse = self.browse
- sheetData = self.getSheetData()
- logData = sheetData.logData
-
- logData.append((isError, indent, msg))
-
- if not browse:
- self.consoleLine(isError, indent, msg)
-
- def getSheetData(self):
- """Deliver the current sheet."""
- sheetName = self.sheetName
- sheetsData = self.sheets
-
- return sheetsData.get(sheetName, AttrDict())
-
- def setSheet(self, newSheet):
+ def setSheet(self, newSheet, force=False, caseSensitive=False):
"""Switch to a named ner sheet.
After the switch, the new sheet will be loaded into memory.
@@ -196,22 +148,22 @@ Module tf.browser.ner.sheets
sheetDir = self.sheetDir
sheetFile = f"{sheetDir}/{newSheet}.xlsx"
- if newSheet not in sheetNames or not fileExists(sheetFile):
+ if newSheet not in sheetNames and not fileExists(sheetFile):
if not browse:
- console(f"Ner sheet {newSheet} ({sheetFile}) does not exist")
+ console(f"NER sheet {newSheet} ({sheetFile}) does not exist")
if newSheet is not None:
return
- if not browse:
- self.setSet("" if newSheet is None else f".{newSheet}")
+ # if not browse:
+ # self.setSet("" if newSheet is None else f".{newSheet}")
if newSheet != sheetName:
sheetName = newSheet
self.sheetName = sheetName
- self.loadSheetData()
+ self.loadSheetData(force=force, caseSensitive=caseSensitive)
- def loadSheetData(self):
+ def loadSheetData(self, force=False, caseSensitive=False):
"""Loads the current ner sheet into memory, if there is one.
If the current ner sheet is None, nothing has to be done.
@@ -232,6 +184,12 @@ Module tf.browser.ner.sheets
return
sheetName = self.sheetName
+ setName = self.setName
+ annoDir = self.annoDir
+ setDir = f"{annoDir}/{setName}"
+ dataFile = f"{setDir}/data.gz"
+ timeFile = f"{setDir}/time.txt"
+ timeKey = "time"
if sheetName is None:
return None
@@ -242,71 +200,151 @@ Module tf.browser.ner.sheets
sheets[sheetName] = AttrDict()
browse = self.browse
- sheetData = self.sheets[sheetName]
+ sheetData = self.getSheetData()
sheetDir = self.sheetDir
sheetFile = f"{sheetDir}/{sheetName}.xlsx"
sheetExists = fileExists(sheetFile)
-
- def timeXls(sheetRela):
- sep = "/" if sheetRela else ""
- sheetPath = f"{sheetDir}/{sheetName}{sep}{sheetRela}.xlsx"
- return mTime(sheetPath)
-
- def timeDir(sheetRela):
- sep = "/" if sheetRela else ""
- dirPath = f"{sheetDir}/{sheetName}{sep}{sheetRela}"
-
- (files, dirs) = dirContents(f"{sheetDir}/{sheetName}{sep}{sheetRela}")
- mFiles = max((mTime(f"{dirPath}/{fl}") for fl in files), default=0)
- mDirs = max((timeDir(f"{sheetRela}{sep}{dr}") for dr in dirs), default=0)
-
- return max((mFiles, mDirs))
-
- sheetUpdated = max((timeXls(""), timeDir(""))) if sheetExists else 0
- needReload = sheetData.get("time", 0) < sheetUpdated
+ sheetUpdated = mTime(sheetFile) if sheetExists else 0
+ needReload = (
+ force
+ or sheetData.caseSensitive != caseSensitive
+ or sheetData.get(timeKey, 0) < sheetUpdated
+ )
showLog = True
if needReload:
- loaded = self.readData(sheetData, sheetUpdated)
+ # try to load from zipped data file first
+ tm = 0
+ dataUptodate = not force and fileExists(timeFile)
+
+ if dataUptodate:
+ with fileOpen(timeFile) as fh:
+ info = fh.read().strip()
+ try:
+ tm = float(info)
+ dataUptodate = sheetUpdated < tm
+ except Exception:
+ dataUptodate = False
+
+ # only load if that data exists and is still up to date
+
+ if dataUptodate:
+ if fileExists(dataFile):
+ with gzip.open(dataFile, mode="rb") as f:
+ data = pickle.load(f)
+
+ if data["caseSensitive"] != caseSensitive:
+ loaded = False
+ else:
+ for k in CLEAR_KEYS:
+ if k in sheetData:
+ sheetData[k].clear()
+
+ for k in SHEET_KEYS:
+ v = data.get(k, None)
+
+ if v is None:
+ if k in sheetData:
+ if type(sheetData[k]) in {list, dict, set}:
+ sheetData[k].clear()
+ else:
+ sheetData[k] = None
+ else:
+ sheetData[k] = v
+
+ sheetData[timeKey] = tm
+ loaded = True
+ else:
+ loaded = False
+ else:
+ loaded = False
if loaded:
+ # we have loaded valid, up-to-date, previously compiled spreadsheet data
self.console("SHEET data: loaded from disk")
else:
+ # now we really heave to read and compile the spreadsheet
self.console("SHEET data: computing from scratch ...", newline=False)
sheetData.logData = []
+ sheetData.caseSensitive = caseSensitive
- self._readSheets(sheetData)
+ self._readSheet()
tm = time.time()
- sheetData.time = tm
+ sheetData[timeKey] = tm
- self._combineSheets(sheetData)
- self._compileSheets(sheetData)
- self._prepareSheets(sheetData)
+ self._compileSheet()
+ self._prepareSheet()
self._processSheet()
- writeKeys = [
- "logData",
- "nameMap",
- "instructions",
- "inventory",
- "triggerFromMatch",
- "hitData",
- ]
- self.writeData(
- {k: sheetData[k] for k in writeKeys},
- tm,
- )
+ with gzip.open(dataFile, mode="wb", compresslevel=GZIP_LEVEL) as f:
+ f.write(
+ pickle.dumps(
+ {k: sheetData[k] for k in SHEET_KEYS},
+ protocol=PICKLE_PROTOCOL,
+ )
+ )
+
+ with fileOpen(timeFile, "w") as fh:
+ fh.write(f"{tm}\n")
+
self.console("done")
showLog = False
else:
+ # the compiled spreadsheet data we have in memory is still up to date
self.console("SHEET data: already in memory and uptodate")
if showLog and not browse:
for x in sheetData.logData:
self.consoleLine(*x)
- def _readSheets(self, sheetData):
+ def makeSheetOfSingleTokens(self):
+ Workbook = self.Workbook
+ sheetName = self.sheetName
+ sheetDir = self.sheetDir
+ sheetPath = f"{sheetDir}/{sheetName}-single.xlsx"
+ sheetData = self.getSheetData()
+ raw = sheetData.raw
+
+ # raw.setdefault(eidkind, {})[normScopeStr] = (r + 1, triggers)
+ words = {}
+
+ for eidkind, eData in raw.items():
+ for scopeStr, (r, triggers) in eData.items():
+ for trigger in triggers:
+ for word in trigger.split():
+ if word.isalpha():
+ words.setdefault(word, {}).setdefault(scopeStr, set()).add(
+ r
+ )
+
+ wb = Workbook()
+ ws = wb.active
+ ws.append(("name", "kind", "scope", "triggers", "origrow"))
+ ws.append(("", "", "", "", ""))
+
+ eids = {}
+
+ for word in sorted(words):
+ wordData = words[word]
+ eid = toSmallId(word).replace(".", " ")
+ n = eids.get(eid, 0)
+
+ eidDis = eid if n == 0 else f"{eid}({n})"
+
+ n += 1
+ eids[eid] = n
+ rows = set()
+
+ for scopeStr in sorted(wordData):
+ rows |= set(wordData[scopeStr])
+
+ rowRep = ",".join(str(r) for r in sorted(rows))
+ ws.append((eidDis, "x", "", word, rowRep))
+
+ wb.save(sheetPath)
+
+ def _readSheet(self):
"""Read all the spreadsheets, the main one and the tweaks.
Store the results in a hierarchy that mimicks the way they are organized in the
@@ -326,307 +364,362 @@ Module tf.browser.ner.sheets
def log(msg):
self.log(False, 0, msg)
- def log1(msg):
- self.log(False, 1, msg)
-
- def log2(msg):
- self.log(False, 2, msg)
-
def err(msg):
self.log(True, 0, msg)
def err1(msg):
self.log(True, 1, msg)
- spec("Reading sheets")
+ sheetData = self.getSheetData()
+ caseSensitive = sheetData.caseSensitive
- sheetData.nameMap = {}
+ nameMap = {}
+ sheetData.nameMap = nameMap
"""Will contain a mapping from entities to names.
The entities are keyed by their (eid, kind) tuple.
The values are names plus the sheet where they are first defined.
"""
- nameMap = sheetData.nameMap
- def readXls(sheetRela):
- sep = "/" if sheetRela else ""
- sheetRep = f"[{sheetRela}]"
+ metaData = {}
+ sheetData.metaData = metaData
+ """Will contain the data of most columns in the spreadsheet.
- sheetPath = f"{sheetDir}/{sheetName}{sep}{sheetRela}.xlsx"
+ The scope and triggers columns are not stored.
- wb = loadXls(sheetPath, data_only=True)
- ws = wb.active
+ For each row, a key is computed (the small eid based on the name in the first
+ column). Under this key we store the list of values.
+ """
- (headRow, subHeadRow, *rows) = list(ws.rows)
- rows = [row for row in rows if any(c.value for c in row)]
+ rowMap = {}
+ sheetData.rowMap = rowMap
- sheet = {}
+ for k in CLEAR_KEYS:
+ if k in sheetData:
+ sheetData[k].clear()
- idFirstRow = {}
- noTrig = 0
+ for k in SHEET_KEYS:
+ if k in sheetData:
+ if type(sheetData[k]) in {list, dict, set}:
+ sheetData[k].clear()
- def myNormalize(x):
- return normalize(x if normalizeChars is None else normalizeChars(x))
+ spec("Reading sheets")
- for r, row in enumerate(ws.rows):
- if r in {0, 1}:
- continue
- if not any(c.value for c in row):
- continue
+ scopeMap = {}
+ sheetData.scopeMap = scopeMap
- (name, kind, triggerStr) = (
- myNormalize(row[i].value or "") for i in range(3)
- )
- triggers = (
- set()
- if not triggerStr
- else {
- y
- for x in triggerStr.split(";")
- if (y := tnorm(x, spaceEscaped=spaceEscaped)) != ""
- }
- )
- if not name:
- name = list(triggers)[0] if len(triggers) else ""
-
- if name == "":
- if kind:
- msg = (
- f"{sheetRep} row {r + 1:>3}: {kind}: "
- "no entity name and no triggers"
- )
- log(msg)
- continue
- else:
- msg = (
- f"{sheetRep} row {r + 1:>3}: {kind}: "
- f"no entity name, supplied synonym {name}"
- )
- log(msg)
+ sheetPath = f"{sheetDir}/{sheetName}.xlsx"
- if not kind:
- kind = defaultKind
- msg = (
- f"{sheetRep} row {r + 1:>3}: "
- f"no kind name, supplied {defaultKind}"
- )
- log(msg)
+ wb = loadXls(sheetPath, data_only=True)
+ ws = wb.active
+ maxCol = ws.max_column
+ maxRow = ws.max_row
- eid = toSmallId(name, transform=transform)
- eidkind = (eid, kind)
- ekRep = f"{kind}-{eid}"
- firstRowEid = idFirstRow.get((eidkind), None)
+ self.console(f"Sheet with {maxRow} rows and {maxCol} columns")
- if firstRowEid is None:
- idFirstRow[eidkind] = (r, name)
+ raw = {}
+ sheetData.raw = raw
- if len(triggers) == 0:
- if sheetRela == "":
- noTrig += 1
- else:
- sheet[eidkind] = triggers
- else:
- sheet[eidkind] = triggers
+ multiNames = {}
+ noNames = set()
+ noTrigs = set()
+ emptyLines = set()
+ scopeMistakes = {}
- prev = nameMap.get(eidkind, None)
-
- if prev is None:
- nameMap[eidkind] = (name, sheetRela)
- else:
- (prevName, prevSheet) = prev
-
- if prevName != name:
- err(f"{ekRep} in {sheetRep}:")
- log1(f"{prevName}: ✅ in {prevSheet}")
- log1(f"{name}: ❌ in {sheetRep}")
+ def myNormalize(x):
+ return normalize(
+ str(x) if normalizeChars is None else normalizeChars(str(x))
+ )
+ for r, row in enumerate(ws.rows):
+ if r == 0:
+ metaFields = [
+ myNormalize(row[i].value or "")
+ for i in range(maxCol)
+ if i not in {2, 3}
+ ]
+ sheetData.metaFields = metaFields
+
+ continue
+ if r == 1:
+ continue
+ if not any(c.value for c in row):
+ continue
+
+ triggerStr = row[3].value
+
+ if triggerStr is not None and "\n" in triggerStr:
+ triggerRep = triggerStr.replace("\n", "\\n")
+ msg = f"r{r + 1:>3}: newline in trigger string: {triggerRep}"
+ log(msg)
+ continue
+
+ (name, kind, scopeStr, triggerStr) = (
+ myNormalize(row[i].value or "") for i in range(4)
+ )
+ if not name or not triggerStr:
+ if name:
+ noTrigs.add(r + 1)
+ elif triggerStr:
+ noNames.add(r + 1)
else:
- (firstRow, firstName) = firstRowEid
- err(f"{ekRep} in {sheetRep}:")
+ emptyLines.add(r + 1)
+ continue
+
+ triggers = {
+ y
+ for x in triggerStr.split(";")
+ if (
+ y := tnorm(
+ x, spaceEscaped=spaceEscaped, caseSensitive=caseSensitive
+ )
+ )
+ != ""
+ }
- if firstName == name:
- log1(f"{name}: in {firstRow + 1:<3} and {r + 1}")
- log2(repSet(triggers))
- log2(repSet(sheet.get(eidkind, set())))
- else:
- log1(f"{firstName}: in {firstRow + 1:<3}")
- log2(repSet(triggers))
- log1(f"{name}: in {r + 1:<3}")
- log2(repSet(sheet.get(eidkind, set())))
+ for trigger in triggers:
+ rowMap.setdefault(trigger, []).append(r + 1)
- theseTriggers = sheet.get(eidkind, set()) | triggers
- if len(theseTriggers):
- sheet[eidkind] = theseTriggers
+ if len(triggers) == 0:
+ noTrigs.add(r + 1)
+ continue
- if noTrig > 0:
- err(f"{sheetRep} has {noTrig} names without triggers")
- return sheet
+ if not kind:
+ kind = defaultKind
+ msg = f"r{r + 1:>3}: " f"no kind name, supplied {defaultKind}"
+ log(msg)
- def readDir(sheetRela, level):
- sheetRep = f"[{sheetRela}]"
- sep = "/" if sheetRela else ""
+ info = parseScopes(scopeStr, plain=False)
+ warnings = info["warning"]
- (files, dirs) = dirContents(f"{sheetDir}/{sheetName}{sep}{sheetRela}")
+ if len(warnings):
+ scopeMistakes[r + 1] = "; ".join(warnings)
+ continue
- sheetSingle = {}
- sheetRange = {}
+ normScopeStr = info["normal"]
- for file in files:
- if file == DS_STORE:
- continue
+ if normScopeStr != "":
+ scopes = info["result"]
- match = SHEET_RE.match(file)
- if not match:
- log(f"{sheetRep} contains unrecognized file {file}")
- continue
+ if normScopeStr not in scopeMap:
+ scopeMap[normScopeStr] = scopes
- (start, end) = match.group(1, 2)
- fileBase = f"{start}{end}"
+ eid = toSmallId(name, transform=transform)
+ eidkind = (eid, kind)
- start = int(start)
- end = int(end[1:]) if end else None
- key = start if end is None else (start, end)
+ if normScopeStr in raw.get(eidkind, {}):
+ ar = raw[eidkind][normScopeStr][0]
+ multiNames[r + 1] = f"({normScopeStr}) {nameMap[eidkind]} also in r{ar}"
+ continue
+ else:
+ nameMap[eidkind] = name
+
+ raw.setdefault(eidkind, {})[normScopeStr] = (r + 1, triggers)
+
+ metaKey = f"{eid}-{kind}"
+ metaData[metaKey] = [
+ myNormalize(row[i].value or "")
+ for i in range(maxCol)
+ if i not in {2, 3}
+ ]
+
+ for diags, isdict, label in (
+ (emptyLines, False, "without a name and triggers"),
+ (multiNames, True, "with a duplicate name"),
+ (noNames, False, "without a name"),
+ (scopeMistakes, True, "with scope mistakes"),
+ (noTrigs, False, "without triggers"),
+ ):
+ n = len(diags)
+
+ if n > 0:
+ if isdict:
+ if n > 0:
+ plural = "" if n == 1 else "s"
+ err(f"{n} row{plural} {label}:")
+
+ for r in sorted(diags):
+ msg = diags[r]
+ err1(f"r{r:>3}: {msg}")
+ else:
+ rep = ", ".join(str(x) for x in sorted(diags)[0:10])
+ plural = "" if n == 1 else "s"
+ err(f"{n} row{plural} {label}:\n\te.g.: {rep}")
- sheetDest = sheetSingle if end is None else sheetRange
- sheetDest[key] = readXls(f"{sheetRela}/{fileBase}")
+ def _compileSheet(self):
+ """Compiles the info in tweaked sheets into complete sheets.
- sheetSubdirs = {}
+ For every tweak spreadsheet, a copy of its parent sheet will be made,
+ and the info of the tweak sheet will be applied to that copy,
+ adding to or overriding the parent sheet.
- for dr in dirs:
- if level >= 3:
- log(f"{sheetRep} is at max depth, yet contains subdir {dr}")
- continue
+ A sheet is basically a mapping of triggers to names.
- if not dr.isdecimal():
- log(f"{sheetRep} contains non-numeric subdir {dr}")
- continue
+ We also maintain a mapping from tweak sheets to triggers, so that we can
+ know later on which sheet assigned which trigger to which name.
- sheetSubdirs[int(dr)] = readDir(f"{sheetRela}{sep}{dr}", level + 1)
+ The tweak may remove triggers from the sheet. We have to adapt the tMap
+ for that.
+ """
+ sheetData = self.getSheetData()
+ compiled = AttrDict()
+ sheetData.compiled = compiled
+ nameMap = sheetData.nameMap
- return AttrDict(sng=sheetSingle, rng=sheetRange, sdr=sheetSubdirs)
+ def spec(msg):
+ self.log(None, 0, msg)
- sheetMain = readXls("")
- sheetSubdirs = readDir("", 1)
- sheetData.raw = AttrDict(main=sheetMain, sdr=sheetSubdirs)
+ def log(msg):
+ self.log(False, 0, msg)
- def _combineSheets(self, sheetData):
- """Combines the spreadsheet info in single-section spreadsheets.
+ def err(msg):
+ self.log(True, 0, msg)
- Among the tweaks, there may be *ranged* spreadsheets, i.e. having the name
- *start*`-`*end*, which indicate that they contain tweaks for sections
- *start* to *end*. These will be converted to individual spreadsheet
- *start*, *start + 1*, ..., *end - 1*, *end*.
- """
- raw = sheetData.raw
+ def errProblem(problem):
+ for indent, msg in problem:
+ self.log(True, indent, msg)
- sheetMain = raw.main
- sheetTweaked = raw.sdr
+ tFullMap = {}
- # combine the info in ranged sheets into single number sheets
+ def getTriggerRep(tr):
+ triggerInfo = tFullMap[tr]
- combined = AttrDict(sheet=sheetMain, tweaks=AttrDict())
- sheetData.combined = combined
+ result = []
- def combineDir(info, dest):
- ranged = info.rng or {}
- single = info.sng or {}
- subdirs = info.sdr or {}
+ for scope in sorted(triggerInfo):
+ scopeRep = "" if not scope else f"({scope})"
+ scopeInfo = triggerInfo[scope]
- for (start, end), sheet in sorted(ranged.items()):
- for i in range(start, end + 1):
- updateDest = dest.setdefault(i, AttrDict()).setdefault("sheet", {})
- for eidkind, triggers in sheet.items():
- updateDest[eidkind] = triggers
+ for eidkind in sorted(scopeInfo):
+ name = sheetData.nameMap[eidkind]
+ rowRep = ",".join(str(r) for r in sorted(scopeInfo[eidkind]))
+ result.append(f"'{tr}'{scopeRep} r{rowRep} for {name}")
+ return result
- for i, sheet in single.items():
- updateDest = dest.setdefault(i, AttrDict()).setdefault("sheet", {})
- for eidkind, triggers in sheet.items():
- updateDest[eidkind] = triggers
+ spec("Checking scopes ...")
- for i, tweaks in subdirs.items():
- updateDest = dest.setdefault(i, AttrDict()).setdefault("tweaks", {})
- combineDir(tweaks, updateDest)
+ raw = sheetData.raw
+ scopeMap = sheetData.scopeMap
+ intervals = partitionScopes(scopeMap)
- combineDir(sheetTweaked, combined.tweaks)
+ problems = set()
- def _compileSheets(self, sheetData):
- """Compiles the info in tweaked sheets into complete sheets.
+ for b, e, scopeStrs in [[None, None, None]] + intervals:
+ scopeStrSet = {""} if scopeStrs is None else set(scopeStrs)
+ intv = (b, e) if b is not None and e is not None else ()
- For every tweak spreadsheet, a copy of its parent sheet will be made,
- and the info of the tweak sheet will be applied to that copy,
- adding to or overriding the parent sheet.
+ spec(repScope(intv))
- A sheet is basically a mapping of triggers to names.
+ thisCompiled = AttrDict()
+ compiled[intv] = thisCompiled
- We also maintain a mapping from tweak sheets to triggers, so that we can
- know later on which sheet assigned which trigger to which name.
+ # check which triggers are a substring in which other triggers
+ # these triggers can never be found!
- The tweak may remove triggers from the sheet. We have to adapt the tMap
- for that.
- """
- nameMap = sheetData.nameMap
- combined = sheetData.combined
+ tFullMap.clear()
- compiled = AttrDict()
- sheetData.compiled = compiled
+ for eidkind, info in raw.items():
+ if "" in info:
+ (r, triggers) = info[""]
- def spec(msg):
- self.log(None, 0, msg)
+ for trigger in triggers:
+ tFullMap.setdefault(trigger, {}).setdefault("", {}).setdefault(
+ eidkind, set()
+ ).add(r)
- def err(msg):
- self.log(True, 0, msg)
+ for scopeStr, (r, triggers) in info.items():
+ if scopeStr == "":
+ continue
+
+ if scopeStr in scopeStrSet:
+ for trigger in triggers:
+ tFullMap.setdefault(trigger, {}).setdefault(
+ scopeStr, {}
+ ).setdefault(eidkind, set()).add(r)
- spec("Compiling data")
+ # now compile the result information: tMap, newSheet
- def compileSheet(path, parentData, info, dest):
- parentSheet = parentData.sheet
- sheet = info.sheet
newSheet = {}
- dest.sheet = newSheet
- parentTMap = parentData["tMap"] or {}
- newTMap = {}
- dest["tMap"] = newTMap
- pathRep = f"[{'.'.join(str(i) for i in path)}]"
+ thisCompiled.sheet = newSheet
- for eidkind, triggers in parentSheet.items():
- newSheet[eidkind] = triggers
+ tMap = {}
+ thisCompiled.tMap = tMap
- for eidkind, triggers in sheet.items():
- if len(triggers) == 0:
- if eidkind in newSheet:
- del newSheet[eidkind]
- else:
- err(f"{pathRep} {nameMap[eidkind][0]} has no triggers")
- else:
+ for eidkind, info in raw.items():
+ if "" in info:
+ (r, triggers) = info[""]
newSheet[eidkind] = triggers
for trigger in triggers:
- newTMap[trigger] = tuple(str(k) for k in path)
+ tMap[trigger] = ""
- for eidkind, triggers in newSheet.items():
- for trigger in triggers:
- if trigger not in newTMap:
- newTMap[trigger] = parentTMap[trigger]
-
- def compileDir(path, parentData, info, dest):
- if "sheet" in info:
- compileSheet(path, parentData, info, dest)
- parentData = AttrDict(sheet=dest.sheet, tMap=dest["tMap"])
-
- tweaks = info.tweaks or {}
- tweakDest = dest.setdefault("tweaks", {})
-
- for k in sorted(tweaks):
- compileDir(
- path + (k,),
- parentData,
- tweaks[k],
- tweakDest.setdefault(k, AttrDict()),
- )
+ for scopeStr, (r, triggers) in info.items():
+ if scopeStr == "":
+ continue
+
+ if scopeStr in scopeStrSet:
+ newSheet[eidkind] = triggers
+
+ for trigger in triggers:
+ tMap[trigger] = scopeStr
+
+ # the sheet is now compiled, all compiled data has been created
+ # the rest only deals with error reporting
+
+ ambi = 0
+ clashes = 0
+
+ for trigger in sorted(tFullMap):
+ problem = []
+ triggerInfo = tFullMap[trigger]
- compileDir((), combined, combined, compiled)
+ for scopeStr in sorted(triggerInfo):
+ scopeInfo = triggerInfo[scopeStr]
+ scopeRep = f" in scope {scopeStr}" if scopeStr else ""
- def _prepareSheets(self, sheetData):
+ thisAmbi = False
+ theseClashes = 0
+
+ if len(scopeInfo) > 1:
+ problem.append((0, f"Ambi: '{trigger}'{scopeRep}: "))
+ thisAmbi = True
+
+ for eidkind, rs in scopeInfo.items():
+ name = nameMap[eidkind]
+ rowRep = ",".join(str(r) for r in rs)
+ nRs = len(rs)
+
+ if nRs > 1:
+ theseClashes += 1
+
+ if thisAmbi:
+ problem.append((1, f"{name}: {rowRep}"))
+ elif nRs > 1:
+ problem.append(
+ (0, f"Clash: '{trigger}' for {name}: {rowRep}")
+ )
+
+ clashes += theseClashes
+
+ if len(problem):
+ problem = tuple(problem)
+
+ if problem not in problems:
+ if thisAmbi:
+ ambi += 1
+ if theseClashes:
+ clashes += theseClashes
+
+ problems.add(problem)
+ errProblem(problem)
+
+ if ambi > 0:
+ err(f"Ambiguous triggers: {ambi} x")
+
+ if clashes > 0:
+ err(f"Reused triggers scope: {clashes} x")
+
+ def _prepareSheet(self):
"""Transform the sheets into instructions.
Now we have complete sheets for every context, the inheritance is resolved.
@@ -665,55 +758,30 @@ Module tf.browser.ner.sheets
"""
spaceEscaped = self.spaceEscaped
- nameMap = sheetData.nameMap
+
+ sheetData = self.getSheetData()
compiled = sheetData.compiled
+ caseSensitive = sheetData.caseSensitive
instructions = {}
sheetData.instructions = instructions
- def spec(msg):
- self.log(None, 0, msg)
-
- def log(msg):
- self.log(False, 0, msg)
-
- def log1(msg):
- self.log(False, 1, msg)
-
- def log2(msg):
- self.log(False, 2, msg)
-
- def log3(msg):
- self.log(False, 3, msg)
-
- def err(msg):
- self.log(True, 0, msg)
-
- def err1(msg):
- self.log(True, 1, msg)
-
- def err2(msg):
- self.log(True, 2, msg)
-
- spec("Checking triggers ...")
-
- def prepareSheet(path, info):
- sheet = info.sheet
- tMap = info["tMap"]
- sheetR = ".".join(path)
- sheetRep = f"[{sheetR}]"
+ for intv, thisCompiled in compiled.items():
+ sheet = thisCompiled.sheet
+ tMap = thisCompiled.tMap
triggerSet = set()
tPos = {}
idMap = {}
- prepared = dict(tPos=tPos, tMap=tMap)
-
- instructions[path] = prepared
+ theseInstructions = dict(tPos=tPos, tMap=tMap)
+ instructions[intv] = theseInstructions
for eidkind, triggers in sheet.items():
for trigger in triggers:
- triggerT = toTokens(trigger, spaceEscaped=spaceEscaped)
+ triggerT = toTokens(
+ trigger, spaceEscaped=spaceEscaped, caseSensitive=caseSensitive
+ )
triggerSet.add(triggerT)
idMap.setdefault(trigger, []).append(eidkind)
@@ -721,56 +789,10 @@ Module tf.browser.ner.sheets
for i, token in enumerate(triggerT):
tPos.setdefault(i, {}).setdefault(token, set()).add(triggerT)
- prepared["idMap"] = {
+ theseInstructions["idMap"] = {
trigger: eidkinds[0] for (trigger, eidkinds) in idMap.items()
}
- nEnt = len(sheet)
- nTriggers = sum(len(triggers) for triggers in sheet.values())
- noTriggers = sum(1 for triggers in sheet.values() if len(triggers) == 0)
-
- ambis = collections.defaultdict(list)
-
- for trigger, eidkinds in sorted(idMap.items()):
- if len(eidkinds) <= 1:
- continue
-
- tPath = tMap[trigger]
-
- if path != tPath:
- continue
-
- for eidkind in eidkinds:
- name = nameMap[eidkind][0]
- ambis[trigger].append(name)
-
- log(f"{sheetRep} {nTriggers} triggers for {nEnt} entities")
-
- if noTriggers > 0:
- err1(f"{noTriggers} without triggers")
-
- nAmbi = len(ambis)
-
- if nAmbi > 0:
- err1(f"{nAmbi} ambiguous triggers")
-
- for trigger, names in ambis.items():
- err2(trigger)
-
- for name in names:
- log3(name)
-
- def prepareDir(path, info):
- if "sheet" in info:
- prepareSheet(path, info)
-
- tweaks = info.tweaks or {}
-
- for k in sorted(tweaks):
- prepareDir(path + (str(k),), tweaks[k])
-
- prepareDir((), compiled)
-
def _processSheet(self):
"""Generates derived data structures out of the source sheet.
@@ -806,8 +828,7 @@ Module tf.browser.ner.sheets
if not self.properlySetup:
return
- sheetName = self.sheetName
- sheetData = self.sheets[sheetName]
+ sheetData = self.getSheetData()
inventory = sheetData.inventory
instructions = sheetData.instructions
@@ -822,13 +843,13 @@ Module tf.browser.ner.sheets
idMap = data["idMap"]
tMap = data["tMap"]
- for trigger, tPath in tMap.items():
- eidkind = idMap[trigger]
- sheet = ".".join(tPath)
-
- occs = inventory.get(eidkind, {}).get(trigger, {}).get(tPath, {})
+ for trigger, scope in tMap.items():
+ eidkind = idMap.get(trigger, None)
+ if eidkind is None:
+ continue
- hitData.setdefault(eidkind, {})[(trigger, sheet)] = len(occs)
+ occs = inventory.get(eidkind, {}).get(trigger, {}).get(scope, [])
+ hitData.setdefault(eidkind, {})[(trigger, scope)] = len(occs)
def _markEntities(self):
"""Marks up the members of the inventory as entities.
@@ -842,8 +863,7 @@ Module tf.browser.ner.sheets
if not self.properlySetup:
return
- sheetName = self.sheetName
- sheetData = self.sheets[sheetName]
+ sheetData = self.getSheetData()
inventory = sheetData.inventory
newEntities = []
@@ -852,145 +872,35 @@ Module tf.browser.ner.sheets
for eidkind, entData in inventory.items():
for trigger, triggerData in entData.items():
- for sheet, matches in triggerData.items():
- newEntities.append((eidkind, matches))
+ for scope, matches in triggerData.items():
for match in matches:
- triggerFromMatch[match] = (trigger, ",".join(sheet))
-
- self._addToSet(newEntities)
-
- def showSheetsRaw(self, main=False):
- src = self.getSheetData()
-
- if src is None:
- self.console("Nothing to show")
-
- src = src.raw
- nameMap = src.nameMap
-
- def showSheet(sheet, tab):
- for eidkind, triggers in sorted(sheet.items()):
- (name, sheetRela) = nameMap[eidkind]
- triggerRep = "|".join(
- t for t in sorted(triggers, key=lambda x: (-len(x), x))
- )
- self.console(f"{tab} '{name}' {eidkind} : {triggerRep}")
- self.console(f"{tab} ---")
-
- def showDir(head, tweaks, level):
- tab = " " * level
- self.console(f"{tab}{head}")
-
- rng = tweaks.rng or {}
- for b, e in sorted(rng):
- self.console(f"{tab} {b}-{e}.xslx")
- showSheet(rng[(b, e)], tab)
-
- sng = tweaks.sng or {}
- for k in sorted(sng):
- self.console(f"{tab} {k}.xslx")
- showSheet(sng[k], tab)
-
- sdr = tweaks.sdr or {}
-
- for k in sorted(sdr):
- showDir(k, sdr[k], level + 1)
-
- if main:
- showSheet(src.main, "")
-
- showDir("", src.sdr, 0)
-
- def showSheetsCombined(self, main=False):
- src = self.getSheetData()
-
- if src is None:
- self.console("Nothing to show")
-
- src = src.combined
- nameMap = src.nameMap
+ triggerFromMatch[match] = (trigger, scope)
- def showSheet(sheet, tab):
- for eidkind, triggers in sorted(sheet.items()):
- (name, sheetRela) = nameMap[eidkind]
- triggerRep = "|".join(
- t for t in sorted(triggers, key=lambda x: (-len(x), x))
- )
- self.console(f"{tab} '{name}' {eidkind} : {triggerRep}")
- self.console(f"{tab} ---")
-
- def showDir(head, info, level):
- tab = " " * level
- self.console(f"{tab}{head}")
-
- if "sheet" in info:
- if main or level > 0:
- showSheet(info.sheet, tab)
-
- tweaks = info.tweaks or {}
-
- for k in sorted(tweaks):
- showDir(k, tweaks[k], level + 1)
-
- showDir("", src, 0)
-
- def showSheetsCompiled(self, main=False):
- nameMap = self.nameMap
- src = self.compiled
-
- def showSheet(info, tab):
- sheet = info.sheet
- tMap = info["tMap"]
-
- for eidkind, triggers in sorted(sheet.items()):
- triggerSources = {tMap[t] for t in triggers}
- nTriggers = len(triggers)
-
- if not main and (nTriggers == 0 or triggerSources == {()}):
- continue
-
- (name, sheetRela) = nameMap[eidkind]
-
- if nTriggers == 0:
- triggerInfo = "X"
- else:
- sourceRep = (
- "X"
- if len(triggerSources) == 0
- else list(list(triggerSources)[0])
- )
- triggerRep = "|".join(
- t for t in sorted(triggers, key=lambda x: (-len(x), x))
- )
- triggerInfo = f"{sourceRep} => {triggerRep}"
-
- self.console(f"{tab} '{name}' {eidkind} : {triggerInfo}")
- self.console(f"{tab} ---")
-
- def showDir(head, info, level):
- tab = " " * level
- self.console(f"{tab}{head}")
+ newEntities.append((eidkind, matches))
- if "sheet" in info:
- if main or level > 0:
- showSheet(info, tab)
+ self._addToSet(newEntities)
- tweaks = info.tweaks or {}
+ def getSheetData(self):
+ """Deliver the current sheet."""
+ sheetName = self.sheetName
+ sheetsData = self.sheets
- for k in sorted(tweaks):
- showDir(k, tweaks[k], level + 1)
+ return sheetsData.setdefault(sheetName, AttrDict())
- showDir("", src, 0)
+ def log(self, isError, indent, msg):
+ silent = self.silent
- def showInstructions(self):
- src = self.getSheetData()
+ if silent and not isError:
+ return
- if src is None:
- self.console("Nothing to show")
+ browse = self.browse
+ sheetData = self.getSheetData()
+ logData = sheetData.logData
- instructions = src.instructions
+ logData.append((isError, indent, msg))
- self.console("\n".join(f"{path}" for path in instructions))
+ if not browse:
+ self.consoleLine(isError, indent, msg)
@@ -1011,7 +921,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Sheets:
def __init__(self, sheets=None):
@@ -1019,6 +929,7 @@ Classes
if CI.importOK(hint=True):
openpyxl = CI.importGet()
self.loadXls = openpyxl.load_workbook
+ self.Workbook = openpyxl.Workbook
else:
self.properlySetup = False
return None
@@ -1044,9 +955,6 @@ Classes
self.readSheets()
- # if not browse:
- # self.loadSheetData()
-
def readSheets(self):
"""Read the list current ner sheets (again).
@@ -1060,84 +968,12 @@ Classes
self.sheetNames = sheetNames
for sheetFile in dirContents(sheetDir)[0]:
- if extNm(sheetFile) == "xlsx":
- sheetName = sheetFile.removesuffix(".xlsx")
- sheetNames.add(sheetName)
- setNames.add(f".{sheetName}")
-
- def readData(self, sheetData, sheetUpdated):
- setName = self.setName
- annoDir = self.annoDir
- setDir = f"{annoDir}/{setName}"
- dataFile = f"{setDir}/data.gz"
- timeFile = f"{setDir}/time.txt"
-
- tm = 0
-
- uptodate = fileExists(timeFile)
-
- if uptodate:
- with fileOpen(timeFile) as fh:
- info = fh.read().strip()
-
- try:
- tm = float(info)
- uptodate = sheetUpdated < tm
- except Exception:
- uptodate = False
-
- if not uptodate:
- return False
-
- if fileExists(dataFile):
- with gzip.open(dataFile, mode="rb") as f:
- data = pickle.load(f)
-
- for k, v in data.items():
- sheetData[k] = v
-
- sheetData.time = tm
-
- return True
- else:
- return False
-
- def writeData(self, sheetData, tm):
- setName = self.setName
- annoDir = self.annoDir
- setDir = f"{annoDir}/{setName}"
- dataFile = f"{setDir}/data.gz"
- timeFile = f"{setDir}/time.txt"
-
- with gzip.open(dataFile, mode="wb", compresslevel=GZIP_LEVEL) as f:
- f.write(pickle.dumps(sheetData, protocol=PICKLE_PROTOCOL))
-
- with fileOpen(timeFile, "w") as fh:
- fh.write(f"{tm}\n")
-
- def log(self, isError, indent, msg):
- silent = self.silent
-
- if silent:
- return
-
- browse = self.browse
- sheetData = self.getSheetData()
- logData = sheetData.logData
-
- logData.append((isError, indent, msg))
-
- if not browse:
- self.consoleLine(isError, indent, msg)
-
- def getSheetData(self):
- """Deliver the current sheet."""
- sheetName = self.sheetName
- sheetsData = self.sheets
-
- return sheetsData.get(sheetName, AttrDict())
+ if extNm(sheetFile) == "xlsx":
+ sheetName = sheetFile.removesuffix(".xlsx")
+ sheetNames.add(sheetName)
+ setNames.add(f".{sheetName}")
- def setSheet(self, newSheet):
+ def setSheet(self, newSheet, force=False, caseSensitive=False):
"""Switch to a named ner sheet.
After the switch, the new sheet will be loaded into memory.
@@ -1158,22 +994,22 @@ Classes
sheetDir = self.sheetDir
sheetFile = f"{sheetDir}/{newSheet}.xlsx"
- if newSheet not in sheetNames or not fileExists(sheetFile):
+ if newSheet not in sheetNames and not fileExists(sheetFile):
if not browse:
- console(f"Ner sheet {newSheet} ({sheetFile}) does not exist")
+ console(f"NER sheet {newSheet} ({sheetFile}) does not exist")
if newSheet is not None:
return
- if not browse:
- self.setSet("" if newSheet is None else f".{newSheet}")
+ # if not browse:
+ # self.setSet("" if newSheet is None else f".{newSheet}")
if newSheet != sheetName:
sheetName = newSheet
self.sheetName = sheetName
- self.loadSheetData()
+ self.loadSheetData(force=force, caseSensitive=caseSensitive)
- def loadSheetData(self):
+ def loadSheetData(self, force=False, caseSensitive=False):
"""Loads the current ner sheet into memory, if there is one.
If the current ner sheet is None, nothing has to be done.
@@ -1194,6 +1030,12 @@ Classes
return
sheetName = self.sheetName
+ setName = self.setName
+ annoDir = self.annoDir
+ setDir = f"{annoDir}/{setName}"
+ dataFile = f"{setDir}/data.gz"
+ timeFile = f"{setDir}/time.txt"
+ timeKey = "time"
if sheetName is None:
return None
@@ -1204,71 +1046,151 @@ Classes
sheets[sheetName] = AttrDict()
browse = self.browse
- sheetData = self.sheets[sheetName]
+ sheetData = self.getSheetData()
sheetDir = self.sheetDir
sheetFile = f"{sheetDir}/{sheetName}.xlsx"
sheetExists = fileExists(sheetFile)
-
- def timeXls(sheetRela):
- sep = "/" if sheetRela else ""
- sheetPath = f"{sheetDir}/{sheetName}{sep}{sheetRela}.xlsx"
- return mTime(sheetPath)
-
- def timeDir(sheetRela):
- sep = "/" if sheetRela else ""
- dirPath = f"{sheetDir}/{sheetName}{sep}{sheetRela}"
-
- (files, dirs) = dirContents(f"{sheetDir}/{sheetName}{sep}{sheetRela}")
- mFiles = max((mTime(f"{dirPath}/{fl}") for fl in files), default=0)
- mDirs = max((timeDir(f"{sheetRela}{sep}{dr}") for dr in dirs), default=0)
-
- return max((mFiles, mDirs))
-
- sheetUpdated = max((timeXls(""), timeDir(""))) if sheetExists else 0
- needReload = sheetData.get("time", 0) < sheetUpdated
+ sheetUpdated = mTime(sheetFile) if sheetExists else 0
+ needReload = (
+ force
+ or sheetData.caseSensitive != caseSensitive
+ or sheetData.get(timeKey, 0) < sheetUpdated
+ )
showLog = True
if needReload:
- loaded = self.readData(sheetData, sheetUpdated)
+ # try to load from zipped data file first
+ tm = 0
+ dataUptodate = not force and fileExists(timeFile)
+
+ if dataUptodate:
+ with fileOpen(timeFile) as fh:
+ info = fh.read().strip()
+ try:
+ tm = float(info)
+ dataUptodate = sheetUpdated < tm
+ except Exception:
+ dataUptodate = False
+
+ # only load if that data exists and is still up to date
+
+ if dataUptodate:
+ if fileExists(dataFile):
+ with gzip.open(dataFile, mode="rb") as f:
+ data = pickle.load(f)
+
+ if data["caseSensitive"] != caseSensitive:
+ loaded = False
+ else:
+ for k in CLEAR_KEYS:
+ if k in sheetData:
+ sheetData[k].clear()
+
+ for k in SHEET_KEYS:
+ v = data.get(k, None)
+
+ if v is None:
+ if k in sheetData:
+ if type(sheetData[k]) in {list, dict, set}:
+ sheetData[k].clear()
+ else:
+ sheetData[k] = None
+ else:
+ sheetData[k] = v
+
+ sheetData[timeKey] = tm
+ loaded = True
+ else:
+ loaded = False
+ else:
+ loaded = False
if loaded:
+ # we have loaded valid, up-to-date, previously compiled spreadsheet data
self.console("SHEET data: loaded from disk")
else:
+ # now we really heave to read and compile the spreadsheet
self.console("SHEET data: computing from scratch ...", newline=False)
sheetData.logData = []
+ sheetData.caseSensitive = caseSensitive
- self._readSheets(sheetData)
+ self._readSheet()
tm = time.time()
- sheetData.time = tm
+ sheetData[timeKey] = tm
- self._combineSheets(sheetData)
- self._compileSheets(sheetData)
- self._prepareSheets(sheetData)
+ self._compileSheet()
+ self._prepareSheet()
self._processSheet()
- writeKeys = [
- "logData",
- "nameMap",
- "instructions",
- "inventory",
- "triggerFromMatch",
- "hitData",
- ]
- self.writeData(
- {k: sheetData[k] for k in writeKeys},
- tm,
- )
+ with gzip.open(dataFile, mode="wb", compresslevel=GZIP_LEVEL) as f:
+ f.write(
+ pickle.dumps(
+ {k: sheetData[k] for k in SHEET_KEYS},
+ protocol=PICKLE_PROTOCOL,
+ )
+ )
+
+ with fileOpen(timeFile, "w") as fh:
+ fh.write(f"{tm}\n")
+
self.console("done")
showLog = False
else:
+ # the compiled spreadsheet data we have in memory is still up to date
self.console("SHEET data: already in memory and uptodate")
if showLog and not browse:
for x in sheetData.logData:
self.consoleLine(*x)
- def _readSheets(self, sheetData):
+ def makeSheetOfSingleTokens(self):
+ Workbook = self.Workbook
+ sheetName = self.sheetName
+ sheetDir = self.sheetDir
+ sheetPath = f"{sheetDir}/{sheetName}-single.xlsx"
+ sheetData = self.getSheetData()
+ raw = sheetData.raw
+
+ # raw.setdefault(eidkind, {})[normScopeStr] = (r + 1, triggers)
+ words = {}
+
+ for eidkind, eData in raw.items():
+ for scopeStr, (r, triggers) in eData.items():
+ for trigger in triggers:
+ for word in trigger.split():
+ if word.isalpha():
+ words.setdefault(word, {}).setdefault(scopeStr, set()).add(
+ r
+ )
+
+ wb = Workbook()
+ ws = wb.active
+ ws.append(("name", "kind", "scope", "triggers", "origrow"))
+ ws.append(("", "", "", "", ""))
+
+ eids = {}
+
+ for word in sorted(words):
+ wordData = words[word]
+ eid = toSmallId(word).replace(".", " ")
+ n = eids.get(eid, 0)
+
+ eidDis = eid if n == 0 else f"{eid}({n})"
+
+ n += 1
+ eids[eid] = n
+ rows = set()
+
+ for scopeStr in sorted(wordData):
+ rows |= set(wordData[scopeStr])
+
+ rowRep = ",".join(str(r) for r in sorted(rows))
+ ws.append((eidDis, "x", "", word, rowRep))
+
+ wb.save(sheetPath)
+
+ def _readSheet(self):
"""Read all the spreadsheets, the main one and the tweaks.
Store the results in a hierarchy that mimicks the way they are organized in the
@@ -1288,307 +1210,362 @@ Classes
def log(msg):
self.log(False, 0, msg)
- def log1(msg):
- self.log(False, 1, msg)
-
- def log2(msg):
- self.log(False, 2, msg)
-
def err(msg):
self.log(True, 0, msg)
def err1(msg):
self.log(True, 1, msg)
- spec("Reading sheets")
+ sheetData = self.getSheetData()
+ caseSensitive = sheetData.caseSensitive
- sheetData.nameMap = {}
+ nameMap = {}
+ sheetData.nameMap = nameMap
"""Will contain a mapping from entities to names.
The entities are keyed by their (eid, kind) tuple.
The values are names plus the sheet where they are first defined.
"""
- nameMap = sheetData.nameMap
- def readXls(sheetRela):
- sep = "/" if sheetRela else ""
- sheetRep = f"[{sheetRela}]"
+ metaData = {}
+ sheetData.metaData = metaData
+ """Will contain the data of most columns in the spreadsheet.
- sheetPath = f"{sheetDir}/{sheetName}{sep}{sheetRela}.xlsx"
+ The scope and triggers columns are not stored.
- wb = loadXls(sheetPath, data_only=True)
- ws = wb.active
-
- (headRow, subHeadRow, *rows) = list(ws.rows)
- rows = [row for row in rows if any(c.value for c in row)]
-
- sheet = {}
+ For each row, a key is computed (the small eid based on the name in the first
+ column). Under this key we store the list of values.
+ """
- idFirstRow = {}
- noTrig = 0
+ rowMap = {}
+ sheetData.rowMap = rowMap
- def myNormalize(x):
- return normalize(x if normalizeChars is None else normalizeChars(x))
+ for k in CLEAR_KEYS:
+ if k in sheetData:
+ sheetData[k].clear()
- for r, row in enumerate(ws.rows):
- if r in {0, 1}:
- continue
- if not any(c.value for c in row):
- continue
+ for k in SHEET_KEYS:
+ if k in sheetData:
+ if type(sheetData[k]) in {list, dict, set}:
+ sheetData[k].clear()
- (name, kind, triggerStr) = (
- myNormalize(row[i].value or "") for i in range(3)
- )
- triggers = (
- set()
- if not triggerStr
- else {
- y
- for x in triggerStr.split(";")
- if (y := tnorm(x, spaceEscaped=spaceEscaped)) != ""
- }
- )
- if not name:
- name = list(triggers)[0] if len(triggers) else ""
-
- if name == "":
- if kind:
- msg = (
- f"{sheetRep} row {r + 1:>3}: {kind}: "
- "no entity name and no triggers"
- )
- log(msg)
- continue
- else:
- msg = (
- f"{sheetRep} row {r + 1:>3}: {kind}: "
- f"no entity name, supplied synonym {name}"
- )
- log(msg)
+ spec("Reading sheets")
- if not kind:
- kind = defaultKind
- msg = (
- f"{sheetRep} row {r + 1:>3}: "
- f"no kind name, supplied {defaultKind}"
- )
- log(msg)
+ scopeMap = {}
+ sheetData.scopeMap = scopeMap
- eid = toSmallId(name, transform=transform)
- eidkind = (eid, kind)
- ekRep = f"{kind}-{eid}"
- firstRowEid = idFirstRow.get((eidkind), None)
+ sheetPath = f"{sheetDir}/{sheetName}.xlsx"
- if firstRowEid is None:
- idFirstRow[eidkind] = (r, name)
+ wb = loadXls(sheetPath, data_only=True)
+ ws = wb.active
+ maxCol = ws.max_column
+ maxRow = ws.max_row
- if len(triggers) == 0:
- if sheetRela == "":
- noTrig += 1
- else:
- sheet[eidkind] = triggers
- else:
- sheet[eidkind] = triggers
+ self.console(f"Sheet with {maxRow} rows and {maxCol} columns")
- prev = nameMap.get(eidkind, None)
+ raw = {}
+ sheetData.raw = raw
- if prev is None:
- nameMap[eidkind] = (name, sheetRela)
- else:
- (prevName, prevSheet) = prev
+ multiNames = {}
+ noNames = set()
+ noTrigs = set()
+ emptyLines = set()
+ scopeMistakes = {}
- if prevName != name:
- err(f"{ekRep} in {sheetRep}:")
- log1(f"{prevName}: ✅ in {prevSheet}")
- log1(f"{name}: ❌ in {sheetRep}")
+ def myNormalize(x):
+ return normalize(
+ str(x) if normalizeChars is None else normalizeChars(str(x))
+ )
+ for r, row in enumerate(ws.rows):
+ if r == 0:
+ metaFields = [
+ myNormalize(row[i].value or "")
+ for i in range(maxCol)
+ if i not in {2, 3}
+ ]
+ sheetData.metaFields = metaFields
+
+ continue
+ if r == 1:
+ continue
+ if not any(c.value for c in row):
+ continue
+
+ triggerStr = row[3].value
+
+ if triggerStr is not None and "\n" in triggerStr:
+ triggerRep = triggerStr.replace("\n", "\\n")
+ msg = f"r{r + 1:>3}: newline in trigger string: {triggerRep}"
+ log(msg)
+ continue
+
+ (name, kind, scopeStr, triggerStr) = (
+ myNormalize(row[i].value or "") for i in range(4)
+ )
+ if not name or not triggerStr:
+ if name:
+ noTrigs.add(r + 1)
+ elif triggerStr:
+ noNames.add(r + 1)
else:
- (firstRow, firstName) = firstRowEid
- err(f"{ekRep} in {sheetRep}:")
+ emptyLines.add(r + 1)
+ continue
+
+ triggers = {
+ y
+ for x in triggerStr.split(";")
+ if (
+ y := tnorm(
+ x, spaceEscaped=spaceEscaped, caseSensitive=caseSensitive
+ )
+ )
+ != ""
+ }
- if firstName == name:
- log1(f"{name}: in {firstRow + 1:<3} and {r + 1}")
- log2(repSet(triggers))
- log2(repSet(sheet.get(eidkind, set())))
- else:
- log1(f"{firstName}: in {firstRow + 1:<3}")
- log2(repSet(triggers))
- log1(f"{name}: in {r + 1:<3}")
- log2(repSet(sheet.get(eidkind, set())))
+ for trigger in triggers:
+ rowMap.setdefault(trigger, []).append(r + 1)
- theseTriggers = sheet.get(eidkind, set()) | triggers
- if len(theseTriggers):
- sheet[eidkind] = theseTriggers
+ if len(triggers) == 0:
+ noTrigs.add(r + 1)
+ continue
- if noTrig > 0:
- err(f"{sheetRep} has {noTrig} names without triggers")
- return sheet
+ if not kind:
+ kind = defaultKind
+ msg = f"r{r + 1:>3}: " f"no kind name, supplied {defaultKind}"
+ log(msg)
- def readDir(sheetRela, level):
- sheetRep = f"[{sheetRela}]"
- sep = "/" if sheetRela else ""
+ info = parseScopes(scopeStr, plain=False)
+ warnings = info["warning"]
- (files, dirs) = dirContents(f"{sheetDir}/{sheetName}{sep}{sheetRela}")
+ if len(warnings):
+ scopeMistakes[r + 1] = "; ".join(warnings)
+ continue
- sheetSingle = {}
- sheetRange = {}
+ normScopeStr = info["normal"]
- for file in files:
- if file == DS_STORE:
- continue
+ if normScopeStr != "":
+ scopes = info["result"]
- match = SHEET_RE.match(file)
- if not match:
- log(f"{sheetRep} contains unrecognized file {file}")
- continue
+ if normScopeStr not in scopeMap:
+ scopeMap[normScopeStr] = scopes
- (start, end) = match.group(1, 2)
- fileBase = f"{start}{end}"
+ eid = toSmallId(name, transform=transform)
+ eidkind = (eid, kind)
- start = int(start)
- end = int(end[1:]) if end else None
- key = start if end is None else (start, end)
+ if normScopeStr in raw.get(eidkind, {}):
+ ar = raw[eidkind][normScopeStr][0]
+ multiNames[r + 1] = f"({normScopeStr}) {nameMap[eidkind]} also in r{ar}"
+ continue
+ else:
+ nameMap[eidkind] = name
+
+ raw.setdefault(eidkind, {})[normScopeStr] = (r + 1, triggers)
+
+ metaKey = f"{eid}-{kind}"
+ metaData[metaKey] = [
+ myNormalize(row[i].value or "")
+ for i in range(maxCol)
+ if i not in {2, 3}
+ ]
+
+ for diags, isdict, label in (
+ (emptyLines, False, "without a name and triggers"),
+ (multiNames, True, "with a duplicate name"),
+ (noNames, False, "without a name"),
+ (scopeMistakes, True, "with scope mistakes"),
+ (noTrigs, False, "without triggers"),
+ ):
+ n = len(diags)
+
+ if n > 0:
+ if isdict:
+ if n > 0:
+ plural = "" if n == 1 else "s"
+ err(f"{n} row{plural} {label}:")
+
+ for r in sorted(diags):
+ msg = diags[r]
+ err1(f"r{r:>3}: {msg}")
+ else:
+ rep = ", ".join(str(x) for x in sorted(diags)[0:10])
+ plural = "" if n == 1 else "s"
+ err(f"{n} row{plural} {label}:\n\te.g.: {rep}")
- sheetDest = sheetSingle if end is None else sheetRange
- sheetDest[key] = readXls(f"{sheetRela}/{fileBase}")
+ def _compileSheet(self):
+ """Compiles the info in tweaked sheets into complete sheets.
- sheetSubdirs = {}
+ For every tweak spreadsheet, a copy of its parent sheet will be made,
+ and the info of the tweak sheet will be applied to that copy,
+ adding to or overriding the parent sheet.
- for dr in dirs:
- if level >= 3:
- log(f"{sheetRep} is at max depth, yet contains subdir {dr}")
- continue
+ A sheet is basically a mapping of triggers to names.
- if not dr.isdecimal():
- log(f"{sheetRep} contains non-numeric subdir {dr}")
- continue
+ We also maintain a mapping from tweak sheets to triggers, so that we can
+ know later on which sheet assigned which trigger to which name.
- sheetSubdirs[int(dr)] = readDir(f"{sheetRela}{sep}{dr}", level + 1)
+ The tweak may remove triggers from the sheet. We have to adapt the tMap
+ for that.
+ """
+ sheetData = self.getSheetData()
+ compiled = AttrDict()
+ sheetData.compiled = compiled
+ nameMap = sheetData.nameMap
- return AttrDict(sng=sheetSingle, rng=sheetRange, sdr=sheetSubdirs)
+ def spec(msg):
+ self.log(None, 0, msg)
- sheetMain = readXls("")
- sheetSubdirs = readDir("", 1)
- sheetData.raw = AttrDict(main=sheetMain, sdr=sheetSubdirs)
+ def log(msg):
+ self.log(False, 0, msg)
- def _combineSheets(self, sheetData):
- """Combines the spreadsheet info in single-section spreadsheets.
+ def err(msg):
+ self.log(True, 0, msg)
- Among the tweaks, there may be *ranged* spreadsheets, i.e. having the name
- *start*`-`*end*, which indicate that they contain tweaks for sections
- *start* to *end*. These will be converted to individual spreadsheet
- *start*, *start + 1*, ..., *end - 1*, *end*.
- """
- raw = sheetData.raw
+ def errProblem(problem):
+ for indent, msg in problem:
+ self.log(True, indent, msg)
- sheetMain = raw.main
- sheetTweaked = raw.sdr
+ tFullMap = {}
- # combine the info in ranged sheets into single number sheets
+ def getTriggerRep(tr):
+ triggerInfo = tFullMap[tr]
- combined = AttrDict(sheet=sheetMain, tweaks=AttrDict())
- sheetData.combined = combined
+ result = []
- def combineDir(info, dest):
- ranged = info.rng or {}
- single = info.sng or {}
- subdirs = info.sdr or {}
+ for scope in sorted(triggerInfo):
+ scopeRep = "" if not scope else f"({scope})"
+ scopeInfo = triggerInfo[scope]
- for (start, end), sheet in sorted(ranged.items()):
- for i in range(start, end + 1):
- updateDest = dest.setdefault(i, AttrDict()).setdefault("sheet", {})
- for eidkind, triggers in sheet.items():
- updateDest[eidkind] = triggers
+ for eidkind in sorted(scopeInfo):
+ name = sheetData.nameMap[eidkind]
+ rowRep = ",".join(str(r) for r in sorted(scopeInfo[eidkind]))
+ result.append(f"'{tr}'{scopeRep} r{rowRep} for {name}")
+ return result
- for i, sheet in single.items():
- updateDest = dest.setdefault(i, AttrDict()).setdefault("sheet", {})
- for eidkind, triggers in sheet.items():
- updateDest[eidkind] = triggers
+ spec("Checking scopes ...")
- for i, tweaks in subdirs.items():
- updateDest = dest.setdefault(i, AttrDict()).setdefault("tweaks", {})
- combineDir(tweaks, updateDest)
+ raw = sheetData.raw
+ scopeMap = sheetData.scopeMap
+ intervals = partitionScopes(scopeMap)
- combineDir(sheetTweaked, combined.tweaks)
+ problems = set()
- def _compileSheets(self, sheetData):
- """Compiles the info in tweaked sheets into complete sheets.
+ for b, e, scopeStrs in [[None, None, None]] + intervals:
+ scopeStrSet = {""} if scopeStrs is None else set(scopeStrs)
+ intv = (b, e) if b is not None and e is not None else ()
- For every tweak spreadsheet, a copy of its parent sheet will be made,
- and the info of the tweak sheet will be applied to that copy,
- adding to or overriding the parent sheet.
+ spec(repScope(intv))
- A sheet is basically a mapping of triggers to names.
+ thisCompiled = AttrDict()
+ compiled[intv] = thisCompiled
- We also maintain a mapping from tweak sheets to triggers, so that we can
- know later on which sheet assigned which trigger to which name.
+ # check which triggers are a substring in which other triggers
+ # these triggers can never be found!
- The tweak may remove triggers from the sheet. We have to adapt the tMap
- for that.
- """
- nameMap = sheetData.nameMap
- combined = sheetData.combined
+ tFullMap.clear()
- compiled = AttrDict()
- sheetData.compiled = compiled
+ for eidkind, info in raw.items():
+ if "" in info:
+ (r, triggers) = info[""]
- def spec(msg):
- self.log(None, 0, msg)
+ for trigger in triggers:
+ tFullMap.setdefault(trigger, {}).setdefault("", {}).setdefault(
+ eidkind, set()
+ ).add(r)
- def err(msg):
- self.log(True, 0, msg)
+ for scopeStr, (r, triggers) in info.items():
+ if scopeStr == "":
+ continue
- spec("Compiling data")
+ if scopeStr in scopeStrSet:
+ for trigger in triggers:
+ tFullMap.setdefault(trigger, {}).setdefault(
+ scopeStr, {}
+ ).setdefault(eidkind, set()).add(r)
+
+ # now compile the result information: tMap, newSheet
- def compileSheet(path, parentData, info, dest):
- parentSheet = parentData.sheet
- sheet = info.sheet
newSheet = {}
- dest.sheet = newSheet
- parentTMap = parentData["tMap"] or {}
- newTMap = {}
- dest["tMap"] = newTMap
- pathRep = f"[{'.'.join(str(i) for i in path)}]"
+ thisCompiled.sheet = newSheet
- for eidkind, triggers in parentSheet.items():
- newSheet[eidkind] = triggers
+ tMap = {}
+ thisCompiled.tMap = tMap
- for eidkind, triggers in sheet.items():
- if len(triggers) == 0:
- if eidkind in newSheet:
- del newSheet[eidkind]
- else:
- err(f"{pathRep} {nameMap[eidkind][0]} has no triggers")
- else:
+ for eidkind, info in raw.items():
+ if "" in info:
+ (r, triggers) = info[""]
newSheet[eidkind] = triggers
for trigger in triggers:
- newTMap[trigger] = tuple(str(k) for k in path)
+ tMap[trigger] = ""
- for eidkind, triggers in newSheet.items():
- for trigger in triggers:
- if trigger not in newTMap:
- newTMap[trigger] = parentTMap[trigger]
-
- def compileDir(path, parentData, info, dest):
- if "sheet" in info:
- compileSheet(path, parentData, info, dest)
- parentData = AttrDict(sheet=dest.sheet, tMap=dest["tMap"])
-
- tweaks = info.tweaks or {}
- tweakDest = dest.setdefault("tweaks", {})
-
- for k in sorted(tweaks):
- compileDir(
- path + (k,),
- parentData,
- tweaks[k],
- tweakDest.setdefault(k, AttrDict()),
- )
+ for scopeStr, (r, triggers) in info.items():
+ if scopeStr == "":
+ continue
+
+ if scopeStr in scopeStrSet:
+ newSheet[eidkind] = triggers
+
+ for trigger in triggers:
+ tMap[trigger] = scopeStr
+
+ # the sheet is now compiled, all compiled data has been created
+ # the rest only deals with error reporting
+
+ ambi = 0
+ clashes = 0
+
+ for trigger in sorted(tFullMap):
+ problem = []
+ triggerInfo = tFullMap[trigger]
- compileDir((), combined, combined, compiled)
+ for scopeStr in sorted(triggerInfo):
+ scopeInfo = triggerInfo[scopeStr]
+ scopeRep = f" in scope {scopeStr}" if scopeStr else ""
- def _prepareSheets(self, sheetData):
+ thisAmbi = False
+ theseClashes = 0
+
+ if len(scopeInfo) > 1:
+ problem.append((0, f"Ambi: '{trigger}'{scopeRep}: "))
+ thisAmbi = True
+
+ for eidkind, rs in scopeInfo.items():
+ name = nameMap[eidkind]
+ rowRep = ",".join(str(r) for r in rs)
+ nRs = len(rs)
+
+ if nRs > 1:
+ theseClashes += 1
+
+ if thisAmbi:
+ problem.append((1, f"{name}: {rowRep}"))
+ elif nRs > 1:
+ problem.append(
+ (0, f"Clash: '{trigger}' for {name}: {rowRep}")
+ )
+
+ clashes += theseClashes
+
+ if len(problem):
+ problem = tuple(problem)
+
+ if problem not in problems:
+ if thisAmbi:
+ ambi += 1
+ if theseClashes:
+ clashes += theseClashes
+
+ problems.add(problem)
+ errProblem(problem)
+
+ if ambi > 0:
+ err(f"Ambiguous triggers: {ambi} x")
+
+ if clashes > 0:
+ err(f"Reused triggers scope: {clashes} x")
+
+ def _prepareSheet(self):
"""Transform the sheets into instructions.
Now we have complete sheets for every context, the inheritance is resolved.
@@ -1627,55 +1604,30 @@ Classes
"""
spaceEscaped = self.spaceEscaped
- nameMap = sheetData.nameMap
+
+ sheetData = self.getSheetData()
compiled = sheetData.compiled
+ caseSensitive = sheetData.caseSensitive
instructions = {}
sheetData.instructions = instructions
- def spec(msg):
- self.log(None, 0, msg)
-
- def log(msg):
- self.log(False, 0, msg)
-
- def log1(msg):
- self.log(False, 1, msg)
-
- def log2(msg):
- self.log(False, 2, msg)
-
- def log3(msg):
- self.log(False, 3, msg)
-
- def err(msg):
- self.log(True, 0, msg)
-
- def err1(msg):
- self.log(True, 1, msg)
-
- def err2(msg):
- self.log(True, 2, msg)
-
- spec("Checking triggers ...")
-
- def prepareSheet(path, info):
- sheet = info.sheet
- tMap = info["tMap"]
- sheetR = ".".join(path)
- sheetRep = f"[{sheetR}]"
+ for intv, thisCompiled in compiled.items():
+ sheet = thisCompiled.sheet
+ tMap = thisCompiled.tMap
triggerSet = set()
tPos = {}
idMap = {}
- prepared = dict(tPos=tPos, tMap=tMap)
-
- instructions[path] = prepared
+ theseInstructions = dict(tPos=tPos, tMap=tMap)
+ instructions[intv] = theseInstructions
for eidkind, triggers in sheet.items():
for trigger in triggers:
- triggerT = toTokens(trigger, spaceEscaped=spaceEscaped)
+ triggerT = toTokens(
+ trigger, spaceEscaped=spaceEscaped, caseSensitive=caseSensitive
+ )
triggerSet.add(triggerT)
idMap.setdefault(trigger, []).append(eidkind)
@@ -1683,56 +1635,10 @@ Classes
for i, token in enumerate(triggerT):
tPos.setdefault(i, {}).setdefault(token, set()).add(triggerT)
- prepared["idMap"] = {
+ theseInstructions["idMap"] = {
trigger: eidkinds[0] for (trigger, eidkinds) in idMap.items()
}
- nEnt = len(sheet)
- nTriggers = sum(len(triggers) for triggers in sheet.values())
- noTriggers = sum(1 for triggers in sheet.values() if len(triggers) == 0)
-
- ambis = collections.defaultdict(list)
-
- for trigger, eidkinds in sorted(idMap.items()):
- if len(eidkinds) <= 1:
- continue
-
- tPath = tMap[trigger]
-
- if path != tPath:
- continue
-
- for eidkind in eidkinds:
- name = nameMap[eidkind][0]
- ambis[trigger].append(name)
-
- log(f"{sheetRep} {nTriggers} triggers for {nEnt} entities")
-
- if noTriggers > 0:
- err1(f"{noTriggers} without triggers")
-
- nAmbi = len(ambis)
-
- if nAmbi > 0:
- err1(f"{nAmbi} ambiguous triggers")
-
- for trigger, names in ambis.items():
- err2(trigger)
-
- for name in names:
- log3(name)
-
- def prepareDir(path, info):
- if "sheet" in info:
- prepareSheet(path, info)
-
- tweaks = info.tweaks or {}
-
- for k in sorted(tweaks):
- prepareDir(path + (str(k),), tweaks[k])
-
- prepareDir((), compiled)
-
def _processSheet(self):
"""Generates derived data structures out of the source sheet.
@@ -1768,8 +1674,7 @@ Classes
if not self.properlySetup:
return
- sheetName = self.sheetName
- sheetData = self.sheets[sheetName]
+ sheetData = self.getSheetData()
inventory = sheetData.inventory
instructions = sheetData.instructions
@@ -1784,13 +1689,13 @@ Classes
idMap = data["idMap"]
tMap = data["tMap"]
- for trigger, tPath in tMap.items():
- eidkind = idMap[trigger]
- sheet = ".".join(tPath)
-
- occs = inventory.get(eidkind, {}).get(trigger, {}).get(tPath, {})
+ for trigger, scope in tMap.items():
+ eidkind = idMap.get(trigger, None)
+ if eidkind is None:
+ continue
- hitData.setdefault(eidkind, {})[(trigger, sheet)] = len(occs)
+ occs = inventory.get(eidkind, {}).get(trigger, {}).get(scope, [])
+ hitData.setdefault(eidkind, {})[(trigger, scope)] = len(occs)
def _markEntities(self):
"""Marks up the members of the inventory as entities.
@@ -1804,8 +1709,7 @@ Classes
if not self.properlySetup:
return
- sheetName = self.sheetName
- sheetData = self.sheets[sheetName]
+ sheetData = self.getSheetData()
inventory = sheetData.inventory
newEntities = []
@@ -1814,145 +1718,35 @@ Classes
for eidkind, entData in inventory.items():
for trigger, triggerData in entData.items():
- for sheet, matches in triggerData.items():
- newEntities.append((eidkind, matches))
+ for scope, matches in triggerData.items():
for match in matches:
- triggerFromMatch[match] = (trigger, ",".join(sheet))
-
- self._addToSet(newEntities)
-
- def showSheetsRaw(self, main=False):
- src = self.getSheetData()
-
- if src is None:
- self.console("Nothing to show")
-
- src = src.raw
- nameMap = src.nameMap
+ triggerFromMatch[match] = (trigger, scope)
- def showSheet(sheet, tab):
- for eidkind, triggers in sorted(sheet.items()):
- (name, sheetRela) = nameMap[eidkind]
- triggerRep = "|".join(
- t for t in sorted(triggers, key=lambda x: (-len(x), x))
- )
- self.console(f"{tab} '{name}' {eidkind} : {triggerRep}")
- self.console(f"{tab} ---")
-
- def showDir(head, tweaks, level):
- tab = " " * level
- self.console(f"{tab}{head}")
-
- rng = tweaks.rng or {}
- for b, e in sorted(rng):
- self.console(f"{tab} {b}-{e}.xslx")
- showSheet(rng[(b, e)], tab)
-
- sng = tweaks.sng or {}
- for k in sorted(sng):
- self.console(f"{tab} {k}.xslx")
- showSheet(sng[k], tab)
-
- sdr = tweaks.sdr or {}
-
- for k in sorted(sdr):
- showDir(k, sdr[k], level + 1)
-
- if main:
- showSheet(src.main, "")
-
- showDir("", src.sdr, 0)
-
- def showSheetsCombined(self, main=False):
- src = self.getSheetData()
-
- if src is None:
- self.console("Nothing to show")
-
- src = src.combined
- nameMap = src.nameMap
-
- def showSheet(sheet, tab):
- for eidkind, triggers in sorted(sheet.items()):
- (name, sheetRela) = nameMap[eidkind]
- triggerRep = "|".join(
- t for t in sorted(triggers, key=lambda x: (-len(x), x))
- )
- self.console(f"{tab} '{name}' {eidkind} : {triggerRep}")
- self.console(f"{tab} ---")
-
- def showDir(head, info, level):
- tab = " " * level
- self.console(f"{tab}{head}")
-
- if "sheet" in info:
- if main or level > 0:
- showSheet(info.sheet, tab)
-
- tweaks = info.tweaks or {}
-
- for k in sorted(tweaks):
- showDir(k, tweaks[k], level + 1)
-
- showDir("", src, 0)
-
- def showSheetsCompiled(self, main=False):
- nameMap = self.nameMap
- src = self.compiled
-
- def showSheet(info, tab):
- sheet = info.sheet
- tMap = info["tMap"]
-
- for eidkind, triggers in sorted(sheet.items()):
- triggerSources = {tMap[t] for t in triggers}
- nTriggers = len(triggers)
-
- if not main and (nTriggers == 0 or triggerSources == {()}):
- continue
-
- (name, sheetRela) = nameMap[eidkind]
-
- if nTriggers == 0:
- triggerInfo = "X"
- else:
- sourceRep = (
- "X"
- if len(triggerSources) == 0
- else list(list(triggerSources)[0])
- )
- triggerRep = "|".join(
- t for t in sorted(triggers, key=lambda x: (-len(x), x))
- )
- triggerInfo = f"{sourceRep} => {triggerRep}"
-
- self.console(f"{tab} '{name}' {eidkind} : {triggerInfo}")
- self.console(f"{tab} ---")
-
- def showDir(head, info, level):
- tab = " " * level
- self.console(f"{tab}{head}")
+ newEntities.append((eidkind, matches))
- if "sheet" in info:
- if main or level > 0:
- showSheet(info, tab)
+ self._addToSet(newEntities)
- tweaks = info.tweaks or {}
+ def getSheetData(self):
+ """Deliver the current sheet."""
+ sheetName = self.sheetName
+ sheetsData = self.sheets
- for k in sorted(tweaks):
- showDir(k, tweaks[k], level + 1)
+ return sheetsData.setdefault(sheetName, AttrDict())
- showDir("", src, 0)
+ def log(self, isError, indent, msg):
+ silent = self.silent
- def showInstructions(self):
- src = self.getSheetData()
+ if silent and not isError:
+ return
- if src is None:
- self.console("Nothing to show")
+ browse = self.browse
+ sheetData = self.getSheetData()
+ logData = sheetData.logData
- instructions = src.instructions
+ logData.append((isError, indent, msg))
- self.console("\n".join(f"{path}" for path in instructions))
+ if not browse:
+ self.consoleLine(isError, indent, msg)
Subclasses
@@ -1978,7 +1772,7 @@ Methods
Deliver the current sheet.
-def loadSheetData(self)
+def loadSheetData(self, force=False, caseSensitive=False)
-
Loads the current ner sheet into memory, if there is one.
@@ -1998,8 +1792,8 @@ Methods
-
-
-def readData(self, sheetData, sheetUpdated)
+
+def makeSheetOfSingleTokens(self)
-
@@ -2013,7 +1807,7 @@
Methods
by editing the spreadsheets in Excel.
-def setSheet(self, newSheet)
+def setSheet(self, newSheet, force=False, caseSensitive=False)
-
Switch to a named ner sheet.
@@ -2024,36 +1818,6 @@ Parameters
- The name of the new ner sheet to switch to.
-
-def showInstructions(self)
--
-
-
-
-def showSheetsCombined(self, main=False)
--
-
-
-
-def showSheetsCompiled(self, main=False)
--
-
-
-
-def showSheetsRaw(self, main=False)
--
-
-
-
-def writeData(self, sheetData, tm)
--
-
-
@@ -2125,20 +1889,15 @@ Index
-
Sheets
-
diff --git a/tf/browser/ner/show.html b/tf/browser/ner/show.html
index c85731d6a..902d37cc1 100644
--- a/tf/browser/ner/show.html
+++ b/tf/browser/ner/show.html
@@ -34,7 +34,7 @@ Module tf.browser.ner.show
Expand source code
-Browse git
+Browse git
"""Rendering of corpus extracts with annotations.
@@ -288,6 +288,7 @@ Module tf.browser.ner.show
sheetData = self.getSheetData()
hitData = sheetData.hitData or {}
nameMap = sheetData.nameMap or {}
+ rowMap = sheetData.rowMap or {}
hasEnt = activeEntity is not None
hasTrig = activeTrigger is not None
@@ -296,7 +297,11 @@ Module tf.browser.ner.show
entries = []
for eidkind, triggers in data.items():
- name = nameMap[eidkind][0]
+ if eidkind not in nameMap:
+ # should not occur!
+ continue
+
+ name = nameMap[eidkind]
tOccs = sum(triggers.values())
subtle = any(t[1] != "" for t in triggers)
@@ -308,7 +313,13 @@ Module tf.browser.ner.show
and not subtle
):
entries.append(
- (eidkind, name, tOccs, subtle, entryTriggers(triggers))
+ (
+ eidkind,
+ name,
+ tOccs,
+ subtle,
+ entryTriggers(triggers),
+ )
)
return entries
@@ -352,14 +363,14 @@ Module tf.browser.ner.show
continue
occsCls = (
- "nooccs" if hasNoOccs else "warnoccs" if hasTriggerNoOccs else ""
+ "nooccs" if hasNoOccs else ("warnoccs" if hasTriggerNoOccs else "")
)
subtleCls = "subtle" if subtle else ""
active = "queried" if hasEnt and eidkind == activeEntity else ""
content.append(
H.div(
- H.div(H.span(f"{tOccs} ", cls="stat")),
+ H.div((H.span(f"{tOccs} ", cls="stat"))),
H.div(
(
H.div(name, cls=f"ntx {occsCls} {subtleCls}"),
@@ -376,19 +387,21 @@ Module tf.browser.ner.show
def genTriggers(data):
content = []
- for k, nOccs, subtle in data:
- (trigger, sheet) = k
+ for tInfo, nOccs, subtle in data:
+ (trigger, scope) = tInfo
hasNoOccs = nOccs == 0
if zeroFilter == -1 and not hasNoOccs:
continue
+ rows = rowMap.get(trigger, [])
+ r = "?" if len(rows) == 0 else ", ".join(str(x) for x in rows)
occsCls = "nooccs" if nOccs == 0 else ""
subtleCls = "subtle" if subtle else ""
- sheetRep = f"{sheet}{SET_SHEET}" if subtle else sheet
- triggerRep = "⊙".join(k)
+ scopeRep = f"{SET_SHEET}{scope}"
+ triggerRep = "⊙".join(str(x) for x in tInfo)
active = (
- "tqueried" if hasTrig and (trigger, sheet) == activeTrigger else ""
+ "tqueried" if hasTrig and (trigger, scope) == activeTrigger else ""
)
content.append(
H.div(
@@ -396,8 +409,8 @@ Module tf.browser.ner.show
H.span(f"{nOccs} ", cls="stat"),
H.span(
(
- H.span(f"{sheetRep} "),
- H.code(trigger, cls="ttx"),
+ H.span(f"{scopeRep} "),
+ H.code(trigger, cls="ttx", title=f"r{r}"),
),
cls=f"{occsCls} {subtleCls}",
),
@@ -673,7 +686,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Show:
def showEntityOverview(self):
@@ -903,6 +916,7 @@ Classes
sheetData = self.getSheetData()
hitData = sheetData.hitData or {}
nameMap = sheetData.nameMap or {}
+ rowMap = sheetData.rowMap or {}
hasEnt = activeEntity is not None
hasTrig = activeTrigger is not None
@@ -911,7 +925,11 @@ Classes
entries = []
for eidkind, triggers in data.items():
- name = nameMap[eidkind][0]
+ if eidkind not in nameMap:
+ # should not occur!
+ continue
+
+ name = nameMap[eidkind]
tOccs = sum(triggers.values())
subtle = any(t[1] != "" for t in triggers)
@@ -923,7 +941,13 @@ Classes
and not subtle
):
entries.append(
- (eidkind, name, tOccs, subtle, entryTriggers(triggers))
+ (
+ eidkind,
+ name,
+ tOccs,
+ subtle,
+ entryTriggers(triggers),
+ )
)
return entries
@@ -967,14 +991,14 @@ Classes
continue
occsCls = (
- "nooccs" if hasNoOccs else "warnoccs" if hasTriggerNoOccs else ""
+ "nooccs" if hasNoOccs else ("warnoccs" if hasTriggerNoOccs else "")
)
subtleCls = "subtle" if subtle else ""
active = "queried" if hasEnt and eidkind == activeEntity else ""
content.append(
H.div(
- H.div(H.span(f"{tOccs} ", cls="stat")),
+ H.div((H.span(f"{tOccs} ", cls="stat"))),
H.div(
(
H.div(name, cls=f"ntx {occsCls} {subtleCls}"),
@@ -991,19 +1015,21 @@ Classes
def genTriggers(data):
content = []
- for k, nOccs, subtle in data:
- (trigger, sheet) = k
+ for tInfo, nOccs, subtle in data:
+ (trigger, scope) = tInfo
hasNoOccs = nOccs == 0
if zeroFilter == -1 and not hasNoOccs:
continue
+ rows = rowMap.get(trigger, [])
+ r = "?" if len(rows) == 0 else ", ".join(str(x) for x in rows)
occsCls = "nooccs" if nOccs == 0 else ""
subtleCls = "subtle" if subtle else ""
- sheetRep = f"{sheet}{SET_SHEET}" if subtle else sheet
- triggerRep = "⊙".join(k)
+ scopeRep = f"{SET_SHEET}{scope}"
+ triggerRep = "⊙".join(str(x) for x in tInfo)
active = (
- "tqueried" if hasTrig and (trigger, sheet) == activeTrigger else ""
+ "tqueried" if hasTrig and (trigger, scope) == activeTrigger else ""
)
content.append(
H.div(
@@ -1011,8 +1037,8 @@ Classes
H.span(f"{nOccs} ", cls="stat"),
H.span(
(
- H.span(f"{sheetRep} "),
- H.code(trigger, cls="ttx"),
+ H.span(f"{scopeRep} "),
+ H.code(trigger, cls="ttx", title=f"r{r}"),
),
cls=f"{occsCls} {subtleCls}",
),
diff --git a/tf/browser/ner/web.html b/tf/browser/ner/web.html
index ec017770a..0d6207335 100644
--- a/tf/browser/ner/web.html
+++ b/tf/browser/ner/web.html
@@ -34,7 +34,7 @@ Module tf.browser.ner.web
Expand source code
-Browse git
+Browse git
"""Flask sub web app in the TF browser.
diff --git a/tf/browser/serve.html b/tf/browser/serve.html
index 0a9669ab3..4efee1f00 100644
--- a/tf/browser/serve.html
+++ b/tf/browser/serve.html
@@ -37,7 +37,7 @@ Dress TF nodes up for serving
Expand source code
-Browse git
+Browse git
"""
# Dress TF nodes up for serving on the web
diff --git a/tf/browser/servelib.html b/tf/browser/servelib.html
index 0ccf0fbe4..87d732922 100644
--- a/tf/browser/servelib.html
+++ b/tf/browser/servelib.html
@@ -39,7 +39,7 @@ About
Expand source code
-Browse git
+Browse git
"""
# Common Server Related Functions
diff --git a/tf/browser/start.html b/tf/browser/start.html
index f8a49e730..7fc2c4968 100644
--- a/tf/browser/start.html
+++ b/tf/browser/start.html
@@ -89,7 +89,7 @@ Implementation notes
Expand source code
-Browse git
+Browse git
"""
# Start the TF browser
diff --git a/tf/browser/web.html b/tf/browser/web.html
index 40d7d3a03..9087e7ffc 100644
--- a/tf/browser/web.html
+++ b/tf/browser/web.html
@@ -114,7 +114,7 @@ JavaScript
Expand source code
-Browse git
+Browse git
"""
# Web interface
@@ -431,7 +431,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Web:
def __init__(self, kernelApi):
diff --git a/tf/browser/wrap.html b/tf/browser/wrap.html
index 9e3dfeee9..438dada30 100644
--- a/tf/browser/wrap.html
+++ b/tf/browser/wrap.html
@@ -33,7 +33,7 @@ Wrap material into HTML
Expand source code
-Browse git
+Browse git
"""
# Wrap material into HTML
diff --git a/tf/capable.html b/tf/capable.html
index 1fc98f01e..9bcf603e0 100644
--- a/tf/capable.html
+++ b/tf/capable.html
@@ -86,7 +86,7 @@ Optional dependencies
Expand source code
-Browse git
+Browse git
"""Dependency management.
@@ -500,7 +500,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Capable:
"""Import modules for back-end communication if possible.
@@ -684,7 +684,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class CheckImport:
def __init__(self, *modules, optional=False):
diff --git a/tf/cheatsheet.html b/tf/cheatsheet.html
index d5406bbca..a7012dfd5 100644
--- a/tf/cheatsheet.html
+++ b/tf/cheatsheet.html
@@ -1669,7 +1669,7 @@ Command-line tools
Expand source code
-Browse git
+Browse git
"""
.. include:: docs/main/cheatsheet.md
diff --git a/tf/clean.html b/tf/clean.html
index e59e38e88..0f805ad2a 100644
--- a/tf/clean.html
+++ b/tf/clean.html
@@ -40,7 +40,7 @@ Clean
Expand source code
-Browse git
+Browse git
"""
# Clean
diff --git a/tf/client/index.html b/tf/client/index.html
index acdfb6218..c27d472d9 100644
--- a/tf/client/index.html
+++ b/tf/client/index.html
@@ -57,7 +57,7 @@ Acknowledgments
Expand source code
-Browse git
+Browse git
"""
# Layered Search
diff --git a/tf/client/make/build.html b/tf/client/make/build.html
index d113a51ef..d815a251c 100644
--- a/tf/client/make/build.html
+++ b/tf/client/make/build.html
@@ -81,7 +81,7 @@ Commands
Expand source code
-Browse git
+Browse git
"""
# Usage
@@ -1532,7 +1532,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Make:
def __init__(
diff --git a/tf/client/make/gh.html b/tf/client/make/gh.html
index 969ae09e2..27b396c14 100644
--- a/tf/client/make/gh.html
+++ b/tf/client/make/gh.html
@@ -31,7 +31,7 @@ Module tf.client.make.gh
Expand source code
-Browse git
+Browse git
import sys
import os
diff --git a/tf/client/make/help.html b/tf/client/make/help.html
index e1e711876..e1ba5633d 100644
--- a/tf/client/make/help.html
+++ b/tf/client/make/help.html
@@ -31,7 +31,7 @@ Module tf.client.make.help
Expand source code
-Browse git
+Browse git
HELP = """
tf-make [--backend={backend} {org}/{repo} serve {folder}
diff --git a/tf/client/make/index.html b/tf/client/make/index.html
index 14ab8f026..1c6b3fbca 100644
--- a/tf/client/make/index.html
+++ b/tf/client/make/index.html
@@ -32,7 +32,7 @@ Module tf.client.make
Expand source code
-Browse git
+Browse git
"""
The search app make program
diff --git a/tf/convert/addnlp.html b/tf/convert/addnlp.html
index c008f2874..94a28ff7f 100644
--- a/tf/convert/addnlp.html
+++ b/tf/convert/addnlp.html
@@ -114,7 +114,7 @@ Examples
Expand source code
-Browse git
+Browse git
"""Add data from an NLP pipeline.
@@ -1844,7 +1844,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class NLPipeline(CheckImport):
def __init__(
diff --git a/tf/convert/app/app.html b/tf/convert/app/app.html
index a4b81a41e..243682ede 100644
--- a/tf/convert/app/app.html
+++ b/tf/convert/app/app.html
@@ -31,7 +31,7 @@ Module tf.convert.app.app
Expand source code
-Browse git
+Browse git
import types
from tf.advanced.app import App
@@ -138,7 +138,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class TfApp(App):
def __init__(app, *args, **kwargs):
diff --git a/tf/convert/helpers.html b/tf/convert/helpers.html
index 903c4cc46..e354ad84a 100644
--- a/tf/convert/helpers.html
+++ b/tf/convert/helpers.html
@@ -31,7 +31,7 @@ Module tf.convert.helpers
Expand source code
-Browse git
+Browse git
import re
from textwrap import dedent
@@ -390,7 +390,7 @@ Module tf.convert.helpers
helpText = f"""
Convert {kind} to TF.
- There are also commands to check the {kind} and to load the TF."""
+ There are also commands to check the {kind} and to load the resulting TF."""
taskSpec = dict(
check="reports on the elements in the source",
@@ -400,7 +400,6 @@ Module tf.convert.helpers
apptoken="modifies the TF app to make it token- instead of character-based",
browse="starts the TF browser on the result",
)
- taskExcluded = {"apptoken", "browse"}
paramSpec = {
"tf": (
@@ -411,6 +410,13 @@ Module tf.convert.helpers
),
"latest",
),
+ "sourceBase": (
+ (
+ "empty: refDir/{kind};\n\t\t"
+ "any directory of choice."
+ ),
+ "",
+ ),
kind.lower(): (
(
"0 or latest: latest version;\n\t\t"
@@ -425,8 +431,9 @@ Module tf.convert.helpers
flagSpec = dict(
verbose=("Produce less or more progress and reporting messages", -1, 3),
+ doc=("Do only this document", None, 0),
)
- return (helpText, taskSpec, taskExcluded, paramSpec, flagSpec)
+ return (helpText, taskSpec, paramSpec, flagSpec)
def tweakTrans(
@@ -720,10 +727,11 @@ Module tf.convert.helpers
tpd = type(data)
if tpd is str:
- for k, v in constants.items():
- if len(data) > 1 and data[0] == "!" and data[1:] in macros:
- data = macros[data[1:]]
+ for k, v in macros.items():
+ pattern = f"<{k}>"
+ data = data.replace(pattern, str(v))
+ for k, v in constants.items():
pattern = f"«{k}»"
if type(v) is int and data == pattern:
@@ -780,11 +788,18 @@ Module tf.convert.helpers
if len(feat) == 1:
parent = None
+ child = None
feat = feat[0]
else:
parent, feat = feat
- newVars[name] = (parent, feat, shift)
+ if parent.startswith("-"):
+ child = parent[1:]
+ parent = None
+ else:
+ child = None
+
+ newVars[name] = (parent, child, feat, shift)
scanInfo.setdefault(nodeType, []).append((extraFeat, value, newVars))
diff --git a/tf/convert/iiif.html b/tf/convert/iiif.html
index d2e397502..e1ac0abce 100644
--- a/tf/convert/iiif.html
+++ b/tf/convert/iiif.html
@@ -31,7 +31,7 @@ Module tf.convert.iiif
Expand source code
-Browse git
+Browse git
from ..core.files import (
readYaml,
@@ -42,6 +42,7 @@ Module tf.convert.iiif
dirExists,
dirCopy,
dirContents,
+ stripExt,
)
from ..core.helpers import console
from .helpers import parseIIIF, fillinIIIF
@@ -50,16 +51,19 @@ Module tf.convert.iiif
class IIIF:
- def __init__(self, teiVersion, app, prod=False, silent=False):
+ def __init__(self, teiVersion, app, pageInfoFile, prod=False, silent=False):
self.teiVersion = teiVersion
self.app = app
+ self.pageInfoFile = pageInfoFile
self.prod = prod
self.silent = silent
+ teiVersionRep = f"/{teiVersion}" if teiVersion else teiVersion
+
F = app.api.F
repoLocation = app.repoLocation
- staticDir = f"{repoLocation}/{teiVersion}/{'prod' if prod else 'dev'}"
+ staticDir = f"{repoLocation}/static{teiVersionRep}/{'prod' if prod else 'dev'}"
self.staticDir = staticDir
self.manifestDir = f"{staticDir}/manifests"
self.thumbDir = (
@@ -72,23 +76,24 @@ Module tf.convert.iiif
self.pagesDir = f"{scanDir}/pages"
self.logoInDir = f"{scanDir}/logo"
self.logoDir = f"{staticDir}/logo"
- self.reportDir = f"{repoLocation}/report/{teiVersion}"
+ self.coversHtmlIn = f"{repoLocation}/programs/covers.html"
+ self.coversHtmlOut = f"{staticDir}/covers.html"
settings = readYaml(asFile=f"{repoLocation}/programs/iiif.yaml", plain=True)
+ self.settings = settings
self.templates = parseIIIF(settings, prod, "templates")
self.getSizes()
+ self.getRotations()
self.getPageSeq()
pages = self.pages
- covers = sorted(f for f in dirContents(coversDir)[0] if f is not DS_STORE)
- self.covers = covers
folders = [F.folder.v(f) for f in F.otype.s("folder")]
self.folders = folders
self.console("Collections:")
for folder in folders:
- n = len(pages[folder])
+ n = len(pages["pages"][folder])
self.console(f"{folder:>5} with {n:>4} pages")
def console(self, msg, **kwargs):
@@ -103,6 +108,24 @@ Module tf.convert.iiif
if not silent:
console(msg, **kwargs)
+ def getRotations(self):
+ prod = self.prod
+ thumbDir = self.thumbDir
+ scanDir = self.scanDir
+
+ rotateFile = f"{scanDir if prod else thumbDir}/rotation_pages.tsv"
+
+ rotateInfo = {}
+ self.rotateInfo = rotateInfo
+
+ with fileOpen(rotateFile) as rh:
+ next(rh)
+ for line in rh:
+ fields = line.rstrip("\n").split("\t")
+ p = fields[0]
+ rot = int(fields[1])
+ rotateInfo[p] = rot
+
def getSizes(self):
prod = self.prod
thumbDir = self.thumbDir
@@ -153,13 +176,15 @@ Module tf.convert.iiif
self.console(f"Average deviation: W = {devW:>4} H = {devH:>4}")
def getPageSeq(self):
- reportDir = self.reportDir
- covers = self.covers
- pageSeqFile = f"{reportDir}/pageseq.json"
+ coversDir = self.coversDir
+ pageInfoFile = self.pageInfoFile
- self.pages = {}
+ covers = sorted(
+ stripExt(f) for f in dirContents(coversDir)[0] if f is not DS_STORE
+ )
+ self.covers = covers
self.pages = dict(
- pages=readJson(asFile=pageSeqFile, plain=True), covers=dict(covers=covers)
+ pages=readJson(asFile=pageInfoFile, plain=True), covers=dict(covers=covers)
)
def genPages(self, kind, folder=None):
@@ -167,6 +192,7 @@ Module tf.convert.iiif
folder = kind
templates = self.templates
sizeInfo = self.sizeInfo[kind]
+ rotateInfo = None if kind == "covers" else self.rotateInfo
pages = self.pages[kind]
thesePages = pages[folder]
@@ -177,9 +203,10 @@ Module tf.convert.iiif
for p in thesePages:
item = {}
w, h = sizeInfo.get(p, (0, 0))
+ rot = 0 if rotateInfo is None else rotateInfo.get(p, 0)
for k, v in pageItem.items():
- v = fillinIIIF(v, folder=folder, page=p, width=w, height=h)
+ v = fillinIIIF(v, folder=folder, page=p, width=w, height=h, rot=rot)
item[k] = v
items.append(item)
@@ -204,6 +231,19 @@ Module tf.convert.iiif
manifestDir = self.manifestDir
logoInDir = self.logoInDir
logoDir = self.logoDir
+ coversHtmlIn = self.coversHtmlIn
+ coversHtmlOut = self.coversHtmlOut
+ prod = self.prod
+ settings = self.settings
+
+ with fileOpen(coversHtmlIn) as fh:
+ coversHtml = fh.read()
+
+ server = settings["switches"]["prod" if prod else "dev"]["server"]
+ coversHtml = coversHtml.replace("«server»", server)
+
+ with fileOpen(coversHtmlOut, "w") as fh:
+ fh.write(coversHtml)
initTree(manifestDir, fresh=True)
@@ -231,26 +271,29 @@ Classes
class IIIF
-(teiVersion, app, prod=False, silent=False)
+(teiVersion, app, pageInfoFile, prod=False, silent=False)
-
Expand source code
-Browse git
+Browse git
class IIIF:
- def __init__(self, teiVersion, app, prod=False, silent=False):
+ def __init__(self, teiVersion, app, pageInfoFile, prod=False, silent=False):
self.teiVersion = teiVersion
self.app = app
+ self.pageInfoFile = pageInfoFile
self.prod = prod
self.silent = silent
+ teiVersionRep = f"/{teiVersion}" if teiVersion else teiVersion
+
F = app.api.F
repoLocation = app.repoLocation
- staticDir = f"{repoLocation}/{teiVersion}/{'prod' if prod else 'dev'}"
+ staticDir = f"{repoLocation}/static{teiVersionRep}/{'prod' if prod else 'dev'}"
self.staticDir = staticDir
self.manifestDir = f"{staticDir}/manifests"
self.thumbDir = (
@@ -263,23 +306,24 @@ Classes
self.pagesDir = f"{scanDir}/pages"
self.logoInDir = f"{scanDir}/logo"
self.logoDir = f"{staticDir}/logo"
- self.reportDir = f"{repoLocation}/report/{teiVersion}"
+ self.coversHtmlIn = f"{repoLocation}/programs/covers.html"
+ self.coversHtmlOut = f"{staticDir}/covers.html"
settings = readYaml(asFile=f"{repoLocation}/programs/iiif.yaml", plain=True)
+ self.settings = settings
self.templates = parseIIIF(settings, prod, "templates")
self.getSizes()
+ self.getRotations()
self.getPageSeq()
pages = self.pages
- covers = sorted(f for f in dirContents(coversDir)[0] if f is not DS_STORE)
- self.covers = covers
folders = [F.folder.v(f) for f in F.otype.s("folder")]
self.folders = folders
self.console("Collections:")
for folder in folders:
- n = len(pages[folder])
+ n = len(pages["pages"][folder])
self.console(f"{folder:>5} with {n:>4} pages")
def console(self, msg, **kwargs):
@@ -294,6 +338,24 @@ Classes
if not silent:
console(msg, **kwargs)
+ def getRotations(self):
+ prod = self.prod
+ thumbDir = self.thumbDir
+ scanDir = self.scanDir
+
+ rotateFile = f"{scanDir if prod else thumbDir}/rotation_pages.tsv"
+
+ rotateInfo = {}
+ self.rotateInfo = rotateInfo
+
+ with fileOpen(rotateFile) as rh:
+ next(rh)
+ for line in rh:
+ fields = line.rstrip("\n").split("\t")
+ p = fields[0]
+ rot = int(fields[1])
+ rotateInfo[p] = rot
+
def getSizes(self):
prod = self.prod
thumbDir = self.thumbDir
@@ -344,13 +406,15 @@ Classes
self.console(f"Average deviation: W = {devW:>4} H = {devH:>4}")
def getPageSeq(self):
- reportDir = self.reportDir
- covers = self.covers
- pageSeqFile = f"{reportDir}/pageseq.json"
+ coversDir = self.coversDir
+ pageInfoFile = self.pageInfoFile
- self.pages = {}
+ covers = sorted(
+ stripExt(f) for f in dirContents(coversDir)[0] if f is not DS_STORE
+ )
+ self.covers = covers
self.pages = dict(
- pages=readJson(asFile=pageSeqFile, plain=True), covers=dict(covers=covers)
+ pages=readJson(asFile=pageInfoFile, plain=True), covers=dict(covers=covers)
)
def genPages(self, kind, folder=None):
@@ -358,6 +422,7 @@ Classes
folder = kind
templates = self.templates
sizeInfo = self.sizeInfo[kind]
+ rotateInfo = None if kind == "covers" else self.rotateInfo
pages = self.pages[kind]
thesePages = pages[folder]
@@ -368,9 +433,10 @@ Classes
for p in thesePages:
item = {}
w, h = sizeInfo.get(p, (0, 0))
+ rot = 0 if rotateInfo is None else rotateInfo.get(p, 0)
for k, v in pageItem.items():
- v = fillinIIIF(v, folder=folder, page=p, width=w, height=h)
+ v = fillinIIIF(v, folder=folder, page=p, width=w, height=h, rot=rot)
item[k] = v
items.append(item)
@@ -395,6 +461,19 @@ Classes
manifestDir = self.manifestDir
logoInDir = self.logoInDir
logoDir = self.logoDir
+ coversHtmlIn = self.coversHtmlIn
+ coversHtmlOut = self.coversHtmlOut
+ prod = self.prod
+ settings = self.settings
+
+ with fileOpen(coversHtmlIn) as fh:
+ coversHtml = fh.read()
+
+ server = settings["switches"]["prod" if prod else "dev"]["server"]
+ coversHtml = coversHtml.replace("«server»", server)
+
+ with fileOpen(coversHtmlOut, "w") as fh:
+ fh.write(coversHtml)
initTree(manifestDir, fresh=True)
@@ -432,6 +511,12 @@ Methods
-
+
+def getRotations(self)
+-
+
+
def getSizes(self)
Index
-
IIIF
-
+
diff --git a/tf/convert/index.html b/tf/convert/index.html
index 1f467a909..4ba6af1c7 100644
--- a/tf/convert/index.html
+++ b/tf/convert/index.html
@@ -39,7 +39,7 @@ Various forms of data interchange
Expand source code
-Browse git
+Browse git
"""
# Various forms of data interchange
diff --git a/tf/convert/makewatm.html b/tf/convert/makewatm.html
index a632c2764..4baa87b25 100644
--- a/tf/convert/makewatm.html
+++ b/tf/convert/makewatm.html
@@ -31,7 +31,7 @@ Module tf.convert.makewatm
Expand source code
-Browse git
+Browse git
import sys
from textwrap import dedent
@@ -83,10 +83,15 @@ Module tf.convert.makewatm
(flag2, docLine2),
...
),
+ paramSpecs=(
+ (param1, docLine1),
+ (param2, docLine2),
+ ...
+ ),
)
"""
- def __init__(self, fileLoc):
+ def __init__(self, fileLoc, **kwargs):
"""Localize upon creation.
When an object of this class is initialized, we assume that the script
@@ -104,15 +109,21 @@ Module tf.convert.makewatm
USAGE
- python make.py tasks
+ python make.py flags params tasks
Converts data from TEI files
ARGS
+ flags
+ Any argument starting with --
+ params
+ Any argument with an = in it
tfversion
Any arg that contains a . is considered to be the tf version number.
If no version is passed, we resort to the hard coded default.
+ tasks
+ Any remaining argument
all
run all (enabled) tasks
@@ -132,6 +143,9 @@ Module tf.convert.makewatm
prod="Delivers WATM in production location",
force="Force execution even if (uptodate) data is present",
)
+ self.BASE_PARAMS = dict(
+ sourceBase="Base directory under which the sources are",
+ )
self.BASE_TASKS = ("tei2tf", "page2tf", "watm", "watms")
self.TF_VERSION = "0.0.0test"
@@ -144,7 +158,7 @@ Module tf.convert.makewatm
targetDir=repoBase
)
self.repoBase = repoBase
- self.setOptions()
+ self.setOptions(**kwargs)
def setOptions(
self,
@@ -153,17 +167,23 @@ Module tf.convert.makewatm
("watm", None),
),
flagSpecs=(
- ("silent", None),
+ ("silent", True),
("relaxed", None),
("usenlp", None),
- ("prod", None),
+ ("prod", True),
("force", None),
),
+ paramSpecs=(
+ ("sourceBase", None),
+ ),
intro=None,
+ **kwargs,
):
self.TASKS = []
self.DOCS = {}
self.FLAGS = {}
+ self.PARAMS = {}
+ self.PARAM_DEFAULTS = {}
good = True
@@ -199,6 +219,23 @@ Module tf.convert.makewatm
self.FLAGS[name] = doc
+ for name, doc in paramSpecs:
+ if name not in self.BASE_PARAMS:
+ if doc is None:
+ console(f"param {name}=: no help text given", error=True)
+ good = False
+
+ if not good:
+ continue
+
+ self.PARAMS[name] = doc
+
+ for (param, value) in kwargs.items():
+ if param in self.PARAMS:
+ self.PARAM_DEFAULTS[param] = value
+ else:
+ console(f"Unrecognized parameter {param}='{value}'", error=True)
+
self.good = good
self.HELP = (
@@ -206,13 +243,29 @@ Module tf.convert.makewatm
+ self.BASE_HELP
+ "\nFLAGS\n\n"
+ "".join(f"{name}\n\t{doc}" for (name, doc) in self.FLAGS.items())
+ + "\nPARAMS\n\n"
+ + "".join(f"{name}\n\t{doc}" for (name, doc) in self.PARAMS.items())
+ "\nTASKS\n\n"
+ "".join(f"{name}\n\t{doc}" for (name, doc) in self.DOCS.items())
+ "\n\nall\n\trun all (enabled) tasks\n\n"
)
+ def warnVersion(self):
+ version = self.version
+ silent = self.flag_silent
+
+ if version is None:
+ console(
+ f"No version for the TF data given. Using default: {self.TF_VERSION}"
+ )
+ self.version = self.TF_VERSION
+ else:
+ if not silent:
+ console(f"Using TF version: {version}")
+
def main(self, cmdLine=None, cargs=sys.argv[1:]):
FLAGS = self.FLAGS
+ PARAMS = self.PARAMS
TASKS = self.TASKS
if cmdLine is not None:
@@ -224,26 +277,58 @@ Module tf.convert.makewatm
unrecognized = set()
tasks = set()
+ srcVersion = None
version = None
for flag in FLAGS:
setattr(self, f"flag_{flag}", False)
+ for param in PARAMS:
+ setattr(self, f"param_{param}", self.PARAM_DEFAULTS.get(param, ""))
+
for carg in cargs:
- if carg.startswith("--"):
+ if carg.startswith("--no-"):
flag = carg[2:]
+ if flag not in FLAGS:
+ unrecognized.add(carg)
+ else:
+ setattr(self, f"flag_{flag}", False)
+
+ elif carg.startswith("--"):
+ flag = carg[5:]
+
if flag not in FLAGS:
unrecognized.add(carg)
else:
setattr(self, f"flag_{flag}", True)
+
+ elif "=" in carg and not carg.startswith("="):
+ parts = carg.split("=", 1)
+
+ if len(parts) == 1:
+ parts.append("")
+
+ (param, value) = parts
+
+ if param not in PARAMS:
+ unrecognized.add(carg)
+ else:
+ setattr(self, f"param_{param}", True)
+
elif carg == "all":
for task in TASKS:
tasks.add(task)
+
elif carg in TASKS:
tasks.add(carg)
+
+ elif "-" in carg:
+ srcVersion = carg
+
elif "." in carg:
version = carg
+
else:
unrecognized.add(carg)
@@ -251,15 +336,10 @@ Module tf.convert.makewatm
if not silent:
console(f"Enabled tasks: {' '.join(self.TASKS)}")
- if version is None:
- console(
- f"No version for the TF data given. Using default: {self.TF_VERSION}"
- )
- version = self.TF_VERSION
- else:
- if not silent:
- console(f"Using TF version: {version}")
+ self.srcVersion = (
+ 0 if srcVersion is None else None if srcVersion == "-" else srcVersion
+ )
self.version = version
if len(unrecognized):
@@ -297,20 +377,28 @@ Module tf.convert.makewatm
def doTask_tei2tf(self):
good = self.good
+ sourceBase = self.param_sourceBase
silent = self.flag_silent
relaxed = self.flag_relaxed
usenlp = self.flag_usenlp
+ srcVersion = self.srcVersion
if not good:
if not silent:
console("Skipping 'produce TF' because of an error condition")
return
+ self.warnVersion()
tfVersion = self.version
verbose = -1 if silent else 0
loadVerbose = DEEP if silent else TERSE
- Tei = TEI(verbose=verbose, tei=0, tf=f"{tfVersion}pre" if usenlp else tfVersion)
+ Tei = TEI(
+ verbose=verbose,
+ sourceBase=sourceBase,
+ tei=srcVersion,
+ tf=f"{tfVersion}pre" if usenlp else tfVersion,
+ )
self.teiVersion = Tei.teiVersion
console("\tValidating TEI ...")
@@ -384,6 +472,7 @@ Module tf.convert.makewatm
console("Skipping 'produce TF' because of an error condition")
return
+ self.warnVersion()
tfVersion = self.version
repoBase = self.repoBase
sourceDir = f"{repoBase}/organized/source"
@@ -474,7 +563,7 @@ Classes
class MakeWATM
-(fileLoc)
+(fileLoc, **kwargs)
-
Base class for running conversions to WATM.
@@ -502,6 +591,11 @@ Classes
(flag2, docLine2),
...
),
+ paramSpecs=(
+ (param1, docLine1),
+ (param2, docLine2),
+ ...
+ ),
)
Localize upon creation.
@@ -515,7 +609,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class MakeWATM:
"""Base class for running conversions to WATM.
@@ -550,10 +644,15 @@ Parameters
(flag2, docLine2),
...
),
+ paramSpecs=(
+ (param1, docLine1),
+ (param2, docLine2),
+ ...
+ ),
)
"""
- def __init__(self, fileLoc):
+ def __init__(self, fileLoc, **kwargs):
"""Localize upon creation.
When an object of this class is initialized, we assume that the script
@@ -571,15 +670,21 @@ Parameters
USAGE
- python make.py tasks
+ python make.py flags params tasks
Converts data from TEI files
ARGS
+ flags
+ Any argument starting with --
+ params
+ Any argument with an = in it
tfversion
Any arg that contains a . is considered to be the tf version number.
If no version is passed, we resort to the hard coded default.
+ tasks
+ Any remaining argument
all
run all (enabled) tasks
@@ -599,6 +704,9 @@ Parameters
prod="Delivers WATM in production location",
force="Force execution even if (uptodate) data is present",
)
+ self.BASE_PARAMS = dict(
+ sourceBase="Base directory under which the sources are",
+ )
self.BASE_TASKS = ("tei2tf", "page2tf", "watm", "watms")
self.TF_VERSION = "0.0.0test"
@@ -611,7 +719,7 @@ Parameters
targetDir=repoBase
)
self.repoBase = repoBase
- self.setOptions()
+ self.setOptions(**kwargs)
def setOptions(
self,
@@ -620,17 +728,23 @@ Parameters
("watm", None),
),
flagSpecs=(
- ("silent", None),
+ ("silent", True),
("relaxed", None),
("usenlp", None),
- ("prod", None),
+ ("prod", True),
("force", None),
),
+ paramSpecs=(
+ ("sourceBase", None),
+ ),
intro=None,
+ **kwargs,
):
self.TASKS = []
self.DOCS = {}
self.FLAGS = {}
+ self.PARAMS = {}
+ self.PARAM_DEFAULTS = {}
good = True
@@ -666,6 +780,23 @@ Parameters
self.FLAGS[name] = doc
+ for name, doc in paramSpecs:
+ if name not in self.BASE_PARAMS:
+ if doc is None:
+ console(f"param {name}=: no help text given", error=True)
+ good = False
+
+ if not good:
+ continue
+
+ self.PARAMS[name] = doc
+
+ for (param, value) in kwargs.items():
+ if param in self.PARAMS:
+ self.PARAM_DEFAULTS[param] = value
+ else:
+ console(f"Unrecognized parameter {param}='{value}'", error=True)
+
self.good = good
self.HELP = (
@@ -673,13 +804,29 @@ Parameters
+ self.BASE_HELP
+ "\nFLAGS\n\n"
+ "".join(f"{name}\n\t{doc}" for (name, doc) in self.FLAGS.items())
+ + "\nPARAMS\n\n"
+ + "".join(f"{name}\n\t{doc}" for (name, doc) in self.PARAMS.items())
+ "\nTASKS\n\n"
+ "".join(f"{name}\n\t{doc}" for (name, doc) in self.DOCS.items())
+ "\n\nall\n\trun all (enabled) tasks\n\n"
)
+ def warnVersion(self):
+ version = self.version
+ silent = self.flag_silent
+
+ if version is None:
+ console(
+ f"No version for the TF data given. Using default: {self.TF_VERSION}"
+ )
+ self.version = self.TF_VERSION
+ else:
+ if not silent:
+ console(f"Using TF version: {version}")
+
def main(self, cmdLine=None, cargs=sys.argv[1:]):
FLAGS = self.FLAGS
+ PARAMS = self.PARAMS
TASKS = self.TASKS
if cmdLine is not None:
@@ -691,26 +838,58 @@ Parameters
unrecognized = set()
tasks = set()
+ srcVersion = None
version = None
for flag in FLAGS:
setattr(self, f"flag_{flag}", False)
+ for param in PARAMS:
+ setattr(self, f"param_{param}", self.PARAM_DEFAULTS.get(param, ""))
+
for carg in cargs:
- if carg.startswith("--"):
+ if carg.startswith("--no-"):
flag = carg[2:]
+ if flag not in FLAGS:
+ unrecognized.add(carg)
+ else:
+ setattr(self, f"flag_{flag}", False)
+
+ elif carg.startswith("--"):
+ flag = carg[5:]
+
if flag not in FLAGS:
unrecognized.add(carg)
else:
setattr(self, f"flag_{flag}", True)
+
+ elif "=" in carg and not carg.startswith("="):
+ parts = carg.split("=", 1)
+
+ if len(parts) == 1:
+ parts.append("")
+
+ (param, value) = parts
+
+ if param not in PARAMS:
+ unrecognized.add(carg)
+ else:
+ setattr(self, f"param_{param}", True)
+
elif carg == "all":
for task in TASKS:
tasks.add(task)
+
elif carg in TASKS:
tasks.add(carg)
+
+ elif "-" in carg:
+ srcVersion = carg
+
elif "." in carg:
version = carg
+
else:
unrecognized.add(carg)
@@ -718,15 +897,10 @@ Parameters
if not silent:
console(f"Enabled tasks: {' '.join(self.TASKS)}")
- if version is None:
- console(
- f"No version for the TF data given. Using default: {self.TF_VERSION}"
- )
- version = self.TF_VERSION
- else:
- if not silent:
- console(f"Using TF version: {version}")
+ self.srcVersion = (
+ 0 if srcVersion is None else None if srcVersion == "-" else srcVersion
+ )
self.version = version
if len(unrecognized):
@@ -764,20 +938,28 @@ Parameters
def doTask_tei2tf(self):
good = self.good
+ sourceBase = self.param_sourceBase
silent = self.flag_silent
relaxed = self.flag_relaxed
usenlp = self.flag_usenlp
+ srcVersion = self.srcVersion
if not good:
if not silent:
console("Skipping 'produce TF' because of an error condition")
return
+ self.warnVersion()
tfVersion = self.version
verbose = -1 if silent else 0
loadVerbose = DEEP if silent else TERSE
- Tei = TEI(verbose=verbose, tei=0, tf=f"{tfVersion}pre" if usenlp else tfVersion)
+ Tei = TEI(
+ verbose=verbose,
+ sourceBase=sourceBase,
+ tei=srcVersion,
+ tf=f"{tfVersion}pre" if usenlp else tfVersion,
+ )
self.teiVersion = Tei.teiVersion
console("\tValidating TEI ...")
@@ -851,6 +1033,7 @@ Parameters
console("Skipping 'produce TF' because of an error condition")
return
+ self.warnVersion()
tfVersion = self.version
repoBase = self.repoBase
sourceDir = f"{repoBase}/organized/source"
@@ -974,7 +1157,13 @@ Methods
-def setOptions(self, taskSpecs=(('tei2tf', None), ('watm', None)), flagSpecs=(('silent', None), ('relaxed', None), ('usenlp', None), ('prod', None), ('force', None)), intro=None)
+def setOptions(self, taskSpecs=(('tei2tf', None), ('watm', None)), flagSpecs=(('silent', True), ('relaxed', None), ('usenlp', None), ('prod', True), ('force', None)), paramSpecs=(('sourceBase', None),), intro=None, **kwargs)
+-
+
+
+
+def warnVersion(self)
-
@@ -1059,6 +1248,7 @@
prepareRun
runsetOptionswarnVersionNaming of features in MQL
Expand source code
-Browse git
+Browse git
"""
# MQL
@@ -1385,7 +1385,7 @@ Classes
Expand source code
-Browse git
+Browse git
class MQL:
def __init__(self, app, mqlDb, exportDir, silent=SILENT_D):
diff --git a/tf/convert/pagexml.html b/tf/convert/pagexml.html
index 4a731ba41..d296cc66a 100644
--- a/tf/convert/pagexml.html
+++ b/tf/convert/pagexml.html
@@ -31,7 +31,7 @@ Module tf.convert.pagexml
Expand source code
-Browse git
+Browse git
import sys
from subprocess import run
@@ -61,55 +61,15 @@ Module tf.convert.pagexml
from ..fabric import Fabric
from ..convert.walker import CV
-from .helpers import FILE, PAGE, DOC, REGION, LINE, NODE, tokenize
+from .helpers import FILE, PAGE, DOC, REGION, LINE, NODE, tokenize, setUp
TOKEN = "token"
SOFT_HYPHEN_CHARS = {"¬", "\u00ad"}
-def setUp():
- helpText = """
- Convert PageXML to TF.
-
- There are also commands to check the to load and browse the resulting TF."""
-
- taskSpec = dict(
- convert="converts PageXML to TF",
- load="loads the generated TF",
- app="configures the TF app for the result",
- browse="starts the TF browser on the result",
- )
- taskExcluded = {"browse"}
-
- paramSpec = {
- "tf": (
- (
- "0 or latest: update latest version;\n\t\t"
- "1 2 3: increase major, intermediate, minor TF version;\n\t\t"
- "rest: explicit version."
- ),
- "latest",
- ),
- "source": (
- (
- "0 or latest: latest version;\n\t\t"
- "-1 -2 etc: previous version, before previous, ...;\n\t\t"
- "1 2 etc: first version, second version, ...;\n\t\t"
- "rest: explicit version."
- ),
- "latest",
- ),
- }
-
- flagSpec = dict(
- verbose=("Produce less or more progress and reporting messages", -1, 3),
- doc=("Do only this document", None, 0),
- )
- return (helpText, taskSpec, taskExcluded, paramSpec, flagSpec)
-
-
-(HELP, TASKS, TASKS_EXCLUDED, PARAMS, FLAGS) = setUp()
+TASKS_EXCLUDED = {"apptoken", "browse"}
+(HELP, TASKS, PARAMS, FLAGS) = setUp("source")
def diverge(cv, s, rtx, rsp, ltx, lsp):
@@ -1129,12 +1089,6 @@ Functions
-
-
-def setUp()
--
-
-
def tokenLogic(cv, s, token, hangover, isFirst, isSecondLast, isLast)
Parameters
Expand source code
-Browse git
+Browse git
class PageXML(CheckImport):
def __init__(
@@ -2337,7 +2291,6 @@ Index
emptySlotlinebreakSlotmainsetUptokenLogicwalkObjectExamples
Expand source code
-Browse git
+Browse git
"""
# Export a TF dataset to a `pandas` data frame.
diff --git a/tf/convert/recorder.html b/tf/convert/recorder.html
index 5c537ab36..27e115ab0 100644
--- a/tf/convert/recorder.html
+++ b/tf/convert/recorder.html
@@ -185,7 +185,7 @@ Usage
Expand source code
-Browse git
+Browse git
"""
# Recorder
@@ -964,7 +964,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class Recorder:
def __init__(self, api=None):
diff --git a/tf/convert/tei.html b/tf/convert/tei.html
index 534d6faae..a704e9be2 100644
--- a/tf/convert/tei.html
+++ b/tf/convert/tei.html
@@ -461,7 +461,7 @@ Examples
Expand source code
-Browse git
+Browse git
"""
# TEI import
@@ -1094,7 +1094,8 @@ Examples
from ..tools.xmlschema import Analysis
-(HELP, TASKS, TASKS_EXCLUDED, PARAMS, FLAGS) = setUp("TEI")
+TASKS_EXCLUDED = {"apptoken", "browse"}
+(HELP, TASKS, PARAMS, FLAGS) = setUp("TEI")
CSS_REND = dict(
h1=(
@@ -1331,6 +1332,7 @@ Examples
class TEI(CheckImport):
def __init__(
self,
+ sourceBase=PARAMS["sourceBase"][1],
tei=PARAMS["tei"][1],
tf=PARAMS["tf"][1],
validate=PARAMS["validate"][1],
@@ -1382,6 +1384,13 @@ Examples
## Input directories
+ ### `sourceBase`
+
+ *Top-level directory of the TEI-XML sources.*
+
+ By default (if left empty) it is the `tei` directory at the top-level of the
+ repo, but you can specify any directory you want.
+
### `tei`
*Location of the TEI-XML sources.*
@@ -1512,10 +1521,13 @@ Examples
repoDir = f"{base}/{org}/{repo}"
refDir = f"{repoDir}{relative}"
programDir = f"{refDir}/programs"
- schemaDir = f"{refDir}/schema"
convertSpec = f"{programDir}/tei.yaml"
convertCustom = f"{programDir}/tei.py"
+ sourceRefDir = sourceBase if sourceBase else refDir
+ teiDir = f"{sourceRefDir}/tei"
+ reportDir = f"{sourceRefDir}/report"
+ schemaDir = f"{sourceRefDir}/schema"
self.schemaDir = schemaDir
settings = readYaml(asFile=convertSpec, plain=True)
@@ -1685,10 +1697,8 @@ Examples
self.siblingEdges = siblingEdges
self.procins = procins
- reportDir = f"{refDir}/report"
appDir = f"{refDir}/app"
docsDir = f"{refDir}/docs"
- teiDir = f"{refDir}/tei"
tfDir = f"{refDir}/tf"
if tei in {"", None}:
@@ -1696,6 +1706,7 @@ Examples
reportPath = f"{reportDir}"
errMsg = f"source directory does not exist: {ux(teiDir)}"
teiVersion = ""
+ teiVersionRep = ""
else:
teiVersions = sorted(dirContents(teiDir)[1], key=versionSort)
nTeiVersions = len(teiVersions)
@@ -1723,9 +1734,11 @@ Examples
else:
teiVersion = tei
- teiPath = f"{teiDir}/{teiVersion}"
- reportPath = f"{reportDir}/{teiVersion}"
errMsg = f"source version {teiVersion} does not exists in {ux(teiDir)}"
+ teiVersionRep = f"/{teiVersion}"
+
+ teiPath = f"{teiDir}{teiVersionRep}"
+ reportPath = f"{reportDir}{teiVersionRep}"
if not dirExists(teiPath):
console(errMsg, error=True)
@@ -1942,7 +1955,7 @@ Examples
generic["sourceFormat"] = "TEI"
generic["version"] = tfVersion
- generic["teiVersion"] = teiVersion
+ generic["teiVersion"] = teiVersion or "not specified"
generic["schema"] = "TEI" + (" + " + (" + ".join(models))) if models else ""
extraInstructions = []
@@ -4959,8 +4972,8 @@ Examples
itemPre = f"{targetDir}/{fileBase}_orig{fileExt}"
justCopy = info["justCopy"]
- teiDir = f"{myDir}/{sourceBit}"
- itemSource = f"{teiDir}/{file}"
+ srcDir = f"{myDir}/{sourceBit}"
+ itemSource = f"{srcDir}/{file}"
# If there is custom info, we do not have to preserve the previous version.
# Otherwise we save the target before overwriting it; # unless it
@@ -5205,7 +5218,7 @@ Classes
class TEI
-(tei='latest', tf='latest', validate=True, verbose=-1)
+(sourceBase='', tei='latest', tf='latest', validate=True, verbose=-1)
-
Converts TEI to TF.
@@ -5241,6 +5254,10 @@ Classes
Relative to this directory the program expects and creates
input / output directories.
Input directories
+sourceBase
+Top-level directory of the TEI-XML sources.
+By default (if left empty) it is the tei directory at the top-level of the
+repo, but you can specify any directory you want.
tei
Location of the TEI-XML sources.
If it does not exist, the program aborts with an error.
@@ -5327,11 +5344,12 @@ Parameters
Expand source code
-Browse git
+Browse git
class TEI(CheckImport):
def __init__(
self,
+ sourceBase=PARAMS["sourceBase"][1],
tei=PARAMS["tei"][1],
tf=PARAMS["tf"][1],
validate=PARAMS["validate"][1],
@@ -5383,6 +5401,13 @@ Parameters
## Input directories
+ ### `sourceBase`
+
+ *Top-level directory of the TEI-XML sources.*
+
+ By default (if left empty) it is the `tei` directory at the top-level of the
+ repo, but you can specify any directory you want.
+
### `tei`
*Location of the TEI-XML sources.*
@@ -5513,10 +5538,13 @@ Parameters
repoDir = f"{base}/{org}/{repo}"
refDir = f"{repoDir}{relative}"
programDir = f"{refDir}/programs"
- schemaDir = f"{refDir}/schema"
convertSpec = f"{programDir}/tei.yaml"
convertCustom = f"{programDir}/tei.py"
+ sourceRefDir = sourceBase if sourceBase else refDir
+ teiDir = f"{sourceRefDir}/tei"
+ reportDir = f"{sourceRefDir}/report"
+ schemaDir = f"{sourceRefDir}/schema"
self.schemaDir = schemaDir
settings = readYaml(asFile=convertSpec, plain=True)
@@ -5686,10 +5714,8 @@ Parameters
self.siblingEdges = siblingEdges
self.procins = procins
- reportDir = f"{refDir}/report"
appDir = f"{refDir}/app"
docsDir = f"{refDir}/docs"
- teiDir = f"{refDir}/tei"
tfDir = f"{refDir}/tf"
if tei in {"", None}:
@@ -5697,6 +5723,7 @@ Parameters
reportPath = f"{reportDir}"
errMsg = f"source directory does not exist: {ux(teiDir)}"
teiVersion = ""
+ teiVersionRep = ""
else:
teiVersions = sorted(dirContents(teiDir)[1], key=versionSort)
nTeiVersions = len(teiVersions)
@@ -5724,9 +5751,11 @@ Parameters
else:
teiVersion = tei
- teiPath = f"{teiDir}/{teiVersion}"
- reportPath = f"{reportDir}/{teiVersion}"
errMsg = f"source version {teiVersion} does not exists in {ux(teiDir)}"
+ teiVersionRep = f"/{teiVersion}"
+
+ teiPath = f"{teiDir}{teiVersionRep}"
+ reportPath = f"{reportDir}{teiVersionRep}"
if not dirExists(teiPath):
console(errMsg, error=True)
@@ -5943,7 +5972,7 @@ Parameters
generic["sourceFormat"] = "TEI"
generic["version"] = tfVersion
- generic["teiVersion"] = teiVersion
+ generic["teiVersion"] = teiVersion or "not specified"
generic["schema"] = "TEI" + (" + " + (" + ".join(models))) if models else ""
extraInstructions = []
@@ -8960,8 +8989,8 @@ Parameters
itemPre = f"{targetDir}/{fileBase}_orig{fileExt}"
justCopy = info["justCopy"]
- teiDir = f"{myDir}/{sourceBit}"
- itemSource = f"{teiDir}/{file}"
+ srcDir = f"{myDir}/{sourceBit}"
+ itemSource = f"{srcDir}/{file}"
# If there is custom info, we do not have to preserve the previous version.
# Otherwise we save the target before overwriting it; # unless it
diff --git a/tf/convert/tf.html b/tf/convert/tf.html
index 722aa8081..1357e7dcd 100644
--- a/tf/convert/tf.html
+++ b/tf/convert/tf.html
@@ -32,7 +32,7 @@ Raw, unoptimised data from TF files<
Expand source code
-Browse git
+Browse git
"""
# Raw, unoptimised data from TF files
diff --git a/tf/convert/variants.html b/tf/convert/variants.html
index c5f83901c..fc3d50e56 100644
--- a/tf/convert/variants.html
+++ b/tf/convert/variants.html
@@ -38,7 +38,7 @@ Variants
Expand source code
-Browse git
+Browse git
"""
# Variants
@@ -676,7 +676,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class Variants:
def __init__(self, cv, cur, baseWitness, sentType, checkPunc, addWarning, addError):
diff --git a/tf/convert/walker.html b/tf/convert/walker.html
index fc345c788..f034fb931 100644
--- a/tf/convert/walker.html
+++ b/tf/convert/walker.html
@@ -157,7 +157,7 @@ Action methods
Expand source code
-Browse git
+Browse git
"""
# Walker
@@ -1973,7 +1973,7 @@ Classes
Expand source code
-Browse git
+Browse git
class CV:
S = "slot"
diff --git a/tf/convert/watm.html b/tf/convert/watm.html
index ed2df9def..b1b5ba23c 100644
--- a/tf/convert/watm.html
+++ b/tf/convert/watm.html
@@ -55,107 +55,6 @@ The situation
pipeline
from a corpus of letters by P.C. Hooft in Folia format to text segments and
web annotations.
-Excursion: STAM
-
-We now have two sytems,
-STAM
-and Text-Fabric that can untangle text and markup.
-They are implemented very differently, and have a different flavour, but at the
-same time they share the preference of separating the textual data from all the data
-around the text.
-
--
-
intent
-
-- STAM: make it easier for tools and infrastructure to handle texts with
-annotations.
-- TF: support researchers in analysing textual corpora.
-
-
--
-
implementation
-
-- STAM: Rust + Python bindings.
-- TF: Pure Python.
-
-
--
-
organization
-
-
--
-
standards
-
-- STAM: actively seeks to interoperate with existing standards, but
-internally it uses its own way of organizing the data.
-- TF: also relies on a few simple conventions w.r.t. data
-organization and efficient serialization. These conventions are
-documented. It has several import and export functions, e.g. from TEI,
-PageXML, MQL, and to MQL, TSV. But it prefers to input and output data
-in minimalistic streams, without the often redundant strings that are
-attached to standard formats.
-
-
--
-
model
-
-- STAM: very generic w.r.t. annotations, annotations can target
-annotations and /or text segments.
-- TF:
-graph model
-where nodes stand for textual positions and subsets of them, nodes and
-edges can have features, which are the raw material of annotations, but
-annotations are not a TF concept.
-
-
--
-
query language
-
-
--
-
display
-
-- STAM: In development, see
stam view in
-STAM tools.
-- TF: Powerful functions to display corpus fragments with highlighting in
-
tf.advanced. The challenge is to build generic display functions that detect
-the peculiarities of the corpora.
-
-
--
-
API
-
-- STAM: in Rust and Python.
-- TF: Python.
-
-
--
-
GUI
-
-- STAM: not yet.
-- TF: locally served web interface for browsing and searching the corpus.
-
-
-
-Both libraries can be used to manage corpus data in intricate ways for research
-and publishing purposes.
-How STAM and Text-Fabric will evolve in the dynamic landscape of corpora, analytical
-methods and AI, is something we cannot predict.
-For now, their different flavour and intent will define their appeal to the different
-categories of users.
The general idea
The idea of WATM is, like the idea of Text-Fabric, to untangle the text from its
markup. Everything outside the text itself is coded in annotations.
@@ -200,6 +99,10 @@ The specifics
thetext-`i.json files!
- a pair of files
anno2node.tsv and pos2node.tsv that map annotations resp. text
positions to their corresponding TF nodes.
+- a file
logicalpairs.tsv (only if it is configured to produce it).
+For each pair of tokens across a line boundary of which the first ends
+with an hyphen, it contains three rows: two with the node numbers of both
+tokens, and one with the combined text of the tokens without the hyphen.
Format of the text files
A text-i.json is a JSON file with the following structure:
@@ -535,6 +438,12 @@ Configuration
Any tab or newline in the body must be written as resp. The tsv files will have exactly one header line.
+-
+
hyphenation: information on combining tokens around an end-of-line hyphen.
+The information consists of the lineType (the node type of lines in the
+corpus).
+It is assumed that the node type of the tokens is the slot type.
+
Caveat
The WATM representation of the corpus is a faithful and complete representation
@@ -589,7 +498,7 @@
Caveat
Expand source code
-Browse git
+Browse git
"""Export to Web Annotation Text Model
@@ -626,91 +535,6 @@ Caveat
from a corpus of letters by P.C. Hooft in Folia format to text segments and
web annotations.
-# Excursion: STAM
-
-
-
-We now have two sytems,
-[STAM](https://github.com/annotation/stam)
-and Text-Fabric that can untangle text and markup.
-They are implemented very differently, and have a different flavour, but at the
-same time they share the preference of separating the textual data from all the data
-around the text.
-
-* *intent*
-
- * **STAM**: make it easier for tools and infrastructure to handle texts with
- annotations.
- * **TF**: support researchers in analysing textual corpora.
-
-* *implementation*
-
- * **STAM**: Rust + [Python bindings](https://github.com/annotation/stam-python).
- * **TF**: Pure Python.
-
-* *organization*
-
- * **STAM**: very neatly in a core with extensions.
- * **TF**: core data functionality in `tf.core` modules, search functionality in
- `tf.search` modules, lots of other functions are included in the code with
- varying degrees of integration and orderliness!
-
-* *standards*
-
- * **STAM**: actively seeks to interoperate with existing standards, but
- internally it uses its own way of organizing the data.
- * **TF**: also relies on a few simple conventions w.r.t. data
- organization and efficient serialization. These conventions are
- documented. It has several import and export functions, e.g. from TEI,
- PageXML, MQL, and to MQL, TSV. But it prefers to input and output data
- in minimalistic streams, without the often redundant strings that are
- attached to standard formats.
-
-* *model*
-
- * **STAM**: very generic w.r.t. annotations, annotations can target
- annotations and /or text segments.
- * **TF**:
- [graph model](https://annotation.github.io/text-fabric/tf/about/datamodel.html)
- where nodes stand for textual positions and subsets of them, nodes and
- edges can have features, which are the raw material of annotations, but
- annotations are not a TF concept.
-
-* *query language*
-
- * **STAM**:
- [STAMQL](https://github.com/annotation/stam/tree/master/extensions/stam-query),
- evolving as an SQL-like language, with user-friendly primitives for
- annotations.
- * **TF**:
- [TF-Query](https://annotation.github.io/text-fabric/tf/about/searchusage.html),
- a noise-free language for graph templates with reasonable performance.
-
-* *display*
-
- * **STAM**: In development, see `stam view` in
- [STAM tools](https://github.com/annotation/stam-tools).
- * **TF**: Powerful functions to display corpus fragments with highlighting in
- `tf.advanced`. The challenge is to build generic display functions that detect
- the peculiarities of the corpora.
-
-* *API*
-
- * **STAM**: in Rust and Python.
- * **TF**: Python.
-
-* *GUI*
-
- * **STAM**: not yet.
- * **TF**: locally served web interface for browsing and searching the corpus.
-
-Both libraries can be used to manage corpus data in intricate ways for research
-and publishing purposes.
-How STAM and Text-Fabric will evolve in the dynamic landscape of corpora, analytical
-methods and AI, is something we cannot predict.
-For now, their different flavour and intent will define their appeal to the different
-categories of users.
-
# The general idea
The idea of WATM is, like the idea of Text-Fabric, to untangle the text from its
@@ -763,6 +587,10 @@ Caveat
the `text-`*i*`.json` files!
* a pair of files `anno2node.tsv` and `pos2node.tsv` that map annotations resp. text
positions to their corresponding TF nodes.
+* a file `logicalpairs.tsv` (only if it is configured to produce it).
+ For each pair of tokens across a line boundary of which the first ends
+ with an hyphen, it contains three rows: two with the node numbers of both
+ tokens, and one with the combined text of the tokens without the hyphen.
## Format of the text files
@@ -1073,6 +901,11 @@ Caveat
The tsv files will have exactly one header line.
+* `hyphenation`: information on combining tokens around an end-of-line hyphen.
+ The information consists of the `lineType` (the node type of lines in the
+ corpus).
+ It is assumed that the node type of the tokens is the slot type.
+
# Caveat
The WATM representation of the corpus is a faithful and complete representation
@@ -1135,6 +968,7 @@ Caveat
from ..core.helpers import console
from ..core.files import (
fileOpen,
+ fileCopy,
initTree,
dirContents,
dirExists,
@@ -1162,6 +996,7 @@ Caveat
IIIF_FILE = "iiif.yaml"
NODEMAP_FILE = "anno2node.tsv"
SLOTMAP_FILE = "pos2node.tsv"
+LOGICAL_FILE = "logicalpairs.tsv"
TT_NAME = "watm"
@@ -1239,16 +1074,13 @@ Caveat
* the repo location
* the relative location of the tf directory in the repo
- In development, the WATM is written to a `watm` directory inside the
- `_temp` directory of the repo.
- The version of the WATM is the version of the TF from which it is generated.
+ The WATM is written to somewhere under the `watm` directory at the toplevel
+ of the repo.
- In production, the WATM is written to the `watm` directory outside the
- `_temp` directory of the repo.
The version of the WATM is the version of the TF from which it is generated plus
a suffix, containing a sequence number.
- Whenever a prodcution generation is executed, a new sequence number is chosen,
+ Whenever a pgeneration is executed, a new sequence number is chosen,
so that no previous version gets overwritten.
If there is a file `latest` in the directory of the WATM versions, its number is
@@ -1259,6 +1091,9 @@ Caveat
Whatever we got, we increment it and write it to the file `latest`.
That is the new version suffix.
+ Inside the resulting directory, the production WATM resides under `prod` and
+ the development WATM resides under `dev`.
+
Parameters
----------
baseDir: string
@@ -1276,50 +1111,51 @@ Caveat
-------
The path to the versioned dir where the WATM ends up.
"""
- if prod:
- resultDirBase = f"{baseDir}{headPart}/{TT_NAME}"
- latestFile = f"{resultDirBase}/latest"
- prefix = f"{version}-"
- prodNumReps = {
- pv.removeprefix(prefix)
- for pv in dirContents(resultDirBase)[1]
- if pv.startswith(prefix)
- }
-
- maxProdNum = max(
- (
- int(prodNum)
- for prodNumRep in prodNumReps
- if (prodNum := prodNumRep.lstrip("0")).isdecimal()
- ),
- default=0,
- )
-
- if not fileExists(latestFile):
- latestNum = 0
- else:
- with fileOpen(latestFile) as fh:
- contents = fh.read().strip()
- latestNum = int(contents) if contents.isdecimal() else 0
+ prodRep = "production" if prod else "development"
+ prodFix = "prod" if prod else "dev"
+ resultDirBase = f"{baseDir}{headPart}/{TT_NAME}"
+ latestFile = f"{resultDirBase}/latest"
+
+ prefix = f"{version}-"
+ numReps = {
+ pv.removeprefix(prefix)
+ for pv in dirContents(resultDirBase)[1]
+ if pv.startswith(prefix)
+ }
+
+ maxNum = max(
+ (int(num) for numRep in numReps if (num := numRep.lstrip("0")).isdecimal()),
+ default=0,
+ )
+
+ if not fileExists(latestFile):
+ latestNum = 0
+ else:
+ with fileOpen(latestFile) as fh:
+ contents = fh.read().strip().split("-", 1)
+ latestNumStr = (contents[1] if len(contents) == 2 else "").lstrip("0")
+ latestNum = int(latestNumStr) if latestNumStr.isdecimal() else 0
- newNum = max((latestNum, maxProdNum))
+ if prod:
+ newNum = max((latestNum, maxNum))
while True:
newNum += 1
newNumRep = f"{newNum:>03}"
- if newNumRep in prodNumReps:
+ if newNumRep in numReps:
continue
break
with fileOpen(latestFile, "w") as fh:
- fh.write(f"{newNum}\n")
+ fh.write(f"{version}-{newNumRep}\n")
+ else:
+ newNumRep = f"{latestNum:>03}"
- resultVersionDir = f"{resultDirBase}/{version}-{newNumRep}"
+ resultVersionDir = f"{resultDirBase}/{version}-{newNumRep}/{prodFix}"
- if not silent:
- console(f"Writing production data to {resultVersionDir}")
+ if not silent:
+ console(f"Writing {prodRep} data to {resultVersionDir}")
else:
- resultDirBase = f"{baseDir}/_temp{headPart}/{TT_NAME}"
resultVersionDir = f"{resultDirBase}/{version}"
if not silent:
@@ -1390,7 +1226,12 @@ Caveat
F = api.F
E = api.E
T = api.T
+ Fall = api.Fall
+ Eall = api.Eall
+ FAllSet = set(Fall())
+
sectionTypes = T.sectionTypes
+ strv = F.str.v if STR in FAllSet else None
cfg = readYaml(asFile=CONFIG_FILE)
self.cfg = cfg
@@ -1417,6 +1258,29 @@ Caveat
)
self.error = True
+ hyphenation = cfg.hyphenation
+
+ if hyphenation is not None:
+ if strv is None:
+ console(
+ f"{CONFIG_FILE}: hyphenation cannot be solved "
+ "because feature str does not exist in the TF dataset",
+ error=True,
+ )
+ hyphenation = None
+ self.error = True
+
+ if hyphenation.lineType not in F.otype.all:
+ console(
+ f"{CONFIG_FILE}: hyphenation.lineType "
+ "must be an existing node type",
+ error=True,
+ )
+ hyphenation = None
+ self.error = True
+
+ self.hyphenation = hyphenation
+
if self.error:
return
@@ -1435,26 +1299,23 @@ Caveat
self.E = E
self.Fs = api.Fs
self.slotType = self.F.otype.slotType
+ self.maxSlot = self.F.otype.maxSlot
self.maxSlotPlus = self.F.otype.maxSlot + 1
self.maxNodePlus = self.F.otype.maxNode + 1
self.otypes = self.F.otype.all
self.info = app.info
self.repoLocation = app.repoLocation
- Fall = api.Fall
- Eall = api.Eall
self.Fall = Fall
self.Eall = Eall
self.nodeFeatures = [f for f in Fall() if f not in TF_SPECIFIC_FEATURES]
self.edgeFeatures = [f for f in Eall() if f not in TF_SPECIFIC_FEATURES]
- FAllSet = set(Fall())
-
self.fotypev = F.otype.v
self.eoslots = E.oslots.s
self.emptyv = F.empty.v if "empty" in FAllSet else None
- self.strv = F.str.v if STR in FAllSet else None
+ self.strv = strv
self.rstrv = F.rstr.v if "rstr" in FAllSet else None
self.afterv = F.after.v if AFTER in FAllSet else None
self.rafterv = F.rafter.v if "rafter" in FAllSet else None
@@ -1669,16 +1530,27 @@ Caveat
for i, n in enumerate(nodes):
for extraFeat, value, variables in thisScanInfo:
values = {}
+ valuesOK = True
+
+ for var, (parent, child, feat, shift) in variables.items():
+ refI = i + shift
+
+ if refI < 0 or refI > lastI:
+ valuesOK = False
+ break
- for var, (parent, feat, shift) in variables.items():
- refI = min((max((0, i + shift)), lastI))
refN = nodes[refI]
if parent is not None:
refN = L.u(refN, otype=parent)[0]
+ elif child is not None:
+ refN = L.d(refN, otype=child)[0]
values[var] = refN if feat == "=" else Fs(feat).v(refN)
+ if not valuesOK:
+ continue
+
val = value.format(**values)
self.mkAnno(
@@ -1797,6 +1669,39 @@ Caveat
if nFarTargets > 10:
self.console(f"... and {nFarTargets - 10} more.")
+ def getLogicalPairs(self):
+ hyphenation = self.hyphenation
+ F = self.F
+ L = self.L
+ slotType = self.slotType
+ strv = self.strv
+ afterv = self.afterv
+ maxSlot = self.maxSlot
+
+ if hyphenation is None:
+ return []
+
+ lineType = hyphenation.lineType
+
+ pairs = []
+ lines = F.otype.s(lineType)
+
+ for line in lines:
+ lastSlot = L.d(line, otype=slotType)[-1]
+
+ if strv(lastSlot) == "-" and lastSlot > 1 and lastSlot < maxSlot:
+ s1 = lastSlot - 1
+ s2 = lastSlot + 1
+ pairs.append(
+ (
+ s1,
+ s2,
+ f"{strv(s1)}{strv(s2)}{afterv(s2)}",
+ )
+ )
+
+ return tuple(pairs)
+
def writeAll(self, resultVersion=None):
"""Write text and annotation data to disk.
@@ -1814,7 +1719,9 @@ Caveat
This is needed if WATM is generated for a series of related datasets that
have the same result base.
"""
-
+ cfg = self.cfg
+ extra = cfg.extra
+ hyphenation = self.hyphenation
maxNodePlus = self.maxNodePlus
maxSlotPlus = self.maxSlotPlus
@@ -1975,6 +1882,30 @@ Caveat
self.console(f"Slot mapping written to {slotmapFile}")
self.console(f"Node mapping written to {nodemapFile}")
+ # hyphenation information files
+
+ if hyphenation is not None:
+ logicalFile = f"{resultDir}/{LOGICAL_FILE}"
+
+ with fileOpen(logicalFile, "w") as fh:
+ fh.write("token1\ttoken2\tstr\n")
+
+ pairs = self.getLogicalPairs()
+
+ for t1, t2, text in pairs:
+ fh.write(f"{t1}\t{t2}\t{text}\n")
+
+ self.console(f"Logical pairs ({len(pairs)}) written to {logicalFile}")
+
+ # extra files
+
+ for extraDir, files in extra.items():
+ for file in files:
+ fileCopy(f"{baseDir}/{extraDir}/{file}", f"{resultDir}/{file}")
+ self.console(
+ f"Extra file {extraDir}/{file} copied to the WATM directory"
+ )
+
@staticmethod
def numEqual(nTF, nWA, silent):
"""Compare two numbers and report the outcome.
@@ -2126,6 +2057,7 @@ Caveat
# collect the files
asTsv = self.asTsv
+ hyphenation = self.hyphenation
resultDir = self.resultDir
resultFiles = dirContents(resultDir)[0]
@@ -2155,6 +2087,9 @@ Caveat
if SLOTMAP_FILE not in mapFiles:
console(f"ERROR: Missing map file {SLOTMAP_FILE}")
self.error = True
+ if hyphenation is not None and LOGICAL_FILE not in resultFiles:
+ console(f"ERROR: Missing logical file {LOGICAL_FILE}")
+ self.error = True
if self.error:
return
@@ -3120,14 +3055,11 @@ Functions
- the repo location
- the relative location of the tf directory in the repo
-In development, the WATM is written to a watm directory inside the
-_temp directory of the repo.
-The version of the WATM is the version of the TF from which it is generated.
-In production, the WATM is written to the watm directory outside the
-_temp directory of the repo.
-The version of the WATM is the version of the TF from which it is generated plus
+
The WATM is written to somewhere under the watm directory at the toplevel
+of the repo.
+The version of the WATM is the version of the TF from which it is generated plus
a suffix, containing a sequence number.
-Whenever a prodcution generation is executed, a new sequence number is chosen,
+
Whenever a pgeneration is executed, a new sequence number is chosen,
so that no previous version gets overwritten.
If there is a file latest in the directory of the WATM versions, its number is
taken as the previous version suffix. But if there are versions with a higher
@@ -3135,6 +3067,8 @@
Functions
suffix.
Whatever we got, we increment it and write it to the file latest.
That is the new version suffix.
+Inside the resulting directory, the production WATM resides under prod and
+the development WATM resides under dev.
Parameters
baseDir : stringParameters
Expand source code
-Browse git
+Browse git
class WATM:
"""The export machinery is exposed as a class, wrapped around a TF dataset."""
@@ -3284,7 +3218,12 @@ Parameters
F = api.F
E = api.E
T = api.T
+ Fall = api.Fall
+ Eall = api.Eall
+ FAllSet = set(Fall())
+
sectionTypes = T.sectionTypes
+ strv = F.str.v if STR in FAllSet else None
cfg = readYaml(asFile=CONFIG_FILE)
self.cfg = cfg
@@ -3311,6 +3250,29 @@ Parameters
)
self.error = True
+ hyphenation = cfg.hyphenation
+
+ if hyphenation is not None:
+ if strv is None:
+ console(
+ f"{CONFIG_FILE}: hyphenation cannot be solved "
+ "because feature str does not exist in the TF dataset",
+ error=True,
+ )
+ hyphenation = None
+ self.error = True
+
+ if hyphenation.lineType not in F.otype.all:
+ console(
+ f"{CONFIG_FILE}: hyphenation.lineType "
+ "must be an existing node type",
+ error=True,
+ )
+ hyphenation = None
+ self.error = True
+
+ self.hyphenation = hyphenation
+
if self.error:
return
@@ -3329,26 +3291,23 @@ Parameters
self.E = E
self.Fs = api.Fs
self.slotType = self.F.otype.slotType
+ self.maxSlot = self.F.otype.maxSlot
self.maxSlotPlus = self.F.otype.maxSlot + 1
self.maxNodePlus = self.F.otype.maxNode + 1
self.otypes = self.F.otype.all
self.info = app.info
self.repoLocation = app.repoLocation
- Fall = api.Fall
- Eall = api.Eall
self.Fall = Fall
self.Eall = Eall
self.nodeFeatures = [f for f in Fall() if f not in TF_SPECIFIC_FEATURES]
self.edgeFeatures = [f for f in Eall() if f not in TF_SPECIFIC_FEATURES]
- FAllSet = set(Fall())
-
self.fotypev = F.otype.v
self.eoslots = E.oslots.s
self.emptyv = F.empty.v if "empty" in FAllSet else None
- self.strv = F.str.v if STR in FAllSet else None
+ self.strv = strv
self.rstrv = F.rstr.v if "rstr" in FAllSet else None
self.afterv = F.after.v if AFTER in FAllSet else None
self.rafterv = F.rafter.v if "rafter" in FAllSet else None
@@ -3563,16 +3522,27 @@ Parameters
for i, n in enumerate(nodes):
for extraFeat, value, variables in thisScanInfo:
values = {}
+ valuesOK = True
+
+ for var, (parent, child, feat, shift) in variables.items():
+ refI = i + shift
+
+ if refI < 0 or refI > lastI:
+ valuesOK = False
+ break
- for var, (parent, feat, shift) in variables.items():
- refI = min((max((0, i + shift)), lastI))
refN = nodes[refI]
if parent is not None:
refN = L.u(refN, otype=parent)[0]
+ elif child is not None:
+ refN = L.d(refN, otype=child)[0]
values[var] = refN if feat == "=" else Fs(feat).v(refN)
+ if not valuesOK:
+ continue
+
val = value.format(**values)
self.mkAnno(
@@ -3691,6 +3661,39 @@ Parameters
if nFarTargets > 10:
self.console(f"... and {nFarTargets - 10} more.")
+ def getLogicalPairs(self):
+ hyphenation = self.hyphenation
+ F = self.F
+ L = self.L
+ slotType = self.slotType
+ strv = self.strv
+ afterv = self.afterv
+ maxSlot = self.maxSlot
+
+ if hyphenation is None:
+ return []
+
+ lineType = hyphenation.lineType
+
+ pairs = []
+ lines = F.otype.s(lineType)
+
+ for line in lines:
+ lastSlot = L.d(line, otype=slotType)[-1]
+
+ if strv(lastSlot) == "-" and lastSlot > 1 and lastSlot < maxSlot:
+ s1 = lastSlot - 1
+ s2 = lastSlot + 1
+ pairs.append(
+ (
+ s1,
+ s2,
+ f"{strv(s1)}{strv(s2)}{afterv(s2)}",
+ )
+ )
+
+ return tuple(pairs)
+
def writeAll(self, resultVersion=None):
"""Write text and annotation data to disk.
@@ -3708,7 +3711,9 @@ Parameters
This is needed if WATM is generated for a series of related datasets that
have the same result base.
"""
-
+ cfg = self.cfg
+ extra = cfg.extra
+ hyphenation = self.hyphenation
maxNodePlus = self.maxNodePlus
maxSlotPlus = self.maxSlotPlus
@@ -3869,6 +3874,30 @@ Parameters
self.console(f"Slot mapping written to {slotmapFile}")
self.console(f"Node mapping written to {nodemapFile}")
+ # hyphenation information files
+
+ if hyphenation is not None:
+ logicalFile = f"{resultDir}/{LOGICAL_FILE}"
+
+ with fileOpen(logicalFile, "w") as fh:
+ fh.write("token1\ttoken2\tstr\n")
+
+ pairs = self.getLogicalPairs()
+
+ for t1, t2, text in pairs:
+ fh.write(f"{t1}\t{t2}\t{text}\n")
+
+ self.console(f"Logical pairs ({len(pairs)}) written to {logicalFile}")
+
+ # extra files
+
+ for extraDir, files in extra.items():
+ for file in files:
+ fileCopy(f"{baseDir}/{extraDir}/{file}", f"{resultDir}/{file}")
+ self.console(
+ f"Extra file {extraDir}/{file} copied to the WATM directory"
+ )
+
@staticmethod
def numEqual(nTF, nWA, silent):
"""Compare two numbers and report the outcome.
@@ -4020,6 +4049,7 @@ Parameters
# collect the files
asTsv = self.asTsv
+ hyphenation = self.hyphenation
resultDir = self.resultDir
resultFiles = dirContents(resultDir)[0]
@@ -4049,6 +4079,9 @@ Parameters
if SLOTMAP_FILE not in mapFiles:
console(f"ERROR: Missing map file {SLOTMAP_FILE}")
self.error = True
+ if hyphenation is not None and LOGICAL_FILE not in resultFiles:
+ console(f"ERROR: Missing logical file {LOGICAL_FILE}")
+ self.error = True
if self.error:
return
@@ -4892,6 +4925,12 @@ Methods
This works exactly as console()
When the silent member of the object is True, the message will be suppressed.
+
+def getLogicalPairs(self)
+-
+
+
def makeAnno(self)
Parameters
Expand source code
-Browse git
+Browse git
class WATMS:
"""Export corpora that are divided over multiple TF datasets.
@@ -5335,7 +5374,6 @@ Index
- The situation
-- Excursion: STAM
- The general idea
- The specifics
- Format of the text files
@@ -5372,6 +5410,7 @@ Index
WATM
consolegetLogicalPairsmakeAnnomakeTextmkAnnoExamples
Expand source code
-Browse git
+Browse git
"""
# XML import
@@ -318,7 +318,8 @@ Examples
APP_CONFIG
)
-(HELP, TASKS, TASKS_EXCLUDED, PARAMS, FLAGS) = setUp("XML")
+TASKS_EXCLUDED = {"apptoken", "browse"}
+(HELP, TASKS, PARAMS, FLAGS) = setUp("XML")
TRIM_ATTS = set(
"""
@@ -337,6 +338,7 @@ Examples
trimAtts=set(),
keywordAtts=set(),
renameAtts={},
+ sourceBase=PARAMS["sourceBase"][1],
xml=PARAMS["xml"][1],
tf=PARAMS["tf"][1],
verbose=FLAGS["verbose"][1],
@@ -509,6 +511,10 @@ Examples
convertSpec = f"{programDir}/xml.yaml"
convertCustom = f"{programDir}/xml.py"
+ sourceRefDir = sourceBase if sourceBase else refDir
+ xmlDir = f"{sourceRefDir}/xml"
+ reportDir = f"{sourceRefDir}/report"
+
settings = readYaml(asFile=convertSpec, plain=True)
self.transform = None
@@ -535,9 +541,7 @@ Examples
procins = settings.get("procins", False)
self.procins = procins
- reportDir = f"{refDir}/report"
appDir = f"{refDir}/app"
- xmlDir = f"{refDir}/xml"
tfDir = f"{refDir}/tf"
xmlVersions = sorted(dirContents(xmlDir)[1], key=versionSort)
@@ -1404,7 +1408,7 @@ Classes
class XML
-(convertTaskCustom=None, trimAtts=set(), keywordAtts=set(), renameAtts={}, xml='latest', tf='latest', verbose=-1)
+(convertTaskCustom=None, trimAtts=set(), keywordAtts=set(), renameAtts={}, sourceBase='', xml='latest', tf='latest', verbose=-1)
-
Converts XML to TF.
@@ -1522,7 +1526,7 @@ Parameters
Expand source code
-Browse git
+Browse git
class XML(CheckImport):
def __init__(
@@ -1531,6 +1535,7 @@ Parameters
trimAtts=set(),
keywordAtts=set(),
renameAtts={},
+ sourceBase=PARAMS["sourceBase"][1],
xml=PARAMS["xml"][1],
tf=PARAMS["tf"][1],
verbose=FLAGS["verbose"][1],
@@ -1703,6 +1708,10 @@ Parameters
convertSpec = f"{programDir}/xml.yaml"
convertCustom = f"{programDir}/xml.py"
+ sourceRefDir = sourceBase if sourceBase else refDir
+ xmlDir = f"{sourceRefDir}/xml"
+ reportDir = f"{sourceRefDir}/report"
+
settings = readYaml(asFile=convertSpec, plain=True)
self.transform = None
@@ -1729,9 +1738,7 @@ Parameters
procins = settings.get("procins", False)
self.procins = procins
- reportDir = f"{refDir}/report"
appDir = f"{refDir}/app"
- xmlDir = f"{refDir}/xml"
tfDir = f"{refDir}/tf"
xmlVersions = sorted(dirContents(xmlDir)[1], key=versionSort)
diff --git a/tf/convert/xmlCustom.html b/tf/convert/xmlCustom.html
index f357d6edd..d54afb611 100644
--- a/tf/convert/xmlCustom.html
+++ b/tf/convert/xmlCustom.html
@@ -31,7 +31,7 @@ Module tf.convert.xmlCustom
Expand source code
-Browse git
+Browse git
import re
from io import BytesIO
diff --git a/tf/core/api.html b/tf/core/api.html
index 9aa08ede3..7ae929ce3 100644
--- a/tf/core/api.html
+++ b/tf/core/api.html
@@ -33,7 +33,7 @@ The core API of TF.
Expand source code
-Browse git
+Browse git
"""
# The core API of TF.
@@ -675,7 +675,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Api:
def __init__(self, TF):
diff --git a/tf/core/command.html b/tf/core/command.html
index c6a0df401..c037e1758 100644
--- a/tf/core/command.html
+++ b/tf/core/command.html
@@ -31,7 +31,7 @@ Module tf.core.command
Expand source code
-Browse git
+Browse git
import sys
from textwrap import dedent
diff --git a/tf/core/computed.html b/tf/core/computed.html
index 4c085bfd4..9486b5a8c 100644
--- a/tf/core/computed.html
+++ b/tf/core/computed.html
@@ -52,7 +52,7 @@ Pre-computed data components
Expand source code
-Browse git
+Browse git
"""
# Pre-computed data components
@@ -117,7 +117,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Computed:
"""Provides access to pre-computed data.
@@ -138,7 +138,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Computeds:
pass
diff --git a/tf/core/data.html b/tf/core/data.html
index bf3f45efd..6ab67f9ab 100644
--- a/tf/core/data.html
+++ b/tf/core/data.html
@@ -31,7 +31,7 @@ Module tf.core.data
Expand source code
-Browse git
+Browse git
import array
import gc
@@ -775,7 +775,7 @@ Classes
Expand source code
-Browse git
+Browse git
class Data:
def __init__(
diff --git a/tf/core/edgefeature.html b/tf/core/edgefeature.html
index 7018ac673..0991c0327 100644
--- a/tf/core/edgefeature.html
+++ b/tf/core/edgefeature.html
@@ -46,7 +46,7 @@ Module tf.core.edgefeature
Expand source code
-Browse git
+Browse git
"""
Mappings from edges to values.
@@ -347,7 +347,7 @@ Classes
Expand source code
-Browse git
+Browse git
class EdgeFeature:
"""Provides access to (edge) feature data.
@@ -771,7 +771,7 @@ Returns
Expand source code
-Browse git
+Browse git
class EdgeFeatures:
pass
diff --git a/tf/core/fabric.html b/tf/core/fabric.html
index 4a600b7c1..c53a33061 100644
--- a/tf/core/fabric.html
+++ b/tf/core/fabric.html
@@ -41,7 +41,7 @@