This unit relies on the oscal-convert project in this - repository, with its files. Like all projects in the repo, it is designed to be self-contained and - self-explanatory to the extent practical. Use your search engine and XProc resources to learn background and - terminology.
+This unit relies on the oscal-convert project in + this repository, with its files. Like all projects in the repo, it aims to be reasonably self-contained and + self-explanatory. Use your search engine and XProc resources to learn background and terminology.
Also like other projects, there are preliminaries for acquiring resources, along with pipelines to run.
To verify syntactic correctness (well-formedness) - does it parse?
+To transcode a file from one encoding to another
+Any reader who feels a 599 lesson on encodings and XProc should pipe up! (Feel free to post an Issue, + Discussion, or PR with a draft.)
+XML and JSON are both data serializations, a term that designates how each of them is to be + considered and treated – irrespective of any questions of information storage – as a sequence of + characters. This is a very important commonality, which makes it possible to bring them together in a + single processing environment such as XProc.
+Along with plain text, perhaps the most important data serialization or format
as we call
+ them, of the three.
A simple XProc pipeline can be used to demonstrate this. While doing so, it shows also while this is not as + simple a process as it seems. Merely to convert from format to format is not enough.
+An acquisition pipeline in the project folder collects some OSCAL onto the local system, where it can be managed, easily inspected, controlled, and edited if necessary.
-But sources and results from a conversion process do not always have to work with local resources. XProc
- sees the Internet itself - whatever protocols are supported by the processor (the file and
- http protocols are required for conformance) work as they do on the Worldwide Web. Of
- course, permissions must be in place to read and write files from and into file system locations. But when
- authentication is configured or resources are openly available, using http to reach resources
- or sources can be a very convenient option.
TBD / this all incoherent so far
+Keep in mind that XProc in theory, and your XProc engine in practice, may read its inputs using whatever
+ protocols it supports, while the file and http protocols are required for
+ conformance, and work as they do on the Worldwide Web. Of course, permissions must be in place to read
+ and write files from and into file system locations. But when authentication is configured or resources
+ are openly available, using http to reach resources or sources can be a very convenient
+ option.
Keep in mind that where resources take the form of serializations of structured data such as XML or JSON,
+ what is read
(in the sense of presented to a consuming application) may not be quite exactly the
+ same data serialization, since as a serialization, it is parsed into an object or representation of some
+ kind,
TBD - TODO - question - how many and of what sort of source data files - so far there is only the cat catalog
In the publishing demonstration project folder is an XProc that valides XML against an OSCAL schema, before formatting it. The same could be done for an XProc that converts the data into JSON - either or both before or after conversion.
-Learn more about recognizing and dealing with errors in Lesson
+ Learn more about recognizing and dealing with errors in Lesson
102, or continue on to the next project, oscal-validate, for more on validation of documents and
sets of documents.
Same as the 101 lesson.
+Same as the 101 lesson.
As usual, the project contains pipelines to be used to acquire copies of resources, placing them in the project lib directory . These may include dependencies for steps in the pipeline (read details in the 102 lesson) so it is a good idea to run them all.
+ href="oscal-produce_102_src.html" class="LessonUnit">102 lesson) so it is a good idea to run them all.These include a pipeline that copies the PDF original from which the HTML input (next step) was derived,
using a COTS (commercial off-the-shelf) tool. Inspect and read this PDF to see what the upstream data looks
like - and get a sense of what the data set looks like when the structures are painted
on the
diff --git a/tutorial/source/oscal-produce/oscal-produce_102_src.html b/tutorial/source/oscal-produce/oscal-produce_102_src.html
index fdda0dc2..c9747647 100644
--- a/tutorial/source/oscal-produce/oscal-produce_102_src.html
+++ b/tutorial/source/oscal-produce/oscal-produce_102_src.html
@@ -2,6 +2,7 @@
Run the pipelines described in the 101 Lesson
+Run the pipelines described in the 101 Lesson
Like the 101 lesson, this lesson unit uses the Like the 101 lesson, this lesson unit uses the oscal-import project in this repo.
Run the pipelines described in the 101 Lesson
+Run the pipelines described in the 101 Lesson
Like the 101 lesson, this lesson unit uses the Like the 101 lesson, this lesson unit uses the oscal-import project in this repo.
Publishing your own OSCAL in these pipelines is also worth trying - the worst they can do is fail. Bring OSCAL, especially OSCAL catalogs.
diff --git a/tutorial/source/oscal-validate/oscal-validate_101_src.html b/tutorial/source/oscal-validate/oscal-validate_101_src.html index f7f8b60c..37dc669c 100644 --- a/tutorial/source/oscal-validate/oscal-validate_101_src.html +++ b/tutorial/source/oscal-validate/oscal-validate_101_src.html @@ -2,7 +2,7 @@The best of it is, as you learn how a traceback is a tasty treat, you enter a virtuous learning cycle. Repair a few bugs and you might get a taste for it. (Try the 102 Lesson unit )
+ class="LessonUnit">102 Lesson unit )If you'd rather keep your feet dry, advance to the next Lesson.
The setup script is a bash script: ./setup.sh, to be run with no arguments.
For XProc runtime -- to execute pipelines -- use either of the scripts ./xp3.sh (under bash) or ./xp3.bat (for Windows). These scripts are
+
For XProc runtime — to execute pipelines — use either of the scripts ./xp3.sh (under bash) or ./xp3.bat (for Windows). These scripts are
used for all pipelines (basically, for everything) unless otherwise noted.
The following pipelines will be run. They are described in top-level README documentation and the expected places.
Effectively (and much more could be said about the processing stack, dependency management and so forth) - what this means is that XProc offers the user and the developer (in either or both roles) very focused and - concentrated points of control or points of adjustment. Out in the field - where software is deployed and - used - it may never be like that. User interfaces constrain what users can do, and even developers are - rarely free to just mess around, as it might be thought of.
-To the extent this is the case, this project only works if things are actually simple enough to pick up and - use.
+ what this means is that XProc offers the user and the developer (in either or both roles) with focused and + concentrated points of control or points of adjustment. In the field -- where software is deployed and + used -- things almost never justdrop in. User interfaces, APIs, dependencies and platform quirks: all these + constrain what users can do, and even developers are rarely free to just mess around, as it might be thought of. +
To the extent this is the case, this project only works if things are actually simple enough to pick up, + use, learn and adapt.
xp3.sh and xp3.bat represent attempts at this. Each of them (on its execution
platform) enables a user to run, without further configuration, the Morgana XProcIIIse processor on any XProc
@@ -111,28 +112,32 @@
As simple examples, these scripts show only one way of running XProc. Keep in mind that even simple scripts - can be used in more than one way.
-For example, a pipeline can be executed from the project root:
-$ ./xp3.sh smoketest/TEST-XPROC3.xpl-
Alternatively, a pipeline can be executed from its home directory, for example if currently in the
- smoketest directory (note the path to the script):
$ ../xp3.sh TEST-XPROC3.xpl-
This works the same ways on Windows, with adjustments:
-> ..\xp3 TEST-XPROC3.xpl-
(On Windows a bat file suffix marks it as executable and does not have to be given explicitly
- when calling.)
Windows users (and others to varying degrees) can set up a drag-and-drop based workflow - using your mouse - or pointer, select an XProc pipeline file and drag it to a shortcut for the executable (Windows batch file). - A command window opens to show the operation of the pipeline. See the README - for more information.
-It is important to try things out since any of these methods can be the basis of a workflow.
-For the big picture, keep in mind that while the command line is useful for development and demonstration – - and however familiar XProc itself may become to the developer – to the uninitiated it remains obscure and - cryptic. XProc-based systems, when integrated into tools or developer editors and environments, can look - much nicer than tracebacks in a console window. The beauty we are looking for here is in a different kind of - elegance and power.
+ (XSLT, XSpec) to show how this exploration can be done. We are an outfitter at the beginning of what + we hope will be a long and profitable voyage. +As simple examples, these scripts show only one way of running XProc. Keep in mind that even simple + scripts can be used in more than one way.
+For example, a pipeline can be executed from the project root:
+$ ./xp3.sh smoketest/TEST-XPROC3.xpl+
Alternatively, a pipeline can be executed from its home directory, for example if currently in the
+ smoketest directory (note the path to the script):
$ ../xp3.sh TEST-XPROC3.xpl+
This works the same ways on Windows, with adjustments:
+> ..\xp3 TEST-XPROC3.xpl+
(On Windows a bat file suffix marks it as executable and does not have to be given
+ explicitly when calling.)
Windows users (and others to varying degrees) can set up a drag-and-drop based workflow - using your + mouse or pointer, select an XProc pipeline file and drag it to a shortcut for the executable (Windows + batch file). A command window opens to show the operation of the pipeline. See the README for more information.
+It is important to try things out since any of these methods can be the basis of a workflow.
+For the big picture, keep in mind that while the command line is useful for development and demonstration + – and however familiar XProc itself may become to the developer – to the uninitiated it remains obscure + and cryptic. XProc-based systems, when integrated into tools or developer editors and environments, can + look much nicer than tracebacks in a console window. The beauty we are looking for here is in a different + kind of elegance and power.
+Hello World. In that spirit, feel free to edit and adjust this file.
You have done Setup 101, Setup 102 and Unpack 101.
+ class="LessonUnit">Setup 102 and Unpack 101.XProc3or
XProc 3.0to help distinguish from 1.0 - technologies
XProc3or
XProc 3.0to help distinguish from 1.0 technologies
An example of a step library in this repository is xpec-execute.xpl, which collects several steps supporting XSpec, one each for supporting the + >xspec-execute.xpl, which collects several steps supporting XSpec, one each for supporting the XSpec testing framework for XSLT, XQuery and Schematron respectively.
The advantage of defining a step at the top level, rather than putting all steps into libraries, is that such a step can be invoked without prior knowledge of its type name, which is used by XProc to @@ -115,15 +114,15 @@
Keep in mind that every XProc pipeline is also, potentially and actually, a step. There are two things we need to know about steps - how to define them, and how to use them.
-We begin with how to recognize and use steps, but we can't avoid how to define them: because an XProc - pipeline is also an XProc step, we can't use steps without ending up with a pipeline. We have only to - look at the working pipeline we make with our steps, to see how a step is made.
+We begin with how to recognize and use steps. But because an XProc pipeline is also an XProc step, we + can't use steps without ending up with a pipeline. We have only to look at the working pipeline we make + with our steps, to see how a step is made.
As described in the XProc 3.0 specification, XProc step declarations can be divided into an initial set of elements for setup - and configuration, followed by what the specification calls a subpipeline, which is typically a + and configuration, followed by what the specification calls a subpipeline, consisting of a sequence of steps to be executed – any steps available, which could be anything. Think of the subpipeline as the working parts of the pipeline, while the rest is all about how it is set up.
-The list of elements that come before the steps is short, which helps: p:import,
+
The list of elements that come before the subpipeline is short, which helps: p:import,
p:import-functions, p:input, p:output, p:option or
p:declare-step. Everything coming after is a step.
Within this set of elements (all preceding, none following the subpipeline) XProc further distinguishes @@ -138,10 +137,9 @@
Following the prologue, a step may also have local step definitions (p:declare-step).
After imports, prologue and (optional) step declarations, the step sequence that follows comprises the subpipeline.
-One other complication: for purposes of exposition, we have pretended that p:variable (a
- variable declaration) and p:documentation (for out-of-band documentation) are steps, which
- is not how the XProc recommendation describes them, although they appears in the subpipeline with and
- among its steps.
One other complication: among the steps in the subpipeline, p:variable (a variable
+ declaration) and p:documentation (for out-of-band documentation) are also permitted – these
+ are not properly steps, but can be useful to have with them.
In summary: any XProc pipeline, viewed as a step declaration, can have the following --
source primary input port, permitting a
single documentp:declare-step. Each of these will have its own name, type, prologue and stepsp:)Given an understanding of the organization of an XProc pipeline, all that remains to understand of its
syntax is the steps themselves, which follow a common pattern. Briefly put, atomic steps are any steps
you use by simply invoking it with inputs and options: its logic is self-contained, and the operation it
- carries out is (at least conceptually) single
. Compound steps, instead, are used to execute more
+ carries out is (at least conceptually) single
. Compound steps, instead, are used to deploy more
than one subpipeline, with settings determined dynamically for the step.
Fortunately XProc keeps things simple by providing only a few compound steps supporting the identified range of needs – and no way for users to define their own. This does not prove to be a practical @@ -228,7 +226,7 @@
See the 599-level coverage in this lesson unit for a discussion of the +
See the 599-level coverage in this lesson unit for a discussion of the schema for XProc.
More in depth.
@@ -35,7 +35,7 @@It may often be considered gratuitous to validate XProc files against a schema, when the application (for - us, Morgana)must in any case take responsibility for conformance issues, as it sees fit. The reference + us, Morgana) must in any case take responsibility for conformance issues, as it sees fit. The reference schema becomes useful if we find or suspect bugs in Morgana, but until then it need not have any direct role in any runtime operation.
Nevertheless the RNG schema still serves as a reference and an object for querying -- queries whose results @@ -47,14 +47,14 @@
Like other XDM-based technologies, XProc embeds and incorporates XPath, an expression language for XML.
XPath 3.0 is a functional language in its own right, although not designed for end-to-end processing of
encoded source data into encoded results, but only for certain critical operations that ordinarily need to
- be performed within such end-to-end processing. Since it is partial in this way, XPath is defined not in
- terms of any data notation (such as XML syntax or any other) but rather against an abstract data
- object, namely an XDM instance (XML data model),
- a putative information object that may be provided to the system by parsing an XML (syntax) instance, or by
- other means. As the query language for XDM and the
- basis for XQuery, XPath is the other half
of the data
- model, which any architect of a system using this technology must know. Learning XPath equips you mentally
- for dealing with the XDM in XQuery, XSLT, XProc or anywhere you find it.
other halfof the data model, which any + architect of a system using this technology must know. Learning XPath equips you mentally for dealing with + the XDM in XQuery, XSLT, XProc or anywhere you find it.
For those not already familiar with XPath, on line resources can be helpful. Keep in mind that XPath 3.1 outstrips XPath 1.0 in many important respects.
@@ -64,8 +64,8 @@unstructured, but that contrast with more common or usual structures of data formats, whether they be tabular data, serialization formats for - object models, or some other regular and fore-ordained arrangement. One might say
commonor -
usualwith reservation, since of course documents are not uncommon where they are common. + object models, or some other regular (formalized and codified) arrangement. One might say
common+ or
usualwith reservation, since of course documents are not uncommon where they are common.
We see a great deal of structured data these days if only because it is so easy to make structured data
with machines, and we now have the machines. What remains difficult is to translate what has not been
created by a machine, into a form that a machine can recognize
, or rather into a form we can
@@ -94,6 +94,9 @@
All elements defined by XProc are listed in this analytical breakout.
TODO - tbd - reformat this table for legibility (CSS grids); validate its completeness against XProc RNG?
+NB - XProc 3.1 makes this a moving target.
+NB also – see Erik Siegel's XProcRef indexing project for more + detailed summaries.
| Function | @@ -199,7 +202,8 @@|||||
|---|---|---|---|---|---|
| 1987 | SGML (ISO-IEC 8879-1) | -parsing logic; schema validation; configurable syntax; tree of elements and attributes | +parsing logic; schema validation; configurable syntax; (implicit) tree of elements and + attributes | Proprietary stacks | Mainframes, workstations |
| 2000 | - | + | XML-configured software builds | Apache Ant | Java |
The technologies we rely on share a common foundation in XML and XDM, technologies developed under the - auspices of the World Wide Web Consortium.
-They are commodity tools in the sense that they are freely available to use without restriction, an
- important qualification for this distribution, which has a prior commitment not to endorse particular
- technological solutions to any problem, however posed or circumscribed. Accordingly, solutions here
- are not offered as recommendations, but rather as stipulations of (minimum) viable functionality in tools or
- capabilities, and not only using tools as black boxes
, but under control and conformant to external
- specifications – i.e., standards.
The technologies we rely on share a common foundation in XML and XDM (the XML data model), technologies + developed under the auspices of the World Wide Web Consortium.
+These are supported by commodity tools that are freely available to use without restriction, an important
+ qualification for this distribution, which has a prior commitment not to endorse particular technological
+ solutions to any problem, however posed or circumscribed. Accordingly, solutions here are not offered
+ as recommendations, but rather as stipulations of (minimum) viable functionality in tools or capabilities,
+ and not only using tools as black boxes
, but under control and conformant to external specifications
+ – i.e., standards.
Users of these tools should keep in mind the model whereby we imagine the viability of a tools market and
- ecosystem that enables both large and small software developers including independent developers, academic
- researchers, and students, to participate meaningfully, finding an appropriate value or service proposition
- to provide their long-term goals. Translated, this means the tools must be capable enough for industrial use
- at scale, while also scale down
to demonstration or classroom use.
scale downto demonstration or classroom use.
In web standards including HTML and Javascript (ECMAScript) we arguably have the beginnings of such an
ecosystem, while it is also contested and turbulent. Within the publishing sector more broadly and
intersecting with the web, the XML family of standards arguably provides the best demonstration of complete
diff --git a/tutorial/src/lesson-html.sch b/tutorial/src/lesson-html.sch
index ad290d82..94121e7b 100644
--- a/tutorial/src/lesson-html.sch
+++ b/tutorial/src/lesson-html.sch
@@ -6,11 +6,12 @@
bash, the setup can be done by hand (downloading and unpacking a package from
SourceForge).
After installation, subsequent work on Windows does not require bash unless you choose to use
- it – a Windows CMD or Powershell can serve as your environment and the processor invoked with a
- Windows bat file (as described in the documentation). Mac and LInux (and WSL) users can
+ it – a Windows CMD or Powershell can serve as your environment and the processor invoked with a
+ Windows bat file (as described in the documentation). Mac and Linux (and WSL) users can
continue to use bash.
The setup script is a bash script: ./setup.sh, to be run with no arguments.
For XProc runtime -- to execute pipelines -- use either of the scripts ./xp3.sh (under bash) or ./xp3.bat (for Windows). These scripts are
+
For XProc runtime — to execute pipelines — use either of the scripts ./xp3.sh (under bash) or ./xp3.bat (for Windows). These scripts are
used for all pipelines (basically, for everything) unless otherwise noted.
The following pipelines will be run. They are described in top-level README documentation and the expected places.
Find setup instructions for the repository in the Project README and in the linked Setup Notes.
+Find setup instructions for the repository in the Project README and in the linked Setup Notes.
After reading and reviewing these documents, perform the setup as instructed. To do this you can either - fork or clone the repository in Github or simply download and decompress a zip of the current + fork or clone the repository in GitHub or simply download and decompress a zip of the current distribution.
After running the setup script, or performing the installation by hand, make sure you can run all the smoke tests successfully.
@@ -534,11 +546,11 @@If you want to run through the tutorial exercises but you are unsure of how deeply you will delve, you can postpone two of the installations until later:
When you see tracebacks suggesting one of these is not supported, you can return to setup.
-Since almost any pipeline will use XSLT and since we do use the latest version (XSLT 3.0/3.1), consider +
Since almost any pipeline will use XSLT and since we do use the latest version (XSLT 3.0 with XPath 3.1), consider the Saxon installation an essential requirement.
Effectively (and much more could be said about the processing stack, dependency management and so forth) - what this means is that XProc offers the user and the developer (in either or both roles) very focused and - concentrated points of control or points of adjustment. Out in the field - where software is deployed and - used - it may never be like that. User interfaces constrain what users can do, and even developers are - rarely free to just mess around, as it might be thought of.
-To the extent this is the case, this project only works if things are actually simple enough to pick up and - use.
+ what this means is that XProc offers the user and the developer (in either or both roles) with focused and + concentrated points of control or points of adjustment. In the field -- where software is deployed and + used -- things almost never justdrop in. User interfaces, APIs, dependencies and platform quirks: all these + constrain what users can do, and even developers are rarely free to just mess around, as it might be thought of. +
To the extent this is the case, this project only works if things are actually simple enough to pick up, + use, learn and adapt.
xp3.sh and xp3.bat represent attempts at this. Each of them (on its execution
platform) enables a user to run, without further configuration, the Morgana XProcIIIse processor on any XProc
@@ -580,54 +592,60 @@
As simple examples, these scripts show only one way of running XProc. Keep in mind that even simple scripts - can be used in more than one way.
-For example, a pipeline can be executed from the project root:
-$ ./xp3.sh smoketest/TEST-XPROC3.xpl-
Alternatively, a pipeline can be executed from its home directory, for example if currently in the
- smoketest directory (note the path to the script):
$ ../xp3.sh TEST-XPROC3.xpl-
This works the same ways on Windows, with adjustments:
-> ..\xp3 TEST-XPROC3.xpl-
(On Windows a bat file suffix marks it as executable and does not have to be given explicitly
- when calling.)
Windows users (and others to varying degrees) can set up a drag-and-drop based workflow - using your mouse - or pointer, select an XProc pipeline file and drag it to a shortcut for the executable (Windows batch file). - A command window opens to show the operation of the pipeline. See the README - for more information.
-It is important to try things out since any of these methods can be the basis of a workflow.
-For the big picture, keep in mind that while the command line is useful for development and demonstration – - and however familiar XProc itself may become to the developer – to the uninitiated it remains obscure and - cryptic. XProc-based systems, when integrated into tools or developer editors and environments, can look - much nicer than tracebacks in a console window. The beauty we are looking for here is in a different kind of - elegance and power.
+ project not to explore and map this space – which is vast – but to show off enough XProc and related logic + (XSLT, XSpec) to show how this exploration can be done. We are an outfitter at the beginning of what + we hope will be a long and profitable voyage. +As simple examples, these scripts show only one way of running XProc. Keep in mind that even simple + scripts can be used in more than one way.
+For example, a pipeline can be executed from the project root:
+$ ./xp3.sh smoketest/TEST-XPROC3.xpl+
Alternatively, a pipeline can be executed from its home directory, for example if currently in the
+ smoketest directory (note the path to the script):
$ ../xp3.sh TEST-XPROC3.xpl+
This works the same ways on Windows, with adjustments:
+> ..\xp3 TEST-XPROC3.xpl+
(On Windows a bat file suffix marks it as executable and does not have to be given
+ explicitly when calling.)
Windows users (and others to varying degrees) can set up a drag-and-drop based workflow - using your + mouse or pointer, select an XProc pipeline file and drag it to a shortcut for the executable (Windows + batch file). A command window opens to show the operation of the pipeline. See the README for more information.
+It is important to try things out since any of these methods can be the basis of a workflow.
+For the big picture, keep in mind that while the command line is useful for development and demonstration + – and however familiar XProc itself may become to the developer – to the uninitiated it remains obscure + and cryptic. XProc-based systems, when integrated into tools or developer editors and environments, can + look much nicer than tracebacks in a console window. The beauty we are looking for here is in a different + kind of elegance and power.
+-
Same as Setup 101.
+Same as Setup 101.
Please complete the repository setup and smoke tests as described in the 101 lesson. In this lesson, we will run these pipelines with adjustments, or similar pipelines.
-This discussion assumes basic knowledge of coding, the Internet (including retrieving resources via file and http protocols, and web-based technologies including HTML.
XML knowledge is also assumed, while at the same time we are interested to know where this gap needs to be - filled.
+Please complete the repository setup and smoke tests as described in the 101 lesson. In this lesson, we will run these pipelines with adjustments, or + similar pipelines.
+This discussion assumes basic knowledge of coding, the Internet (including retrieving resources via
+ file and http protocols), and web-based technologies including HTML.
XML knowledge is also assumed. In particular, XProc uses XPath + 3.1, the query language for XML. This latest version of XPath builds on XPath 1.0, so any XPath + experience will help. In general, any XSLT or XQuery experience will be invaluable.
You will also need a programmer's plain text editor, XML/XSLT editor or IDE (integrated development environment) for more interactive testing of the code.
The two groupings of pipelines used in setup and testing can be considered separately.
The key to understanding both groups is to know that once the initial Setup script is run, Morgana can be invoked directly, as paths and scripts are already in place. In doing - so – before extension libraries are in place – it can use only basic XProc steps, but those are enough for + so – before extension libraries are in place – it can use only basic XProc steps, but those are enough for these purposes.
Specifically, the pipelines can acquire resources from the Internet, save them locally, and perform unarchiving (unzipping). Having been downloaded, each library provides software that the pipeline engine @@ -659,15 +677,15 @@
Take a look at these. If it is helpful, try visualize the XML syntax as a set of nested frames with labels - and connectors.
+Take a look at these. If it is helpful, try to see the XML syntax as a set of nested frames with labels and + connectors.
Try more than one way of looking at the XProc source code: in the Github repository, on your file system, in a plain text editor, in an XML editor.
Use a text editor or IDE for this exercise.
If you have any concepts for improvements to the pipelines, or other resources that might be acquired this way, copy and modify one of the pipelines given to achieve those results.
-Even if not - be sure to break the pipelines given -- or of copies under new names -- in any of several - ways. Then run the modified pipelines, as a safe way to familiarize yourself with error messages:
+Even if not - be sure to break the pipelines given -- or copies under new names -- in any of several ways. + Then run the modified pipelines, as a safe way to familiarize yourself with error messages:
This larger fitting of solutions to problems is a responsibility for both CMEs (content matter experts) and +
This larger fitting of solutions to problems is a responsibility for both SMEs (subject matter experts) and developers together, who must define problems to be solved before approaches to them can be found.
The open questions are: who can use XProc pipelines; and how can they be made more useful? The questions come up in an OSCAL context or any context where XML is demonstrably capable.
@@ -710,8 +728,8 @@Because it is now centered on pipelines as much as on software packages, dependency management is - different from other technologies including Java and NodeJS - how so?
+Because it is now centered on pipelines as much as on files and software packages, dependency + management is different from other technologies including Java and NodeJS - how so?
MorganaXProc-III is implemented in Scala, and Saxon is built in Java, but otherwise distributions including the SchXSLT and XSpec distributions consist mainly of XSLT. This is either very good (with development and maintenance requirements in view), or not good at all.
@@ -783,12 +801,12 @@Hello World. In that spirit, feel free to edit and adjust this file.
You have done Setup 101, Setup 102 and Unpack 101.
+ class="LessonUnit">Setup 102 and Unpack 101.XProc3or
XProc 3.0to help distinguish from 1.0 - technologies
XProc3or
XProc 3.0to help distinguish from 1.0 technologies
Additionally, step declarations can include their own pipeline (step) declarations, making a hybrid architecture: the pipeline comprises a step, with its own library not imported but in line. This can be useful.
-An example of a step library in this repository is xpec-execute.xpl, which collects several steps supporting XSpec, one each for supporting the +
An example of a step library in this repository is xspec-execute.xpl, which collects several steps supporting XSpec, one each for supporting the XSpec testing framework for XSLT, XQuery and Schematron respectively.
The advantage of defining a step at the top level, rather than putting all steps into libraries, is that such a step can be invoked without prior knowledge of its type name, which is used by XProc to @@ -954,8 +972,8 @@
An example of this is the XProc p:documentation element. This element is designed to
carry documentation to a consuming application. Rather than mandate some kind of behavior for
- p:documentation – something difficult or impossible to do for the general case, or to
- test –- the XProc rule is anything marked as documentation is for some other consumer
, i.e. a
+ p:documentation – something difficult or impossible to do for the general case, or to
+ test –- the XProc rule is anything marked as documentation is for some other consumer
, i.e. a
documentation engine, not the XProc processor. In other words, a conformant processor
must ignore anything it sees
inside p:documentation.
Keep in mind that every XProc pipeline is also, potentially and actually, a step. There are two things we need to know about steps - how to define them, and how to use them.
-We begin with how to recognize and use steps, but we can't avoid how to define them: because an XProc - pipeline is also an XProc step, we can't use steps without ending up with a pipeline. We have only to - look at the working pipeline we make with our steps, to see how a step is made.
+We begin with how to recognize and use steps. But because an XProc pipeline is also an XProc step, we + can't use steps without ending up with a pipeline. We have only to look at the working pipeline we make + with our steps, to see how a step is made.
As described in the XProc 3.0 specification, XProc step declarations can be divided into an initial set of elements for setup - and configuration, followed by what the specification calls a subpipeline, which is typically a - sequence of steps to be executed – any steps available, which could be anything. Think of the subpipeline + and configuration, followed by what the specification calls a subpipeline, consisting of a + sequence of steps to be executed – any steps available, which could be anything. Think of the subpipeline as the working parts of the pipeline, while the rest is all about how it is set up.
-The list of elements that come before the steps is short, which helps: p:import,
+
The list of elements that come before the subpipeline is short, which helps: p:import,
p:import-functions, p:input, p:output, p:option or
p:declare-step. Everything coming after is a step.
Within this set of elements (all preceding, none following the subpipeline) XProc further distinguishes
between the imports for steps and functions, appearing first (elements p:import and
p:import-functions), to be followed by elements configuring the step:
- p:input, p:output, p:option – elements together called the prologue.
p:input, p:output, p:option – elements together called the prologue.
The prologue is used to define ports and options for the pipeline - the points of control for its
interfaces. (Technically: runtime bindings, and parameter or option settings.) If only a single input is
needed, a single input port (named source) can be assumed, so prologues can be empty (and
invisible, or not there).
Following the prologue, a step may also have local step definitions (p:declare-step).
After imports, prologue and (optional) step declarations, the step sequence that follows comprises the subpipeline.
-One other complication: for purposes of exposition, we have pretended that p:variable (a
- variable declaration) and p:documentation (for out-of-band documentation) are steps, which
- is not how the XProc recommendation describes them, although they appears in the subpipeline with and
- among its steps.
One other complication: among the steps in the subpipeline, p:variable (a variable
+ declaration) and p:documentation (for out-of-band documentation) are also permitted – these
+ are not properly steps, but can be useful to have with them.
In summary: any XProc pipeline, viewed as a step declaration, can have the following --
p:declare-step. Each of these will have its own name, type, prologue and stepsp:)Given an understanding of the organization of an XProc pipeline, all that remains to understand of its
syntax is the steps themselves, which follow a common pattern. Briefly put, atomic steps are any steps
you use by simply invoking it with inputs and options: its logic is self-contained, and the operation it
- carries out is (at least conceptually) single
. Compound steps, instead, are used to execute more
+ carries out is (at least conceptually) single
. Compound steps, instead, are used to deploy more
than one subpipeline, with settings determined dynamically for the step.
Fortunately XProc keeps things simple by providing only a few compound steps supporting the identified - range of needs – and no way for users to define their own. This does not prove to be a practical + range of needs – and no way for users to define their own. This does not prove to be a practical limitation, since atomic steps can have multiple inputs and outputs, distinguished by type and role, and indeed since atomic steps used in a pipeline can be defined with compound steps in their own subpipelines, either externally or even within the same step declaration.
@@ -1102,7 +1119,8 @@See the 599-level coverage in this lesson unit for a discussion of the +
See the 599-level coverage in this lesson unit for a discussion of the schema for XProc.
The last lesson unit already provided an opportunity to alter pipelines and see how they fail when not
- encoded correctly – when broken
, any way we can think of breaking them. (Then we had an opportunity
+ encoded correctly – when broken
, any way we can think of breaking them. (Then we had an opportunity
to put them back.)
After reading this page, do this again, except focusing on a few key areas:
It may often be considered gratuitous to validate XProc files against a schema, when the application (for - us, Morgana)must in any case take responsibility for conformance issues, as it sees fit. The reference + us, Morgana) must in any case take responsibility for conformance issues, as it sees fit. The reference schema becomes useful if we find or suspect bugs in Morgana, but until then it need not have any direct role in any runtime operation.
Nevertheless the RNG schema still serves as a reference and an object for querying -- queries whose results @@ -1231,14 +1249,12 @@
Like other XDM-based technologies, XProc embeds and incorporates XPath, an expression language for XML.
XPath 3.0 is a functional language in its own right, although not designed for end-to-end processing of
encoded source data into encoded results, but only for certain critical operations that ordinarily need to
- be performed within such end-to-end processing. Since it is partial in this way, XPath is defined not in
- terms of any data notation (such as XML syntax or any other) but rather against an abstract data
- object, namely an XDM instance (XML data model),
- a putative information object that may be provided to the system by parsing an XML (syntax) instance, or by
- other means. As the query language for XDM and the
- basis for XQuery, XPath is the other half
of the data
- model, which any architect of a system using this technology must know. Learning XPath equips you mentally
- for dealing with the XDM in XQuery, XSLT, XProc or anywhere you find it.
other halfof the data model, which any + architect of a system using this technology must know. Learning XPath equips you mentally for dealing with + the XDM in XQuery, XSLT, XProc or anywhere you find it.
For those not already familiar with XPath, on line resources can be helpful. Keep in mind that XPath 3.1 outstrips XPath 1.0 in many important respects.
unstructured, but that contrast with more common or usual structures of data formats, whether they be tabular data, serialization formats for - object models, or some other regular and fore-ordained arrangement. One might say
commonor -
usualwith reservation, since of course documents are not uncommon where they are common. + object models, or some other regular (formalized and codified) arrangement. One might say
common+ or
usualwith reservation, since of course documents are not uncommon where they are common.
We see a great deal of structured data these days if only because it is so easy to make structured data
with machines, and we now have the machines. What remains difficult is to translate what has not been
created by a machine, into a form that a machine can recognize
, or rather into a form we can
@@ -1259,7 +1275,7 @@
And this is to give no consideration to the fact that these structures can be described at multiple levels of generality or specificity with regard to either their supposed semantics, or their configuration in operation.
@@ -1268,7 +1284,7 @@A definition for what is a document is out of scope for this tutorial – an interesting topic but not only a +
A definition for what is a document is out of scope for this tutorial – an interesting topic but not only a technical one.
All elements defined by XProc are listed in this analytical breakout.
TODO - tbd - reformat this table for legibility (CSS grids); validate its completeness against XProc RNG?
+NB - XProc 3.1 makes this a moving target.
+NB also – see Erik Siegel's XProcRef indexing project for more + detailed summaries.
| 1987 | SGML (ISO-IEC 8879-1) | -parsing logic; schema validation; configurable syntax; tree of elements and attributes | +parsing logic; schema validation; configurable syntax; (implicit) tree of elements and + attributes | Proprietary stacks | Mainframes, workstations | 2000 | - | - | +XML-configured software builds | Apache Ant | Java | @@ -1705,25 +1724,25 @@
The technologies we rely on share a common foundation in XML and XDM, technologies developed under the - auspices of the World Wide Web Consortium.
-They are commodity tools in the sense that they are freely available to use without restriction, an
- important qualification for this distribution, which has a prior commitment not to endorse particular
- technological solutions to any problem, however posed or circumscribed. Accordingly, solutions here
- are not offered as recommendations, but rather as stipulations of (minimum) viable functionality in tools or
- capabilities, and not only using tools as black boxes
, but under control and conformant to external
- specifications – i.e., standards.
The technologies we rely on share a common foundation in XML and XDM (the XML data model), technologies + developed under the auspices of the World Wide Web Consortium.
+These are supported by commodity tools that are freely available to use without restriction, an important
+ qualification for this distribution, which has a prior commitment not to endorse particular technological
+ solutions to any problem, however posed or circumscribed. Accordingly, solutions here are not offered
+ as recommendations, but rather as stipulations of (minimum) viable functionality in tools or capabilities,
+ and not only using tools as black boxes
, but under control and conformant to external specifications
+ – i.e., standards.
Users of these tools should keep in mind the model whereby we imagine the viability of a tools market and
- ecosystem that enables both large and small software developers including independent developers, academic
- researchers, and students, to participate meaningfully, finding an appropriate value or service proposition
- to provide their long-term goals. Translated, this means the tools must be capable enough for industrial use
- at scale, while also scale down
to demonstration or classroom use.
scale downto demonstration or classroom use.
In web standards including HTML and Javascript (ECMAScript) we arguably have the beginnings of such an ecosystem, while it is also contested and turbulent. Within the publishing sector more broadly and intersecting with the web, the XML family of standards arguably provides the best demonstration of complete @@ -1773,24 +1792,34 @@
This unit relies on the oscal-convert project in this - repository, with its files. Like all projects in the repo, it is designed to be self-contained and - self-explanatory to the extent practical. Use your search engine and XProc resources to learn background and - terminology.
+This unit relies on the oscal-convert project in + this repository, with its files. Like all projects in the repo, it aims to be reasonably self-contained and + self-explanatory. Use your search engine and XProc resources to learn background and terminology.
Also like other projects, there are preliminaries for acquiring resources, along with pipelines to run.
An acquisition pipeline in the project folder collects some OSCAL onto the local system, where it can be managed, easily inspected, controlled, and edited if necessary.
-But sources and results from a conversion process do not always have to work with local resources. XProc
- sees the Internet itself - whatever protocols are supported by the processor (the file and
- http protocols are required for conformance) work as they do on the Worldwide Web. Of
- course, permissions must be in place to read and write files from and into file system locations. But when
- authentication is configured or resources are openly available, using http to reach resources
- or sources can be a very convenient option.
TBD / this all incoherent so far
+Keep in mind that XProc in theory, and your XProc engine in practice, may read its inputs using whatever
+ protocols it supports, while the file and http protocols are required for
+ conformance, and work as they do on the Worldwide Web. Of course, permissions must be in place to read
+ and write files from and into file system locations. But when authentication is configured or resources
+ are openly available, using http to reach resources or sources can be a very convenient
+ option.
Keep in mind that where resources take the form of serializations of structured data such as XML or JSON,
+ what is read
(in the sense of presented to a consuming application) may not be quite exactly the
+ same data serialization, since as a serialization, it is parsed into an object or representation of some
+ kind,
TBD - TODO - question - how many and of what sort of source data files - so far there is only the cat catalog
In later units we will see how using the XProc steps described, rudimentary data manipulations can be done using XProc by itself, without entailing the use of either XSLT or XQuery (another capability invoked with a different step). At the same time, while pipelines are based on the idea of passing data through a series of - processes, there are many cases where logic is sufficiently complex that it becomes essential to maintain – - and test – that logic externally from the XProc. At what point it becomes more efficient to encapsulate + processes, there are many cases where logic is sufficiently complex that it becomes essential to maintain – + and test – that logic externally from the XProc. At what point it becomes more efficient to encapsulate logic separately (whether by XSLT, XQuery or other means), depends very much on the case.
The p:xslt pipeline step in particular is so important for real-world uses of XProc that it is
introduced early, to show such black-box application. XProc also makes a fine environment for testing XSLT
@@ -1852,7 +1881,8 @@
In the publishing demonstration project folder is an XProc that valides XML against an OSCAL schema, before formatting it. The same could be done for an XProc that converts the data into JSON - either or both before or after conversion.
-Learn more about recognizing and dealing with errors in Lesson
+ Learn more about recognizing and dealing with errors in Lesson
102, or continue on to the next project, oscal-validate, for more on validation of documents and
sets of documents.
Same as the 101 lesson.
+Same as the 101 lesson.
XML comment syntax takes the form of angle-bracketed text with dashes (hyphens) and a bang: XML comment syntax takes the form of angle-bracketed text with dashes (hyphens) and a bang: The intrepid explorer will try both tweaking, around the edges, then adapting, with new
ideas. The best of it is, as you learn how a traceback is a tasty treat, you enter a virtuous learning
- cycle. Repair a few bugs and you might get a taste for it. (Try the 102 Lesson unit )<!–
+ <!–
-->. You will see comments throughout the code examples, sometimes as inline documentation, and
sometimes for entire blocks of inactive code.
If you'd rather keep your feet dry, advance to the next Lesson.
Such a rules set is called a schema
and deploying a schema to impose regularity and predictability
- over an open-ended body of data – including data that has not yet been created – is a primary responsibility
+ over an open-ended body of data – including data that has not yet been created – is a primary responsibility
of an entity that seeks to define an interoperable language, supporting robust data interchange across
organizational boundaries.
Run the pipelines described in the 101 Lesson +
Run the pipelines described in the 101 Lesson
Like the 101 lesson, this lesson unit uses the oscal-import project in this repo.
+Like the 101 lesson, this lesson unit uses the oscal-import project in this repo.
Publishing your own OSCAL in these pipelines is also worth trying - the worst they can do is fail. Bring OSCAL, especially OSCAL catalogs.
OSCAL JSON catalogs can be accommodated by splicing together a conversion pipeline with a publishing @@ -2267,7 +2301,8 @@
As usual, the project contains pipelines to be used to acquire copies of resources, placing them in the - project lib directory . These may include dependencies for steps in the pipeline (read details in the 102 lesson) so it is a good idea to run them all.
+ project lib directory . These may include dependencies for steps in the pipeline (read details in the 102 lesson) so it is a good idea to run them all.These include a pipeline that copies the PDF original from which the HTML input (next step) was derived,
using a COTS (commercial off-the-shelf) tool. Inspect and read this PDF to see what the upstream data looks
like - and get a sense of what the data set looks like when the structures are painted
on the
@@ -2277,41 +2312,62 @@
For this exercise, we have already produced HTML from the PDF source: find it cached as an input file in the project folder. Next to it is a copy with whitespace added for legibility, which may be easier to inspect.
-
- Do not delete, replace, rename or alter this file, as all the subsequent processing is tuned
- specifically to it, its internal organization and regularities and anomalies. Another process input - even
- an HTML file showing the same text
- is highly unlikely to deliver acceptable results from the
- pipeline. This is an important factor to keep in mind when considering the scalability of the exercise here.
- It does not require less expertise than producing the original document (in some respects it requires
- more).
For some data sets, however, and especially if the exercise can also be used advantageously in other ways - - that is, beyond its considerabl eimpacts on the availability and utility of the information - even a bespoke - transformation pipeline such as this can be useful. This occurs mainly when the source document is of such - high quality and consisteny instrinsically, that exposing its semantics via a rational and explicable text - encoding - a standards-based, platform- and application-independent encoding - such as OSCAL - becomes - attractive for the leverage it gives over the information. In FM 6-22 Chapter 4, we have exactly such - a document, which fits well into standards-based documentary formats such as NISO STS and, eventually, - OSCAL.
-Because PDF is relatively opaque, extracting data from a PDF file with format intact is not a trivial - exercise. Even tools designed, sold and supported by the proprietors of PDF format (Adobe, Inc.) are able at - best to create outputs that appear to present the document with its organization and information - intact, if rendered on screen with an appropriate tool (in the case of HTML, a web browser). In so doing it - may meet a low baseline of functional requirement - the document is legible - while failing to meet others, - and in particular the requirements we have for clean and well-structured data fit for reuse in a broad range - of scenarios and applications, not only page production.
-Indeed this HTML file makes an excellent example of the kind of rather-poor encoding produced by a - market-leading HTML export utility. Experts in data conversion might find other ways to make better encoding - from the PDF - a task out of scope for us. (As we are not trying to optimize but only to demonstrate the - methods applied.) This HTML is good enough, and that is what is often available.
-The pipeline to be run in the next step starts by loading source/export/fm6_22.html - do not - rename, move or alter this file, or expect errors!
+Compare this to the PDF document you have downloaded (or the document on line). Open the HTML file in a web + browser to see what it looks like in display, but also inspect its source code. You don't have to read it: + just seeing it is enough.
+
+ Do not delete, replace, rename or alter this file, as all the subsequent processing is tuned
+ specifically to it, its internal organization, regularities and anomalies. Another process input - even
+ an HTML file showing the same text (to any definition of same
) - is highly unlikely to deliver
+ acceptable results from the pipeline. This is an important factor to keep in mind when considering the
+ scalability of the operation demonstrated. Producing a clean and legible encoding for the document
+ does not require less expertise than producing the original; in some ways (for example with respect to
+ knowledge of text encoding formats) it requires more.
For some data sets, however, even a bespoke transformation pipeline such as this can be profitable, even
+ beyond its impacts in greater availability and utility of the information. The expense and effort may be
+ warranted when the source document is of such high quality and consistency instrinsically, that exposing
+ its native semantics
via a rational and explicable text encoding - a standards-based, platform-
+ and application-independent encoding - such as OSCAL - becomes attractive for the specific kinds of
+ affordances – the leverage – it gives for processing the information. In FM 6-22 Chapter 4, we have
+ exactly such a document, which fits well into standards-based documentary formats such as NISO STS and,
+ eventually, OSCAL.
Because PDF is relatively opaque, extracting data from a PDF file with format intact is not a trivial
+ exercise. PDF is arguably not a data format
at all, but a language used to control a device for
+ casting pixels and glyphs onto a page or screen. This being its purpose, no tool can do more – at least
+ without the benefit of an external information source – than create outputs that appear to present
+ the document with its organization and information intact, if rendered on screen with an appropriate tool
+ (such as, in the case of an HTML rendition of the page, a web browser). In so doing it may meet a low
+ baseline of functional requirement - the document is legible - while failing to meet others. In
+ particular: failing to meet requirements for clean and well-structured data fit for reuse in a broad
+ range of scenarios and applications, not only page production.
Indeed this HTML file makes an excellent example of the kind of rather-poor encoding produced by a + market-leading HTML export utility. Experts in data conversion might find other ways to make better + encoding from the PDF - a task out of scope for us. (As we are not trying to find the best export + utility, but on the contrary to demonstrate methods available so we do not have to do so.) This HTML is + good enough, and that is what is often available.
+The pipeline starts by loading source/export/fm6_22.html +
+It produces output files in a temp directory. If you want to save out any of these outputs, you
+ can copy files out, or edit the pipeline.
Two files are created, both representing the Field Manual, each in a different format:
+semanticview of the data + in OSCAL format, again presenting the full text of Chapter 4, but this time with externalized control + sets, representing 25 attributes and 50 competencies as these are described in Field Manual + 6-22 and its related documents such as ADRP 6-22 (describing the US Army Leadership Requirements + Model).
Run the pipelines described in the 101 Lesson +
Run the pipelines described in the 101 Lesson
Like the 101 lesson, this lesson unit uses the oscal-import project in this repo.
+Like the 101 lesson, this lesson unit uses the oscal-import project in this repo.