sync doc with wiki in preparation for the 1000th commit

HouzuoGuo · Feb 26, 2021 · dcda5d7 · dcda5d7
1 parent 04d3bfb
commit dcda5d7
Show file tree

Hide file tree

Showing 29 changed files with 618 additions and 110 deletions.
diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 <img src="https://raw.githubusercontent.com/HouzuoGuo/laitos/master/doc/cosmetic/20200825-poster.png" alt="poster image" />
 
-# laitos - Top geek's software kit
+# laitos - Top geek's chindogu
 
 **laitos** software suite offers all you need for hosting a personal website, receiving Emails, blocking ads with a DNS server.
 And for the geeks 🤓 - as a professional geek, you need Internet access whenever and wherever!
@@ -34,7 +34,7 @@ I'd love to hear from your feedback, feel free to [Email me](mailto:guohouzuo@gm
 and visit my home page [hz.gl](https://hz.gl).
 
 ## Copyright
-Copyright (C) 2016-2020 Houzuo (Howard) Guo <[email protected]>
+Copyright (C) 2016-2021 Houzuo (Howard) Guo <[email protected]>
 
 This program is free software subject to the terms of Mozilla Public License, v. 2.0. You may find the license text in
 [the LICENSE file](https://github.com/HouzuoGuo/laitos/blob/master/LICENSE)

diff --git a/doc/Cloud-tips.md b/doc/Cloud-tips.md
@@ -53,8 +53,8 @@ right away. All flavours of Linux distributions that run on EC2 can run laitos.
 Lambda is the flagship Function-as-a-Service product offered by AWS. As a serverless offering, the product does not support
 general purpose computing. For laitos, lambda can run its web server without having to prepare an EC2 instance manually.
 
-Lambda offers pre-built runtime environment (including Go) with built-in web server. Because laitos runs its own web server,
-we will build a "lambda custom runtime", start with creating a shell script and name it `bootstrap`:
+Even though lambda offers pre-built runtime environment (including Go) with built-in web server, laitos will not be using it
+because laitos runs its own web server, we will build a "lambda custom runtime" by first creating a script and name it `bootstrap`:
 
     #!/bin/sh
     ./laitos -config config.json -daemons insecurehttpd -awslambda
@@ -142,6 +142,64 @@ Here are some tips for using laitos on ElasticBeanstalk:
 
 - Remember to adjust firewall (security group) to open ports for all services (e.g. DNS, SMTP, HTTPS) served by laitos.
 
+## Integrate with AWS Kinesis Firehose, S3, SQS, and SNS
+Beyond using AWS as the computing foundation for running laitos server, it may also integrate with the following AWS products:
+- When a program component emits a warning log message, send the message to SQS (simple queue service).
+- Upon receiving a [phone home telemetry](https://github.com/HouzuoGuo/laitos/wiki/%5BApp%5D-phone-home-telemetry-handler) report,
+  send the report to Kinesis Firehose.
+- Upon receiving a [phone home telemetry](https://github.com/HouzuoGuo/laitos/wiki/%5BApp%5D-phone-home-telemetry-handler) report,
+  send the report to SNS (simple notification service).
+- After completing a round of [system maintenance](https://github.com/HouzuoGuo/laitos/wiki/%5BDaemon%5D-system-maintenance),
+  upload the report of results to S3.
+- Use x-ray to trace HTTP requests handled by the laitos web servers.
+
+The points of integration are entirely optional, they may be individually enabled as needed. In order to enable any of them, you have
+to tweak the laitos configuration, program environment and launch command in these ways:
+
+- Set environment variable `AWS_REGION` to the API name of AWS region in which where all of the involved AWS resources are located.
+  laitos program assumes that the SQS queue, Kinesis Firehose stream, SNS topic, and S3 bucket are located in the same region.
+- Add CLI parameter `-awsinteg` to the command line. The parameter acts as a master switch to turn on/off all AWS integration features.
+- Supply AWS access key in one of the several ways:
+  * Via Lambda execution role, ECS task role, or EC2 instance role (also called "instance profile").
+  * Via environment variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and the optional `AWS_SESSION_TOKEN`.
+  * Via environment variable `AWS_PROFILE` and the credentials file from user's home directory.
+- Tweak the laitos program configuration file to mention the ID or names of the invovled AWS resources.
+
+For example, if the command line for launching laitos was:
+
+    ./laitos -config my-config.json -daemons httpd,maintenance,phonehome
+
+Then change it to (the example specifies Ireland as the region):
+
+    env AWS_REGION=eu-west-1 ./laitos -awsinteg -config my-config.json -daemons httpd,maintenance,phonehome
+
+And tell laitos the AWS resource names/IDs in its program configuration:
+
+```
+{
+    "AWSIntegration": {
+        "ForwardMessageProcessorReportsToFirehoseStreamName": "laitos-subject-reports-stream-name",
+        "ForwardMessageProcessorReportsToSNSTopicARN": "arn:aws:sns:eu-west-1:123484198765:laitos-subject-report-topic",
+        "SendWarningLogToSQSURL": "https://sqs.eu-west-1.amazonaws.com/123484198765/laitos-warnings-queue"
+    },
+    ...
+    "Maintenance": {
+        ...
+        "UploadReportToS3Bucket": "laitos-maintenance-report-bucket",
+        ...
+    },
+    ...
+}
+```
+
+Be aware that it is often a bad idea to keep AWS access key in a program configuration file, therefore the configuration file
+of laitos does not have AWS access keys. For detailed instructions on how to supply the access key, check out AWS documentation ["Configuring the AWS SDK for Go"](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials)
+
+To use x-ray for tracing HTTP requests, turn on the AWS integration master switch with CLI parameter `-awsinteg`, and then install
+the AWS x-ray daemon program on the server host by following [AWS X-Ray Daemon guide](https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon.html).
+
+All interactions between laitos and AWS generate info-level log messages for diagnosis and inspection.
+
 ## Deploy on Microsoft Azure and Google Compute Engine
 Simply copy laitos program and its data onto a Linux virtual machine and start laitos right away. It is often useful to
 use systemd integration to launch laitos automatically upon system boot. All flavours of Linux distributions supported
@@ -153,3 +211,4 @@ instance, you can be almost certain that it will run laitos smoothly and well.
 
 Beyond AWS, Azure, and GCE, the author of laitos has also successfully deployed it on generic KVM virtual machine,
 OpenStack, Linode, and several cheap hosting services advertised on [lowendbox.com](https://lowendbox.com/).
+
diff --git a/doc/Command-processor.md b/doc/Command-processor.md
@@ -43,7 +43,8 @@ Mandatory `PINAndShortcuts` - define access password and shortcut command entrie
     <td>Passwords</td>
     <td>array of strings</td>
     <td>
-        Put any of these passwords at the very beginning of app command input to gain access to apps.
+        Put any of the passwords at the very beginning of app command input to gain access to apps.
+        A password consists of letters and numbers. Avoid using spaces and special symbols.
         <br/>
         See "Usage" for more information.
     </td>
@@ -189,6 +190,7 @@ App command looks like:
     Password .app_identifier parameter1 parameter2 parameter3 ...
 
 Where:
+- `Password` is one of the passwords from the `Passwords` array of valid password strings.
 - `.app_identifier` is a short text string that identifies the app to invoke. Pay attention to the mandatory leading `.` dot.
 - Parameters are passed as-is to the specified app as its input.
 
@@ -232,16 +234,27 @@ OTP 1 is `123123` and OTP 2 is `789789`:
   will refuse to execute the second "echo hi" command by saying "the TOTP has already been used with a different command".
 
 ### Override output length and timeout restriction
-Prepend the lower case string "plt" and three parameters to an app command, to position (skip) first N characters from the command output,
-override the output length restriction, and/or override the command execution timeout:
+By default, daemons that are capable of receiving app commands, executing them, and respond with execution result will impose several
+restrictions on:
+- Maximum length of the app command input.
+- Maximum length of the app command execution result (output).
+- Maximum duration (in seconds) that the app command execution may take.
 
-    .plt <SKIP> <MAX LENGTH> <TIMEOUT SECONDS> Password .app_identifier parameter1 parameter 2 parameter 3 ...
+The restrictions are often configurable, and they are crucial in preventing a malfunctioning app command from exhausting server resources.
+However, in certain cases, you may wish to temporarily lift the restriction for an individual app command, for example when executing an
+app command over SMS the response is usually restricted to ~130 characters (max. 1 SMS), and by using the special `.plt` string, you may
+then read an a response 300 characters long (in 3 texts).
+
+An app command that uses `.plt` string looks like:
+
+    PasswordPIN .plt SKIP MaxLength TimeoutSeconds .app_identifier parameter1 parameter 2 parameter 3 ...
 
 Where:
-- `<SKIP>` is the number of characters to discard from beginning of the result output.
-- `<MAX LENGTH>` is the maximum number of characters to collect from command response. It overrides `MaxLength` of `LintText` as well as
-  certain daemons' internal default limit.
-- `<TIMEOUT SECONDS>` is the maximum number of seconds the app may spend to execute the command. It overrides daemon's internal default limit.
+- `PasswordPIN` is the password PIN to authorise app command execution.
+- `SKIP` is the number of characters to discard from beginning of the result output.
+- `MaxLength` is the maximum number of characters to collect from command response. It overrides `MaxLength` of `LintText`, or the default
+  limit imposed by a daemon internally.
+- `TimeoutSeconds` is the maximum number of seconds the app may spend to execute the command. It overrides daemon's internal default limit.
 
 Take an example - a user uses the Telegram bot daemon to execute command `mypassword .il work-mail 0 10` (get the latest 10 Email subjects).
 The user previously configured `LintText` to restrict output to only 76 characters, and Telegram bot internally spends at most 30 seconds to
@@ -251,9 +264,10 @@ execute a command. These constraints would result in this incomplete response:
     2 [email protected] Holiday greetings
     3
 
-Let us try to retrieve the full output - skip the 2 Email subjects already seen, override `MaxLength` to 10000 and timeout to 60 seconds:
+Let us try to retrieve the full output - skip the 2 Email subjects already seen (the first 75 characters), then override maximum output length
+to 10000 and timeout to 60 seconds:
 
-    .plt 75 10000 60 mypassword .il work-mail 0 10
+    mypassword .plt 75 10000 60 .il work-mail 0 10
 
 And we will get the desirable result:
 
@@ -268,13 +282,12 @@ And we will get the desirable result:
 
 ## Tips
 Regarding password:
-- It must be at least 7 characters long.
-- Do not use space character in the password, or it might not be validated successfully during a command invocation.
-- Use a strong password that is hard to guess.
+- It must be at least 7 characters long. Use a strong password that is hard to guess.
+- Feel free to use numbers, upper and lower case letters, but please refrain from using other special characters or space characters in the password.
 - All daemons capable of invoking app commands offer rate limit mechanism to reduce impact of brute-force password guessing. Pay special
   attention to the rate limit settings in individual daemon configuration.
-- For prevention of brute-force guessing of password via DDoS, each laitos daemon will execute a maximum of 1000 commands per second, regardless
-  of their rate limit configuration.
+- For prevention of brute-force guessing of password via DDoS, laitos enforces a global limitation of maximum 1000 command executions per second,
+  regardless of how many daemons are processing the app commands.
 - Incorrect password PIN entry does not result in an Email notification, however, the attempts are logged in warnings and can be inspected via
   [environment inspection](https://github.com/HouzuoGuo/laitos/wiki/%5BApp%5D-inspect-and-control-server-environment)
   or [program health report](https://github.com/HouzuoGuo/laitos/wiki/%5BWeb-service%5D-program-health-report).

diff --git a/doc/Component-list.md b/doc/Component-list.md
@@ -31,6 +31,11 @@ the apps.
         <td>Web server hosts a static personal website made of text and media files, along with rich web services (see below).</td>
         <td><a href="https://github.com/HouzuoGuo/laitos/wiki/%5BDaemon%5D-web-server" target="_blank">Link</a></td>
     </tr>
+    <tr>
+        <td>Web proxy server</td>
+        <td>The general purpose web proxy server is capable of handling both HTTP and HTTPS destinations.</td>
+        <td><a href="https://github.com/HouzuoGuo/laitos/wiki/%5BDaemon%5D-web-proxy" target="_blank">Link</a></td>
+    </tr>
     <tr>
         <td>Telnet server</td>
         <td>Telnet server provides unencrypted access to all apps via basic tools such HyperTerminal.</td>
@@ -87,6 +92,11 @@ The following services are hosted by web server and enabled on your demand:
         <td>Run app commands on Skype and Cortana via Microsoft Bot Framework.</td>
         <td><a href="https://github.com/HouzuoGuo/laitos/wiki/%5BWeb-service%5D-Microsoft-bot-hook" target="_blank">Link</a></td>
     </tr>
+    <tr>
+        <td>The Things Network LORA tracker integration</td>
+        <td>Collect location telemetry from your LoRa IoT devices that run The Things Network Mapper program.</td>
+        <td><a href="https://github.com/HouzuoGuo/laitos/wiki/%5BWeb-service%5D-the-things-network-LORA-tracker-integration" target="_blank">Link</a></td>
+    </tr>
     <tr>
         <td>Recurring commands</td>
         <td>Run app commands at regular interval, and retrieve their result.</td>
@@ -132,20 +142,30 @@ The following services are hosted by web server and enabled on your demand:
         <td>Present you with a fully functional computer desktop running on laitos server as a virtual machine.</td>
         <td><a href="https://github.com/HouzuoGuo/laitos/wiki/%5BWeb-service%5D-desktop-on-a-page-(virtual-machine)" target="_blank">Link</a></td>
     </tr>
+    <tr>
+        <td>Read telemetry records</td>
+        <td>Read phone-home telemetry records collected by this server.</td>
+        <td><a href="https://github.com/HouzuoGuo/laitos/wiki/%5BWeb-service%5D-read-telemetry-records" target="_blank">Link</a></td>
+    </tr>
     <tr>
         <td>Program health report</td>
         <td>Display program stats, log entries, and system resource usage in a comprehensive report.</td>
         <td><a href="https://github.com/HouzuoGuo/laitos/wiki/%5BWeb-service%5D-program-health-report" target="_blank">Link</a></td>
     </tr>
     <tr>
-        <td>Read telemetry records</td>
-        <td>Read phone-home telemetry records collected by this server.</td>
-        <td><a href="https://github.com/HouzuoGuo/laitos/wiki/%5BWeb-service%5D-read-telemetry-records" target="_blank">Link</a></td>
+        <td>System process explorer</td>
+        <td>Find all processes running on the host OS and inspect the status and resource usage of individual process.</td>
+        <td><a href="https://github.com/HouzuoGuo/laitos/wiki/%5BWeb-service%5D-system-process-explorer" target="_blank">Link</a></td>
     </tr>
     <tr>
-        <td>The Things Network LORA tracker integration</td>
-        <td>Collect location telemetry from your LoRa IoT devices that run The Things Network Mapper program.</td>
-        <td><a href="https://github.com/HouzuoGuo/laitos/wiki/%5BWeb-service%5D-the-things-network-LORA-tracker-integration" target="_blank">Link</a></td>
+        <td>Prometheus metrics exporter</td>
+        <td>Serve metrics info collected from web server, web proxy server, program resource usage, in prometheus exporter format.</td>
+        <td><a href="https://github.com/HouzuoGuo/laitos/wiki/%5BWeb-service%5D-prometheus-metrics-exporter" target="_blank">Link</a></td>
+    </tr>
+    <tr>
+        <td>HTTP request inspector</td>
+        <td>Dump the incoming HTTP request for your inspection.</td>
+        <td><a href="https://github.com/HouzuoGuo/laitos/wiki/%5BWeb-service%5D-request-inspector" target="_blank">Link</a></td>
     </tr>
 </table>
 

diff --git a/doc/Get-started.md b/doc/Get-started.md
@@ -102,18 +102,22 @@ Use the following command line options with extra care:
 <table>
 <tr>
     <th>Flag</th>
+    <th>Value data type</th>
     <th>Meaning</th>
 </tr>
 <tr>
     <td>-debug</td>
+    <td>true/false</td>
     <td>Print stack traces to standard error upon receiving the interrupt signal SIGINT.</td>
 </tr>
 <tr>
-    <td>-gomaxprocs</td>
+    <td>-gomaxprocs Num</td>
+    <td>Integer</td>
     <td>Specify maximum number of concurrent goroutines. The default value is the number of CPU cores/threads.</td>
 </tr>
 <tr>
     <td>-disableconflicts</td>
+    <td>true/false</td>
     <td>
         Automatically disable the following system softwares that may run into resource conflict:<br>
         <ul>
@@ -128,10 +132,53 @@ Use the following command line options with extra care:
 </tr>
 <tr>
     <td>-awslambda</td>
+    <td>true/false</td>
     <td>
       Launch laitos as a handler for AWS Lambda function.
       <br/>
       See <a href="https://github.com/HouzuoGuo/laitos/wiki/Cloud-tips">cloud deployment tips</a> for the detailed usage.
     </td>
 </tr>
+<tr>
+    <td>-awsinteg</td>
+    <td>true/false</td>
+    <td>
+      The master switch for turning on all points of integration with AWS infrastructure resources such as S3, SNS, SQS, Kinesis Firehose.
+      <br/>
+      See <a href="https://github.com/HouzuoGuo/laitos/wiki/Cloud-tips">cloud deployment tips</a> for the detailed usage.
+    </td>
+</tr>
+<tr>
+    <td>-prominteg</td>
+    <td>true/false</td>
+    <td>
+      The master switch for turning on all points of integration with prometheus metrics exporter.
+      <br/>
+      See <a href="https://github.com/HouzuoGuo/laitos/wiki/%5BWeb-service%5D-prometheus-metrics-exporter">Web service - prometheus metrics exporter</a>
+      for the detailed usage.
+    </td>
+</tr>
+<tr>
+    <td>-profhttpport PORT</td>
+    <td>Integer</td>
+    <td>
+      Start an HTTP server on localhost:PORT to serve program profiling data at URL location "/debug/pprof/{cmdline,profile,symbol,trace}".
+    </td>
+</tr>
 </table>
+
+### Build a container image
+Images of a (usually) up-to-date version of laitos are uploaded to Docker Hub [hzgl/laitos](https://hub.docker.com/r/hzgl/laitos).
+
+If you wish to customise the image to your needs, feel free to use the [`Dockerfile`](https://github.com/HouzuoGuo/laitos/blob/master/Dockerfile)
+from GitHub repository as a reference.
+
+### Supply program configuration in an environment variable
+Usually, the program configuration is kept in a JSON file, the path of which is specified in the laitos launch command line (`-config my-laitos-config.json`).
+However, if the program configuration is short enough to fit into an environment variable, laitos can also get its configuration
+from there. This can be rather useful for testing a configuration snippet or starting a small number of daemons in a container.
+
+The following example starts the HTTP daemon (without TLS) on the default port number (80), the web server comes with app-command runner endpoint:
+
+    sudo env 'LAITOS_CONFIG={"HTTPFilters": {"PINAndShortcuts": {"Passwords": ["abcdefgh"]},"LintText": {"MaxLength": 1000}},"HTTPHandlers": {"AppCommandEndpoint": "/cmd"}}' ./laitos -daemons insecurehttpd
+    # Try running an app command: curl http://0.0.0.0:80/cmd?cmd=abcdefgh.sdate