adds introduction module

reader/content/_index.md

@@ -1,4 +1,13 @@
 ---
 archetype: "home"
 title: "Introduction to Software Engineering"
----
+---
+
+This is the course reader for CPSC 310 at the University of British Columbia.
+This course is a comprehensive introduction to how modern software systems are designed, constructed, and evolved.
+It is intended to be paired with a development-heavy project to better reinforce the core concepts from lecture materials and to enable concepts to be applied in practice.
+
+The materials have been adapted over several years of use, although each section of the course may use different subsets of the readings and videos.
+This is a high-level overview of what the course will be about.
+The materials roughly break down into 6 high-level modules that are spread across the 13 week of standard academic semester at UBC.
+Readings and videos are available for most course concepts.

reader/content/introduction/_index.md

@@ -3,3 +3,72 @@ archetype: "chapter"
 weight: 1
 title: "Introduction"
 ---
+
+Software engineering is a broad and relatively young field. Here we will briefly touch on what software is, why it needs to be engineered, and what role the field of software engineering plays in the overall lifecycle of modern software systems.
+
+
+{{< youtube 3tXzYUVQlrM >}}
+
+
+{{% notice default "Learning Outcomes" "graduation-cap" %}}
+After completing this chapter, you should understand the following differences
+
+* [x] [Syntax vs Semantics]({{< ref "languages#language-properties" >}})
+* [x] [Interpreted vs Complied Languages]({{< ref "languages#interpreted-vs-compiled" >}})
+* [x] Static vs Dynamic Representations of the System
+* [x] [Static vs Dynamic Types]({{< ref "languages#static-vs-dynamic-types" >}})
+{{% /notice %}}
+
+
+## What is software?
+
+Software is everywhere. It is in our spacecraft, our cars, our fridges, and our phones. While software plays a key role in a diverse set of domains, at its core all software is the same: Software comprises a series of instructions for a computer to execute that consume input, perform computation, and emit output.
+
+Ed Lazowska provides a great overview of the kinds of impact computer science has on our modern society (I added the software doughnut to his original figure):
+
+
+{{< figure src="software-impact.png" >}}
+
+## Why do we need to engineer software?
+
+Given that a computer will blindly execute any set of syntactically valid instructions it is given, it is extremely important that these instructions be correct. We have all experienced software failures (be them problems that cause a program to crash or require you to reboot your computer) and defects (where the program continues to execute but behaves incorrectly). While these failures and defects are frustrating, the pervasiveness and utility of software means that these problems can have much more serious (e.g., loss of life) or expensive (e.g., hundreds of millions of dollars) implications in some domains.
+
+During his his 1972 Turing award lecture Dijkstra said,
+
+> As long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming has become an equally gigantic problem.
+
+Another way to think about this is in terms of trajectory: do you believe that software will play a greater or lesser role in your life 5 years from now? We are increasingly reliant on software systems for a myriad of both life-critical and convenient aspects of our modern lives.
+
+In terms of size, software is rapidly growing. While a decade ago 1 MLOC (Million Lines of Code) systems were somewhat rare, they are commonplace today (with many systems now ranging into hundreds of millions of lines). This explosion in complexity has been driven by our ever-expanding expectations of what software systems should be able to accomplish. The growth in size of some common systems is shown below. Managing this level of complexity requires disciplined and intentional engineering techniques.
+
+{{< figure src="system-loc-growth.png" >}}
+
+Along with the size of software systems, their overall cost has also become a major component of industries that were not historically software-heavy. For example, the electronics and software in modern cars cost 40% of the price of the car in 2015 compared to 16% in 1990.[^car-software] Similarly, with the complexity issues above, understanding, predicting, and controlling costs of complex systems requires incredible engineering effort and skill.
+
+## What is software engineering?
+
+Given the description of software above, it might be tempting to define software engineering like:
+
+> The process of transforming a mental plan of desired actions for a computer into a representation that can be understood by the computer.
+[Jean-Michel Hoc and Anh Nguyen-Xuan]
+
+This is a great definition that clearly captures the challenges of mapping from our human-defined goals for a piece of software to a representation of these goals that is executable by a computer. While the mismatch between these two representations might not seem like a big deal, it is at the root of many of the hardest aspects of software development. That said, while computers need to be able to execute software, other engineers need to be able to understand it (as observed by Knuth):
+
+> Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.
+
+
+Ultimately though, programming is only a small part of what software engineering is all about; a more comprehensive definition is:
+
+> The establishment and application of scientific, economic, social, and practical knowledge in order to specify, invent, design, build, validate, deploy, maintain, research, and improve software that is correct, reliable, and works efficiently on real machines.
+
+{{< figure src="se-tasks.png" >}}
+
+This definition captures the range of input skills required to build software (scientific, economic, social, and practical knowledge), the tasks performed by software engineers themselves (specify, invent, design, build, validate, deploy, maintain, research, and improve), and the goals of the process (to build correct, reliable, and efficient systems).
+
+The most obvious difference between the definition of programming and software engineering is that programming itself really only captures the 'build' task that is only a small piece of the spectrum of software engineering tasks. While programming software is hard, and is an important and necessary part of the software engineering process, the tasks that come before (like figuring out what to build and how to do it) and after programming (like deploying, maintaining, and evolving an existing system) are often crucial for ensuring that programming effort is effectively applied.
+
+One of the main goals of this course is to give you a deeper understanding and appreciation for the complexity involved in designing, building, and evolving high-quality software systems. To that end, we will be investigating many of the different dimensions of software engineering captured in the definition above.
+
+[^car-software]: [How much does software add to the cost of today's vehicles?](http://www.autoblog.com/2010/06/08/how-much-does-software-add-to-the-cost-of-todays-vehicles-how/)
+
+[//]: http://andrewbegel.com/info461/readings/history.html

reader/content/introduction/async/index.md

@@ -0,0 +1,166 @@
+---
+title: "Asynchronous Programming"
+weight: 2
+---
+
+{{< youtube T__CJfXswbA >}}
+
+
+Over the past decade, advances in computers have happened through increasing the number of cores within a processor rather than raw clock speed. To take advantage of this, most modern programming languages support some model concurrent execution. While at its core the code you write in JavaScript (and by extension TypeScript) can only execute in one thread a thread pool does exist in both Node and the browser to ensure that long-running tasks can be offloaded in a way that does not block the main thread. The model supported by JavaScript is that of asynchronous execution.
+
+<!---
+TODO: call stack / heap / queue / event loop figure
+--->
+
+In JavaScript functions execute on the call stack, and they always execute to completion. Since only one method can execute at a time and another cannot run until the first finishes, it is important that JavaScript functions do not take 'too long' to execute. While 'too long' can be difficult to define, in JavaScript the most commonly-held measure is 16ms; exceeding this value will cause the UI to miss a refresh on a 60Hz monitor. Stuttering caused by computations exceeding these thresholds are referred to as '[jank](http://www.html5rocks.com/en/tutorials/speed/rendering/)' within the browser. [Rick Byers](https://rbyers.github.io/scroll-latency.html) has a nice Jank demo where you can see the impact of work on the main thread with respect to UI updates.
+
+In practice, many operations take more than 16ms to happen: network latency can range into hundreds of milliseconds, and reading or writing from disk can easily take more than a second for larger files. Since these are relatively common operations, JavaScript has a thread pool that has limited capabilities but is ideal for handling blocking operations (e.g., network and file IO, long-running tasks, timeouts, etc.). Rather than executing on the main thread, these execute in a separate thread pool and are returned to the call stack via callbacks when the event loop is idle.
+
+<!---
+### JavaScript event loop example
+
+TODO: EXAMPLE
+--->
+
+[Philip Roberts](https://www.youtube.com/watch?v=8aGhZQkoFbQ) has created a great video for understanding the event loop. He also developed a cool [tool](http://latentflip.com/loupe/) for visualizing how the event loop works.
+
+{{% notice tip %}}
+The [Async Cookbook](https://github.com/ubccpsc/310/blob/main/resources/readings/cookbooks/async.md) contains many code-forward examples demonstrating how to call and test asynchronous methods using callbacks, promises, and async/await.
+{{% /notice %}}
+
+## Callbacks 
+
+{{< youtube ZyiLFip7nqk >}}
+
+
+### Shortcomings
+
+Two common problems manifest with callbacks in practice. First, handling errors is difficult because callbacks are invoked, but they do not _return_ a value. This means any information about the execution of the function making the callback must take place in the parameters. Node-based JavaScript addresses this with the [error-first](http://fredkschott.com/post/2014/03/understanding-error-first-callbacks-in-node-js/) idiom. For example:
+
+```javascript
+fs.readFile('/cpsc310.csv', function(err, data) {
+  if (err) {
+    // handle
+    return;
+  }
+  console.log(data);
+});
+```
+
+It is only by convention that `err` is passed first; this depends on the developer doing the right thing and on clients checking `err` to see if it contains a value and acting appropriately if it has.
+
+This problem becomes harder to manage once callbacks depend on the output of previous callback functions; this is extremely common (e.g., read a directory list (async) and then read a specific file (async)). This pattern arises from our natural desire to want the apparent execution to proceed from top-to-bottom in the source code file. This style of error handling is much like you would see in C code where return values were checked for error statuses (instead of the error being in a function param). One significant problem with callback-based error handling is that exceptions cannot be effectively caught. That is, since the callback is not being executed in the context of its originating function there is no 'parent' method for the catch to exist in. More details and tips about this can be found [here](https://ruben.verborgh.org/blog/2012/12/31/asynchronous-error-handling-in-javascript/). Nested callbacks are often referred to as [callback hell](http://callbackhell.com/).
+
+```javascript
+fs.readdir(source, function (err, files) {
+  if (err) {
+    // handle
+  } else {
+    files.forEach(function (name, index) {
+      fs.stat(files[index], function(err, data) {
+        if (err) {
+          // handle
+        } else {
+          if (data.isFile()) {
+            fs.readFile(files[index], function(err, data) {
+                if (err) {
+                    // handle
+                } else {
+                    // process data
+                }
+            });
+          }
+        }
+      });
+    });
+  }
+});
+```
+
+## Promises
+
+{{< youtube xwSUBY2tIm4 >}}
+
+[Promises](http://colintoh.com/blog/staying-sane-with-asynchronous-programming-promises-and-generators) are a mechanism to address two of the largest shortcoming of callback hell, namely:
+
+1. Flattening callback chains; and
+1. Enabling try/catch for error handling.
+
+Although promises do enable try/catch to work properly, they do not provide any functionality that would not be possible with callbacks alone. Their main benefit is that they are easier for developers to read and understand making it more likely that developers will 'do the right thing' in these often-complex program sequences. Adapting the callback example from above, a promise-ified version would look like:
+
+```typescript
+var file = null;
+fs.readdir(source).then(function(files) {
+    file = files[0];
+    return fs.stat(file);
+}).then(function(fileStat) {
+    if (fileStat.isFile())
+        return fs.readFile(file);
+}).then(function(fileData) {
+    // process data
+}).catch(function(err) {
+    // handle errors
+});
+```
+
+Promises can only have three states, _pending_ before it is done its task, _fulfilled_ if the task has completed successfully, and _rejected_ if the task has completed erroneously. Promises can only transition to _fulfilled_ or _rejected_ once and cannot change between _fulfilled_ and _rejected_; this process is called _settling_. [HTML5Rocks](http://www.html5rocks.com/en/tutorials/es6/promises/) has an extremely thorough walkthrough of promises where you can see how many of its features are used in practice.
+
+[//]: # (<img src="./figures/promise-states.png" width="512px" alt="promise states">)
+{{< figure src="promise-states.png" >}}
+
+At their core, promises are objects with two methods `then` and `catch`. `then` is called when the promise is settled successfully while `catch` is called when the promise is settled with an error. These methods both themselves return promises enabling them to be chained as in the example above (three `then` functions called in sequence). It is important to note that there can also be multiple `catch` statements (e.g., in a 5-step flow you could catch after the first two, fix the problem, and then continue the flow while still having a final catch at the end).
+
+One nice feature of Promises is that they can be composed; this is especially helpful when you are trying to manage multiple concurrent promise executions in parallel. This is the explicit goal of `Promise.all`:
+
+```typescript
+let processList:GroupRepoDescription[] = [];
+for (const toProcess of completeGroups) {
+    processList.push(gpc.getStats("CS310-2016Fall", toProcess.name)); // getStats is async
+}
+
+Promise.all(processList).then(function (provisionedRepos: GroupRepoResult[]) {
+    Log.info("Creation complete: " + provisionedRepos.length);
+    // can also iterate through the individual results
+}).catch(function(err) {
+    Log.error("Creation failed: "+err);
+});
+```
+
+Analogous to `Promise.all` is `Promise.race`. This allows for starting multiple async calls; `then` is called when the _first_ promise fulfills, rather than when _all_ of them fulfill as is required for `Promise.all`. `Promise.race` can be handy when there are multiple ways to accomplish a task, but you cannot know which will be fastest in advance, or if you want to show progress on a task by updating the user when data starts to become available rather than waiting for a full operation to be complete.
+
+When working with promises it is _strongly_ encouraged that you end every `then` sequence with a `catch` even if you do not do any meaningful error handling (like we do above). This is because promises settle asynchronously and will fail silently if rejections are not caught; they will not escape to a global error handler. Another common mistake when working with promises is that `then` is not applied to the right object. In the above example if you called `then` inside the `for` loop you would not be notified when all promises are done but one-by-one as they execute. Adding verbose debug messages can really help here, so you can keep track of the order your functions and callbacks are executing.
+
+<!-- 
+TODO async/await
+-->
+
+## References
+
+* Discussion of some [challenges](https://www.quora.com/What-is-the-difference-between-deadlock-and-livelock-deadlock-infinite-recursion-and-starvation/answer/Akash-Kava) with concurrent programming.
+
+* [Mozilla](https://developer.mozilla.org/en/docs/Web/JavaScript/EventLoop) event loop overview.
+
+* Article about the event loop with [more examples](http://altitudelabs.com/blog/what-is-the-javascript-event-loop/).
+
+* Article about the event loop that includes some detail about [closures](http://blog.carbonfive.com/2013/10/27/the-javascript-event-loop-explained/).
+
+* A collection of promise use cases can be found [here](http://exploringjs.com/es6/ch_promises.html) and [further examples](https://www.promisejs.org/patterns/).
+
+* While promises seem really fancy, [how they are implemented](http://www.mattgreer.org/articles/promises-in-wicked-detail/) is relatively straightforward (or even more [straightforward](https://www.promisejs.org/implementing/)).
+
+* Sencha also has a nice [promise intro](https://www.sencha.com/blog/asynchronous-javascript-promises/).
+
+* Callback, Promise, async/await [concrete example](https://medium.com/@gab_montes/is-async-await-a-step-back-to-javascript-95e31263dd31#.8jtvqy8fb).
+
+* [Async/await](https://hackernoon.com/javascript-es7-async-await-bible-tutorial-example-32294f6133ab#.4p2stibtt) will eventually supplement promises, although they are still really new language features.
+
+<!---
+Another good reference:
+http://www.2ality.com/2014/10/es6-promises-api.html
+--->
+
+
+
+
+
+