# notes

path: zk/Redirection_in_Bash.md
sortable_path: zkRedirection_in_Bash.md
title: Redirection
lead: ## Redirecting outputs
checksum: e64c3fd247bb692dbe02ecb5f5d645298857b8d8675be4a8a9e3532475b2fc00
created: 2024-06-16 17:37:31.825481372+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"tags":["shell"]}
body:
  ## Redirecting outputs

  The symbol `>` is called the **redirection operator** because it redirects
  `stdout` to another program or file. You most frequently use this when you want
  to save contents to a file rather than standard output.

  ```bash
  ls | grep d* > result.txt
  ```

  ### Combining redirection with file escriptors

  It is common practice to combine redirection with the
  [file descriptors](File_descriptors.md) to redirect
  the output of `stdout` and `stderr`. A common case is to
  [redirect error output to `/dev/null`](Redirect_to_dev_null.md).

  Redirection defaults to interpreting `>` as the redirection of `stdout` (`1`);

  ## Redirecting inputs

  We can also switch the direction of the redirection symbol and pass in a file to
  a command rather than command ouput to a file:

  ```bash
  sql-processing-program < data.sql
  ```

  We can redirect a string with three carets:

  ```bash
  program <<< "this is a string input"
  ```

  ## Appending

  We use `>>` to append contents on the next available line of a pre-existing
  file. Continuing on from the example above:

  ```bash
  echo 'These are the files I just grepped' >> result.txt
  ```

raw_content:
  ---
  tags:
    - shell
  ---

  # Redirection

  ## Redirecting outputs

  The symbol `>` is called the **redirection operator** because it redirects
  `stdout` to another program or file. You most frequently use this when you want
  to save contents to a file rather than standard output.

  ```bash
  ls | grep d* > result.txt
  ```

  ### Combining redirection with file escriptors

  It is common practice to combine redirection with the
  [file descriptors](File_descriptors.md) to redirect
  the output of `stdout` and `stderr`. A common case is to
  [redirect error output to `/dev/null`](Redirect_to_dev_null.md).

  Redirection defaults to interpreting `>` as the redirection of `stdout` (`1`);

  ## Redirecting inputs

  We can also switch the direction of the redirection symbol and pass in a file to
  a command rather than command ouput to a file:

  ```bash
  sql-processing-program < data.sql
  ```

  We can redirect a string with three carets:

  ```bash
  program <<< "this is a string input"
  ```

  ## Appending

  We use `>>` to append contents on the next available line of a pre-existing
  file. Continuing on from the example above:

  ```bash
  echo 'These are the files I just grepped' >> result.txt
  ```


path: zk/Restructure_URLs_lambda.md
sortable_path: zkRestructure_URLs_lambda.md
title: AWS Lambda examples
lead: ## Reconstruct CloudFront URLs
checksum: 6a911ce9d3aa21da4b42240dc00369ab6f54b0e35dca6f1cd3d11e804463b857
created: 2024-06-16 17:37:31.847347532+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"tags":["AWS","aws-lambda"]}
body:
  ## Reconstruct CloudFront URLs

  ```js
  exports.handler = (event, context, callback) => {
    // Extract the request from the CloudFront event that is sent to Lambda@Edge
    var request = event.Records[0].cf.request;

    // Extract the URI from the request
    var olduri = request.uri;

    // Match any '/' that occurs at the end of a URI. Replace it with a default index
    var newuri = olduri.replace(/\/$/, "/index.html");

    // Replace the received URI with the URI that includes the index page
    request.uri = newuri;

    // Return to CloudFront
    return callback(null, request);
  };
  ```

  Let's walk through the above example making reference to the general structure
  of the
  [AWS Lambda handler function](Lambda_handler_function.md):

  1. The `event` parameter is invoked to gain access to the event that triggered
     the function. This is a CloudFront request event which is targetted with
     `event.Records[0].cf.request`
  2. We classify this request as the `olduri` since it doesn't have the form we
     want.
  3. We mutate the original request by replacing `/` with `/index.html`
  4. We then re-assign the `request.uri` value to the value of the new URI.
  5. Finally we invoke the `callback` parameter to signal the completion of the
     transformation and return the new request as the return value.

raw_content:
  ---
  tags:
    - AWS
    - aws-lambda
  ---

  # AWS Lambda examples

  ## Reconstruct CloudFront URLs

  ```js
  exports.handler = (event, context, callback) => {
    // Extract the request from the CloudFront event that is sent to Lambda@Edge
    var request = event.Records[0].cf.request;

    // Extract the URI from the request
    var olduri = request.uri;

    // Match any '/' that occurs at the end of a URI. Replace it with a default index
    var newuri = olduri.replace(/\/$/, "/index.html");

    // Replace the received URI with the URI that includes the index page
    request.uri = newuri;

    // Return to CloudFront
    return callback(null, request);
  };
  ```

  Let's walk through the above example making reference to the general structure
  of the
  [AWS Lambda handler function](Lambda_handler_function.md):

  1. The `event` parameter is invoked to gain access to the event that triggered
     the function. This is a CloudFront request event which is targetted with
     `event.Records[0].cf.request`
  2. We classify this request as the `olduri` since it doesn't have the form we
     want.
  3. We mutate the original request by replacing `/` with `/index.html`
  4. We then re-assign the `request.uri` value to the value of the new URI.
  5. Finally we invoke the `callback` parameter to signal the completion of the
     transformation and return the new request as the return value.


path: zk/Routing_on_link_click_in_React.md
sortable_path: zkRouting_on_link_click_in_React.md
title: Test: Routing on link click
lead: ## Component
checksum: 0935aa4e5e1e4051e4ca684aa0735d3b0fb3a555a93412746712f20724dee40b
created: 2024-06-16 17:37:31.858850096+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"tags":["javascript","react","testing"]}
body:
  ## Component

  ```jsx
  const CertificateHtmlLink = () => {
    return (
      <div className="certificate-html-link html-version-message">
        <Link className="btn btn-primary ml-2" to="/some-location">
          View HTML version
        </Link>
      </div>
    );
  };
  ```

  ## Test

  ```js
  import { render, fireEvent } from "@testing-library/react";
  import { Router } from "react-router-dom";

  describe("<CertificateHtmlLink", () => {
    it("should link to dynamic HTML certificate page URL on click event", () => {
      // Arrange:
      const history = createMemoryHistory({ initialEntries: ["/"] });

      // Act:
      const { getByText } = render(
        <Router history={history}>
          <CertificateHtmlLink to="/certificate/123" />
        </Router>
      );

      // Assert:
      expect(history.location.pathname).toBe("/");
      fireEvent.click(getByText("View HTML version"));
      expect(history.location.pathname).toBe("/certificate/123");
    });
  });
  ```

raw_content:
  ---
  tags:
    - javascript
    - react
    - testing
  ---

  # Test: Routing on link click

  ## Component

  ```jsx
  const CertificateHtmlLink = () => {
    return (
      <div className="certificate-html-link html-version-message">
        <Link className="btn btn-primary ml-2" to="/some-location">
          View HTML version
        </Link>
      </div>
    );
  };
  ```

  ## Test

  ```js
  import { render, fireEvent } from "@testing-library/react";
  import { Router } from "react-router-dom";

  describe("<CertificateHtmlLink", () => {
    it("should link to dynamic HTML certificate page URL on click event", () => {
      // Arrange:
      const history = createMemoryHistory({ initialEntries: ["/"] });

      // Act:
      const { getByText } = render(
        <Router history={history}>
          <CertificateHtmlLink to="/certificate/123" />
        </Router>
      );

      // Assert:
      expect(history.location.pathname).toBe("/");
      fireEvent.click(getByText("View HTML version"));
      expect(history.location.pathname).toBe("/certificate/123");
    });
  });
  ```


path: zk/Schema_Definition_Language_in_GraphQL.md
sortable_path: zkSchema_Definition_Language_in_GraphQL.md
title: Schema Definition Language
lead: SDL is the formal name for the syntax of GraphQL schemas.
checksum: c50d5b10c7f9a915e1d4d5d9bb67ef337b59f126f97bfda5c6bfeba74d25ac1c
created: 2024-06-16 17:37:31.863579167+00:00
modified: 2024-10-18 17:55:56.292394043+00:00
metadata: {"tags":["graphql"]}
body:
  SDL is the formal name for the syntax of GraphQL schemas.

  ## Types

  A schema is a collection of object types that contain fields. Each field has a
  type of its own. A field's type can be a primitive/scalar value (such as an
  `Int` or a `String`), or it can be another object type (just like a custom type
  in TS).

  A schema's type can be non-nullable which is to say, a required field. We
  indicate this with `!`.

  A type for a field can be a collection/array of a given type.

  The following example indicates these properties:

  ```gql
  type Person {
    age: Int
    name: String
    pets: [Pet]!
  }

  type Pet {
    species: String
    name: String
    age: Int
  }
  ```

  ## Queries

  A query is also a schema type but of a special sort.

  > The fields of this type are entry points into the rest of the schema. These
  > are the top-level fields that the client can query for.

  For example if we had this type:

  ```graphql
  type Track {
    id: ID!
      author: Author!
    thumbnail: String
    length: Int
    modulesCount: Int
  }
  ```

  We could define a type to access a give `Track` as follows:

  ```graphql
  type Query {
    tracksForHomePage: [Track!]!
  }
  ```

  Then use this type as the basis for a query:

  ```

  ```

  ## Descriptions

  Descriptions are comments that allow you to document your Schema

  Single line:

  ```gql
  "Single line comment"
  ```

  ```gql
  """
  Multi
  line
  comment
  """
  ```

raw_content:
  ---
  tags: [graphql]
  ---

  # Schema Definition Language

  SDL is the formal name for the syntax of GraphQL schemas.

  ## Types

  A schema is a collection of object types that contain fields. Each field has a
  type of its own. A field's type can be a primitive/scalar value (such as an
  `Int` or a `String`), or it can be another object type (just like a custom type
  in TS).

  A schema's type can be non-nullable which is to say, a required field. We
  indicate this with `!`.

  A type for a field can be a collection/array of a given type.

  The following example indicates these properties:

  ```gql
  type Person {
    age: Int
    name: String
    pets: [Pet]!
  }

  type Pet {
    species: String
    name: String
    age: Int
  }
  ```

  ## Queries

  A query is also a schema type but of a special sort.

  > The fields of this type are entry points into the rest of the schema. These
  > are the top-level fields that the client can query for.

  For example if we had this type:

  ```graphql
  type Track {
    id: ID!
      author: Author!
    thumbnail: String
    length: Int
    modulesCount: Int
  }
  ```

  We could define a type to access a give `Track` as follows:

  ```graphql
  type Query {
    tracksForHomePage: [Track!]!
  }
  ```

  Then use this type as the basis for a query:

  ```

  ```

  ## Descriptions

  Descriptions are comments that allow you to document your Schema

  Single line:

  ```gql
  "Single line comment"
  ```

  ```gql
  """
  Multi
  line
  comment
  """
  ```


path: zk/Secrets_or_env_vars_in_AWS.md
sortable_path: zkSecrets_or_env_vars_in_AWS.md
title: Secrets or environment variables?
checksum: 5dbb25aea37698bb6cf861e39dfd15ee9823b0e0e928f76e9d1d4e3899538a9b
created: 2024-06-16 17:37:31.867266988+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"tags":["AWS","aws-lambda"]}
lead:
  Choosing between environment variables and AWS Secrets Manager hinges on factors
  such as use case, security needs, and the data type. Environment variables are
  easier to use, free, and offer fast access but are less secure and lack features
  like versioning. On the other hand, AWS Secrets Manager provides robust security
  through encryption, access control, audit trails, and supports secret versioning
  and rotation. However, it comes with additional costs and can introduce latency
  and rate limits.

body:
  Choosing between environment variables and AWS Secrets Manager hinges on factors
  such as use case, security needs, and the data type. Environment variables are
  easier to use, free, and offer fast access but are less secure and lack features
  like versioning. On the other hand, AWS Secrets Manager provides robust security
  through encryption, access control, audit trails, and supports secret versioning
  and rotation. However, it comes with additional costs and can introduce latency
  and rate limits.

  General recommendations suggest using AWS Secrets Manager for storing sensitive
  data like API keys and passwords, due to its enhanced security features. For
  less sensitive, non-critical data such as feature flags or environment-specific
  URLs, environment variables are often adequate. Cost-conscious users who don't
  need advanced features may find environment variables more economical.
  Ultimately, the best choice depends on your specific requirements and
  limitations.

raw_content:
  ---
  tags: [AWS, aws-lambda]
  ---

  # Secrets or environment variables?

  Choosing between environment variables and AWS Secrets Manager hinges on factors
  such as use case, security needs, and the data type. Environment variables are
  easier to use, free, and offer fast access but are less secure and lack features
  like versioning. On the other hand, AWS Secrets Manager provides robust security
  through encryption, access control, audit trails, and supports secret versioning
  and rotation. However, it comes with additional costs and can introduce latency
  and rate limits.

  General recommendations suggest using AWS Secrets Manager for storing sensitive
  data like API keys and passwords, due to its enhanced security features. For
  less sensitive, non-critical data such as feature flags or environment-specific
  URLs, environment variables are often adequate. Cost-conscious users who don't
  need advanced features may find environment variables more economical.
  Ultimately, the best choice depends on your specific requirements and
  limitations.


path: zk/Setting_up_and_configuring_TypeScript.md
sortable_path: zkSetting_up_and_configuring_TypeScript.md
title: Setup and configuration
lead: ## Installation and configuration
checksum: 7ad962fc516dcea03815580140ae449dfaeb4c74b9e330cc4b8199af5454edf1
created: 2024-06-16 17:37:31.871578494+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"tags":["typescript"]}
body:
  ## Installation and configuration

  TypeScript offers custom installations for most modern JS-based frameworks
  including Webpack, React.js and Vue.js. The instructions below cover minimal TS
  set-up outside of a specific framework. TypeScript adds an additional step to
  any build process since your code must compile to JS before any bundling and
  transpilation can occur.

  ## Installation

  ```
  mkdir typescript-project
  cd typescript-project
  npm i typescript --save-dev
  ```

  ## Initialise project

  ```
  npx tsc --init
  ```

  ## Basic configuration

  ```json
  "compilerOptions": {
  	"target" : "es2020", //es2015 for prod
  	"module" : "es2020",
    	"allowJs": true,
    	"checkJs": true,
    	"strict": true,
    	"esModuleInterop": true
  }
  ```

  ### Specify output paths (for production code)

  ```json
  "compilerOptions": {
  	...
  	"outDir": "dist",
  	"sourceMap": true,
  }
  ```

  ### Compile-time commands

  ```
  tsc --noEmit
  ```

  ```
  tsc --noEmit --watch
  ```

  ## Global requirements

  ```
  npm install -g typescript
  npm install -g ts-node
  ```

  `ts-node` allows you to run TS files the way you would with standard JS, i.e.
  `node [filename]`, you just use `ts-node filename` .

  This is a convenience that saves you compiling from TS → JS and then running
  Node against the compiled JS. Which is useful for debugging and checking values
  as you work.

  ## Imports and exports

  You may wish to define your custom types outside of their application, for
  instance in a `/types/` directory. The custom type or types can then be imported
  using standard ES6 imports:

  ```tsx
  export type Article = {
  		price: number,
  }

  export type AnotherType = {
  ...
  }
  ```

  ```tsx
  import { Article, AnotherType } from "./types/allTypes.js";
  import Article from "./types/allTypes.js";
  ```

  TypeScript also provides a specific import keyword `type` so that you can
  distinguish type imports from other module imports. This is handy when you have
  multiple components in the one file:

  ```tsx
  import type { Article } from "./types/allTypes.js";
  ```

raw_content:
  ---
  tags:
    - typescript
  ---

  # Setup and configuration

  ## Installation and configuration

  TypeScript offers custom installations for most modern JS-based frameworks
  including Webpack, React.js and Vue.js. The instructions below cover minimal TS
  set-up outside of a specific framework. TypeScript adds an additional step to
  any build process since your code must compile to JS before any bundling and
  transpilation can occur.

  ## Installation

  ```
  mkdir typescript-project
  cd typescript-project
  npm i typescript --save-dev
  ```

  ## Initialise project

  ```
  npx tsc --init
  ```

  ## Basic configuration

  ```json
  "compilerOptions": {
  	"target" : "es2020", //es2015 for prod
  	"module" : "es2020",
    	"allowJs": true,
    	"checkJs": true,
    	"strict": true,
    	"esModuleInterop": true
  }
  ```

  ### Specify output paths (for production code)

  ```json
  "compilerOptions": {
  	...
  	"outDir": "dist",
  	"sourceMap": true,
  }
  ```

  ### Compile-time commands

  ```
  tsc --noEmit
  ```

  ```
  tsc --noEmit --watch
  ```

  ## Global requirements

  ```
  npm install -g typescript
  npm install -g ts-node
  ```

  `ts-node` allows you to run TS files the way you would with standard JS, i.e.
  `node [filename]`, you just use `ts-node filename` .

  This is a convenience that saves you compiling from TS → JS and then running
  Node against the compiled JS. Which is useful for debugging and checking values
  as you work.

  ## Imports and exports

  You may wish to define your custom types outside of their application, for
  instance in a `/types/` directory. The custom type or types can then be imported
  using standard ES6 imports:

  ```tsx
  export type Article = {
  		price: number,
  }

  export type AnotherType = {
  ...
  }
  ```

  ```tsx
  import { Article, AnotherType } from "./types/allTypes.js";
  import Article from "./types/allTypes.js";
  ```

  TypeScript also provides a specific import keyword `type` so that you can
  distinguish type imports from other module imports. This is handy when you have
  multiple components in the one file:

  ```tsx
  import type { Article } from "./types/allTypes.js";
  ```


path: zk/Split_into_array_in_Bash.md
sortable_path: zkSplit_into_array_in_Bash.md
title: Splitting input into an array
lead: ## readarray
checksum: b613cfe6a16f00e820880c671aeeccbf3f0ecdae9e3770fc68016d7b50c6b413
created: 2024-06-16 17:37:31.906852767+00:00
modified: 2024-10-18 17:55:56.292394043+00:00
metadata: {"tags":["shell"]}
body:
  ## readarray

  `readarray` makes it really easy to split input into an array based on new
  lines. Say we have this file as input:

  ```
  123
  456
  789
  ```

  Then we can split like so:

  ```bash
  readarray -t name_for_array < ./input.text

  # Print all elements
  echo "${name_for_array[@]}"

  # Print element by index

  echo "${name_for_array[1]}"
  456
  ```

  If we want to read a string directly:

  ```bash
  readarray -t new_name_for_array <<< "here
  is
  some
  text"

  echo "${new_name_for_array[1]}"
  is
  ```

  > The _-t_ flag removes the trailing newline

  See more: https://linuxhint.com/split-string-array-bash/

  ## read

  For delimiters other than a space we have to use `read`, combined with `IFS` the
  **Internal Field Separator**.

  For example, to split by comma:

  ```plaintext
  # comma-input.txt
  something,something else,something more
  ```

  ```bash
  IFS=',' read -a arr < ./comma_inputs.txt
  ```

  > We use the `-a` flag to signal that we want each item read returned as an
  > element in an array

  ## mapfile

  We can use `mapfile` to read the lines of a input file in set segments.

  For the following file:

  ```
  line one
  line two
  line three
  line four
  ```

  We could generate the following output:

  ```
  -----SNIP-----
  line one
  line two
  -----SNIP-----
  line three
  line four
  ```

  With:

  ```bash
  while mapfile -t -n 2 ary && ((${#ary[@]})); do
    printf '%s\n' "${ary[@]}"
    printf -- '--- SNIP ---\n'
   done < $testInput
  ```

  So, while there are input lines, map each line into an array and return it in
  groups of 2.

raw_content:
  ---
  tags:
    - shell
  ---

  # Splitting input into an array

  ## readarray

  `readarray` makes it really easy to split input into an array based on new
  lines. Say we have this file as input:

  ```
  123
  456
  789
  ```

  Then we can split like so:

  ```bash
  readarray -t name_for_array < ./input.text

  # Print all elements
  echo "${name_for_array[@]}"

  # Print element by index

  echo "${name_for_array[1]}"
  456
  ```

  If we want to read a string directly:

  ```bash
  readarray -t new_name_for_array <<< "here
  is
  some
  text"

  echo "${new_name_for_array[1]}"
  is
  ```

  > The _-t_ flag removes the trailing newline

  See more: https://linuxhint.com/split-string-array-bash/

  ## read

  For delimiters other than a space we have to use `read`, combined with `IFS` the
  **Internal Field Separator**.

  For example, to split by comma:

  ```plaintext
  # comma-input.txt
  something,something else,something more
  ```

  ```bash
  IFS=',' read -a arr < ./comma_inputs.txt
  ```

  > We use the `-a` flag to signal that we want each item read returned as an
  > element in an array

  ## mapfile

  We can use `mapfile` to read the lines of a input file in set segments.

  For the following file:

  ```
  line one
  line two
  line three
  line four
  ```

  We could generate the following output:

  ```
  -----SNIP-----
  line one
  line two
  -----SNIP-----
  line three
  line four
  ```

  With:

  ```bash
  while mapfile -t -n 2 ary && ((${#ary[@]})); do
    printf '%s\n' "${ary[@]}"
    printf -- '--- SNIP ---\n'
   done < $testInput
  ```

  So, while there are input lines, map each line into an array and return it in
  groups of 2.


path: zk/Streams_in_NodeJS.md
sortable_path: zkStreams_in_NodeJS.md
title: Handling streams with fs
checksum: f8299b5e6a1242c1e024952c4e37aaa85dbc54a9b22f1a262b3658605e2ec655
created: 2024-06-16 17:37:31.912375997+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"tags":["node-js"]}
lead:
  When reading from a file, the
  [`fs.readFile()`](fs.md) method waits
  until the entire file has been read before executing the callback. It's obvious
  why this might not be ideal in certain cases. If the file is very large you are
  utilising a lot of [memory](Memory.md) for a
  single process. Additionally, the data you need might appear early in the file,
  in which case, once you find the data you want, there is no need to read to the
  end of the file.

body:
  When reading from a file, the
  [`fs.readFile()`](fs.md) method waits
  until the entire file has been read before executing the callback. It's obvious
  why this might not be ideal in certain cases. If the file is very large you are
  utilising a lot of [memory](Memory.md) for a
  single process. Additionally, the data you need might appear early in the file,
  in which case, once you find the data you want, there is no need to read to the
  end of the file.

  > An example of this in practice is watching a Netflix film: we don't have to
  > wait for the whole film to download, we can start watching it immediately
  > because it is passed to us in chunks.

  ## Reading streams

  This is why the ability to read files and data as streams exists in Node.js.
  Streams promote memory and time efficiency. As you are not necessarily reading
  the whole file, you can extract the data you need quickly and you are not
  putting the whole file in memory.

  When read as a stream, the file is broken up into smaller chunks. Readable
  streams raise data events and pass the chunks to our callbacks. Hence we don't
  have to wait for the whole file before registering data chunks.

  ## Implementation

  Instead of, e.g :

  ```js
  fs.readFile("./lorem.md", "UTF-8", function (err, fileContents) {
    console.log(fileContents);
  });
  ```

  We would:

  ```js
  let stream = fs.createReadStream("./lorem.md", "UTF-8");

  let data;

  stream.once("data", (chunk) => {
    console.log("read stream started");
  });

  stream.on("data", (chunk) => {
    data += chunk;
    console.log(data);
  });

  stream.on("end", () => console.log("stream ends"));
  ```

  We use `once` to do something at the moment the stream starts to be read.

  We use `on` to target the `"data"` and `"end"` events. We use this to log the
  chunks via the `data` variable.

  ## Writing streams

  When we write as a stream, we break data up and pass this to a reader as a
  stream. It is what creates a stream in the first place.

  For example we could create a stream that handles `stdin` and writes this to a
  file in chunks:

  ```js
  var stream = fs.createWriteStream("lorem.txt")

  process.stdin.once("data", (data) => {
      stream.write(data, function() => {
          // data is now written.
      })
  })

  process.on("exit", function () {
      stream.close()
  })
  ```

  Note that we always have to close the stream on exit

raw_content:
  ---
  tags:
    - node-js
  ---

  # Handling streams with fs

  When reading from a file, the
  [`fs.readFile()`](fs.md) method waits
  until the entire file has been read before executing the callback. It's obvious
  why this might not be ideal in certain cases. If the file is very large you are
  utilising a lot of [memory](Memory.md) for a
  single process. Additionally, the data you need might appear early in the file,
  in which case, once you find the data you want, there is no need to read to the
  end of the file.

  > An example of this in practice is watching a Netflix film: we don't have to
  > wait for the whole film to download, we can start watching it immediately
  > because it is passed to us in chunks.

  ## Reading streams

  This is why the ability to read files and data as streams exists in Node.js.
  Streams promote memory and time efficiency. As you are not necessarily reading
  the whole file, you can extract the data you need quickly and you are not
  putting the whole file in memory.

  When read as a stream, the file is broken up into smaller chunks. Readable
  streams raise data events and pass the chunks to our callbacks. Hence we don't
  have to wait for the whole file before registering data chunks.

  ## Implementation

  Instead of, e.g :

  ```js
  fs.readFile("./lorem.md", "UTF-8", function (err, fileContents) {
    console.log(fileContents);
  });
  ```

  We would:

  ```js
  let stream = fs.createReadStream("./lorem.md", "UTF-8");

  let data;

  stream.once("data", (chunk) => {
    console.log("read stream started");
  });

  stream.on("data", (chunk) => {
    data += chunk;
    console.log(data);
  });

  stream.on("end", () => console.log("stream ends"));
  ```

  We use `once` to do something at the moment the stream starts to be read.

  We use `on` to target the `"data"` and `"end"` events. We use this to log the
  chunks via the `data` variable.

  ## Writing streams

  When we write as a stream, we break data up and pass this to a reader as a
  stream. It is what creates a stream in the first place.

  For example we could create a stream that handles `stdin` and writes this to a
  file in chunks:

  ```js
  var stream = fs.createWriteStream("lorem.txt")

  process.stdin.once("data", (data) => {
      stream.write(data, function() => {
          // data is now written.
      })
  })

  process.on("exit", function () {
      stream.close()
  })
  ```

  Note that we always have to close the stream on exit


path: zk/Strings_in_Bash.md
sortable_path: zkStrings_in_Bash.md
title: Strings in bash
lead: ## Return a substring by index
checksum: 2530f2db53c8899fd0d79f4d754052143e4479eb4e64620e2cc4cc233aca8321
created: 2024-06-16 17:37:31.925959714+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"tags":["shell","data-types"]}
body:
  ## Return a substring by index

  ```bash
  myString="hello"
  substring=${myString:0:3}
  # hel
  ```

  This is often used when looping through each character in a string.

  ## Loop through each character in a string

  ```bash
  str="hallelujah"
  stringLength=$(expr length str)

  for (( i=0; i<=${stringLength}; i++ )); do
      echo "${str:$i:1}"
  done
  ```

raw_content:
  ---
  tags:
    - shell
    - data-types
  ---

  # Strings in bash

  ## Return a substring by index

  ```bash
  myString="hello"
  substring=${myString:0:3}
  # hel
  ```

  This is often used when looping through each character in a string.

  ## Loop through each character in a string

  ```bash
  str="hallelujah"
  stringLength=$(expr length str)

  for (( i=0; i<=${stringLength}; i++ )); do
      echo "${str:$i:1}"
  done
  ```


path: zk/Text_manipulation_in_Bash.md
sortable_path: zkText_manipulation_in_Bash.md
title: Text manipulation
lead: ## Sorting strings: `sort`
checksum: 16bd858534a5e8bb07228d40c6a8984bad8a4b74cf7a8c1b32bae9ae2ba8dfe1
created: 2024-06-16 17:37:31.952865941+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"tags":["shell"]}
body:
  ## Sorting strings: `sort`

  If you have a `.txt` file containing text strings, each on a new line you can
  use the sort function to quickly put them in alphabetical order:

  ```bash
  sort file.txt
  ```

  Note that this will not save the sort, it only presents it as a standard output.
  To save the sort you need to direct the sort to a file in the standard way:

  ```bash
  sort file.txt > output.txt
  ```

  ### Options

  - `-r`
    - reverse sort
  - `c`
    - check if file is already sorted. If not, it will highlight the strings which
      are not sorted

  ## Find and replace: `sed`

  The `sed` programme can be used to implement find and replace procedures. In
  `sed`, find and replace are covered by the substitution option: `/s` :

  ```bash
  sed ‘s/word/replacement word/’ file.txt
  ```

  This however will only change the first instance of word to be replaced, in
  order to apply to every instance you need to add the global option: `-g` .

  As sed is a stream editor, any changes you make using it, will only occur within
  the standard output , they will not be saved to file. In order to save to file
  you need to specify a new file output (using `> output.txt`) in addition to the
  original file. This hasthe benefit of leaving the original file untouched whilst
  ensuring the desired outcome is stored permanently.

  Alternatively, you can use the `-i` option which will make the changes take
  place in the source file as well as in standard input.

  Note that this will overwrite the original version of the file and it cannot be
  regained. If this is an issue then it is recommended to include a backup command
  in the overall argument like so:

  ```bash
  sed -i.bak ‘s/word/replacement word/’ file.txt
  ```

  This will create the file `file.txt.bak` in the directory you are working within
  which is the original file before the replacement was carried out.

  ### Remove duplicates

  We can use the `sort -u` command can be used to remove duplicates:

  ```bash
  sort -u file.txt
  ```

  It is important to sort before attempting to remove duplicates since the `-u`
  flag works on the basis of the strings being adjacent.

  ## Split a large file into multiple smaller files: `split`

  Suppose you have a file containing 1000 lines. You want to break the file up
  into five separate files, each containing two hundred lines. You can use `split`
  to accomplish this, like so:

  ```bash
  split -l 200 big-file.txt new-files
  ```

  `split` will categorise the resulting five files as follows:

  - new-file-aa,
  - new-file-ab
  - new-file-ac,
  - newfile-ad,
  - new-file-ae.

  If you would rather have numeric suffixes, use the option `-d` . You can also
  split a file by its number of bytes, using the option `-b` and specifying a
  constituent file size.

  ## Merge multiple files into one with `cat`

  We can use `cat` read multiple files at once and then append a redirect to save
  them to a file:

  ```bash
  cat file_a.txt file_b.txt file_c.txt > merged-file.txt
  ```

  ## Count lines, words, etc: `wc`

  To count words:

  ```bash
  wc file.txt
  ```

  When we use the command three numbers are outputted, in order: lines, words,
  bytes.

  You can use modifiers to get just one of the numbers: `-l`, `-w` , `-b` .

raw_content:
  ---
  tags:
    - shell
  ---

  # Text manipulation

  ## Sorting strings: `sort`

  If you have a `.txt` file containing text strings, each on a new line you can
  use the sort function to quickly put them in alphabetical order:

  ```bash
  sort file.txt
  ```

  Note that this will not save the sort, it only presents it as a standard output.
  To save the sort you need to direct the sort to a file in the standard way:

  ```bash
  sort file.txt > output.txt
  ```

  ### Options

  - `-r`
    - reverse sort
  - `c`
    - check if file is already sorted. If not, it will highlight the strings which
      are not sorted

  ## Find and replace: `sed`

  The `sed` programme can be used to implement find and replace procedures. In
  `sed`, find and replace are covered by the substitution option: `/s` :

  ```bash
  sed ‘s/word/replacement word/’ file.txt
  ```

  This however will only change the first instance of word to be replaced, in
  order to apply to every instance you need to add the global option: `-g` .

  As sed is a stream editor, any changes you make using it, will only occur within
  the standard output , they will not be saved to file. In order to save to file
  you need to specify a new file output (using `> output.txt`) in addition to the
  original file. This hasthe benefit of leaving the original file untouched whilst
  ensuring the desired outcome is stored permanently.

  Alternatively, you can use the `-i` option which will make the changes take
  place in the source file as well as in standard input.

  Note that this will overwrite the original version of the file and it cannot be
  regained. If this is an issue then it is recommended to include a backup command
  in the overall argument like so:

  ```bash
  sed -i.bak ‘s/word/replacement word/’ file.txt
  ```

  This will create the file `file.txt.bak` in the directory you are working within
  which is the original file before the replacement was carried out.

  ### Remove duplicates

  We can use the `sort -u` command can be used to remove duplicates:

  ```bash
  sort -u file.txt
  ```

  It is important to sort before attempting to remove duplicates since the `-u`
  flag works on the basis of the strings being adjacent.

  ## Split a large file into multiple smaller files: `split`

  Suppose you have a file containing 1000 lines. You want to break the file up
  into five separate files, each containing two hundred lines. You can use `split`
  to accomplish this, like so:

  ```bash
  split -l 200 big-file.txt new-files
  ```

  `split` will categorise the resulting five files as follows:

  - new-file-aa,
  - new-file-ab
  - new-file-ac,
  - newfile-ad,
  - new-file-ae.

  If you would rather have numeric suffixes, use the option `-d` . You can also
  split a file by its number of bytes, using the option `-b` and specifying a
  constituent file size.

  ## Merge multiple files into one with `cat`

  We can use `cat` read multiple files at once and then append a redirect to save
  them to a file:

  ```bash
  cat file_a.txt file_b.txt file_c.txt > merged-file.txt
  ```

  ## Count lines, words, etc: `wc`

  To count words:

  ```bash
  wc file.txt
  ```

  When we use the command three numbers are outputted, in order: lines, words,
  bytes.

  You can use modifiers to get just one of the numbers: `-l`, `-w` , `-b` .


path: zk/The_kernel.md
sortable_path: zkThe_kernel.md
title: The kernel
checksum: 96c9ce4e7d5964c10b9af81b81e350210f24146bc8a16245169ff057cdb3dd4f
created: 2024-06-16 17:37:31.958371212+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"tags":["systems-programming","memory","operating-systems","computer-architecture"]}
lead:
  Within the [hierarchy of the OS](./Basic_model_of_the_operating_system.md), the
  kernel acts as the primary mediator between the hardware (CPU, memory) and
  [user](./User_Space.md) [processes](Processes.md). Let's look at each of its
  responsibilities in greater depth:

body:
  Within the [hierarchy of the OS](./Basic_model_of_the_operating_system.md), the
  kernel acts as the primary mediator between the hardware (CPU, memory) and
  [user](./User_Space.md) [processes](Processes.md). Let's look at each of its
  responsibilities in greater depth:

  - process management
  - memory management
  - device drivers
  - system calls

  ## Process management

  > A process is just another name for a running program. Process management is
  > the starting, pausing, resuming, scheduling and terminating of processes.

  On modern computers it appears that multiple processes can run simultaneously at
  once. This is only because the processor is so fast that we do not detect
  changes. In fact access to the CPU is always sequential. The sequence in which
  multiple programs are allowed to access the CPU is managed by the kernel.

  > Consider a system with a one-core CPU. Many processes may be _able_ to use the
  > CPU, but only one process can actually use the CPU at any given time...Each
  > process uses the CPU for a fraction of a second, then pauses, then another
  > process uses it for a fraction of a second and so on... (_How Linux Works:
  > Third Edition_, Brian Ward 2021)

  This process of the CPU shuffling between multiple processes is called _context
  switching_.

  The role of the kernel in facilitating this, is as follows:

  1. CPU runs process for a time slice based on its internal time. Then hands
     control back to the kernel (kernel mode)
  2. Kernel records current state of CPU and memory. This is necessary in order to
     resume the progress that was just interupted.
  3. The kernel executes any tasks that arose in the last timeslice executed by
     the CPU (e.g. collecting data from I/0)
  4. Kernel then analyses the list of processes that are ready to run next and
     chooses one.
  5. Kernel prepares memory for this new process and prepares the CPU.
  6. Kernel tells CPU how long the time slice for the new process will last.
  7. Kernel switches the CPU into user mode and hands control of CPU to the
     process.

  ## Memory management

  During the context switch from CPU to user space, the kernel allocates memory.
  It has the following jobs to manage:

  - Keeping its own private area in memory for itself that user processes cannot
    access
  - Assigning each user process its own section of memory
  - Managing shared memory between processes and ensuring the private memory of
    processes is not accessed by others
  - Managing read-only memory
  - Allowing for the use of disk space as auxiliary memory

  > Modern CPUs include a
  > [memory management unit](Virtual_memory_and_the_MMU_in_Linux.md#the-memory-management-unit-mmu)
  > which provides the kernel with **virtual** memory. In this scenario, memory
  > isn't directly accessed by the process instead it works on the assumption that
  > is has access to the entire memory of the machine and this is then translated
  > into a map that is applied to the real memory and managed by the kernel.

  ## Device drivers

  Devices are managed by the kernal and are not accessible directly via user
  space, since improper usage could crash the whole machine. There is little
  uniformity between devices and as a result drivers are needed. Thes are kernl
  code that enable different OS kernels to access and control the devices.

  ## System calls

  Syscalls are what enable programs to start and are required for the acts of
  opening, reading and writing files. System calls in Linux are typically managed
  via C.

  In Linux there are two particularly important system calls:

  - `fork()`
    - When a process calls fork, the kernel creates a nearly identical copy of
      this running process
  - `exec()`
    - When a process calls exec it passes a program name as a parameter. Then the
      kernel loads and starts this program, replacing the current process.

  Example with a terminal program like `ls`:

  > When you enter `ls` into the terminal window, the shell that's running inside
  > the terminal window calls `fork()` to create a copy of the shell, and then the
  > new copy of the shell calls `exec(ls)` to run `ls`. (_Ibid._)

  ## Controlling processes

  In Linux we can view, kill, pause and resume processes using [ps](Processes.md).

raw_content:
  ---
  tags:
    - systems-programming
    - memory
    - operating-systems
    - computer-architecture
  ---

  # The kernel

  Within the [hierarchy of the OS](./Basic_model_of_the_operating_system.md), the
  kernel acts as the primary mediator between the hardware (CPU, memory) and
  [user](./User_Space.md) [processes](Processes.md). Let's look at each of its
  responsibilities in greater depth:

  - process management
  - memory management
  - device drivers
  - system calls

  ## Process management

  > A process is just another name for a running program. Process management is
  > the starting, pausing, resuming, scheduling and terminating of processes.

  On modern computers it appears that multiple processes can run simultaneously at
  once. This is only because the processor is so fast that we do not detect
  changes. In fact access to the CPU is always sequential. The sequence in which
  multiple programs are allowed to access the CPU is managed by the kernel.

  > Consider a system with a one-core CPU. Many processes may be _able_ to use the
  > CPU, but only one process can actually use the CPU at any given time...Each
  > process uses the CPU for a fraction of a second, then pauses, then another
  > process uses it for a fraction of a second and so on... (_How Linux Works:
  > Third Edition_, Brian Ward 2021)

  This process of the CPU shuffling between multiple processes is called _context
  switching_.

  The role of the kernel in facilitating this, is as follows:

  1. CPU runs process for a time slice based on its internal time. Then hands
     control back to the kernel (kernel mode)
  2. Kernel records current state of CPU and memory. This is necessary in order to
     resume the progress that was just interupted.
  3. The kernel executes any tasks that arose in the last timeslice executed by
     the CPU (e.g. collecting data from I/0)
  4. Kernel then analyses the list of processes that are ready to run next and
     chooses one.
  5. Kernel prepares memory for this new process and prepares the CPU.
  6. Kernel tells CPU how long the time slice for the new process will last.
  7. Kernel switches the CPU into user mode and hands control of CPU to the
     process.

  ## Memory management

  During the context switch from CPU to user space, the kernel allocates memory.
  It has the following jobs to manage:

  - Keeping its own private area in memory for itself that user processes cannot
    access
  - Assigning each user process its own section of memory
  - Managing shared memory between processes and ensuring the private memory of
    processes is not accessed by others
  - Managing read-only memory
  - Allowing for the use of disk space as auxiliary memory

  > Modern CPUs include a
  > [memory management unit](Virtual_memory_and_the_MMU_in_Linux.md#the-memory-management-unit-mmu)
  > which provides the kernel with **virtual** memory. In this scenario, memory
  > isn't directly accessed by the process instead it works on the assumption that
  > is has access to the entire memory of the machine and this is then translated
  > into a map that is applied to the real memory and managed by the kernel.

  ## Device drivers

  Devices are managed by the kernal and are not accessible directly via user
  space, since improper usage could crash the whole machine. There is little
  uniformity between devices and as a result drivers are needed. Thes are kernl
  code that enable different OS kernels to access and control the devices.

  ## System calls

  Syscalls are what enable programs to start and are required for the acts of
  opening, reading and writing files. System calls in Linux are typically managed
  via C.

  In Linux there are two particularly important system calls:

  - `fork()`
    - When a process calls fork, the kernel creates a nearly identical copy of
      this running process
  - `exec()`
    - When a process calls exec it passes a program name as a parameter. Then the
      kernel loads and starts this program, replacing the current process.

  Example with a terminal program like `ls`:

  > When you enter `ls` into the terminal window, the shell that's running inside
  > the terminal window calls `fork()` to create a copy of the shell, and then the
  > new copy of the shell calls `exec(ls)` to run `ls`. (_Ibid._)

  ## Controlling processes

  In Linux we can view, kill, pause and resume processes using [ps](Processes.md).


path: zk/Troubleshooting_MongoDB.md
sortable_path: zkTroubleshooting_MongoDB.md
title: Troubleshooting MongoDB connection issues
lead: ## Arch Linux troubleshooting
checksum: 764b19f1d36a6c0d05f28bf1f30d52c829fe3030cef31f8e3c8efabf28f1f4c3
created: 2024-06-16 17:37:32.020694929+00:00
modified: 2024-10-18 17:55:56.292394043+00:00
metadata: {"tags":["mongo-db"]}
body:
  ## Arch Linux troubleshooting

  Most times any problems will be a result of a Mongo process that is already
  running. Resolve with:

  ```bash
  # Kill running Mongo process...
  # Identify PID
  sudo lsof -iTCP -sTCP:LISTEN -n -P
  sudo kill [pid]
  ```

  Otherwise try the below.

  ```bash
  # Check that the systemd service is runnign for Mongo
  sudo systemctl status mongodb

  # If not, enable and start service (only need to do this once for machine)
  sudo systemctl enable --now mongodb
  sudo systemctl start --now mongodb

  # If issues with Mongo daemon...
  # Remove the current socket
  rm /tmp/mongodb-27017.sock
  ```

  For additional command line utilities you will also need to install
  `mongodb-tools` from the AUR.

raw_content:
  ---
  tags: [mongo-db]
  ---

  # Troubleshooting MongoDB connection issues

  ## Arch Linux troubleshooting

  Most times any problems will be a result of a Mongo process that is already
  running. Resolve with:

  ```bash
  # Kill running Mongo process...
  # Identify PID
  sudo lsof -iTCP -sTCP:LISTEN -n -P
  sudo kill [pid]
  ```

  Otherwise try the below.

  ```bash
  # Check that the systemd service is runnign for Mongo
  sudo systemctl status mongodb

  # If not, enable and start service (only need to do this once for machine)
  sudo systemctl enable --now mongodb
  sudo systemctl start --now mongodb

  # If issues with Mongo daemon...
  # Remove the current socket
  rm /tmp/mongodb-27017.sock
  ```

  For additional command line utilities you will also need to install
  `mongodb-tools` from the AUR.


path: zk/Type_guarding_and_narrowing_in_TS.md
sortable_path: zkType_guarding_and_narrowing_in_TS.md
title: Type narrowing and guarding
checksum: 490017e14d24edb1d0dbb214c845e7c0f56679e29856822fd555a7f4d46e88b5
created: 2024-06-16 17:37:32.164007308+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"tags":["typescript"]}
lead:
  Type narrowing is the process of working out from a supertype like
  [any](Any.md) or [unknown](Unknown_type_in_TS.md) whic type the value should be in the
  x course of your code. This is necessary since a process will not necessarily
  always return or involve homogenous types.

body:
  Type narrowing is the process of working out from a supertype like
  [any](Any.md) or [unknown](Unknown_type_in_TS.md) whic type the value should be in the
  x course of your code. This is necessary since a process will not necessarily
  always return or involve homogenous types.

  Related to this is type guarding: ensuring that a value is of the suitable type
  as a factor of control flow. For instance using typeof to ensure that an input
  is numerical before proceeding with a function's logic.

raw_content:
  ---
  tags:
    - typescript
  ---

  # Type narrowing and guarding

  Type narrowing is the process of working out from a supertype like
  [any](Any.md) or [unknown](Unknown_type_in_TS.md) whic type the value should be in the
  x course of your code. This is necessary since a process will not necessarily
  always return or involve homogenous types.

  Related to this is type guarding: ensuring that a value is of the suitable type
  as a factor of control flow. For instance using typeof to ensure that an input
  is numerical before proceeding with a function's logic.


path: zk/Union_types_in_TS.md
sortable_path: zkUnion_types_in_TS.md
title: Union types
checksum: c821ae262bc36f541ebebc61d22c3d5bd95bb51fa3dd7b44522c80fe2956ee3f
created: 2024-06-16 17:37:32.182332489+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"tags":["typescript"]}
lead:
  Like intersection types, union types allow us to combine the properties of
  existing primitive and custom types to make composites.

body:
  Like intersection types, union types allow us to combine the properties of
  existing primitive and custom types to make composites.

  Intersection types work well when you have complete knowledge of what form your
  data will take. This approach only works if you know in advance that your data
  will be and that it will share properties from more than one type. But what if
  you don't have this knowledge? You only know that it will be one of the three
  possible types? This is where union types become helpful. Instead of saying that
  event X will be of type A we say that event X may be one of types A or B or C.

  > A union type is a type formed from two or more other types, representing
  > values that may be any one of those types. We refer to each of these types as
  > the union’s members.

  ## Demonstration

  ```ts
  type Petrol = {
    maxSpeed: number;
    tankVolume: number; // unique property
  };

  type Electric = {
    maxSpeed: number;
    maxCharge: number; // unique property
  };

  type Car = Petrol | Electric;

  const tesla: Car = {
    maxSpeed: 120,
    maxCharge: 22,
  };
  ```

  - We have defined two custom types: `Petrol` and `Electric`
  - We have unified them to make a new custom type of `Car`
  - `tesla` is an instance of the union type

  ## Usage with functions

  Union type syntax is not limited to the declaration phase. You can also use them
  as parameters to functions or function return types:

  ```ts
  function generateCar(car: Petrol | Electric): void {
    console.log(`The car has a max speed of ${car.maxSpeed}`);
  }
  ```

  ## Unifying primitive and custom types

  We can also use unions with primitive types as well as custom types:

  ```ts
  function printId(id: number | string): void {
    console.log(`Your id is ${id}`);
  }
  ```

  We can also combine primitive and custom types in the same union:

  ```ts
  function printStringOrNumber(): string | Petrol {}
  ```

  ## Set theory: understanding errors

  In JavaScript we have the primitive types of string , number , boolean etc. In
  TS, these types are distinct and exclusive: if you declare that x : string you
  cannot redefine it as boolean or expect Boolean properties and methods to belong
  to a string type. For instance a string type is incapable of having as its value
  false.

  This is not true in the case of unions.

  With a union of `string | boolean` a value `x` could be `'banana'` or `true`. We
  are allowing for all values that are either from the set string or the set
  boolean. Logically this is equivalent to the OR operator: a value of type
  `string | boolean` could be one of infinite possible strings or true/false. With
  union types then, we radically widen the scope of possible values for the type.
  a

  These constraints don't just apply to the values themselves but the methods and
  properties you can use with the values. For instance a boolean doesn't have the
  property of `length`. Therefore if you have a union type of `string|boolean` you
  are going to get an error if you apply `length` to it since **the property must
  exist on both types that comprise the union**. The way round this is to use type
  narrowing and execute control flow based on the parameter that is passed to a
  function that is declared to be of this mixed type.

  Here is an example of this:

  ```ts
  function printId(id: number | string) {
    if (typeof id === "string") {
      // In this branch, id is of type 'string'
      console.log(id.toUpperCase());
    } else {
      // Here, id is of type 'number'
      console.log(id);
    }
  }
  ```

raw_content:
  ---
  tags:
    - typescript
  ---

  # Union types

  Like intersection types, union types allow us to combine the properties of
  existing primitive and custom types to make composites.

  Intersection types work well when you have complete knowledge of what form your
  data will take. This approach only works if you know in advance that your data
  will be and that it will share properties from more than one type. But what if
  you don't have this knowledge? You only know that it will be one of the three
  possible types? This is where union types become helpful. Instead of saying that
  event X will be of type A we say that event X may be one of types A or B or C.

  > A union type is a type formed from two or more other types, representing
  > values that may be any one of those types. We refer to each of these types as
  > the union’s members.

  ## Demonstration

  ```ts
  type Petrol = {
    maxSpeed: number;
    tankVolume: number; // unique property
  };

  type Electric = {
    maxSpeed: number;
    maxCharge: number; // unique property
  };

  type Car = Petrol | Electric;

  const tesla: Car = {
    maxSpeed: 120,
    maxCharge: 22,
  };
  ```

  - We have defined two custom types: `Petrol` and `Electric`
  - We have unified them to make a new custom type of `Car`
  - `tesla` is an instance of the union type

  ## Usage with functions

  Union type syntax is not limited to the declaration phase. You can also use them
  as parameters to functions or function return types:

  ```ts
  function generateCar(car: Petrol | Electric): void {
    console.log(`The car has a max speed of ${car.maxSpeed}`);
  }
  ```

  ## Unifying primitive and custom types

  We can also use unions with primitive types as well as custom types:

  ```ts
  function printId(id: number | string): void {
    console.log(`Your id is ${id}`);
  }
  ```

  We can also combine primitive and custom types in the same union:

  ```ts
  function printStringOrNumber(): string | Petrol {}
  ```

  ## Set theory: understanding errors

  In JavaScript we have the primitive types of string , number , boolean etc. In
  TS, these types are distinct and exclusive: if you declare that x : string you
  cannot redefine it as boolean or expect Boolean properties and methods to belong
  to a string type. For instance a string type is incapable of having as its value
  false.

  This is not true in the case of unions.

  With a union of `string | boolean` a value `x` could be `'banana'` or `true`. We
  are allowing for all values that are either from the set string or the set
  boolean. Logically this is equivalent to the OR operator: a value of type
  `string | boolean` could be one of infinite possible strings or true/false. With
  union types then, we radically widen the scope of possible values for the type.
  a

  These constraints don't just apply to the values themselves but the methods and
  properties you can use with the values. For instance a boolean doesn't have the
  property of `length`. Therefore if you have a union type of `string|boolean` you
  are going to get an error if you apply `length` to it since **the property must
  exist on both types that comprise the union**. The way round this is to use type
  narrowing and execute control flow based on the parameter that is passed to a
  function that is declared to be of this mixed type.

  Here is an example of this:

  ```ts
  function printId(id: number | string) {
    if (typeof id === "string") {
      // In this branch, id is of type 'string'
      console.log(id.toUpperCase());
    } else {
      // Here, id is of type 'number'
      console.log(id);
    }
  }
  ```


path: zk/Unknown_type_in_TS.md
sortable_path: zkUnknown_type_in_TS.md
title: Unknown
checksum: 8c0f68fad261ac22cf13cf968e917e2308e19247798725d27b71e2be1a1a3da2
created: 2024-06-16 17:37:32.188926698+00:00
modified: 2024-10-18 17:55:56.292394043+00:00
metadata: {"tags":["typescript"]}
lead:
  You might think that a good use case for `any` is a scenario where you don't
  know in advance what the data type is going to be.

body:
  You might think that a good use case for `any` is a scenario where you don't
  know in advance what the data type is going to be.

  In order to mark that this is the case, you type the value as `any`. In fact,
  TypeScript provides a type for this specific scenario: `unknown`.

  Like `any`, `unknown` is equivalent to every type in TS (it is also a supertype)
  but it is deliberately inhibiting, in contrast to `any`. When you use `unknown`
  you have to use type narrowing to avoid it being returned. So if your code
  starts with `unknown`, you should filter out scenarios where it evaluates to
  each of the possible types otherwise if `unknown` is returned it will cause an
  error.

raw_content:
  ---
  tags:
    - typescript
  ---

  # Unknown

  You might think that a good use case for `any` is a scenario where you don't
  know in advance what the data type is going to be.

  In order to mark that this is the case, you type the value as `any`. In fact,
  TypeScript provides a type for this specific scenario: `unknown`.

  Like `any`, `unknown` is equivalent to every type in TS (it is also a supertype)
  but it is deliberately inhibiting, in contrast to `any`. When you use `unknown`
  you have to use type narrowing to avoid it being returned. So if your code
  starts with `unknown`, you should filter out scenarios where it evaluates to
  each of the possible types otherwise if `unknown` is returned it will cause an
  error.


path: zk/User_management_in_Linux.md
sortable_path: zkUser_management_in_Linux.md
title: User management
lead: ## Switch user
checksum: e5c96a6157409e2d25e0f6278475b4bb97323774450952af78f3ad3ea1369622
created: 2024-06-16 17:37:32.196102048+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"tags":["Linux","procedural"]}
body:
  ## Switch user

  If already logged in as a user you can switch users with the command
  `su - [username]`.

  ## Login as root

  If you are logged in as a standard user, if you use `su -` without specifying a
  username, then it will assume you wish to log in as root.

  If you wish to login as root in the tty at startup, then use `root` as the
  username.

raw_content:
  ---
  tags:
    - Linux
    - procedural
  ---

  # User management

  ## Switch user

  If already logged in as a user you can switch users with the command
  `su - [username]`.

  ## Login as root

  If you are logged in as a standard user, if you use `su -` without specifying a
  username, then it will assume you wish to log in as root.

  If you wish to login as root in the tty at startup, then use `root` as the
  username.


path: zk/Utilities_operators_flags.md
sortable_path: zkUtilities_operators_flags.md
title: Utilities, operators, flags in Bash
checksum: 05c70aee147aa773a0a5398a3250697f6d1ff38a8998eeb9f614efd16dbb7e6c
created: 2024-06-16 17:37:32.208963277+00:00
modified: 2024-10-19 09:01:23.725350032+00:00
metadata: {"tags":["shell"]}
lead:
  The following are useful built-in utility methods that you can use for checking
  and validation in the course of your bash scripts.

body:
  The following are useful built-in utility methods that you can use for checking
  and validation in the course of your bash scripts.

  ## Flags

  ### Prevent newline

  Prevent bash from adding a new line after an echo:

  ```bash
  echo 'Your name is Thomas'
  echo 'and you are 33 years old'

  # Your name is Thomas
  # and you are 33 years old
  ```

  ```bash
  echo -n 'Your name is Thomas '
  echo 'and you are 33 years old'

  # Your name is Thomas and you are 33 years old
  ```

  ## Operators

  ### Mathematical

  ```bash
  -lt , -gt,
  ```

raw_content:
  ---
  tags:
    - shell
  ---

  # Utilities, operators, flags in Bash

  The following are useful built-in utility methods that you can use for checking
  and validation in the course of your bash scripts.

  ## Flags

  ### Prevent newline

  Prevent bash from adding a new line after an echo:

  ```bash
  echo 'Your name is Thomas'
  echo 'and you are 33 years old'

  # Your name is Thomas
  # and you are 33 years old
  ```

  ```bash
  echo -n 'Your name is Thomas '
  echo 'and you are 33 years old'

  # Your name is Thomas and you are 33 years old
  ```

  ## Operators

  ### Mathematical

  ```bash
  -lt , -gt,
  ```


path: zk/Validation_in_NodeJS.md
sortable_path: zkValidation_in_NodeJS.md
title: Creating a RESTful API: Validation
checksum: 6f0c4e7f0af95a7522c442d4991bd06ab0a817dac6eabb19ed81afb036089759
created: 2024-06-16 17:37:32.228913914+00:00
modified: 2024-10-18 17:55:56.292394043+00:00
metadata: {"tags":["node-js","REST","APIs"]}
lead:
  We can provide server-side validation for our projects by using a **schema
  validator**. This is a program that declaratively parses the JSON values
  received as requests from the client. This makes it easy to systematically
  validate the data that we receive from any HTTP requests where the client sends
  a body to the endpoint.

body:
  We can provide server-side validation for our projects by using a **schema
  validator**. This is a program that declaratively parses the JSON values
  received as requests from the client. This makes it easy to systematically
  validate the data that we receive from any HTTP requests where the client sends
  a body to the endpoint.

  One of the most popular schema validators for NodeJS is
  [joi](joi).

  ## Demonstration

  Let's say we have a POST request that expects a single field as the body that
  must be a string and greater than two characters long. First we define our
  schema:

  ```js
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });

  const { error } = schema.validate(req.body);
  ```

  The `schema` variable is an object whose keys should match those of the intended
  request body. Instead of actual values we provide Joi's in-built validators,
  concatenated as necessary. We then store the results of the validation in a
  variable.

  Next we add handling in the case of errors:

  ```js
  if (error) {
    error.details.map((joiErr) => res.status(400).send(joiErr.message));
    return;
  }
  ```

  We loop through the error array and return 400s as the response if they are
  found. If there are no errors, the Joi object will return `undefined`.

raw_content:
  ---
  tags:
    - node-js
    - REST
    - APIs
  ---

  # Creating a RESTful API: Validation

  We can provide server-side validation for our projects by using a **schema
  validator**. This is a program that declaratively parses the JSON values
  received as requests from the client. This makes it easy to systematically
  validate the data that we receive from any HTTP requests where the client sends
  a body to the endpoint.

  One of the most popular schema validators for NodeJS is
  [joi](joi).

  ## Demonstration

  Let's say we have a POST request that expects a single field as the body that
  must be a string and greater than two characters long. First we define our
  schema:

  ```js
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });

  const { error } = schema.validate(req.body);
  ```

  The `schema` variable is an object whose keys should match those of the intended
  request body. Instead of actual values we provide Joi's in-built validators,
  concatenated as necessary. We then store the results of the validation in a
  variable.

  Next we add handling in the case of errors:

  ```js
  if (error) {
    error.details.map((joiErr) => res.status(400).send(joiErr.message));
    return;
  }
  ```

  We loop through the error array and return 400s as the response if they are
  found. If there are no errors, the Joi object will return `undefined`.


path: zk/Variable_indirection_in_Bash.md
sortable_path: zkVariable_indirection_in_Bash.md
title: Variable indirection
lead: > Write proper notes on this
checksum: 6682743efcc1a97b2ba0760899b1690e9af193e72e3df5a957904ab8e01726a5
created: 2024-06-16 17:37:32.243804241+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"tags":["shell"]}
body:
  > Write proper notes on this

  ```sh
  function array_empty() {
      declare -n arr=$1
      # Proceed if array not empty:
      if [ ${#arr[@]} -gt 0 ]; then
          return 1 # array is not empty
      else
          return 0 # array is empty
      fi
  }

  my_array=(1 2 3)

  function push() {
      # $1 = stack name
      local stack_name="$1"
      if [ "${stack_name}" ]; then # stack exists
          if array_empty "${!stack_name}"; then
              echo "stack is empty"
          else
              echo "stack is not empty"
          fi
      else
          echo "Error: ${stack_name} does not exist."
      fi
  }

  push "my_array"

  ```

  In Bash scripting, ${!stack_name} is an example of indirect variable referencing
  or variable indirection. It's a way to use a variable whose name is stored in
  another variable.

  Let's break it down:

  `${stack_name}`: This syntax is used to reference the value of a variable. So if
  stack_name="my_array", then ${stack_name} would return "my_array".

  `${!stack_name}`: Adding the ! introduces indirection. Now, instead of getting
  the value of stack_name, you get the value of the variable whose name is stored
  in stack_name. So if stack_name="my_array" and my_array=(1 2 3), then
  ${!stack_name} would return (1 2 3).

  So, in the context of your script, `${!stack_name} allows you to pass the name
  of an array (as a string) to your push function, and then use that string to
  indirectly reference the actual array within the function.

raw_content:
  ---
  tags:
    - shell
  ---

  # Variable indirection

  > Write proper notes on this

  ```sh
  function array_empty() {
      declare -n arr=$1
      # Proceed if array not empty:
      if [ ${#arr[@]} -gt 0 ]; then
          return 1 # array is not empty
      else
          return 0 # array is empty
      fi
  }

  my_array=(1 2 3)

  function push() {
      # $1 = stack name
      local stack_name="$1"
      if [ "${stack_name}" ]; then # stack exists
          if array_empty "${!stack_name}"; then
              echo "stack is empty"
          else
              echo "stack is not empty"
          fi
      else
          echo "Error: ${stack_name} does not exist."
      fi
  }

  push "my_array"

  ```

  In Bash scripting, ${!stack_name} is an example of indirect variable referencing
  or variable indirection. It's a way to use a variable whose name is stored in
  another variable.

  Let's break it down:

  `${stack_name}`: This syntax is used to reference the value of a variable. So if
  stack_name="my_array", then ${stack_name} would return "my_array".

  `${!stack_name}`: Adding the ! introduces indirection. Now, instead of getting
  the value of stack_name, you get the value of the variable whose name is stored
  in stack_name. So if stack_name="my_array" and my_array=(1 2 3), then
  ${!stack_name} would return (1 2 3).

  So, in the context of your script, `${!stack_name} allows you to pass the name
  of an array (as a string) to your push function, and then use that string to
  indirectly reference the actual array within the function.


path: zk/Variables_and_data_types_in_Bash.md
sortable_path: zkVariables_and_data_types_in_Bash.md
title: Variables and datatypes in Bash
lead: ## Types
checksum: fc6431165e27bec3e2955dbf547f4094db1ea5c48191596be901043599a966a2
created: 2024-06-16 17:37:32.248470584+00:00
modified: 2024-10-19 09:01:58.29266654+00:00
metadata: {"tags":["shell"]}
body:
  ## Types

  ## Variables

  We use the equality symbol to create a variable:

  ```bash
  stringVar="My first variable"
  floatVar="50.3"
  boolVar="true"
  ```

  As there is no typing in bash, the names of these variables are purely notional.

  To invoke a variable we use special brackets:

  ```bash
  echo ${stringVar} # My first variable
  echo ${floatVar} # 50.3
  echo ${boolVar} # true
  ```

  - Quotation marks at declaration are also not strictly necessary however they
    can help avoid bugs. Also serves as a reminder that every type is basically a
    string at the end of the day

  ## Common operations

  ### Check if a variable exists

  ```sh
  if [ -v variable_name ]; then
      echo "The variable exists."
  else
      echo "The variable does not exist."
  fi
  ```

  ### Check if a variable is set (defined)

  ```sh
  if [ -z "${variable_name+x}" ]; then
      echo "The variable does not exist."
  else
      echo "The variable exists."
  fi
  ```

  ## Parameter expansion

  Still not very clear on this.

raw_content:
  ---
  tags:
    - shell
  ---

  # Variables and datatypes in Bash

  ## Types

  ## Variables

  We use the equality symbol to create a variable:

  ```bash
  stringVar="My first variable"
  floatVar="50.3"
  boolVar="true"
  ```

  As there is no typing in bash, the names of these variables are purely notional.

  To invoke a variable we use special brackets:

  ```bash
  echo ${stringVar} # My first variable
  echo ${floatVar} # 50.3
  echo ${boolVar} # true
  ```

  - Quotation marks at declaration are also not strictly necessary however they
    can help avoid bugs. Also serves as a reminder that every type is basically a
    string at the end of the day

  ## Common operations

  ### Check if a variable exists

  ```sh
  if [ -v variable_name ]; then
      echo "The variable exists."
  else
      echo "The variable does not exist."
  fi
  ```

  ### Check if a variable is set (defined)

  ```sh
  if [ -z "${variable_name+x}" ]; then
      echo "The variable does not exist."
  else
      echo "The variable exists."
  fi
  ```

  ## Parameter expansion

  Still not very clear on this.


path: zk/Variables_in_C.md
sortable_path: zkVariables_in_C.md
title: Variables in C
lead:
  ```c
  int age = 25
  ```

body:
  ```c
  int age = 25
  ```

checksum: 5a795d624f330f27608912116a4e66a003e6534c585d38e5adfe4b8d7df0ee99
created: 2024-06-16 17:37:32.252810994+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Thursday, February 29, 2024 | 17:20","id":"r4i16p1x","tags":["C"]}
raw_content:
  ---
  id: r4i16p1x
  tags: [C]
  created: Thursday, February 29, 2024 | 17:20
  ---

  # Variables in C

  ```c
  int age = 25
  ```


path: zk/Virtual_private_cloud.md
sortable_path: zkVirtual_private_cloud.md
title: Virtual Private Cloud
checksum: 34438b2d0b9aad65bd3698fdee839c0a140c5900550d8ed3f306cb75f0cd3306
created: 2024-06-16 17:37:32.272794495+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"created":"Tuesday, February 20, 2024 | 08:31","id":"mdw5fe5a","tags":["AWS","networks"]}
lead:
  AWS VPC is used to create a virtual network. This is typically used in concert
  with [[zk/Elastic_Compute_Cloud]] to manage connections to a virtual server both
  privately and accross the internet.

body:
  AWS VPC is used to create a virtual network. This is typically used in concert
  with [[zk/Elastic_Compute_Cloud]] to manage connections to a virtual server both
  privately and accross the internet.

  You define a network address range and then create subnets for managing
  different connections and functionality. You use a public subnet for resources
  that must be connected to the internet and a private subnet for resources that
  are to remain isolated from the internet.

  The diagram below details a basic VPC configuration:

  ![](../img/aws-vpc-example.png)

  - Within a given AWS region we have created a VPC network.
  - This comprises public and private subnets
  - Both subnets host an EC2 instance
  - The public subnet has a bridge to the internet through the Internet Gateway
  - Both subnets have a routing table to manage requests and access

raw_content:
  ---
  id: mdw5fe5a
  tags: [AWS, networks]
  created: Tuesday, February 20, 2024 | 08:31
  ---

  # Virtual Private Cloud

  AWS VPC is used to create a virtual network. This is typically used in concert
  with [[zk/Elastic_Compute_Cloud]] to manage connections to a virtual server both
  privately and accross the internet.

  You define a network address range and then create subnets for managing
  different connections and functionality. You use a public subnet for resources
  that must be connected to the internet and a private subnet for resources that
  are to remain isolated from the internet.

  The diagram below details a basic VPC configuration:

  ![](../img/aws-vpc-example.png)

  - Within a given AWS region we have created a VPC network.
  - This comprises public and private subnets
  - Both subnets host an EC2 instance
  - The public subnet has a bridge to the internet through the Internet Gateway
  - Both subnets have a routing table to manage requests and access


path: zk/http_in_Node.md
sortable_path: zkhttp_in_Node.md
title: http module
checksum: d74610fc301d67759fe76291d7e99eda1b918720fe38a694e5d28775c0b7ee25
created: 2024-06-16 17:37:32.313365813+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"tags":["node-js"]}
lead:
  The HTTP Module allows us to create a web server that listens for HTTP requests
  on a given port. It is therefore perfect for creating backends for client-side
  JavaScript.

body:
  The HTTP Module allows us to create a web server that listens for HTTP requests
  on a given port. It is therefore perfect for creating backends for client-side
  JavaScript.

  ## Creating a server

  An HTTP server is another instance of an
  [event emitter](events.md)). It
  therefore has all the same methods as the `EventEmitter` class: `on`, `emit`,
  `addListener` etc. This demonstrates again how much of Node's core functionality
  is based on event emitters.

  _Creating a server_

  ```js
  const http = require("http");

  const server = http.createServer(); // Create server as emitter

  // Register functions to run when listener is triggered
  server.on("connection", (socket) => {
    console.log("new connection...");
  });

  server.listen(3000);
  console.log("Listening on port 3000");
  ```

  This server is functionally equivalent to a generic event emitter:

  ```js
  // Raise an event
  const emitter = new EventEmitter("messageLogged");

  // Register a listener
  emitter.on("messagedLogged", function () {
    console.log("The listener was called.");
  });
  ```

  Whenever a request is made to this server, it raises an event. We can therefore
  target it with the `on` method and make it execute a function when requests are
  made.

  If we were to start the server by running the file and we then used a browser to
  navigate to the port, we would see `new connection` logged every time we refresh
  the page.

  ### Sockets and `req, res`

  A socket is a generic protocol for client-server communication. Crucially it
  **allows simultaneous communication both ways**. The client can contact the
  server but the server can also contact the client. Our listener function above
  uses a socket as the callback function but in most cases this is quite
  low-level, not distinguishing responses from requests. It is more likely that
  you would initiate a `request, resource` architecture in place of a socket:

  ```js
  const server = http.createServer((req, res) => {
    if (req.url === "/") {
      res.write("hello");
      res.end();
    }
  });
  ```

  #### Return JSON

  Below is an example of using this architecture to return JSON to the client:

  ```js
  const server = http.createServer((req, res) => {
    if (req.url === "/products") {
      res.write(JSON.stringify(["shoes", "lipstick", "cups"]));
      res.end();
    }
  });
  ```

  ### Express

  In reality you would rarely use the `http` module directly to create a server.
  This is because it is quite low level and each response must be written in a
  linear fashion as with the two URLs in the previous example. Instead we use
  Express which is a framework for creating servers and routing that is an
  abstraction on top of the core HTTP module.

raw_content:
  ---
  tags:
    - node-js
  ---

  # `http` module

  The HTTP Module allows us to create a web server that listens for HTTP requests
  on a given port. It is therefore perfect for creating backends for client-side
  JavaScript.

  ## Creating a server

  An HTTP server is another instance of an
  [event emitter](events.md)). It
  therefore has all the same methods as the `EventEmitter` class: `on`, `emit`,
  `addListener` etc. This demonstrates again how much of Node's core functionality
  is based on event emitters.

  _Creating a server_

  ```js
  const http = require("http");

  const server = http.createServer(); // Create server as emitter

  // Register functions to run when listener is triggered
  server.on("connection", (socket) => {
    console.log("new connection...");
  });

  server.listen(3000);
  console.log("Listening on port 3000");
  ```

  This server is functionally equivalent to a generic event emitter:

  ```js
  // Raise an event
  const emitter = new EventEmitter("messageLogged");

  // Register a listener
  emitter.on("messagedLogged", function () {
    console.log("The listener was called.");
  });
  ```

  Whenever a request is made to this server, it raises an event. We can therefore
  target it with the `on` method and make it execute a function when requests are
  made.

  If we were to start the server by running the file and we then used a browser to
  navigate to the port, we would see `new connection` logged every time we refresh
  the page.

  ### Sockets and `req, res`

  A socket is a generic protocol for client-server communication. Crucially it
  **allows simultaneous communication both ways**. The client can contact the
  server but the server can also contact the client. Our listener function above
  uses a socket as the callback function but in most cases this is quite
  low-level, not distinguishing responses from requests. It is more likely that
  you would initiate a `request, resource` architecture in place of a socket:

  ```js
  const server = http.createServer((req, res) => {
    if (req.url === "/") {
      res.write("hello");
      res.end();
    }
  });
  ```

  #### Return JSON

  Below is an example of using this architecture to return JSON to the client:

  ```js
  const server = http.createServer((req, res) => {
    if (req.url === "/products") {
      res.write(JSON.stringify(["shoes", "lipstick", "cups"]));
      res.end();
    }
  });
  ```

  ### Express

  In reality you would rarely use the `http` module directly to create a server.
  This is because it is quite low level and each response must be written in a
  linear fashion as with the two URLs in the previous example. Instead we use
  Express which is a framework for creating servers and routing that is an
  abstraction on top of the core HTTP module.


path: zk/Network_debugging.md
sortable_path: zkNetwork_debugging.md
title: Network debugging
lead: ## Related notes
body:
  ## Related notes

  ![Network scanning](Network_scanning.md)

checksum: b18f726199d234e7e7f8592d035aba3e276a636627c070c18a07d9b6211d9698
created: 2024-06-17 19:00:36.317063339+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"created":"Monday, June 17, 2024","id":"x44w","tags":["procedural","Linux","networks"]}
raw_content:
  ---
  id: x44w
  tags: [procedural, Linux, networks]
  created: Monday, June 17, 2024
  ---

  # Network debugging

  ## Related notes

  ![Network scanning](Network_scanning.md)


path: zk/Passing_arguments_to_Python_scripts.md
sortable_path: zkPassing_arguments_to_Python_scripts.md
title: Passing arguments to Python scripts
lead:
  `sys.argv` is a list that contains the command-line arguments passed to a Python
  scripts.

checksum: 8b5c6497c1b17aaf8a5ad1d2d7f86edf63f3825229136af7133af23238556c9a
created: 2024-06-19 16:57:35.57855372+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Wednesday, June 19, 2024","id":"44li","tags":["python"]}
body:
  `sys.argv` is a list that contains the command-line arguments passed to a Python
  scripts.

  - `sys.argv[0]` = the name of script
  - `sys.argv[1]` = the first argument
  - `sys.argv[2]` = the second argument, and so on

  Example invocation:

  ```sh
  python3 ./my_script.py argument_one argument_two

  ```

  ```python
  import sys

  print(sys.argv[0])
  print(sys.arg)
  print(sys.argv[1])

  # my_script.py
  # ['my_script.py', 'argyment_one', 'argument_two']
  # argument_one

  ```

raw_content:
  ---
  id: 44li
  tags: [python]
  created: Wednesday, June 19, 2024
  ---

  # Passing arguments to Python scripts

  `sys.argv` is a list that contains the command-line arguments passed to a Python
  scripts.

  - `sys.argv[0]` = the name of script
  - `sys.argv[1]` = the first argument
  - `sys.argv[2]` = the second argument, and so on

  Example invocation:

  ```sh
  python3 ./my_script.py argument_one argument_two

  ```

  ```python
  import sys

  print(sys.argv[0])
  print(sys.arg)
  print(sys.argv[1])

  # my_script.py
  # ['my_script.py', 'argyment_one', 'argument_two']
  # argument_one

  ```


path: zk/Basic_model_of_the_operating_system.md
sortable_path: zkBasic_model_of_the_operating_system.md
title: Basic model of a nix operating system
lead:
  We can abstract the Linux OS into three operational levels or tiers, from the
  bottom up:

checksum: adb2cb2eb5321b12cb4fe19d75c059b192eb7b8440a10b6661014617a2c99742
created: 2024-06-21 05:06:21.717225019+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"tags":["operating-systems","Linux"]}
body:
  We can abstract the Linux OS into three operational levels or tiers, from the
  bottom up:

  <dl>
  <dt>User processes: user space</dt>
  <dd>The running programs that the kernel manages. Also known as the user space which is the memory that the kernal assigns for user processes. Comprising:
  <ul>
      <li>Graphical user interface</li>
      <li>Servers</li> 
      <li>Shell</li>    
    </ul>
  <dt>Kernel: kernel space</dt>
  </dd>
  <dd>The core of the operating system. Software residing in memory that tells the CPU where to look for its next task. Acts as a mediator and primary interface between the hardware and the user processes. Known as kernel space: the memory that the kernel allocates for itself. Comprising:
    <ul>
      <li>System calls</li>
      <li>Process management</li> 
      <li>Memory management</li>    
      <li>Device drivers</li>    
    </ul>
  </dd>
  <dt>Hardware</dt>
  <dd>The base: one or more CPUs and RAM memory performing computations and writing to memory. Comprising:
    <ul>
      <li>Processor (CPU)</li>
      <li>Main memory (RAM)</li> 
      <li>Disks</li>    
      <li>Network ports</li>    
    </ul>
  </dl>

  !! Add info on kernel mode and user mode
  https://www.geeksforgeeks.org/user-mode-and-kernel-mode-switching/

raw_content:
  ---
  tags:
    - operating-systems
    - Linux
  ---

  # Basic model of a nix operating system

  We can abstract the Linux OS into three operational levels or tiers, from the
  bottom up:

  <dl>
  <dt>User processes: user space</dt>
  <dd>The running programs that the kernel manages. Also known as the user space which is the memory that the kernal assigns for user processes. Comprising:
  <ul>
      <li>Graphical user interface</li>
      <li>Servers</li> 
      <li>Shell</li>    
    </ul>
  <dt>Kernel: kernel space</dt>
  </dd>
  <dd>The core of the operating system. Software residing in memory that tells the CPU where to look for its next task. Acts as a mediator and primary interface between the hardware and the user processes. Known as kernel space: the memory that the kernel allocates for itself. Comprising:
    <ul>
      <li>System calls</li>
      <li>Process management</li> 
      <li>Memory management</li>    
      <li>Device drivers</li>    
    </ul>
  </dd>
  <dt>Hardware</dt>
  <dd>The base: one or more CPUs and RAM memory performing computations and writing to memory. Comprising:
    <ul>
      <li>Processor (CPU)</li>
      <li>Main memory (RAM)</li> 
      <li>Disks</li>    
      <li>Network ports</li>    
    </ul>
  </dl>

  !! Add info on kernel mode and user mode
  https://www.geeksforgeeks.org/user-mode-and-kernel-mode-switching/


path: zk/ps.md
sortable_path: zkps.md
title: Processes (ps)
lead:
  `ps` allows us to control [user processes](The_kernel.md)
  from the shell.

checksum: db2045cff446875dd957c96c1b0b2dcaf4c16fed9ceb7c1f0f91146ea1f07297
created: 2024-06-21 05:23:32.570424822+00:00
modified: 2024-10-18 17:55:56.292394043+00:00
metadata: {"tags":["shell","Linux"]}
body:
  `ps` allows us to control [user processes](The_kernel.md)
  from the shell.

  The command in its most minimal application returns the following

  ```
    PID TTY       TIME     CMD
    2437 pts/2    00:00:01 zsh
    7112 pts/2    00:00:00 ps
  ```

  With the `-e` modifier we can list more processes:

  ```
   PID TTY          TIME CMD
        1 ?        00:00:05 systemd
        2 ?        00:00:00 kthreadd
        3 ?        00:00:00 rcu_gp
        4 ?        00:00:00 rcu_par_gp
        5 ?        00:00:00 netns
        7 ?        00:00:00 kworker/0:0H-events_highpri
        9 ?        00:00:00 mm_percpu_wq
       11 ?        00:00:00 rcu_tasks_kthread
       12 ?        00:00:00 rcu_tasks_rude_kthread
       13 ?        00:00:00 rcu_tasks_trace_kthread
       14 ?        00:00:08 ksoftirqd/0
       15 ?        00:03:20 rcu_preempt
       16 ?        00:00:00 rcub/0
       17 ?        00:00:00 migration/0
       18 ?        00:00:00 idle_inject/0
       20 ?        00:00:00 cpuhp/0
       21 ?        00:00:00 cpuhp/1
       22 ?        00:00:00 idle_inject/1
       23 ?        00:00:00 migration/1
  ```

  <dl>
  <dt>pid</dt>
  <dd>Process ID: every currently running process has a unique ID<dd>
  <dt>tty</dt>
  <dd>The terminal device where the process is running<dd>
  <dt>Time</dt>
  <dd>The amount of CPU time in minutes and seconds that the process has used so far. The total amount of time that the process has spent running instructions on the processor.<dd>
  <dt>cmd</dt>
  <dd>The command used to run the program. Note this can change during the running of the program.<dd>
  </dl>

  ## Modifiers

  <dl>
  <dt><code>ps x</code></dt>
  <dd>Show all running processes<dd>
  <dt><code>ps ax</code></dt>
  <dd>Show all processes not just the ones you, the current user, own<dd>
  <dt><code>ps u</code></dt>
  <dd>Show detailed info on processes<dd>
  <dt><code>ps w</code></dt>
  <dd>Show full command names<dd>
  </dl>

  ## Process termination

  The general schema is: `kill [pid]`. This allows for process clean-up. If this
  doesn't succeed you can force with `KILL [pid]` which will terminate the process
  immediately but is obviously more risky.

  We can also start/stop processes with modifiers on `kill`:

  - `kill -STOP pid`
  - `kill -CONT pid`

raw_content:
  ---
  tags:
    - shell
    - Linux
  ---

  # Processes (`ps`)

  `ps` allows us to control [user processes](The_kernel.md)
  from the shell.

  The command in its most minimal application returns the following

  ```
    PID TTY       TIME     CMD
    2437 pts/2    00:00:01 zsh
    7112 pts/2    00:00:00 ps
  ```

  With the `-e` modifier we can list more processes:

  ```
   PID TTY          TIME CMD
        1 ?        00:00:05 systemd
        2 ?        00:00:00 kthreadd
        3 ?        00:00:00 rcu_gp
        4 ?        00:00:00 rcu_par_gp
        5 ?        00:00:00 netns
        7 ?        00:00:00 kworker/0:0H-events_highpri
        9 ?        00:00:00 mm_percpu_wq
       11 ?        00:00:00 rcu_tasks_kthread
       12 ?        00:00:00 rcu_tasks_rude_kthread
       13 ?        00:00:00 rcu_tasks_trace_kthread
       14 ?        00:00:08 ksoftirqd/0
       15 ?        00:03:20 rcu_preempt
       16 ?        00:00:00 rcub/0
       17 ?        00:00:00 migration/0
       18 ?        00:00:00 idle_inject/0
       20 ?        00:00:00 cpuhp/0
       21 ?        00:00:00 cpuhp/1
       22 ?        00:00:00 idle_inject/1
       23 ?        00:00:00 migration/1
  ```

  <dl>
  <dt>pid</dt>
  <dd>Process ID: every currently running process has a unique ID<dd>
  <dt>tty</dt>
  <dd>The terminal device where the process is running<dd>
  <dt>Time</dt>
  <dd>The amount of CPU time in minutes and seconds that the process has used so far. The total amount of time that the process has spent running instructions on the processor.<dd>
  <dt>cmd</dt>
  <dd>The command used to run the program. Note this can change during the running of the program.<dd>
  </dl>

  ## Modifiers

  <dl>
  <dt><code>ps x</code></dt>
  <dd>Show all running processes<dd>
  <dt><code>ps ax</code></dt>
  <dd>Show all processes not just the ones you, the current user, own<dd>
  <dt><code>ps u</code></dt>
  <dd>Show detailed info on processes<dd>
  <dt><code>ps w</code></dt>
  <dd>Show full command names<dd>
  </dl>

  ## Process termination

  The general schema is: `kill [pid]`. This allows for process clean-up. If this
  doesn't succeed you can force with `KILL [pid]` which will terminate the process
  immediately but is obviously more risky.

  We can also start/stop processes with modifiers on `kill`:

  - `kill -STOP pid`
  - `kill -CONT pid`


path: zk/Processes.md
sortable_path: zkProcesses.md
title: Processes
checksum: 2b331b124e4ae2014628e74cbe62db17966948066c36c6a255fecd473f58e76b
created: 2024-06-21 05:23:32.592788276+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"created":"Friday, June 21, 2024","id":"t70u","tags":["operating-systems"]}
lead:
  Programs are sequences of machine instructions stored in a file. However they do
  not work by themselves. Something needs to load the file's intructions into
  memory, direct the CPU to run the program and manage it during runtime. The OS
  does this via processes.

body:
  Programs are sequences of machine instructions stored in a file. However they do
  not work by themselves. Something needs to load the file's intructions into
  memory, direct the CPU to run the program and manage it during runtime. The OS
  does this via processes.

  A process **is a running instance of a given program**. It can be thought of as
  a container in which a program runs. This container includes:

  - a copy of the program code loaded into memory
  - a memory address
  - other information about the state of the process

  Other than the `init` process started by the kernel (PID1) (see
  ![systemd](systemd.md)), every process has a parent process that started it.
  This parent-child relationship creates a tree of processes.

  It is possible that a parent process will terminate before one of its child
  processes. In this instance the child becomes an orphan. When this occurs in
  #Linux, the orphan process is adopted by `init`.

  Below, I have used the `pstree` utility to list all the running processes on my
  machine hierarchically.

  ![diagram of `pstree` output](../img/ps-tree.png)

  Children are represented vertically and horizontally.

  ```
  ├─terminator─┬─zsh───tmux: client
  │            ├─zsh───pstree
  │            └─6*[{terminator}]

  ```

  For instance here, `terminator` is a child of `init`, as are `zsh` and `tmux`
  but they are also children of `terminator`.`pstree` is a child of `zsh` and
  therefore also a child (grandchild) of `terminator`.

  Each process has a unique identifier called a _process identifier_, a
  _processID_ or just _PID_.

  ## Related notes

  ![systemd](./systemd.md)

  ![ps](./ps.md)

  ![Monitoring processes and resources](Monitoring_processes_and_resources.md)

raw_content:
  ---
  id: t70u
  tags: [operating-systems]
  created: Friday, June 21, 2024
  ---

  # Processes

  Programs are sequences of machine instructions stored in a file. However they do
  not work by themselves. Something needs to load the file's intructions into
  memory, direct the CPU to run the program and manage it during runtime. The OS
  does this via processes.

  A process **is a running instance of a given program**. It can be thought of as
  a container in which a program runs. This container includes:

  - a copy of the program code loaded into memory
  - a memory address
  - other information about the state of the process

  Other than the `init` process started by the kernel (PID1) (see
  ![systemd](systemd.md)), every process has a parent process that started it.
  This parent-child relationship creates a tree of processes.

  It is possible that a parent process will terminate before one of its child
  processes. In this instance the child becomes an orphan. When this occurs in
  #Linux, the orphan process is adopted by `init`.

  Below, I have used the `pstree` utility to list all the running processes on my
  machine hierarchically.

  ![diagram of `pstree` output](../img/ps-tree.png)

  Children are represented vertically and horizontally.

  ```
  ├─terminator─┬─zsh───tmux: client
  │            ├─zsh───pstree
  │            └─6*[{terminator}]

  ```

  For instance here, `terminator` is a child of `init`, as are `zsh` and `tmux`
  but they are also children of `terminator`.`pstree` is a child of `zsh` and
  therefore also a child (grandchild) of `terminator`.

  Each process has a unique identifier called a _process identifier_, a
  _processID_ or just _PID_.

  ## Related notes

  ![systemd](./systemd.md)

  ![ps](./ps.md)

  ![Monitoring processes and resources](Monitoring_processes_and_resources.md)


path: zk/DynamoDB_data_structure.md
sortable_path: zkDynamoDB_data_structure.md
title: DynamoDB
lead: ## Data structure
checksum: 2d5c18608853ef243aa5d83efba55f37af0a4ff04d2a64527076313f81ce6ad7
created: 2024-06-22 09:38:00.123616477+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"created":"Sunday, June 09, 2024","id":"xowl","tags":["AWS","databases","dynamodb"]}
body:
  ## Data structure

  ### Non-relational tables

  DynamoDB is "NoSQL" because it does not support #SQL queries and is
  non-relational meaning there cannot be JOIN operations via
  [foreign_keys](Foreign_keys_in_SQL.md)

  ![](../img/dynamodb.png)

  ### Primary key

  Although the data is stored as a table, one of the attributes is a primary key
  and the rest of the attributes are effectively the "value" associated with it.

  Because DynamoDB is schemaless, other than the primary key, neither the
  attributes or their data types need to be defined beforehand and each item can
  have its own distinct attributes.

  Each item in the table is uniquely identifiable by its primary key.

  There are two types of primary key available:

  - partition key: a simple primary key composed of one attribute only. Because
    the primary key is hash-mapped items can be retrieved very rapidly using the
    primary key. This would be the `personId` alone.

  - composite key: this comprises a partition key and a _sort key_ both of which
    are attributes. In a table that has a partition key and a sort key, it's
    possible for multiple items to have the same partition key value. However,
    those items must have different sort key values. You could then query by
    either key or both. For instance using the `PersonId` along with `LastName`

  ### Secondary index

  As well as the index provided by the primary key, you can set one or more
  **secondary indices**. A secondary index lets you query the data in the table
  using an alternate key.

  A **global secondary index** is useful for querying data that needs to be
  accessed using non-primary key attributes. For example, if you have a Users
  table with `UserID` as the primary key but often need to fetch users by their
  `Email`, a GSI on `Email` would be appropriate.

  There are also **local secondary indices** but I don't understand the
  difference.

  ## Real example

  Below is a specification of the DynamoDB table I am using for my time-entries
  project:

  ```json
  {
    "TableName": "TimeEntries",
    "KeyAttributes": {
      "PartitionKey": {
        "AttributeName": "activity_start_end",
        "AttributeType": "S"
      }
    },
    "NonKeyAttributes": [
      {
        "AttributeName": "activity_type",
        "AttributeType": "S"
      },
      {
        "AttributeName": "start",
        "AttributeType": "S"
      },
      {
        "AttributeName": "end",
        "AttributeType": "S"
      },
      {
        "AttributeName": "duration",
        "AttributeType": "N"
      },
      {
        "AttributeName": "description",
        "AttributeType": "S"
      },
      {
        "AttributeName": "year",
        "AttributeType": "S"
      }
    ],
    "GlobalSecondaryIndexes": [
      {
        "IndexName": "YearIndex",
        "KeyAttributes": {
          "PartitionKey": {
            "AttributeName": "year",
            "AttributeType": "S"
          },
          "SortKey": {
            "AttributeName": "start",
            "AttributeType": "S"
          }
        },
        "Projection": {
          "ProjectionType": "ALL"
        }
      }
    ]
  }
  ```

  This defines the attribute `activity_start_end` as the primary key. This string
  (`S`) value is a concatenation of three attributes, which is a way of ensuring
  each entry for the attribute will be unique.

  The `NonKeyAttributes` are all the other attributes in addition to the primary
  key. As mentioned these do not actually need to be defined when setting up the
  table but they are listed here for clarity.

  I have also defined a GSI. This is derived from the `Year` attribute. This will
  group all the items by their `Year`, allowing me to query directly by year but
  also helping to chunk the entries which will make look-ups quicker and less
  expensive.

  ## Related notes

raw_content:
  ---
  id: xowl
  tags:
    - AWS
    - databases
    - dynamodb
  created: Sunday, June 09, 2024
  ---

  # DynamoDB

  ## Data structure

  ### Non-relational tables

  DynamoDB is "NoSQL" because it does not support #SQL queries and is
  non-relational meaning there cannot be JOIN operations via
  [foreign_keys](Foreign_keys_in_SQL.md)

  ![](../img/dynamodb.png)

  ### Primary key

  Although the data is stored as a table, one of the attributes is a primary key
  and the rest of the attributes are effectively the "value" associated with it.

  Because DynamoDB is schemaless, other than the primary key, neither the
  attributes or their data types need to be defined beforehand and each item can
  have its own distinct attributes.

  Each item in the table is uniquely identifiable by its primary key.

  There are two types of primary key available:

  - partition key: a simple primary key composed of one attribute only. Because
    the primary key is hash-mapped items can be retrieved very rapidly using the
    primary key. This would be the `personId` alone.

  - composite key: this comprises a partition key and a _sort key_ both of which
    are attributes. In a table that has a partition key and a sort key, it's
    possible for multiple items to have the same partition key value. However,
    those items must have different sort key values. You could then query by
    either key or both. For instance using the `PersonId` along with `LastName`

  ### Secondary index

  As well as the index provided by the primary key, you can set one or more
  **secondary indices**. A secondary index lets you query the data in the table
  using an alternate key.

  A **global secondary index** is useful for querying data that needs to be
  accessed using non-primary key attributes. For example, if you have a Users
  table with `UserID` as the primary key but often need to fetch users by their
  `Email`, a GSI on `Email` would be appropriate.

  There are also **local secondary indices** but I don't understand the
  difference.

  ## Real example

  Below is a specification of the DynamoDB table I am using for my time-entries
  project:

  ```json
  {
    "TableName": "TimeEntries",
    "KeyAttributes": {
      "PartitionKey": {
        "AttributeName": "activity_start_end",
        "AttributeType": "S"
      }
    },
    "NonKeyAttributes": [
      {
        "AttributeName": "activity_type",
        "AttributeType": "S"
      },
      {
        "AttributeName": "start",
        "AttributeType": "S"
      },
      {
        "AttributeName": "end",
        "AttributeType": "S"
      },
      {
        "AttributeName": "duration",
        "AttributeType": "N"
      },
      {
        "AttributeName": "description",
        "AttributeType": "S"
      },
      {
        "AttributeName": "year",
        "AttributeType": "S"
      }
    ],
    "GlobalSecondaryIndexes": [
      {
        "IndexName": "YearIndex",
        "KeyAttributes": {
          "PartitionKey": {
            "AttributeName": "year",
            "AttributeType": "S"
          },
          "SortKey": {
            "AttributeName": "start",
            "AttributeType": "S"
          }
        },
        "Projection": {
          "ProjectionType": "ALL"
        }
      }
    ]
  }
  ```

  This defines the attribute `activity_start_end` as the primary key. This string
  (`S`) value is a concatenation of three attributes, which is a way of ensuring
  each entry for the attribute will be unique.

  The `NonKeyAttributes` are all the other attributes in addition to the primary
  key. As mentioned these do not actually need to be defined when setting up the
  table but they are listed here for clarity.

  I have also defined a GSI. This is derived from the `Year` attribute. This will
  group all the items by their `Year`, allowing me to query directly by year but
  also helping to chunk the entries which will make look-ups quicker and less
  expensive.

  ## Related notes


path: zk/DynamoDB_CLI_commands.md
sortable_path: zkDynamoDB_CLI_commands.md
title: DynamoDB CLI commands
lead: ## Connecting to a local (Docker)/prod (AWS) DynamoDB instance
checksum: d5c52d2d041c61617fa5af1ec2f57f94f1888571c56e3cd0e1993f9777610cb4
created: 2024-06-23 18:20:42.429528536+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Saturday, June 22, 2024","id":"l045","tags":["AWS","databases","dynamodb"]}
body:
  ## Connecting to a local (Docker)/prod (AWS) DynamoDB instance

  In order to distinguish between local and production accounts you should keep
  seperate profiles for each (via `.aws/config` and `.aws/credentials`).

  When connecting to a local DB use the local profile and the local URL. Without
  the `--profile` flag, AWS will default to the `default` profile which will
  typically be your credentials for accessing AWS on the remote.

  It also sometimes required to add the endpoint-url when working locally:

  ```
  --endpoint-url http://localhost:800

  > When using the commans below locally, assume `--endpoint-url` and `--profile`
  > have been appended

  ```

  For example:

  ```sh
  aws dynamodb list-tables \
  --profile timetracking_dev \
  --endpoint-url http://localhost:800
  ```

  ## Delete a table

  ```sh
  aws dynamodb delete-table \
  --table-name TableName
  ```

  ## Create table from JSON schema

  ```sh
  aws dynamodb create-table \
  --cli-input-json file://create-timeentries-table.json \

  ```

  ## Describe the table

  View the table schema:

  ```sh

  aws dynamodb describe-table \
  --table-name TimeEntries
  ```

  ## Scan the table

  ```sh
  aws dynamodb scan \
  --table-name TimeEntries \
  --output table
  ```

  ## Related notes

raw_content:
  ---
  id: l045
  tags: [AWS, databases, dynamodb]
  created: Saturday, June 22, 2024
  ---

  # DynamoDB CLI commands

  ## Connecting to a local (Docker)/prod (AWS) DynamoDB instance

  In order to distinguish between local and production accounts you should keep
  seperate profiles for each (via `.aws/config` and `.aws/credentials`).

  When connecting to a local DB use the local profile and the local URL. Without
  the `--profile` flag, AWS will default to the `default` profile which will
  typically be your credentials for accessing AWS on the remote.

  It also sometimes required to add the endpoint-url when working locally:

  ```
  --endpoint-url http://localhost:800

  > When using the commans below locally, assume `--endpoint-url` and `--profile`
  > have been appended

  ```

  For example:

  ```sh
  aws dynamodb list-tables \
  --profile timetracking_dev \
  --endpoint-url http://localhost:800
  ```

  ## Delete a table

  ```sh
  aws dynamodb delete-table \
  --table-name TableName
  ```

  ## Create table from JSON schema

  ```sh
  aws dynamodb create-table \
  --cli-input-json file://create-timeentries-table.json \

  ```

  ## Describe the table

  View the table schema:

  ```sh

  aws dynamodb describe-table \
  --table-name TimeEntries
  ```

  ## Scan the table

  ```sh
  aws dynamodb scan \
  --table-name TimeEntries \
  --output table
  ```

  ## Related notes


path: zk/DynamoDB_SDK_commands.md
sortable_path: zkDynamoDB_SDK_commands.md
title: DynamoDB SDK commands
lead: The following commands are for using `@aws-sdk/client-dynamodb` (the JS SDK).
checksum: 56668c1e36e4fbaadbb8d50f9b9a63de9be5cb42f5010b739711aa662cd6acfa
created: 2024-06-23 18:20:42.435974932+00:00
modified: 2024-10-18 17:55:56.292394043+00:00
metadata: {"created":"Sunday, June 23, 2024","id":"8b6h","tags":["AWS","databases","dynamodb"]}
body:
  The following commands are for using `@aws-sdk/client-dynamodb` (the JS SDK).

  ## Create client

  ```js
  import DynamoDBClient from "@aws-sdk/client-dynamodb";

  const client = new DynamoDBClient({
    region: "eu-west-2",
    accessKeyId: process.env.ACCESS_KEY_ID,
    secretAccessKey: process.env.ACCESS_KEY_ID,
  });
  ```

  ### DynamoDB Document Client

  The Document Client returns the data as a standard JSON document in the manner
  of a document-based NoSQL database. Without it, DynamoDB will return the data
  with type data which is unwieldy and hard to parse.

  To transform into a document structure:

  ```js
  import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
  import { DynamoDBDocumentClient } from "@aws-sdk/lib-dynamodb";

  const client = new DynamoDBClient({
    region: "eu-west-2",
    accessKeyId: process.env.ACCESS_KEY_ID,
    secretAccessKey: process.env.ACCESS_KEY_ID,
  });

  const dynamoDbDocumentClient = DynamoDBDocumentClient.from(client);
  ```

  ## Query a table

  Querying is the most performant and cost-effective method since it is an O(1)
  lookup against a partition key or secondary index.

  ```js
  import { DynamoDBDocumentClient, QueryCommand } from "@aws-sdk/lib-dynamodb";

  const params = {
    TableName: "YourTableName",
    KeyConditionExpression: "#pk = :pk",
    ExpressionAttributeNames: { "#pk": "PrimaryKeyAttributeName" },
    ExpressionAttributeValues: { ":pk": "PrimaryKeyValue" },
  };

  const dynamoDbDocumentClient = DynamoDBDocumentClient.from(client);
  const data = await dynamoDbDocumentClient.send(new QueryCommand(params));
  console.log(data);
  ```

  ## Scan a table

  Scanning is less performant (O(n)) and most expensive since it requires checking
  every item in the database.

  ```js
  import { DynamoDBDocumentClient, ScanCommand } from "@aws-sdk/lib-dynamodb";

  const params = {
    TableName: "YourTableName",
  };

  const dynamoDbDocumentClient = DynamoDBDocumentClient.from(client);
  const data = await dynamoDbDocumentClient.send(new ScanCommand(params));
  ```

  ## Related notes

raw_content:
  ---
  id: 8b6h
  tags: [AWS, databases, dynamodb]
  created: Sunday, June 23, 2024
  ---

  # DynamoDB SDK commands

  The following commands are for using `@aws-sdk/client-dynamodb` (the JS SDK).

  ## Create client

  ```js
  import DynamoDBClient from "@aws-sdk/client-dynamodb";

  const client = new DynamoDBClient({
    region: "eu-west-2",
    accessKeyId: process.env.ACCESS_KEY_ID,
    secretAccessKey: process.env.ACCESS_KEY_ID,
  });
  ```

  ### DynamoDB Document Client

  The Document Client returns the data as a standard JSON document in the manner
  of a document-based NoSQL database. Without it, DynamoDB will return the data
  with type data which is unwieldy and hard to parse.

  To transform into a document structure:

  ```js
  import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
  import { DynamoDBDocumentClient } from "@aws-sdk/lib-dynamodb";

  const client = new DynamoDBClient({
    region: "eu-west-2",
    accessKeyId: process.env.ACCESS_KEY_ID,
    secretAccessKey: process.env.ACCESS_KEY_ID,
  });

  const dynamoDbDocumentClient = DynamoDBDocumentClient.from(client);
  ```

  ## Query a table

  Querying is the most performant and cost-effective method since it is an O(1)
  lookup against a partition key or secondary index.

  ```js
  import { DynamoDBDocumentClient, QueryCommand } from "@aws-sdk/lib-dynamodb";

  const params = {
    TableName: "YourTableName",
    KeyConditionExpression: "#pk = :pk",
    ExpressionAttributeNames: { "#pk": "PrimaryKeyAttributeName" },
    ExpressionAttributeValues: { ":pk": "PrimaryKeyValue" },
  };

  const dynamoDbDocumentClient = DynamoDBDocumentClient.from(client);
  const data = await dynamoDbDocumentClient.send(new QueryCommand(params));
  console.log(data);
  ```

  ## Scan a table

  Scanning is less performant (O(n)) and most expensive since it requires checking
  every item in the database.

  ```js
  import { DynamoDBDocumentClient, ScanCommand } from "@aws-sdk/lib-dynamodb";

  const params = {
    TableName: "YourTableName",
  };

  const dynamoDbDocumentClient = DynamoDBDocumentClient.from(client);
  const data = await dynamoDbDocumentClient.send(new ScanCommand(params));
  ```

  ## Related notes


path: zk/Breadboards.md
sortable_path: zkBreadboards.md
title: Breadboards
lead: ![](../img/breadboard-diagram.jpg)
checksum: c30e28a837b489ecbebb55a6b34153401f1007ceafd0a4318766d0bb0246307e
created: 2024-06-23 18:20:42.444058773+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Sunday, June 23, 2024","id":"jbf1","tags":["electronics"]}
body:
  ![](../img/breadboard-diagram.jpg)

  ## Contact holes

  Beneath the plastic casing, the contact holes are connected to one another
  making a continuous circuit connection

  ## Terminal strips

  The contact holes in the middle section, arranged in groups of five and
  designated by letters

  The terminal strips are not connected accross the ditch thus each of the two
  strips are electrically separate regions (A-E and F-J).

  ## Bus strips

  The contact holes on the outer edges of the breadboard, designated by numbers
  (1-30). The chief role of the bus strips is to manage the power to the circuit.
  You will typically use the positive bus strip on one side for the voltage source
  and the negative bus strip on the other side for the ground outlet.

raw_content:
  ---
  id: jbf1
  tags: [electronics]
  created: Sunday, June 23, 2024
  ---

  # Breadboards

  ![](../img/breadboard-diagram.jpg)

  ## Contact holes

  Beneath the plastic casing, the contact holes are connected to one another
  making a continuous circuit connection

  ## Terminal strips

  The contact holes in the middle section, arranged in groups of five and
  designated by letters

  The terminal strips are not connected accross the ditch thus each of the two
  strips are electrically separate regions (A-E and F-J).

  ## Bus strips

  The contact holes on the outer edges of the breadboard, designated by numbers
  (1-30). The chief role of the bus strips is to manage the power to the circuit.
  You will typically use the positive bus strip on one side for the voltage source
  and the negative bus strip on the other side for the ground outlet.


path: zk/ROM_versus_disk_loading.md
sortable_path: zkROM_versus_disk_loading.md
title: ROM versus disk loading
checksum: f12aa9254eb7ecf2c9c48b2957218ddc91aa497e6babda04ce7811a64e14c47c
created: 2024-06-24 16:45:46.215734756+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"created":"Monday, June 24, 2024","id":"rmvm","tags":["computer-architecture","memory"]}
lead:
  I will explore the concept via the practical examples of cartridge-based and
  disk-based video games console.

body:
  I will explore the concept via the practical examples of cartridge-based and
  disk-based video games console.

  When you insert a disk into an Xbox, the game data is typically read from the
  disk and transferred into the console's memory by the OS. If the game is
  installed to the hard drive, this initially involves transferring substantial
  amounts of data to speed up future access. When you insert a cartridge into a
  console like the SNES, the data is not transferred in the same way. Instead, the
  cartridge contains a ROM chip that, when inserted, integrates directly with the
  motherboard. From the operating system’s point of view, this ROM chip behaves as
  an extension of the system’s memory architecture.

  The primary advantage of a ROM-based game is significantly reduced latency. The
  game data can be accessed as rapidly as any other data moving along the buses of
  the motherboard. With a disk, the data must first be read into a buffer and then
  transferred to a memory address where it can be accessed by the CPU.

  With a ROM, while the console CPU still uses its own RAM to run the game
  (similar to disk-based systems), the game data on the ROM can be loaded into
  this RAM more quickly because it is directly attached to the motherboard. This
  makes the ROM cartridge effectively a fast, directly connected storage device.

  However, despite its speed, the limitation of a ROM cartridge is its storage
  capacity—it can hold only as much data as the chip can accommodate. In contrast,
  a disk can store much more data. This larger storage capacity allows not just
  for bigger and more complex games but also for the possibility of expanding the
  game through additional data downloaded to the console's hard drive. Such
  expansions or patches represent additional data from the console’s storage that,
  combined with the original disk data, are synthesized in memory at runtime for
  enhanced or updated gameplay experiences.

  (Note however that modern cartridge-based consoles like the Nintendo Switch can
  also combine ROM-data with expanded data from the console HDD, sourced from the
  internet.)

  Both disks and ROMs are forms of static storage in the context of a game
  console. While disks can technically be overwritten with new data, in practice,
  manufacturers implement various locks and protections to prevent this. Any
  changes or additions to the game, such as patches or expansions, are not written
  back to the original disk. Instead, these updates are stored on the console's
  hard drive.

  Therefore, if you download an expansion for a game on your Xbox, the original
  disk remains unchanged as when you first purchased it. Consequently, if you were
  to take that disk and load it on another Xbox, it would not include any of the
  expansions or updates. The game would function in its original form, as the
  additional content and updates reside solely on the hard drive of the first
  console where they were downloaded.

  ## Related notes

raw_content:
  ---
  id: rmvm
  tags: [computer-architecture, memory]
  created: Monday, June 24, 2024
  ---

  # ROM versus disk loading

  I will explore the concept via the practical examples of cartridge-based and
  disk-based video games console.

  When you insert a disk into an Xbox, the game data is typically read from the
  disk and transferred into the console's memory by the OS. If the game is
  installed to the hard drive, this initially involves transferring substantial
  amounts of data to speed up future access. When you insert a cartridge into a
  console like the SNES, the data is not transferred in the same way. Instead, the
  cartridge contains a ROM chip that, when inserted, integrates directly with the
  motherboard. From the operating system’s point of view, this ROM chip behaves as
  an extension of the system’s memory architecture.

  The primary advantage of a ROM-based game is significantly reduced latency. The
  game data can be accessed as rapidly as any other data moving along the buses of
  the motherboard. With a disk, the data must first be read into a buffer and then
  transferred to a memory address where it can be accessed by the CPU.

  With a ROM, while the console CPU still uses its own RAM to run the game
  (similar to disk-based systems), the game data on the ROM can be loaded into
  this RAM more quickly because it is directly attached to the motherboard. This
  makes the ROM cartridge effectively a fast, directly connected storage device.

  However, despite its speed, the limitation of a ROM cartridge is its storage
  capacity—it can hold only as much data as the chip can accommodate. In contrast,
  a disk can store much more data. This larger storage capacity allows not just
  for bigger and more complex games but also for the possibility of expanding the
  game through additional data downloaded to the console's hard drive. Such
  expansions or patches represent additional data from the console’s storage that,
  combined with the original disk data, are synthesized in memory at runtime for
  enhanced or updated gameplay experiences.

  (Note however that modern cartridge-based consoles like the Nintendo Switch can
  also combine ROM-data with expanded data from the console HDD, sourced from the
  internet.)

  Both disks and ROMs are forms of static storage in the context of a game
  console. While disks can technically be overwritten with new data, in practice,
  manufacturers implement various locks and protections to prevent this. Any
  changes or additions to the game, such as patches or expansions, are not written
  back to the original disk. Instead, these updates are stored on the console's
  hard drive.

  Therefore, if you download an expansion for a game on your Xbox, the original
  disk remains unchanged as when you first purchased it. Consequently, if you were
  to take that disk and load it on another Xbox, it would not include any of the
  expansions or updates. The game would function in its original form, as the
  additional content and updates reside solely on the hard drive of the first
  console where they were downloaded.

  ## Related notes


path: zk/Push_buttons.md
sortable_path: zkPush_buttons.md
title: Push buttons
checksum: 04e804313c68988cb974b6293dd1807087288f983f73cecf701708e088b5522b
created: 2024-06-25 04:45:25.356583385+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Tuesday, June 25, 2024","id":"nsrv","tags":["electronics"]}
lead:
  Push buttons are push-to-make switches. This means that they are **on when they
  pressed and off when they are not pressed**. A classic example would be the
  switches on a keyboard.

body:
  Push buttons are push-to-make switches. This means that they are **on when they
  pressed and off when they are not pressed**. A classic example would be the
  switches on a keyboard.

  Push buttons have four legs that are connected in signal-in and signal-out
  pairs.

  To create a breakable circuit we would connect a signal-in cable to the top left
  at row 23 and a signal-out cable to the bottom-right at row 25.

  ![Push button legs diagram](../img/push-button-legs.png)

  _The correct use of a push button where the button breaks the circuit_

  ![](../img/correct_push_button.jpg)

  To override the switch functionality and just have the button work as a
  connector we would connect the signal-in to the row 23 input and the row 23
  output.

  _The push button being used as simple connector which does not break the
  circuit:_

  ![](../img/incorrect_push_button.jpg)

  ## Related notes

  https://projects.raspberrypi.org/en/projects/button-switch-scratch-pi/1

raw_content:
  ---
  id: nsrv
  tags: [electronics]
  created: Tuesday, June 25, 2024
  ---

  # Push buttons

  Push buttons are push-to-make switches. This means that they are **on when they
  pressed and off when they are not pressed**. A classic example would be the
  switches on a keyboard.

  Push buttons have four legs that are connected in signal-in and signal-out
  pairs.

  To create a breakable circuit we would connect a signal-in cable to the top left
  at row 23 and a signal-out cable to the bottom-right at row 25.

  ![Push button legs diagram](../img/push-button-legs.png)

  _The correct use of a push button where the button breaks the circuit_

  ![](../img/correct_push_button.jpg)

  To override the switch functionality and just have the button work as a
  connector we would connect the signal-in to the row 23 input and the row 23
  output.

  _The push button being used as simple connector which does not break the
  circuit:_

  ![](../img/incorrect_push_button.jpg)

  ## Related notes

  https://projects.raspberrypi.org/en/projects/button-switch-scratch-pi/1


path: zk/Working_with_JSON_in_Python.md
sortable_path: zkWorking_with_JSON_in_Python.md
title: Working with JSON in Python
lead: ## `json.loads()`
checksum: 7ceffd0f3280c65bdd423a236eadedbc95416fcea3b0c4331d12be2b47fefdb2
created: 2024-06-25 05:55:58.93390879+00:00
modified: 2024-10-18 17:55:56.292394043+00:00
metadata: {"created":"Tuesday, June 25, 2024","id":"mjia","tags":["python","JSON"]}
body:
  ## `json.loads()`

  Convert a string containing JSON data into the native Python dictionary object.
  Equivalent to `JSON.parse()` in JavaScript.

  Will error if the input string is not properly formatted JSON. This will be
  `json.JSONDecodeError`, a subclass of the `ValueError`
  [exception](./Error_handling_in_Python.md)

  ## `json.dumps()`

  Take a JSON-formatted dictionary and convert it into a string. The reverse of
  `json.loads()` and equivalent to `JSON.stringify()` in JavaScript.

  ## JSON errors

  The `.loads()` and `.parse()` methods will throw an error if the input string is

  ## Related notes

raw_content:
  ---
  id: mjia
  tags: [python, JSON]
  created: Tuesday, June 25, 2024
  ---

  # Working with JSON in Python

  ## `json.loads()`

  Convert a string containing JSON data into the native Python dictionary object.
  Equivalent to `JSON.parse()` in JavaScript.

  Will error if the input string is not properly formatted JSON. This will be
  `json.JSONDecodeError`, a subclass of the `ValueError`
  [exception](./Error_handling_in_Python.md)

  ## `json.dumps()`

  Take a JSON-formatted dictionary and convert it into a string. The reverse of
  `json.loads()` and equivalent to `JSON.stringify()` in JavaScript.

  ## JSON errors

  The `.loads()` and `.parse()` methods will throw an error if the input string is

  ## Related notes


path: zk/Threads.md
sortable_path: zkThreads.md
title: Threads
checksum: b205ac9e81526fd4a3e0afc1bb2e543180916a4eca63752de926b504259e37f4
created: 2024-06-26 04:51:56.925975752+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Wednesday, June 26, 2024","id":"zadl","tags":["operating-systems"]}
lead:
  A ![process](Processes.md) is a running instance of a given program. A program
  runs sequentially handling one task at a time, however we may need to run
  certain tasks in parallel.

body:
  A ![process](Processes.md) is a running instance of a given program. A program
  runs sequentially handling one task at a time, however we may need to run
  certain tasks in parallel.

  Think of a program that is downloading a resource from the internet and wants to
  update the UI at the same time to show the download process.

  If the program is strictly sequential, once the program starts the download, the
  UI is neglected since the CPU time dedicated to the process must be focused
  elsewhere.

  We need the UI update and the download to execute in parallel. This is achieved
  via operating system **threads of execution**.

  > A thread is a schedulable unit of execution within a process. It runs within a
  > process, shares its memory address and can execute any program code loaded
  > within it.

  Each thread has an identifier called a _thread ID_ (TID).

  Both procesess in [user space](User_Space.md) and the [kernel](The_kernel.md)
  utilise threads.

  ## Hardware considerations

  Each [processor core](./CPU_architecture.md) can only run one thread at a time
  so the number of cores that a device's CPU has deterimines how many threads can
  run at once. In practice, then, the CPU shuffles between multiple threads in
  [clock segments](./Clock_signals.md). This is known as scheduling and is managed
  by the process that the thread belongs to.

  ## Related notes

  ![Processes](Processes.md)

raw_content:
  ---
  id: zadl
  tags: [operating-systems]
  created: Wednesday, June 26, 2024
  ---

  # Threads

  A ![process](Processes.md) is a running instance of a given program. A program
  runs sequentially handling one task at a time, however we may need to run
  certain tasks in parallel.

  Think of a program that is downloading a resource from the internet and wants to
  update the UI at the same time to show the download process.

  If the program is strictly sequential, once the program starts the download, the
  UI is neglected since the CPU time dedicated to the process must be focused
  elsewhere.

  We need the UI update and the download to execute in parallel. This is achieved
  via operating system **threads of execution**.

  > A thread is a schedulable unit of execution within a process. It runs within a
  > process, shares its memory address and can execute any program code loaded
  > within it.

  Each thread has an identifier called a _thread ID_ (TID).

  Both procesess in [user space](User_Space.md) and the [kernel](The_kernel.md)
  utilise threads.

  ## Hardware considerations

  Each [processor core](./CPU_architecture.md) can only run one thread at a time
  so the number of cores that a device's CPU has deterimines how many threads can
  run at once. In practice, then, the CPU shuffles between multiple threads in
  [clock segments](./Clock_signals.md). This is known as scheduling and is managed
  by the process that the thread belongs to.

  ## Related notes

  ![Processes](Processes.md)


path: zk/Concise_subfield_mapping_JS.md
sortable_path: zkConcise_subfield_mapping_JS.md
title: Concise mapping of object subfileds in JS
lead: ## Scenario
checksum: 858f366081f302311228af222dc1f012097ae888b16700c8824c42b6b3b578f2
created: 2024-06-28 08:36:18.842907046+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Friday, June 28, 2024","id":"dv3u","tags":[]}
body:
  ## Scenario

  You have an array of objects and you want to return the objects with only a
  subset of the fields.

  ## Implementation

  Standard approach with a map:

  ```js
  const arrayOfObjs = [
    { id: 12, name: "Thomas" },
    { id: 3, name: "Gerald" },
  ];

  // We just want the `name` property

  const subset = arrayOfObjs.map((obj) => {
    name: obj.name;
  });
  ```

  More concise approach with destructuring:

  ```js
  const subset = arrayOfObjs.map(({ name }) => ({ name }));
  ```

  ## Related notes

raw_content:
  ---
  id: dv3u
  tags: []
  created: Friday, June 28, 2024
  ---

  # Concise mapping of object subfileds in JS

  ## Scenario

  You have an array of objects and you want to return the objects with only a
  subset of the fields.

  ## Implementation

  Standard approach with a map:

  ```js
  const arrayOfObjs = [
    { id: 12, name: "Thomas" },
    { id: 3, name: "Gerald" },
  ];

  // We just want the `name` property

  const subset = arrayOfObjs.map((obj) => {
    name: obj.name;
  });
  ```

  More concise approach with destructuring:

  ```js
  const subset = arrayOfObjs.map(({ name }) => ({ name }));
  ```

  ## Related notes


path: zk/Memory_Management_Unit.md
sortable_path: zkMemory_Management_Unit.md
title: Memory Management Unit (MMU)
checksum: 872a7691c78622056fe66b79197c0f6f9df56d1dba50321e9f4a17e5c3b52da0
created: 2024-07-08 05:22:28.885795567+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Monday, July 08, 2024","tags":["memory","Linux"]}
lead:
  The MMU is the bridge between the physical memory devices of the machine and the
  virtual memory space. It is a chip that sits between the CPU and the RAM and
  determines the physical location of the memory requested by the kernel as
  virtual memory.

body:
  The MMU is the bridge between the physical memory devices of the machine and the
  virtual memory space. It is a chip that sits between the CPU and the RAM and
  determines the physical location of the memory requested by the kernel as
  virtual memory.

  ![Virtual memory diagram](/img/virtual-memory-diagram.jpg)

  ## Pages

  We use the term **pages** to denote blocks of virtual memory and to distinguish
  them from **addresses** as physical blocks. The MMU possesses a **page table**
  which is registry logging which pages correspond to which physical blocks.

  ## Shared pages

  Virtual memory allows the sharing of files and memory by multiple processes.
  Crucially the shared data doesn't have to be within the address of each process,
  instead there is a reference in the page table that each process has access to
  the shared data.

  ## Page faults

  There are two kinds of error that can occur with relation to paged memory:

  - minor page faults
    - The desired page is in main memory but the MMU doesn't currently know where
      it is
  - major page faults
    - The desired page is not in main memory at all. Therefore the kernel must
      fetch it from disk

  Minor page faults are very common and are to be expected; they resolve quickly.
  On the other hand too many major page faults can slow the system down both
  because of the time-costly process of fetching data from disk and because it
  demands more kernel resources to locate the missing page, which puts other
  processes on hold.

raw_content:
  ---
  tags: [memory, Linux]
  created: Monday, July 08, 2024
  ---

  # Memory Management Unit (MMU)

  The MMU is the bridge between the physical memory devices of the machine and the
  virtual memory space. It is a chip that sits between the CPU and the RAM and
  determines the physical location of the memory requested by the kernel as
  virtual memory.

  ![Virtual memory diagram](/img/virtual-memory-diagram.jpg)

  ## Pages

  We use the term **pages** to denote blocks of virtual memory and to distinguish
  them from **addresses** as physical blocks. The MMU possesses a **page table**
  which is registry logging which pages correspond to which physical blocks.

  ## Shared pages

  Virtual memory allows the sharing of files and memory by multiple processes.
  Crucially the shared data doesn't have to be within the address of each process,
  instead there is a reference in the page table that each process has access to
  the shared data.

  ## Page faults

  There are two kinds of error that can occur with relation to paged memory:

  - minor page faults
    - The desired page is in main memory but the MMU doesn't currently know where
      it is
  - major page faults
    - The desired page is not in main memory at all. Therefore the kernel must
      fetch it from disk

  Minor page faults are very common and are to be expected; they resolve quickly.
  On the other hand too many major page faults can slow the system down both
  because of the time-costly process of fetching data from disk and because it
  demands more kernel resources to locate the missing page, which puts other
  processes on hold.


path: zk/VirtualMemory.md
sortable_path: zkVirtualMemory.md
title: Virtual memory
checksum: d53518d5476f53f2549469757a80180fa1c29f287964788ec6185e7fa28c3f86
created: 2024-07-08 05:28:27.921902606+00:00
modified: 2024-10-19 09:18:48.224092511+00:00
metadata: {"tags":["memory","Linux","kernel"]}
lead:
  Virtual memory is an abstracted and idealised representation of the physical
  memory capacity of a machine that is presented to user space for its memory
  operations.

body:
  Virtual memory is an abstracted and idealised representation of the physical
  memory capacity of a machine that is presented to user space for its memory
  operations.

  When an OS implements virtual memory, [processes](./Processes.md) in
  [user space](./User_Space.md) cannot directly read or write to the physical
  memory. Instead they execute memory operations against virtual memory and the
  [kernel](./The_kernel.md) translates these into the actual operations against
  the memory hardware.

  The main benefits:

  - User mode processes do not have to be concerned with the physical memory
    management
  - There is a buffer between user mode processes and physical memory, meaning
    that memory cannot be accidentally corrupted by other processes in user space.

  Because the physical memory is abstracted, it can be the case that the physical
  [memory addresses](./Memory_addresses.md) are non-contiguous or even distributed
  accross different hardware components (such as the
  [cache](./Register_and_cache_memory.md) and [swap](./Swap_space.md)). Despite
  this, the memory addresses will appear contiguous in virtual memory. Each user
  space process is presented with the same range of available memory addresses and
  the same total capacity.

  It is also possible for the kernel to present user space with an available
  virtual memory capcacity that exceeds the current physical capacity of the
  machine:

  > _It's possible for the kernel and all running processes to request more
  > byte<!--  -->s of virtual memory than the total size of RAM. In that
  > situation, the OS can move move bytes of memory to secondary storage to make
  > room in RAM for newly requested memory._

  _How Computers Really Work_ (2021) p.206

  ## Virtual memory and the kernel

  The kernel itself utilises virtual memory.

  During the [boot_process](Boot_process.md), it runs in physical memory mode
  however it rapidly creates a 1:1 mapping of physical to virtual memory for
  itself. This direct mapping allows the kernel to easily access any physical
  memory location as needed, whilst retaining the benefits of virtual memory.

  With its own virtual memory established, it can then build additional virtual
  memory structures for itself and user processes.

  While the kernel uses virtual memory it can also access physical memory at any
  time wherease user space processes can only ever access virtual memory.

  The kernel virtual memory has a different range of virtual addresses to work
  with than user space virtual memory.

  Unlike user space virtual memory, the kernel has access to everything running in
  kernel address space whereas processes in user address space are partitioned
  from each other with separate address spaces that cannot interact.

  In addition to being able to access its own virtual memory (and physical memory,
  as required) the kernel can also access any user processes' virtual address.

  This said, the virtual memory space of the kernel and the virtual memory space
  of the user processes are distinct. It is not the case that the kernel is
  superset of all available virtual memory. It can access user space virtual
  memory because it sets up the tables and locations, not because it is a subset
  of its own virtual memory.

  ![Virtual memory diagram](../img/virtual-memory-diagram.jpg)

raw_content:
  ---
  tags:
    - memory
    - Linux
    - kernel
  ---

  # Virtual memory

  Virtual memory is an abstracted and idealised representation of the physical
  memory capacity of a machine that is presented to user space for its memory
  operations.

  When an OS implements virtual memory, [processes](./Processes.md) in
  [user space](./User_Space.md) cannot directly read or write to the physical
  memory. Instead they execute memory operations against virtual memory and the
  [kernel](./The_kernel.md) translates these into the actual operations against
  the memory hardware.

  The main benefits:

  - User mode processes do not have to be concerned with the physical memory
    management
  - There is a buffer between user mode processes and physical memory, meaning
    that memory cannot be accidentally corrupted by other processes in user space.

  Because the physical memory is abstracted, it can be the case that the physical
  [memory addresses](./Memory_addresses.md) are non-contiguous or even distributed
  accross different hardware components (such as the
  [cache](./Register_and_cache_memory.md) and [swap](./Swap_space.md)). Despite
  this, the memory addresses will appear contiguous in virtual memory. Each user
  space process is presented with the same range of available memory addresses and
  the same total capacity.

  It is also possible for the kernel to present user space with an available
  virtual memory capcacity that exceeds the current physical capacity of the
  machine:

  > _It's possible for the kernel and all running processes to request more
  > byte<!--  -->s of virtual memory than the total size of RAM. In that
  > situation, the OS can move move bytes of memory to secondary storage to make
  > room in RAM for newly requested memory._

  _How Computers Really Work_ (2021) p.206

  ## Virtual memory and the kernel

  The kernel itself utilises virtual memory.

  During the [boot_process](Boot_process.md), it runs in physical memory mode
  however it rapidly creates a 1:1 mapping of physical to virtual memory for
  itself. This direct mapping allows the kernel to easily access any physical
  memory location as needed, whilst retaining the benefits of virtual memory.

  With its own virtual memory established, it can then build additional virtual
  memory structures for itself and user processes.

  While the kernel uses virtual memory it can also access physical memory at any
  time wherease user space processes can only ever access virtual memory.

  The kernel virtual memory has a different range of virtual addresses to work
  with than user space virtual memory.

  Unlike user space virtual memory, the kernel has access to everything running in
  kernel address space whereas processes in user address space are partitioned
  from each other with separate address spaces that cannot interact.

  In addition to being able to access its own virtual memory (and physical memory,
  as required) the kernel can also access any user processes' virtual address.

  This said, the virtual memory space of the kernel and the virtual memory space
  of the user processes are distinct. It is not the case that the kernel is
  superset of all available virtual memory. It can access user space virtual
  memory because it sets up the tables and locations, not because it is a subset
  of its own virtual memory.

  ![Virtual memory diagram](../img/virtual-memory-diagram.jpg)


path: zk/Memory_addresses.md
sortable_path: zkMemory_addresses.md
title: Memory addresses
checksum: cc19bb80c91935b16385412e0ca63bfa50e40344fd948e1b2782feeee141d9b2
created: 2024-07-12 05:50:46.67372147+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Friday, July 12, 2024","tags":["memory"]}
lead:
  > Computers assign numeric addresses to bytes of memory and the CPU can read or
  > write to those addresses

body:
  > Computers assign numeric addresses to bytes of memory and the CPU can read or
  > write to those addresses

  We can think of the internals of RAM as grids of memory cells.

  Each single-bit cell in the grid can be identified using two dimensional
  coordinates, like in a graph. The coordinates are the location of that cell in
  the grid.

  Handling one bit at a time isn't very efficient so RAM accesses **multiple
  grids** of 1-bit memory cells in parallel. This allows for reads or writes of
  multiple bits at once, such as a whole byte.

  The location of a set of bits in memory is known as a **memory address**.

  ### Demonstration

  Let's imagine we have a computer system that can address up to 64KB of memory
  and our system is byte addressable. This means there are
  $64 \cdot 1024 = 65,536$ bytes of memory because 1KB = 1024 bytes.

  We therefore have 65,536 addresses and each address can store one byte. So our
  addresses go from 0 to 65, 535.

  We now need to consider how many bits we need to uniquely represent an address
  on this system.

  What does this mean? Although there are approximately 64 thousand bytes of
  memory, to refer to each byte we can't just use 1, 2, 3... because computers use
  binary numbers. We need a binary number to refer to a given byte in the the 64KB
  of memory. The question we are asking is: how long does this binary number need
  to be to be able to represent each of the 64 thousand bytes?

  1 bit can represent two addresses: 0 and 1. 2 bits can represent four addresses:
  00, 01, 10, 11. The formula is as follows: number of addresses = $2^n$ where $n$
  is the number of bits.

  We need to reverse this formula to find out how many bits we need to represent a
  given number of addresses. We can do this with a [logarithm](Logarithms.md).

  We can reverse the formula as follows: number of bits = $\log_2$ (number of
  addresses).

  In our case we have 65,536 addresses so we need $\log_2(65,536)$ bits to
  represent each address. This is approximately 16 bits. Thus a 16 bit memory
  iaddress is needed to address 65, 546 bytes.

  Using memory addresses we end up with tables like the following:

  | Memory address   | Data             |
  | ---------------- | ---------------- |
  | 0000000000000000 | 1010101010101010 |
  | 0000000000000001 | 0010001001001011 |
  | 0000000000000010 | 0010001001001010 |

  This is hard to parse so we can instead use
  [hexadecimal numbers](Hexadecimal_number_system.md) to represent the addresses:

  | Memory address (as hex) | Data (as binary) |
  | ----------------------- | ---------------- |
  | 0x0000                  | 1010101010101010 |
  | 0x0001                  | 0010001001001011 |
  | 0x0002                  | 0010001001001010 |

  By itself, the the data is meaningless but we know from
  [binary encoding](Binary_encoding.md) that the binary data will correspond to
  some meaningful data, such as a character or a colour, depending on the encoding
  scheme used. The above table could correspond to the characters for 'A', 'B' and
  'C' in the ASCII encoding scheme:

  | Memory address (as hex) | Data (as binary) | Data (as ASCII) |
  | ----------------------- | ---------------- | --------------- |
  | 0x0000                  | 1010101010101010 | A               |
  | 0x0001                  | 0010001001001011 | B               |
  | 0x0002                  | 0010001001001010 | C               |

raw_content:
  ---
  tags: [memory]
  created: Friday, July 12, 2024
  ---

  # Memory addresses

  > Computers assign numeric addresses to bytes of memory and the CPU can read or
  > write to those addresses

  We can think of the internals of RAM as grids of memory cells.

  Each single-bit cell in the grid can be identified using two dimensional
  coordinates, like in a graph. The coordinates are the location of that cell in
  the grid.

  Handling one bit at a time isn't very efficient so RAM accesses **multiple
  grids** of 1-bit memory cells in parallel. This allows for reads or writes of
  multiple bits at once, such as a whole byte.

  The location of a set of bits in memory is known as a **memory address**.

  ### Demonstration

  Let's imagine we have a computer system that can address up to 64KB of memory
  and our system is byte addressable. This means there are
  $64 \cdot 1024 = 65,536$ bytes of memory because 1KB = 1024 bytes.

  We therefore have 65,536 addresses and each address can store one byte. So our
  addresses go from 0 to 65, 535.

  We now need to consider how many bits we need to uniquely represent an address
  on this system.

  What does this mean? Although there are approximately 64 thousand bytes of
  memory, to refer to each byte we can't just use 1, 2, 3... because computers use
  binary numbers. We need a binary number to refer to a given byte in the the 64KB
  of memory. The question we are asking is: how long does this binary number need
  to be to be able to represent each of the 64 thousand bytes?

  1 bit can represent two addresses: 0 and 1. 2 bits can represent four addresses:
  00, 01, 10, 11. The formula is as follows: number of addresses = $2^n$ where $n$
  is the number of bits.

  We need to reverse this formula to find out how many bits we need to represent a
  given number of addresses. We can do this with a [logarithm](Logarithms.md).

  We can reverse the formula as follows: number of bits = $\log_2$ (number of
  addresses).

  In our case we have 65,536 addresses so we need $\log_2(65,536)$ bits to
  represent each address. This is approximately 16 bits. Thus a 16 bit memory
  iaddress is needed to address 65, 546 bytes.

  Using memory addresses we end up with tables like the following:

  | Memory address   | Data             |
  | ---------------- | ---------------- |
  | 0000000000000000 | 1010101010101010 |
  | 0000000000000001 | 0010001001001011 |
  | 0000000000000010 | 0010001001001010 |

  This is hard to parse so we can instead use
  [hexadecimal numbers](Hexadecimal_number_system.md) to represent the addresses:

  | Memory address (as hex) | Data (as binary) |
  | ----------------------- | ---------------- |
  | 0x0000                  | 1010101010101010 |
  | 0x0001                  | 0010001001001011 |
  | 0x0002                  | 0010001001001010 |

  By itself, the the data is meaningless but we know from
  [binary encoding](Binary_encoding.md) that the binary data will correspond to
  some meaningful data, such as a character or a colour, depending on the encoding
  scheme used. The above table could correspond to the characters for 'A', 'B' and
  'C' in the ASCII encoding scheme:

  | Memory address (as hex) | Data (as binary) | Data (as ASCII) |
  | ----------------------- | ---------------- | --------------- |
  | 0x0000                  | 1010101010101010 | A               |
  | 0x0001                  | 0010001001001011 | B               |
  | 0x0002                  | 0010001001001010 | C               |


path: zk/What_is_memory.md
sortable_path: zkWhat_is_memory.md
title: What is memory ?
checksum: a8bdfe8e8fba49ccf433d7a0d90744c2b434915a5f1a4206998c53908a9d521a
created: 2024-07-12 06:24:41.184588602+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"created":"Friday, July 12, 2024","tags":["memory"]}
lead:
  > A CPU is just an operator on memory. It reads its instructions and data from
  > the memory and writes back out to the memory. (Ward 2021)

body:
  > A CPU is just an operator on memory. It reads its instructions and data from
  > the memory and writes back out to the memory. (Ward 2021)

  When a [CPU](CPU_architecture.md) executes a program, it needs a place to store
  the program's **instructions** and **related data**. This is the role of memory.

  The data that comprises a program is a series of bits. The basic unit of memory
  storage is a **memory cell**: a circuit that can store a single bit.

  [RAM](./DRAM_and_SRAM_memory.md) memory is _volatile_ : the memory is only
  retained whilst the computer has a power supply and is wiped when the computer
  is rebooted. This contrasts with the memory of the harddisk which is
  non-volatile and is retained after a reboot.

  Programs that are executing are loaded into memory because the chips that
  comprise memory can read and store data much faster than the harddisk. It would
  be possible to run a program from the harddisk but it would be 500 - 1000 times
  slower than memory.

raw_content:
  ---
  tags: [memory]
  created: Friday, July 12, 2024
  ---

  # What is memory ?

  > A CPU is just an operator on memory. It reads its instructions and data from
  > the memory and writes back out to the memory. (Ward 2021)

  When a [CPU](CPU_architecture.md) executes a program, it needs a place to store
  the program's **instructions** and **related data**. This is the role of memory.

  The data that comprises a program is a series of bits. The basic unit of memory
  storage is a **memory cell**: a circuit that can store a single bit.

  [RAM](./DRAM_and_SRAM_memory.md) memory is _volatile_ : the memory is only
  retained whilst the computer has a power supply and is wiped when the computer
  is rebooted. This contrasts with the memory of the harddisk which is
  non-volatile and is retained after a reboot.

  Programs that are executing are loaded into memory because the chips that
  comprise memory can read and store data much faster than the harddisk. It would
  be possible to run a program from the harddisk but it would be 500 - 1000 times
  slower than memory.


path: zk/DRAM_and_SRAM_memory.md
sortable_path: zkDRAM_and_SRAM_memory.md
title: DRAM and SRAM memory
lead: There are two types of RAM memory: [SRAM and DRAM](./DRAM_and_SRAM_memory.md).
checksum: 70881014b2548697de343ae573dcc59348fe3b75eacbbc3f6d4a638f48b98d8a
created: 2024-07-12 06:25:44.482673223+00:00
modified: 2024-10-18 18:12:33.821491436+00:00
metadata: {"created":"Friday, July 12, 2024","tags":["memory","hardware"]}
body:
  There are two types of RAM memory: [SRAM and DRAM](./DRAM_and_SRAM_memory.md).

  ### DRAM

  DRAM uses capacitors to create the memory cell:

  > a **capacitor** is an electronic component that stores electrical energy in an
  > electrical field. A device which can accumulate and release electrical charge.

  In a DRAM cell, each bit of data is stored as a charge in a capacitor. The
  presence of charge represents a '1' bit and the absence of charge represents a
  '0' bit. Each of these cells is paired with a [transistor](Transistors.md) that
  controls the reading and writing of data.

  However capacitors lose [charge](Current.md) over time due to leaks. As a result
  DRAM is memory that needs to be refreshed (recharged) frequently. For this
  reason and because it only uses one transistor and capacitor per bit, DRAM is
  the less expensive form of volatile memory.

  ### SRAM

  SRAM (Static Random Access Memory) is also volatile memory but its electronical
  implementation is different. Unlike DRAM it doesn't use capacitors. Consequently
  the transistors do not leak or need to be refreshed, hence why SRAM is _static_
  and DRAM is _dynamic_.

  SRAM uses [flip flops](Flip_flops.md) to store the bits. It also uses multiple
  transistors per bit. This makes it faster than DRAM but more expensive. DRAM is
  at least ten times slower than SRAM.

raw_content:
  ---
  tags: [memory, hardware]
  created: Friday, July 12, 2024
  ---

  # DRAM and SRAM memory

  There are two types of RAM memory: [SRAM and DRAM](./DRAM_and_SRAM_memory.md).

  ### DRAM

  DRAM uses capacitors to create the memory cell:

  > a **capacitor** is an electronic component that stores electrical energy in an
  > electrical field. A device which can accumulate and release electrical charge.

  In a DRAM cell, each bit of data is stored as a charge in a capacitor. The
  presence of charge represents a '1' bit and the absence of charge represents a
  '0' bit. Each of these cells is paired with a [transistor](Transistors.md) that
  controls the reading and writing of data.

  However capacitors lose [charge](Current.md) over time due to leaks. As a result
  DRAM is memory that needs to be refreshed (recharged) frequently. For this
  reason and because it only uses one transistor and capacitor per bit, DRAM is
  the less expensive form of volatile memory.

  ### SRAM

  SRAM (Static Random Access Memory) is also volatile memory but its electronical
  implementation is different. Unlike DRAM it doesn't use capacitors. Consequently
  the transistors do not leak or need to be refreshed, hence why SRAM is _static_
  and DRAM is _dynamic_.

  SRAM uses [flip flops](Flip_flops.md) to store the bits. It also uses multiple
  transistors per bit. This makes it faster than DRAM but more expensive. DRAM is
  at least ten times slower than SRAM.


path: zk/Role_of_memory_in_computation.md
sortable_path: zkRole_of_memory_in_computation.md
title: The role of memory in computation
checksum: 201e429a3109d53637859ac37e568ac5254ff329ef4620a4d68cc4b4cb3d4708
created: 2024-07-12 07:36:37.987161729+00:00
modified: 2024-10-18 17:55:56.292394043+00:00
metadata: {"created":"Friday, July 12, 2024","tags":["memory"]}
lead:
  The following steps outline the way in which memory interacts with the processor
  during computational cycles, once the [bootstrapping](Boot_process.md) process
  has completed and the OS kernel is itself loaded into memory.

body:
  The following steps outline the way in which memory interacts with the processor
  during computational cycles, once the [bootstrapping](Boot_process.md) process
  has completed and the OS kernel is itself loaded into memory.

  1. A file is loaded from the harddisk into memory.
  2. The instruction at the first address is sent to the CPU, travelling accross
     the data bus part of the [system bus](Bus.md).
  3. The CPU processes this instruction and then sends a request accross the
     address bus part of the system bus for the next instruction to the memory
     controller within the [chipset](Chipset_and_controllers.md).
  4. The chipset finds where this instruction is stored within the
     [DRAM](Memory.md#dram) and issues a request to have it read out and send to
     the CPU over the data bus.

  > This is a simplified account; it is not the case that only single requests are
  > passed back and forth. This would be inefficient and time-wasting. The kernel
  > sends to the CPU not just the first instruction in the requested file but also
  > a number of instructions that immediately follow it.

  ![Memory flow diagram](/img/memory-flow.svg)

  Every part of the above process - the journey accross the bus, the lookup in the
  controller, the operations on the DRAM, the journey back accross the bus - takes
  multiple CPU clock cycles.

raw_content:
  ---
  tags: [memory]
  created: Friday, July 12, 2024
  ---

  # The role of memory in computation

  The following steps outline the way in which memory interacts with the processor
  during computational cycles, once the [bootstrapping](Boot_process.md) process
  has completed and the OS kernel is itself loaded into memory.

  1. A file is loaded from the harddisk into memory.
  2. The instruction at the first address is sent to the CPU, travelling accross
     the data bus part of the [system bus](Bus.md).
  3. The CPU processes this instruction and then sends a request accross the
     address bus part of the system bus for the next instruction to the memory
     controller within the [chipset](Chipset_and_controllers.md).
  4. The chipset finds where this instruction is stored within the
     [DRAM](Memory.md#dram) and issues a request to have it read out and send to
     the CPU over the data bus.

  > This is a simplified account; it is not the case that only single requests are
  > passed back and forth. This would be inefficient and time-wasting. The kernel
  > sends to the CPU not just the first instruction in the requested file but also
  > a number of instructions that immediately follow it.

  ![Memory flow diagram](/img/memory-flow.svg)

  Every part of the above process - the journey accross the bus, the lookup in the
  controller, the operations on the DRAM, the journey back accross the bus - takes
  multiple CPU clock cycles.


path: zk/Register_and_cache_memory.md
sortable_path: zkRegister_and_cache_memory.md
title: Register and cache memory
lead: The CPU has its own memory devices in the form of registers and cache memory.
checksum: 51b2d2998fb892d96af6bbd7f6f496ba943b2aac343058bda58d60208e691687
created: 2024-07-12 07:38:42.783507026+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"created":"Friday, July 12, 2024","tags":["memory","CPU"]}
body:
  The CPU has its own memory devices in the form of registers and cache memory.

  [Registers](./CPU_architecture.md#Registers) are a form of memory that are
  positioned on the same chip as the CPU. They are very fast but can only store a
  small amount of data. They are used to store the results of calculations and the
  addresses of the next instructions to be processed.

  The cache is SRAM memory that is separate from the DRAM memory which comprises
  the main memory. It exists in order to boost perfomance when executing the
  read/request cycles of the steps detailed above.

  There are two types of cache memory:

  - L1 cache
    - Situated on the CPU chip itself
    - Fastest to access but stores less
  - L2 cache
    - Situated outside of the CPU on its own chip
    - Slower to access than L1 but can store more data

  The L1 cache is the fastest since the data has less distance to travel when
  moving to and from the CPU. This said, the L2 cache is still very fast when
  compared to the main memory, both because it is SRAM rather than DRAM and
  because it is closer to the processor than the main memory.

  Cache controllers use complex algorithms to determine what should go into the
  cache to facilitate the best performance, but generally they work on the
  principle that what has been previously used by the CPU will be requested again
  soon. If the CPU has just asked for an instruction at memory location 555 it's
  very likely that it will next ask for the one at 556, and after that the one at
  557 and so on. The cache's controller circuits therefore go ahead and fetch
  these from slow [DRAM to fast SRAM](./DRAM_and_SRAM_memory.md).

raw_content:
  ---
  tags: [memory, CPU]
  created: Friday, July 12, 2024
  ---

  # Register and cache memory

  The CPU has its own memory devices in the form of registers and cache memory.

  [Registers](./CPU_architecture.md#Registers) are a form of memory that are
  positioned on the same chip as the CPU. They are very fast but can only store a
  small amount of data. They are used to store the results of calculations and the
  addresses of the next instructions to be processed.

  The cache is SRAM memory that is separate from the DRAM memory which comprises
  the main memory. It exists in order to boost perfomance when executing the
  read/request cycles of the steps detailed above.

  There are two types of cache memory:

  - L1 cache
    - Situated on the CPU chip itself
    - Fastest to access but stores less
  - L2 cache
    - Situated outside of the CPU on its own chip
    - Slower to access than L1 but can store more data

  The L1 cache is the fastest since the data has less distance to travel when
  moving to and from the CPU. This said, the L2 cache is still very fast when
  compared to the main memory, both because it is SRAM rather than DRAM and
  because it is closer to the processor than the main memory.

  Cache controllers use complex algorithms to determine what should go into the
  cache to facilitate the best performance, but generally they work on the
  principle that what has been previously used by the CPU will be requested again
  soon. If the CPU has just asked for an instruction at memory location 555 it's
  very likely that it will next ask for the one at 556, and after that the one at
  557 and so on. The cache's controller circuits therefore go ahead and fetch
  these from slow [DRAM to fast SRAM](./DRAM_and_SRAM_memory.md).


path: zk/The_memory_hierarchy.md
sortable_path: zkThe_memory_hierarchy.md
title: The memory hierarchy
checksum: 520f038e59b4ff5a9e388265ab28df99ee8da22437dbdbe7be84333c974b1aac
created: 2024-07-12 07:43:51.419849451+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"created":"Friday, July 12, 2024","tags":["memory"]}
lead:
  The diagram below compares the different forms of memory within a computing
  device in terms of speed, monetary cost and capacity:

body:
  The diagram below compares the different forms of memory within a computing
  device in terms of speed, monetary cost and capacity:

  ![Memory hierarchy diagram](/img/Memory-Hierarchy.jpg)

raw_content:
  ---
  tags: [memory]
  created: Friday, July 12, 2024
  ---

  # The memory hierarchy

  The diagram below compares the different forms of memory within a computing
  device in terms of speed, monetary cost and capacity:

  ![Memory hierarchy diagram](/img/Memory-Hierarchy.jpg)


path: zk/Operating_system_API.md
sortable_path: zkOperating_system_API.md
title: Operating system API
checksum: 7aeb44d375bb920e23b475fc0948194c841f603bdf7371ea17aa577bcf8e1d13
created: 2024-07-17 05:57:15.872985575+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"created":"Wednesday, July 17, 2024","tags":["operating-systems"]}
lead:
  A user can interact with the operating system either via the GUI or the
  [shell](Intro_to_the_Unix_shell.md). (Both ultimately access the OS via the
  shell since the UI is an abstraction layer of the shell.)

body:
  A user can interact with the operating system either via the GUI or the
  [shell](Intro_to_the_Unix_shell.md). (Both ultimately access the OS via the
  shell since the UI is an abstraction layer of the shell.)

  When _programs_ interact with the operating system they use the **operating
  system's API** directly. The GUI and shell also invoke the OS API but the access
  is mediated.

  Every OS has an API. This specification details how a program should interact
  with the OS, exposing functions, properties and data types.

  Whether using the GUI or a terminal emulator, the shell translates the user's
  commands into API calls. The API the invokes internal operating system code to
  perform the action.

  ![OS API diagram](../img/OS-api.svg)

  ## Example: opening a file

  A user can use a file explorer and the mouse to open a file. Or they can open
  the file by invoking it from the terminal.

  A running application doesn't have to do this. If it's remit includes opening
  files, it can do so by calling the OS API.

  On a Unix-like system the applicaton would use the `open` API function to open
  the file:

  ```
  open("foo.txt", 0_WRONLY|0_CREAT)
  ```

  This command makes the action write-only and creates the file because it does
  not yet exist. When the user opens a file in their File Explorer, the
  application translates their graphical actions into this API call.

  ## Different OSs have different APIs

  Unix-like and Linux operating systems use variations on the **Portable Operating
  System Interface** (POSIX) standard. This defines the OS API and also the rules
  for the shell's behaviour and its included utilities.

  Variations on this include _Cocoa_, macOS's API and Android's _Android Platform_
  API.

  Windows has its own API called _The Windows API_. The original version was
  16-bit called _Win16_, then when it moved to 32-bits it became _Win32_. When it
  moved to an x86 [instruction_set_architecture](Instruction_set_architectures.md)
  it became _Win64_. Starting with Windows 10, it introduced the _Universal
  Windows Platform_ which aims to make app development consistent accross
  different devices that run Windows.

raw_content:
  ---
  tags: [operating-systems]
  created: Wednesday, July 17, 2024
  ---

  # Operating system API

  A user can interact with the operating system either via the GUI or the
  [shell](Intro_to_the_Unix_shell.md). (Both ultimately access the OS via the
  shell since the UI is an abstraction layer of the shell.)

  When _programs_ interact with the operating system they use the **operating
  system's API** directly. The GUI and shell also invoke the OS API but the access
  is mediated.

  Every OS has an API. This specification details how a program should interact
  with the OS, exposing functions, properties and data types.

  Whether using the GUI or a terminal emulator, the shell translates the user's
  commands into API calls. The API the invokes internal operating system code to
  perform the action.

  ![OS API diagram](../img/OS-api.svg)

  ## Example: opening a file

  A user can use a file explorer and the mouse to open a file. Or they can open
  the file by invoking it from the terminal.

  A running application doesn't have to do this. If it's remit includes opening
  files, it can do so by calling the OS API.

  On a Unix-like system the applicaton would use the `open` API function to open
  the file:

  ```
  open("foo.txt", 0_WRONLY|0_CREAT)
  ```

  This command makes the action write-only and creates the file because it does
  not yet exist. When the user opens a file in their File Explorer, the
  application translates their graphical actions into this API call.

  ## Different OSs have different APIs

  Unix-like and Linux operating systems use variations on the **Portable Operating
  System Interface** (POSIX) standard. This defines the OS API and also the rules
  for the shell's behaviour and its included utilities.

  Variations on this include _Cocoa_, macOS's API and Android's _Android Platform_
  API.

  Windows has its own API called _The Windows API_. The original version was
  16-bit called _Win16_, then when it moved to 32-bits it became _Win32_. When it
  moved to an x86 [instruction_set_architecture](Instruction_set_architectures.md)
  it became _Win64_. Starting with Windows 10, it introduced the _Universal
  Windows Platform_ which aims to make app development consistent accross
  different devices that run Windows.


path: zk/Memory_versus_processor.md
sortable_path: zkMemory_versus_processor.md
title: Memory versus processor
checksum: 6934369b9b4e8234521db8a1b40ca54359cf20097fb1930b87ed62bdf8ea3206
created: 2024-08-01 17:07:42.214100662+00:00
modified: 2024-10-18 18:17:01.726525085+00:00
metadata: {"created":"Thursday, August 01, 2024","tags":["question"]}
lead:
  > Would a more powerful processor with average or reduced memory capacity
  > perform better than, equal to, or less than a less powerful processor with
  > more memory, in principle?

body:
  > Would a more powerful processor with average or reduced memory capacity
  > perform better than, equal to, or less than a less powerful processor with
  > more memory, in principle?

raw_content:
  ---
  tags: ["question"]
  created: Thursday, August 01, 2024
  ---

  # Memory versus processor

  > Would a more powerful processor with average or reduced memory capacity
  > perform better than, equal to, or less than a less powerful processor with
  > more memory, in principle?


path: zk/Headless_Raspi_network_setup.md
sortable_path: zkHeadless_Raspi_network_setup.md
title: Headless network setup
checksum: 1130cb421dc289d445aaedfbcfa79cac0079753aa0c992958503737758883f4b
created: 2024-08-03 06:05:43.878164659+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
lead:
  If you are using a headless system and you aren't using an ethernet connection,
  you face a chicken-egg issue of how to connect the device to WiFi via `ssh` when
  the device isn't yet on the network.

body:
  If you are using a headless system and you aren't using an ethernet connection,
  you face a chicken-egg issue of how to connect the device to WiFi via `ssh` when
  the device isn't yet on the network.

  You can get around this by saving a file called `wp_supplicant.conf` to the
  `/boot` directory of the device. This specifies the SSID name and password and
  will be used to auto-connect when the device boots.

  For example:

  ```
  country=GB
  ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
  update_config=1
  network={
       ssid="SKYVL7XP"
       scan_ssid=1
       psk="WIFI_PASSWORD"
       key_mgmt=WPA-PSK
  }
  ```

  ## Related notes

raw_content:
  ---
  id: hb10
  tags:
    - networks
    - Linux
    - raspberry-pi
    - procedural
  created: Wednesday, June 12, 2024
  ---

  # Headless network setup

  If you are using a headless system and you aren't using an ethernet connection,
  you face a chicken-egg issue of how to connect the device to WiFi via `ssh` when
  the device isn't yet on the network.

  You can get around this by saving a file called `wp_supplicant.conf` to the
  `/boot` directory of the device. This specifies the SSID name and password and
  will be used to auto-connect when the device boots.

  For example:

  ```
  country=GB
  ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
  update_config=1
  network={
       ssid="SKYVL7XP"
       scan_ssid=1
       psk="WIFI_PASSWORD"
       key_mgmt=WPA-PSK
  }
  ```

  ## Related notes

metadata:
  {"created":"Wednesday, June 12, 2024","id":"hb10","tags":["networks","Linux","raspberry-pi","procedural"]}


path: zk/Network_fundamentals.md
sortable_path: zkNetwork_fundamentals.md
title: Network fundamentals
checksum: cc82f0dcd8355ab2f7af5eda572679f060a725888b69ea268ca70bdaacb8a794
created: 2024-08-03 06:06:05.867268631+00:00
modified: 2024-10-18 17:55:56.292394043+00:00
metadata: {"created":"Saturday, August 03, 2024","tags":["networks","network-protocols"]}
lead:
  > A network is a system that allows computing devices to communicate and
  > exchange information with each other.

body:
  > A network is a system that allows computing devices to communicate and
  > exchange information with each other.

  In order for devices to be able to communicate they must share a common
  **communication protocol**.

  A protocol specifies **rules determining how information is to be exchanged**.
  (Simply connecting two devices is not sufficient for them to be able to
  communicate, they must have a shared language.)

  The nodes of a network are called [hosts](Network_hosts.md).

raw_content:
  ---
  tags: [networks, network-protocols]
  created: Saturday, August 03, 2024
  ---

  # Network fundamentals

  > A network is a system that allows computing devices to communicate and
  > exchange information with each other.

  In order for devices to be able to communicate they must share a common
  **communication protocol**.

  A protocol specifies **rules determining how information is to be exchanged**.
  (Simply connecting two devices is not sufficient for them to be able to
  communicate, they must have a shared language.)

  The nodes of a network are called [hosts](Network_hosts.md).


path: zk/Torrenting.md
sortable_path: zkTorrenting.md
title: Torrenting
checksum: 5fbc5690a97e5fb717e54d69288e597c1f5003ed492b254c9f864e4b38bd948b
created: 2024-08-03 06:10:21.423432559+00:00
modified: 2024-10-18 17:55:56.292394043+00:00
metadata: {"created":"Saturday, August 03, 2024","tags":["question","networks"]}
lead:
  **What is torrenting? Is this the same thing as a peer-to-peer network? How do
  these technologies fit within the client-server paradigm?**

body:
  **What is torrenting? Is this the same thing as a peer-to-peer network? How do
  these technologies fit within the client-server paradigm?**

  Torrenting is a specific application of
  [peer_to_peer_networking](Peer_to_peer_network.md) applied to file-sharing.

  Each peer on the network both consumes resources and makes resources available
  to others.

  ## Procedure

  Large files are divided into small equal sized pieces and each piece is assigned
  a unique hash identifier. The `.torrent` file details the number of pieces,
  their sizes and hash values.

  Clients request different pieces from different peers simultaneously. As pieces
  are received, they are verified against their hash. Corrupted or incorrect
  pieces are discarded and re-downloaded.

  The client software reassembles the pieces in the correct order and the final
  file will be identical to the original which can be verified by comparing
  hashes.

  ## Key terms

  - _tracker_
    - a server that coordinates communication between peers (so there is a server
      but its role is more lightweight than client-server, coordinating the peers
      rather than hosting or distributing file content)
  - _swarm_
    - a group of peers sharing the same file
  - _seeding_
    - uploading file pieces to other peers
  - _leeching_
    - downloading file pieces from other peers

  > An alternative to a tracker server is to use trackerless methods such as
  > distributed hash tables

  ## Terminological clarification

  - _BitTorrent_ is the communications [protocols](Network_protocols.md) for P2P
    file sharing. It was also a client software developed by the company
    BitTorrent Inc.

  - _torrenting_ is the general terms for that act of sharing files using the
    BitTorrent protocol

  ## Napster

  Napster pre-dated the emergence of the BitTorrent protocol. It did not segment
  files and files were downloaded directly from peers. It was a centralised P2P
  network because there was a central server that indexed all available files
  which kept a database of all connected users and their files.

  When you searched Napster you would be querying this DB on the server. The
  server would facilitate a direct connection between the consumer and provider
  peer for the data exchange.

  When users share with BitTorrent the `.torrent` file is the source of truth and
  a swarm is created around the users who have possess the file.

raw_content:
  ---
  tags: [question, networks]
  created: Saturday, August 03, 2024
  ---

  # Torrenting

  **What is torrenting? Is this the same thing as a peer-to-peer network? How do
  these technologies fit within the client-server paradigm?**

  Torrenting is a specific application of
  [peer_to_peer_networking](Peer_to_peer_network.md) applied to file-sharing.

  Each peer on the network both consumes resources and makes resources available
  to others.

  ## Procedure

  Large files are divided into small equal sized pieces and each piece is assigned
  a unique hash identifier. The `.torrent` file details the number of pieces,
  their sizes and hash values.

  Clients request different pieces from different peers simultaneously. As pieces
  are received, they are verified against their hash. Corrupted or incorrect
  pieces are discarded and re-downloaded.

  The client software reassembles the pieces in the correct order and the final
  file will be identical to the original which can be verified by comparing
  hashes.

  ## Key terms

  - _tracker_
    - a server that coordinates communication between peers (so there is a server
      but its role is more lightweight than client-server, coordinating the peers
      rather than hosting or distributing file content)
  - _swarm_
    - a group of peers sharing the same file
  - _seeding_
    - uploading file pieces to other peers
  - _leeching_
    - downloading file pieces from other peers

  > An alternative to a tracker server is to use trackerless methods such as
  > distributed hash tables

  ## Terminological clarification

  - _BitTorrent_ is the communications [protocols](Network_protocols.md) for P2P
    file sharing. It was also a client software developed by the company
    BitTorrent Inc.

  - _torrenting_ is the general terms for that act of sharing files using the
    BitTorrent protocol

  ## Napster

  Napster pre-dated the emergence of the BitTorrent protocol. It did not segment
  files and files were downloaded directly from peers. It was a centralised P2P
  network because there was a central server that indexed all available files
  which kept a database of all connected users and their files.

  When you searched Napster you would be querying this DB on the server. The
  server would facilitate a direct connection between the consumer and provider
  peer for the data exchange.

  When users share with BitTorrent the `.torrent` file is the source of truth and
  a swarm is created around the users who have possess the file.


path: zk/Network_hosts.md
sortable_path: zkNetwork_hosts.md
title: Network hosts
checksum: b4ac6a2c1a8d9cbdeb0ebd886484e545c5d398dfb3ef09c0c4aa8380b3d430f1
created: 2024-08-03 06:14:23.069677144+00:00
modified: 2024-10-18 18:18:24.428070835+00:00
metadata: {"created":"Saturday, August 03, 2024","tags":["networks"]}
lead:
  A network **host** or **node** is a single computing device attached to a
  ![network](Network_fundamentals.md)

body:
  A network **host** or **node** is a single computing device attached to a
  ![network](Network_fundamentals.md)

  Hosts can act as servers or clients, or both.

  ## Server

  A network server is a host that listens for inbound network connections and
  provides services to other hosts. Examples include web and email servers.

  ## Client

  A network client is a host that makes outbound connections and requests servcies
  from network servers.

raw_content:
  ---
  tags: [networks]
  created: Saturday, August 03, 2024
  ---

  # Network hosts

  A network **host** or **node** is a single computing device attached to a
  ![network](Network_fundamentals.md)

  Hosts can act as servers or clients, or both.

  ## Server

  A network server is a host that listens for inbound network connections and
  provides services to other hosts. Examples include web and email servers.

  ## Client

  A network client is a host that makes outbound connections and requests servcies
  from network servers.


path: zk/Internet_fundamentals.md
sortable_path: zkInternet_fundamentals.md
title: Internet fundamentals
checksum: 74fbd8a154de3ccf6b6fc4bba7e3b09e2c7c260d38dd2e21d11b735908d767e2
created: 2024-08-03 06:51:56.584079758+00:00
modified: 2024-10-18 17:55:56.289060649+00:00
metadata: {"created":"Saturday, August 03, 2024","tags":["networks"]}
lead:
  The internet is a _[network](Network_fundamentals.md) of networks_. A globally
  connected set of computer networks that all use a suite of common
  [protocols](Network_protocols.md).

body:
  The internet is a _[network](Network_fundamentals.md) of networks_. A globally
  connected set of computer networks that all use a suite of common
  [protocols](Network_protocols.md).

  The protocols governing the communication between networks on the internet are
  specified in the **Internet Protocol Suite**.

  This combines two protocols:

  - **TCP**
    - The Transmission Control Protocol
  - **IP**
    - The Internet Protocol

  Hence the Internet Protocol Suite is also known as "TCP/IP".

  The TCP/IP is a network stack comprising four layers, each with its own
  protocols. From the top:

  ![TCP/IP stack diagram](../img/TCP_IP.png)

  - The [Link Layer](Link_Layer_of_Internet_Protocol.md) concerns communication
    between devices on the same local network. The transfer of information once it
    has been received from the internet or its transfer between devices before it
    sent out into the internet. Ethernet and WiFi are two core protocols utilised
    at this layer.

  - The [Internet Layer](Internet_Layer_of_Internet_Protocol.md) is concerned with
    communication accross networks. It is responsible for identifying hosts on the
    inter-network by their addresses and routing data from network to network.

  - The [Transport Layer](Transport_Layer_of_Internet_Protocol.md) creates a
    commmunication channel between two hosts using ports and sockets.

  - The [Application Layer](Application_Layer_of_Internet_Protocol.md) concerned
    with software that utilises information received and sent over the internet,
    for instance web pages and emails. This is the highest level of abstraction
    for which all the lower levels serve as "plumbing". Examples of protocols
    operative at the Application Layer include HTTP and SSH.

  See:
  [Example_scenario_internet_data_transfer](Example_scenario_internet_data_transfer.md)

raw_content:
  ---
  tags: [networks]
  created: Saturday, August 03, 2024
  ---

  # Internet fundamentals

  The internet is a _[network](Network_fundamentals.md) of networks_. A globally
  connected set of computer networks that all use a suite of common
  [protocols](Network_protocols.md).

  The protocols governing the communication between networks on the internet are
  specified in the **Internet Protocol Suite**.

  This combines two protocols:

  - **TCP**
    - The Transmission Control Protocol
  - **IP**
    - The Internet Protocol

  Hence the Internet Protocol Suite is also known as "TCP/IP".

  The TCP/IP is a network stack comprising four layers, each with its own
  protocols. From the top:

  ![TCP/IP stack diagram](../img/TCP_IP.png)

  - The [Link Layer](Link_Layer_of_Internet_Protocol.md) concerns communication
    between devices on the same local network. The transfer of information once it
    has been received from the internet or its transfer between devices before it
    sent out into the internet. Ethernet and WiFi are two core protocols utilised
    at this layer.

  - The [Internet Layer](Internet_Layer_of_Internet_Protocol.md) is concerned with
    communication accross networks. It is responsible for identifying hosts on the
    inter-network by their addresses and routing data from network to network.

  - The [Transport Layer](Transport_Layer_of_Internet_Protocol.md) creates a
    commmunication channel between two hosts using ports and sockets.

  - The [Application Layer](Application_Layer_of_Internet_Protocol.md) concerned
    with software that utilises information received and sent over the internet,
    for instance web pages and emails. This is the highest level of abstraction
    for which all the lower levels serve as "plumbing". Examples of protocols
    operative at the Application Layer include HTTP and SSH.

  See:
  [Example_scenario_internet_data_transfer](Example_scenario_internet_data_transfer.md)


path: zk/Network_protocols.md
sortable_path: zkNetwork_protocols.md
title: Network protocols
checksum: 25b22a05f48950de12dbb593e2bde94b3da329d5b20dec546c4b3624f40f3767
created: 2024-08-03 06:53:42.143909607+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"created":"Saturday, August 03, 2024","tags":["networks"]}
lead:
  Protocols are agreed standards specifying and governing communication between
  nodes on a [network](Network_fundamentals.md).

body:
  Protocols are agreed standards specifying and governing communication between
  nodes on a [network](Network_fundamentals.md).

  Network protocols are typically _layered_. The implmentation of a network layer
  is a _stack_.

  Stacks enshrine encapsulation: not every layer has to be directly concerned with
  every other layer. Only adjacent layers that interface with each other need to
  be in contact.

  A prime example of a layered network stack is the
  [internet_protocol_suite](Internet_fundamentals.md).

raw_content:
  ---
  tags: [networks]
  created: Saturday, August 03, 2024
  ---

  # Network protocols

  Protocols are agreed standards specifying and governing communication between
  nodes on a [network](Network_fundamentals.md).

  Network protocols are typically _layered_. The implmentation of a network layer
  is a _stack_.

  Stacks enshrine encapsulation: not every layer has to be directly concerned with
  every other layer. Only adjacent layers that interface with each other need to
  be in contact.

  A prime example of a layered network stack is the
  [internet_protocol_suite](Internet_fundamentals.md).


path: zk/Link_Layer_of_Internet_Protocol.md
sortable_path: zkLink_Layer_of_Internet_Protocol.md
title: Link Layer of the Internet Protocol Suite
checksum: aa296b50c848330270028a57ee3f3d5ebe1b7a401db10a81319b40e651035091
created: 2024-08-06 05:27:00.733151391+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Tuesday, August 06, 2024","tags":["networks","internet"]}
lead:
  > The Link Layer is the lowest level of the network stack that makes up the
  > Internet Protocol Suite. It is concerned with the physical and logical
  > connections between [hosts](./Network_hosts.md) on the same local network.

body:
  > The Link Layer is the lowest level of the network stack that makes up the
  > Internet Protocol Suite. It is concerned with the physical and logical
  > connections between [hosts](./Network_hosts.md) on the same local network.

  The physical and logical connections are known as **Links**.

  Link Layer protocols goven communication between devices. Examples include:

  - **WiFi**
    - For wireless communication between devices using radio waves
  - **Ethernet**
    - Wired communication

  All devices connected to the internet participate in the Link Layer since it is
  the base level and the gateway out of the local network to other networks. The
  same is not true for other layers. For example a device like a laptop
  participates in all four layers up to the Application Layer but a network router
  does not:

  ![Link Layer diagram](../img/link-layer-frame-revised.png)

  ## MAC addresses

  Each device on a link has a network address that uniquely identifies it. For
  most Link Layer Protocols this is a
  [Media Access Control (MAC) address](MAC_addresses.md).

  A device receives data addressed to its MAC address.

  MAC addresses are limited to local networks. Remote networks cannot send data to
  a specific MAC address. This is off-limits from the outside.

  ## Frames

  Link Layer data is divided into small units called "frames". The anatomy of a
  frame is as follows:

  ![Link Layer frame](../img/link-layer-frame-revised.png)

  The header contains the source and destination MAC address plus a descriptor of
  the type of data it contains. The data is the payload. The footer is used to
  detect errors.

raw_content:
  ---
  tags: [networks, internet]
  created: Tuesday, August 06, 2024
  ---

  # Link Layer of the Internet Protocol Suite

  > The Link Layer is the lowest level of the network stack that makes up the
  > Internet Protocol Suite. It is concerned with the physical and logical
  > connections between [hosts](./Network_hosts.md) on the same local network.

  The physical and logical connections are known as **Links**.

  Link Layer protocols goven communication between devices. Examples include:

  - **WiFi**
    - For wireless communication between devices using radio waves
  - **Ethernet**
    - Wired communication

  All devices connected to the internet participate in the Link Layer since it is
  the base level and the gateway out of the local network to other networks. The
  same is not true for other layers. For example a device like a laptop
  participates in all four layers up to the Application Layer but a network router
  does not:

  ![Link Layer diagram](../img/link-layer-frame-revised.png)

  ## MAC addresses

  Each device on a link has a network address that uniquely identifies it. For
  most Link Layer Protocols this is a
  [Media Access Control (MAC) address](MAC_addresses.md).

  A device receives data addressed to its MAC address.

  MAC addresses are limited to local networks. Remote networks cannot send data to
  a specific MAC address. This is off-limits from the outside.

  ## Frames

  Link Layer data is divided into small units called "frames". The anatomy of a
  frame is as follows:

  ![Link Layer frame](../img/link-layer-frame-revised.png)

  The header contains the source and destination MAC address plus a descriptor of
  the type of data it contains. The data is the payload. The footer is used to
  detect errors.


path: zk/MAC_addresses.md
sortable_path: zkMAC_addresses.md
title: MAC addresses
checksum: d385cd4964d14e1ef14c8f39104492844bd52abd4e3d9b6e0532bf89f721339d
created: 2024-08-06 05:48:26.021725255+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Tuesday, August 06, 2024","tags":["networks"]}
lead:
  A Media Access Control (MAC) address is a physical address in contrast to, say,
  an IP address which is logical.

body:
  A Media Access Control (MAC) address is a physical address in contrast to, say,
  an IP address which is logical.

  Each device on a [local network](./Link_Layer_of_Internet_Protocol.md) has a MAC
  as a unique identifier. It is a hardware-based, physical attribute of the
  device, typically permanently encoded onto a non-volatile memory chip attached
  to the ![network_card](Network_card.md).

  MAC addresses consist of 6 bytes (48-bits) represented as 12
  [hexadecimal_digits](Hexadecimal_number_system.md).

raw_content:
  ---
  tags: [networks]
  created: Tuesday, August 06, 2024
  ---

  # MAC addresses

  A Media Access Control (MAC) address is a physical address in contrast to, say,
  an IP address which is logical.

  Each device on a [local network](./Link_Layer_of_Internet_Protocol.md) has a MAC
  as a unique identifier. It is a hardware-based, physical attribute of the
  device, typically permanently encoded onto a non-volatile memory chip attached
  to the ![network_card](Network_card.md).

  MAC addresses consist of 6 bytes (48-bits) represented as 12
  [hexadecimal_digits](Hexadecimal_number_system.md).


path: zk/Network_card.md
sortable_path: zkNetwork_card.md
title: Network card
checksum: 76c1d9ecb72c7e71e23119bd1159d51e40a3a9da798a073ba9888eddeedad437
created: 2024-08-06 06:08:04.807273649+00:00
modified: 2024-10-18 18:18:10.66781386+00:00
metadata: {"created":"Tuesday, August 06, 2024","tags":["networks","hardware"]}
lead:
  The network card (technically know as the Network Interface Card (NIC)) or
  _network adapter_ is the hardware device that allows a computer to connect to a
  network.

body:
  The network card (technically know as the Network Interface Card (NIC)) or
  _network adapter_ is the hardware device that allows a computer to connect to a
  network.

  They are the physical interface between the computer and the network
  [link](./Link_Layer_of_Internet_Protocol.md), e.g. WiFi or ethernet.

  They can be internal and built into the [motherboard](./Motherboard.md) or
  external as in the case of USB adaptors.

  Each network card has a unique [MAC address](./MAC_addresses.md) to identify the
  device on the network.

  Network cards on consumer-grade devices typically utilise the devices resources
  and the kernel manages its actions via a driver. More high-end devices
  (typically used in servers) have their own processors and memory.

raw_content:
  ---
  tags: [networks, hardware]
  created: Tuesday, August 06, 2024
  ---

  # Network card

  The network card (technically know as the Network Interface Card (NIC)) or
  _network adapter_ is the hardware device that allows a computer to connect to a
  network.

  They are the physical interface between the computer and the network
  [link](./Link_Layer_of_Internet_Protocol.md), e.g. WiFi or ethernet.

  They can be internal and built into the [motherboard](./Motherboard.md) or
  external as in the case of USB adaptors.

  Each network card has a unique [MAC address](./MAC_addresses.md) to identify the
  device on the network.

  Network cards on consumer-grade devices typically utilise the devices resources
  and the kernel manages its actions via a driver. More high-end devices
  (typically used in servers) have their own processors and memory.


path: zk/Peer_to_peer_network.md
sortable_path: zkPeer_to_peer_network.md
title: Peer to peer network
lead:
  A decentralised network model where each participant (peer) acts as both a
  client and a server.

checksum: bb1364bf5d9ed4a089d9697585e4832a7907e2142a23af856133de69e365e9e9
created: 2024-08-09 14:05:44.108407682+00:00
modified: 2024-10-18 17:55:56.292394043+00:00
metadata: {"created":"Friday, August 09, 2024","tags":["networks"]}
body:
  A decentralised network model where each participant (peer) acts as both a
  client and a server.

  Resources are shared directly between peers rather than being coordinated via a
  server.

  Under a server architecture, multiple requests are made from different hosts for
  resources. The server manages this load and is in control of what is being
  shared, to whom, and when. With P2P there is no central authority equivalent to
  this. Each peer both shares and consumes resources and in this sense is both a
  client and a server.

  A practical example of this is [torrenting](Torrenting.md) - an applicaton of
  P2P technology to file sharing.

  ## Benefits

  - Decentralisation, no central authority, also means no single point of failure.
  - Scalable: the network capacity grows with the number of users (contrast
    servers)
  - Efficiency: idle resources of peers are put to use
  - Improved performance for popular content

  ## Drawbacks

  - Security: potential for malicious peers and content
  - Inconsistent availability of resources

raw_content:
  ---
  tags: [networks]
  created: Friday, August 09, 2024
  ---

  # Peer to peer network

  A decentralised network model where each participant (peer) acts as both a
  client and a server.

  Resources are shared directly between peers rather than being coordinated via a
  server.

  Under a server architecture, multiple requests are made from different hosts for
  resources. The server manages this load and is in control of what is being
  shared, to whom, and when. With P2P there is no central authority equivalent to
  this. Each peer both shares and consumes resources and in this sense is both a
  client and a server.

  A practical example of this is [torrenting](Torrenting.md) - an applicaton of
  P2P technology to file sharing.

  ## Benefits

  - Decentralisation, no central authority, also means no single point of failure.
  - Scalable: the network capacity grows with the number of users (contrast
    servers)
  - Efficiency: idle resources of peers are put to use
  - Improved performance for popular content

  ## Drawbacks

  - Security: potential for malicious peers and content
  - Inconsistent availability of resources


path: zk/Internet_Layer_of_Internet_Protocol.md
sortable_path: zkInternet_Layer_of_Internet_Protocol.md
title: Internet Layer of the Internet Protocol Suite
checksum: fd97eae4f1441c51acff3d52fc5d636be1e7c59d68b435c45a6495faca4fc21e
created: 2024-08-15 05:53:06.099345649+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"created":"Thursday, August 15, 2024","tags":["internet","networks"]}
lead:
  The Internet Layer is concerned with establishing the basis for the connection
  and communication between hosts accross the internet.

body:
  The Internet Layer is concerned with establishing the basis for the connection
  and communication between hosts accross the internet.

  In order for the data to get from a machine on one local network to a machine on
  another local network, each machine must have a way of identifying and
  interfacing with the other.

  The Internet Layer utilises the **Internet Protocol** to determine how devices
  will be publicly identified to devices on other networks and how different
  devices on the same local network will be distinguished from outside of this
  network. This is achieved through
  [Internet Protocol (IP) addresses](./IP_addresses.md).

  ## Packets

  Data is sent accross the internet between hosts as _packets_. These packets,
  when sent and received within the
  [Link Layer](Link_Layer_of_Internet_Protocol.md) on the local network, are
  enclosed within a Link Layer frame (in its payload section):

  ![Internet Layer packet](../img/internet-layer-packet-revised.png)

  Like a frame, an IP packet has a header and a payload. The payload comprises the
  data between sent between hosts. The header contains a source IP address and a
  destination IP address and also specifies which version of the Internet Protocol
  is being used.

  // Need to find out more about packets work.

raw_content:
  ---
  tags:
    - internet
    - networks
  created: Thursday, August 15, 2024
  ---

  # Internet Layer of the Internet Protocol Suite

  The Internet Layer is concerned with establishing the basis for the connection
  and communication between hosts accross the internet.

  In order for the data to get from a machine on one local network to a machine on
  another local network, each machine must have a way of identifying and
  interfacing with the other.

  The Internet Layer utilises the **Internet Protocol** to determine how devices
  will be publicly identified to devices on other networks and how different
  devices on the same local network will be distinguished from outside of this
  network. This is achieved through
  [Internet Protocol (IP) addresses](./IP_addresses.md).

  ## Packets

  Data is sent accross the internet between hosts as _packets_. These packets,
  when sent and received within the
  [Link Layer](Link_Layer_of_Internet_Protocol.md) on the local network, are
  enclosed within a Link Layer frame (in its payload section):

  ![Internet Layer packet](../img/internet-layer-packet-revised.png)

  Like a frame, an IP packet has a header and a payload. The payload comprises the
  data between sent between hosts. The header contains a source IP address and a
  destination IP address and also specifies which version of the Internet Protocol
  is being used.

  // Need to find out more about packets work.


path: zk/IP_addresses.md
sortable_path: zkIP_addresses.md
title: IP addresses
lead: ## IPv4 vs IPv6
checksum: 0f01198098c3d02c6940f70369dd6d8e152f046585adc2f39f3fc788135e834a
created: 2024-08-16 14:21:22.886184519+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"created":"Friday, August 16, 2024","tags":["networks","internet"]}
body:
  ## IPv4 vs IPv6

  IP addresses come in two formats and bit-lengths depending on which version of
  the [Internet Protocol](Internet_Layer_of_Internet_Protocol.md) is being used.

  - IPv4 addresses are 32-bits in length
  - IPv6 addresses are 128-bits in length

  The larger the bit-length, the more unique addresses that can be generated. Thus
  IPv6 can create more addresses that IPv4.

  This entry focuses on IPv4 addresses.

  ## Anatomy of an IPv4 address

  Each of the 32 bits are divided into four groups which are each 8 bits in
  length. Each group therefore comprises an octet and is separated by dots.

  The decimal number system, rather than binary is used to represent the values.
  This means each binary octet is three decimal digits in length.

  The 32 bits comprise two major groupings

  - the **network prefix**
  - the **host**

  ![Different sections of IP address](../img/ip-address-anatomy.png)

  Devices connected to the same network share the same network prefix. They are
  said to be on the same **subnet**. However each device will have a unique value
  for the host octet.

  (This account is somewhat idealised. The length of the prefix does not always
  align with the octet boundary. For instance, the prefix could be 25 bits in
  length, rather than 24. In this case it would "steal" one bit from the host
  section making it 23 bits in length.)

  Any changes to the length of the network prefix change the number of hosts
  available on the subnet. If the network prefix is longer, the number of unique
  hosts is reduced. If the network prefix is shorted, the number of unique hosts
  is increased.

  ## Identifying the network and host groupings

  Given that the network prefix and host identifier do not always stick to set
  boundaries, we require a way of distinguishing the two values. There are two
  dominant methods.

  ### CIDR notation

  This stands for _Classless Inter-Domain Routing_. An example:

  > 192.168.1.23/24

  The value after the forward slash is the number of bits that specify the network
  address. In the example, it is the standard 24-bits. Thus 192.168.1.23 stands
  for the network.

  ### Subnet masking

  The subnet mask is another number _in addition to_ the network address. It is a
  kind of superimposed map on top of the address.

  Here is an example of a subnet mask:

  > 11111111.11111111.11111111.00000000

  The denary form would be as follows (as 255 is the maximal decimal number that
  can be represented with a single 8-bit number and 0 is the smallest):

  > 255.255.255.0

  In the binary form, the 1 values represent the bits which designate the network
  address and the 0 values represent the bits that designate the host.

  In the example above this corresponds to the idealised 32-bit:8-bit ratio of
  CIDR /24.

  There is a clever consequence of the subnet mask: if you apply a bitwise AND
  operator against the IP address and mask (both in their binary form) you can
  determine whether two addresses are on the same network.

  To compare the IP address 192.168.1.23 against 192.168.1.100 to demonstrate:

  ```
  192.168.1.23:
  IP:     11000000.10101000.00000001.00010111
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000001.00000000
  ___________________________________

  192.168.1.100:
  IP:     11000000.10101000.00000001.00010111
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000001.00000000
  ```

  After applying the bitwise AND logic we see that the result is identical for
  both IPs indicating they are on the same network (share the same network
  prefix), whilst the host value is "masked".

  Here is a scenario where the subnet mask indicates that two IPs are not on the
  shared network:

  ```
  192.168.1.23:
  IP:     11000000.10101000.00000001.00010111
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000001.00000000
  ___________________________________

  192.168.2.1
  IP:     11000000.10101000.00000010.00000001
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000010.00000000

  ```

  255.255.255.0 is not the only possible subnet mask. There are masks
  corresponding to ratios other than CIDR /24. For example:

  - 255.0.0.0 (/8)
  - 255.255.0.0 (/8)
  - 255.255.255.192 (/26)
  - 255.255.255.240 (/28)

  As noted previously, the different ratios will obviously affect the number of
  unique hosts available on the subnet.

  ## Determining the range of hosts: practical example

  Let's say we have the following IP address expressed in CIDR: 192.168.0.133/27.
  In binary this would be:

  ```
  110000000.10101000.00000000.100--00101
  ```

  The `--` indicates the demarcation point between the network prefix and the bits
  designated for the host. 32 - 27 leaves us 5 bits for our range of hosts. This
  gives us 2^5 = 32 unique host values.

  In fact it will actually be 30 values since the first value (`00000`) will be
  used to identify the network itself and the last value (`11111`) will be the
  broadcast address - the address used when a message needs to be sent to all
  hosts on the network.

  An actual host address in this range is included in the example: `00101`.

raw_content:
  ---
  tags: [networks, internet]
  created: Friday, August 16, 2024
  ---

  # IP addresses

  ## IPv4 vs IPv6

  IP addresses come in two formats and bit-lengths depending on which version of
  the [Internet Protocol](Internet_Layer_of_Internet_Protocol.md) is being used.

  - IPv4 addresses are 32-bits in length
  - IPv6 addresses are 128-bits in length

  The larger the bit-length, the more unique addresses that can be generated. Thus
  IPv6 can create more addresses that IPv4.

  This entry focuses on IPv4 addresses.

  ## Anatomy of an IPv4 address

  Each of the 32 bits are divided into four groups which are each 8 bits in
  length. Each group therefore comprises an octet and is separated by dots.

  The decimal number system, rather than binary is used to represent the values.
  This means each binary octet is three decimal digits in length.

  The 32 bits comprise two major groupings

  - the **network prefix**
  - the **host**

  ![Different sections of IP address](../img/ip-address-anatomy.png)

  Devices connected to the same network share the same network prefix. They are
  said to be on the same **subnet**. However each device will have a unique value
  for the host octet.

  (This account is somewhat idealised. The length of the prefix does not always
  align with the octet boundary. For instance, the prefix could be 25 bits in
  length, rather than 24. In this case it would "steal" one bit from the host
  section making it 23 bits in length.)

  Any changes to the length of the network prefix change the number of hosts
  available on the subnet. If the network prefix is longer, the number of unique
  hosts is reduced. If the network prefix is shorted, the number of unique hosts
  is increased.

  ## Identifying the network and host groupings

  Given that the network prefix and host identifier do not always stick to set
  boundaries, we require a way of distinguishing the two values. There are two
  dominant methods.

  ### CIDR notation

  This stands for _Classless Inter-Domain Routing_. An example:

  > 192.168.1.23/24

  The value after the forward slash is the number of bits that specify the network
  address. In the example, it is the standard 24-bits. Thus 192.168.1.23 stands
  for the network.

  ### Subnet masking

  The subnet mask is another number _in addition to_ the network address. It is a
  kind of superimposed map on top of the address.

  Here is an example of a subnet mask:

  > 11111111.11111111.11111111.00000000

  The denary form would be as follows (as 255 is the maximal decimal number that
  can be represented with a single 8-bit number and 0 is the smallest):

  > 255.255.255.0

  In the binary form, the 1 values represent the bits which designate the network
  address and the 0 values represent the bits that designate the host.

  In the example above this corresponds to the idealised 32-bit:8-bit ratio of
  CIDR /24.

  There is a clever consequence of the subnet mask: if you apply a bitwise AND
  operator against the IP address and mask (both in their binary form) you can
  determine whether two addresses are on the same network.

  To compare the IP address 192.168.1.23 against 192.168.1.100 to demonstrate:

  ```
  192.168.1.23:
  IP:     11000000.10101000.00000001.00010111
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000001.00000000
  ___________________________________

  192.168.1.100:
  IP:     11000000.10101000.00000001.00010111
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000001.00000000
  ```

  After applying the bitwise AND logic we see that the result is identical for
  both IPs indicating they are on the same network (share the same network
  prefix), whilst the host value is "masked".

  Here is a scenario where the subnet mask indicates that two IPs are not on the
  shared network:

  ```
  192.168.1.23:
  IP:     11000000.10101000.00000001.00010111
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000001.00000000
  ___________________________________

  192.168.2.1
  IP:     11000000.10101000.00000010.00000001
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000010.00000000

  ```

  255.255.255.0 is not the only possible subnet mask. There are masks
  corresponding to ratios other than CIDR /24. For example:

  - 255.0.0.0 (/8)
  - 255.255.0.0 (/8)
  - 255.255.255.192 (/26)
  - 255.255.255.240 (/28)

  As noted previously, the different ratios will obviously affect the number of
  unique hosts available on the subnet.

  ## Determining the range of hosts: practical example

  Let's say we have the following IP address expressed in CIDR: 192.168.0.133/27.
  In binary this would be:

  ```
  110000000.10101000.00000000.100--00101
  ```

  The `--` indicates the demarcation point between the network prefix and the bits
  designated for the host. 32 - 27 leaves us 5 bits for our range of hosts. This
  gives us 2^5 = 32 unique host values.

  In fact it will actually be 30 values since the first value (`00000`) will be
  used to identify the network itself and the last value (`11111`) will be the
  broadcast address - the address used when a message needs to be sent to all
  hosts on the network.

  An actual host address in this range is included in the example: `00101`.


path: zk/The_History_of_Computing_Swade.md
sortable_path: zkThe_History_of_Computing_Swade.md
title: History of Computing (Swade, 2022 )
checksum: aa8215af979c47e01476b79b15378cf67294b028b02f92bb020ff379363c0b21
created: 2024-08-23 07:46:48.602264538+00:00
modified: 2024-10-19 08:59:26.629863039+00:00
metadata: {"created":"Friday, August 23, 2024","tags":["literature","computer-history"]}
lead:
  | Title                                               | Author      | Publication date | Resource type |
  | --------------------------------------------------- | ----------- | ---------------- | ------------- |
  | The History of Computing: A Very Short Introduction | Doron Swade | 2022             | Book          |

body:
  | Title                                               | Author      | Publication date | Resource type |
  | --------------------------------------------------- | ----------- | ---------------- | ------------- |
  | The History of Computing: A Very Short Introduction | Doron Swade | 2022             | Book          |

  ## Timeline

  A typical timeline approach rooted in major innovations.

  - Ancient aids to counting: knotted cords and notched sticks
  - Ancient aids to calculation: counting boards and abacii
  - Early mechanical calculator devices in the 17th century (number wheels,
    Pascal, Leibniz)
  - Modern aids to calculation: slide rules following the discovery of
    [logarithms](Logarithms.md)
  - Mechanised, automated calculating engines of Babbage in the 19th century
  - Punched-card machines leading to IBM in the early 20th century
  - Analogue and electro-mechanical computers of the early 20th century inclusive
    of wartime computers
  - Early valve-based (vacuum-tubed) digital computers (again wartime and early
    Cold War)
  - The invention of the transistor and first fully-digital computers
  - The invention of [integrated_circuits](Integrated_circuits.md)
  - Supercomputers
  - Minicomputers
  - Consumer personal computers
  - Internet and later, Web
  - Smart phones

  First three phases of digital electronic computers:

  - Wartime up to 1950s vacuum-tube era
  - Transistor era up to 1963
  - First microchip era ending in early 1970s

  ## Terms

  | Term            | Definition                                                                                                                                                                  |
  | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  | automatic       | can execute programs, process data and perform functions on its own once it has been started. Can monitor its own processes and make adjustments without human intervention |
  | general-purpose | a computer designed to be able to solve a wide variety of problems and perform many different types of tasks as opposed to a computer built for a specific, limited purpose |
  | programmable    | the ability to change the operations a computer peforms without physically altering its hardware                                                                            |

  ## Mechanical calculating devices in the 17th century

  Focus was chiefly on creating a desktop calculator capable of four-function
  arithmetic.

  ![Photograph of Pascaline](../img/pascaline.jpg)

  ![Photograph of replica of Leibniz stepped drum machine](../img/leibniz-stepped-drum.jpg)

  The main contenders were the Pascaline of #Pascal (which only did cumulative
  addition) and the wheel or "stepped drum" calculator of #Leibniz that could do
  all operations (in theory).

  Subsequent designs were based on these artefacts. In practice, neither worked
  consistently well with the carriage of tens remaining a sticking point.

  ![Photograph of Arithmometer](../img/arithmometer.jpg)

  ![Photograph of Comptometer](../img/comptometer.png)

  The arithmometer (crank driven) and comptometer (key-driven) were descendents of
  the Leibniz design that became commercially viable by the 19th century along
  with other mechanical calculators. In the US, Burroughs dominated the market.

  ## Babbage: mechanized, automated calculation

  > I wish to God these calculations had been executed by Steam (#Babbage)

  With Babbage's machines we see an approach to computation that can only be
  understood against the backdrop of the Industrial Revolution in which they were
  conceived.

  The idea is that the machine is a factory and number is the product. In the same
  way as the mechanised looms created textiles. It is the extension of a model of
  industrial production from goods/commodities to information.

  Babbage conceived two machines: the Difference Engine (DE) and the Analytical
  Engine (AE). Neither were successfully built in his lifetime. The DE preceded
  the AE and was basically an advanced mechanical calculator whereas the AE
  approximated a general purpose computer.

  Despite this, with the Difference Engine, in contrast to preceding _aids to
  calculation_, the steps of the computational algorithm were no longer directed
  by human intelligence but by internal rules embodied in the mechanism and
  automatically generated.

  ### Difference Engine

  The DE's single purpose was to calculate and output mathematical tables such as
  the results of polynomial equations. The idea was that you would input the
  answer a bell would ring and the machine would _halt_. This influenced #Turing
  later. It was non-programmable and designed for a specific set of calculations.

  ### Analytical Engine

  Conceived as a general-purpose computing machine capable of perfoming a wide
  range of calculations, programmable using punched cards similar to those used
  with Jacquard looms.

  It more resembled modern computers in that Babbage used concepts that would
  later translate into the #vonNeumann architecture. There was a "mill" (CPU),
  "store" (memory) and input/output mechanisms. It also had a concept of looping
  and conditional branching.

  ### Lovelace's insight

  A central idea of Ada #Lovelace, expressed in her notes on the Analytical Engine
  is that **number can represent entities other than quantity**.

  If we assign meaning to number, results arrived at by operating on number
  according to rules can say things about the world when mapped back onto the
  world using the meanings assigned to them.

  Lovelace's insight was that the potential of computing lay in the power of
  machines to manipulate representations of the world contained in symbols.

  ## Analogue computers

  Both digital and analogue computers are automatic. They differ in _how they
  represent quantities_ and how their outputs are derived.

  With digital machines, quantity is represented as a string of discrete digits.

  With analogue machines, quantity is a physical property _in itself_ rather than
  a representation. This could be, for example: the lowering of a weight, the flow
  of a liquid, or an electrical charge. This physical behaviour is **analagous**
  to the system that is being modelled. Quantities are continuously variable
  values rather than discrete (discontinuous values).

  Digital machines produce results by _calculation_ whereas analogue machines
  produce results by _measurement_, e.g. the height of liquid in a tank or the
  time it taks for a tank to be emptied.

  ### Examples of analogue computers

  #### The Phillips Hydraulic Computer

  This used fluid to model the workings of the British economy. It consisted of a
  series of transparent plastic tanks and pipes which were fastened to a wooden
  board.

  ![Philips Hydraulic Computer](../img/hydraulic_computer.jpg)

  Each tank represented some aspect of the UK national economy and the flow of
  money around the economy was illustrated by coloured water. At the top of the
  board was a large tank called the treasury. Water (representing money) flowed
  from the treasury to other tanks representing the various ways in which a
  country could spend its money. For example, there were tanks for health and
  education. To increase spending on health care a tap could be opened to drain
  water from the treasury to the tank which represented health spending.

  #### Bush's Differential Analyser

  This was designed to solve differential equations by integration. In contrast to
  the Philips Computer it was general enough to be used to solve problems from
  different contexts. Examples of these contexts: heat flow, ballistics,
  mechanics, population growth, chemical interactions, astronomy.

  ![Differential Analyser](../img/differential_analyser_bush.jpg)

  It was about the size of a room and used shafts, motors, discs and wheels to
  work.

  ### Historiography

  There is a tendency in the history of computing to downplay or diminish the
  contribution of analogue computing devices and to present them as just an
  inferior precursor to the inevitable dominance of digital electronic computers.

  This is ahistorical and inaccurate.

  Analogue (and electromechanical devices) overlapped with and coexisted with
  digital devices for 40 years, spanning the first three generations of digital
  electronic devices. The term "analogue" itself only came about when the need
  arose to distinguish digital devices from other types of computer. They were not
  "rivals" before this.

  ### Electro-mechanical computers

  Electro-mechanical devices (also known as "electronic analogue computers") are a
  sort of midway between full digital devices and analogue computers, forming a
  bridge between the two eras.

  Their heyday was roughly 1935 - 1945.

  Their key components were:

  - [relays](Relays.md) for logic operations
  - rotating shafts and gears for performing calculations
  - punched cards or paper tape for input instructions and outputs

  They were slower than fully digital computers as they were limited by the speed
  of moving components rather than the flow of electric charge. In addition the
  various mechanical parts were prone to wear and needed frequent replacement.

  ### Examples of electro-mechanical computers

  #### Harvard Mark 1 (1937-1944) - Aikin, IBM

  Designed by Howard Aiken and built by IBM (1937-1944). Also known as the
  _Automatic Sequence Controlled Calculator_ (ASCC). A general-purpose
  electro-mechanical computer it was most famously used at Los Alamos by
  #vonNeumann to calculate the blast yield of the atomic bomb.

  ![Harvard Mark One](../img/Harvard_Mark_1.jpg)

  It was more than 15m in length and weighed 5 tonnes comprising over 750,000
  parts. It used paper tape and punched cards for input/output.

  #### Complex Number Calculator (1940) - Bell Labs, Stibitz

  Not general purpose nor was it programmable. It was hardwired to perform a
  specific set of operations on numbers and nothing else. It used relays like the
  others. Its distinguishing feature was that it used a teletype for input rather
  than cards or paper tape.

  ![Calculating panel of the CMC](../img/complex-number-calculator-bell-labs.jpg)

  ![Teletype input of CMC](../img/stibitz-teletype.jpg)

  It comprised a panel (the calculating unit) and teletype (the input). One could
  remotely access the computer from the teletype in another location, providing it
  was connected to Bell Lab's telephone network.

  #### Z3 (1941) - Konrad Zuse

  ![Z3](../img/Zuse-Z3.jpeg)

  First designed in 1938 and completed in 1941. Considered the first fully
  automatic, programmable digital computer although relay-based. Programs were
  stored on external punched tape and binary was used as the number system.
  Destroyed by Allied bombing in the War but a replica (pictured) was built
  in 1998.

  ## Vacuum-tube devices

  Vacuum-tube devices were the first fully electronic digital devices as they used
  vacuum-tubes for implementing binary logic rather than relay switches.

  Canonical examples from this era are the ABC computer, the Collossus and most
  famously, the ENIAC.

  ### The Antanasoff-Berry Computer (1939 - 1942)

  ![ABC Computer](../img/abc-computer.jpg)

  Fully automatic, digital electronic computer but not programmable or really
  general-purpose, being created to do linear equations. It took decimal input and
  output but operated internally using binary arithmetic. The input was punched
  cards and the output was via a front-panel display.

  Its importance stems from the fact that it was the first to use vacuum-tubes for
  the arithmetic calculations of the processor.

  ### Colossus (1943-1945) - Tommy Flowers/Max Newman

  Designed and built at the Post Office Research Station at Dollis Hill. Used for
  cracking the Lorenz messages of the German high command.

  Mainly important (beyond its role in the War) for establishing the viability of
  vacuum-tubes over relays. It comprised 2.4 thousand tubes.

  Debatable the extent to which the Collosus can be called the _first digital
  computer_ since it was not general purpose and not fully programmable.

  Furthermore it was not even a complete solution in its context of application.
  The codebreakers would be given a certain advantage using the Collosus
  (narrowing the field of possible deciphering keys) but still need to use manual
  methods to break the code. It made decipher time shorter rather than executing
  the full diciphering. In contrast to, say, the Bombe computer that would
  identify the key for the days transmissions.

  It's actual influence is also hard to gauge because of the secrecy that remained
  in place after the War. Apart from those who worked on it, its technical
  advancements were not public knowledge until the late 1970s.

  ### ENIAC (1945) John Mauchly/J.P Eckert

  The _Electronic Numerical Calculator and Integrator_

  The main motivation for creating the ENIAC was military: the US Army needed
  speed and accuracy when calculating artiliary firing tables and munition
  trajectory so that shells could hit their intended targets. Accuracy depended on
  myriad factors: gun elevation, shell shape and weight, explosive charge,
  distance, wind, temperature etc. Previously this had been done by human
  computers under the aegis of the Ballistic Research Laboratory.

  ![The main control panel of the ENIAC](../img/eniac.jpg)

  It was not completed until after VE day but was used at Los Alamos after the War
  and retired in 1955.

  The BRL commissioned the Moore School of Engineering at the University of
  Pennsylvania for the construction of an automatic electronic calculator.

  It used vacuum-tubes, along with switches and realys. It used decimal rather
  than binary and had punch card input and output. It was crudely programmable and
  could be set up to work on different problems by rearranging the different
  components using switches and plugboards. But this took roughly two days.

  It's design was very influential because it was made public and not classified.

  ### EDVAC (1944-49) John Mauchly/J.P Eckert/ John von Neumann

  Built at the Moore School for the BRL as a successor to ENIAC by Mauchly and
  Eckert whilst ENIAC was still operational. It's architecture arose out of
  discussions between #vonNeumann and Mauchly and Eckert and was necessitated by
  the need to calculate the behaviour of the fissile materials in the atomic bomb.

  It resulted in the famous _First Draft of a Report on the EDVAC_ by #vonNeumann
  which is seen as the definitive statement of the modern **digital programmable
  general-purpose computer**.

  As a result of the paper, #vonNeumann effectively got the credit for the
  invention even though it synthesised ideas from the ENIAC and the contributions
  of others to the EDVAC.

  There was lots of disputed provenance and claims of plagiarism generally.
  Antanasoff won a legal claim against Eckert and Mauchly for stealing ideas from
  the ABC to make ENIAC.

  // Add more on the hardware.

  ### Concept of the _internal stored program_

  - Key innovation of the EDVAC was the _internal stored program_ (ISP).

  - Solution to the problem of preceding computers requiring down-time when
    switching between applications and problem sets.

  - Circuitry would need to be reconfigured before the computer could run on the
    next problem

  - ISP removed the friction:

    - the instructions comprising the program would be prepared on tape or punched
      cards and read into electronic memory
    - the hardware configuration of the machine would remain the same accross
      different programs and require no reconfiguration, only the input program
      data would change

  - This was basically the invention of software. Both instructions and data
    shared the same memory space once they were read-in. Before, the data would be
    stored in memory but the instructions would be read one-by-one from the
    external storage media.

  - Key consequences:
    - flexibility (no need for re-wiring)
    - speed: fetching instructions from memory is quicker than fetching from
      external storage devicde
    - self-modification: since they were in memory, programs could modify
      themselves during their execution (e.g. use branching conditions) and
      treating one program as data for another. This enabled more complex
      algorithms and also opened the possibility of higher-level languages and
      compilers
    - simplified architecture: both programs and data in memory
    - generalisation: computers could switch between different tasks simply by
      loading different programs

  ### Post-EDVAC devices

  There were several notable devices which attempted to implement the architecture
  described in the _First Draft_ using vacuum tubes for logic operations and a
  variety of different electronic methods for memory:

  - the Manchester "Baby"
  - the Cambridge EDSAC
  - Turing's ACE computer

  #### Manchester Baby (1948)

  ![The Manchester Baby Computer](../img/manchester_baby.png)

  - An experimental computer intended to create the
    [von Neumann architecture](CPU_architecture.md) using
    [Williams_Tube_RAM](Williams_Tube_RAM.md)

  - Developed at the Univesity of Manchester and completed in 1948.

  - Considered the first electronic stored-program computer and first to contain
    all the elements of a modern electronic digital computer.

  #### EDSAC (1949) J.Wilkes et al.

  ![EDSAC](../img/EDSAC.jpg)

  - _Electronic Delay Storge Automatic Computer_

  - Constructed by Maurice Wilkes and others at the Mathematical Laboratory of the
    University of Cambridge.

  - The second digital stored-program computer after the Manchester Baby.

  - It used vacuum-tubes for the arithmetical operations in the ALU and mercury
    [delay line memory](Delay_line_memory.md) for the RAM.

  - Designed to be used by relatively non-specialist practitioners from other
    university departments who were expected to program it themselves. To this
    end, a formal programming paradigm was forged for the EDSAC which established
    the following:

    - subroutines as a library of common procedures available to programmers (e.g.
      printing a result, reading input tape, program checking, mathematical
      operations)
    - diagnostics: techniques for verifying program code and its correctness

  ![Programming Digital Electronic Computer book](../img/programming_a_digital_electronic_computter.jpg)

  - The key players wrote the first textbook on programming in 1951: _The
    Preparation of Programs for an Electronic Digital Computer_ (Wilkes, Wheeler,
    and Gill)

  #### The Automatic Computing Engine (ACE) (1946) Alan Turing

  - Technical design provided by Alan #Turing working at the Mathematics Division
    of the National Physical Laboratory. The product of Turing's theoretical work
    in "On Computable Numbers" where he proposes the concept of a
    [Turing_machine](Turing_machines.md) and based on his experience with early
    single-purpose computing devices at Bletchley.

  - Due to secrecy over wartime work, it was hard for get Turing to build a fully
    electronic implementation since this would disclose advances made during the
    War. Tommy Flowers was intended to be brought in to build it but again this
    was scuppered over the classification of wartime technology. Had this not been
    the case, it is likely it would've been seen as on par with the EDVAC in its
    design, if not dwarfing it.

  - Turing proposed a stored program architecture with high-speed memory. It would
    be more perfomant than the EDVAC as a result.

  ![The Pilot ACE computer](../img/pilot-ace.jpg)

  - Turing's actual design was not implemented as it was thought too ambitious
    given how powerful it would need to be. Instead a smaller prototype was made,
    the Pilot ACE. It used vacuum-tubes for logic, and mercury delay lines for
    memory. Data and memory was input via punched cards.

  ### Further applications of the EDSAC architecture

  ### LEO I

  - Built by Lyons Tea Company to manage business affairs (payroll, inventory,
    stock management)

  - Marked a shift from military and academic contexts for computers to business
    and data management.

  ### IAS machines

  ![The MANIAC computer](../img/MANIAC_computer.jpg)

  Several machines were built at the Institute for Advanced Study utilising the
  "von Neumann" architecture and associated advancements such as vacuum-tubes and
  Williams' tubes. There was the original IAS machine (1952) as well as the
  JOHNNIAC (1954) and MANIAC (1956).

  Although their purpose was military (Los Alamos), their designs were public and
  widely studied making them influential outside of academia.

  ### UNIVAC (1951)

  ![UNIVAC_computer](../img/univac-1.jpg)

  Mauchley and Eckert, who had designed the ENIAC left the Moore School and went
  into business: Eckert-Mauchley Computer Corporation. This was bought by
  Remington Rand and in subsequent years became the main competitor to IBM.

  In this capacity they built the UNIVAC: _Universal Automatic Computer_. It's
  name being an embodiment of its nature as a general-purpose, electronic digital
  computer.

  It used vacuum-tubes for logic and mercury delay lines for memory. It had
  multiple means of input/output including: directly via an operator console
  (basically a typewriter keyboard), magnetic tape for input and output, along
  with punched cards.

  ![UNIVAC operator console](../img/univac-operator-console.jpeg)

  It was the first computer specifically designed to include business and
  administrative use. This was underscored by its first client: the US Census
  Bureau.

  A key event was its succesful prediction of the 1952 general election. It
  correctly predicted a landslide for Eisenhower (against expectations). (This was
  so unlikely, they actually fudged the data because they thought the machine was
  way off.) It was a novelty on the results night but it cemented a certain
  concept of the computer in the public imagination - large, room sized machines
  with blinking lights.

  Swade notes that Eckert and Mauchley effectively launched the US commercial
  computer industry with the UNIVAC.

  ## IBM

  - First created in 1911 as a merger of four companies all involved in some
    capacity in business data processing. The most significant of these was the
    Hollerith Tabulating Machine Company who's punched card machines had been used
    in the 1890 census, greatly reducing its collation time

  - Named IBM in 1924

  - UNIVAC represented the encroachment of computing devices into IBM's market of
    punched-card based data-processing and effectively put IBM on notice.

  - To attack the threat IBM brought out three crucial machines: the 701, 702
    and 650.

  - IBM's approach to marketing their computers was to emphasise the comfort of
    continuity over the shock of the new represented by the UNIVAC. IBM had
    already captured the business market. Their computers would be more of the
    same, just faster and more powerful than their punched card machines due to
    the electronics. To this end they initially did not call their devices
    'computers'.

  - IBM were ultimately totally successful in their pivot to the computing market.
    By 1960 it owned over 70% of the data processing and computer market.

  ### IBM 701 ("Defense Calculator") (1952)

  ![Control unit of the IBM 701](../img/ibm-701-control-unit.jpeg)

  IBM's first electronic computer. It directly competed with the UNIVAC for
  government contracts. It followed the prevailing approach of vaccuum tubes for
  logic and had a variety of storage methods: Williams tubes,
  [magnetic tape](Magnetic_tape.md) , and magentic drums.

  701s were used at Los Alamos and avionics companies. Some were also used to
  manage payroll in business contexts.

  ### IBM 702 ("Tape Processing Machine") (1953)

  ![Advertisement for the IBM 702](../img/ibm-add.jpeg)

  Focused primarily on business applications and targetted at businesses rather
  than government contracts. Less powerful than the 701.

  ### IBM 650 (1953)

  ![IBM 650](../img/ibm-650.jpg)

  Low-cost general purpose machine using magnetic drum memory. Marketed as slower
  but more affordable than the 700 range. It was mass-produced unlike the others
  which were built for specific customers. It was in fact the first mass-produced
  computer in the world. It proved the breakout star in IBMs initial line up and
  sometimes called "IBM's Model T".

  ![IBM_650 console panel](../img/ibm-650-console-panel.jpg)

  IBM offerred 650s to univesities at a 60% discount on the condition that the
  universities would establish courses in computing. This was shrewd as it meant
  that a whole generation of engineers and computer scientists learned on 650s and
  would prefer them in their research. It also created a pool of skilled users
  that would go on to work at IBM.

  Donald Knuth dedicated _The Art of Computer Programming_ to the 650.

  It used [magnetic drum memory](./Magnetic_drum_memory.md) along with
  vacuum-tubes.

  ### IBM 1401 (1959)

  ![IBM 1401](../img/ibm-1401.jpg)

  Another IBM mainframe. Transistors replaced vacuum-tubes. Magnetic core storage
  replaced magnetic drum storage. It was housed in rectangular light-blue cabinets
  and the ubiquity of the 1401 in industry earned IBM the moniker 'Big Blue'.

  ### IBM System/360 (1964)

  ![IBM 360](../img/ibm-360.jpg)

  Named to suggest all-round compatibility - a family of mainframes designed to
  cover commercial and scientific applications. Considered one of history's most
  successful computers.

  Up until this point all IBM computers had a programming language unique to the
  specific processor. At that point there were about seven IBM computer models in
  active use and they were all incompatible with each other.

  ![IBM 360 promo poster](../img/ibm-360-in-use.jpeg)

  In contrast the 360 computers all used the same programming language. This meant
  they were interoperable with each others. Because the there were variants in the
  360 ranges, customers could purchase a smaller system knowing they could expand
  it if their needs grew, without reprogramming their application software.

  ## Magnetic core devices: Whirlwind and SAGE

  Two devices that leveraged the new technology of
  [magnetic_core_memory](Magnetic_core_memory.md) where the Whirlwind and SAGE
  computers. The ENIAC was also updated to use magnetic cores.

  ![Whirlwind magnetic core memory](../img/whirlwind-magnetic-cores.JPG)

  The Whirlwind computer (1953) was a flight simulator and the first to use
  magnetic cores. Crucially was able to operate in realtime for output.

  The SAGE computer (_Semi-Automatic Ground Environment_) emerged as a response to
  the Soviet acquisition of the atomic bomb in 1949. The Americans realised they
  needed a much better early warning and air defence system that would allow them
  to identify bombers in their airspace and dispatch fighters.

  To achieve this, it was necessary to detect enemy bombers with radar and compute
  an interception course. Alterations in the bomber flight path required
  continuous updates and real-time computation of directions for the responding
  fighter.

  The resulting computer was SAGE (made by IBM) which was modelled on Whirlwind,
  using magnetic cores. SAGE computers were spread accross the continental US in
  sectors and managed by NORAD.

  ![SAGE command post](../img/SAGE_command_post.jpg)

  The SAGE system was operation between 1958-1984, receiving continual
  improvements and updates. In addition to magnetic cores it introduced many
  technological innovations that influenced computers generally: interactive
  screens, printed circuit boards, mass-storage devices, digital communication
  over telephone lines, time-sharing, along with advances in software.

  Because so many private companies worked as contractors on SAGE (IBM, Burroughs,
  Bell Labs, RAND Corp), the military technology (both hardware and software) made
  its way from the military context to business and consumer computing.

  An example of a civilian application of SAGE technology was airline booking
  systems, where booking reservation data needed to be processed in realtime. IBM
  worked with American Airlines to introduce this.

  ## Transistor and integrated circuit revolution

  The arc of modern computing history (and its main eras) can be presented as

  - vacuum-tubes
  - [transistors](Transistors.md)
  - [integrated_circuits](Integrated_circuits.md)

  The transistor was invented at Bell Labs in 1947 by William Shockley and others.
  They were not created with computers in mind. Instead they were first used in
  hearing aids (1953) and transistor radios (1954).

  From the mid-1950s onwards they started being used in military computers.

  They had several advantages over vacuum-tubes: they were durable and reliable
  with no required startup time, they were power efficient and small.

  The actual period where transistors alone were supreme in the form of a
  transistor board was relatively short-lived and rapidly gave way to ICs.

  Computers of the transistor era had the following components separate from each
  other: the transistors, wired connections, resistors and capacitors. Integrated
  circuits put these all on the same piece of silicon.

  They were first manufactured by Fairchild Semiconductor which was an offshoot of
  Shockley's Shockley Semiconductor Laboratory.

  The initial marked for ICs was military and scientific: the Minuteman ICBM and
  computers of the Apollo spacecraft. This laid the groundworkd for later
  commercial use and the personal computer revolution.

  ## Mini-computers

  Mini computers did not threaten mainframes but they opened up a new class of
  user. They were vastly cheaper than a mainframe and only the size of a small
  fridge.

  The main player was DEC (Digital Equipment Corporation) starting in the
  mid-1960s. The introduced the PDP-8 in 1965 (Programmed Data Processor). The
  PDP-8 used transisotrs and magnetic core memory. It was affordable to smaller
  businesses if not yet, consumers.

  ![The PDP-8](<../img/540px-PDP-8_(1).jpg>)

  The internals were made public and DEC encouraged making the machine extensible
  by users being permitted to create their own programs and specialised
  applications. This was a very different culture to IBM where technicians were
  required and machines were typically leased.

  By 1988, DEC was the second largest computer company after IBM. This led to
  additional companies entering the mini-computer market: Data General, Honeywell,
  Hewlett-Packard.

  Mini-computers died as a category in the mid-1990s in the wake of the PC.

  ## The Personal Computer

  The PC emerged out of advances in IC chip design - this was the engine of the
  PC.

  Intel (newly founded by Robert Noyce and Gordon Moore) were charged with making
  custom chips for a Japanese calculator company. Instead of creating the custom
  chips, they advised the company to use a general purpose chip that could handle
  general computing as well as the basic calculator functions. This generic chip -
  the Intel 4004 - was the microprocessor combined with three other ICs
  (comprising 2.3k transistors) that comprised the fundamental building blocks of
  the #vonNeumann architecture.

  ![Intel 4004 chip](../img/intel-4004.jpg)

  However it took until the Apple II in 1977 for the phrase "personal computer" to
  enter the lexicon and become an identifiable class.

  The broader context for this was the computer hobbyists in Silicon Valley/Menlo
  Park (particularly the Homebrew Club of amateur computer engineers) along with
  idealistic figures from the 60s counterculture (such as Stuart Brand) who began
  associating computing technology as a tool of personal liberty and fulfillment.

  ### Microsoft and Apple

  The PC "boom" began with the Altair 8800. This was a mail-order kit that
  required manual assembly. It was released by Micro Instrumentation Telemetry
  Systems (MITS) in 1975. It couldn't do much but had expansion capability
  (memory, teletype interface, casette player for data storage).

  ![Altair 8800](../img/altair-8800.jpg)

  Bill Gates and Paul Allen made a proposal to MITS: they would write software
  that would allow users to program the Altair in BASIC. They agreed and were
  prepared to distribute the software. As a result Gates and Allen founded
  Microsoft in 1975.

  Around the same time, Steve Wozniak (member of the Homebrew club) built the
  Apple I as a single-board hobbyist project. He made this available to buy via
  mail order and formed Apple with Steve Jobs in 1976 to manage the enterprise.

  ![Apple I](../img/apple1.jpg)

  By 1977 they had investment capital and brought out the Apple II. This was sold
  preassembled with casing and required no soldering. It had expansion slots for
  third-party vendors to create compatible devices and had colour graphics.

  The design of the Apple II was published in _Byte_ in order to invite
  crowd-sourced plugins and extensions. One of these was the Z-80 softcard which
  allowed it to run the CP/M OS (created by the company Digital Research) giving
  ready-made access to software such as a word-processor, spreadsheets and
  databases.

  ![Apple 2](../img/apple2.jpg)

  Other competitors in the PC market at this time were Radioshack's TRS-80 and the
  Commodore PET.

  ### IBM

  All the while it was hobbyist and tinkerers driving the PC market, IBM had no
  real interest. When PCSs started being used by businesses and their capacity
  grew to include typical business applications (databases, spreadsheets) it
  sensed an incursion into its market.

  ![IBM PC](../img/ibm-pc.jpg)

  Thus, in 1981, IBM launched the "IBM Personal Computer" which rapidly became the
  industry standard. This had the effect of legitimising the concept of a PC.
  There was overwhelming demand. In their marketing IBM melded home and office use
  to capture both markets at once.

  The internal specification of the IBM PC was freely available (dubbed 'open
  architecture'), in contrast to IBM's business computers, to encourage software
  expansion. Cards could be created to fit in its expansion slots adding graphics,
  sound, additional memory and networking capability.

  IBM originally contracted Digital Research to create software based on CP/M but
  this fell through. Microsoft got the deal to supply the OS which became MS-DOS
  (Microsoft Disk Operating System). Henceforth every PC and PC-clone came with
  DOS bundled.

raw_content:
  ---
  tags: [literature, computer-history]
  created: Friday, August 23, 2024
  ---

  # History of Computing (Swade, 2022 )

  | Title                                               | Author      | Publication date | Resource type |
  | --------------------------------------------------- | ----------- | ---------------- | ------------- |
  | The History of Computing: A Very Short Introduction | Doron Swade | 2022             | Book          |

  ## Timeline

  A typical timeline approach rooted in major innovations.

  - Ancient aids to counting: knotted cords and notched sticks
  - Ancient aids to calculation: counting boards and abacii
  - Early mechanical calculator devices in the 17th century (number wheels,
    Pascal, Leibniz)
  - Modern aids to calculation: slide rules following the discovery of
    [logarithms](Logarithms.md)
  - Mechanised, automated calculating engines of Babbage in the 19th century
  - Punched-card machines leading to IBM in the early 20th century
  - Analogue and electro-mechanical computers of the early 20th century inclusive
    of wartime computers
  - Early valve-based (vacuum-tubed) digital computers (again wartime and early
    Cold War)
  - The invention of the transistor and first fully-digital computers
  - The invention of [integrated_circuits](Integrated_circuits.md)
  - Supercomputers
  - Minicomputers
  - Consumer personal computers
  - Internet and later, Web
  - Smart phones

  First three phases of digital electronic computers:

  - Wartime up to 1950s vacuum-tube era
  - Transistor era up to 1963
  - First microchip era ending in early 1970s

  ## Terms

  | Term            | Definition                                                                                                                                                                  |
  | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  | automatic       | can execute programs, process data and perform functions on its own once it has been started. Can monitor its own processes and make adjustments without human intervention |
  | general-purpose | a computer designed to be able to solve a wide variety of problems and perform many different types of tasks as opposed to a computer built for a specific, limited purpose |
  | programmable    | the ability to change the operations a computer peforms without physically altering its hardware                                                                            |

  ## Mechanical calculating devices in the 17th century

  Focus was chiefly on creating a desktop calculator capable of four-function
  arithmetic.

  ![Photograph of Pascaline](../img/pascaline.jpg)

  ![Photograph of replica of Leibniz stepped drum machine](../img/leibniz-stepped-drum.jpg)

  The main contenders were the Pascaline of #Pascal (which only did cumulative
  addition) and the wheel or "stepped drum" calculator of #Leibniz that could do
  all operations (in theory).

  Subsequent designs were based on these artefacts. In practice, neither worked
  consistently well with the carriage of tens remaining a sticking point.

  ![Photograph of Arithmometer](../img/arithmometer.jpg)

  ![Photograph of Comptometer](../img/comptometer.png)

  The arithmometer (crank driven) and comptometer (key-driven) were descendents of
  the Leibniz design that became commercially viable by the 19th century along
  with other mechanical calculators. In the US, Burroughs dominated the market.

  ## Babbage: mechanized, automated calculation

  > I wish to God these calculations had been executed by Steam (#Babbage)

  With Babbage's machines we see an approach to computation that can only be
  understood against the backdrop of the Industrial Revolution in which they were
  conceived.

  The idea is that the machine is a factory and number is the product. In the same
  way as the mechanised looms created textiles. It is the extension of a model of
  industrial production from goods/commodities to information.

  Babbage conceived two machines: the Difference Engine (DE) and the Analytical
  Engine (AE). Neither were successfully built in his lifetime. The DE preceded
  the AE and was basically an advanced mechanical calculator whereas the AE
  approximated a general purpose computer.

  Despite this, with the Difference Engine, in contrast to preceding _aids to
  calculation_, the steps of the computational algorithm were no longer directed
  by human intelligence but by internal rules embodied in the mechanism and
  automatically generated.

  ### Difference Engine

  The DE's single purpose was to calculate and output mathematical tables such as
  the results of polynomial equations. The idea was that you would input the
  answer a bell would ring and the machine would _halt_. This influenced #Turing
  later. It was non-programmable and designed for a specific set of calculations.

  ### Analytical Engine

  Conceived as a general-purpose computing machine capable of perfoming a wide
  range of calculations, programmable using punched cards similar to those used
  with Jacquard looms.

  It more resembled modern computers in that Babbage used concepts that would
  later translate into the #vonNeumann architecture. There was a "mill" (CPU),
  "store" (memory) and input/output mechanisms. It also had a concept of looping
  and conditional branching.

  ### Lovelace's insight

  A central idea of Ada #Lovelace, expressed in her notes on the Analytical Engine
  is that **number can represent entities other than quantity**.

  If we assign meaning to number, results arrived at by operating on number
  according to rules can say things about the world when mapped back onto the
  world using the meanings assigned to them.

  Lovelace's insight was that the potential of computing lay in the power of
  machines to manipulate representations of the world contained in symbols.

  ## Analogue computers

  Both digital and analogue computers are automatic. They differ in _how they
  represent quantities_ and how their outputs are derived.

  With digital machines, quantity is represented as a string of discrete digits.

  With analogue machines, quantity is a physical property _in itself_ rather than
  a representation. This could be, for example: the lowering of a weight, the flow
  of a liquid, or an electrical charge. This physical behaviour is **analagous**
  to the system that is being modelled. Quantities are continuously variable
  values rather than discrete (discontinuous values).

  Digital machines produce results by _calculation_ whereas analogue machines
  produce results by _measurement_, e.g. the height of liquid in a tank or the
  time it taks for a tank to be emptied.

  ### Examples of analogue computers

  #### The Phillips Hydraulic Computer

  This used fluid to model the workings of the British economy. It consisted of a
  series of transparent plastic tanks and pipes which were fastened to a wooden
  board.

  ![Philips Hydraulic Computer](../img/hydraulic_computer.jpg)

  Each tank represented some aspect of the UK national economy and the flow of
  money around the economy was illustrated by coloured water. At the top of the
  board was a large tank called the treasury. Water (representing money) flowed
  from the treasury to other tanks representing the various ways in which a
  country could spend its money. For example, there were tanks for health and
  education. To increase spending on health care a tap could be opened to drain
  water from the treasury to the tank which represented health spending.

  #### Bush's Differential Analyser

  This was designed to solve differential equations by integration. In contrast to
  the Philips Computer it was general enough to be used to solve problems from
  different contexts. Examples of these contexts: heat flow, ballistics,
  mechanics, population growth, chemical interactions, astronomy.

  ![Differential Analyser](../img/differential_analyser_bush.jpg)

  It was about the size of a room and used shafts, motors, discs and wheels to
  work.

  ### Historiography

  There is a tendency in the history of computing to downplay or diminish the
  contribution of analogue computing devices and to present them as just an
  inferior precursor to the inevitable dominance of digital electronic computers.

  This is ahistorical and inaccurate.

  Analogue (and electromechanical devices) overlapped with and coexisted with
  digital devices for 40 years, spanning the first three generations of digital
  electronic devices. The term "analogue" itself only came about when the need
  arose to distinguish digital devices from other types of computer. They were not
  "rivals" before this.

  ### Electro-mechanical computers

  Electro-mechanical devices (also known as "electronic analogue computers") are a
  sort of midway between full digital devices and analogue computers, forming a
  bridge between the two eras.

  Their heyday was roughly 1935 - 1945.

  Their key components were:

  - [relays](Relays.md) for logic operations
  - rotating shafts and gears for performing calculations
  - punched cards or paper tape for input instructions and outputs

  They were slower than fully digital computers as they were limited by the speed
  of moving components rather than the flow of electric charge. In addition the
  various mechanical parts were prone to wear and needed frequent replacement.

  ### Examples of electro-mechanical computers

  #### Harvard Mark 1 (1937-1944) - Aikin, IBM

  Designed by Howard Aiken and built by IBM (1937-1944). Also known as the
  _Automatic Sequence Controlled Calculator_ (ASCC). A general-purpose
  electro-mechanical computer it was most famously used at Los Alamos by
  #vonNeumann to calculate the blast yield of the atomic bomb.

  ![Harvard Mark One](../img/Harvard_Mark_1.jpg)

  It was more than 15m in length and weighed 5 tonnes comprising over 750,000
  parts. It used paper tape and punched cards for input/output.

  #### Complex Number Calculator (1940) - Bell Labs, Stibitz

  Not general purpose nor was it programmable. It was hardwired to perform a
  specific set of operations on numbers and nothing else. It used relays like the
  others. Its distinguishing feature was that it used a teletype for input rather
  than cards or paper tape.

  ![Calculating panel of the CMC](../img/complex-number-calculator-bell-labs.jpg)

  ![Teletype input of CMC](../img/stibitz-teletype.jpg)

  It comprised a panel (the calculating unit) and teletype (the input). One could
  remotely access the computer from the teletype in another location, providing it
  was connected to Bell Lab's telephone network.

  #### Z3 (1941) - Konrad Zuse

  ![Z3](../img/Zuse-Z3.jpeg)

  First designed in 1938 and completed in 1941. Considered the first fully
  automatic, programmable digital computer although relay-based. Programs were
  stored on external punched tape and binary was used as the number system.
  Destroyed by Allied bombing in the War but a replica (pictured) was built
  in 1998.

  ## Vacuum-tube devices

  Vacuum-tube devices were the first fully electronic digital devices as they used
  vacuum-tubes for implementing binary logic rather than relay switches.

  Canonical examples from this era are the ABC computer, the Collossus and most
  famously, the ENIAC.

  ### The Antanasoff-Berry Computer (1939 - 1942)

  ![ABC Computer](../img/abc-computer.jpg)

  Fully automatic, digital electronic computer but not programmable or really
  general-purpose, being created to do linear equations. It took decimal input and
  output but operated internally using binary arithmetic. The input was punched
  cards and the output was via a front-panel display.

  Its importance stems from the fact that it was the first to use vacuum-tubes for
  the arithmetic calculations of the processor.

  ### Colossus (1943-1945) - Tommy Flowers/Max Newman

  Designed and built at the Post Office Research Station at Dollis Hill. Used for
  cracking the Lorenz messages of the German high command.

  Mainly important (beyond its role in the War) for establishing the viability of
  vacuum-tubes over relays. It comprised 2.4 thousand tubes.

  Debatable the extent to which the Collosus can be called the _first digital
  computer_ since it was not general purpose and not fully programmable.

  Furthermore it was not even a complete solution in its context of application.
  The codebreakers would be given a certain advantage using the Collosus
  (narrowing the field of possible deciphering keys) but still need to use manual
  methods to break the code. It made decipher time shorter rather than executing
  the full diciphering. In contrast to, say, the Bombe computer that would
  identify the key for the days transmissions.

  It's actual influence is also hard to gauge because of the secrecy that remained
  in place after the War. Apart from those who worked on it, its technical
  advancements were not public knowledge until the late 1970s.

  ### ENIAC (1945) John Mauchly/J.P Eckert

  The _Electronic Numerical Calculator and Integrator_

  The main motivation for creating the ENIAC was military: the US Army needed
  speed and accuracy when calculating artiliary firing tables and munition
  trajectory so that shells could hit their intended targets. Accuracy depended on
  myriad factors: gun elevation, shell shape and weight, explosive charge,
  distance, wind, temperature etc. Previously this had been done by human
  computers under the aegis of the Ballistic Research Laboratory.

  ![The main control panel of the ENIAC](../img/eniac.jpg)

  It was not completed until after VE day but was used at Los Alamos after the War
  and retired in 1955.

  The BRL commissioned the Moore School of Engineering at the University of
  Pennsylvania for the construction of an automatic electronic calculator.

  It used vacuum-tubes, along with switches and realys. It used decimal rather
  than binary and had punch card input and output. It was crudely programmable and
  could be set up to work on different problems by rearranging the different
  components using switches and plugboards. But this took roughly two days.

  It's design was very influential because it was made public and not classified.

  ### EDVAC (1944-49) John Mauchly/J.P Eckert/ John von Neumann

  Built at the Moore School for the BRL as a successor to ENIAC by Mauchly and
  Eckert whilst ENIAC was still operational. It's architecture arose out of
  discussions between #vonNeumann and Mauchly and Eckert and was necessitated by
  the need to calculate the behaviour of the fissile materials in the atomic bomb.

  It resulted in the famous _First Draft of a Report on the EDVAC_ by #vonNeumann
  which is seen as the definitive statement of the modern **digital programmable
  general-purpose computer**.

  As a result of the paper, #vonNeumann effectively got the credit for the
  invention even though it synthesised ideas from the ENIAC and the contributions
  of others to the EDVAC.

  There was lots of disputed provenance and claims of plagiarism generally.
  Antanasoff won a legal claim against Eckert and Mauchly for stealing ideas from
  the ABC to make ENIAC.

  // Add more on the hardware.

  ### Concept of the _internal stored program_

  - Key innovation of the EDVAC was the _internal stored program_ (ISP).

  - Solution to the problem of preceding computers requiring down-time when
    switching between applications and problem sets.

  - Circuitry would need to be reconfigured before the computer could run on the
    next problem

  - ISP removed the friction:

    - the instructions comprising the program would be prepared on tape or punched
      cards and read into electronic memory
    - the hardware configuration of the machine would remain the same accross
      different programs and require no reconfiguration, only the input program
      data would change

  - This was basically the invention of software. Both instructions and data
    shared the same memory space once they were read-in. Before, the data would be
    stored in memory but the instructions would be read one-by-one from the
    external storage media.

  - Key consequences:
    - flexibility (no need for re-wiring)
    - speed: fetching instructions from memory is quicker than fetching from
      external storage devicde
    - self-modification: since they were in memory, programs could modify
      themselves during their execution (e.g. use branching conditions) and
      treating one program as data for another. This enabled more complex
      algorithms and also opened the possibility of higher-level languages and
      compilers
    - simplified architecture: both programs and data in memory
    - generalisation: computers could switch between different tasks simply by
      loading different programs

  ### Post-EDVAC devices

  There were several notable devices which attempted to implement the architecture
  described in the _First Draft_ using vacuum tubes for logic operations and a
  variety of different electronic methods for memory:

  - the Manchester "Baby"
  - the Cambridge EDSAC
  - Turing's ACE computer

  #### Manchester Baby (1948)

  ![The Manchester Baby Computer](../img/manchester_baby.png)

  - An experimental computer intended to create the
    [von Neumann architecture](CPU_architecture.md) using
    [Williams_Tube_RAM](Williams_Tube_RAM.md)

  - Developed at the Univesity of Manchester and completed in 1948.

  - Considered the first electronic stored-program computer and first to contain
    all the elements of a modern electronic digital computer.

  #### EDSAC (1949) J.Wilkes et al.

  ![EDSAC](../img/EDSAC.jpg)

  - _Electronic Delay Storge Automatic Computer_

  - Constructed by Maurice Wilkes and others at the Mathematical Laboratory of the
    University of Cambridge.

  - The second digital stored-program computer after the Manchester Baby.

  - It used vacuum-tubes for the arithmetical operations in the ALU and mercury
    [delay line memory](Delay_line_memory.md) for the RAM.

  - Designed to be used by relatively non-specialist practitioners from other
    university departments who were expected to program it themselves. To this
    end, a formal programming paradigm was forged for the EDSAC which established
    the following:

    - subroutines as a library of common procedures available to programmers (e.g.
      printing a result, reading input tape, program checking, mathematical
      operations)
    - diagnostics: techniques for verifying program code and its correctness

  ![Programming Digital Electronic Computer book](../img/programming_a_digital_electronic_computter.jpg)

  - The key players wrote the first textbook on programming in 1951: _The
    Preparation of Programs for an Electronic Digital Computer_ (Wilkes, Wheeler,
    and Gill)

  #### The Automatic Computing Engine (ACE) (1946) Alan Turing

  - Technical design provided by Alan #Turing working at the Mathematics Division
    of the National Physical Laboratory. The product of Turing's theoretical work
    in "On Computable Numbers" where he proposes the concept of a
    [Turing_machine](Turing_machines.md) and based on his experience with early
    single-purpose computing devices at Bletchley.

  - Due to secrecy over wartime work, it was hard for get Turing to build a fully
    electronic implementation since this would disclose advances made during the
    War. Tommy Flowers was intended to be brought in to build it but again this
    was scuppered over the classification of wartime technology. Had this not been
    the case, it is likely it would've been seen as on par with the EDVAC in its
    design, if not dwarfing it.

  - Turing proposed a stored program architecture with high-speed memory. It would
    be more perfomant than the EDVAC as a result.

  ![The Pilot ACE computer](../img/pilot-ace.jpg)

  - Turing's actual design was not implemented as it was thought too ambitious
    given how powerful it would need to be. Instead a smaller prototype was made,
    the Pilot ACE. It used vacuum-tubes for logic, and mercury delay lines for
    memory. Data and memory was input via punched cards.

  ### Further applications of the EDSAC architecture

  ### LEO I

  - Built by Lyons Tea Company to manage business affairs (payroll, inventory,
    stock management)

  - Marked a shift from military and academic contexts for computers to business
    and data management.

  ### IAS machines

  ![The MANIAC computer](../img/MANIAC_computer.jpg)

  Several machines were built at the Institute for Advanced Study utilising the
  "von Neumann" architecture and associated advancements such as vacuum-tubes and
  Williams' tubes. There was the original IAS machine (1952) as well as the
  JOHNNIAC (1954) and MANIAC (1956).

  Although their purpose was military (Los Alamos), their designs were public and
  widely studied making them influential outside of academia.

  ### UNIVAC (1951)

  ![UNIVAC_computer](../img/univac-1.jpg)

  Mauchley and Eckert, who had designed the ENIAC left the Moore School and went
  into business: Eckert-Mauchley Computer Corporation. This was bought by
  Remington Rand and in subsequent years became the main competitor to IBM.

  In this capacity they built the UNIVAC: _Universal Automatic Computer_. It's
  name being an embodiment of its nature as a general-purpose, electronic digital
  computer.

  It used vacuum-tubes for logic and mercury delay lines for memory. It had
  multiple means of input/output including: directly via an operator console
  (basically a typewriter keyboard), magnetic tape for input and output, along
  with punched cards.

  ![UNIVAC operator console](../img/univac-operator-console.jpeg)

  It was the first computer specifically designed to include business and
  administrative use. This was underscored by its first client: the US Census
  Bureau.

  A key event was its succesful prediction of the 1952 general election. It
  correctly predicted a landslide for Eisenhower (against expectations). (This was
  so unlikely, they actually fudged the data because they thought the machine was
  way off.) It was a novelty on the results night but it cemented a certain
  concept of the computer in the public imagination - large, room sized machines
  with blinking lights.

  Swade notes that Eckert and Mauchley effectively launched the US commercial
  computer industry with the UNIVAC.

  ## IBM

  - First created in 1911 as a merger of four companies all involved in some
    capacity in business data processing. The most significant of these was the
    Hollerith Tabulating Machine Company who's punched card machines had been used
    in the 1890 census, greatly reducing its collation time

  - Named IBM in 1924

  - UNIVAC represented the encroachment of computing devices into IBM's market of
    punched-card based data-processing and effectively put IBM on notice.

  - To attack the threat IBM brought out three crucial machines: the 701, 702
    and 650.

  - IBM's approach to marketing their computers was to emphasise the comfort of
    continuity over the shock of the new represented by the UNIVAC. IBM had
    already captured the business market. Their computers would be more of the
    same, just faster and more powerful than their punched card machines due to
    the electronics. To this end they initially did not call their devices
    'computers'.

  - IBM were ultimately totally successful in their pivot to the computing market.
    By 1960 it owned over 70% of the data processing and computer market.

  ### IBM 701 ("Defense Calculator") (1952)

  ![Control unit of the IBM 701](../img/ibm-701-control-unit.jpeg)

  IBM's first electronic computer. It directly competed with the UNIVAC for
  government contracts. It followed the prevailing approach of vaccuum tubes for
  logic and had a variety of storage methods: Williams tubes,
  [magnetic tape](Magnetic_tape.md) , and magentic drums.

  701s were used at Los Alamos and avionics companies. Some were also used to
  manage payroll in business contexts.

  ### IBM 702 ("Tape Processing Machine") (1953)

  ![Advertisement for the IBM 702](../img/ibm-add.jpeg)

  Focused primarily on business applications and targetted at businesses rather
  than government contracts. Less powerful than the 701.

  ### IBM 650 (1953)

  ![IBM 650](../img/ibm-650.jpg)

  Low-cost general purpose machine using magnetic drum memory. Marketed as slower
  but more affordable than the 700 range. It was mass-produced unlike the others
  which were built for specific customers. It was in fact the first mass-produced
  computer in the world. It proved the breakout star in IBMs initial line up and
  sometimes called "IBM's Model T".

  ![IBM_650 console panel](../img/ibm-650-console-panel.jpg)

  IBM offerred 650s to univesities at a 60% discount on the condition that the
  universities would establish courses in computing. This was shrewd as it meant
  that a whole generation of engineers and computer scientists learned on 650s and
  would prefer them in their research. It also created a pool of skilled users
  that would go on to work at IBM.

  Donald Knuth dedicated _The Art of Computer Programming_ to the 650.

  It used [magnetic drum memory](./Magnetic_drum_memory.md) along with
  vacuum-tubes.

  ### IBM 1401 (1959)

  ![IBM 1401](../img/ibm-1401.jpg)

  Another IBM mainframe. Transistors replaced vacuum-tubes. Magnetic core storage
  replaced magnetic drum storage. It was housed in rectangular light-blue cabinets
  and the ubiquity of the 1401 in industry earned IBM the moniker 'Big Blue'.

  ### IBM System/360 (1964)

  ![IBM 360](../img/ibm-360.jpg)

  Named to suggest all-round compatibility - a family of mainframes designed to
  cover commercial and scientific applications. Considered one of history's most
  successful computers.

  Up until this point all IBM computers had a programming language unique to the
  specific processor. At that point there were about seven IBM computer models in
  active use and they were all incompatible with each other.

  ![IBM 360 promo poster](../img/ibm-360-in-use.jpeg)

  In contrast the 360 computers all used the same programming language. This meant
  they were interoperable with each others. Because the there were variants in the
  360 ranges, customers could purchase a smaller system knowing they could expand
  it if their needs grew, without reprogramming their application software.

  ## Magnetic core devices: Whirlwind and SAGE

  Two devices that leveraged the new technology of
  [magnetic_core_memory](Magnetic_core_memory.md) where the Whirlwind and SAGE
  computers. The ENIAC was also updated to use magnetic cores.

  ![Whirlwind magnetic core memory](../img/whirlwind-magnetic-cores.JPG)

  The Whirlwind computer (1953) was a flight simulator and the first to use
  magnetic cores. Crucially was able to operate in realtime for output.

  The SAGE computer (_Semi-Automatic Ground Environment_) emerged as a response to
  the Soviet acquisition of the atomic bomb in 1949. The Americans realised they
  needed a much better early warning and air defence system that would allow them
  to identify bombers in their airspace and dispatch fighters.

  To achieve this, it was necessary to detect enemy bombers with radar and compute
  an interception course. Alterations in the bomber flight path required
  continuous updates and real-time computation of directions for the responding
  fighter.

  The resulting computer was SAGE (made by IBM) which was modelled on Whirlwind,
  using magnetic cores. SAGE computers were spread accross the continental US in
  sectors and managed by NORAD.

  ![SAGE command post](../img/SAGE_command_post.jpg)

  The SAGE system was operation between 1958-1984, receiving continual
  improvements and updates. In addition to magnetic cores it introduced many
  technological innovations that influenced computers generally: interactive
  screens, printed circuit boards, mass-storage devices, digital communication
  over telephone lines, time-sharing, along with advances in software.

  Because so many private companies worked as contractors on SAGE (IBM, Burroughs,
  Bell Labs, RAND Corp), the military technology (both hardware and software) made
  its way from the military context to business and consumer computing.

  An example of a civilian application of SAGE technology was airline booking
  systems, where booking reservation data needed to be processed in realtime. IBM
  worked with American Airlines to introduce this.

  ## Transistor and integrated circuit revolution

  The arc of modern computing history (and its main eras) can be presented as

  - vacuum-tubes
  - [transistors](Transistors.md)
  - [integrated_circuits](Integrated_circuits.md)

  The transistor was invented at Bell Labs in 1947 by William Shockley and others.
  They were not created with computers in mind. Instead they were first used in
  hearing aids (1953) and transistor radios (1954).

  From the mid-1950s onwards they started being used in military computers.

  They had several advantages over vacuum-tubes: they were durable and reliable
  with no required startup time, they were power efficient and small.

  The actual period where transistors alone were supreme in the form of a
  transistor board was relatively short-lived and rapidly gave way to ICs.

  Computers of the transistor era had the following components separate from each
  other: the transistors, wired connections, resistors and capacitors. Integrated
  circuits put these all on the same piece of silicon.

  They were first manufactured by Fairchild Semiconductor which was an offshoot of
  Shockley's Shockley Semiconductor Laboratory.

  The initial marked for ICs was military and scientific: the Minuteman ICBM and
  computers of the Apollo spacecraft. This laid the groundworkd for later
  commercial use and the personal computer revolution.

  ## Mini-computers

  Mini computers did not threaten mainframes but they opened up a new class of
  user. They were vastly cheaper than a mainframe and only the size of a small
  fridge.

  The main player was DEC (Digital Equipment Corporation) starting in the
  mid-1960s. The introduced the PDP-8 in 1965 (Programmed Data Processor). The
  PDP-8 used transisotrs and magnetic core memory. It was affordable to smaller
  businesses if not yet, consumers.

  ![The PDP-8](<../img/540px-PDP-8_(1).jpg>)

  The internals were made public and DEC encouraged making the machine extensible
  by users being permitted to create their own programs and specialised
  applications. This was a very different culture to IBM where technicians were
  required and machines were typically leased.

  By 1988, DEC was the second largest computer company after IBM. This led to
  additional companies entering the mini-computer market: Data General, Honeywell,
  Hewlett-Packard.

  Mini-computers died as a category in the mid-1990s in the wake of the PC.

  ## The Personal Computer

  The PC emerged out of advances in IC chip design - this was the engine of the
  PC.

  Intel (newly founded by Robert Noyce and Gordon Moore) were charged with making
  custom chips for a Japanese calculator company. Instead of creating the custom
  chips, they advised the company to use a general purpose chip that could handle
  general computing as well as the basic calculator functions. This generic chip -
  the Intel 4004 - was the microprocessor combined with three other ICs
  (comprising 2.3k transistors) that comprised the fundamental building blocks of
  the #vonNeumann architecture.

  ![Intel 4004 chip](../img/intel-4004.jpg)

  However it took until the Apple II in 1977 for the phrase "personal computer" to
  enter the lexicon and become an identifiable class.

  The broader context for this was the computer hobbyists in Silicon Valley/Menlo
  Park (particularly the Homebrew Club of amateur computer engineers) along with
  idealistic figures from the 60s counterculture (such as Stuart Brand) who began
  associating computing technology as a tool of personal liberty and fulfillment.

  ### Microsoft and Apple

  The PC "boom" began with the Altair 8800. This was a mail-order kit that
  required manual assembly. It was released by Micro Instrumentation Telemetry
  Systems (MITS) in 1975. It couldn't do much but had expansion capability
  (memory, teletype interface, casette player for data storage).

  ![Altair 8800](../img/altair-8800.jpg)

  Bill Gates and Paul Allen made a proposal to MITS: they would write software
  that would allow users to program the Altair in BASIC. They agreed and were
  prepared to distribute the software. As a result Gates and Allen founded
  Microsoft in 1975.

  Around the same time, Steve Wozniak (member of the Homebrew club) built the
  Apple I as a single-board hobbyist project. He made this available to buy via
  mail order and formed Apple with Steve Jobs in 1976 to manage the enterprise.

  ![Apple I](../img/apple1.jpg)

  By 1977 they had investment capital and brought out the Apple II. This was sold
  preassembled with casing and required no soldering. It had expansion slots for
  third-party vendors to create compatible devices and had colour graphics.

  The design of the Apple II was published in _Byte_ in order to invite
  crowd-sourced plugins and extensions. One of these was the Z-80 softcard which
  allowed it to run the CP/M OS (created by the company Digital Research) giving
  ready-made access to software such as a word-processor, spreadsheets and
  databases.

  ![Apple 2](../img/apple2.jpg)

  Other competitors in the PC market at this time were Radioshack's TRS-80 and the
  Commodore PET.

  ### IBM

  All the while it was hobbyist and tinkerers driving the PC market, IBM had no
  real interest. When PCSs started being used by businesses and their capacity
  grew to include typical business applications (databases, spreadsheets) it
  sensed an incursion into its market.

  ![IBM PC](../img/ibm-pc.jpg)

  Thus, in 1981, IBM launched the "IBM Personal Computer" which rapidly became the
  industry standard. This had the effect of legitimising the concept of a PC.
  There was overwhelming demand. In their marketing IBM melded home and office use
  to capture both markets at once.

  The internal specification of the IBM PC was freely available (dubbed 'open
  architecture'), in contrast to IBM's business computers, to encourage software
  expansion. Cards could be created to fit in its expansion slots adding graphics,
  sound, additional memory and networking capability.

  IBM originally contracted Digital Research to create software based on CP/M but
  this fell through. Microsoft got the deal to supply the OS which became MS-DOS
  (Microsoft Disk Operating System). Henceforth every PC and PC-clone came with
  DOS bundled.


path: zk/Python_application_structure.md
sortable_path: zkPython_application_structure.md
title: Python application structure
lead: ## Directory layout
checksum: 49847157a3a69633258b7d00d93230bf9d910ce35d8d9c93ba462884d16c2071
created: 2024-08-27 14:33:04.042157825+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"created":"Tuesday, August 27, 2024","tags":["python"]}
body:
  ## Directory layout

  ```
  src/
      cli.py OR app.py
      constants.py
      __init__.py
  tests/
  requirements.txt
  setup.py
  ```

  ## `requirements.txt`

  Dependencies used within the venv for the project, e.g:

  ```
  google-auth==1.35.0
  google-auth-httplib2==0.1.0
  google-auth-oauthlib==0.4.6
  google-api-python-client==2.26.1
  gspread==3.7.0
  requests==2.25.1
  oauth2client==4.1.3
  ```

  ## `setup.py`

  The declaration file for the application, equivalent to `package.json`, e.g:

  ```py
  from setuptools import setup, find_packages

  setup(
      name="eolas-db",
      version="0.1",
      packages=find_packages(where="src"),
      package_dir={"": "src"},
      install_requires=[
          # List your project dependencies here
      ],
      entry_points={
          "console_scripts": [
              "eolas-db=cli:main",
          ],
      },
  )
  ```

  The `entry_points` key is important. This is the first execution point of the
  application. Typically you would name your entry-point script as `app.py` or
  `cli.py` and then specify its `main` function.

  E.g. in `cli.py`:

  ```py
  def main():
      print('run the application')

  if __name__ == "__main__":
      main()

  ```

  ## `__init__.py`

  See
  [[Python_modules_and_imports#distinguishing-_modules_-from-_packages_]]|Python
  modules]]

raw_content:
  ---
  tags: [python]
  created: Tuesday, August 27, 2024
  ---

  # Python application structure

  ## Directory layout

  ```
  src/
      cli.py OR app.py
      constants.py
      __init__.py
  tests/
  requirements.txt
  setup.py
  ```

  ## `requirements.txt`

  Dependencies used within the venv for the project, e.g:

  ```
  google-auth==1.35.0
  google-auth-httplib2==0.1.0
  google-auth-oauthlib==0.4.6
  google-api-python-client==2.26.1
  gspread==3.7.0
  requests==2.25.1
  oauth2client==4.1.3
  ```

  ## `setup.py`

  The declaration file for the application, equivalent to `package.json`, e.g:

  ```py
  from setuptools import setup, find_packages

  setup(
      name="eolas-db",
      version="0.1",
      packages=find_packages(where="src"),
      package_dir={"": "src"},
      install_requires=[
          # List your project dependencies here
      ],
      entry_points={
          "console_scripts": [
              "eolas-db=cli:main",
          ],
      },
  )
  ```

  The `entry_points` key is important. This is the first execution point of the
  application. Typically you would name your entry-point script as `app.py` or
  `cli.py` and then specify its `main` function.

  E.g. in `cli.py`:

  ```py
  def main():
      print('run the application')

  if __name__ == "__main__":
      main()

  ```

  ## `__init__.py`

  See
  [[Python_modules_and_imports#distinguishing-_modules_-from-_packages_]]|Python
  modules]]


path: zk/Transport_Layer_of_Internet_Protocol.md
sortable_path: zkTransport_Layer_of_Internet_Protocol.md
title: Transport Layer of the Internet Protocol
checksum: 0e37694c48b92f88286f2e70bec9ccf9f0085572171cd1f2f1a2d44c0db5d344
created: 2024-09-03 05:37:32.871908625+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"created":"Tuesday, September 03, 2024","tags":["internet","networks","ports"]}
lead:
  Transport Layer protocols provide a communications channel for data to be sent
  between hosts. The Transport Layer is dependent upon the
  [Internet Layer](Internet_Layer_of_Internet_Protocol.md) for addressing and
  routing the data and the
  [Link_Layer_of_Internet_Protocol](Link_Layer_of_Internet_Protocol.md) for
  transferring the data to a specific host once it has reached the local network.

body:
  Transport Layer protocols provide a communications channel for data to be sent
  between hosts. The Transport Layer is dependent upon the
  [Internet Layer](Internet_Layer_of_Internet_Protocol.md) for addressing and
  routing the data and the
  [Link_Layer_of_Internet_Protocol](Link_Layer_of_Internet_Protocol.md) for
  transferring the data to a specific host once it has reached the local network.

  The protocol governing the Transport Layer is the **Transmission Control
  Protocol** (TCP).

  A TCP **fragment** fits within an IP packet's data section, giving us the
  following iteration of the network stack:

  ![Transport layer fragment](../img/transport-layer-segment.png)

  The segment header contains a destination network port number. The port number
  identifies the specific service or process on the host device which will receive
  the data.

  A useful analogy:

  _The IP address is like the street address of an office building. The port
  number is like the office number of a specific worker within the building. Once
  a letter arrives at the building, the mail processing office (like the OS) is
  not the intended recipient. Instead the mail worker passes on the letter to the
  intended recipient. Likewise, the OS examines the port number and delivers the
  inbound data to the process listening on the specified port._

  ## Ports

  A host with a single IP address can have multiple active ports, each used for
  performing a different type of activity on the network.

  Both servers and clients possess ports. A server will have a dedicated port
  which it listens for requests on (typically port 80 or port 443). A client will
  have an "ephemeral" port where it listens for responses from a server. They are
  ephemeral because they are open only temporarily to receive a response to a
  certain request and are assigned dynamically by the networking components of the
  OS.

  An IP address plus a port (e.g: 192.168.2.1:3000) constitutes an **endpoint**.
  An instance of an endpoint is a **socket**.

  If multiple clients connect to the same endpoint each has its own socket (for
  instance on a host that is a server). A socket can listen for new connections or
  it can represent an established, ongoing connection.

raw_content:
  ---
  tags: [internet, networks, ports]
  created: Tuesday, September 03, 2024
  ---

  # Transport Layer of the Internet Protocol

  Transport Layer protocols provide a communications channel for data to be sent
  between hosts. The Transport Layer is dependent upon the
  [Internet Layer](Internet_Layer_of_Internet_Protocol.md) for addressing and
  routing the data and the
  [Link_Layer_of_Internet_Protocol](Link_Layer_of_Internet_Protocol.md) for
  transferring the data to a specific host once it has reached the local network.

  The protocol governing the Transport Layer is the **Transmission Control
  Protocol** (TCP).

  A TCP **fragment** fits within an IP packet's data section, giving us the
  following iteration of the network stack:

  ![Transport layer fragment](../img/transport-layer-segment.png)

  The segment header contains a destination network port number. The port number
  identifies the specific service or process on the host device which will receive
  the data.

  A useful analogy:

  _The IP address is like the street address of an office building. The port
  number is like the office number of a specific worker within the building. Once
  a letter arrives at the building, the mail processing office (like the OS) is
  not the intended recipient. Instead the mail worker passes on the letter to the
  intended recipient. Likewise, the OS examines the port number and delivers the
  inbound data to the process listening on the specified port._

  ## Ports

  A host with a single IP address can have multiple active ports, each used for
  performing a different type of activity on the network.

  Both servers and clients possess ports. A server will have a dedicated port
  which it listens for requests on (typically port 80 or port 443). A client will
  have an "ephemeral" port where it listens for responses from a server. They are
  ephemeral because they are open only temporarily to receive a response to a
  certain request and are assigned dynamically by the networking components of the
  OS.

  An IP address plus a port (e.g: 192.168.2.1:3000) constitutes an **endpoint**.
  An instance of an endpoint is a **socket**.

  If multiple clients connect to the same endpoint each has its own socket (for
  instance on a host that is a server). A socket can listen for new connections or
  it can represent an established, ongoing connection.


path: zk/Application_Layer_of_Internet_Protocol.md
sortable_path: zkApplication_Layer_of_Internet_Protocol.md
title: Application Layer of the Internet Protocol
lead: The final, topmost layer of the Internet Protocol suite.
checksum: b1dff8e8911ef498988c406e96ff83d64f86f048499cc3c958ca5d4b29b5edfa
created: 2024-09-06 06:11:59.753192243+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Friday, September 06, 2024","tags":["internet","networks"]}
body:
  The final, topmost layer of the Internet Protocol suite.

  ![Data at the Application Layer](../img/application-layer-data.png)

  The Application Layer is where we get to the protocols that describe the
  behaviour of applications. All the preceding lower levels are effectively
  **plumbing that enables applications to do the things they want accross the
  internet**.

  Whereas the lower levels facilitate communication accross the internet, the
  Application Layer is concerned with achieving a specific task once the ability
  to communicate is assured.

  Application Layer protocols include HTTP, the Simple Mail Transfer Protocol
  (SMTP), and FTP.

raw_content:
  ---
  tags: [internet, networks]
  created: Friday, September 06, 2024
  ---

  # Application Layer of the Internet Protocol

  The final, topmost layer of the Internet Protocol suite.

  ![Data at the Application Layer](../img/application-layer-data.png)

  The Application Layer is where we get to the protocols that describe the
  behaviour of applications. All the preceding lower levels are effectively
  **plumbing that enables applications to do the things they want accross the
  internet**.

  Whereas the lower levels facilitate communication accross the internet, the
  Application Layer is concerned with achieving a specific task once the ability
  to communicate is assured.

  Application Layer protocols include HTTP, the Simple Mail Transfer Protocol
  (SMTP), and FTP.


path: zk/Example_scenario_internet_data_transfer.md
sortable_path: zkExample_scenario_internet_data_transfer.md
title: Example scenario of data transfer accross the internet
lead: ![Path of a network request accross the internet](../img/network-request-layers.png)
checksum: 4d2c68558e76d803d975f06cae05c6a43a97ef0bde82be9a1c805c7206750565
created: 2024-09-06 08:39:27.364238465+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Friday, September 06, 2024","tags":["networks","internet"]}
body:
  ![Path of a network request accross the internet](../img/network-request-layers.png)

  ![All layers of the IP](../img/all-layers-of-IP.png)

  - A client device is connected to a wireless WiFi network.
  - This network is connected to the internet via a router.
  - Elsewhere on the internet is a server also connected to the internet via
    router.
  - A user of the client device opens a web browser and requests a web page hosted
    on the server (we will assume it already knows the IP address)
  - The web browser knows HTTP - a protocol of the
    [Application Layer](Application_Layer_of_Internet_Protocol.md) - and forms an
    HTTP request intended for the destination server.
  - The browser hands off the HTTP request to the TCP/IP software stack of the
    device's [operating system](Basic_model_of_the_operating_system.md), asking
    that the data be delivered to the server on its IP address and at port 80.
  - The TCP/IP software stack on the client OS encapsulates the HTTP payload in a
    TCP segment (at the
    [Transport Layer](Transport_Layer_of_Internet_Protocol.md)), setting the
    destination port to 80 in the segment header.
  - The [Internet Layer](Internet_Layer_of_Internet_Protocol.md) software on the
    client then wraps the TCP segment in an IP packet, which includes the
    destination IP address of the server in the packet header.
  - At the [Link Layer](Link_Layer_of_Internet_Protocol.md) of the client device,
    the IP packet is encapsulated in a frame with the
    [MAC_address](MAC_addresses.md) of the local router in its header. The frame
    is wirelessly transmitted by the client device's WiFi hardware.
  - The wireless access point receives the frame and passes to the router. The
    router examines the internet layer packet to determine the destination IP
    address.
  - To reach the server, the request needs to travel through multiple routers on
    the internet and the process is replicated (looking up MAC address of the
    device and determining IP target). Eventually, the request reaches the router
    on the network where the server is connected.
  - The last router puts the packet in a frame suitable for the server's local
    network and in the frame's header is the MAC address of the server.
  - The server receives the frame and the packet is passed to the TCP/IP software
    stack. This passes the HTTP data to the process listening on TCP port 80.
  - Web server software, listening on port 80 handles the request and replies to
    the client, start ing the process again in reverse order.

raw_content:
  ---
  tags: [networks, internet]
  created: Friday, September 06, 2024
  ---

  # Example scenario of data transfer accross the internet

  ![Path of a network request accross the internet](../img/network-request-layers.png)

  ![All layers of the IP](../img/all-layers-of-IP.png)

  - A client device is connected to a wireless WiFi network.
  - This network is connected to the internet via a router.
  - Elsewhere on the internet is a server also connected to the internet via
    router.
  - A user of the client device opens a web browser and requests a web page hosted
    on the server (we will assume it already knows the IP address)
  - The web browser knows HTTP - a protocol of the
    [Application Layer](Application_Layer_of_Internet_Protocol.md) - and forms an
    HTTP request intended for the destination server.
  - The browser hands off the HTTP request to the TCP/IP software stack of the
    device's [operating system](Basic_model_of_the_operating_system.md), asking
    that the data be delivered to the server on its IP address and at port 80.
  - The TCP/IP software stack on the client OS encapsulates the HTTP payload in a
    TCP segment (at the
    [Transport Layer](Transport_Layer_of_Internet_Protocol.md)), setting the
    destination port to 80 in the segment header.
  - The [Internet Layer](Internet_Layer_of_Internet_Protocol.md) software on the
    client then wraps the TCP segment in an IP packet, which includes the
    destination IP address of the server in the packet header.
  - At the [Link Layer](Link_Layer_of_Internet_Protocol.md) of the client device,
    the IP packet is encapsulated in a frame with the
    [MAC_address](MAC_addresses.md) of the local router in its header. The frame
    is wirelessly transmitted by the client device's WiFi hardware.
  - The wireless access point receives the frame and passes to the router. The
    router examines the internet layer packet to determine the destination IP
    address.
  - To reach the server, the request needs to travel through multiple routers on
    the internet and the process is replicated (looking up MAC address of the
    device and determining IP target). Eventually, the request reaches the router
    on the network where the server is connected.
  - The last router puts the packet in a frame suitable for the server's local
    network and in the frame's header is the MAC address of the server.
  - The server receives the frame and the packet is passed to the TCP/IP software
    stack. This passes the HTTP data to the process listening on TCP port 80.
  - Web server software, listening on port 80 handles the request and replies to
    the client, start ing the process again in reverse order.


path: zk/Relays.md
sortable_path: zkRelays.md
title: Relays
lead: A relay is an electrically operated switch.
checksum: 9ac0d0a04e61d1e64dd26710c70c57a00103a50502c4bc15fb7f696c9336bc8d
created: 2024-09-09 15:47:45.553399322+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"created":"Monday, September 09, 2024","tags":["electronics"]}
body:
  A relay is an electrically operated switch.

  It uses an electromagnet to mechancially operate a switch. A switch in one
  circuit can be used the trigger the operation of another circuit.

  ## Core components

  - electromagnet
    - a coil of wire around an iron core
  - armature
    - a movable magnetic strip
  - contacts
    - electrical connections that can be open or closed
  - spring
    - returns the armature to its resting position when not energised

  ## Operation

  ![Operation of a relay](../img/relay.gif)

  Current flows through the electromagnet which creates a magnetic field. This
  field attracts the armature which then moves to open or close the contacts. When
  the current stops, this returns the armature to its original position.

  ## Usage

  During the
  [electro-mechanical era of computing](The_History_of_Computing_Swade.md) they
  were used as precursors to vacuum tubes (and later, transistors) to implement
  logic gates. While they were superseded in this domain, they are still used in
  industrial control systems and automobiles.

raw_content:
  ---
  tags: [electronics]
  created: Monday, September 09, 2024
  ---

  # Relays

  A relay is an electrically operated switch.

  It uses an electromagnet to mechancially operate a switch. A switch in one
  circuit can be used the trigger the operation of another circuit.

  ## Core components

  - electromagnet
    - a coil of wire around an iron core
  - armature
    - a movable magnetic strip
  - contacts
    - electrical connections that can be open or closed
  - spring
    - returns the armature to its resting position when not energised

  ## Operation

  ![Operation of a relay](../img/relay.gif)

  Current flows through the electromagnet which creates a magnetic field. This
  field attracts the armature which then moves to open or close the contacts. When
  the current stops, this returns the armature to its original position.

  ## Usage

  During the
  [electro-mechanical era of computing](The_History_of_Computing_Swade.md) they
  were used as precursors to vacuum tubes (and later, transistors) to implement
  logic gates. While they were superseded in this domain, they are still used in
  industrial control systems and automobiles.


path: zk/Turing_completeness.md
sortable_path: zkTuring_completeness.md
title: Turing Completeness
lead:
  We know that a [Turing machine](Turing_machines.md) is a theoretical construct
  of a computer that:

checksum: 6955170d1e0c4ae74beaea68d24b8d72bbe977f01608376028bdbd8b888f430e
created: 2024-09-13 17:04:46.513183149+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Friday, September 13, 2024","tags":["theory-of-computation","Turing"]}
body:
  We know that a [Turing machine](Turing_machines.md) is a theoretical construct
  of a computer that:

  > contains mutable state, executes sequences of simple instructions that read
  > and write that state, and can pick different execution paths depending on the
  > state (via conditional branch instructions.)

  A Turing Complete (TC) system is a system that abides by, or can be reduced to,
  the above description.

  TC also serves as a _definition of computability_ and provides a formal basis
  for conceiving of computation at a theoretical level.

  All Turing Complete systems are functionally equivalent. This means they can
  simulate each other given enough time and memory. Similarly a TC system can in
  principle perform any computation that any other programmable computer can
  perform. This is true for _other_ TC systems and also those that are not TC
  however the inverse doesn't hold: a non-TC system cannot emulate a TS system.
  For instance a calculator cannot do what a TC smart phone can do. But a smart
  phone can act as a calculator.

  This concept of completeness is also expressed in terms of a Universal Turing
  Machine - a TM capable of simulating any other Turing machine. Given that the
  ability of a TM to simulate another TM is the condition for completeness, a TC
  system is also a UTM.

  Completeness applies to the hardware of computers as well as their software.

  Turing Completeness is the theoretical basis of the practical concept of a
  "general-purpose computer": a general-purpose computer is such because it is
  TC - it can in theory compute anything that is computable.

  Most modern programming languages are Turing Complete in that they can, in
  theory, be used to compute anything that is computable.

raw_content:
  ---
  tags: [theory-of-computation, Turing]
  created: Friday, September 13, 2024
  ---

  # Turing Completeness

  We know that a [Turing machine](Turing_machines.md) is a theoretical construct
  of a computer that:

  > contains mutable state, executes sequences of simple instructions that read
  > and write that state, and can pick different execution paths depending on the
  > state (via conditional branch instructions.)

  A Turing Complete (TC) system is a system that abides by, or can be reduced to,
  the above description.

  TC also serves as a _definition of computability_ and provides a formal basis
  for conceiving of computation at a theoretical level.

  All Turing Complete systems are functionally equivalent. This means they can
  simulate each other given enough time and memory. Similarly a TC system can in
  principle perform any computation that any other programmable computer can
  perform. This is true for _other_ TC systems and also those that are not TC
  however the inverse doesn't hold: a non-TC system cannot emulate a TS system.
  For instance a calculator cannot do what a TC smart phone can do. But a smart
  phone can act as a calculator.

  This concept of completeness is also expressed in terms of a Universal Turing
  Machine - a TM capable of simulating any other Turing machine. Given that the
  ability of a TM to simulate another TM is the condition for completeness, a TC
  system is also a UTM.

  Completeness applies to the hardware of computers as well as their software.

  Turing Completeness is the theoretical basis of the practical concept of a
  "general-purpose computer": a general-purpose computer is such because it is
  TC - it can in theory compute anything that is computable.

  Most modern programming languages are Turing Complete in that they can, in
  theory, be used to compute anything that is computable.


path: zk/Delay_line_memory.md
sortable_path: zkDelay_line_memory.md
title: Delay line memory
lead:
  - First described in the _First Draft_ by #vonNeumann based on work by Eckert
  and Mauchley.

checksum: 4537d92f87f1506fe9062b26ba88cd6874b08310592a3ee423201512025bc777
created: 2024-09-18 05:49:18.875020915+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Wednesday, September 18, 2024","tags":["computer-history","memory"]}
body:
  - First described in the _First Draft_ by #vonNeumann based on work by Eckert
    and Mauchley.

  - Information is stored as acoustic waves travelling through a medium, typically
    mercury. As sound travels more slowly than electric charge (light), if
    electric signals are converted to sound, the delay time involved can be used
    as a type of store.

  - It worked as follows:
    - Imagine we have an 8-bit number. The number is input as a sequence of pulses
      where a pulse is 1 and the absence of a pulse is 0.
    - The pulses are converted into sound waves (using a transducer, as with a
      speaker) and sent through a mercury-filled tube.
    - The length of the tube is calculated so that it takes exactly the time of
      one machine cycle for a pulse to travel from one end to the other
    - At the other end of the tube, another transducer converts the sound waves
      back to electrical pulses
    - This operates in a loop: the pulses go back into the input end of the tube
      and the cycle runs again.

  To read the data, the computer would "listen" to the pulses coming out of the
  receiving end. To write new data, the computer would inject new pulses at the
  precise moment when it wants to change a '0' to a '1' or vice versa.

  For an 8-bit number there would be 8 pulses for each bit and it would be
  necessary to keep track of 8 specific positions in the delay line where each bit
  would be stored.

  The limitations were timing - difficult to keep track of each precise bit and
  when it would arrive at the output. Also it was sequential rather than random
  access. Not every bit could be accessed at once, you had to wait for its time
  slot to come around to access it.

raw_content:
  ---
  tags: [computer-history, memory]
  created: Wednesday, September 18, 2024
  ---

  # Delay line memory

  - First described in the _First Draft_ by #vonNeumann based on work by Eckert
    and Mauchley.

  - Information is stored as acoustic waves travelling through a medium, typically
    mercury. As sound travels more slowly than electric charge (light), if
    electric signals are converted to sound, the delay time involved can be used
    as a type of store.

  - It worked as follows:
    - Imagine we have an 8-bit number. The number is input as a sequence of pulses
      where a pulse is 1 and the absence of a pulse is 0.
    - The pulses are converted into sound waves (using a transducer, as with a
      speaker) and sent through a mercury-filled tube.
    - The length of the tube is calculated so that it takes exactly the time of
      one machine cycle for a pulse to travel from one end to the other
    - At the other end of the tube, another transducer converts the sound waves
      back to electrical pulses
    - This operates in a loop: the pulses go back into the input end of the tube
      and the cycle runs again.

  To read the data, the computer would "listen" to the pulses coming out of the
  receiving end. To write new data, the computer would inject new pulses at the
  precise moment when it wants to change a '0' to a '1' or vice versa.

  For an 8-bit number there would be 8 pulses for each bit and it would be
  necessary to keep track of 8 specific positions in the delay line where each bit
  would be stored.

  The limitations were timing - difficult to keep track of each precise bit and
  when it would arrive at the output. Also it was sequential rather than random
  access. Not every bit could be accessed at once, you had to wait for its time
  slot to come around to access it.


path: zk/GPU_versus_CPU.md
sortable_path: zkGPU_versus_CPU.md
title: GPU versus CPU
checksum: 132c91acd7f4f95fe91448d7b07df376dc5f8028fee99951dcb411a17cec74ae
created: 2024-09-22 16:35:15.847621095+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"created":"Sunday, September 22, 2024","tags":["computer-architecture"]}
lead:
  - GPU stands for _Graphics Processing Unit_. Originally designed for rendering
    graphics and video but now used for other applications of parallel computing
    eg. machine learning.

body:
  - GPU stands for _Graphics Processing Unit_. Originally designed for rendering
    graphics and video but now used for other applications of parallel computing
    eg. machine learning.

  - The standard CPU uses sequential processing whereas the GPU is designed for
    parallel processing, specialising in performing many simple calculations
    simultaneously.

  - Harnessing the GPU requires specialised programming approaches.

  - Most standard laptops and desktops (if they are not designed for
    graphically-intensive tasks) incorporate graphics processing within the CPU.
    The CPU chip is on the same chip as the CPU chip. This is called _integrated_
    graphics processing.

  - Some laptops, especially those marketed for gaming or professional work, have
    a separate GPU chip. This is a more powerful graphics processor, separate from
    the CPU.

raw_content:
  ---
  tags: [computer-architecture]
  created: Sunday, September 22, 2024
  ---

  # GPU versus CPU

  - GPU stands for _Graphics Processing Unit_. Originally designed for rendering
    graphics and video but now used for other applications of parallel computing
    eg. machine learning.

  - The standard CPU uses sequential processing whereas the GPU is designed for
    parallel processing, specialising in performing many simple calculations
    simultaneously.

  - Harnessing the GPU requires specialised programming approaches.

  - Most standard laptops and desktops (if they are not designed for
    graphically-intensive tasks) incorporate graphics processing within the CPU.
    The CPU chip is on the same chip as the CPU chip. This is called _integrated_
    graphics processing.

  - Some laptops, especially those marketed for gaming or professional work, have
    a separate GPU chip. This is a more powerful graphics processor, separate from
    the CPU.


path: zk/Magnetic_tape.md
sortable_path: zkMagnetic_tape.md
title: Magnetic tape
checksum: 0b07c8e2462ce875eda2c00c8e49c806621d49ad7318bbd0ef1056ad3d26a6f1
created: 2024-09-23 06:05:40.653297315+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"created":"Monday, September 23, 2024","tags":["computer-history","storage"]}
lead:
  Used for data storage in early digital computers (broadly from the
  [UNIVAC](./The_History_of_Computing_Swade.md)) era until the mini-computer era.

body:
  Used for data storage in early digital computers (broadly from the
  [UNIVAC](./The_History_of_Computing_Swade.md)) era until the mini-computer era.

  While the UNIVAC was not the first to use magnetic tape, it popularised the
  method due to its success and high profile.

  ![Magnetic tape reader in IBM computer](../img/magnetic-tape-ibm.jpeg)

  It worked as follows. The tape was a long, narrow strip of plastic coated with a
  magnetic material. Data is recorded on the the tape by magnetising tiny
  particles on its surface. It was first used in the 1920s for audio recording.

  Data is accessed in a linear, sequential manner. To reach a specific piece of
  data, you often need to wind through all the preceding data

  It was used for long term storage rather than RAM memory due to its sequential
  nature. It stored data and program instructions that would be read into the
  machine's memory. Data was also written out to magenetic tape, in order to store
  and further process results.

  It could store more data than punched cards and its sequential nature
  accommodated batch processing well. However its sequential nature meant it was
  slow at retreiving specific pieces of data as it would have to cycle through all
  the values stored on the tape.

  ![Magnetic tape stored as casette](../img/magnetic-tape-disk.jpeg)

  It developed from open-reel tapes to cartridges and cassettes. It was superseded
  by hard disk drives and solid state drives.

raw_content:
  ---
  tags: [computer-history, storage]
  created: Monday, September 23, 2024
  ---

  # Magnetic tape

  Used for data storage in early digital computers (broadly from the
  [UNIVAC](./The_History_of_Computing_Swade.md)) era until the mini-computer era.

  While the UNIVAC was not the first to use magnetic tape, it popularised the
  method due to its success and high profile.

  ![Magnetic tape reader in IBM computer](../img/magnetic-tape-ibm.jpeg)

  It worked as follows. The tape was a long, narrow strip of plastic coated with a
  magnetic material. Data is recorded on the the tape by magnetising tiny
  particles on its surface. It was first used in the 1920s for audio recording.

  Data is accessed in a linear, sequential manner. To reach a specific piece of
  data, you often need to wind through all the preceding data

  It was used for long term storage rather than RAM memory due to its sequential
  nature. It stored data and program instructions that would be read into the
  machine's memory. Data was also written out to magenetic tape, in order to store
  and further process results.

  It could store more data than punched cards and its sequential nature
  accommodated batch processing well. However its sequential nature meant it was
  slow at retreiving specific pieces of data as it would have to cycle through all
  the values stored on the tape.

  ![Magnetic tape stored as casette](../img/magnetic-tape-disk.jpeg)

  It developed from open-reel tapes to cartridges and cassettes. It was superseded
  by hard disk drives and solid state drives.


path: zk/Magnetic_drum_memory.md
sortable_path: zkMagnetic_drum_memory.md
title: Magnetic drum memory
checksum: 8ca7a01d80bbcbdcb6dca744f5a122187aa786f3c6b523a3c86f86d5ae870f01
created: 2024-09-24 05:35:27.273338982+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"created":"Tuesday, September 24, 2024","tags":["computer-history","memory"]}
lead:
  Along with the [Williams_Tube](Williams_Tube_memory.md), another early approach
  to RAM used in 1950s-1960s era of computing.

body:
  Along with the [Williams_Tube](Williams_Tube_memory.md), another early approach
  to RAM used in 1950s-1960s era of computing.

  ![Magnetic drum](../img/magnetic-drum-memory.jpg)

  A magnetic drum was a metal cylinder coated with a magnetic material. Data was
  stored by magnetising small regions on the drum's surface. The drum would rotate
  at high speeds and read/write heads were positioned along the length of the drum
  to access data.

  Seen as a improvement on Williams Tubes and
  [delay line memory](Delay_line_memory.md) but superseded by magnetic core memory
  later. It's concept lived on in harddisk drives which became the dominant form
  of secondary storage.

  It had a larger capacity than the technologies that preceded it and it was also
  non-volatile - the data would remain intact when the power was turned off.

  It was used in the [IBM 650](The_History_of_Computing_Swade.md) (1953) and
  Ferranti Mark I (1951)

raw_content:
  ---
  tags: [computer-history, memory]
  created: Tuesday, September 24, 2024
  ---

  # Magnetic drum memory

  Along with the [Williams_Tube](Williams_Tube_memory.md), another early approach
  to RAM used in 1950s-1960s era of computing.

  ![Magnetic drum](../img/magnetic-drum-memory.jpg)

  A magnetic drum was a metal cylinder coated with a magnetic material. Data was
  stored by magnetising small regions on the drum's surface. The drum would rotate
  at high speeds and read/write heads were positioned along the length of the drum
  to access data.

  Seen as a improvement on Williams Tubes and
  [delay line memory](Delay_line_memory.md) but superseded by magnetic core memory
  later. It's concept lived on in harddisk drives which became the dominant form
  of secondary storage.

  It had a larger capacity than the technologies that preceded it and it was also
  non-volatile - the data would remain intact when the power was turned off.

  It was used in the [IBM 650](The_History_of_Computing_Swade.md) (1953) and
  Ferranti Mark I (1951)


path: zk/Williams_Tube_memory.md
sortable_path: zkWilliams_Tube_memory.md
title: Williams Tube memory
checksum: cafd2be17ab55aa9487b07d6eb6cc4b0f994a323b35a2654af5858c733326885
created: 2024-09-24 05:37:12.171282575+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"created":"Tuesday, September 17, 2024","tags":["computer-history","memory"]}
lead:
  - One of the early attempts to create RAM in computers based on the EDVAC
    stored-program architecture.

body:
  - One of the early attempts to create RAM in computers based on the EDVAC
    stored-program architecture.

  - Worked by displaying a grid of dots on a cathode-ray tube ("screen")

  - Due to the way CRTs work, this creates a small charge of static electricity
    over each dot. The charge at the location of each of the dots is read by a
    thin metal sheet just in front of the display.

  - Each dot position could be written to and read from and the pattern was
    constantly refreshed as the dots would fade over time.

  ![A Williams CRT tube](../img/williams-tube.jpg)

  ![Memory dot pattern from a Williams Tube](../img/williams-tube-dots.jpg)

raw_content:
  ---
  tags: [computer-history, memory]
  created: Tuesday, September 17, 2024
  ---

  # Williams Tube memory

  - One of the early attempts to create RAM in computers based on the EDVAC
    stored-program architecture.

  - Worked by displaying a grid of dots on a cathode-ray tube ("screen")

  - Due to the way CRTs work, this creates a small charge of static electricity
    over each dot. The charge at the location of each of the dots is read by a
    thin metal sheet just in front of the display.

  - Each dot position could be written to and read from and the pattern was
    constantly refreshed as the dots would fade over time.

  ![A Williams CRT tube](../img/williams-tube.jpg)

  ![Memory dot pattern from a Williams Tube](../img/williams-tube-dots.jpg)


path: zk/Magnetic_core_memory.md
sortable_path: zkMagnetic_core_memory.md
title: Magnetic core memory
lead: ![](../img/magnetic-core-memory.jpg)
checksum: 946a0b8cf1656b420be858bad9e76ca1c4974a70b88e00088d16b74e3055d3c4
created: 2024-09-27 06:04:33.788716527+00:00
modified: 2024-10-18 18:16:37.832744547+00:00
metadata: {"created":"Friday, September 27, 2024","tags":["computer-history","memory"]}
body:
  ![](../img/magnetic-core-memory.jpg)

  Like [Magnetic_drum_memory](Magnetic_drum_memory.md), magnetic core memory was
  faster and more reliable than [delay_line_memory](Delay_line_memory.md) and
  vacuum-tubes, and was also persistent. Thus it was used for volatile memory as
  well as storage.

  It worked as follows. Magnetic beads made of ferrite and threaded with copper
  were arranged in a crisscross grid. A bead would be placed at the overlap of two
  copper threads. This constitutes a "core".A pulse of electric current would
  magnetise/de-magnetise the cores.

  ![](../img/magnetic-core-closeup.jpg)

raw_content:
  ---
  tags: [computer-history, memory]
  created: Friday, September 27, 2024
  ---

  # Magnetic core memory

  ![](../img/magnetic-core-memory.jpg)

  Like [Magnetic_drum_memory](Magnetic_drum_memory.md), magnetic core memory was
  faster and more reliable than [delay_line_memory](Delay_line_memory.md) and
  vacuum-tubes, and was also persistent. Thus it was used for volatile memory as
  well as storage.

  It worked as follows. Magnetic beads made of ferrite and threaded with copper
  were arranged in a crisscross grid. A bead would be placed at the overlap of two
  copper threads. This constitutes a "core".A pulse of electric current would
  magnetise/de-magnetise the cores.

  ![](../img/magnetic-core-closeup.jpg)


path: zk/Mainframe_computers.md
sortable_path: zkMainframe_computers.md
title: Mainframe computers
checksum: 75b912ea66c5f6b06475187d504b2c873b6d4f161c78d8aa54a9935bf2a75545
created: 2024-09-27 12:43:45.801890468+00:00
modified: 2024-10-18 18:16:47.869599104+00:00
metadata: {"created":"Friday, September 27, 2024","tags":["computer-history"]}
lead:
  A mainframe computer, also known as a "big iron" is a high performance computer
  used for resource-intensive tasks that require greater security and availability
  than smaller scale machines. They are typically used for centralised rather than
  distributed computing.

body:
  A mainframe computer, also known as a "big iron" is a high performance computer
  used for resource-intensive tasks that require greater security and availability
  than smaller scale machines. They are typically used for centralised rather than
  distributed computing.

  The heyday of mainframes was before the mini- and micro-computer revolution but
  they are still used.

  IBM remains the main manufacturer and service provider for mainframes. Example
  use cases include banks who need to process large volumes of transatction and
  high frequency tranding; government departments, the military and internal
  revenue who require huge databases.

raw_content:
  ---
  tags: [computer-history]
  created: Friday, September 27, 2024
  ---

  # Mainframe computers

  A mainframe computer, also known as a "big iron" is a high performance computer
  used for resource-intensive tasks that require greater security and availability
  than smaller scale machines. They are typically used for centralised rather than
  distributed computing.

  The heyday of mainframes was before the mini- and micro-computer revolution but
  they are still used.

  IBM remains the main manufacturer and service provider for mainframes. Example
  use cases include banks who need to process large volumes of transatction and
  high frequency tranding; government departments, the military and internal
  revenue who require huge databases.


path: zk/Compile_Python_app_to_single_executable.md
sortable_path: zkCompile_Python_app_to_single_executable.md
title: Compile Python app to single executable
lead: Assume you are already using a Pip venv.
checksum: b37e6fed8544630f8bc355e7d3d16baeca84cee7f64713fefac23499634281f2
created: 2024-10-19 11:53:04.194129309+00:00
modified: 2024-10-19 11:56:50.287481603+00:00
metadata: {"created":"Saturday, October 19, 2024","tags":["python","procedural"]}
body:
  Assume you are already using a Pip venv.

  Install `pyinstaller` and/or add it to your `requirements.txt`:

  ```sh
  souce venv/bin/activate
  pip install pyinstaller
  pip install -r requirements.txt
  ```

  Then from root of project:

  ```sh
  pyinstall -F src/app.py
  ```

  The `-F` flag compiles the application to a single file containing all necessary
  external packages and local imports.

  It outpute to `/dist`. Say the app is called `hello`, run the executable thus:

  ```sh
  /my-python-application/dist/app
  ```

raw_content:
  ---
  tags: [python, procedural]
  created: Saturday, October 19, 2024
  ---

  # Compile Python app to single executable

  Assume you are already using a Pip venv.

  Install `pyinstaller` and/or add it to your `requirements.txt`:

  ```sh
  souce venv/bin/activate
  pip install pyinstaller
  pip install -r requirements.txt
  ```

  Then from root of project:

  ```sh
  pyinstall -F src/app.py
  ```

  The `-F` flag compiles the application to a single file containing all necessary
  external packages and local imports.

  It outpute to `/dist`. Say the app is called `hello`, run the executable thus:

  ```sh
  /my-python-application/dist/app
  ```


path: zk/Creating_a_RESTful_API_GET.md
sortable_path: zkCreating_a_RESTful_API_GET.md
title: Creating a RESTful API: GET requests
lead: With our GET request we will simply return the array of course objects.
checksum: 007813002026a9ba320d6f0e716d1694c74c92c292f5e3c5718b9fce78ee5417
created: 2024-10-21 13:16:36.089118393+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"tags":["node-js","REST","APIs"]}
body:
  With our GET request we will simply return the array of course objects.

  We create an [event emitter](Events%20module.md#event-emitters) and listener
  that listens for GET requests on a specified port and sends data in response to
  requests.

  ```js
  // Return a value as response from specified URI
  router.get("/", (req, res) => {
    res.send(courses);
  });
  ```

  Our server is now set up:

  ![](/img/server-listening.png)

  > When creating our API this structure of creating handlers for specific routes
  > will be iterated. Every endpoint will be specified with
  > `[app].[http_request_type]` and followed by a callback.

  We can now call the endpoint:

  ```js
  const getAllCourses = async () => {
    try {
      const resp = await axios.get("http://localhost:3000/api/courses");
      console.log(resp.data);
    } catch (err) {
      console.error(err);
    }
  };

  getAllCourses();
  ```

  Returns:

  ```js
  [
    { id: 1, name: "First course" },
    { id: 2, name: "Second course" },
    { id: 3, name: "Third course" },
  ];
  ```

  ## Parameters

  The previous example serves the entire set of our data. But we will also need to
  retrieve specific values, we do this by adapting the GET callback to accept
  parameters. These parameters will correspond to the specific entry in our main
  data array.

  ```js
  router.get("/:id", (req, res) => {
    res.send(req.params.id);
  });
  ```

  We use the `:` symbol in the URI to indicate that we looking to parse for a
  specific value in the data. Now if we call `/api/courses/2`, we will get the
  second item in the array.

  The block above is the most basic format but we would want to add some kind of
  error handling, for example:

  ```js
  router.get("/:id", (req, res) => {
    const course = courses.find((c) => c.id === parseInt(req.params.id));
    if (!course) res.status(404).send("A course with the given ID was not found");
    res.send(course);
  });
  ```

  ## Queries

  Whereas parameters return specific data points, queries don't get data they
  aggregate or present the data that is returned in a certain way, such as for
  instance applying a search function. We indicate queries with a `?` in our URI.
  For example: `/api/posts/2018/1?sortBy=name`. To facilitate a request like this
  we have to create a GET path that allows for it:

  ```js
  router.get("/:year/:month", (req, res) => {
    res.send(req.query);
  })[]();
  ```

  We would get the following back:

  ```json
  {
    "sortBy": "name"
  }
  ```

  Again a JSON object with key-value pairs is returned.

raw_content:
  ---
  tags:
    - node-js
    - REST
    - APIs
  ---

  # Creating a RESTful API: `GET` requests

  With our GET request we will simply return the array of course objects.

  We create an [event emitter](Events%20module.md#event-emitters) and listener
  that listens for GET requests on a specified port and sends data in response to
  requests.

  ```js
  // Return a value as response from specified URI
  router.get("/", (req, res) => {
    res.send(courses);
  });
  ```

  Our server is now set up:

  ![](/img/server-listening.png)

  > When creating our API this structure of creating handlers for specific routes
  > will be iterated. Every endpoint will be specified with
  > `[app].[http_request_type]` and followed by a callback.

  We can now call the endpoint:

  ```js
  const getAllCourses = async () => {
    try {
      const resp = await axios.get("http://localhost:3000/api/courses");
      console.log(resp.data);
    } catch (err) {
      console.error(err);
    }
  };

  getAllCourses();
  ```

  Returns:

  ```js
  [
    { id: 1, name: "First course" },
    { id: 2, name: "Second course" },
    { id: 3, name: "Third course" },
  ];
  ```

  ## Parameters

  The previous example serves the entire set of our data. But we will also need to
  retrieve specific values, we do this by adapting the GET callback to accept
  parameters. These parameters will correspond to the specific entry in our main
  data array.

  ```js
  router.get("/:id", (req, res) => {
    res.send(req.params.id);
  });
  ```

  We use the `:` symbol in the URI to indicate that we looking to parse for a
  specific value in the data. Now if we call `/api/courses/2`, we will get the
  second item in the array.

  The block above is the most basic format but we would want to add some kind of
  error handling, for example:

  ```js
  router.get("/:id", (req, res) => {
    const course = courses.find((c) => c.id === parseInt(req.params.id));
    if (!course) res.status(404).send("A course with the given ID was not found");
    res.send(course);
  });
  ```

  ## Queries

  Whereas parameters return specific data points, queries don't get data they
  aggregate or present the data that is returned in a certain way, such as for
  instance applying a search function. We indicate queries with a `?` in our URI.
  For example: `/api/posts/2018/1?sortBy=name`. To facilitate a request like this
  we have to create a GET path that allows for it:

  ```js
  router.get("/:year/:month", (req, res) => {
    res.send(req.query);
  })[]();
  ```

  We would get the following back:

  ```json
  {
    "sortBy": "name"
  }
  ```

  Again a JSON object with key-value pairs is returned.


path: zk/Creating_a_RESTful_API_Integrating_the_Database.md
sortable_path: zkCreating_a_RESTful_API_Integrating_the_Database.md
title: Creating a RESTful API: Integrating the database
checksum: 538034760405536a2f4949cde906dd4398f0f43c0e8e59a51357aae3c5fe19f0
created: 2024-10-21 13:16:36.137028858+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"tags":["node-js","REST","APIs","mongo-db","databases"]}
lead:
  So far we have set up the application and an `/api/courses` route which handles
  requests for RESTful API operations on a local array of course objects. We now
  want to have the endpoints operate on a MongoDB `courses` collection rather than
  the array.

body:
  So far we have set up the application and an `/api/courses` route which handles
  requests for RESTful API operations on a local array of course objects. We now
  want to have the endpoints operate on a MongoDB `courses` collection rather than
  the array.

  ## Set-up

  We will follow the routine for establishing a MongoDB instance as detailed in
  [my notes](Connect_to_Mongo_database.md) on Mongo:

  - [Create MongoDB database](Create_Mongo_database.md)
  - [Connect to MongoDB database](Connect_to_Mongo_database.md)

  Our `index.js` now looks like the following:

  ```js
  // index.js

  // Connect to database
  mongoose
    .connect("mongodb://127.0.0.1/playground")
    .then(() => console.log("Connected to MongoDB"))
    .catch((err) => console.error(err));

  app.use(express.json());

  // Link to `courses` route which contains our REST request handlers for this part of the API
  app.use("/api/courses", courses);
  ```

  ## Integrating Mongo with our our `courses` module

  ### Create the schema

  Now we go the router module for `courses` and start to use Mongoose, defining
  our `Course` schema:

  ```diff
  // index.js

  // Connect to database
  mongoose
    .connect("mongodb://127.0.0.1/playground")
    .then(() => console.log("Connected to MongoDB"))
    .catch((err) => console.error(err));

  app.use(express.json());

  // Link to `courses` route which contains our REST request handlers for this part of the API
  app.use("/api/courses", courses);

  + const courseSchema = new mongoose.Schema({
  +  name: {type: String, required: true, minlength: 5, maxlength: 255},
  +  author: String,
  +  tags: [String],
  +  data: {type: Date, default: Date.now}, // if unspecified, entry will default to current date
  +  isPublished: Boolean,
  + });

  ```

  ### Create a model

  ```diff
  const courseSchema = new mongoose.Schema({
    name: {type: String, required: true, minlength: 5, maxlength: 255},
    author: String,
    tags: [String],
    data: {type: Date, default: Date.now}, // if unspecified, entry will default to current date
    isPublished: Boolean,
   });

  + const Course = new mongoose.model('Course', courseSchema);
  ```

  With this established we can remove our local array as we are ready to start
  getting our data from the database:

  ```diff
  const Course = mongoose.model('Course', courseSchema);

  - const courses = [
  -  {
  -    id: 1,
  -    name: "First course",
  -  },
  -  ...
  -];
  ```

  We could actually simplify the syntax here and combine our schema and model
  declaration into a single block:

  ```js
  const Course = mongoose.model(
    "Course",
    new mongoose.Schema({
      name: { type: String, required: true, minlength: 5, maxlength: 255 },
      author: String,
      tags: [String],
      data: { type: Date, default: Date.now }, // if unspecified, entry will default to current date
      isPublished: Boolean,
    })
  );
  ```

  > N.B In a real project we wouldn't keep our models in the same file as our
  > handlers. We would keep them in the dedicated `/models/` directory. We should
  > stick to the single responsibility principle and keep `/routes/` for API
  > handlers and `/model/` for schema declarations and models.

  ## Rewriting the REST handlers

  Now we need to rewrite our RESTful request handlers so that the data is sourced
  from and added to the database. We will mainly be using the Mongo syntax defined
  at [Querying a collection](Querying_a_Mongo_collection.md) and
  [Adding documents to a collection](Adding_documents_to_a_Mongo_collection.md).
  We will also keep API validation within the `/model/` file.

  ### GET

  Instead of simply returning the array, we use the Mongoose `find` method.

  ```diff
  - router.get("/", (req, res) => {
  -  res.send(courses);
   });

  + router.get("/", async (res, res) => {
  +  const courses = await Courses.find();
    res.send(courses)
  })
  ```

  ### POST

  Now we make our new course an instance of the `Courses` model:

  ```js
  // Original formulation

  router.post("/", (req, res) => {
  -  const course = {
  -    id: courses.length + 1,
  -    name: req.body.name,
  -  };
    courses.push(course);
    res.send(course);
  });
  ```

  ```diff
  router.post("/", async (req, res) => {
  +  let course = new Course({ // make new course instance of Course model
  -   id: courses.length + 1, // not needed as DB automatically adds an id
      name: req.body.name,
    });
  - courses.push(course); // not pushing to the array anymore
  + await course.save() // save to Mongo
    res.send(course);
  });

  ```

  ### PUT

  When updating a value in the database we are going to use the
  [query-first](Update_a_Mongo_document.md#query-first-document-update)
  approach to updating a Mongo document.

  ```jsconst courseSchema = new mongoose.Schema({
    name: {type: String, required: true, minlength: 5, maxlength: 255},
    author: String,
    tags: [String],
    data: {type: Date, default: Date.now}, // if unspecified, entry will default to current date
    isPublished: Boolean,
   });
  router.put("/:id", (req, res) => {
    const course = courses.find((c) => c.id === parseInt(req.params.id));

    if (!course)
      return res.status(404).send("A course with the given ID was not found");

    const { error } = validateCourse(req.body);
    if (error)
      return error.details.map((joiErr) => res.status(400).send(joiErr.message));
  const courseSchema = new mongoose.Schema({
    name: {type: String, required: true, minlength: 5, maxlength: 255},
    author: String,
    tags: [String],
    data: {type: Date, default: Date.now}, // if unspecified, entry will default to current date
    isPublished: Boolean,
   });
    course.name = req.body.name;
    res.send(course);
  });
  ```

  ```diff
  router.put("/:id", async (req, res) => {
  - const course = courses.find((c) => c.id === parseInt(req.params.id));
   const { error } = validateCourse(req.body);
   if (!course) return res.status(404).send("A course with the given ID was not found");
  + const updatedCourse = await Course.findByIdAndUpdate(req.params.id,
  +  { name: req.body.name },
  +  { new: true}
  + )

  -   if (error)
  -   return error.details.map((joiErr) => res.status(400).send(joiErr.message));
  - })

  - course.name = req.body.name;
    res.send(course);
  ```

  ### DELETE

  ```js
  router.delete("/:id", (req, res) => {
    const course = courses.find((c) => c.id === parseInt(req.params.id));
    if (!course)
      return res.status(404).send("A course with the given ID was not found");

    courses.indexOf(course);
    courses.splice(index, 1);
    res.send(course);
  });
  ```

  ```diff

  router.delete("/:id", async (req, res) => {
   const courseToDelete = await Course.findByIdAndRemove(req.params.id)

   if (!course) return res.status(404).send("A course with the given ID was not found");

  - courses.indexOf(course);
  - courses.splice(index, 1);

  res.send(course);
  })
  ```

raw_content:
  ---
  tags:
    - node-js
    - REST
    - APIs
    - mongo-db
    - databases
  ---

  # Creating a RESTful API: Integrating the database

  So far we have set up the application and an `/api/courses` route which handles
  requests for RESTful API operations on a local array of course objects. We now
  want to have the endpoints operate on a MongoDB `courses` collection rather than
  the array.

  ## Set-up

  We will follow the routine for establishing a MongoDB instance as detailed in
  [my notes](Connect_to_Mongo_database.md) on Mongo:

  - [Create MongoDB database](Create_Mongo_database.md)
  - [Connect to MongoDB database](Connect_to_Mongo_database.md)

  Our `index.js` now looks like the following:

  ```js
  // index.js

  // Connect to database
  mongoose
    .connect("mongodb://127.0.0.1/playground")
    .then(() => console.log("Connected to MongoDB"))
    .catch((err) => console.error(err));

  app.use(express.json());

  // Link to `courses` route which contains our REST request handlers for this part of the API
  app.use("/api/courses", courses);
  ```

  ## Integrating Mongo with our our `courses` module

  ### Create the schema

  Now we go the router module for `courses` and start to use Mongoose, defining
  our `Course` schema:

  ```diff
  // index.js

  // Connect to database
  mongoose
    .connect("mongodb://127.0.0.1/playground")
    .then(() => console.log("Connected to MongoDB"))
    .catch((err) => console.error(err));

  app.use(express.json());

  // Link to `courses` route which contains our REST request handlers for this part of the API
  app.use("/api/courses", courses);

  + const courseSchema = new mongoose.Schema({
  +  name: {type: String, required: true, minlength: 5, maxlength: 255},
  +  author: String,
  +  tags: [String],
  +  data: {type: Date, default: Date.now}, // if unspecified, entry will default to current date
  +  isPublished: Boolean,
  + });

  ```

  ### Create a model

  ```diff
  const courseSchema = new mongoose.Schema({
    name: {type: String, required: true, minlength: 5, maxlength: 255},
    author: String,
    tags: [String],
    data: {type: Date, default: Date.now}, // if unspecified, entry will default to current date
    isPublished: Boolean,
   });

  + const Course = new mongoose.model('Course', courseSchema);
  ```

  With this established we can remove our local array as we are ready to start
  getting our data from the database:

  ```diff
  const Course = mongoose.model('Course', courseSchema);

  - const courses = [
  -  {
  -    id: 1,
  -    name: "First course",
  -  },
  -  ...
  -];
  ```

  We could actually simplify the syntax here and combine our schema and model
  declaration into a single block:

  ```js
  const Course = mongoose.model(
    "Course",
    new mongoose.Schema({
      name: { type: String, required: true, minlength: 5, maxlength: 255 },
      author: String,
      tags: [String],
      data: { type: Date, default: Date.now }, // if unspecified, entry will default to current date
      isPublished: Boolean,
    })
  );
  ```

  > N.B In a real project we wouldn't keep our models in the same file as our
  > handlers. We would keep them in the dedicated `/models/` directory. We should
  > stick to the single responsibility principle and keep `/routes/` for API
  > handlers and `/model/` for schema declarations and models.

  ## Rewriting the REST handlers

  Now we need to rewrite our RESTful request handlers so that the data is sourced
  from and added to the database. We will mainly be using the Mongo syntax defined
  at [Querying a collection](Querying_a_Mongo_collection.md) and
  [Adding documents to a collection](Adding_documents_to_a_Mongo_collection.md).
  We will also keep API validation within the `/model/` file.

  ### GET

  Instead of simply returning the array, we use the Mongoose `find` method.

  ```diff
  - router.get("/", (req, res) => {
  -  res.send(courses);
   });

  + router.get("/", async (res, res) => {
  +  const courses = await Courses.find();
    res.send(courses)
  })
  ```

  ### POST

  Now we make our new course an instance of the `Courses` model:

  ```js
  // Original formulation

  router.post("/", (req, res) => {
  -  const course = {
  -    id: courses.length + 1,
  -    name: req.body.name,
  -  };
    courses.push(course);
    res.send(course);
  });
  ```

  ```diff
  router.post("/", async (req, res) => {
  +  let course = new Course({ // make new course instance of Course model
  -   id: courses.length + 1, // not needed as DB automatically adds an id
      name: req.body.name,
    });
  - courses.push(course); // not pushing to the array anymore
  + await course.save() // save to Mongo
    res.send(course);
  });

  ```

  ### PUT

  When updating a value in the database we are going to use the
  [query-first](Update_a_Mongo_document.md#query-first-document-update)
  approach to updating a Mongo document.

  ```jsconst courseSchema = new mongoose.Schema({
    name: {type: String, required: true, minlength: 5, maxlength: 255},
    author: String,
    tags: [String],
    data: {type: Date, default: Date.now}, // if unspecified, entry will default to current date
    isPublished: Boolean,
   });
  router.put("/:id", (req, res) => {
    const course = courses.find((c) => c.id === parseInt(req.params.id));

    if (!course)
      return res.status(404).send("A course with the given ID was not found");

    const { error } = validateCourse(req.body);
    if (error)
      return error.details.map((joiErr) => res.status(400).send(joiErr.message));
  const courseSchema = new mongoose.Schema({
    name: {type: String, required: true, minlength: 5, maxlength: 255},
    author: String,
    tags: [String],
    data: {type: Date, default: Date.now}, // if unspecified, entry will default to current date
    isPublished: Boolean,
   });
    course.name = req.body.name;
    res.send(course);
  });
  ```

  ```diff
  router.put("/:id", async (req, res) => {
  - const course = courses.find((c) => c.id === parseInt(req.params.id));
   const { error } = validateCourse(req.body);
   if (!course) return res.status(404).send("A course with the given ID was not found");
  + const updatedCourse = await Course.findByIdAndUpdate(req.params.id,
  +  { name: req.body.name },
  +  { new: true}
  + )

  -   if (error)
  -   return error.details.map((joiErr) => res.status(400).send(joiErr.message));
  - })

  - course.name = req.body.name;
    res.send(course);
  ```

  ### DELETE

  ```js
  router.delete("/:id", (req, res) => {
    const course = courses.find((c) => c.id === parseInt(req.params.id));
    if (!course)
      return res.status(404).send("A course with the given ID was not found");

    courses.indexOf(course);
    courses.splice(index, 1);
    res.send(course);
  });
  ```

  ```diff

  router.delete("/:id", async (req, res) => {
   const courseToDelete = await Course.findByIdAndRemove(req.params.id)

   if (!course) return res.status(404).send("A course with the given ID was not found");

  - courses.indexOf(course);
  - courses.splice(index, 1);

  res.send(course);
  })
  ```


path: zk/Creating_a_RESTful_API_Introduction.md
sortable_path: zkCreating_a_RESTful_API_Introduction.md
title: Creating a RESTful API: Introduction
lead:
  We are going to use Express to create a
  [RESTful API](RESTful_APIs.md) in Node.js.

checksum: 693829439b53406aa42623c1431213faf04b7d53ae8ad104884b5541fa45c38b
created: 2024-10-21 13:16:36.267132856+00:00
modified: 2024-10-18 17:55:56.302394226+00:00
metadata: {"tags":["node-js","REST","APIs"]}
body:
  We are going to use Express to create a
  [RESTful API](RESTful_APIs.md) in Node.js.

  ## Request types

  Express provides us with methods corresponding to each of the
  [HTTP request types](HTTP_request_types.md):

  - `.get()`
  - `.post()`
  - `.put()`
  - `.delete()`

  ## Our data set

  > Typically when you create a RESTful API you are going to be returning data
  > from a database. For simplicity we are just going simulate this with a simple
  > data array so that we can focus on the Express syntax rather than database
  > handling. Later we will integrate this with a
  > [MongoDB database](05_%20Integrating_the_database.md).

  We will mainly work with the following array of objects:

  ```js
  const courses = [
    {
      id: 1,
      name: "First course",
    },
    {
      id: 2,
      name: "Second course",
    },
    {
      id: 3,
      name: "Third course",
    },
  ];
  ```

  ## Creating an Express instance

  We first create an instance of Express within `index.js`. This will be the main
  coordinating file and we will aim to minimise the amount of business logic we
  have in this file. It should really just be for initialization and managing
  [middleware](Middleware_in_NodeJS.md).

  ```js
  const express = require("express");
  const app = express();
  ```

  ## Routing

  We are going to receive all our HTTP requests at the path `/api/courses` and we
  will place all code related to this route in a dedicated Node module
  (`routes/courses.js`) rather than within `index.js` which should remain as
  uncluttered as possible.

  First we need to declare this in `index.js`:

  ```js
  const courses = require("./routes/courses");
  app.use("/api/courses", courses);
  ```

  Then within `courses.js` we instantiate an express router that `app.js` will
  route to:

  ```js
  const router = express.Router();
  ```

  Our REST requests will all follow the following pattern:

  ```js
  router.get("/", (req, res) => {
    res.send(courses);
  });
  ```

  We target the `Router` instance we created and all paths will be expressed as
  `/` since the app will route by default to `/courses/api` from `index.js`

  ## Create listener

  With the routing established and the Express instance created we can now listen
  for requests:

  ```js
  app.listen(3000, () => console.log("Listening on port 30000..."));
  ```

  ## REST endpoints

  We can now proceed to set up our RESTful endpoints:

  [GET requests](1_GET.md)

  [POST requests](2_POST.md)

  [PUT requests](3_PUT.md)

  [DELETE requests](4_DELETE.md)

raw_content:
  ---
  tags:
    - node-js
    - REST
    - APIs
  ---

  # Creating a RESTful API: Introduction

  We are going to use Express to create a
  [RESTful API](RESTful_APIs.md) in Node.js.

  ## Request types

  Express provides us with methods corresponding to each of the
  [HTTP request types](HTTP_request_types.md):

  - `.get()`
  - `.post()`
  - `.put()`
  - `.delete()`

  ## Our data set

  > Typically when you create a RESTful API you are going to be returning data
  > from a database. For simplicity we are just going simulate this with a simple
  > data array so that we can focus on the Express syntax rather than database
  > handling. Later we will integrate this with a
  > [MongoDB database](05_%20Integrating_the_database.md).

  We will mainly work with the following array of objects:

  ```js
  const courses = [
    {
      id: 1,
      name: "First course",
    },
    {
      id: 2,
      name: "Second course",
    },
    {
      id: 3,
      name: "Third course",
    },
  ];
  ```

  ## Creating an Express instance

  We first create an instance of Express within `index.js`. This will be the main
  coordinating file and we will aim to minimise the amount of business logic we
  have in this file. It should really just be for initialization and managing
  [middleware](Middleware_in_NodeJS.md).

  ```js
  const express = require("express");
  const app = express();
  ```

  ## Routing

  We are going to receive all our HTTP requests at the path `/api/courses` and we
  will place all code related to this route in a dedicated Node module
  (`routes/courses.js`) rather than within `index.js` which should remain as
  uncluttered as possible.

  First we need to declare this in `index.js`:

  ```js
  const courses = require("./routes/courses");
  app.use("/api/courses", courses);
  ```

  Then within `courses.js` we instantiate an express router that `app.js` will
  route to:

  ```js
  const router = express.Router();
  ```

  Our REST requests will all follow the following pattern:

  ```js
  router.get("/", (req, res) => {
    res.send(courses);
  });
  ```

  We target the `Router` instance we created and all paths will be expressed as
  `/` since the app will route by default to `/courses/api` from `index.js`

  ## Create listener

  With the routing established and the Express instance created we can now listen
  for requests:

  ```js
  app.listen(3000, () => console.log("Listening on port 30000..."));
  ```

  ## REST endpoints

  We can now proceed to set up our RESTful endpoints:

  [GET requests](1_GET.md)

  [POST requests](2_POST.md)

  [PUT requests](3_PUT.md)

  [DELETE requests](4_DELETE.md)


path: zk/Creating_a_RESTful_API_POST.md
sortable_path: zkCreating_a_RESTful_API_POST.md
title: Creating a RESTful API: POST requests
checksum: 88ee1290e5a804acf0e89181b6a40a96465ee9a05779f69abc73419210ab6340
created: 2024-10-21 13:16:36.449624595+00:00
modified: 2024-10-18 17:55:56.299060832+00:00
metadata: {"tags":["node-js","REST","APIs"]}
lead:
  To demonstrate the handling of POST requests, we will create a handler that add
  a new element to the array of courses.

body:
  To demonstrate the handling of POST requests, we will create a handler that add
  a new element to the array of courses.

  ```js
  router.post("/", (req, res) => {
    const course = {
      id: courses.length + 1,
      name: req.body.name,
    };
    courses.push(course);
    res.send(course);
  });
  ```

  Here we use the body that is sent from the client and isolate the field `name`.
  This presupposes that the client is sending us data with the following shape as
  the body:

  ```json
  {
    "name": "Biology and Life Sciences"
  }
  ```

  To execute the PUT request from the frontend:

  ```js
  const addCourse = async (newCourse) => {
    try {
      const resp = await axios.post("http://localhost:3000/api/courses", {
        name: newCourse,
      });
      console.log(resp.data);
    } catch (err) {
      console.error(err);
    }
  };
  addCourse("Biology and Life Sciences");
  ```

  Which returns:

  ```js
  { id: 4, name: 'Biology and Life Sciences' }
  ```

  The `id` is added by the server, not the client. Having created the new value we
  add it to our `courses` array. (In reality we would be creating a new entry in a
  database.) Then we follow the convention of returning the new value back to the
  client.

  ## Validation

  We should accept alterations to the database that are not first validated. We
  can use the
  [Joi validator](Validation_in_NodeJS.md) to vet
  the request:

  ```js
  function validateCourse(course) {
    const schema = Joi.object({
      name: Joi.string().min(3).required(),
    });

    const { error } = schema.validate(course);
    return error;
  }
  ```

  We can then add the validation as part of our general error handling:

  ```js
  router.post("/", (req, res) => {
    const course = {
      id: courses.length + 1,
      name: req.body.name,
    };
    const { error } = schema.validate(req.body);
    if (error)
      return error.details.map((joiErr) => res.status(400).send(joiErr.message));

    courses.push(course);
    res.send(course);
  });
  ```

raw_content:
  ---
  tags:
    - node-js
    - REST
    - APIs
  ---

  # Creating a RESTful API: `POST` requests

  To demonstrate the handling of POST requests, we will create a handler that add
  a new element to the array of courses.

  ```js
  router.post("/", (req, res) => {
    const course = {
      id: courses.length + 1,
      name: req.body.name,
    };
    courses.push(course);
    res.send(course);
  });
  ```

  Here we use the body that is sent from the client and isolate the field `name`.
  This presupposes that the client is sending us data with the following shape as
  the body:

  ```json
  {
    "name": "Biology and Life Sciences"
  }
  ```

  To execute the PUT request from the frontend:

  ```js
  const addCourse = async (newCourse) => {
    try {
      const resp = await axios.post("http://localhost:3000/api/courses", {
        name: newCourse,
      });
      console.log(resp.data);
    } catch (err) {
      console.error(err);
    }
  };
  addCourse("Biology and Life Sciences");
  ```

  Which returns:

  ```js
  { id: 4, name: 'Biology and Life Sciences' }
  ```

  The `id` is added by the server, not the client. Having created the new value we
  add it to our `courses` array. (In reality we would be creating a new entry in a
  database.) Then we follow the convention of returning the new value back to the
  client.

  ## Validation

  We should accept alterations to the database that are not first validated. We
  can use the
  [Joi validator](Validation_in_NodeJS.md) to vet
  the request:

  ```js
  function validateCourse(course) {
    const schema = Joi.object({
      name: Joi.string().min(3).required(),
    });

    const { error } = schema.validate(course);
    return error;
  }
  ```

  We can then add the validation as part of our general error handling:

  ```js
  router.post("/", (req, res) => {
    const course = {
      id: courses.length + 1,
      name: req.body.name,
    };
    const { error } = schema.validate(req.body);
    if (error)
      return error.details.map((joiErr) => res.status(400).send(joiErr.message));

    courses.push(course);
    res.send(course);
  });
  ```


path: zk/Creating_a_RESTful_API_PUT.md
sortable_path: zkCreating_a_RESTful_API_PUT.md
title: Creating a RESTful API: PUT requests
checksum: 272bbcfe9b28d7a73070ac46a3a75f0d862d795d04e50770ca77f399f46b7365
created: 2024-10-21 13:16:36.48825746+00:00
modified: 2024-10-18 17:55:56.295727437+00:00
metadata: {"tags":["node-js","REST","APIs"]}
lead:
  To demonstrate the handling of PUT requests, we will create a handler that
  updates an element in the course array, based on its `id` and return the updated
  entry:

body:
  To demonstrate the handling of PUT requests, we will create a handler that
  updates an element in the course array, based on its `id` and return the updated
  entry:

  ```js
  router.put("/:id", (req, res) => {
    const course = courses.find((c) => c.id === parseInt(req.params.id));

    if (!course)
      return res.status(404).send("A course with the given ID was not found");

    const { error } = validateCourse(req.body);

    if (error)
      return error.details.map((joiErr) => res.status(400).send(joiErr.message));

    course.name = req.body.name;
    res.send(course);
  });
  ```

  Our request:

  ```js
  const updateCourse = async (courseChange) => {
    try {
      const resp = await axios.put("http://localhost:3000/api/courses/1", {
        name: courseChange.name,
      });
      console.log(resp.data);
    } catch (err) {
      console.error(err);
    }
  };

  updateCourse({ name: "A new course" });
  ```

  Returns:

  ```js
  { id: 1, name: 'A new course' }
  ```

raw_content:
  ---
  tags:
    - node-js
    - REST
    - APIs
  ---

  # Creating a RESTful API: `PUT` requests

  To demonstrate the handling of PUT requests, we will create a handler that
  updates an element in the course array, based on its `id` and return the updated
  entry:

  ```js
  router.put("/:id", (req, res) => {
    const course = courses.find((c) => c.id === parseInt(req.params.id));

    if (!course)
      return res.status(404).send("A course with the given ID was not found");

    const { error } = validateCourse(req.body);

    if (error)
      return error.details.map((joiErr) => res.status(400).send(joiErr.message));

    course.name = req.body.name;
    res.send(course);
  });
  ```

  Our request:

  ```js
  const updateCourse = async (courseChange) => {
    try {
      const resp = await axios.put("http://localhost:3000/api/courses/1", {
        name: courseChange.name,
      });
      console.log(resp.data);
    } catch (err) {
      console.error(err);
    }
  };

  updateCourse({ name: "A new course" });
  ```

  Returns:

  ```js
  { id: 1, name: 'A new course' }
  ```


path: neuron/4e98d18e-1a79-4e78-a4ac-f7e3a2de9887/861cc26e_5f59_4615_821d_fe3ee5e1014d.md
sortable_path: neuron4e98d18e-1a79-4e78-a4ac-f7e3a2de9887861cc26e_5f59_4615_821d_fe3ee5e1014d.md
title: 861cc26e-5f59-4615-821d-fe3ee5e1014d
checksum: 4048137d16c3d07669399b16b3bd66f03186baae560838f8773a7270dfbb0bce
created: 2024-10-22 05:37:47.387622024+00:00
modified: 2024-10-21 19:43:58.61116373+00:00
metadata: {"created":"Monday, October 14, 2024","tags":["synthesis","computer-history","ARPA"]}
lead:
  After Sputnik, Eisenhower wanted a dedicated, cross-departmental division within
  the US government to focus on advanced technological research so that they would
  not be caught out again. ARPA was the result.

body:
  After Sputnik, Eisenhower wanted a dedicated, cross-departmental division within
  the US government to focus on advanced technological research so that they would
  not be caught out again. ARPA was the result.

  Originally under the control of the Department of Defence (Pentagon).

  Once projects reached maturity, they would be handed over to the armed forces.

  It's name switched back and forth between ARPA/DARPA depending on congressional
  factors and funding. The switch to DARPA (and the associated re-focusing on
  military applications) indirectly precipitated a brain-drain in the late 1970s
  which contributed to the birth of the personal computer around the same time.

  Main technologies it contributed to or invented include the #ARPANET,
  time-sharing and GPS systems.

raw_content:
  ---
  tags: [synthesis, computer-history, ARPA]
  created: Monday, October 14, 2024
  ---

  # 861cc26e-5f59-4615-821d-fe3ee5e1014d

  After Sputnik, Eisenhower wanted a dedicated, cross-departmental division within
  the US government to focus on advanced technological research so that they would
  not be caught out again. ARPA was the result.

  Originally under the control of the Department of Defence (Pentagon).

  Once projects reached maturity, they would be handed over to the armed forces.

  It's name switched back and forth between ARPA/DARPA depending on congressional
  factors and funding. The switch to DARPA (and the associated re-focusing on
  military applications) indirectly precipitated a brain-drain in the late 1970s
  which contributed to the birth of the personal computer around the same time.

  Main technologies it contributed to or invented include the #ARPANET,
  time-sharing and GPS systems.


path: neuron/4e98d18e-1a79-4e78-a4ac-f7e3a2de9887/API_Gateway.md
sortable_path: neuron4e98d18e-1a79-4e78-a4ac-f7e3a2de9887API_Gateway.md
title: AWS API Gateway
checksum: c316c94f4f8bf3331eb058cb97d1ebc9901acecdd4694a32767ebbbfec5b4944
created: 2024-10-22 05:37:47.402999793+00:00
modified: 2024-10-21 22:11:47.065313176+00:00
metadata: {"tags":["AWS","APIs"]}
lead:
  We can use Gateway as the front-door to our application. It will create an
  [HTTP](HTTP_request_types.md) endpoint that you can call from a
  client. In response to a client request you can then call a
  [lambda function](Lambda_handler_function.md) that
  executes a backend process.

body:
  We can use Gateway as the front-door to our application. It will create an
  [HTTP](HTTP_request_types.md) endpoint that you can call from a
  client. In response to a client request you can then call a
  [lambda function](Lambda_handler_function.md) that
  executes a backend process.

  ![](static/gateway-services.png)

  See
  [using API Gateway as Lambda trigger](Practical_walkthrough_Lambda_creation_within_AWS.md)
  for a basic example of usage.

raw_content:
  ---
  tags: [AWS, APIs]
  ---

  # AWS API Gateway

  We can use Gateway as the front-door to our application. It will create an
  [HTTP](HTTP_request_types.md) endpoint that you can call from a
  client. In response to a client request you can then call a
  [lambda function](Lambda_handler_function.md) that
  executes a backend process.

  ![](static/gateway-services.png)

  See
  [using API Gateway as Lambda trigger](Practical_walkthrough_Lambda_creation_within_AWS.md)
  for a basic example of usage.


path: neuron/4e98d18e-1a79-4e78-a4ac-f7e3a2de9887/Adding_documents_to_a_Mongo_collection.md
sortable_path: neuron4e98d18e-1a79-4e78-a4ac-f7e3a2de9887Adding_documents_to_a_Mongo_collection.md
title: Adding documents to a collection
checksum: 0d9a6a29a4effc5fe4cbf5997c7864cbb6b47758bdb5b448c67a272e21fabf78
created: 2024-10-22 05:37:47.717823974+00:00
modified: 2024-10-21 22:11:47.055313007+00:00
metadata: {"tags":["mongo-db","node-js","mongoose","databases"]}
lead:
  We have our database (`playground`) and collection (`courses`) established. We
  now need to add documents to our collection. We will do this via a function
  since this will be an asynchronous process:

body:
  We have our database (`playground`) and collection (`courses`) established. We
  now need to add documents to our collection. We will do this via a function
  since this will be an asynchronous process:

  ```js
  const pythonCourse = new Course({
    name: "Python Course",
    author: "Terry Ogleton",
    tags: ["python", "backend"],
    isPublished: true,
  });

  async function addCourseDocToDb(courseDocument) {
    try {
      const result = await courseDocument.save();
      console.log(result);
    } catch (err) {
      console.error(err.message);
    }
  }

  addCourseDocToDb(nodeCourse);
  ```

  When we run this, we call the `save` method on the Mongoose schema. We will then
  have the Mongo document outputted to the console:

  ```
  {
    name: 'Python Course',
    author: 'Terry Ogleton',
    tags: [ 'python' ],
    isPublished: true,
    _id: new ObjectId("62f4ac989d2fec2f01596b9b"),
    date: 2022-08-11T07:15:36.978Z,
    __v: 0
  }
  ```

  This will also be reflected in Compass:

  ![](static/mongo-doc-added.png)

raw_content:
  ---
  tags:
    - mongo-db
    - node-js
    - mongoose
    - databases
  ---

  # Adding documents to a collection

  We have our database (`playground`) and collection (`courses`) established. We
  now need to add documents to our collection. We will do this via a function
  since this will be an asynchronous process:

  ```js
  const pythonCourse = new Course({
    name: "Python Course",
    author: "Terry Ogleton",
    tags: ["python", "backend"],
    isPublished: true,
  });

  async function addCourseDocToDb(courseDocument) {
    try {
      const result = await courseDocument.save();
      console.log(result);
    } catch (err) {
      console.error(err.message);
    }
  }

  addCourseDocToDb(nodeCourse);
  ```

  When we run this, we call the `save` method on the Mongoose schema. We will then
  have the Mongo document outputted to the console:

  ```
  {
    name: 'Python Course',
    author: 'Terry Ogleton',
    tags: [ 'python' ],
    isPublished: true,
    _id: new ObjectId("62f4ac989d2fec2f01596b9b"),
    date: 2022-08-11T07:15:36.978Z,
    __v: 0
  }
  ```

  This will also be reflected in Compass:

  ![](static/mongo-doc-added.png)


path: neuron/4e98d18e-1a79-4e78-a4ac-f7e3a2de9887/Algorithmic_complexity.md
sortable_path: neuron4e98d18e-1a79-4e78-a4ac-f7e3a2de9887Algorithmic_complexity.md
title: Algorithmic complexity
checksum: 86ea1fe66eaac1b003462ff1509fa6a1ca2d689719deadb154fd0c15972fe4a2
created: 2024-10-22 05:37:47.786508447+00:00
modified: 2024-10-21 22:11:47.051979618+00:00
metadata: {"tags":["algorithms"]}
lead:
  ![Screenshot_2021-05-11_at_18.55.23.png](static/Screenshot_2021-05-11_at_18.55.23.png)
  _Summary of the main classes of algorithmic complexity_

body:
  ![Screenshot_2021-05-11_at_18.55.23.png](static/Screenshot_2021-05-11_at_18.55.23.png)
  _Summary of the main classes of algorithmic complexity_

  ## Distinguish algorithms from programs

  Test commit Algorithms are general sets of instructions that take data in one
  state, follow a prescribed series of steps and return data in another state.
  Programs are a specific application of one or more algorithms to achieve an
  outcome in a specific context. With algorithms, the actual detail of the steps
  is mostly abstracted and it is irrelevant to what end the algorithm is being
  put. For instance you may create a program that returns addresses from a
  database using a postcode. It is irrelevant to the efficiency or value of the
  algorithm whether or not you are looking up postcodes or some other form of
  alphanumeric string.

  ## Algorithmic efficiency

  Algorithms can be classified based on their efficiency. Efficiency is function
  of the runtime speed. However this doesn't always mean that the fastest
  algorithms are best.

  If we are landing the Curiosity Rover on Mars we may choose an algorithm that is
  slower on average for a guarantee that it will never take longer than we find
  acceptable. In other cases for example a video game, we may choose an algorithm
  that keeps the average time down, even if this occasionally leads to processes
  that need to be aborted because they take too long.

  We need a generalised measure of efficiency to compare algorithms, across
  variant hardware. We can't simply use the number of steps, since some steps will
  be quicker to complete than others in the course of the overall algorithm and
  may take longer on different machines. Moreover the same algorithm could run at
  different speeds on the same machine, depending on its internal state at the
  given time that it ran. So we use the following: **the number of steps required
  relative to the input.**

  > Two given computers may differ in how quickly they can run an algorithm
  > depending on clock speed, available memory and so forth. They will however
  > tend to require approximately the same number of instructions and we can
  > measure the rate at which the number of instructions increases with the
  > problem size.

  This is what **asymptotic runtime** means: the rate at which the runtime of an
  algorithm grows compared to the size of its input. For precision and accuracy we
  use the worst case scenario as the benchmark.

  So: the efficiency of algorithm _A_ can be judged relative to the efficiency of
  algorithm _B_ based on the rate at which the runtime of _A_ grows compared to
  its input, compared to the same property in _B_, assuming the worst possible
  performance.

  From now on we will use the word 'input' to denote the data that the algorithm
  receives (in most cases we will envision this as an array containing a certain
  data type) and 'execution' to denote the computation that is applied by the
  algorithm to each item of the data input. Rephrasing the above with these terms
  we can say that 'algorithmic efficiency' is a measure that describes the rate at
  which the execution time of an algorithm increases relative to the size of its
  input.

  We will find that for the runtime of some algorithms, the size of the input does
  not change the execution time. In these cases, the runtime is proportional to
  the input quantity. In this case, regardless of whether the input is an array of
  one hundred elements or an array of ten elements, the amount of work that is
  executed on each element is the same.

  For other cases, this will not hold true. We will find that there is a
  relationship between input size and execution time such that the length of the
  input affects the amount of work that needs to be performed on each item at
  execution.

  ## Linear time

  Let's start with linear time, which is the easiest runtime to grasp.

  We need an example to make this tangible and show how an algorithm's runtime
  changes compared to the size of its input. Let's take a simple function that
  takes a sequence of integers and returns their sum:

  ```js
  function findSum(arr){
  	let total = 0;
  	for (let i = 0; i < arr.length; i++){
  		total = total += arr[i];
  	)
  	return total
  }
  ```

  The input of this function is an array of integers. It returns their sum as the
  output. Let's say that it takes 1ms for the function to sum an array of two
  integers.

  If we passed in an array of four integers, how would this change the runtime?
  The answer is that, providing that the time it takes to sum two integers doesn't
  change, it would take twice as long.

  As the time it takes to execute `findSum` doesn't change, we can say confidently
  that the runtime is as long as the number of integers we pass in.

  A more general way to say this is that the runtime is equal to size of the
  input. For algorithms of the class of which `findSum` is a member: **the total
  runtime is proportional to the number of items to be processed**.

  ## Introducing asymptotic notation

  If we say that it takes 1ms for two integers to be summed, this gives us the
  following data set:

  | Length of input | Runtime |
  | :-------------- | ------: |
  | 2               |       2 |
  | 3               |       3 |
  | 4               |       4 |
  | 5               |       5 |

  If we plotted this as a graph it is clear that this is equivalent to a linear
  distribution:![lin.svg](static/lin.svg)

  Algorithms which display this distribution are therefore called **linear
  algorithms**.

  The crucial point is that the amount of time it takes to sum the integers does
  not increase as the algorithm proceeds and the input size grows. This time
  remains the same. If it did increase, we would have a fluctuating curve on the
  graph. This aspect remains constant, only the instructions increase. This is why
  we have a nice steadily-advancing distribution in the graph.

  We can now introduce notation to formalise the algorithmic properties we have
  been discussing.

  ## Big O notation

  To express linear time algorithms formally, we say that:

  > it takes some constant amount of time ($C$) to sum one integer and n times as
  > long to sum n integers

  Here the constant is the time for each execution to run and n is the length of
  the input. Thus the complexity is equal to that time multiplied by the input.

  The algebraic expression of this is $cn$ : the constant multiplied by the length
  of the input. In algorithmic notation, the reference to the constant is always
  removed. Instead we just use n and combine it with a 'big O' which stands for
  'order of complexity'. Likewise, if we have an array of four integers being
  passed to `findSum` we could technically express it as O(4n), but we don't
  because we are interested in the general case not the specific details of the
  runtime. So a linear algorithm is expressed algebraically as $O(n)$ which is
  read as "oh of n" and means

  > $O(n)$ = with an order of complexity equal to (some constant) multiplied by n

  Applied, this means an input of length 6 ($n$) where runtime is constant ($c$)
  at 1ms has a total runtime of 6 x 1 = 6ms in total. Exactly the same as our
  table and graph. O n is just a mathematical way of saying _the runtime grows on
  the order of the size of the input._

  > It's really important to remember that when we talk about the execution
  > runtime being constant at 1ms, this is just an arbitrary placeholder. We are
  > not really bothered about whether it's 1ms or 100ms: 'constant' in the
  > mathematical sense doesn't mean a unit of time, it means 'unchanging'. We are
  > using 1ms to get traction on this concept but the fundamental point being
  > expressed is that the size of the input doesn't affect the execution time
  > across the length of the execution time.

  ## Constant time

  Constant time is another one of the main classes of algorithmic complexity. It
  is expressed as O(1). Here, we do away with n because with constant time we are
  only ever dealing with a single execution so we don't need a variable to express
  nth in a series or 'more than one'. Constant time covers all singular processes,
  without iteration.

  An example in practice would be printing `array[0]` . Regardless of the size of
  the array, it is only ever going to take one step, or constant times one. On a
  graph this is equivalent to a flat line along the time axis. Since it only
  happens for one instant, it doesn't persist over time or have multiple
  iterations.

  ### Relation to linear time

  If you think about it, there is a clear logical relationship between constant
  and linear time: because the execution time of a linear algorithm is constant,
  regardless of the size of n, each execution of O(n) is equal to O(1). Thus O(n)
  is simply O(1) writ large or iterated. At any given execution of an O(n)
  algorithm n is going to be equal to 1.

  ## Quadratic time

  With the examples of constant and linear time, the total number of instructions
  doesn't change the amount of work that needs to be performed for each item, but
  this only covers one subset of algorithms. In cases other than O(1) and O(n),
  the length of the input **can** affect the amount of work that needs to be
  performed at execution. The most common example of this scenario is known as
  quadratic time, represented as $O(n^2)$.

  Let's start with an example.

  ```js
  const letters = ['A', 'B', 'C'];

  function quadratic(arr) {
    for (let i = 0; i < arr.length; i++) {
      for (let j = 0; j < arr.length j++) {
        console.log(arr[i]);
      }
    }
  }

  quadratic(letters);
  ```

  This function takes an array . The outer loop runs once for each element of the
  array that is passed to the function. For each iteration of the outer loop, the
  inner loop also runs once for each element of the array.

  In the example this means that the following is output:

  ```
  A A A B B B C C C (length: 9)
  ```

  Mathematically this means that n (the size of the input) grows at a rate of n2
  or the input multiplied by itself. Our outer loop (`i`) is performing n
  iterations (just like in linear time) but our inner loop (`j`) is also
  performing n iterations, three `j`s for every one `i` . It is performing n
  iterations for every nth iteration of the outer loop. So runtime here is
  directly proportional to the squared size of the input data set. As the input
  array has a length of 3, and the inner array runs once for every element in the
  array, this is equal to 3 x 3 or 3 squared (9).

  If the input had length 4, the runtime would be 16 or 4x4. For every execution
  of linear time (the outer loop) the inner loop runs as many times as is equal to
  the length of the input.

  This is not a linear algorithm because as n grows the runtime increases as a
  factor of it. Therefore the runtime is not growing proportional to the size of
  the input, it is growing proportional to the size of the input squared.

  Graphically this is represented with a curving lines as follows:
  ![square.svg](static/square.svg)

  We can clearly see that as n grows, the runtime gets steeper and more
  pronounced,

  ## Logarithmic time (log n)

  A logarithm is best understood as the inverse of exponentiation:

  $$ \log \_{2}8 = 3 \leftrightarrow 2^3 = 8 $$

  When we use log in the context of algorithms we are always using the binary
  number system so we omit the 2, we just say log.

  > With base two logarithms, the logarithm of a number roughly measures the
  > number of times you can divide that number by 2 before you get a value that is
  > less than or equal to 1

  So applying this to the example of $\log 8$ , it is borne out as follows:

  - 8 / 2 = 4 — count: 1
  - 4 / 2 = 2 — count: 2
  - 2 / 2 = 1 — count: 3

  As we are now at 1, we can't divide any more, so $\log 8$ is equal to 3.

  Obviously this doesn't work so neatly with odd numbers, so we approximate.

  For example, with $\log 25$:

  - 25 / 2 = 12.5 — count: 1

  - 12.5 / 2 = 6.25 — count: 2

  - 6.25 / 2 = 3.125 — count: 3

  - 3.125 / 2 = 1.5625 — count: 4

  - 1.5625 / 2 = 0.78125

    Now we are lower than 1 so we have to stop. We can only say that the answer to
    $\log 25$ is somewhere between 4 and 5.

    The exact answer is $\log 25 \approx 4.64$

  Back to algorithms: $O(\log n)$ is a really good complexity to have. It is close
  to O(1) and in between O(1) and O(n). Represented graphically, it starts of with
  a slight increase in runtime but then quickly levels off:

  ![Screenshot_2021-05-11_at_18.51.02.png](static/Screenshot_2021-05-11_at_18.51.02.png)

  Many sorting algorithms run in log n time, as does recursion.

  ## Reducing O complexity to the general case

  When we talk about big O we are looking for the most general case, slight
  deviations, additions or diminutions in n are not as important as the big
  picture. We are looking for the underlying logic and patterns that are
  summarised by the classes of O(1), O(n), O(n2) and others.

  For example, with the following function:

  ```js
  function sumAndAddTwo(arr) {
    let total = 0;
    for (let i = 0; i < arr.length; i++) {
      total += arr[i];
    }
    total = total += 2;
  }
  ```

  The formal representation of the above complexity would be O(n) + O(1). But it's
  easier just to say O(n), since the O(1) that comes from adding two to the result
  of the loop, makes a marginal difference overall.

  Similarly, with the following function:

  ```js
  function processSomeIntegers(integers){
  	let sum, product = 0;

  	integers.forEach(function(int){
  		return sum += int;
  	}

  	integers.forEach(function(int){
  		return product *= int;
  	}

  	console.log(`The sum is ${sum} and the product is ${product}`);
  }
  ```

  It might appear to be more complex than the earlier summing function but it
  isn't really. We have one array (`integers` ) and two loops. Each loop is of
  O(n) complexity and does a constant amount of work. If we add O(n) and O(n) we
  still have O(n), not O(2n). The constant isn't changed in any way by the fact
  that we are looping twice through the array in separate processes, it just
  doubles the length of n. So rather than formalising this as O(n) + O(n), we just
  reduce it to O(n).

  When seeking to simplify algorithms to their most general level of complexity,
  we should keep in mind the following shorthands:

  - Arithmetic operations always take constant time
  - Variable assignment always takes constant time
  - Accessing an element in an array by index or an object value by key is always
    constant
  - in a loop the complexity is the length of the loop times the complexity of
    whatever happens inside of the loop

  With this in mind we can break down the `findSum` function like so:

  ![breakdown.svg](static/breakdown.svg)

  This gives us:

  $$ O(1) + O(1) + O(n) $$

  Which, as noted above can just be reduced to O(n).

  ## Space complexity

  So far we have talked about time complexity only: how the runtime changes
  relative to the size of the input. With space complexity, we are interested in
  how much memory (conceived as an abstract spatial quantity corresponding to the
  machine's hardware) is required by the algorithm. We can use Big O notation for
  space complexity as well as time complexity.

  Space complexity in this sense is called 'auxiliary space complexity'. This
  means the space that the algorithm itself takes up, independent of the the size
  of the inputs. We are not focusing on the space that each input item takes up,
  only the overall space of the algorithm.

  Again there are some rules of thumb:

  - Booleans, `undefined`, and `null` take up constant space
  - Strings require O(n) space, where n is the sting length
  - Reference types take up O(n): an array of length 4 takes up twice as much
    space as an array of length 2

  So with space complexity we are not really interested in how many times the
  function executes, if it is a loop. We are looking to where data is stored: how
  many variables are initialised, how many items there are in the array.

raw_content:
  ---
  tags: [algorithms]
  ---

  # Algorithmic complexity

  ![Screenshot_2021-05-11_at_18.55.23.png](static/Screenshot_2021-05-11_at_18.55.23.png)
  _Summary of the main classes of algorithmic complexity_

  ## Distinguish algorithms from programs

  Test commit Algorithms are general sets of instructions that take data in one
  state, follow a prescribed series of steps and return data in another state.
  Programs are a specific application of one or more algorithms to achieve an
  outcome in a specific context. With algorithms, the actual detail of the steps
  is mostly abstracted and it is irrelevant to what end the algorithm is being
  put. For instance you may create a program that returns addresses from a
  database using a postcode. It is irrelevant to the efficiency or value of the
  algorithm whether or not you are looking up postcodes or some other form of
  alphanumeric string.

  ## Algorithmic efficiency

  Algorithms can be classified based on their efficiency. Efficiency is function
  of the runtime speed. However this doesn't always mean that the fastest
  algorithms are best.

  If we are landing the Curiosity Rover on Mars we may choose an algorithm that is
  slower on average for a guarantee that it will never take longer than we find
  acceptable. In other cases for example a video game, we may choose an algorithm
  that keeps the average time down, even if this occasionally leads to processes
  that need to be aborted because they take too long.

  We need a generalised measure of efficiency to compare algorithms, across
  variant hardware. We can't simply use the number of steps, since some steps will
  be quicker to complete than others in the course of the overall algorithm and
  may take longer on different machines. Moreover the same algorithm could run at
  different speeds on the same machine, depending on its internal state at the
  given time that it ran. So we use the following: **the number of steps required
  relative to the input.**

  > Two given computers may differ in how quickly they can run an algorithm
  > depending on clock speed, available memory and so forth. They will however
  > tend to require approximately the same number of instructions and we can
  > measure the rate at which the number of instructions increases with the
  > problem size.

  This is what **asymptotic runtime** means: the rate at which the runtime of an
  algorithm grows compared to the size of its input. For precision and accuracy we
  use the worst case scenario as the benchmark.

  So: the efficiency of algorithm _A_ can be judged relative to the efficiency of
  algorithm _B_ based on the rate at which the runtime of _A_ grows compared to
  its input, compared to the same property in _B_, assuming the worst possible
  performance.

  From now on we will use the word 'input' to denote the data that the algorithm
  receives (in most cases we will envision this as an array containing a certain
  data type) and 'execution' to denote the computation that is applied by the
  algorithm to each item of the data input. Rephrasing the above with these terms
  we can say that 'algorithmic efficiency' is a measure that describes the rate at
  which the execution time of an algorithm increases relative to the size of its
  input.

  We will find that for the runtime of some algorithms, the size of the input does
  not change the execution time. In these cases, the runtime is proportional to
  the input quantity. In this case, regardless of whether the input is an array of
  one hundred elements or an array of ten elements, the amount of work that is
  executed on each element is the same.

  For other cases, this will not hold true. We will find that there is a
  relationship between input size and execution time such that the length of the
  input affects the amount of work that needs to be performed on each item at
  execution.

  ## Linear time

  Let's start with linear time, which is the easiest runtime to grasp.

  We need an example to make this tangible and show how an algorithm's runtime
  changes compared to the size of its input. Let's take a simple function that
  takes a sequence of integers and returns their sum:

  ```js
  function findSum(arr){
  	let total = 0;
  	for (let i = 0; i < arr.length; i++){
  		total = total += arr[i];
  	)
  	return total
  }
  ```

  The input of this function is an array of integers. It returns their sum as the
  output. Let's say that it takes 1ms for the function to sum an array of two
  integers.

  If we passed in an array of four integers, how would this change the runtime?
  The answer is that, providing that the time it takes to sum two integers doesn't
  change, it would take twice as long.

  As the time it takes to execute `findSum` doesn't change, we can say confidently
  that the runtime is as long as the number of integers we pass in.

  A more general way to say this is that the runtime is equal to size of the
  input. For algorithms of the class of which `findSum` is a member: **the total
  runtime is proportional to the number of items to be processed**.

  ## Introducing asymptotic notation

  If we say that it takes 1ms for two integers to be summed, this gives us the
  following data set:

  | Length of input | Runtime |
  | :-------------- | ------: |
  | 2               |       2 |
  | 3               |       3 |
  | 4               |       4 |
  | 5               |       5 |

  If we plotted this as a graph it is clear that this is equivalent to a linear
  distribution:![lin.svg](static/lin.svg)

  Algorithms which display this distribution are therefore called **linear
  algorithms**.

  The crucial point is that the amount of time it takes to sum the integers does
  not increase as the algorithm proceeds and the input size grows. This time
  remains the same. If it did increase, we would have a fluctuating curve on the
  graph. This aspect remains constant, only the instructions increase. This is why
  we have a nice steadily-advancing distribution in the graph.

  We can now introduce notation to formalise the algorithmic properties we have
  been discussing.

  ## Big O notation

  To express linear time algorithms formally, we say that:

  > it takes some constant amount of time ($C$) to sum one integer and n times as
  > long to sum n integers

  Here the constant is the time for each execution to run and n is the length of
  the input. Thus the complexity is equal to that time multiplied by the input.

  The algebraic expression of this is $cn$ : the constant multiplied by the length
  of the input. In algorithmic notation, the reference to the constant is always
  removed. Instead we just use n and combine it with a 'big O' which stands for
  'order of complexity'. Likewise, if we have an array of four integers being
  passed to `findSum` we could technically express it as O(4n), but we don't
  because we are interested in the general case not the specific details of the
  runtime. So a linear algorithm is expressed algebraically as $O(n)$ which is
  read as "oh of n" and means

  > $O(n)$ = with an order of complexity equal to (some constant) multiplied by n

  Applied, this means an input of length 6 ($n$) where runtime is constant ($c$)
  at 1ms has a total runtime of 6 x 1 = 6ms in total. Exactly the same as our
  table and graph. O n is just a mathematical way of saying _the runtime grows on
  the order of the size of the input._

  > It's really important to remember that when we talk about the execution
  > runtime being constant at 1ms, this is just an arbitrary placeholder. We are
  > not really bothered about whether it's 1ms or 100ms: 'constant' in the
  > mathematical sense doesn't mean a unit of time, it means 'unchanging'. We are
  > using 1ms to get traction on this concept but the fundamental point being
  > expressed is that the size of the input doesn't affect the execution time
  > across the length of the execution time.

  ## Constant time

  Constant time is another one of the main classes of algorithmic complexity. It
  is expressed as O(1). Here, we do away with n because with constant time we are
  only ever dealing with a single execution so we don't need a variable to express
  nth in a series or 'more than one'. Constant time covers all singular processes,
  without iteration.

  An example in practice would be printing `array[0]` . Regardless of the size of
  the array, it is only ever going to take one step, or constant times one. On a
  graph this is equivalent to a flat line along the time axis. Since it only
  happens for one instant, it doesn't persist over time or have multiple
  iterations.

  ### Relation to linear time

  If you think about it, there is a clear logical relationship between constant
  and linear time: because the execution time of a linear algorithm is constant,
  regardless of the size of n, each execution of O(n) is equal to O(1). Thus O(n)
  is simply O(1) writ large or iterated. At any given execution of an O(n)
  algorithm n is going to be equal to 1.

  ## Quadratic time

  With the examples of constant and linear time, the total number of instructions
  doesn't change the amount of work that needs to be performed for each item, but
  this only covers one subset of algorithms. In cases other than O(1) and O(n),
  the length of the input **can** affect the amount of work that needs to be
  performed at execution. The most common example of this scenario is known as
  quadratic time, represented as $O(n^2)$.

  Let's start with an example.

  ```js
  const letters = ['A', 'B', 'C'];

  function quadratic(arr) {
    for (let i = 0; i < arr.length; i++) {
      for (let j = 0; j < arr.length j++) {
        console.log(arr[i]);
      }
    }
  }

  quadratic(letters);
  ```

  This function takes an array . The outer loop runs once for each element of the
  array that is passed to the function. For each iteration of the outer loop, the
  inner loop also runs once for each element of the array.

  In the example this means that the following is output:

  ```
  A A A B B B C C C (length: 9)
  ```

  Mathematically this means that n (the size of the input) grows at a rate of n2
  or the input multiplied by itself. Our outer loop (`i`) is performing n
  iterations (just like in linear time) but our inner loop (`j`) is also
  performing n iterations, three `j`s for every one `i` . It is performing n
  iterations for every nth iteration of the outer loop. So runtime here is
  directly proportional to the squared size of the input data set. As the input
  array has a length of 3, and the inner array runs once for every element in the
  array, this is equal to 3 x 3 or 3 squared (9).

  If the input had length 4, the runtime would be 16 or 4x4. For every execution
  of linear time (the outer loop) the inner loop runs as many times as is equal to
  the length of the input.

  This is not a linear algorithm because as n grows the runtime increases as a
  factor of it. Therefore the runtime is not growing proportional to the size of
  the input, it is growing proportional to the size of the input squared.

  Graphically this is represented with a curving lines as follows:
  ![square.svg](static/square.svg)

  We can clearly see that as n grows, the runtime gets steeper and more
  pronounced,

  ## Logarithmic time (log n)

  A logarithm is best understood as the inverse of exponentiation:

  $$ \log \_{2}8 = 3 \leftrightarrow 2^3 = 8 $$

  When we use log in the context of algorithms we are always using the binary
  number system so we omit the 2, we just say log.

  > With base two logarithms, the logarithm of a number roughly measures the
  > number of times you can divide that number by 2 before you get a value that is
  > less than or equal to 1

  So applying this to the example of $\log 8$ , it is borne out as follows:

  - 8 / 2 = 4 — count: 1
  - 4 / 2 = 2 — count: 2
  - 2 / 2 = 1 — count: 3

  As we are now at 1, we can't divide any more, so $\log 8$ is equal to 3.

  Obviously this doesn't work so neatly with odd numbers, so we approximate.

  For example, with $\log 25$:

  - 25 / 2 = 12.5 — count: 1

  - 12.5 / 2 = 6.25 — count: 2

  - 6.25 / 2 = 3.125 — count: 3

  - 3.125 / 2 = 1.5625 — count: 4

  - 1.5625 / 2 = 0.78125

    Now we are lower than 1 so we have to stop. We can only say that the answer to
    $\log 25$ is somewhere between 4 and 5.

    The exact answer is $\log 25 \approx 4.64$

  Back to algorithms: $O(\log n)$ is a really good complexity to have. It is close
  to O(1) and in between O(1) and O(n). Represented graphically, it starts of with
  a slight increase in runtime but then quickly levels off:

  ![Screenshot_2021-05-11_at_18.51.02.png](static/Screenshot_2021-05-11_at_18.51.02.png)

  Many sorting algorithms run in log n time, as does recursion.

  ## Reducing O complexity to the general case

  When we talk about big O we are looking for the most general case, slight
  deviations, additions or diminutions in n are not as important as the big
  picture. We are looking for the underlying logic and patterns that are
  summarised by the classes of O(1), O(n), O(n2) and others.

  For example, with the following function:

  ```js
  function sumAndAddTwo(arr) {
    let total = 0;
    for (let i = 0; i < arr.length; i++) {
      total += arr[i];
    }
    total = total += 2;
  }
  ```

  The formal representation of the above complexity would be O(n) + O(1). But it's
  easier just to say O(n), since the O(1) that comes from adding two to the result
  of the loop, makes a marginal difference overall.

  Similarly, with the following function:

  ```js
  function processSomeIntegers(integers){
  	let sum, product = 0;

  	integers.forEach(function(int){
  		return sum += int;
  	}

  	integers.forEach(function(int){
  		return product *= int;
  	}

  	console.log(`The sum is ${sum} and the product is ${product}`);
  }
  ```

  It might appear to be more complex than the earlier summing function but it
  isn't really. We have one array (`integers` ) and two loops. Each loop is of
  O(n) complexity and does a constant amount of work. If we add O(n) and O(n) we
  still have O(n), not O(2n). The constant isn't changed in any way by the fact
  that we are looping twice through the array in separate processes, it just
  doubles the length of n. So rather than formalising this as O(n) + O(n), we just
  reduce it to O(n).

  When seeking to simplify algorithms to their most general level of complexity,
  we should keep in mind the following shorthands:

  - Arithmetic operations always take constant time
  - Variable assignment always takes constant time
  - Accessing an element in an array by index or an object value by key is always
    constant
  - in a loop the complexity is the length of the loop times the complexity of
    whatever happens inside of the loop

  With this in mind we can break down the `findSum` function like so:

  ![breakdown.svg](static/breakdown.svg)

  This gives us:

  $$ O(1) + O(1) + O(n) $$

  Which, as noted above can just be reduced to O(n).

  ## Space complexity

  So far we have talked about time complexity only: how the runtime changes
  relative to the size of the input. With space complexity, we are interested in
  how much memory (conceived as an abstract spatial quantity corresponding to the
  machine's hardware) is required by the algorithm. We can use Big O notation for
  space complexity as well as time complexity.

  Space complexity in this sense is called 'auxiliary space complexity'. This
  means the space that the algorithm itself takes up, independent of the the size
  of the inputs. We are not focusing on the space that each input item takes up,
  only the overall space of the algorithm.

  Again there are some rules of thumb:

  - Booleans, `undefined`, and `null` take up constant space
  - Strings require O(n) space, where n is the sting length
  - Reference types take up O(n): an array of length 4 takes up twice as much
    space as an array of length 2

  So with space complexity we are not really interested in how many times the
  function executes, if it is a loop. We are looking to where data is stored: how
  many variables are initialised, how many items there are in the array.


path: neuron/4e98d18e-1a79-4e78-a4ac-f7e3a2de9887/Analogue_and_digital.md
sortable_path: neuron4e98d18e-1a79-4e78-a4ac-f7e3a2de9887Analogue_and_digital.md
title: Analogue and digital
checksum: 94c1cd2bdfbc7ff7cf0a43f53ebf0a4a0f855e8bcd860b5f447ceceb32229270
created: 2024-10-22 05:37:47.80171542+00:00
modified: 2024-10-21 22:11:47.058646397+00:00
metadata: {"tags":["analogue"]}
lead:
  Analogue and digital are paradigms for recording information, specifically
  information about an object or state that obtains in the world.

body:
  Analogue and digital are paradigms for recording information, specifically
  information about an object or state that obtains in the world.

  ![](static/analog-digital.svg)

  ## Analogue

  Here are two examples of analogue measurement/ capture:

  - Measuring the temperature of a room with a mercury barometer
    - The mercury in the barometer expands or contracts in response to the heat of
      the ambient environment.
    - We say that the position of the mercury relative to the container represents
      the current temperature
  - Measuring the weight of an object by placing it on a weighing scale
    - We say the position of the needle on the scale represents the weight of the
      object

  > In each case, the position of the needle or the mercury isn't actually the
  > temperature or the weight, it is a representation of this property. Thus in
  > both cases, the analogue device works by analogy hence why we call it
  > analogue.

  This isn't the only sense in which "analogue" is used however. It doesn't always
  mean an analagous representation of a physical property. We talk about
  **analogue signals** as signals that vary continuously and which do not align to
  digital values.

  ## Digital

  ### Limitations of analogue for computation

  From a computing point of view, the data that can be captured and recorded by
  analogue devices is simply too diverse and non-uniform. Computers are universal
  machines whereas analogue devices typically only measure one property well.
  Whereas a computer can record and playback music and take the temperature, to do
  this in an analogue way would require a complex machine that can represent two
  unrelated data sources. Imagine if a computer was able to achieve universality
  through analogue means, it would most likely be very large physically and very
  complex computationally since the computer would need to be able process data
  from, e.g. for grooves on vinyl, needles on weighing sensors and mercury in
  vacuum tubes.

  > The types of analogue systems used are so different and variable that creating
  > a common computing device that can understand all of them is nearly
  > impossible.

  Furthermore:

  > Computers require highly reliable and accurate representations of certain
  > types of data. Analogue representations of data can be difficult to measure
  > precisely (and are potentially infinite), tend to decay over time, and lose
  > fidelity when copied.

  **Computers need a way to represent all types of data in a format that can be
  accurately processed, stored and copied.**

  ### Why digital is better suited to computational requirements

  A digital system represents data as a sequence of symbols where each symbol is
  one of a limited set of values.

  Computers only use two symbols for each value: 0 and 1.

  Although a digital system could use more than two symbols, adding more would
  [increase the complexity](Why_computers_use_binary.md#from-circuits-to-programs)
  and cost of the system. A set of only two symbols allows for simplified hardware
  and improved reliability.

  All data within a computer from images to sounds are sequences of 0s and 1s. In
  practice, the sequences are stored in various ways. On a CD, the binary digits
  are stored as bumps (0) or flat spaces (1). On a flash drive, the 1s and 0s are
  electrical charges etc.

raw_content:
  ---
  tags: [analogue]
  ---

  # Analogue and digital

  Analogue and digital are paradigms for recording information, specifically
  information about an object or state that obtains in the world.

  ![](static/analog-digital.svg)

  ## Analogue

  Here are two examples of analogue measurement/ capture:

  - Measuring the temperature of a room with a mercury barometer
    - The mercury in the barometer expands or contracts in response to the heat of
      the ambient environment.
    - We say that the position of the mercury relative to the container represents
      the current temperature
  - Measuring the weight of an object by placing it on a weighing scale
    - We say the position of the needle on the scale represents the weight of the
      object

  > In each case, the position of the needle or the mercury isn't actually the
  > temperature or the weight, it is a representation of this property. Thus in
  > both cases, the analogue device works by analogy hence why we call it
  > analogue.

  This isn't the only sense in which "analogue" is used however. It doesn't always
  mean an analagous representation of a physical property. We talk about
  **analogue signals** as signals that vary continuously and which do not align to
  digital values.

  ## Digital

  ### Limitations of analogue for computation

  From a computing point of view, the data that can be captured and recorded by
  analogue devices is simply too diverse and non-uniform. Computers are universal
  machines whereas analogue devices typically only measure one property well.
  Whereas a computer can record and playback music and take the temperature, to do
  this in an analogue way would require a complex machine that can represent two
  unrelated data sources. Imagine if a computer was able to achieve universality
  through analogue means, it would most likely be very large physically and very
  complex computationally since the computer would need to be able process data
  from, e.g. for grooves on vinyl, needles on weighing sensors and mercury in
  vacuum tubes.

  > The types of analogue systems used are so different and variable that creating
  > a common computing device that can understand all of them is nearly
  > impossible.

  Furthermore:

  > Computers require highly reliable and accurate representations of certain
  > types of data. Analogue representations of data can be difficult to measure
  > precisely (and are potentially infinite), tend to decay over time, and lose
  > fidelity when copied.

  **Computers need a way to represent all types of data in a format that can be
  accurately processed, stored and copied.**

  ### Why digital is better suited to computational requirements

  A digital system represents data as a sequence of symbols where each symbol is
  one of a limited set of values.

  Computers only use two symbols for each value: 0 and 1.

  Although a digital system could use more than two symbols, adding more would
  [increase the complexity](Why_computers_use_binary.md#from-circuits-to-programs)
  and cost of the system. A set of only two symbols allows for simplified hardware
  and improved reliability.

  All data within a computer from images to sounds are sequences of 0s and 1s. In
  practice, the sequences are stored in various ways. On a CD, the binary digits
  are stored as bumps (0) or flat spaces (1). On a flash drive, the 1s and 0s are
  electrical charges etc.


