# notes

path: zk/Middleware_in_NodeJS.md
sortable_path: zkMiddleware_in_NodeJS.md
title: Middleware
lead: ## What is middleware?
checksum: cc530669082b3716b21388f1c2ef4e95e3e1c2d678e96e586dec2cc8a3f66f4b
created: 2024-06-16 17:37:31.546028105+00:00
modified: 2024-06-16 17:18:22.098336515+00:00
metadata: {"tags":["node-js","middleware"]}
body:
  ## What is middleware?

  - Anything that terminates the `req, res` cycle counts as middleware. It is
    basically anything that acts as an intermediary once the request is received
    but before the resource is sent. A good example would be the
    `app.use(express.json()` or `app.use(bodyParser.json)` functions we call in
    order to be able to parse JSON that is sent from the client.
  - You will most likely have multiple middleware functions running at once. We
    call this intermediary part of the cycle the **request processing pipeline**.
  - Generally all middleware will be added as a property on the Express `app`
    instance with the `app.use(...)` syntax.

  ## Creating custom middleware functions

  ### Basic schema

  ```js
  app.use((req, res, next) => {
    // do some middleware
    next();
  });
  ```

  ### `next`

  The `next` parameter is key, it allows Express to move onto the next middleware
  function once the custom middleware executes. Without it, the request processing
  pipeline will get blocked.

  Middleware functions are basically asynchronous requests and as such they use a
  similar syntax as Promises (e.g `then`) for sequencing processes.

  ### Example of sequence

  ```js
  app.use((req, res, next) => {
    console.log("Do process A...");
    next();
  });

  app.use((req, res, next) => {
    console.log("Do process B...");
    next();
  });
  ```

  > It makes more sense of course to define our middleware within a function and
  > then pass it as an argument to `app.use()`

  ## Including middleware based on environment

  With a full-scale Node application you will typically run three environments:

  - Development
  - Testing
  - Production

  We will not want to run certain types of middleware in all environments. For
  example, it would be costly to run logging in the app's production environment.
  It would make more sense to run this only in development.

  ### Accessing current Node environment

  We can control which middleware we run via the Node envrionment variables:
  `process.env` (see for instance [ports](Ports_in_NodeJS.md)).

  We could set
  [Morgan](Morgan.md) to run
  only in development with:

  ```js
  if (app.get("env") === "development") {
    app.use(morgan("common"));
    console.log("Morgan enabled");
  }
  ```

raw_content:
  ---
  tags:
    - node-js
    - middleware
  ---

  # Middleware

  ## What is middleware?

  - Anything that terminates the `req, res` cycle counts as middleware. It is
    basically anything that acts as an intermediary once the request is received
    but before the resource is sent. A good example would be the
    `app.use(express.json()` or `app.use(bodyParser.json)` functions we call in
    order to be able to parse JSON that is sent from the client.
  - You will most likely have multiple middleware functions running at once. We
    call this intermediary part of the cycle the **request processing pipeline**.
  - Generally all middleware will be added as a property on the Express `app`
    instance with the `app.use(...)` syntax.

  ## Creating custom middleware functions

  ### Basic schema

  ```js
  app.use((req, res, next) => {
    // do some middleware
    next();
  });
  ```

  ### `next`

  The `next` parameter is key, it allows Express to move onto the next middleware
  function once the custom middleware executes. Without it, the request processing
  pipeline will get blocked.

  Middleware functions are basically asynchronous requests and as such they use a
  similar syntax as Promises (e.g `then`) for sequencing processes.

  ### Example of sequence

  ```js
  app.use((req, res, next) => {
    console.log("Do process A...");
    next();
  });

  app.use((req, res, next) => {
    console.log("Do process B...");
    next();
  });
  ```

  > It makes more sense of course to define our middleware within a function and
  > then pass it as an argument to `app.use()`

  ## Including middleware based on environment

  With a full-scale Node application you will typically run three environments:

  - Development
  - Testing
  - Production

  We will not want to run certain types of middleware in all environments. For
  example, it would be costly to run logging in the app's production environment.
  It would make more sense to run this only in development.

  ### Accessing current Node environment

  We can control which middleware we run via the Node envrionment variables:
  `process.env` (see for instance [ports](Ports_in_NodeJS.md)).

  We could set
  [Morgan](Morgan.md) to run
  only in development with:

  ```js
  if (app.get("env") === "development") {
    app.use(morgan("common"));
    console.log("Morgan enabled");
  }
  ```


path: zk/Modules_in_NodeJS.md
sortable_path: zkModules_in_NodeJS.md
title: Modules
checksum: 9d03b5ccc104e11c3ef76f35571497cf375083499eb5f3b5f78548785b8d21f0
created: 2024-06-16 17:37:31.573910162+00:00
modified: 2024-06-15 09:50:29.696637857+00:00
metadata: {"tags":["node-js"]}
lead:
  > Modules are partitioned files where we define our variables and functions.
  > Values defined in modules are scoped to that specific module, constituting a
  > unique name space. This avoids name clashes in large programs.

body:
  > Modules are partitioned files where we define our variables and functions.
  > Values defined in modules are scoped to that specific module, constituting a
  > unique name space. This avoids name clashes in large programs.

  - Every file in a Node application is considered a module.

  - The variables and methods in a module are equivalent to `private` properties
    and methods in object-oriented programming.

  - If you wish to use a function or variable defined in a module outside of its
    modular container you need to explicitly export it and make it public.

  ## Structure of a module

  Node keeps an internal record of the properties of a module. To see this we can
  log the property `module` to the console.

  ```js
  // index.js
  console.log(module);
  ```

  This gives us:

  ```plaintext
  Module {
   id: '.',
   path: '/home/thomas/repos/node-learning',
   exports: {},
   filename: '/home/thomas/repos/node-learning/index.js',
   loaded: false,
   children: [],
   paths: [
     '/home/thomas/repos/node-learning/node_modules',
     '/home/thomas/repos/node_modules',
     '/home/thomas/node_modules',
     '/home/node_modules',
     '/node_modules'
   ]
  }
  ```

  ## Exports

  - Whenever we export a property or method from a module we are directly
    targeting the `exports` property of the module object.
  - Once we add exports to a file they will be displayed under that property of
    the module object.
  - We can export the entire module itself as the export (typically used when the
    module is a single function or class) or individual properties.

  ### Exporting a whole module

  _The example below is a module file that consists in a single function_

  ```js
  module.exports = function (...params) {
    // function body
  };
  ```

  Note the module is unnamed. We would name it when we import:

  ```js
  const myFunction = require("./filenme");
  ```

  ### Exporting sub-components from a module

  In the example below we export a variable and function from the same module.
  Note only those values prefixed with `exports` are exported.

  ```js
  exports.myFunc = (...params) => {
    // function bod[]()y
  };

  exports.aVar = 321.3;

  var nonExportedVar = true;
  ```

  This time the exports are already name so we would import with the following:

  ```js
  const { myFunc, aVar } = require("./filename");
  ```

  We can also do the exporting at the bottom when the individual components are
  named:

  ```js
  const myNamedFunc = (val) => {
    return val + 1;
  };

  function anotherNamedFunc(val) {
    return val * 2;
  }

  // This time we export at the bottom
  exports.myNamedFunc = myNamedFunc;
  exports.differentName = anotherNamedFunc; // We can use different names

  // Or we could export them together
  module.exports = { myNamedFunc, anotherNamedFunc };
  ```

  The import is the same:

  ```js
  const { myNamedFunc, anotherNamedFunc } = require("./modules/multiExports");
  ```

  ## Structuring modules

  The techniques above are useful to know but generally you would want to enforce
  a stricter structure than a mix of exported and private values in the one file.
  The best way to do this is with a single default export.

  Here the thing exported could be a composite function or an object that
  basically acts like a class with methods and properties.

  _Export a composite single function_

  ```js

  module.exports = () => {
  	foo() {...}
  	bar() {...}
  }
  ```

  _Export an object_

  ```js

  module.exports = {
  	foo : () => {...},
      bar: () => {...}
  }
  ```

  **Both of these structures would be referred to in the same way when importing
  and using them.**

  Or you could export an actual class as the default. This is practically the same
  as the two above other than that you would have to use `new` to initiate an
  instance of the class.

  ```js
  export default class {
    foo() {}
    bar() {}
  }
  ```

  Every method and property within the export will be public by default, whether
  it is an object, class or function. If you wanted to keep certain
  methods/properties private, the best approach is to define them as variables and
  functions within the module file but outside of the `export` block.

raw_content:
  ---
  tags:
    - node-js
  ---

  # Modules

  > Modules are partitioned files where we define our variables and functions.
  > Values defined in modules are scoped to that specific module, constituting a
  > unique name space. This avoids name clashes in large programs.

  - Every file in a Node application is considered a module.

  - The variables and methods in a module are equivalent to `private` properties
    and methods in object-oriented programming.

  - If you wish to use a function or variable defined in a module outside of its
    modular container you need to explicitly export it and make it public.

  ## Structure of a module

  Node keeps an internal record of the properties of a module. To see this we can
  log the property `module` to the console.

  ```js
  // index.js
  console.log(module);
  ```

  This gives us:

  ```plaintext
  Module {
   id: '.',
   path: '/home/thomas/repos/node-learning',
   exports: {},
   filename: '/home/thomas/repos/node-learning/index.js',
   loaded: false,
   children: [],
   paths: [
     '/home/thomas/repos/node-learning/node_modules',
     '/home/thomas/repos/node_modules',
     '/home/thomas/node_modules',
     '/home/node_modules',
     '/node_modules'
   ]
  }
  ```

  ## Exports

  - Whenever we export a property or method from a module we are directly
    targeting the `exports` property of the module object.
  - Once we add exports to a file they will be displayed under that property of
    the module object.
  - We can export the entire module itself as the export (typically used when the
    module is a single function or class) or individual properties.

  ### Exporting a whole module

  _The example below is a module file that consists in a single function_

  ```js
  module.exports = function (...params) {
    // function body
  };
  ```

  Note the module is unnamed. We would name it when we import:

  ```js
  const myFunction = require("./filenme");
  ```

  ### Exporting sub-components from a module

  In the example below we export a variable and function from the same module.
  Note only those values prefixed with `exports` are exported.

  ```js
  exports.myFunc = (...params) => {
    // function bod[]()y
  };

  exports.aVar = 321.3;

  var nonExportedVar = true;
  ```

  This time the exports are already name so we would import with the following:

  ```js
  const { myFunc, aVar } = require("./filename");
  ```

  We can also do the exporting at the bottom when the individual components are
  named:

  ```js
  const myNamedFunc = (val) => {
    return val + 1;
  };

  function anotherNamedFunc(val) {
    return val * 2;
  }

  // This time we export at the bottom
  exports.myNamedFunc = myNamedFunc;
  exports.differentName = anotherNamedFunc; // We can use different names

  // Or we could export them together
  module.exports = { myNamedFunc, anotherNamedFunc };
  ```

  The import is the same:

  ```js
  const { myNamedFunc, anotherNamedFunc } = require("./modules/multiExports");
  ```

  ## Structuring modules

  The techniques above are useful to know but generally you would want to enforce
  a stricter structure than a mix of exported and private values in the one file.
  The best way to do this is with a single default export.

  Here the thing exported could be a composite function or an object that
  basically acts like a class with methods and properties.

  _Export a composite single function_

  ```js

  module.exports = () => {
  	foo() {...}
  	bar() {...}
  }
  ```

  _Export an object_

  ```js

  module.exports = {
  	foo : () => {...},
      bar: () => {...}
  }
  ```

  **Both of these structures would be referred to in the same way when importing
  and using them.**

  Or you could export an actual class as the default. This is practically the same
  as the two above other than that you would have to use `new` to initiate an
  instance of the class.

  ```js
  export default class {
    foo() {}
    bar() {}
  }
  ```

  Every method and property within the export will be public by default, whether
  it is an object, class or function. If you wanted to keep certain
  methods/properties private, the best approach is to define them as variables and
  functions within the module file but outside of the `export` block.


path: zk/Non_null_assertion_in_TypeScript.md
sortable_path: zkNon_null_assertion_in_TypeScript.md
title: Non-null assertion
lead: Oftentimes you will encounter errors of the form:
checksum: 3afb6d654d3d9d9c24f9155a366976af6b70dc7f501848f73c2cb17e7c8e6455
created: 2024-06-16 17:37:31.703255445+00:00
modified: 2024-03-27 10:45:52.503525368+00:00
metadata: {"tags":["typescript"]}
body:
  Oftentimes you will encounter errors of the form:

  `Property [property_name] does not exist on type [type_name].ts(2339)`

  This means the interpreter is not able to determine whether the property which
  you are seeking to change exists. It can't be sure that you are not trying to
  update a property that is either `null` or `undefined`. This is good type
  checking but it means that in cases when _you know_ the property either exists
  or _will_ exist at runtime, you are going to face a code-breaking error.

  To get around this you can use non-null assertion. Like the name suggests, here
  you are saying to TypeScript: thanks but I am sure this property exists. When
  you do this you get the value you are trying to change with `null` and
  `undefined` excluded from it.

  This typically occurs when you are assigning a property to a value and TS
  doesn't know whether the value will exist or be accessible at this point in the
  runtime. A good scenario would be dynamic data passed into a function:

  ```ts
  function doSomethingWith(typeAlias: ITypeAlias) {
    let x = typeAlias!.somePropertyEvidentAtRuntime;
  }
  ```

  However it should be used carefully and sparingly because you are obviously
  turning off core type-checking and overuse nullifies the purpose of TypeScript.

  One way to get around it is to use better
  [type-guarding](Type_guarding_and_narrowing.md) and conditionality and to
  cover cases where the value may be undefined.

raw_content:
  ---
  tags:
    - typescript
  ---

  # Non-null assertion

  Oftentimes you will encounter errors of the form:

  `Property [property_name] does not exist on type [type_name].ts(2339)`

  This means the interpreter is not able to determine whether the property which
  you are seeking to change exists. It can't be sure that you are not trying to
  update a property that is either `null` or `undefined`. This is good type
  checking but it means that in cases when _you know_ the property either exists
  or _will_ exist at runtime, you are going to face a code-breaking error.

  To get around this you can use non-null assertion. Like the name suggests, here
  you are saying to TypeScript: thanks but I am sure this property exists. When
  you do this you get the value you are trying to change with `null` and
  `undefined` excluded from it.

  This typically occurs when you are assigning a property to a value and TS
  doesn't know whether the value will exist or be accessible at this point in the
  runtime. A good scenario would be dynamic data passed into a function:

  ```ts
  function doSomethingWith(typeAlias: ITypeAlias) {
    let x = typeAlias!.somePropertyEvidentAtRuntime;
  }
  ```

  However it should be used carefully and sparingly because you are obviously
  turning off core type-checking and overuse nullifies the purpose of TypeScript.

  One way to get around it is to use better
  [type-guarding](Type_guarding_and_narrowing.md) and conditionality and to
  cover cases where the value may be undefined.


path: zk/Package_management_in_NodeJS.md
sortable_path: zkPackage_management_in_NodeJS.md
title: Package management
lead: ## List installed packages
checksum: a4752920727fecb5b8ce97d7572b92652a86d679349fb8b71c6d9695790bc888
created: 2024-06-16 17:37:31.719175457+00:00
modified: 2024-06-15 09:50:35.966759132+00:00
metadata: {"tags":["node-js"]}
body:
  ## List installed packages

  ```bash
  npm list
  ```

  This will return a recursive tree that lists dependencies, dependences of
  dependencies, ... and so on. To limit the depth you can add the `--depth=` flag.
  For example to see only your installed packages and their versions use
  `npm list --depth=0`.

  ## View `package.json` data for an installed package

  We could go to the NPM registry and view details or we can quickly view the
  `package.json` for the dependency with the command `npm view [package_name]`

  We can pinpoint specific dependencies in the `package.json`, e.g.
  `npm view [package_name] dependencies `

  ## View outdated modules

  See whether your dependency version is out of date use `npm outdated`. This
  gives us a table, for example:

  ![Pasted image 20220411082627.png](Pasted_image_20220411082627.png)

  - _Latest_ tells us the latest release available from the developers
  - _Wanted_ tells us the version that our `package.json` rules target. To take
    the first dependency as an example. We must have set our SemVer syntax to
    `^0.4.x` since it is telling us that there is a minor release that is more
    recent than the one we have installed but is not advising that we update to
    the latest major release.
  - _Current_ tells us which version we currently have installed regardless of the
    version that our `package.json` is targeting or the most recent version
    available.

  ## Updating

  `npm update` only updates from _current_ to _wanted_. In other words it only
  updates in accordance with your caret and tilde rules applied to
  [semantic versioning](Semantic_versioning.md).

raw_content:
  ---
  tags:
    - node-js
  ---

  # Package management

  ## List installed packages

  ```bash
  npm list
  ```

  This will return a recursive tree that lists dependencies, dependences of
  dependencies, ... and so on. To limit the depth you can add the `--depth=` flag.
  For example to see only your installed packages and their versions use
  `npm list --depth=0`.

  ## View `package.json` data for an installed package

  We could go to the NPM registry and view details or we can quickly view the
  `package.json` for the dependency with the command `npm view [package_name]`

  We can pinpoint specific dependencies in the `package.json`, e.g.
  `npm view [package_name] dependencies `

  ## View outdated modules

  See whether your dependency version is out of date use `npm outdated`. This
  gives us a table, for example:

  ![Pasted image 20220411082627.png](Pasted_image_20220411082627.png)

  - _Latest_ tells us the latest release available from the developers
  - _Wanted_ tells us the version that our `package.json` rules target. To take
    the first dependency as an example. We must have set our SemVer syntax to
    `^0.4.x` since it is telling us that there is a minor release that is more
    recent than the one we have installed but is not advising that we update to
    the latest major release.
  - _Current_ tells us which version we currently have installed regardless of the
    version that our `package.json` is targeting or the most recent version
    available.

  ## Updating

  `npm update` only updates from _current_ to _wanted_. In other words it only
  updates in accordance with your caret and tilde rules applied to
  [semantic versioning](Semantic_versioning.md).


path: zk/Ports_in_NodeJS.md
sortable_path: zkPorts_in_NodeJS.md
title: Ports
checksum: c9975673e974869e0d02f1f89f50d66c63960e6cdd9b0af1d6d5912241e71bb3
created: 2024-06-16 17:37:31.730278283+00:00
modified: 2024-06-15 09:50:38.086800108+00:00
metadata: {"tags":["node-js"]}
lead:
  When working in development we are able to specify the specific port from which
  we want top serve our application. In production, we do not always have this
  control: the port will most likely be set by the provider of the server
  environment.

body:
  When working in development we are able to specify the specific port from which
  we want top serve our application. In production, we do not always have this
  control: the port will most likely be set by the provider of the server
  environment.

  While we may not know the specific port, whatever it is, it will be accessible
  via the `PORT` environment variable. So we can use this when writing our
  [event listeners](Events%20module.md#event-emitters):

  ```js
  const port = process.env.PORT || 3000;
  ```

  This way, if a port is set by the provider it will use it. If not, it will fall
  back to 3000.

raw_content:
  ---
  tags:
    - node-js
  ---

  # Ports

  When working in development we are able to specify the specific port from which
  we want top serve our application. In production, we do not always have this
  control: the port will most likely be set by the provider of the server
  environment.

  While we may not know the specific port, whatever it is, it will be accessible
  via the `PORT` environment variable. So we can use this when writing our
  [event listeners](Events%20module.md#event-emitters):

  ```js
  const port = process.env.PORT || 3000;
  ```

  This way, if a port is set by the provider it will use it. If not, it will fall
  back to 3000.


path: zk/Printing_values_in_C.md
sortable_path: zkPrinting_values_in_C.md
title: Printing values in C
lead: # Printing values in C
checksum: 16d854f6517315d8456373ef1c9943b600da2024d010a6029f9fc1e6cd9f20a2
created: 2024-06-16 17:37:31.744437412+00:00
modified: 2024-06-16 17:19:46.997337347+00:00
body:
  # Printing values in C

  ## To print a literal

  ```c
  printf("hello world")
  ```

  ## To print a variable

  ```c
  int age = 25
  print("%d", age)

  ```

  The `%d` is an example of a [format specifier](./zk/format-specifiers-in-c.md).
  It is the decimal integer format specifier.

raw_content:
  ---
  id: 3hcv0amc
  title: Printing values in C
  tags:
    - C
  created: Thursday, February 29, 2024 | 17:33
  ---

  # Printing values in C

  ## To print a literal

  ```c
  printf("hello world")
  ```

  ## To print a variable

  ```c
  int age = 25
  print("%d", age)

  ```

  The `%d` is an example of a [format specifier](./zk/format-specifiers-in-c.md).
  It is the decimal integer format specifier.

metadata:
  {"created":"Thursday, February 29, 2024 | 17:33","id":"3hcv0amc","tags":["C"],"title":"Printing values in C"}


path: zk/Process_module_in_NodeJS.md
sortable_path: zkProcess_module_in_NodeJS.md
title: The process module in Node.js
checksum: c0471abbd224662b086b5b3f40c8d67e46decb2adefb72bb979d0b92686a0b94
created: 2024-06-16 17:37:31.760848902+00:00
modified: 2024-06-16 17:15:36.691080654+00:00
metadata: {"tags":["node-js"]}
lead:
  `process` is a core module and a global object accessible from anywhere in a
  Node.js application. It contains functionality that allows us to interact with
  information about the current process instance of Node.

body:
  `process` is a core module and a global object accessible from anywhere in a
  Node.js application. It contains functionality that allows us to interact with
  information about the current process instance of Node.

  For example, we can use it to get environment information, read environment
  variables, parse arguments that are passed to the run command, communicate with
  the terminal and exit the current process.

  ## Managing runtime environments

  See
  [Managing Environments](Managing_environments_in_NodeJS.md).

  ## Accessing arguments: `process.argv`

  We can use `process.argv` to return an array containing the command-line
  arguments passed when a Node process is launched. This could be a whole-app
  entrypoint (i.e. `index.js`) or a single file we are running.

  For instance if we run the following file:

  ```js
  // process-demo.js
  console.log(3 + 3);
  console.log(process.argv);
  ```

  We get:

  ```bash
  6
  [
    '/Users/thomasbishop/.nvm/versions/node/v16.10.0/bin/node',
    '/Users/thomasbishop/prepos/testNode.js'
  ]
  ```

  The first item is the last executed process. The second is the array of
  arguments.

  In the array, the first value is a reference to the Node runtime binary. The
  second is the file that was passed to Node.

  If we passed in a parameter we would get that too:

  ```
  node process-demo.js --fakeFlag
  ```

  Gives us:

  ```bash
  [
    "/Users/thomasbishop/.nvm/versions/node/v16.10.0/bin/node",
    "/Users/thomasbishop/prepos/testNode.js",
    "--fake-flag",
  ]
  ```

  When writing Node CLI applications we can write functions that parse standard
  input. For example:

  ```js
  function parseStandardInput(flag) {
    let indexAfterFlag = process.argv.indexOf(flag);
    console.log(process.argv[indexAfterFlag]);
  }
  ```

  Say we ran a program that took in a username:

  ```bash
  node userName.js --user Thomas
  ```

  Then `parseStandardInput("--user")` would give us `"thomas"`

  ## Standard input and output

  Below is the most basic use case: writing to the output.

  ```js
  // alpha.js
  process.stdout.write("Hello from file \n");
  ```

  ```
  node alpha.js
  Hello from file
  ```

  Typically we want to write and read data that the user provides. To do this we
  need to wait on that event. So we use Node's asynchonous
  [event listeners](events.md). We wait
  on the `data` event:

  ```js
  process.stdin.on("data", (data) => {
    process.stdout.write(data.toString().trim());
  });

  process.stdout.write("\n What is your name? \n");
  process.stdout.write(` \n > `);
  ```

  This waits for whatever input is given and then echoes it.

  Currently there is no way to end this process; it will just keep echoing the
  input.

  So we add an exit clause:

  ```js
  process.stdin.on("data", (data) => {
    process.stdout.write(data.toString().trim());
    process.exit();
  });
  ```

  We can also execute functionality on the exit event:

  ```js
  process.on("exit", () => {
    // Do something
  });
  ```

  > Timings can also obviously be managed via the `setTimeout` and `setInterval`
  > methods.

  ### Additional useful methods

  ```js
  process.stdout.clearLine();
  process.stdout.cursorTo([int]);
  ```

  ### `readline`

  The core `readline` module simplifies the handling of standard input and output
  by providing a wrapper. It reads the input stream line by line. Basic usage:

  ```js
  var readline = require("readline");

  var rl = readline.createInterface(process.stdin, process.stdout);

  rl.question("What is your age? ", (age) => {
    console.log("Your age is: " + age);
  });
  ```

  Usage scenarios at: https://www.geeksforgeeks.org/node-js-readline-module/

raw_content:
  ---
  tags:
    - node-js
  ---

  # The `process` module in Node.js

  `process` is a core module and a global object accessible from anywhere in a
  Node.js application. It contains functionality that allows us to interact with
  information about the current process instance of Node.

  For example, we can use it to get environment information, read environment
  variables, parse arguments that are passed to the run command, communicate with
  the terminal and exit the current process.

  ## Managing runtime environments

  See
  [Managing Environments](Managing_environments_in_NodeJS.md).

  ## Accessing arguments: `process.argv`

  We can use `process.argv` to return an array containing the command-line
  arguments passed when a Node process is launched. This could be a whole-app
  entrypoint (i.e. `index.js`) or a single file we are running.

  For instance if we run the following file:

  ```js
  // process-demo.js
  console.log(3 + 3);
  console.log(process.argv);
  ```

  We get:

  ```bash
  6
  [
    '/Users/thomasbishop/.nvm/versions/node/v16.10.0/bin/node',
    '/Users/thomasbishop/prepos/testNode.js'
  ]
  ```

  The first item is the last executed process. The second is the array of
  arguments.

  In the array, the first value is a reference to the Node runtime binary. The
  second is the file that was passed to Node.

  If we passed in a parameter we would get that too:

  ```
  node process-demo.js --fakeFlag
  ```

  Gives us:

  ```bash
  [
    "/Users/thomasbishop/.nvm/versions/node/v16.10.0/bin/node",
    "/Users/thomasbishop/prepos/testNode.js",
    "--fake-flag",
  ]
  ```

  When writing Node CLI applications we can write functions that parse standard
  input. For example:

  ```js
  function parseStandardInput(flag) {
    let indexAfterFlag = process.argv.indexOf(flag);
    console.log(process.argv[indexAfterFlag]);
  }
  ```

  Say we ran a program that took in a username:

  ```bash
  node userName.js --user Thomas
  ```

  Then `parseStandardInput("--user")` would give us `"thomas"`

  ## Standard input and output

  Below is the most basic use case: writing to the output.

  ```js
  // alpha.js
  process.stdout.write("Hello from file \n");
  ```

  ```
  node alpha.js
  Hello from file
  ```

  Typically we want to write and read data that the user provides. To do this we
  need to wait on that event. So we use Node's asynchonous
  [event listeners](events.md). We wait
  on the `data` event:

  ```js
  process.stdin.on("data", (data) => {
    process.stdout.write(data.toString().trim());
  });

  process.stdout.write("\n What is your name? \n");
  process.stdout.write(` \n > `);
  ```

  This waits for whatever input is given and then echoes it.

  Currently there is no way to end this process; it will just keep echoing the
  input.

  So we add an exit clause:

  ```js
  process.stdin.on("data", (data) => {
    process.stdout.write(data.toString().trim());
    process.exit();
  });
  ```

  We can also execute functionality on the exit event:

  ```js
  process.on("exit", () => {
    // Do something
  });
  ```

  > Timings can also obviously be managed via the `setTimeout` and `setInterval`
  > methods.

  ### Additional useful methods

  ```js
  process.stdout.clearLine();
  process.stdout.cursorTo([int]);
  ```

  ### `readline`

  The core `readline` module simplifies the handling of standard input and output
  by providing a wrapper. It reads the input stream line by line. Basic usage:

  ```js
  var readline = require("readline");

  var rl = readline.createInterface(process.stdin, process.stdout);

  rl.question("What is your age? ", (age) => {
    console.log("Your age is: " + age);
  });
  ```

  Usage scenarios at: https://www.geeksforgeeks.org/node-js-readline-module/


path: zk/Prop_types_in_React.md
sortable_path: zkProp_types_in_React.md
title: Prop types
checksum: 727c77a483cc8cd4beceb26f9145e1f40aef386f77b3ac6c2f4b43d874e30039
created: 2024-06-16 17:37:31.788664835+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["javascript","react","data-types"]}
lead:
  **The `prop-types` package allows us to introduce type-checking during the
  development runtime of a React application. With larger applications or those
  with complex data flows, this can help reduce errors and bugs during
  development. This is a bit like using TypeScript but on a more minimal level.**

body:
  **The `prop-types` package allows us to introduce type-checking during the
  development runtime of a React application. With larger applications or those
  with complex data flows, this can help reduce errors and bugs during
  development. This is a bit like using TypeScript but on a more minimal level.**

  ## Install

  ```
  npm install prop-types
  ```

  ## Usage

  ```jsx
  import PropTypes from "prop-types";
  ```

  ## Examples

  Say we have a component with the following props:

  ```jsx
  function Page(props) {
    const classes = useStyles();
    return (
      <React.Fragment>
        <Typography className={classes.heading} variant="h1">
          {props.h1}
        </Typography>
        {props.h2 !== "" && <Typography variant="h2">{props.h2}</Typography>}
        {props.paras.map((paragraph, index) => (
          <Typography
            className={classes.paragraph}
            key={index}
            variant="body1"
            gutterBottom
          >
            {paragraph}
          </Typography>
        ))}
        {props.children}
        <Button
          className={classes.button}
          variant="contained"
          color="primary"
          disableElevation
          component={Link}
          to={props.buttonLink}
        >
          {props.buttonText}
        </Button>
      </React.Fragment>
    );
  }
  ```

  Before the final `export default` statement, we would add a PropTypes object
  that specifies the data or reference type that is required by each prop cited in
  the component:

  ```jsx
  Page.propTypes = {
    h1: PropTypes.string.isRequired, // should be string data type, must be present
    h2: PropTypes.string, // as above but optional
    paras: PropTypes.array, // should be array reference type
    buttonLink: PropTypes.string,
    buttonText: PropTypes.string,
    children: PropTypes.node, // can be anything (element, type, JSX) that is rendered to the screen
  };
  ```

  If one of these type checks failed you get a very semantic Warning in the
  console specifying which prop is not consistent with your prop type declaration.

  ## Additional checks

  Above are the most frequent and basic checks, but there are many other
  conditions you can check for:

  ```jsx
  // The full list of types
  [propName]: PropTypes.array,
  [propName]: PropTypes.bool,
  [propName]: PropTypes.func,
  [propName]: PropTypes.number,
  [propName]: PropTypes.object,
  [propName]: PropTypes.string,
  [propName]: PropTypes.symbol,

  // Anything that can be rendered: numbers, strings, elements or an arrayor fragment) containing these types.
  [propName]: PropTypes.node,

  // A function

  [propName]: PropTypes.func

  // An object that could be one of many types
  [objectProp]: PropTypes.oneOfType([
      PropTypes.string,
      PropTypes.number,
    ]),

  // An array of a certain type
  [arrayProp]: PropTypes.arrayOf(PropTypes.number),

  // An object with property values of a certain type
  [objectProp]: PropTypes.objectOf(PropTypes.number),
  ```

raw_content:
  ---
  tags:
    - javascript
    - react
    - data-types
  ---

  # Prop types

  **The `prop-types` package allows us to introduce type-checking during the
  development runtime of a React application. With larger applications or those
  with complex data flows, this can help reduce errors and bugs during
  development. This is a bit like using TypeScript but on a more minimal level.**

  ## Install

  ```
  npm install prop-types
  ```

  ## Usage

  ```jsx
  import PropTypes from "prop-types";
  ```

  ## Examples

  Say we have a component with the following props:

  ```jsx
  function Page(props) {
    const classes = useStyles();
    return (
      <React.Fragment>
        <Typography className={classes.heading} variant="h1">
          {props.h1}
        </Typography>
        {props.h2 !== "" && <Typography variant="h2">{props.h2}</Typography>}
        {props.paras.map((paragraph, index) => (
          <Typography
            className={classes.paragraph}
            key={index}
            variant="body1"
            gutterBottom
          >
            {paragraph}
          </Typography>
        ))}
        {props.children}
        <Button
          className={classes.button}
          variant="contained"
          color="primary"
          disableElevation
          component={Link}
          to={props.buttonLink}
        >
          {props.buttonText}
        </Button>
      </React.Fragment>
    );
  }
  ```

  Before the final `export default` statement, we would add a PropTypes object
  that specifies the data or reference type that is required by each prop cited in
  the component:

  ```jsx
  Page.propTypes = {
    h1: PropTypes.string.isRequired, // should be string data type, must be present
    h2: PropTypes.string, // as above but optional
    paras: PropTypes.array, // should be array reference type
    buttonLink: PropTypes.string,
    buttonText: PropTypes.string,
    children: PropTypes.node, // can be anything (element, type, JSX) that is rendered to the screen
  };
  ```

  If one of these type checks failed you get a very semantic Warning in the
  console specifying which prop is not consistent with your prop type declaration.

  ## Additional checks

  Above are the most frequent and basic checks, but there are many other
  conditions you can check for:

  ```jsx
  // The full list of types
  [propName]: PropTypes.array,
  [propName]: PropTypes.bool,
  [propName]: PropTypes.func,
  [propName]: PropTypes.number,
  [propName]: PropTypes.object,
  [propName]: PropTypes.string,
  [propName]: PropTypes.symbol,

  // Anything that can be rendered: numbers, strings, elements or an arrayor fragment) containing these types.
  [propName]: PropTypes.node,

  // A function

  [propName]: PropTypes.func

  // An object that could be one of many types
  [objectProp]: PropTypes.oneOfType([
      PropTypes.string,
      PropTypes.number,
    ]),

  // An array of a certain type
  [arrayProp]: PropTypes.arrayOf(PropTypes.number),

  // An object with property values of a certain type
  [objectProp]: PropTypes.objectOf(PropTypes.number),
  ```


path: zk/Props_in_React.md
sortable_path: zkProps_in_React.md
title: Props
lead: ## Types or interfaces?
checksum: 5932443b8d7abf3608aa338b3410754fc1976a0a2fc49a45a3a865dba90389b0
created: 2024-06-16 17:37:31.795292542+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["typescript","react"]}
body:
  ## Types or interfaces?

  They are mostly interchangeable but there are some specific differences

  ### Interfaces

  - Classes and other interfaces can extend an interface but not a type

    ```ts
    interface Animal {
      name: string;
    }

    interface Bear extends Animal {
      honey: boolean;
    }
    ```

  - You can add properties to an interface without generating a double declaration
    error:

    ```ts
    interface Window {
        }

    interface Window {
      ts: TypeScriptAPI;
    }
    ```

  ### Types

  You can create unions with types but not interfaces.

  > The consensus seems to be that interfaces should be used over types unless
  > there are ocassions when you do not want a type to be extendable /
  > redeclarable. Or when you want to use unions.

  ## How we type props

  The custom is to type props as an interface. This way we can easily type
  function prop types.

  ```tsx
  interface IProps {
    people: {
      name: string;
      age: number;
      note?: string;
    }[];
  }

  const List: React.FC<IProps> = ({people}: IProps) => {
      return()
  }

  // Note we say that the props into the func component are of type IProps
  // And we destructure the people key

  ```

  Then in the parent:

  ```tsx

  const [people, setPeople] = useState<IState['people']>({});

  <List props={people}>

  ```

  ## Common types for props

  ### Object with any number of properties

  ```tsx
  interface IProps {
    obj: {};
  }
  ```

  ### Array of objects

  ```tsx
  interface IProps {
    val1: string;
    val2: boolean;
  }
  [];
  ```

  ### Functions

  ```tsx

  // Function that doesn't receive or return anything
  interface IProps {
      onClick: () => void;
  }[]

  // Function with named parameter
  interface IProps {
      onClick: (id: number) => void;
  }[]

  // Function that takes an event
  onChange: (event: React.ChangeEvent<HTMLInputElement>) => void;
  ```

  ### Other React components

  ```js
  interface IProps {
    children: React.ReactNode;
  }

  interface IProps {
    children: JSX.Element;
  }
  ```

  `React.ReactNode` covers everything that React can render. This should be
  definitely used when the prop is a child component, otherwise `JSX.element` is
  ok.

  <p style="color: red;">Should I use type or interface? What is consensus?</p>

raw_content:
  ---
  tags:
    - typescript
    - react
  ---

  # Props

  ## Types or interfaces?

  They are mostly interchangeable but there are some specific differences

  ### Interfaces

  - Classes and other interfaces can extend an interface but not a type

    ```ts
    interface Animal {
      name: string;
    }

    interface Bear extends Animal {
      honey: boolean;
    }
    ```

  - You can add properties to an interface without generating a double declaration
    error:

    ```ts
    interface Window {
        }

    interface Window {
      ts: TypeScriptAPI;
    }
    ```

  ### Types

  You can create unions with types but not interfaces.

  > The consensus seems to be that interfaces should be used over types unless
  > there are ocassions when you do not want a type to be extendable /
  > redeclarable. Or when you want to use unions.

  ## How we type props

  The custom is to type props as an interface. This way we can easily type
  function prop types.

  ```tsx
  interface IProps {
    people: {
      name: string;
      age: number;
      note?: string;
    }[];
  }

  const List: React.FC<IProps> = ({people}: IProps) => {
      return()
  }

  // Note we say that the props into the func component are of type IProps
  // And we destructure the people key

  ```

  Then in the parent:

  ```tsx

  const [people, setPeople] = useState<IState['people']>({});

  <List props={people}>

  ```

  ## Common types for props

  ### Object with any number of properties

  ```tsx
  interface IProps {
    obj: {};
  }
  ```

  ### Array of objects

  ```tsx
  interface IProps {
    val1: string;
    val2: boolean;
  }
  [];
  ```

  ### Functions

  ```tsx

  // Function that doesn't receive or return anything
  interface IProps {
      onClick: () => void;
  }[]

  // Function with named parameter
  interface IProps {
      onClick: (id: number) => void;
  }[]

  // Function that takes an event
  onChange: (event: React.ChangeEvent<HTMLInputElement>) => void;
  ```

  ### Other React components

  ```js
  interface IProps {
    children: React.ReactNode;
  }

  interface IProps {
    children: JSX.Element;
  }
  ```

  `React.ReactNode` covers everything that React can render. This should be
  definitely used when the prop is a child component, otherwise `JSX.element` is
  ok.

  <p style="color: red;">Should I use type or interface? What is consensus?</p>


path: zk/Read_command__in_Bash.md
sortable_path: zkRead_command__in_Bash.md
title: read
checksum: c9fb9f3512d6c4406ed8ab423f11a31c5c9e23de462b89c7dd18af357cccc1d0
created: 2024-06-16 17:37:31.802939842+00:00
modified: 2024-06-16 17:24:02.465309515+00:00
metadata: {"tags":["shell"]}
lead:
  The primary use of `read` is to capture user input from `stdin`. It can also be
  used to parse strings or files that are redirected to it (with `<` and `<<`) or
  piped to it. In each case, what is read is stored as a variable.

body:
  The primary use of `read` is to capture user input from `stdin`. It can also be
  used to parse strings or files that are redirected to it (with `<` and `<<`) or
  piped to it. In each case, what is read is stored as a variable.

  `read` will parse line by line using a space (`\n`) as the default delimiter.
  You can use IFS to parse by other characters and/or
  [split the contents into an array](Split_into_array_in_Bash.md).

  ## Example of capturing user input

  When we use `read` to gather user input, the script will pause until the input
  is provided.

  ```bash
  echo "What is your name?"
  read name

  echo "What is your password?"
  read -s pass

  read

  ```

  > If you don't specify variables, `read` will automatically parse using
  > whitespace

  ## Example of piping to read

  Here we use [find](Find_Bash_command.md) to collate the files in
  the current directory and then pipe them to read.

  ```bash
  find -type -f  -not -path "./.git/" | read $fname
  ```

  ## Example of parsing a file

  We will typically read from a source and then do something with each variable
  that `read` returns, e.g:

  ```bash
  while read line; do
    if [ var == 'something' ]; then
      # do something
  done < './input-file.txt
  ```

  ## $REPLY

  If you do not assign a variable name to store the value that `read` reads a
  default (`$REPLY`) is applied. You can reference this value in your code.

  For example the following loop does something if `$REPLY` is equal to an empty
  string:

  ```bash
  while read;
  do
      ((count++))
      if [[ -z "$REPLY" ]]; then
          echo "$count"
      fi
  done < "$input
  ```

raw_content:
  ---
  tags:
    - shell
  ---

  # read

  The primary use of `read` is to capture user input from `stdin`. It can also be
  used to parse strings or files that are redirected to it (with `<` and `<<`) or
  piped to it. In each case, what is read is stored as a variable.

  `read` will parse line by line using a space (`\n`) as the default delimiter.
  You can use IFS to parse by other characters and/or
  [split the contents into an array](Split_into_array_in_Bash.md).

  ## Example of capturing user input

  When we use `read` to gather user input, the script will pause until the input
  is provided.

  ```bash
  echo "What is your name?"
  read name

  echo "What is your password?"
  read -s pass

  read

  ```

  > If you don't specify variables, `read` will automatically parse using
  > whitespace

  ## Example of piping to read

  Here we use [find](Find_Bash_command.md) to collate the files in
  the current directory and then pipe them to read.

  ```bash
  find -type -f  -not -path "./.git/" | read $fname
  ```

  ## Example of parsing a file

  We will typically read from a source and then do something with each variable
  that `read` returns, e.g:

  ```bash
  while read line; do
    if [ var == 'something' ]; then
      # do something
  done < './input-file.txt
  ```

  ## $REPLY

  If you do not assign a variable name to store the value that `read` reads a
  default (`$REPLY`) is applied. You can reference this value in your code.

  For example the following loop does something if `$REPLY` is equal to an empty
  string:

  ```bash
  while read;
  do
      ((count++))
      if [[ -z "$REPLY" ]]; then
          echo "$count"
      fi
  done < "$input
  ```


path: zk/Redirection_in_Bash.md
sortable_path: zkRedirection_in_Bash.md
title: Redirection
lead: ## Redirecting outputs
checksum: e64c3fd247bb692dbe02ecb5f5d645298857b8d8675be4a8a9e3532475b2fc00
created: 2024-06-16 17:37:31.825481372+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["shell"]}
body:
  ## Redirecting outputs

  The symbol `>` is called the **redirection operator** because it redirects
  `stdout` to another program or file. You most frequently use this when you want
  to save contents to a file rather than standard output.

  ```bash
  ls | grep d* > result.txt
  ```

  ### Combining redirection with file escriptors

  It is common practice to combine redirection with the
  [file descriptors](File_descriptors.md) to redirect
  the output of `stdout` and `stderr`. A common case is to
  [redirect error output to `/dev/null`](Redirect_to_dev_null.md).

  Redirection defaults to interpreting `>` as the redirection of `stdout` (`1`);

  ## Redirecting inputs

  We can also switch the direction of the redirection symbol and pass in a file to
  a command rather than command ouput to a file:

  ```bash
  sql-processing-program < data.sql
  ```

  We can redirect a string with three carets:

  ```bash
  program <<< "this is a string input"
  ```

  ## Appending

  We use `>>` to append contents on the next available line of a pre-existing
  file. Continuing on from the example above:

  ```bash
  echo 'These are the files I just grepped' >> result.txt
  ```

raw_content:
  ---
  tags:
    - shell
  ---

  # Redirection

  ## Redirecting outputs

  The symbol `>` is called the **redirection operator** because it redirects
  `stdout` to another program or file. You most frequently use this when you want
  to save contents to a file rather than standard output.

  ```bash
  ls | grep d* > result.txt
  ```

  ### Combining redirection with file escriptors

  It is common practice to combine redirection with the
  [file descriptors](File_descriptors.md) to redirect
  the output of `stdout` and `stderr`. A common case is to
  [redirect error output to `/dev/null`](Redirect_to_dev_null.md).

  Redirection defaults to interpreting `>` as the redirection of `stdout` (`1`);

  ## Redirecting inputs

  We can also switch the direction of the redirection symbol and pass in a file to
  a command rather than command ouput to a file:

  ```bash
  sql-processing-program < data.sql
  ```

  We can redirect a string with three carets:

  ```bash
  program <<< "this is a string input"
  ```

  ## Appending

  We use `>>` to append contents on the next available line of a pre-existing
  file. Continuing on from the example above:

  ```bash
  echo 'These are the files I just grepped' >> result.txt
  ```


path: zk/Restructure_URLs_lambda.md
sortable_path: zkRestructure_URLs_lambda.md
title: AWS Lambda examples
lead: ## Reconstruct CloudFront URLs
checksum: 6a911ce9d3aa21da4b42240dc00369ab6f54b0e35dca6f1cd3d11e804463b857
created: 2024-06-16 17:37:31.847347532+00:00
modified: 2024-06-15 09:50:42.956894178+00:00
metadata: {"tags":["AWS","aws-lambda"]}
body:
  ## Reconstruct CloudFront URLs

  ```js
  exports.handler = (event, context, callback) => {
    // Extract the request from the CloudFront event that is sent to Lambda@Edge
    var request = event.Records[0].cf.request;

    // Extract the URI from the request
    var olduri = request.uri;

    // Match any '/' that occurs at the end of a URI. Replace it with a default index
    var newuri = olduri.replace(/\/$/, "/index.html");

    // Replace the received URI with the URI that includes the index page
    request.uri = newuri;

    // Return to CloudFront
    return callback(null, request);
  };
  ```

  Let's walk through the above example making reference to the general structure
  of the
  [AWS Lambda handler function](Lambda_handler_function.md):

  1. The `event` parameter is invoked to gain access to the event that triggered
     the function. This is a CloudFront request event which is targetted with
     `event.Records[0].cf.request`
  2. We classify this request as the `olduri` since it doesn't have the form we
     want.
  3. We mutate the original request by replacing `/` with `/index.html`
  4. We then re-assign the `request.uri` value to the value of the new URI.
  5. Finally we invoke the `callback` parameter to signal the completion of the
     transformation and return the new request as the return value.

raw_content:
  ---
  tags:
    - AWS
    - aws-lambda
  ---

  # AWS Lambda examples

  ## Reconstruct CloudFront URLs

  ```js
  exports.handler = (event, context, callback) => {
    // Extract the request from the CloudFront event that is sent to Lambda@Edge
    var request = event.Records[0].cf.request;

    // Extract the URI from the request
    var olduri = request.uri;

    // Match any '/' that occurs at the end of a URI. Replace it with a default index
    var newuri = olduri.replace(/\/$/, "/index.html");

    // Replace the received URI with the URI that includes the index page
    request.uri = newuri;

    // Return to CloudFront
    return callback(null, request);
  };
  ```

  Let's walk through the above example making reference to the general structure
  of the
  [AWS Lambda handler function](Lambda_handler_function.md):

  1. The `event` parameter is invoked to gain access to the event that triggered
     the function. This is a CloudFront request event which is targetted with
     `event.Records[0].cf.request`
  2. We classify this request as the `olduri` since it doesn't have the form we
     want.
  3. We mutate the original request by replacing `/` with `/index.html`
  4. We then re-assign the `request.uri` value to the value of the new URI.
  5. Finally we invoke the `callback` parameter to signal the completion of the
     transformation and return the new request as the return value.


path: zk/Routing_on_link_click_in_React.md
sortable_path: zkRouting_on_link_click_in_React.md
title: Test: Routing on link click
lead: ## Component
checksum: 0935aa4e5e1e4051e4ca684aa0735d3b0fb3a555a93412746712f20724dee40b
created: 2024-06-16 17:37:31.858850096+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["javascript","react","testing"]}
body:
  ## Component

  ```jsx
  const CertificateHtmlLink = () => {
    return (
      <div className="certificate-html-link html-version-message">
        <Link className="btn btn-primary ml-2" to="/some-location">
          View HTML version
        </Link>
      </div>
    );
  };
  ```

  ## Test

  ```js
  import { render, fireEvent } from "@testing-library/react";
  import { Router } from "react-router-dom";

  describe("<CertificateHtmlLink", () => {
    it("should link to dynamic HTML certificate page URL on click event", () => {
      // Arrange:
      const history = createMemoryHistory({ initialEntries: ["/"] });

      // Act:
      const { getByText } = render(
        <Router history={history}>
          <CertificateHtmlLink to="/certificate/123" />
        </Router>
      );

      // Assert:
      expect(history.location.pathname).toBe("/");
      fireEvent.click(getByText("View HTML version"));
      expect(history.location.pathname).toBe("/certificate/123");
    });
  });
  ```

raw_content:
  ---
  tags:
    - javascript
    - react
    - testing
  ---

  # Test: Routing on link click

  ## Component

  ```jsx
  const CertificateHtmlLink = () => {
    return (
      <div className="certificate-html-link html-version-message">
        <Link className="btn btn-primary ml-2" to="/some-location">
          View HTML version
        </Link>
      </div>
    );
  };
  ```

  ## Test

  ```js
  import { render, fireEvent } from "@testing-library/react";
  import { Router } from "react-router-dom";

  describe("<CertificateHtmlLink", () => {
    it("should link to dynamic HTML certificate page URL on click event", () => {
      // Arrange:
      const history = createMemoryHistory({ initialEntries: ["/"] });

      // Act:
      const { getByText } = render(
        <Router history={history}>
          <CertificateHtmlLink to="/certificate/123" />
        </Router>
      );

      // Assert:
      expect(history.location.pathname).toBe("/");
      fireEvent.click(getByText("View HTML version"));
      expect(history.location.pathname).toBe("/certificate/123");
    });
  });
  ```


path: zk/Schema_Definition_Language_in_GraphQL.md
sortable_path: zkSchema_Definition_Language_in_GraphQL.md
title: Schema Definition Language
lead: SDL is the formal name for the syntax of GraphQL schemas.
checksum: c50d5b10c7f9a915e1d4d5d9bb67ef337b59f126f97bfda5c6bfeba74d25ac1c
created: 2024-06-16 17:37:31.863579167+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["graphql"]}
body:
  SDL is the formal name for the syntax of GraphQL schemas.

  ## Types

  A schema is a collection of object types that contain fields. Each field has a
  type of its own. A field's type can be a primitive/scalar value (such as an
  `Int` or a `String`), or it can be another object type (just like a custom type
  in TS).

  A schema's type can be non-nullable which is to say, a required field. We
  indicate this with `!`.

  A type for a field can be a collection/array of a given type.

  The following example indicates these properties:

  ```gql
  type Person {
    age: Int
    name: String
    pets: [Pet]!
  }

  type Pet {
    species: String
    name: String
    age: Int
  }
  ```

  ## Queries

  A query is also a schema type but of a special sort.

  > The fields of this type are entry points into the rest of the schema. These
  > are the top-level fields that the client can query for.

  For example if we had this type:

  ```graphql
  type Track {
    id: ID!
      author: Author!
    thumbnail: String
    length: Int
    modulesCount: Int
  }
  ```

  We could define a type to access a give `Track` as follows:

  ```graphql
  type Query {
    tracksForHomePage: [Track!]!
  }
  ```

  Then use this type as the basis for a query:

  ```

  ```

  ## Descriptions

  Descriptions are comments that allow you to document your Schema

  Single line:

  ```gql
  "Single line comment"
  ```

  ```gql
  """
  Multi
  line
  comment
  """
  ```

raw_content:
  ---
  tags: [graphql]
  ---

  # Schema Definition Language

  SDL is the formal name for the syntax of GraphQL schemas.

  ## Types

  A schema is a collection of object types that contain fields. Each field has a
  type of its own. A field's type can be a primitive/scalar value (such as an
  `Int` or a `String`), or it can be another object type (just like a custom type
  in TS).

  A schema's type can be non-nullable which is to say, a required field. We
  indicate this with `!`.

  A type for a field can be a collection/array of a given type.

  The following example indicates these properties:

  ```gql
  type Person {
    age: Int
    name: String
    pets: [Pet]!
  }

  type Pet {
    species: String
    name: String
    age: Int
  }
  ```

  ## Queries

  A query is also a schema type but of a special sort.

  > The fields of this type are entry points into the rest of the schema. These
  > are the top-level fields that the client can query for.

  For example if we had this type:

  ```graphql
  type Track {
    id: ID!
      author: Author!
    thumbnail: String
    length: Int
    modulesCount: Int
  }
  ```

  We could define a type to access a give `Track` as follows:

  ```graphql
  type Query {
    tracksForHomePage: [Track!]!
  }
  ```

  Then use this type as the basis for a query:

  ```

  ```

  ## Descriptions

  Descriptions are comments that allow you to document your Schema

  Single line:

  ```gql
  "Single line comment"
  ```

  ```gql
  """
  Multi
  line
  comment
  """
  ```


path: zk/Secrets_or_env_vars_in_AWS.md
sortable_path: zkSecrets_or_env_vars_in_AWS.md
title: Secrets or environment variables?
checksum: 5dbb25aea37698bb6cf861e39dfd15ee9823b0e0e928f76e9d1d4e3899538a9b
created: 2024-06-16 17:37:31.867266988+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["AWS","aws-lambda"]}
lead:
  Choosing between environment variables and AWS Secrets Manager hinges on factors
  such as use case, security needs, and the data type. Environment variables are
  easier to use, free, and offer fast access but are less secure and lack features
  like versioning. On the other hand, AWS Secrets Manager provides robust security
  through encryption, access control, audit trails, and supports secret versioning
  and rotation. However, it comes with additional costs and can introduce latency
  and rate limits.

body:
  Choosing between environment variables and AWS Secrets Manager hinges on factors
  such as use case, security needs, and the data type. Environment variables are
  easier to use, free, and offer fast access but are less secure and lack features
  like versioning. On the other hand, AWS Secrets Manager provides robust security
  through encryption, access control, audit trails, and supports secret versioning
  and rotation. However, it comes with additional costs and can introduce latency
  and rate limits.

  General recommendations suggest using AWS Secrets Manager for storing sensitive
  data like API keys and passwords, due to its enhanced security features. For
  less sensitive, non-critical data such as feature flags or environment-specific
  URLs, environment variables are often adequate. Cost-conscious users who don't
  need advanced features may find environment variables more economical.
  Ultimately, the best choice depends on your specific requirements and
  limitations.

raw_content:
  ---
  tags: [AWS, aws-lambda]
  ---

  # Secrets or environment variables?

  Choosing between environment variables and AWS Secrets Manager hinges on factors
  such as use case, security needs, and the data type. Environment variables are
  easier to use, free, and offer fast access but are less secure and lack features
  like versioning. On the other hand, AWS Secrets Manager provides robust security
  through encryption, access control, audit trails, and supports secret versioning
  and rotation. However, it comes with additional costs and can introduce latency
  and rate limits.

  General recommendations suggest using AWS Secrets Manager for storing sensitive
  data like API keys and passwords, due to its enhanced security features. For
  less sensitive, non-critical data such as feature flags or environment-specific
  URLs, environment variables are often adequate. Cost-conscious users who don't
  need advanced features may find environment variables more economical.
  Ultimately, the best choice depends on your specific requirements and
  limitations.


path: zk/Setting_up_and_configuring_TypeScript.md
sortable_path: zkSetting_up_and_configuring_TypeScript.md
title: Setup and configuration
lead: ## Installation and configuration
checksum: 7ad962fc516dcea03815580140ae449dfaeb4c74b9e330cc4b8199af5454edf1
created: 2024-06-16 17:37:31.871578494+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["typescript"]}
body:
  ## Installation and configuration

  TypeScript offers custom installations for most modern JS-based frameworks
  including Webpack, React.js and Vue.js. The instructions below cover minimal TS
  set-up outside of a specific framework. TypeScript adds an additional step to
  any build process since your code must compile to JS before any bundling and
  transpilation can occur.

  ## Installation

  ```
  mkdir typescript-project
  cd typescript-project
  npm i typescript --save-dev
  ```

  ## Initialise project

  ```
  npx tsc --init
  ```

  ## Basic configuration

  ```json
  "compilerOptions": {
  	"target" : "es2020", //es2015 for prod
  	"module" : "es2020",
    	"allowJs": true,
    	"checkJs": true,
    	"strict": true,
    	"esModuleInterop": true
  }
  ```

  ### Specify output paths (for production code)

  ```json
  "compilerOptions": {
  	...
  	"outDir": "dist",
  	"sourceMap": true,
  }
  ```

  ### Compile-time commands

  ```
  tsc --noEmit
  ```

  ```
  tsc --noEmit --watch
  ```

  ## Global requirements

  ```
  npm install -g typescript
  npm install -g ts-node
  ```

  `ts-node` allows you to run TS files the way you would with standard JS, i.e.
  `node [filename]`, you just use `ts-node filename` .

  This is a convenience that saves you compiling from TS → JS and then running
  Node against the compiled JS. Which is useful for debugging and checking values
  as you work.

  ## Imports and exports

  You may wish to define your custom types outside of their application, for
  instance in a `/types/` directory. The custom type or types can then be imported
  using standard ES6 imports:

  ```tsx
  export type Article = {
  		price: number,
  }

  export type AnotherType = {
  ...
  }
  ```

  ```tsx
  import { Article, AnotherType } from "./types/allTypes.js";
  import Article from "./types/allTypes.js";
  ```

  TypeScript also provides a specific import keyword `type` so that you can
  distinguish type imports from other module imports. This is handy when you have
  multiple components in the one file:

  ```tsx
  import type { Article } from "./types/allTypes.js";
  ```

raw_content:
  ---
  tags:
    - typescript
  ---

  # Setup and configuration

  ## Installation and configuration

  TypeScript offers custom installations for most modern JS-based frameworks
  including Webpack, React.js and Vue.js. The instructions below cover minimal TS
  set-up outside of a specific framework. TypeScript adds an additional step to
  any build process since your code must compile to JS before any bundling and
  transpilation can occur.

  ## Installation

  ```
  mkdir typescript-project
  cd typescript-project
  npm i typescript --save-dev
  ```

  ## Initialise project

  ```
  npx tsc --init
  ```

  ## Basic configuration

  ```json
  "compilerOptions": {
  	"target" : "es2020", //es2015 for prod
  	"module" : "es2020",
    	"allowJs": true,
    	"checkJs": true,
    	"strict": true,
    	"esModuleInterop": true
  }
  ```

  ### Specify output paths (for production code)

  ```json
  "compilerOptions": {
  	...
  	"outDir": "dist",
  	"sourceMap": true,
  }
  ```

  ### Compile-time commands

  ```
  tsc --noEmit
  ```

  ```
  tsc --noEmit --watch
  ```

  ## Global requirements

  ```
  npm install -g typescript
  npm install -g ts-node
  ```

  `ts-node` allows you to run TS files the way you would with standard JS, i.e.
  `node [filename]`, you just use `ts-node filename` .

  This is a convenience that saves you compiling from TS → JS and then running
  Node against the compiled JS. Which is useful for debugging and checking values
  as you work.

  ## Imports and exports

  You may wish to define your custom types outside of their application, for
  instance in a `/types/` directory. The custom type or types can then be imported
  using standard ES6 imports:

  ```tsx
  export type Article = {
  		price: number,
  }

  export type AnotherType = {
  ...
  }
  ```

  ```tsx
  import { Article, AnotherType } from "./types/allTypes.js";
  import Article from "./types/allTypes.js";
  ```

  TypeScript also provides a specific import keyword `type` so that you can
  distinguish type imports from other module imports. This is handy when you have
  multiple components in the one file:

  ```tsx
  import type { Article } from "./types/allTypes.js";
  ```


path: zk/Split_into_array_in_Bash.md
sortable_path: zkSplit_into_array_in_Bash.md
title: Splitting input into an array
lead: ## readarray
checksum: b613cfe6a16f00e820880c671aeeccbf3f0ecdae9e3770fc68016d7b50c6b413
created: 2024-06-16 17:37:31.906852767+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["shell"]}
body:
  ## readarray

  `readarray` makes it really easy to split input into an array based on new
  lines. Say we have this file as input:

  ```
  123
  456
  789
  ```

  Then we can split like so:

  ```bash
  readarray -t name_for_array < ./input.text

  # Print all elements
  echo "${name_for_array[@]}"

  # Print element by index

  echo "${name_for_array[1]}"
  456
  ```

  If we want to read a string directly:

  ```bash
  readarray -t new_name_for_array <<< "here
  is
  some
  text"

  echo "${new_name_for_array[1]}"
  is
  ```

  > The _-t_ flag removes the trailing newline

  See more: https://linuxhint.com/split-string-array-bash/

  ## read

  For delimiters other than a space we have to use `read`, combined with `IFS` the
  **Internal Field Separator**.

  For example, to split by comma:

  ```plaintext
  # comma-input.txt
  something,something else,something more
  ```

  ```bash
  IFS=',' read -a arr < ./comma_inputs.txt
  ```

  > We use the `-a` flag to signal that we want each item read returned as an
  > element in an array

  ## mapfile

  We can use `mapfile` to read the lines of a input file in set segments.

  For the following file:

  ```
  line one
  line two
  line three
  line four
  ```

  We could generate the following output:

  ```
  -----SNIP-----
  line one
  line two
  -----SNIP-----
  line three
  line four
  ```

  With:

  ```bash
  while mapfile -t -n 2 ary && ((${#ary[@]})); do
    printf '%s\n' "${ary[@]}"
    printf -- '--- SNIP ---\n'
   done < $testInput
  ```

  So, while there are input lines, map each line into an array and return it in
  groups of 2.

raw_content:
  ---
  tags:
    - shell
  ---

  # Splitting input into an array

  ## readarray

  `readarray` makes it really easy to split input into an array based on new
  lines. Say we have this file as input:

  ```
  123
  456
  789
  ```

  Then we can split like so:

  ```bash
  readarray -t name_for_array < ./input.text

  # Print all elements
  echo "${name_for_array[@]}"

  # Print element by index

  echo "${name_for_array[1]}"
  456
  ```

  If we want to read a string directly:

  ```bash
  readarray -t new_name_for_array <<< "here
  is
  some
  text"

  echo "${new_name_for_array[1]}"
  is
  ```

  > The _-t_ flag removes the trailing newline

  See more: https://linuxhint.com/split-string-array-bash/

  ## read

  For delimiters other than a space we have to use `read`, combined with `IFS` the
  **Internal Field Separator**.

  For example, to split by comma:

  ```plaintext
  # comma-input.txt
  something,something else,something more
  ```

  ```bash
  IFS=',' read -a arr < ./comma_inputs.txt
  ```

  > We use the `-a` flag to signal that we want each item read returned as an
  > element in an array

  ## mapfile

  We can use `mapfile` to read the lines of a input file in set segments.

  For the following file:

  ```
  line one
  line two
  line three
  line four
  ```

  We could generate the following output:

  ```
  -----SNIP-----
  line one
  line two
  -----SNIP-----
  line three
  line four
  ```

  With:

  ```bash
  while mapfile -t -n 2 ary && ((${#ary[@]})); do
    printf '%s\n' "${ary[@]}"
    printf -- '--- SNIP ---\n'
   done < $testInput
  ```

  So, while there are input lines, map each line into an array and return it in
  groups of 2.


path: zk/Streams_in_NodeJS.md
sortable_path: zkStreams_in_NodeJS.md
title: Handling streams with fs
checksum: f8299b5e6a1242c1e024952c4e37aaa85dbc54a9b22f1a262b3658605e2ec655
created: 2024-06-16 17:37:31.912375997+00:00
modified: 2024-06-15 09:50:45.043601129+00:00
metadata: {"tags":["node-js"]}
lead:
  When reading from a file, the
  [`fs.readFile()`](fs.md) method waits
  until the entire file has been read before executing the callback. It's obvious
  why this might not be ideal in certain cases. If the file is very large you are
  utilising a lot of [memory](Memory.md) for a
  single process. Additionally, the data you need might appear early in the file,
  in which case, once you find the data you want, there is no need to read to the
  end of the file.

body:
  When reading from a file, the
  [`fs.readFile()`](fs.md) method waits
  until the entire file has been read before executing the callback. It's obvious
  why this might not be ideal in certain cases. If the file is very large you are
  utilising a lot of [memory](Memory.md) for a
  single process. Additionally, the data you need might appear early in the file,
  in which case, once you find the data you want, there is no need to read to the
  end of the file.

  > An example of this in practice is watching a Netflix film: we don't have to
  > wait for the whole film to download, we can start watching it immediately
  > because it is passed to us in chunks.

  ## Reading streams

  This is why the ability to read files and data as streams exists in Node.js.
  Streams promote memory and time efficiency. As you are not necessarily reading
  the whole file, you can extract the data you need quickly and you are not
  putting the whole file in memory.

  When read as a stream, the file is broken up into smaller chunks. Readable
  streams raise data events and pass the chunks to our callbacks. Hence we don't
  have to wait for the whole file before registering data chunks.

  ## Implementation

  Instead of, e.g :

  ```js
  fs.readFile("./lorem.md", "UTF-8", function (err, fileContents) {
    console.log(fileContents);
  });
  ```

  We would:

  ```js
  let stream = fs.createReadStream("./lorem.md", "UTF-8");

  let data;

  stream.once("data", (chunk) => {
    console.log("read stream started");
  });

  stream.on("data", (chunk) => {
    data += chunk;
    console.log(data);
  });

  stream.on("end", () => console.log("stream ends"));
  ```

  We use `once` to do something at the moment the stream starts to be read.

  We use `on` to target the `"data"` and `"end"` events. We use this to log the
  chunks via the `data` variable.

  ## Writing streams

  When we write as a stream, we break data up and pass this to a reader as a
  stream. It is what creates a stream in the first place.

  For example we could create a stream that handles `stdin` and writes this to a
  file in chunks:

  ```js
  var stream = fs.createWriteStream("lorem.txt")

  process.stdin.once("data", (data) => {
      stream.write(data, function() => {
          // data is now written.
      })
  })

  process.on("exit", function () {
      stream.close()
  })
  ```

  Note that we always have to close the stream on exit

raw_content:
  ---
  tags:
    - node-js
  ---

  # Handling streams with fs

  When reading from a file, the
  [`fs.readFile()`](fs.md) method waits
  until the entire file has been read before executing the callback. It's obvious
  why this might not be ideal in certain cases. If the file is very large you are
  utilising a lot of [memory](Memory.md) for a
  single process. Additionally, the data you need might appear early in the file,
  in which case, once you find the data you want, there is no need to read to the
  end of the file.

  > An example of this in practice is watching a Netflix film: we don't have to
  > wait for the whole film to download, we can start watching it immediately
  > because it is passed to us in chunks.

  ## Reading streams

  This is why the ability to read files and data as streams exists in Node.js.
  Streams promote memory and time efficiency. As you are not necessarily reading
  the whole file, you can extract the data you need quickly and you are not
  putting the whole file in memory.

  When read as a stream, the file is broken up into smaller chunks. Readable
  streams raise data events and pass the chunks to our callbacks. Hence we don't
  have to wait for the whole file before registering data chunks.

  ## Implementation

  Instead of, e.g :

  ```js
  fs.readFile("./lorem.md", "UTF-8", function (err, fileContents) {
    console.log(fileContents);
  });
  ```

  We would:

  ```js
  let stream = fs.createReadStream("./lorem.md", "UTF-8");

  let data;

  stream.once("data", (chunk) => {
    console.log("read stream started");
  });

  stream.on("data", (chunk) => {
    data += chunk;
    console.log(data);
  });

  stream.on("end", () => console.log("stream ends"));
  ```

  We use `once` to do something at the moment the stream starts to be read.

  We use `on` to target the `"data"` and `"end"` events. We use this to log the
  chunks via the `data` variable.

  ## Writing streams

  When we write as a stream, we break data up and pass this to a reader as a
  stream. It is what creates a stream in the first place.

  For example we could create a stream that handles `stdin` and writes this to a
  file in chunks:

  ```js
  var stream = fs.createWriteStream("lorem.txt")

  process.stdin.once("data", (data) => {
      stream.write(data, function() => {
          // data is now written.
      })
  })

  process.on("exit", function () {
      stream.close()
  })
  ```

  Note that we always have to close the stream on exit


path: zk/Strings_in_Bash.md
sortable_path: zkStrings_in_Bash.md
title: Strings in bash
lead: ## Return a substring by index
checksum: 2530f2db53c8899fd0d79f4d754052143e4479eb4e64620e2cc4cc233aca8321
created: 2024-06-16 17:37:31.925959714+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["shell","data-types"]}
body:
  ## Return a substring by index

  ```bash
  myString="hello"
  substring=${myString:0:3}
  # hel
  ```

  This is often used when looping through each character in a string.

  ## Loop through each character in a string

  ```bash
  str="hallelujah"
  stringLength=$(expr length str)

  for (( i=0; i<=${stringLength}; i++ )); do
      echo "${str:$i:1}"
  done
  ```

raw_content:
  ---
  tags:
    - shell
    - data-types
  ---

  # Strings in bash

  ## Return a substring by index

  ```bash
  myString="hello"
  substring=${myString:0:3}
  # hel
  ```

  This is often used when looping through each character in a string.

  ## Loop through each character in a string

  ```bash
  str="hallelujah"
  stringLength=$(expr length str)

  for (( i=0; i<=${stringLength}; i++ )); do
      echo "${str:$i:1}"
  done
  ```


path: zk/Text_manipulation_in_Bash.md
sortable_path: zkText_manipulation_in_Bash.md
title: Text manipulation
lead: ## Sorting strings: `sort`
checksum: 16bd858534a5e8bb07228d40c6a8984bad8a4b74cf7a8c1b32bae9ae2ba8dfe1
created: 2024-06-16 17:37:31.952865941+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["shell"]}
body:
  ## Sorting strings: `sort`

  If you have a `.txt` file containing text strings, each on a new line you can
  use the sort function to quickly put them in alphabetical order:

  ```bash
  sort file.txt
  ```

  Note that this will not save the sort, it only presents it as a standard output.
  To save the sort you need to direct the sort to a file in the standard way:

  ```bash
  sort file.txt > output.txt
  ```

  ### Options

  - `-r`
    - reverse sort
  - `c`
    - check if file is already sorted. If not, it will highlight the strings which
      are not sorted

  ## Find and replace: `sed`

  The `sed` programme can be used to implement find and replace procedures. In
  `sed`, find and replace are covered by the substitution option: `/s` :

  ```bash
  sed ‘s/word/replacement word/’ file.txt
  ```

  This however will only change the first instance of word to be replaced, in
  order to apply to every instance you need to add the global option: `-g` .

  As sed is a stream editor, any changes you make using it, will only occur within
  the standard output , they will not be saved to file. In order to save to file
  you need to specify a new file output (using `> output.txt`) in addition to the
  original file. This hasthe benefit of leaving the original file untouched whilst
  ensuring the desired outcome is stored permanently.

  Alternatively, you can use the `-i` option which will make the changes take
  place in the source file as well as in standard input.

  Note that this will overwrite the original version of the file and it cannot be
  regained. If this is an issue then it is recommended to include a backup command
  in the overall argument like so:

  ```bash
  sed -i.bak ‘s/word/replacement word/’ file.txt
  ```

  This will create the file `file.txt.bak` in the directory you are working within
  which is the original file before the replacement was carried out.

  ### Remove duplicates

  We can use the `sort -u` command can be used to remove duplicates:

  ```bash
  sort -u file.txt
  ```

  It is important to sort before attempting to remove duplicates since the `-u`
  flag works on the basis of the strings being adjacent.

  ## Split a large file into multiple smaller files: `split`

  Suppose you have a file containing 1000 lines. You want to break the file up
  into five separate files, each containing two hundred lines. You can use `split`
  to accomplish this, like so:

  ```bash
  split -l 200 big-file.txt new-files
  ```

  `split` will categorise the resulting five files as follows:

  - new-file-aa,
  - new-file-ab
  - new-file-ac,
  - newfile-ad,
  - new-file-ae.

  If you would rather have numeric suffixes, use the option `-d` . You can also
  split a file by its number of bytes, using the option `-b` and specifying a
  constituent file size.

  ## Merge multiple files into one with `cat`

  We can use `cat` read multiple files at once and then append a redirect to save
  them to a file:

  ```bash
  cat file_a.txt file_b.txt file_c.txt > merged-file.txt
  ```

  ## Count lines, words, etc: `wc`

  To count words:

  ```bash
  wc file.txt
  ```

  When we use the command three numbers are outputted, in order: lines, words,
  bytes.

  You can use modifiers to get just one of the numbers: `-l`, `-w` , `-b` .

raw_content:
  ---
  tags:
    - shell
  ---

  # Text manipulation

  ## Sorting strings: `sort`

  If you have a `.txt` file containing text strings, each on a new line you can
  use the sort function to quickly put them in alphabetical order:

  ```bash
  sort file.txt
  ```

  Note that this will not save the sort, it only presents it as a standard output.
  To save the sort you need to direct the sort to a file in the standard way:

  ```bash
  sort file.txt > output.txt
  ```

  ### Options

  - `-r`
    - reverse sort
  - `c`
    - check if file is already sorted. If not, it will highlight the strings which
      are not sorted

  ## Find and replace: `sed`

  The `sed` programme can be used to implement find and replace procedures. In
  `sed`, find and replace are covered by the substitution option: `/s` :

  ```bash
  sed ‘s/word/replacement word/’ file.txt
  ```

  This however will only change the first instance of word to be replaced, in
  order to apply to every instance you need to add the global option: `-g` .

  As sed is a stream editor, any changes you make using it, will only occur within
  the standard output , they will not be saved to file. In order to save to file
  you need to specify a new file output (using `> output.txt`) in addition to the
  original file. This hasthe benefit of leaving the original file untouched whilst
  ensuring the desired outcome is stored permanently.

  Alternatively, you can use the `-i` option which will make the changes take
  place in the source file as well as in standard input.

  Note that this will overwrite the original version of the file and it cannot be
  regained. If this is an issue then it is recommended to include a backup command
  in the overall argument like so:

  ```bash
  sed -i.bak ‘s/word/replacement word/’ file.txt
  ```

  This will create the file `file.txt.bak` in the directory you are working within
  which is the original file before the replacement was carried out.

  ### Remove duplicates

  We can use the `sort -u` command can be used to remove duplicates:

  ```bash
  sort -u file.txt
  ```

  It is important to sort before attempting to remove duplicates since the `-u`
  flag works on the basis of the strings being adjacent.

  ## Split a large file into multiple smaller files: `split`

  Suppose you have a file containing 1000 lines. You want to break the file up
  into five separate files, each containing two hundred lines. You can use `split`
  to accomplish this, like so:

  ```bash
  split -l 200 big-file.txt new-files
  ```

  `split` will categorise the resulting five files as follows:

  - new-file-aa,
  - new-file-ab
  - new-file-ac,
  - newfile-ad,
  - new-file-ae.

  If you would rather have numeric suffixes, use the option `-d` . You can also
  split a file by its number of bytes, using the option `-b` and specifying a
  constituent file size.

  ## Merge multiple files into one with `cat`

  We can use `cat` read multiple files at once and then append a redirect to save
  them to a file:

  ```bash
  cat file_a.txt file_b.txt file_c.txt > merged-file.txt
  ```

  ## Count lines, words, etc: `wc`

  To count words:

  ```bash
  wc file.txt
  ```

  When we use the command three numbers are outputted, in order: lines, words,
  bytes.

  You can use modifiers to get just one of the numbers: `-l`, `-w` , `-b` .


path: zk/The_kernel.md
sortable_path: zkThe_kernel.md
title: The kernel
checksum: 96c9ce4e7d5964c10b9af81b81e350210f24146bc8a16245169ff057cdb3dd4f
created: 2024-06-16 17:37:31.958371212+00:00
modified: 2024-06-21 05:06:21.705739867+00:00
metadata: {"tags":["systems-programming","memory","operating-systems","computer-architecture"]}
lead:
  Within the [hierarchy of the OS](./Basic_model_of_the_operating_system.md), the
  kernel acts as the primary mediator between the hardware (CPU, memory) and
  [user](./User_Space.md) [processes](Processes.md). Let's look at each of its
  responsibilities in greater depth:

body:
  Within the [hierarchy of the OS](./Basic_model_of_the_operating_system.md), the
  kernel acts as the primary mediator between the hardware (CPU, memory) and
  [user](./User_Space.md) [processes](Processes.md). Let's look at each of its
  responsibilities in greater depth:

  - process management
  - memory management
  - device drivers
  - system calls

  ## Process management

  > A process is just another name for a running program. Process management is
  > the starting, pausing, resuming, scheduling and terminating of processes.

  On modern computers it appears that multiple processes can run simultaneously at
  once. This is only because the processor is so fast that we do not detect
  changes. In fact access to the CPU is always sequential. The sequence in which
  multiple programs are allowed to access the CPU is managed by the kernel.

  > Consider a system with a one-core CPU. Many processes may be _able_ to use the
  > CPU, but only one process can actually use the CPU at any given time...Each
  > process uses the CPU for a fraction of a second, then pauses, then another
  > process uses it for a fraction of a second and so on... (_How Linux Works:
  > Third Edition_, Brian Ward 2021)

  This process of the CPU shuffling between multiple processes is called _context
  switching_.

  The role of the kernel in facilitating this, is as follows:

  1. CPU runs process for a time slice based on its internal time. Then hands
     control back to the kernel (kernel mode)
  2. Kernel records current state of CPU and memory. This is necessary in order to
     resume the progress that was just interupted.
  3. The kernel executes any tasks that arose in the last timeslice executed by
     the CPU (e.g. collecting data from I/0)
  4. Kernel then analyses the list of processes that are ready to run next and
     chooses one.
  5. Kernel prepares memory for this new process and prepares the CPU.
  6. Kernel tells CPU how long the time slice for the new process will last.
  7. Kernel switches the CPU into user mode and hands control of CPU to the
     process.

  ## Memory management

  During the context switch from CPU to user space, the kernel allocates memory.
  It has the following jobs to manage:

  - Keeping its own private area in memory for itself that user processes cannot
    access
  - Assigning each user process its own section of memory
  - Managing shared memory between processes and ensuring the private memory of
    processes is not accessed by others
  - Managing read-only memory
  - Allowing for the use of disk space as auxiliary memory

  > Modern CPUs include a
  > [memory management unit](Virtual_memory_and_the_MMU_in_Linux.md#the-memory-management-unit-mmu)
  > which provides the kernel with **virtual** memory. In this scenario, memory
  > isn't directly accessed by the process instead it works on the assumption that
  > is has access to the entire memory of the machine and this is then translated
  > into a map that is applied to the real memory and managed by the kernel.

  ## Device drivers

  Devices are managed by the kernal and are not accessible directly via user
  space, since improper usage could crash the whole machine. There is little
  uniformity between devices and as a result drivers are needed. Thes are kernl
  code that enable different OS kernels to access and control the devices.

  ## System calls

  Syscalls are what enable programs to start and are required for the acts of
  opening, reading and writing files. System calls in Linux are typically managed
  via C.

  In Linux there are two particularly important system calls:

  - `fork()`
    - When a process calls fork, the kernel creates a nearly identical copy of
      this running process
  - `exec()`
    - When a process calls exec it passes a program name as a parameter. Then the
      kernel loads and starts this program, replacing the current process.

  Example with a terminal program like `ls`:

  > When you enter `ls` into the terminal window, the shell that's running inside
  > the terminal window calls `fork()` to create a copy of the shell, and then the
  > new copy of the shell calls `exec(ls)` to run `ls`. (_Ibid._)

  ## Controlling processes

  In Linux we can view, kill, pause and resume processes using [ps](Processes.md).

raw_content:
  ---
  tags:
    - systems-programming
    - memory
    - operating-systems
    - computer-architecture
  ---

  # The kernel

  Within the [hierarchy of the OS](./Basic_model_of_the_operating_system.md), the
  kernel acts as the primary mediator between the hardware (CPU, memory) and
  [user](./User_Space.md) [processes](Processes.md). Let's look at each of its
  responsibilities in greater depth:

  - process management
  - memory management
  - device drivers
  - system calls

  ## Process management

  > A process is just another name for a running program. Process management is
  > the starting, pausing, resuming, scheduling and terminating of processes.

  On modern computers it appears that multiple processes can run simultaneously at
  once. This is only because the processor is so fast that we do not detect
  changes. In fact access to the CPU is always sequential. The sequence in which
  multiple programs are allowed to access the CPU is managed by the kernel.

  > Consider a system with a one-core CPU. Many processes may be _able_ to use the
  > CPU, but only one process can actually use the CPU at any given time...Each
  > process uses the CPU for a fraction of a second, then pauses, then another
  > process uses it for a fraction of a second and so on... (_How Linux Works:
  > Third Edition_, Brian Ward 2021)

  This process of the CPU shuffling between multiple processes is called _context
  switching_.

  The role of the kernel in facilitating this, is as follows:

  1. CPU runs process for a time slice based on its internal time. Then hands
     control back to the kernel (kernel mode)
  2. Kernel records current state of CPU and memory. This is necessary in order to
     resume the progress that was just interupted.
  3. The kernel executes any tasks that arose in the last timeslice executed by
     the CPU (e.g. collecting data from I/0)
  4. Kernel then analyses the list of processes that are ready to run next and
     chooses one.
  5. Kernel prepares memory for this new process and prepares the CPU.
  6. Kernel tells CPU how long the time slice for the new process will last.
  7. Kernel switches the CPU into user mode and hands control of CPU to the
     process.

  ## Memory management

  During the context switch from CPU to user space, the kernel allocates memory.
  It has the following jobs to manage:

  - Keeping its own private area in memory for itself that user processes cannot
    access
  - Assigning each user process its own section of memory
  - Managing shared memory between processes and ensuring the private memory of
    processes is not accessed by others
  - Managing read-only memory
  - Allowing for the use of disk space as auxiliary memory

  > Modern CPUs include a
  > [memory management unit](Virtual_memory_and_the_MMU_in_Linux.md#the-memory-management-unit-mmu)
  > which provides the kernel with **virtual** memory. In this scenario, memory
  > isn't directly accessed by the process instead it works on the assumption that
  > is has access to the entire memory of the machine and this is then translated
  > into a map that is applied to the real memory and managed by the kernel.

  ## Device drivers

  Devices are managed by the kernal and are not accessible directly via user
  space, since improper usage could crash the whole machine. There is little
  uniformity between devices and as a result drivers are needed. Thes are kernl
  code that enable different OS kernels to access and control the devices.

  ## System calls

  Syscalls are what enable programs to start and are required for the acts of
  opening, reading and writing files. System calls in Linux are typically managed
  via C.

  In Linux there are two particularly important system calls:

  - `fork()`
    - When a process calls fork, the kernel creates a nearly identical copy of
      this running process
  - `exec()`
    - When a process calls exec it passes a program name as a parameter. Then the
      kernel loads and starts this program, replacing the current process.

  Example with a terminal program like `ls`:

  > When you enter `ls` into the terminal window, the shell that's running inside
  > the terminal window calls `fork()` to create a copy of the shell, and then the
  > new copy of the shell calls `exec(ls)` to run `ls`. (_Ibid._)

  ## Controlling processes

  In Linux we can view, kill, pause and resume processes using [ps](Processes.md).


path: zk/Troubleshooting_MongoDB.md
sortable_path: zkTroubleshooting_MongoDB.md
title: Troubleshooting MongoDB connection issues
lead: ## Arch Linux troubleshooting
checksum: 764b19f1d36a6c0d05f28bf1f30d52c829fe3030cef31f8e3c8efabf28f1f4c3
created: 2024-06-16 17:37:32.020694929+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["mongo-db"]}
body:
  ## Arch Linux troubleshooting

  Most times any problems will be a result of a Mongo process that is already
  running. Resolve with:

  ```bash
  # Kill running Mongo process...
  # Identify PID
  sudo lsof -iTCP -sTCP:LISTEN -n -P
  sudo kill [pid]
  ```

  Otherwise try the below.

  ```bash
  # Check that the systemd service is runnign for Mongo
  sudo systemctl status mongodb

  # If not, enable and start service (only need to do this once for machine)
  sudo systemctl enable --now mongodb
  sudo systemctl start --now mongodb

  # If issues with Mongo daemon...
  # Remove the current socket
  rm /tmp/mongodb-27017.sock
  ```

  For additional command line utilities you will also need to install
  `mongodb-tools` from the AUR.

raw_content:
  ---
  tags: [mongo-db]
  ---

  # Troubleshooting MongoDB connection issues

  ## Arch Linux troubleshooting

  Most times any problems will be a result of a Mongo process that is already
  running. Resolve with:

  ```bash
  # Kill running Mongo process...
  # Identify PID
  sudo lsof -iTCP -sTCP:LISTEN -n -P
  sudo kill [pid]
  ```

  Otherwise try the below.

  ```bash
  # Check that the systemd service is runnign for Mongo
  sudo systemctl status mongodb

  # If not, enable and start service (only need to do this once for machine)
  sudo systemctl enable --now mongodb
  sudo systemctl start --now mongodb

  # If issues with Mongo daemon...
  # Remove the current socket
  rm /tmp/mongodb-27017.sock
  ```

  For additional command line utilities you will also need to install
  `mongodb-tools` from the AUR.


path: zk/Type_guarding_and_narrowing_in_TS.md
sortable_path: zkType_guarding_and_narrowing_in_TS.md
title: Type narrowing and guarding
checksum: 490017e14d24edb1d0dbb214c845e7c0f56679e29856822fd555a7f4d46e88b5
created: 2024-06-16 17:37:32.164007308+00:00
modified: 2024-06-16 17:27:18.625450007+00:00
metadata: {"tags":["typescript"]}
lead:
  Type narrowing is the process of working out from a supertype like
  [any](Any.md) or [unknown](Unknown_type_in_TS.md) whic type the value should be in the
  x course of your code. This is necessary since a process will not necessarily
  always return or involve homogenous types.

body:
  Type narrowing is the process of working out from a supertype like
  [any](Any.md) or [unknown](Unknown_type_in_TS.md) whic type the value should be in the
  x course of your code. This is necessary since a process will not necessarily
  always return or involve homogenous types.

  Related to this is type guarding: ensuring that a value is of the suitable type
  as a factor of control flow. For instance using typeof to ensure that an input
  is numerical before proceeding with a function's logic.

raw_content:
  ---
  tags:
    - typescript
  ---

  # Type narrowing and guarding

  Type narrowing is the process of working out from a supertype like
  [any](Any.md) or [unknown](Unknown_type_in_TS.md) whic type the value should be in the
  x course of your code. This is necessary since a process will not necessarily
  always return or involve homogenous types.

  Related to this is type guarding: ensuring that a value is of the suitable type
  as a factor of control flow. For instance using typeof to ensure that an input
  is numerical before proceeding with a function's logic.


path: zk/Union_types_in_TS.md
sortable_path: zkUnion_types_in_TS.md
title: Union types
checksum: c821ae262bc36f541ebebc61d22c3d5bd95bb51fa3dd7b44522c80fe2956ee3f
created: 2024-06-16 17:37:32.182332489+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["typescript"]}
lead:
  Like intersection types, union types allow us to combine the properties of
  existing primitive and custom types to make composites.

body:
  Like intersection types, union types allow us to combine the properties of
  existing primitive and custom types to make composites.

  Intersection types work well when you have complete knowledge of what form your
  data will take. This approach only works if you know in advance that your data
  will be and that it will share properties from more than one type. But what if
  you don't have this knowledge? You only know that it will be one of the three
  possible types? This is where union types become helpful. Instead of saying that
  event X will be of type A we say that event X may be one of types A or B or C.

  > A union type is a type formed from two or more other types, representing
  > values that may be any one of those types. We refer to each of these types as
  > the union’s members.

  ## Demonstration

  ```ts
  type Petrol = {
    maxSpeed: number;
    tankVolume: number; // unique property
  };

  type Electric = {
    maxSpeed: number;
    maxCharge: number; // unique property
  };

  type Car = Petrol | Electric;

  const tesla: Car = {
    maxSpeed: 120,
    maxCharge: 22,
  };
  ```

  - We have defined two custom types: `Petrol` and `Electric`
  - We have unified them to make a new custom type of `Car`
  - `tesla` is an instance of the union type

  ## Usage with functions

  Union type syntax is not limited to the declaration phase. You can also use them
  as parameters to functions or function return types:

  ```ts
  function generateCar(car: Petrol | Electric): void {
    console.log(`The car has a max speed of ${car.maxSpeed}`);
  }
  ```

  ## Unifying primitive and custom types

  We can also use unions with primitive types as well as custom types:

  ```ts
  function printId(id: number | string): void {
    console.log(`Your id is ${id}`);
  }
  ```

  We can also combine primitive and custom types in the same union:

  ```ts
  function printStringOrNumber(): string | Petrol {}
  ```

  ## Set theory: understanding errors

  In JavaScript we have the primitive types of string , number , boolean etc. In
  TS, these types are distinct and exclusive: if you declare that x : string you
  cannot redefine it as boolean or expect Boolean properties and methods to belong
  to a string type. For instance a string type is incapable of having as its value
  false.

  This is not true in the case of unions.

  With a union of `string | boolean` a value `x` could be `'banana'` or `true`. We
  are allowing for all values that are either from the set string or the set
  boolean. Logically this is equivalent to the OR operator: a value of type
  `string | boolean` could be one of infinite possible strings or true/false. With
  union types then, we radically widen the scope of possible values for the type.
  a

  These constraints don't just apply to the values themselves but the methods and
  properties you can use with the values. For instance a boolean doesn't have the
  property of `length`. Therefore if you have a union type of `string|boolean` you
  are going to get an error if you apply `length` to it since **the property must
  exist on both types that comprise the union**. The way round this is to use type
  narrowing and execute control flow based on the parameter that is passed to a
  function that is declared to be of this mixed type.

  Here is an example of this:

  ```ts
  function printId(id: number | string) {
    if (typeof id === "string") {
      // In this branch, id is of type 'string'
      console.log(id.toUpperCase());
    } else {
      // Here, id is of type 'number'
      console.log(id);
    }
  }
  ```

raw_content:
  ---
  tags:
    - typescript
  ---

  # Union types

  Like intersection types, union types allow us to combine the properties of
  existing primitive and custom types to make composites.

  Intersection types work well when you have complete knowledge of what form your
  data will take. This approach only works if you know in advance that your data
  will be and that it will share properties from more than one type. But what if
  you don't have this knowledge? You only know that it will be one of the three
  possible types? This is where union types become helpful. Instead of saying that
  event X will be of type A we say that event X may be one of types A or B or C.

  > A union type is a type formed from two or more other types, representing
  > values that may be any one of those types. We refer to each of these types as
  > the union’s members.

  ## Demonstration

  ```ts
  type Petrol = {
    maxSpeed: number;
    tankVolume: number; // unique property
  };

  type Electric = {
    maxSpeed: number;
    maxCharge: number; // unique property
  };

  type Car = Petrol | Electric;

  const tesla: Car = {
    maxSpeed: 120,
    maxCharge: 22,
  };
  ```

  - We have defined two custom types: `Petrol` and `Electric`
  - We have unified them to make a new custom type of `Car`
  - `tesla` is an instance of the union type

  ## Usage with functions

  Union type syntax is not limited to the declaration phase. You can also use them
  as parameters to functions or function return types:

  ```ts
  function generateCar(car: Petrol | Electric): void {
    console.log(`The car has a max speed of ${car.maxSpeed}`);
  }
  ```

  ## Unifying primitive and custom types

  We can also use unions with primitive types as well as custom types:

  ```ts
  function printId(id: number | string): void {
    console.log(`Your id is ${id}`);
  }
  ```

  We can also combine primitive and custom types in the same union:

  ```ts
  function printStringOrNumber(): string | Petrol {}
  ```

  ## Set theory: understanding errors

  In JavaScript we have the primitive types of string , number , boolean etc. In
  TS, these types are distinct and exclusive: if you declare that x : string you
  cannot redefine it as boolean or expect Boolean properties and methods to belong
  to a string type. For instance a string type is incapable of having as its value
  false.

  This is not true in the case of unions.

  With a union of `string | boolean` a value `x` could be `'banana'` or `true`. We
  are allowing for all values that are either from the set string or the set
  boolean. Logically this is equivalent to the OR operator: a value of type
  `string | boolean` could be one of infinite possible strings or true/false. With
  union types then, we radically widen the scope of possible values for the type.
  a

  These constraints don't just apply to the values themselves but the methods and
  properties you can use with the values. For instance a boolean doesn't have the
  property of `length`. Therefore if you have a union type of `string|boolean` you
  are going to get an error if you apply `length` to it since **the property must
  exist on both types that comprise the union**. The way round this is to use type
  narrowing and execute control flow based on the parameter that is passed to a
  function that is declared to be of this mixed type.

  Here is an example of this:

  ```ts
  function printId(id: number | string) {
    if (typeof id === "string") {
      // In this branch, id is of type 'string'
      console.log(id.toUpperCase());
    } else {
      // Here, id is of type 'number'
      console.log(id);
    }
  }
  ```


path: zk/Unknown_type_in_TS.md
sortable_path: zkUnknown_type_in_TS.md
title: Unknown
checksum: 8c0f68fad261ac22cf13cf968e917e2308e19247798725d27b71e2be1a1a3da2
created: 2024-06-16 17:37:32.188926698+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["typescript"]}
lead:
  You might think that a good use case for `any` is a scenario where you don't
  know in advance what the data type is going to be.

body:
  You might think that a good use case for `any` is a scenario where you don't
  know in advance what the data type is going to be.

  In order to mark that this is the case, you type the value as `any`. In fact,
  TypeScript provides a type for this specific scenario: `unknown`.

  Like `any`, `unknown` is equivalent to every type in TS (it is also a supertype)
  but it is deliberately inhibiting, in contrast to `any`. When you use `unknown`
  you have to use type narrowing to avoid it being returned. So if your code
  starts with `unknown`, you should filter out scenarios where it evaluates to
  each of the possible types otherwise if `unknown` is returned it will cause an
  error.

raw_content:
  ---
  tags:
    - typescript
  ---

  # Unknown

  You might think that a good use case for `any` is a scenario where you don't
  know in advance what the data type is going to be.

  In order to mark that this is the case, you type the value as `any`. In fact,
  TypeScript provides a type for this specific scenario: `unknown`.

  Like `any`, `unknown` is equivalent to every type in TS (it is also a supertype)
  but it is deliberately inhibiting, in contrast to `any`. When you use `unknown`
  you have to use type narrowing to avoid it being returned. So if your code
  starts with `unknown`, you should filter out scenarios where it evaluates to
  each of the possible types otherwise if `unknown` is returned it will cause an
  error.


path: zk/User_management_in_Linux.md
sortable_path: zkUser_management_in_Linux.md
title: User management
lead: ## Switch user
checksum: e5c96a6157409e2d25e0f6278475b4bb97323774450952af78f3ad3ea1369622
created: 2024-06-16 17:37:32.196102048+00:00
modified: 2024-06-16 17:33:56.283646467+00:00
metadata: {"tags":["Linux","procedural"]}
body:
  ## Switch user

  If already logged in as a user you can switch users with the command
  `su - [username]`.

  ## Login as root

  If you are logged in as a standard user, if you use `su -` without specifying a
  username, then it will assume you wish to log in as root.

  If you wish to login as root in the tty at startup, then use `root` as the
  username.

raw_content:
  ---
  tags:
    - Linux
    - procedural
  ---

  # User management

  ## Switch user

  If already logged in as a user you can switch users with the command
  `su - [username]`.

  ## Login as root

  If you are logged in as a standard user, if you use `su -` without specifying a
  username, then it will assume you wish to log in as root.

  If you wish to login as root in the tty at startup, then use `root` as the
  username.


path: zk/Utilities_operators_flags.md
sortable_path: zkUtilities_operators_flags.md
title: Flags
lead: ### Prevent newline
checksum: a337ba34dc84259e41d04f49430beff149ec176bc74d5f2bf49b7b17623cd857
created: 2024-06-16 17:37:32.208963277+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["shell"]}
body:
  ### Prevent newline

  Prevent bash from adding a new line after an echo:

  ```bash
  echo 'Your name is Thomas'
  echo 'and you are 33 years old'

  # Your name is Thomas
  # and you are 33 years old
  ```

  ```bash
  echo -n 'Your name is Thomas '
  echo 'and you are 33 years old'

  # Your name is Thomas and you are 33 years old
  ```

  ## Operators

  ### Mathematical

  ```bash
  -lt , -gt,
  ```

raw_content:
  ---
  tags:
    - shell
  ---

  The following are useful built-in utility methods that you can use for checking
  and validation in the course of your bash scripts.

  ## Flags

  ### Prevent newline

  Prevent bash from adding a new line after an echo:

  ```bash
  echo 'Your name is Thomas'
  echo 'and you are 33 years old'

  # Your name is Thomas
  # and you are 33 years old
  ```

  ```bash
  echo -n 'Your name is Thomas '
  echo 'and you are 33 years old'

  # Your name is Thomas and you are 33 years old
  ```

  ## Operators

  ### Mathematical

  ```bash
  -lt , -gt,
  ```


path: zk/Validation_in_NodeJS.md
sortable_path: zkValidation_in_NodeJS.md
title: Creating a RESTful API: Validation
checksum: 6f0c4e7f0af95a7522c442d4991bd06ab0a817dac6eabb19ed81afb036089759
created: 2024-06-16 17:37:32.228913914+00:00
modified: 2024-06-15 09:50:54.243778564+00:00
metadata: {"tags":["node-js","REST","APIs"]}
lead:
  We can provide server-side validation for our projects by using a **schema
  validator**. This is a program that declaratively parses the JSON values
  received as requests from the client. This makes it easy to systematically
  validate the data that we receive from any HTTP requests where the client sends
  a body to the endpoint.

body:
  We can provide server-side validation for our projects by using a **schema
  validator**. This is a program that declaratively parses the JSON values
  received as requests from the client. This makes it easy to systematically
  validate the data that we receive from any HTTP requests where the client sends
  a body to the endpoint.

  One of the most popular schema validators for NodeJS is
  [joi](joi).

  ## Demonstration

  Let's say we have a POST request that expects a single field as the body that
  must be a string and greater than two characters long. First we define our
  schema:

  ```js
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });

  const { error } = schema.validate(req.body);
  ```

  The `schema` variable is an object whose keys should match those of the intended
  request body. Instead of actual values we provide Joi's in-built validators,
  concatenated as necessary. We then store the results of the validation in a
  variable.

  Next we add handling in the case of errors:

  ```js
  if (error) {
    error.details.map((joiErr) => res.status(400).send(joiErr.message));
    return;
  }
  ```

  We loop through the error array and return 400s as the response if they are
  found. If there are no errors, the Joi object will return `undefined`.

raw_content:
  ---
  tags:
    - node-js
    - REST
    - APIs
  ---

  # Creating a RESTful API: Validation

  We can provide server-side validation for our projects by using a **schema
  validator**. This is a program that declaratively parses the JSON values
  received as requests from the client. This makes it easy to systematically
  validate the data that we receive from any HTTP requests where the client sends
  a body to the endpoint.

  One of the most popular schema validators for NodeJS is
  [joi](joi).

  ## Demonstration

  Let's say we have a POST request that expects a single field as the body that
  must be a string and greater than two characters long. First we define our
  schema:

  ```js
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });

  const { error } = schema.validate(req.body);
  ```

  The `schema` variable is an object whose keys should match those of the intended
  request body. Instead of actual values we provide Joi's in-built validators,
  concatenated as necessary. We then store the results of the validation in a
  variable.

  Next we add handling in the case of errors:

  ```js
  if (error) {
    error.details.map((joiErr) => res.status(400).send(joiErr.message));
    return;
  }
  ```

  We loop through the error array and return 400s as the response if they are
  found. If there are no errors, the Joi object will return `undefined`.


path: zk/Variable_indirection_in_Bash.md
sortable_path: zkVariable_indirection_in_Bash.md
title: Variable indirection
lead: > Write proper notes on this
checksum: 6682743efcc1a97b2ba0760899b1690e9af193e72e3df5a957904ab8e01726a5
created: 2024-06-16 17:37:32.243804241+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["shell"]}
body:
  > Write proper notes on this

  ```sh
  function array_empty() {
      declare -n arr=$1
      # Proceed if array not empty:
      if [ ${#arr[@]} -gt 0 ]; then
          return 1 # array is not empty
      else
          return 0 # array is empty
      fi
  }

  my_array=(1 2 3)

  function push() {
      # $1 = stack name
      local stack_name="$1"
      if [ "${stack_name}" ]; then # stack exists
          if array_empty "${!stack_name}"; then
              echo "stack is empty"
          else
              echo "stack is not empty"
          fi
      else
          echo "Error: ${stack_name} does not exist."
      fi
  }

  push "my_array"

  ```

  In Bash scripting, ${!stack_name} is an example of indirect variable referencing
  or variable indirection. It's a way to use a variable whose name is stored in
  another variable.

  Let's break it down:

  `${stack_name}`: This syntax is used to reference the value of a variable. So if
  stack_name="my_array", then ${stack_name} would return "my_array".

  `${!stack_name}`: Adding the ! introduces indirection. Now, instead of getting
  the value of stack_name, you get the value of the variable whose name is stored
  in stack_name. So if stack_name="my_array" and my_array=(1 2 3), then
  ${!stack_name} would return (1 2 3).

  So, in the context of your script, `${!stack_name} allows you to pass the name
  of an array (as a string) to your push function, and then use that string to
  indirectly reference the actual array within the function.

raw_content:
  ---
  tags:
    - shell
  ---

  # Variable indirection

  > Write proper notes on this

  ```sh
  function array_empty() {
      declare -n arr=$1
      # Proceed if array not empty:
      if [ ${#arr[@]} -gt 0 ]; then
          return 1 # array is not empty
      else
          return 0 # array is empty
      fi
  }

  my_array=(1 2 3)

  function push() {
      # $1 = stack name
      local stack_name="$1"
      if [ "${stack_name}" ]; then # stack exists
          if array_empty "${!stack_name}"; then
              echo "stack is empty"
          else
              echo "stack is not empty"
          fi
      else
          echo "Error: ${stack_name} does not exist."
      fi
  }

  push "my_array"

  ```

  In Bash scripting, ${!stack_name} is an example of indirect variable referencing
  or variable indirection. It's a way to use a variable whose name is stored in
  another variable.

  Let's break it down:

  `${stack_name}`: This syntax is used to reference the value of a variable. So if
  stack_name="my_array", then ${stack_name} would return "my_array".

  `${!stack_name}`: Adding the ! introduces indirection. Now, instead of getting
  the value of stack_name, you get the value of the variable whose name is stored
  in stack_name. So if stack_name="my_array" and my_array=(1 2 3), then
  ${!stack_name} would return (1 2 3).

  So, in the context of your script, `${!stack_name} allows you to pass the name
  of an array (as a string) to your push function, and then use that string to
  indirectly reference the actual array within the function.


path: zk/Variables_and_data_types_in_Bash.md
sortable_path: zkVariables_and_data_types_in_Bash.md
title: Types
lead: ## Variables
checksum: 5d0734a826d4cd925e0eac4fc3da7eb6b0ad3cbedebb3da59ffefc3189d0347e
created: 2024-06-16 17:37:32.248470584+00:00
modified: 2024-03-27 10:45:52.506858751+00:00
metadata: {"tags":["shell"]}
body:
  ## Variables

  We use the equality symbol to create a variable:

  ```bash
  stringVar="My first variable"
  floatVar="50.3"
  boolVar="true"
  ```

  As there is no typing in bash, the names of these variables are purely notional.

  To invoke a variable we use special brackets:

  ```bash
  echo ${stringVar} # My first variable
  echo ${floatVar} # 50.3
  echo ${boolVar} # true
  ```

  - Quotation marks at declaration are also not strictly necessary however they
    can help avoid bugs. Also serves as a reminder that every type is basically a
    string at the end of the day

  ## Common operations

  ### Check if a variable exists

  ```sh
  if [ -v variable_name ]; then
      echo "The variable exists."
  else
      echo "The variable does not exist."
  fi
  ```

  ### Check if a variable is set (defined)

  ```sh
  if [ -z "${variable_name+x}" ]; then
      echo "The variable does not exist."
  else
      echo "The variable exists."
  fi
  ```

  ## Parameter expansion

  Still not very clear on this.

raw_content:
  ---
  tags:
    - shell
  ---

  ## Types

  ## Variables

  We use the equality symbol to create a variable:

  ```bash
  stringVar="My first variable"
  floatVar="50.3"
  boolVar="true"
  ```

  As there is no typing in bash, the names of these variables are purely notional.

  To invoke a variable we use special brackets:

  ```bash
  echo ${stringVar} # My first variable
  echo ${floatVar} # 50.3
  echo ${boolVar} # true
  ```

  - Quotation marks at declaration are also not strictly necessary however they
    can help avoid bugs. Also serves as a reminder that every type is basically a
    string at the end of the day

  ## Common operations

  ### Check if a variable exists

  ```sh
  if [ -v variable_name ]; then
      echo "The variable exists."
  else
      echo "The variable does not exist."
  fi
  ```

  ### Check if a variable is set (defined)

  ```sh
  if [ -z "${variable_name+x}" ]; then
      echo "The variable does not exist."
  else
      echo "The variable exists."
  fi
  ```

  ## Parameter expansion

  Still not very clear on this.


path: zk/Variables_in_C.md
sortable_path: zkVariables_in_C.md
title: Variables in C
lead: # Variables in C
body:
  # Variables in C

  ```c
  int age = 25
  ```

checksum: b702bec2aebc256d92e83e60ef9beefdd46605d1a435691d4936b840e6ba160c
created: 2024-06-16 17:37:32.252810994+00:00
modified: 2024-03-27 10:45:52.510192133+00:00
raw_content:
  ---
  id: r4i16p1x
  title: Variables in C
  tags: [C]
  created: Thursday, February 29, 2024 | 17:20
  ---

  # Variables in C

  ```c
  int age = 25
  ```

metadata:
  {"created":"Thursday, February 29, 2024 | 17:20","id":"r4i16p1x","tags":["C"],"title":"Variables in C"}


path: zk/Virtual_private_cloud.md
sortable_path: zkVirtual_private_cloud.md
title: Virtual Private Cloud
lead: # Virtual Private Cloud
checksum: 15665690b24aff7e50083932162a8fb8e22aa4065f512ff3495b497587d4bca4
created: 2024-06-16 17:37:32.272794495+00:00
modified: 2024-03-27 10:45:52.510192133+00:00
body:
  # Virtual Private Cloud

  AWS VPC is used to create a virtual network. This is typically used in concert
  with [[zk/Elastic_Compute_Cloud]] to manage connections to a virtual server both
  privately and accross the internet.

  You define a network address range and then create subnets for managing
  different connections and functionality. You use a public subnet for resources
  that must be connected to the internet and a private subnet for resources that
  are to remain isolated from the internet.

  The diagram below details a basic VPC configuration:

  ![](../img/aws-vpc-example.png)

  - Within a given AWS region we have created a VPC network.
  - This comprises public and private subnets
  - Both subnets host an EC2 instance
  - The public subnet has a bridge to the internet through the Internet Gateway
  - Both subnets have a routing table to manage requests and access

raw_content:
  ---
  id: mdw5fe5a
  title: Virtual Private Cloud
  tags: [AWS, networks]
  created: Tuesday, February 20, 2024 | 08:31
  ---

  # Virtual Private Cloud

  AWS VPC is used to create a virtual network. This is typically used in concert
  with [[zk/Elastic_Compute_Cloud]] to manage connections to a virtual server both
  privately and accross the internet.

  You define a network address range and then create subnets for managing
  different connections and functionality. You use a public subnet for resources
  that must be connected to the internet and a private subnet for resources that
  are to remain isolated from the internet.

  The diagram below details a basic VPC configuration:

  ![](../img/aws-vpc-example.png)

  - Within a given AWS region we have created a VPC network.
  - This comprises public and private subnets
  - Both subnets host an EC2 instance
  - The public subnet has a bridge to the internet through the Internet Gateway
  - Both subnets have a routing table to manage requests and access

metadata:
  {"created":"Tuesday, February 20, 2024 | 08:31","id":"mdw5fe5a","tags":["AWS","networks"],"title":"Virtual Private Cloud"}


path: zk/http_in_Node.md
sortable_path: zkhttp_in_Node.md
title: http module
checksum: d74610fc301d67759fe76291d7e99eda1b918720fe38a694e5d28775c0b7ee25
created: 2024-06-16 17:37:32.313365813+00:00
modified: 2024-06-15 09:50:58.8172+00:00
metadata: {"tags":["node-js"]}
lead:
  The HTTP Module allows us to create a web server that listens for HTTP requests
  on a given port. It is therefore perfect for creating backends for client-side
  JavaScript.

body:
  The HTTP Module allows us to create a web server that listens for HTTP requests
  on a given port. It is therefore perfect for creating backends for client-side
  JavaScript.

  ## Creating a server

  An HTTP server is another instance of an
  [event emitter](events.md)). It
  therefore has all the same methods as the `EventEmitter` class: `on`, `emit`,
  `addListener` etc. This demonstrates again how much of Node's core functionality
  is based on event emitters.

  _Creating a server_

  ```js
  const http = require("http");

  const server = http.createServer(); // Create server as emitter

  // Register functions to run when listener is triggered
  server.on("connection", (socket) => {
    console.log("new connection...");
  });

  server.listen(3000);
  console.log("Listening on port 3000");
  ```

  This server is functionally equivalent to a generic event emitter:

  ```js
  // Raise an event
  const emitter = new EventEmitter("messageLogged");

  // Register a listener
  emitter.on("messagedLogged", function () {
    console.log("The listener was called.");
  });
  ```

  Whenever a request is made to this server, it raises an event. We can therefore
  target it with the `on` method and make it execute a function when requests are
  made.

  If we were to start the server by running the file and we then used a browser to
  navigate to the port, we would see `new connection` logged every time we refresh
  the page.

  ### Sockets and `req, res`

  A socket is a generic protocol for client-server communication. Crucially it
  **allows simultaneous communication both ways**. The client can contact the
  server but the server can also contact the client. Our listener function above
  uses a socket as the callback function but in most cases this is quite
  low-level, not distinguishing responses from requests. It is more likely that
  you would initiate a `request, resource` architecture in place of a socket:

  ```js
  const server = http.createServer((req, res) => {
    if (req.url === "/") {
      res.write("hello");
      res.end();
    }
  });
  ```

  #### Return JSON

  Below is an example of using this architecture to return JSON to the client:

  ```js
  const server = http.createServer((req, res) => {
    if (req.url === "/products") {
      res.write(JSON.stringify(["shoes", "lipstick", "cups"]));
      res.end();
    }
  });
  ```

  ### Express

  In reality you would rarely use the `http` module directly to create a server.
  This is because it is quite low level and each response must be written in a
  linear fashion as with the two URLs in the previous example. Instead we use
  Express which is a framework for creating servers and routing that is an
  abstraction on top of the core HTTP module.

raw_content:
  ---
  tags:
    - node-js
  ---

  # `http` module

  The HTTP Module allows us to create a web server that listens for HTTP requests
  on a given port. It is therefore perfect for creating backends for client-side
  JavaScript.

  ## Creating a server

  An HTTP server is another instance of an
  [event emitter](events.md)). It
  therefore has all the same methods as the `EventEmitter` class: `on`, `emit`,
  `addListener` etc. This demonstrates again how much of Node's core functionality
  is based on event emitters.

  _Creating a server_

  ```js
  const http = require("http");

  const server = http.createServer(); // Create server as emitter

  // Register functions to run when listener is triggered
  server.on("connection", (socket) => {
    console.log("new connection...");
  });

  server.listen(3000);
  console.log("Listening on port 3000");
  ```

  This server is functionally equivalent to a generic event emitter:

  ```js
  // Raise an event
  const emitter = new EventEmitter("messageLogged");

  // Register a listener
  emitter.on("messagedLogged", function () {
    console.log("The listener was called.");
  });
  ```

  Whenever a request is made to this server, it raises an event. We can therefore
  target it with the `on` method and make it execute a function when requests are
  made.

  If we were to start the server by running the file and we then used a browser to
  navigate to the port, we would see `new connection` logged every time we refresh
  the page.

  ### Sockets and `req, res`

  A socket is a generic protocol for client-server communication. Crucially it
  **allows simultaneous communication both ways**. The client can contact the
  server but the server can also contact the client. Our listener function above
  uses a socket as the callback function but in most cases this is quite
  low-level, not distinguishing responses from requests. It is more likely that
  you would initiate a `request, resource` architecture in place of a socket:

  ```js
  const server = http.createServer((req, res) => {
    if (req.url === "/") {
      res.write("hello");
      res.end();
    }
  });
  ```

  #### Return JSON

  Below is an example of using this architecture to return JSON to the client:

  ```js
  const server = http.createServer((req, res) => {
    if (req.url === "/products") {
      res.write(JSON.stringify(["shoes", "lipstick", "cups"]));
      res.end();
    }
  });
  ```

  ### Express

  In reality you would rarely use the `http` module directly to create a server.
  This is because it is quite low level and each response must be written in a
  linear fashion as with the two URLs in the previous example. Instead we use
  Express which is a framework for creating servers and routing that is an
  abstraction on top of the core HTTP module.


path: zk/Network_debugging.md
sortable_path: zkNetwork_debugging.md
title: Network debugging
lead: # Network debugging
body:
  # Network debugging

  ## Related notes

  ![Network scanning](Network_scanning.md)

checksum: c9c358edc6e51a8d38781b824600abdf15dcc85e98385bfaf50fd1264927c2b1
created: 2024-06-17 19:00:36.317063339+00:00
modified: 2024-06-17 18:57:53.073398693+00:00
raw_content:
  ---
  id: x44w
  title: Network debugging
  tags: [procedural, Linux, networks]
  created: Monday, June 17, 2024
  ---

  # Network debugging

  ## Related notes

  ![Network scanning](Network_scanning.md)

metadata:
  {"created":"Monday, June 17, 2024","id":"x44w","tags":["procedural","Linux","networks"],"title":"Network debugging"}


path: zk/Network debugging.md
sortable_path: zkNetwork debugging.md
title: Network debugging
lead: # Network debugging
checksum: a8dee7b683041f9119953bb6a3356cf170d05e27e4e02b58e8a1cb3a7e8f9726
created: 2024-06-18 06:43:30.382978733+00:00
modified: 2024-06-17 19:33:19.773645673+00:00
body:
  # Network debugging

  ### For DNS resolution problems

  ```sh
  sudo systemctl restart resolvd

  ```

  ## Related notes

  ![Network scanning](Network_scanning.md)

raw_content:
  ---
  id: x44w
  title: Network debugging
  tags: [procedural, Linux, networks]
  created: Monday, June 17, 2024
  ---

  # Network debugging

  ### For DNS resolution problems

  ```sh
  sudo systemctl restart resolvd

  ```

  ## Related notes

  ![Network scanning](Network_scanning.md)

metadata:
  {"created":"Monday, June 17, 2024","id":"x44w","tags":["procedural","Linux","networks"],"title":"Network debugging"}


path: zk/Passing_arguments_to_Python_scripts.md
sortable_path: zkPassing_arguments_to_Python_scripts.md
title: Passing arguments to Python scripts
lead: # Passing arguments to Python scripts
checksum: 5e1ef301b1aa124717c0ce90a1868111cc2b7e21fab8d21ef063d144db35393b
created: 2024-06-19 16:57:35.57855372+00:00
modified: 2024-06-19 06:39:17.552249291+00:00
body:
  # Passing arguments to Python scripts

  `sys.argv` is a list that contains the command-line arguments passed to a Python
  scripts.

  - `sys.argv[0]` = the name of script
  - `sys.argv[1]` = the first argument
  - `sys.argv[2]` = the second argument, and so on

  Example invocation:

  ```sh
  python3 ./my_script.py argument_one argument_two

  ```

  ```python
  import sys

  print(sys.argv[0])
  print(sys.arg)
  print(sys.argv[1])

  # my_script.py
  # ['my_script.py', 'argyment_one', 'argument_two']
  # argument_one

  ```

raw_content:
  ---
  id: 44li
  title: Passing arguments to Python scripts
  tags: [python]
  created: Wednesday, June 19, 2024
  ---

  # Passing arguments to Python scripts

  `sys.argv` is a list that contains the command-line arguments passed to a Python
  scripts.

  - `sys.argv[0]` = the name of script
  - `sys.argv[1]` = the first argument
  - `sys.argv[2]` = the second argument, and so on

  Example invocation:

  ```sh
  python3 ./my_script.py argument_one argument_two

  ```

  ```python
  import sys

  print(sys.argv[0])
  print(sys.arg)
  print(sys.argv[1])

  # my_script.py
  # ['my_script.py', 'argyment_one', 'argument_two']
  # argument_one

  ```

metadata:
  {"created":"Wednesday, June 19, 2024","id":"44li","tags":["python"],"title":"Passing arguments to Python scripts"}


path: zk/Basic_model_of_the_operating_system.md
sortable_path: zkBasic_model_of_the_operating_system.md
title: Basic model of a nix operating system
lead:
  We can abstract the Linux OS into three operational levels or tiers, from the
  bottom up:

checksum: adb2cb2eb5321b12cb4fe19d75c059b192eb7b8440a10b6661014617a2c99742
created: 2024-06-21 05:06:21.717225019+00:00
modified: 2024-07-26 07:16:00.321010139+00:00
metadata: {"tags":["operating-systems","Linux"]}
body:
  We can abstract the Linux OS into three operational levels or tiers, from the
  bottom up:

  <dl>
  <dt>User processes: user space</dt>
  <dd>The running programs that the kernel manages. Also known as the user space which is the memory that the kernal assigns for user processes. Comprising:
  <ul>
      <li>Graphical user interface</li>
      <li>Servers</li> 
      <li>Shell</li>    
    </ul>
  <dt>Kernel: kernel space</dt>
  </dd>
  <dd>The core of the operating system. Software residing in memory that tells the CPU where to look for its next task. Acts as a mediator and primary interface between the hardware and the user processes. Known as kernel space: the memory that the kernel allocates for itself. Comprising:
    <ul>
      <li>System calls</li>
      <li>Process management</li> 
      <li>Memory management</li>    
      <li>Device drivers</li>    
    </ul>
  </dd>
  <dt>Hardware</dt>
  <dd>The base: one or more CPUs and RAM memory performing computations and writing to memory. Comprising:
    <ul>
      <li>Processor (CPU)</li>
      <li>Main memory (RAM)</li> 
      <li>Disks</li>    
      <li>Network ports</li>    
    </ul>
  </dl>

  !! Add info on kernel mode and user mode
  https://www.geeksforgeeks.org/user-mode-and-kernel-mode-switching/

raw_content:
  ---
  tags:
    - operating-systems
    - Linux
  ---

  # Basic model of a nix operating system

  We can abstract the Linux OS into three operational levels or tiers, from the
  bottom up:

  <dl>
  <dt>User processes: user space</dt>
  <dd>The running programs that the kernel manages. Also known as the user space which is the memory that the kernal assigns for user processes. Comprising:
  <ul>
      <li>Graphical user interface</li>
      <li>Servers</li> 
      <li>Shell</li>    
    </ul>
  <dt>Kernel: kernel space</dt>
  </dd>
  <dd>The core of the operating system. Software residing in memory that tells the CPU where to look for its next task. Acts as a mediator and primary interface between the hardware and the user processes. Known as kernel space: the memory that the kernel allocates for itself. Comprising:
    <ul>
      <li>System calls</li>
      <li>Process management</li> 
      <li>Memory management</li>    
      <li>Device drivers</li>    
    </ul>
  </dd>
  <dt>Hardware</dt>
  <dd>The base: one or more CPUs and RAM memory performing computations and writing to memory. Comprising:
    <ul>
      <li>Processor (CPU)</li>
      <li>Main memory (RAM)</li> 
      <li>Disks</li>    
      <li>Network ports</li>    
    </ul>
  </dl>

  !! Add info on kernel mode and user mode
  https://www.geeksforgeeks.org/user-mode-and-kernel-mode-switching/


path: zk/ps.md
sortable_path: zkps.md
title: Processes (ps)
lead:
  `ps` allows us to control [user processes](The_kernel.md)
  from the shell.

checksum: db2045cff446875dd957c96c1b0b2dcaf4c16fed9ceb7c1f0f91146ea1f07297
created: 2024-06-21 05:23:32.570424822+00:00
modified: 2024-06-16 17:26:03.906136212+00:00
metadata: {"tags":["shell","Linux"]}
body:
  `ps` allows us to control [user processes](The_kernel.md)
  from the shell.

  The command in its most minimal application returns the following

  ```
    PID TTY       TIME     CMD
    2437 pts/2    00:00:01 zsh
    7112 pts/2    00:00:00 ps
  ```

  With the `-e` modifier we can list more processes:

  ```
   PID TTY          TIME CMD
        1 ?        00:00:05 systemd
        2 ?        00:00:00 kthreadd
        3 ?        00:00:00 rcu_gp
        4 ?        00:00:00 rcu_par_gp
        5 ?        00:00:00 netns
        7 ?        00:00:00 kworker/0:0H-events_highpri
        9 ?        00:00:00 mm_percpu_wq
       11 ?        00:00:00 rcu_tasks_kthread
       12 ?        00:00:00 rcu_tasks_rude_kthread
       13 ?        00:00:00 rcu_tasks_trace_kthread
       14 ?        00:00:08 ksoftirqd/0
       15 ?        00:03:20 rcu_preempt
       16 ?        00:00:00 rcub/0
       17 ?        00:00:00 migration/0
       18 ?        00:00:00 idle_inject/0
       20 ?        00:00:00 cpuhp/0
       21 ?        00:00:00 cpuhp/1
       22 ?        00:00:00 idle_inject/1
       23 ?        00:00:00 migration/1
  ```

  <dl>
  <dt>pid</dt>
  <dd>Process ID: every currently running process has a unique ID<dd>
  <dt>tty</dt>
  <dd>The terminal device where the process is running<dd>
  <dt>Time</dt>
  <dd>The amount of CPU time in minutes and seconds that the process has used so far. The total amount of time that the process has spent running instructions on the processor.<dd>
  <dt>cmd</dt>
  <dd>The command used to run the program. Note this can change during the running of the program.<dd>
  </dl>

  ## Modifiers

  <dl>
  <dt><code>ps x</code></dt>
  <dd>Show all running processes<dd>
  <dt><code>ps ax</code></dt>
  <dd>Show all processes not just the ones you, the current user, own<dd>
  <dt><code>ps u</code></dt>
  <dd>Show detailed info on processes<dd>
  <dt><code>ps w</code></dt>
  <dd>Show full command names<dd>
  </dl>

  ## Process termination

  The general schema is: `kill [pid]`. This allows for process clean-up. If this
  doesn't succeed you can force with `KILL [pid]` which will terminate the process
  immediately but is obviously more risky.

  We can also start/stop processes with modifiers on `kill`:

  - `kill -STOP pid`
  - `kill -CONT pid`

raw_content:
  ---
  tags:
    - shell
    - Linux
  ---

  # Processes (`ps`)

  `ps` allows us to control [user processes](The_kernel.md)
  from the shell.

  The command in its most minimal application returns the following

  ```
    PID TTY       TIME     CMD
    2437 pts/2    00:00:01 zsh
    7112 pts/2    00:00:00 ps
  ```

  With the `-e` modifier we can list more processes:

  ```
   PID TTY          TIME CMD
        1 ?        00:00:05 systemd
        2 ?        00:00:00 kthreadd
        3 ?        00:00:00 rcu_gp
        4 ?        00:00:00 rcu_par_gp
        5 ?        00:00:00 netns
        7 ?        00:00:00 kworker/0:0H-events_highpri
        9 ?        00:00:00 mm_percpu_wq
       11 ?        00:00:00 rcu_tasks_kthread
       12 ?        00:00:00 rcu_tasks_rude_kthread
       13 ?        00:00:00 rcu_tasks_trace_kthread
       14 ?        00:00:08 ksoftirqd/0
       15 ?        00:03:20 rcu_preempt
       16 ?        00:00:00 rcub/0
       17 ?        00:00:00 migration/0
       18 ?        00:00:00 idle_inject/0
       20 ?        00:00:00 cpuhp/0
       21 ?        00:00:00 cpuhp/1
       22 ?        00:00:00 idle_inject/1
       23 ?        00:00:00 migration/1
  ```

  <dl>
  <dt>pid</dt>
  <dd>Process ID: every currently running process has a unique ID<dd>
  <dt>tty</dt>
  <dd>The terminal device where the process is running<dd>
  <dt>Time</dt>
  <dd>The amount of CPU time in minutes and seconds that the process has used so far. The total amount of time that the process has spent running instructions on the processor.<dd>
  <dt>cmd</dt>
  <dd>The command used to run the program. Note this can change during the running of the program.<dd>
  </dl>

  ## Modifiers

  <dl>
  <dt><code>ps x</code></dt>
  <dd>Show all running processes<dd>
  <dt><code>ps ax</code></dt>
  <dd>Show all processes not just the ones you, the current user, own<dd>
  <dt><code>ps u</code></dt>
  <dd>Show detailed info on processes<dd>
  <dt><code>ps w</code></dt>
  <dd>Show full command names<dd>
  </dl>

  ## Process termination

  The general schema is: `kill [pid]`. This allows for process clean-up. If this
  doesn't succeed you can force with `KILL [pid]` which will terminate the process
  immediately but is obviously more risky.

  We can also start/stop processes with modifiers on `kill`:

  - `kill -STOP pid`
  - `kill -CONT pid`


path: zk/Processes.md
sortable_path: zkProcesses.md
title: Processes
lead: # Processes
checksum: 3c9be297bb6c2b03442c98e4f41d1367bdf6e8ecea735f36236082157eac49a4
created: 2024-06-21 05:23:32.592788276+00:00
modified: 2024-06-26 05:12:56.918584949+00:00
metadata: {"created":"Friday, June 21, 2024","id":"t70u","tags":["operating-systems"],"title":"Processes"}
body:
  # Processes

  Programs are sequences of machine instructions stored in a file. However they do
  not work by themselves. Something needs to load the file's intructions into
  memory, direct the CPU to run the program and manage it during runtime. The OS
  does this via processes.

  A process **is a running instance of a given program**. It can be thought of as
  a container in which a program runs. This container includes:

  - a copy of the program code loaded into memory
  - a memory address
  - other information about the state of the process

  Other than the `init` process started by the kernel (PID1) (see
  ![systemd](systemd.md)), every process has a parent process that started it.
  This parent-child relationship creates a tree of processes.

  It is possible that a parent process will terminate before one of its child
  processes. In this instance the child becomes an orphan. When this occurs in
  #Linux, the orphan process is adopted by `init`.

  Below, I have used the `pstree` utility to list all the running processes on my
  machine hierarchically.

  ![diagram of `pstree` output](../img/ps-tree.png)

  Children are represented vertically and horizontally.

  ```
  ├─terminator─┬─zsh───tmux: client
  │            ├─zsh───pstree
  │            └─6*[{terminator}]

  ```

  For instance here, `terminator` is a child of `init`, as are `zsh` and `tmux`
  but they are also children of `terminator`.`pstree` is a child of `zsh` and
  therefore also a child (grandchild) of `terminator`.

  Each process has a unique identifier called a _process identifier_, a
  _processID_ or just _PID_.

  ## Related notes

  ![systemd](./systemd.md)

  ![ps](./ps.md)

  ![Monitoring processes and resources](Monitoring_processes_and_resources.md)

raw_content:
  ---
  id: t70u
  title: Processes
  tags: [operating-systems]
  created: Friday, June 21, 2024
  ---

  # Processes

  Programs are sequences of machine instructions stored in a file. However they do
  not work by themselves. Something needs to load the file's intructions into
  memory, direct the CPU to run the program and manage it during runtime. The OS
  does this via processes.

  A process **is a running instance of a given program**. It can be thought of as
  a container in which a program runs. This container includes:

  - a copy of the program code loaded into memory
  - a memory address
  - other information about the state of the process

  Other than the `init` process started by the kernel (PID1) (see
  ![systemd](systemd.md)), every process has a parent process that started it.
  This parent-child relationship creates a tree of processes.

  It is possible that a parent process will terminate before one of its child
  processes. In this instance the child becomes an orphan. When this occurs in
  #Linux, the orphan process is adopted by `init`.

  Below, I have used the `pstree` utility to list all the running processes on my
  machine hierarchically.

  ![diagram of `pstree` output](../img/ps-tree.png)

  Children are represented vertically and horizontally.

  ```
  ├─terminator─┬─zsh───tmux: client
  │            ├─zsh───pstree
  │            └─6*[{terminator}]

  ```

  For instance here, `terminator` is a child of `init`, as are `zsh` and `tmux`
  but they are also children of `terminator`.`pstree` is a child of `zsh` and
  therefore also a child (grandchild) of `terminator`.

  Each process has a unique identifier called a _process identifier_, a
  _processID_ or just _PID_.

  ## Related notes

  ![systemd](./systemd.md)

  ![ps](./ps.md)

  ![Monitoring processes and resources](Monitoring_processes_and_resources.md)


path: zk/DynamoDB_data_structure.md
sortable_path: zkDynamoDB_data_structure.md
title: DynamoDB
lead: # DynamoDB
checksum: b845f451ebb9f0ddb43aa8046e9bfd71b87046576647b64f257d216ae09b4c75
created: 2024-06-22 09:38:00.123616477+00:00
modified: 2024-06-21 06:28:08.053871627+00:00
body:
  # DynamoDB

  ## Data structure

  ### Non-relational tables

  DynamoDB is "NoSQL" because it does not support #SQL queries and is
  non-relational meaning there cannot be JOIN operations via
  [foreign_keys](Foreign_keys_in_SQL.md)

  ![](../img/dynamodb.png)

  ### Primary key

  Although the data is stored as a table, one of the attributes is a primary key
  and the rest of the attributes are effectively the "value" associated with it.

  Because DynamoDB is schemaless, other than the primary key, neither the
  attributes or their data types need to be defined beforehand and each item can
  have its own distinct attributes.

  Each item in the table is uniquely identifiable by its primary key.

  There are two types of primary key available:

  - partition key: a simple primary key composed of one attribute only. Because
    the primary key is hash-mapped items can be retrieved very rapidly using the
    primary key. This would be the `personId` alone.

  - composite key: this comprises a partition key and a _sort key_ both of which
    are attributes. In a table that has a partition key and a sort key, it's
    possible for multiple items to have the same partition key value. However,
    those items must have different sort key values. You could then query by
    either key or both. For instance using the `PersonId` along with `LastName`

  ### Secondary index

  As well as the index provided by the primary key, you can set one or more
  **secondary indices**. A secondary index lets you query the data in the table
  using an alternate key.

  A **global secondary index** is useful for querying data that needs to be
  accessed using non-primary key attributes. For example, if you have a Users
  table with `UserID` as the primary key but often need to fetch users by their
  `Email`, a GSI on `Email` would be appropriate.

  There are also **local secondary indices** but I don't understand the
  difference.

  ## Real example

  Below is a specification of the DynamoDB table I am using for my time-entries
  project:

  ```json
  {
    "TableName": "TimeEntries",
    "KeyAttributes": {
      "PartitionKey": {
        "AttributeName": "activity_start_end",
        "AttributeType": "S"
      }
    },
    "NonKeyAttributes": [
      {
        "AttributeName": "activity_type",
        "AttributeType": "S"
      },
      {
        "AttributeName": "start",
        "AttributeType": "S"
      },
      {
        "AttributeName": "end",
        "AttributeType": "S"
      },
      {
        "AttributeName": "duration",
        "AttributeType": "N"
      },
      {
        "AttributeName": "description",
        "AttributeType": "S"
      },
      {
        "AttributeName": "year",
        "AttributeType": "S"
      }
    ],
    "GlobalSecondaryIndexes": [
      {
        "IndexName": "YearIndex",
        "KeyAttributes": {
          "PartitionKey": {
            "AttributeName": "year",
            "AttributeType": "S"
          },
          "SortKey": {
            "AttributeName": "start",
            "AttributeType": "S"
          }
        },
        "Projection": {
          "ProjectionType": "ALL"
        }
      }
    ]
  }
  ```

  This defines the attribute `activity_start_end` as the primary key. This string
  (`S`) value is a concatenation of three attributes, which is a way of ensuring
  each entry for the attribute will be unique.

  The `NonKeyAttributes` are all the other attributes in addition to the primary
  key. As mentioned these do not actually need to be defined when setting up the
  table but they are listed here for clarity.

  I have also defined a GSI. This is derived from the `Year` attribute. This will
  group all the items by their `Year`, allowing me to query directly by year but
  also helping to chunk the entries which will make look-ups quicker and less
  expensive.

  ## Related notes

raw_content:
  ---
  id: xowl
  title: DynamoDB
  tags:
    - AWS
    - databases
    - dynamodb
  created: Sunday, June 09, 2024
  ---

  # DynamoDB

  ## Data structure

  ### Non-relational tables

  DynamoDB is "NoSQL" because it does not support #SQL queries and is
  non-relational meaning there cannot be JOIN operations via
  [foreign_keys](Foreign_keys_in_SQL.md)

  ![](../img/dynamodb.png)

  ### Primary key

  Although the data is stored as a table, one of the attributes is a primary key
  and the rest of the attributes are effectively the "value" associated with it.

  Because DynamoDB is schemaless, other than the primary key, neither the
  attributes or their data types need to be defined beforehand and each item can
  have its own distinct attributes.

  Each item in the table is uniquely identifiable by its primary key.

  There are two types of primary key available:

  - partition key: a simple primary key composed of one attribute only. Because
    the primary key is hash-mapped items can be retrieved very rapidly using the
    primary key. This would be the `personId` alone.

  - composite key: this comprises a partition key and a _sort key_ both of which
    are attributes. In a table that has a partition key and a sort key, it's
    possible for multiple items to have the same partition key value. However,
    those items must have different sort key values. You could then query by
    either key or both. For instance using the `PersonId` along with `LastName`

  ### Secondary index

  As well as the index provided by the primary key, you can set one or more
  **secondary indices**. A secondary index lets you query the data in the table
  using an alternate key.

  A **global secondary index** is useful for querying data that needs to be
  accessed using non-primary key attributes. For example, if you have a Users
  table with `UserID` as the primary key but often need to fetch users by their
  `Email`, a GSI on `Email` would be appropriate.

  There are also **local secondary indices** but I don't understand the
  difference.

  ## Real example

  Below is a specification of the DynamoDB table I am using for my time-entries
  project:

  ```json
  {
    "TableName": "TimeEntries",
    "KeyAttributes": {
      "PartitionKey": {
        "AttributeName": "activity_start_end",
        "AttributeType": "S"
      }
    },
    "NonKeyAttributes": [
      {
        "AttributeName": "activity_type",
        "AttributeType": "S"
      },
      {
        "AttributeName": "start",
        "AttributeType": "S"
      },
      {
        "AttributeName": "end",
        "AttributeType": "S"
      },
      {
        "AttributeName": "duration",
        "AttributeType": "N"
      },
      {
        "AttributeName": "description",
        "AttributeType": "S"
      },
      {
        "AttributeName": "year",
        "AttributeType": "S"
      }
    ],
    "GlobalSecondaryIndexes": [
      {
        "IndexName": "YearIndex",
        "KeyAttributes": {
          "PartitionKey": {
            "AttributeName": "year",
            "AttributeType": "S"
          },
          "SortKey": {
            "AttributeName": "start",
            "AttributeType": "S"
          }
        },
        "Projection": {
          "ProjectionType": "ALL"
        }
      }
    ]
  }
  ```

  This defines the attribute `activity_start_end` as the primary key. This string
  (`S`) value is a concatenation of three attributes, which is a way of ensuring
  each entry for the attribute will be unique.

  The `NonKeyAttributes` are all the other attributes in addition to the primary
  key. As mentioned these do not actually need to be defined when setting up the
  table but they are listed here for clarity.

  I have also defined a GSI. This is derived from the `Year` attribute. This will
  group all the items by their `Year`, allowing me to query directly by year but
  also helping to chunk the entries which will make look-ups quicker and less
  expensive.

  ## Related notes

metadata:
  {"created":"Sunday, June 09, 2024","id":"xowl","tags":["AWS","databases","dynamodb"],"title":"DynamoDB"}


path: zk/DynamoDB_CLI_commands.md
sortable_path: zkDynamoDB_CLI_commands.md
title: DynamoDB CLI commands
lead: # DynamoDB CLI commands
checksum: 1a00b260b0e3389d043fd16674eb547ec691c4d7f33d0b966b93c86b632bda4b
created: 2024-06-23 18:20:42.429528536+00:00
modified: 2024-06-23 18:02:16.326234491+00:00
body:
  # DynamoDB CLI commands

  ## Connecting to a local (Docker)/prod (AWS) DynamoDB instance

  In order to distinguish between local and production accounts you should keep
  seperate profiles for each (via `.aws/config` and `.aws/credentials`).

  When connecting to a local DB use the local profile and the local URL. Without
  the `--profile` flag, AWS will default to the `default` profile which will
  typically be your credentials for accessing AWS on the remote.

  It also sometimes required to add the endpoint-url when working locally:

  ```
  --endpoint-url http://localhost:800

  > When using the commans below locally, assume `--endpoint-url` and `--profile`
  > have been appended

  ```

  For example:

  ```sh
  aws dynamodb list-tables \
  --profile timetracking_dev \
  --endpoint-url http://localhost:800
  ```

  ## Delete a table

  ```sh
  aws dynamodb delete-table \
  --table-name TableName
  ```

  ## Create table from JSON schema

  ```sh
  aws dynamodb create-table \
  --cli-input-json file://create-timeentries-table.json \

  ```

  ## Describe the table

  View the table schema:

  ```sh

  aws dynamodb describe-table \
  --table-name TimeEntries
  ```

  ## Scan the table

  ```sh
  aws dynamodb scan \
  --table-name TimeEntries \
  --output table
  ```

  ## Related notes

raw_content:
  ---
  id: l045
  title: DynamoDB CLI commands
  tags: [AWS, databases, dynamodb]
  created: Saturday, June 22, 2024
  ---

  # DynamoDB CLI commands

  ## Connecting to a local (Docker)/prod (AWS) DynamoDB instance

  In order to distinguish between local and production accounts you should keep
  seperate profiles for each (via `.aws/config` and `.aws/credentials`).

  When connecting to a local DB use the local profile and the local URL. Without
  the `--profile` flag, AWS will default to the `default` profile which will
  typically be your credentials for accessing AWS on the remote.

  It also sometimes required to add the endpoint-url when working locally:

  ```
  --endpoint-url http://localhost:800

  > When using the commans below locally, assume `--endpoint-url` and `--profile`
  > have been appended

  ```

  For example:

  ```sh
  aws dynamodb list-tables \
  --profile timetracking_dev \
  --endpoint-url http://localhost:800
  ```

  ## Delete a table

  ```sh
  aws dynamodb delete-table \
  --table-name TableName
  ```

  ## Create table from JSON schema

  ```sh
  aws dynamodb create-table \
  --cli-input-json file://create-timeentries-table.json \

  ```

  ## Describe the table

  View the table schema:

  ```sh

  aws dynamodb describe-table \
  --table-name TimeEntries
  ```

  ## Scan the table

  ```sh
  aws dynamodb scan \
  --table-name TimeEntries \
  --output table
  ```

  ## Related notes

metadata:
  {"created":"Saturday, June 22, 2024","id":"l045","tags":["AWS","databases","dynamodb"],"title":"DynamoDB CLI commands"}


path: zk/DynamoDB_SDK_commands.md
sortable_path: zkDynamoDB_SDK_commands.md
title: DynamoDB SDK commands
lead: # DynamoDB SDK commands
checksum: 2e375094fdb1f167a15e1dbcfc55f5e2653f990ea82816debc3ad47839ed09fe
created: 2024-06-23 18:20:42.435974932+00:00
modified: 2024-06-23 18:09:03.599881207+00:00
body:
  # DynamoDB SDK commands

  The following commands are for using `@aws-sdk/client-dynamodb` (the JS SDK).

  ## Create client

  ```js
  import DynamoDBClient from "@aws-sdk/client-dynamodb";

  const client = new DynamoDBClient({
    region: "eu-west-2",
    accessKeyId: process.env.ACCESS_KEY_ID,
    secretAccessKey: process.env.ACCESS_KEY_ID,
  });
  ```

  ### DynamoDB Document Client

  The Document Client returns the data as a standard JSON document in the manner
  of a document-based NoSQL database. Without it, DynamoDB will return the data
  with type data which is unwieldy and hard to parse.

  To transform into a document structure:

  ```js
  import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
  import { DynamoDBDocumentClient } from "@aws-sdk/lib-dynamodb";

  const client = new DynamoDBClient({
    region: "eu-west-2",
    accessKeyId: process.env.ACCESS_KEY_ID,
    secretAccessKey: process.env.ACCESS_KEY_ID,
  });

  const dynamoDbDocumentClient = DynamoDBDocumentClient.from(client);
  ```

  ## Query a table

  Querying is the most performant and cost-effective method since it is an O(1)
  lookup against a partition key or secondary index.

  ```js
  import { DynamoDBDocumentClient, QueryCommand } from "@aws-sdk/lib-dynamodb";

  const params = {
    TableName: "YourTableName",
    KeyConditionExpression: "#pk = :pk",
    ExpressionAttributeNames: { "#pk": "PrimaryKeyAttributeName" },
    ExpressionAttributeValues: { ":pk": "PrimaryKeyValue" },
  };

  const dynamoDbDocumentClient = DynamoDBDocumentClient.from(client);
  const data = await dynamoDbDocumentClient.send(new QueryCommand(params));
  console.log(data);
  ```

  ## Scan a table

  Scanning is less performant (O(n)) and most expensive since it requires checking
  every item in the database.

  ```js
  import { DynamoDBDocumentClient, ScanCommand } from "@aws-sdk/lib-dynamodb";

  const params = {
    TableName: "YourTableName",
  };

  const dynamoDbDocumentClient = DynamoDBDocumentClient.from(client);
  const data = await dynamoDbDocumentClient.send(new ScanCommand(params));
  ```

  ## Related notes

raw_content:
  ---
  id: 8b6h
  title: DynamoDB SDK commands
  tags: [AWS, databases, dynamodb]
  created: Sunday, June 23, 2024
  ---

  # DynamoDB SDK commands

  The following commands are for using `@aws-sdk/client-dynamodb` (the JS SDK).

  ## Create client

  ```js
  import DynamoDBClient from "@aws-sdk/client-dynamodb";

  const client = new DynamoDBClient({
    region: "eu-west-2",
    accessKeyId: process.env.ACCESS_KEY_ID,
    secretAccessKey: process.env.ACCESS_KEY_ID,
  });
  ```

  ### DynamoDB Document Client

  The Document Client returns the data as a standard JSON document in the manner
  of a document-based NoSQL database. Without it, DynamoDB will return the data
  with type data which is unwieldy and hard to parse.

  To transform into a document structure:

  ```js
  import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
  import { DynamoDBDocumentClient } from "@aws-sdk/lib-dynamodb";

  const client = new DynamoDBClient({
    region: "eu-west-2",
    accessKeyId: process.env.ACCESS_KEY_ID,
    secretAccessKey: process.env.ACCESS_KEY_ID,
  });

  const dynamoDbDocumentClient = DynamoDBDocumentClient.from(client);
  ```

  ## Query a table

  Querying is the most performant and cost-effective method since it is an O(1)
  lookup against a partition key or secondary index.

  ```js
  import { DynamoDBDocumentClient, QueryCommand } from "@aws-sdk/lib-dynamodb";

  const params = {
    TableName: "YourTableName",
    KeyConditionExpression: "#pk = :pk",
    ExpressionAttributeNames: { "#pk": "PrimaryKeyAttributeName" },
    ExpressionAttributeValues: { ":pk": "PrimaryKeyValue" },
  };

  const dynamoDbDocumentClient = DynamoDBDocumentClient.from(client);
  const data = await dynamoDbDocumentClient.send(new QueryCommand(params));
  console.log(data);
  ```

  ## Scan a table

  Scanning is less performant (O(n)) and most expensive since it requires checking
  every item in the database.

  ```js
  import { DynamoDBDocumentClient, ScanCommand } from "@aws-sdk/lib-dynamodb";

  const params = {
    TableName: "YourTableName",
  };

  const dynamoDbDocumentClient = DynamoDBDocumentClient.from(client);
  const data = await dynamoDbDocumentClient.send(new ScanCommand(params));
  ```

  ## Related notes

metadata:
  {"created":"Sunday, June 23, 2024","id":"8b6h","tags":["AWS","databases","dynamodb"],"title":"DynamoDB SDK commands"}


path: zk/Breadboards.md
sortable_path: zkBreadboards.md
title: Breadboards
lead: # Breadboards
checksum: a75f8e322e550c69d0f56ba2b927043a213d1d213097c26a94636245fa825c86
created: 2024-06-23 18:20:42.444058773+00:00
modified: 2024-07-04 05:19:33.039109238+00:00
metadata: {"created":"Sunday, June 23, 2024","id":"jbf1","tags":["electronics"],"title":"Breadboards"}
body:
  # Breadboards

  ![](../img/breadboard-diagram.jpg)

  ## Contact holes

  Beneath the plastic casing, the contact holes are connected to one another
  making a continuous circuit connection

  ## Terminal strips

  The contact holes in the middle section, arranged in groups of five and
  designated by letters

  The terminal strips are not connected accross the ditch thus each of the two
  strips are electrically separate regions (A-E and F-J).

  ## Bus strips

  The contact holes on the outer edges of the breadboard, designated by numbers
  (1-30). The chief role of the bus strips is to manage the power to the circuit.
  You will typically use the positive bus strip on one side for the voltage source
  and the negative bus strip on the other side for the ground outlet.

raw_content:
  ---
  id: jbf1
  title: Breadboards
  tags: [electronics]
  created: Sunday, June 23, 2024
  ---

  # Breadboards

  ![](../img/breadboard-diagram.jpg)

  ## Contact holes

  Beneath the plastic casing, the contact holes are connected to one another
  making a continuous circuit connection

  ## Terminal strips

  The contact holes in the middle section, arranged in groups of five and
  designated by letters

  The terminal strips are not connected accross the ditch thus each of the two
  strips are electrically separate regions (A-E and F-J).

  ## Bus strips

  The contact holes on the outer edges of the breadboard, designated by numbers
  (1-30). The chief role of the bus strips is to manage the power to the circuit.
  You will typically use the positive bus strip on one side for the voltage source
  and the negative bus strip on the other side for the ground outlet.


path: zk/ROM versus disk loading.md
sortable_path: zkROM versus disk loading.md
title: ROM versus disk loading
lead: # ROM versus disk loading
checksum: d600733b132ab09bab56b1b41a87ffede900adc285cc732de7dbd37743f0da40
created: 2024-06-24 16:30:55.619239265+00:00
modified: 2024-06-24 16:57:05.137930828+00:00
body:
  # ROM versus disk loading

  I will explore the concept via the practical examples of cartridge-based and
  disk-based video games console.

  When you insert a disk into an Xbox, the game data is typically read from the
  disk and transferred into the console's memory by the OS. If the game is
  installed to the hard drive, this initially involves transferring substantial
  amounts of data to speed up future access. When you insert a cartridge into a
  console like the SNES, the data is not transferred in the same way. Instead, the
  cartridge contains a ROM chip that, when inserted, integrates directly with the
  motherboard. From the operating system’s point of view, this ROM chip behaves as
  an extension of the system’s memory architecture.

  The primary advantage of a ROM-based game is significantly reduced latency. The
  game data can be accessed as rapidly as any other data moving along the buses of
  the motherboard. With a disk, the data must first be read into a buffer and then
  transferred to a memory address where it can be accessed by the CPU.

  With a ROM, while the console CPU still uses its own RAM to run the game
  (similar to disk-based systems), the game data on the ROM can be loaded into
  this RAM more quickly because it is directly attached to the motherboard. This
  makes the ROM cartridge effectively a fast, directly connected storage device.

  However, despite its speed, the limitation of a ROM cartridge is its storage
  capacity—it can hold only as much data as the chip can accommodate. In contrast,
  a disk can store much more data. This larger storage capacity allows not just
  for bigger and more complex games but also for the possibility of expanding the
  game through additional data downloaded to the console's hard drive. Such
  expansions or patches represent additional data from the console’s storage that,
  combined with the original disk data, are synthesized in memory at runtime for
  enhanced or updated gameplay experiences.

  (Note however that modern cartridge-based consoles like the Nintendo Switch can
  also combine ROM-data with expanded data from the console HDD, sourced from the
  internet.)

  Both disks and ROMs are forms of static storage in the context of a game
  console. While disks can technically be overwritten with new data, in practice,
  manufacturers implement various locks and protections to prevent this. Any
  changes or additions to the game, such as patches or expansions, are not written
  back to the original disk. Instead, these updates are stored on the console's
  hard drive.

  Therefore, if you download an expansion for a game on your Xbox, the original
  disk remains unchanged as when you first purchased it. Consequently, if you were
  to take that disk and load it on another Xbox, it would not include any of the
  expansions or updates. The game would function in its original form, as the
  additional content and updates reside solely on the hard drive of the first
  console where they were downloaded.

  ## Related notes

raw_content:
  ---
  id: rmvm
  title: ROM versus disk loading
  tags: [computer-architecture, memory]
  created: Monday, June 24, 2024
  ---

  # ROM versus disk loading

  I will explore the concept via the practical examples of cartridge-based and
  disk-based video games console.

  When you insert a disk into an Xbox, the game data is typically read from the
  disk and transferred into the console's memory by the OS. If the game is
  installed to the hard drive, this initially involves transferring substantial
  amounts of data to speed up future access. When you insert a cartridge into a
  console like the SNES, the data is not transferred in the same way. Instead, the
  cartridge contains a ROM chip that, when inserted, integrates directly with the
  motherboard. From the operating system’s point of view, this ROM chip behaves as
  an extension of the system’s memory architecture.

  The primary advantage of a ROM-based game is significantly reduced latency. The
  game data can be accessed as rapidly as any other data moving along the buses of
  the motherboard. With a disk, the data must first be read into a buffer and then
  transferred to a memory address where it can be accessed by the CPU.

  With a ROM, while the console CPU still uses its own RAM to run the game
  (similar to disk-based systems), the game data on the ROM can be loaded into
  this RAM more quickly because it is directly attached to the motherboard. This
  makes the ROM cartridge effectively a fast, directly connected storage device.

  However, despite its speed, the limitation of a ROM cartridge is its storage
  capacity—it can hold only as much data as the chip can accommodate. In contrast,
  a disk can store much more data. This larger storage capacity allows not just
  for bigger and more complex games but also for the possibility of expanding the
  game through additional data downloaded to the console's hard drive. Such
  expansions or patches represent additional data from the console’s storage that,
  combined with the original disk data, are synthesized in memory at runtime for
  enhanced or updated gameplay experiences.

  (Note however that modern cartridge-based consoles like the Nintendo Switch can
  also combine ROM-data with expanded data from the console HDD, sourced from the
  internet.)

  Both disks and ROMs are forms of static storage in the context of a game
  console. While disks can technically be overwritten with new data, in practice,
  manufacturers implement various locks and protections to prevent this. Any
  changes or additions to the game, such as patches or expansions, are not written
  back to the original disk. Instead, these updates are stored on the console's
  hard drive.

  Therefore, if you download an expansion for a game on your Xbox, the original
  disk remains unchanged as when you first purchased it. Consequently, if you were
  to take that disk and load it on another Xbox, it would not include any of the
  expansions or updates. The game would function in its original form, as the
  additional content and updates reside solely on the hard drive of the first
  console where they were downloaded.

  ## Related notes

metadata:
  {"created":"Monday, June 24, 2024","id":"rmvm","tags":["computer-architecture","memory"],"title":"ROM versus disk loading"}


path: zk/ROM_versus_disk_loading.md
sortable_path: zkROM_versus_disk_loading.md
title: ROM versus disk loading
lead: # ROM versus disk loading
checksum: 477d6de8d99f3fb26277cc5fe6d64cc8b3071b4eb7e53c51f593ba53d9887a0e
created: 2024-06-24 16:45:46.215734756+00:00
modified: 2024-06-24 16:42:46.898181187+00:00
body:
  # ROM versus disk loading

  I will explore the concept via the practical examples of cartridge-based and
  disk-based video games console.

  When you insert a disk into an XBox, the game data is transferred from the disk
  into the console's memory by the OS. When you insert a cartridge into a SNES,
  there is no transferral of data. The game data is stored on a chip and this chip
  is attached to the motherboard. From the point of view of the OS, it is just
  another chip along with the others that comprise the motherboard.

  The chief advantage with a ROM game is that the latency is greatly reduced. The
  game data can be accessed as rapidly as any other data travelling along the
  busses of the motherboard. With a disk, the bytes have to be transferred in a
  buffer from the disk into a memory address and from there, accessed by the CPU.

  With a ROM, the console CPU still uses its own memory to run the game (as with
  disk-based consoles), it is just that the game data on the ROM can be loaded
  into memory quicker because it is attached to the motherboard.

  becomes part of the motherboard just like the other chips that comprise the
  motherboard.

  ## Related notes

raw_content:
  ---
  id: rmvm
  title: ROM versus disk loading
  tags: [computer-architecture, memory]
  created: Monday, June 24, 2024
  ---

  # ROM versus disk loading

  I will explore the concept via the practical examples of cartridge-based and
  disk-based video games console.

  When you insert a disk into an XBox, the game data is transferred from the disk
  into the console's memory by the OS. When you insert a cartridge into a SNES,
  there is no transferral of data. The game data is stored on a chip and this chip
  is attached to the motherboard. From the point of view of the OS, it is just
  another chip along with the others that comprise the motherboard.

  The chief advantage with a ROM game is that the latency is greatly reduced. The
  game data can be accessed as rapidly as any other data travelling along the
  busses of the motherboard. With a disk, the bytes have to be transferred in a
  buffer from the disk into a memory address and from there, accessed by the CPU.

  With a ROM, the console CPU still uses its own memory to run the game (as with
  disk-based consoles), it is just that the game data on the ROM can be loaded
  into memory quicker because it is attached to the motherboard.

  becomes part of the motherboard just like the other chips that comprise the
  motherboard.

  ## Related notes

metadata:
  {"created":"Monday, June 24, 2024","id":"rmvm","tags":["computer-architecture","memory"],"title":"ROM versus disk loading"}


path: zk/Push_buttons.md
sortable_path: zkPush_buttons.md
title: Push buttons
lead: # Push buttons
checksum: 9de375b26634dde0143d49b59f0d69edf73b1aab4c31a33a077d721bcd316d07
created: 2024-06-25 04:45:25.356583385+00:00
modified: 2024-07-04 05:18:07.968396987+00:00
metadata: {"created":"Tuesday, June 25, 2024","id":"nsrv","tags":["electronics"],"title":"Push buttons"}
body:
  # Push buttons

  Push buttons are push-to-make switches. This means that they are **on when they
  pressed and off when they are not pressed**. A classic example would be the
  switches on a keyboard.

  Push buttons have four legs that are connected in signal-in and signal-out
  pairs.

  To create a breakable circuit we would connect a signal-in cable to the top left
  at row 23 and a signal-out cable to the bottom-right at row 25.

  ![Push button legs diagram](../img/push-button-legs.png)

  _The correct use of a push button where the button breaks the circuit_

  ![](../img/correct_push_button.jpg)

  To override the switch functionality and just have the button work as a
  connector we would connect the signal-in to the row 23 input and the row 23
  output.

  _The push button being used as simple connector which does not break the
  circuit:_

  ![](../img/incorrect_push_button.jpg)

  ## Related notes

  https://projects.raspberrypi.org/en/projects/button-switch-scratch-pi/1

raw_content:
  ---
  id: nsrv
  title: Push buttons
  tags: [electronics]
  created: Tuesday, June 25, 2024
  ---

  # Push buttons

  Push buttons are push-to-make switches. This means that they are **on when they
  pressed and off when they are not pressed**. A classic example would be the
  switches on a keyboard.

  Push buttons have four legs that are connected in signal-in and signal-out
  pairs.

  To create a breakable circuit we would connect a signal-in cable to the top left
  at row 23 and a signal-out cable to the bottom-right at row 25.

  ![Push button legs diagram](../img/push-button-legs.png)

  _The correct use of a push button where the button breaks the circuit_

  ![](../img/correct_push_button.jpg)

  To override the switch functionality and just have the button work as a
  connector we would connect the signal-in to the row 23 input and the row 23
  output.

  _The push button being used as simple connector which does not break the
  circuit:_

  ![](../img/incorrect_push_button.jpg)

  ## Related notes

  https://projects.raspberrypi.org/en/projects/button-switch-scratch-pi/1


path: zk/Working_with_JSON_in_Python.md
sortable_path: zkWorking_with_JSON_in_Python.md
title: Working with JSON in Python
lead: # Working with JSON in Python
checksum: 1c8c9d545b9864da4a876487a7e9a3a0b74a5e43f395d78179ef45ce4794d1fd
created: 2024-06-25 05:55:58.93390879+00:00
modified: 2024-06-25 05:57:27.664935775+00:00
body:
  # Working with JSON in Python

  ## `json.loads()`

  Convert a string containing JSON data into the native Python dictionary object.
  Equivalent to `JSON.parse()` in JavaScript.

  Will error if the input string is not properly formatted JSON. This will be
  `json.JSONDecodeError`, a subclass of the `ValueError`
  [exception](./Error_handling_in_Python.md)

  ## `json.dumps()`

  Take a JSON-formatted dictionary and convert it into a string. The reverse of
  `json.loads()` and equivalent to `JSON.stringify()` in JavaScript.

  ## JSON errors

  The `.loads()` and `.parse()` methods will throw an error if the input string is

  ## Related notes

raw_content:
  ---
  id: mjia
  title: Working with JSON in Python
  tags: [python, JSON]
  created: Tuesday, June 25, 2024
  ---

  # Working with JSON in Python

  ## `json.loads()`

  Convert a string containing JSON data into the native Python dictionary object.
  Equivalent to `JSON.parse()` in JavaScript.

  Will error if the input string is not properly formatted JSON. This will be
  `json.JSONDecodeError`, a subclass of the `ValueError`
  [exception](./Error_handling_in_Python.md)

  ## `json.dumps()`

  Take a JSON-formatted dictionary and convert it into a string. The reverse of
  `json.loads()` and equivalent to `JSON.stringify()` in JavaScript.

  ## JSON errors

  The `.loads()` and `.parse()` methods will throw an error if the input string is

  ## Related notes

metadata:
  {"created":"Tuesday, June 25, 2024","id":"mjia","tags":["python","JSON"],"title":"Working with JSON in Python"}


path: zk/Threads.md
sortable_path: zkThreads.md
title: Threads
lead: # Threads
checksum: 558f3dc51c16180831dfe1d1f7ab799df37f9208a64ad07407f565de34bcaa82
created: 2024-06-26 04:51:56.925975752+00:00
modified: 2024-07-07 17:14:47.423025923+00:00
metadata: {"created":"Wednesday, June 26, 2024","id":"zadl","tags":["operating-systems"],"title":"Threads"}
body:
  # Threads

  A ![process](Processes.md) is a running instance of a given program. A program
  runs sequentially handling one task at a time, however we may need to run
  certain tasks in parallel.

  Think of a program that is downloading a resource from the internet and wants to
  update the UI at the same time to show the download process.

  If the program is strictly sequential, once the program starts the download, the
  UI is neglected since the CPU time dedicated to the process must be focused
  elsewhere.

  We need the UI update and the download to execute in parallel. This is achieved
  via operating system **threads of execution**.

  > A thread is a schedulable unit of execution within a process. It runs within a
  > process, shares its memory address and can execute any program code loaded
  > within it.

  Each thread has an identifier called a _thread ID_ (TID).

  Both procesess in [user space](User_Space.md) and the [kernel](The_kernel.md)
  utilise threads.

  ## Hardware considerations

  Each [processor core](./CPU_architecture.md) can only run one thread at a time
  so the number of cores that a device's CPU has deterimines how many threads can
  run at once. In practice, then, the CPU shuffles between multiple threads in
  [clock segments](./Clock_signals.md). This is known as scheduling and is managed
  by the process that the thread belongs to.

  ## Related notes

  ![Processes](Processes.md)

raw_content:
  ---
  id: zadl
  title: Threads
  tags: [operating-systems]
  created: Wednesday, June 26, 2024
  ---

  # Threads

  A ![process](Processes.md) is a running instance of a given program. A program
  runs sequentially handling one task at a time, however we may need to run
  certain tasks in parallel.

  Think of a program that is downloading a resource from the internet and wants to
  update the UI at the same time to show the download process.

  If the program is strictly sequential, once the program starts the download, the
  UI is neglected since the CPU time dedicated to the process must be focused
  elsewhere.

  We need the UI update and the download to execute in parallel. This is achieved
  via operating system **threads of execution**.

  > A thread is a schedulable unit of execution within a process. It runs within a
  > process, shares its memory address and can execute any program code loaded
  > within it.

  Each thread has an identifier called a _thread ID_ (TID).

  Both procesess in [user space](User_Space.md) and the [kernel](The_kernel.md)
  utilise threads.

  ## Hardware considerations

  Each [processor core](./CPU_architecture.md) can only run one thread at a time
  so the number of cores that a device's CPU has deterimines how many threads can
  run at once. In practice, then, the CPU shuffles between multiple threads in
  [clock segments](./Clock_signals.md). This is known as scheduling and is managed
  by the process that the thread belongs to.

  ## Related notes

  ![Processes](Processes.md)


path: zk/Concise_subfield_mapping_JS.md
sortable_path: zkConcise_subfield_mapping_JS.md
title: Concise mapping of object subfileds in JS
lead: # Concise mapping of object subfileds in JS
checksum: 7367a3a8eb32ffc1bf9e7a454022658265ae66e709755533a9a54d8b358f5544
created: 2024-06-28 08:36:18.842907046+00:00
modified: 2024-06-28 08:38:31.208728774+00:00
body:
  # Concise mapping of object subfileds in JS

  ## Scenario

  You have an array of objects and you want to return the objects with only a
  subset of the fields.

  ## Implementation

  Standard approach with a map:

  ```js
  const arrayOfObjs = [
    { id: 12, name: "Thomas" },
    { id: 3, name: "Gerald" },
  ];

  // We just want the `name` property

  const subset = arrayOfObjs.map((obj) => {
    name: obj.name;
  });
  ```

  More concise approach with destructuring:

  ```js
  const subset = arrayOfObjs.map(({ name }) => ({ name }));
  ```

  ## Related notes

raw_content:
  ---
  id: dv3u
  title: Concise mapping of object subfileds in JS
  tags: []
  created: Friday, June 28, 2024
  ---

  # Concise mapping of object subfileds in JS

  ## Scenario

  You have an array of objects and you want to return the objects with only a
  subset of the fields.

  ## Implementation

  Standard approach with a map:

  ```js
  const arrayOfObjs = [
    { id: 12, name: "Thomas" },
    { id: 3, name: "Gerald" },
  ];

  // We just want the `name` property

  const subset = arrayOfObjs.map((obj) => {
    name: obj.name;
  });
  ```

  More concise approach with destructuring:

  ```js
  const subset = arrayOfObjs.map(({ name }) => ({ name }));
  ```

  ## Related notes

metadata:
  {"created":"Friday, June 28, 2024","id":"dv3u","tags":[],"title":"Concise mapping of object subfileds in JS"}


path: zk/Memory_Management_Unit.md
sortable_path: zkMemory_Management_Unit.md
title: Memory_Management_Unit
lead: # Memory Management Unit (MMU)
checksum: 3ce6bc8f79f37f9f2cb61dd5444c355eaeb0df48f2bb079fef9d7243f2017cdd
created: 2024-07-08 05:22:28.885795567+00:00
modified: 2024-07-16 06:22:12.036189248+00:00
metadata: {"created":"Monday, July 08, 2024","tags":["memory","Linux"],"title":"Memory_Management_Unit"}
body:
  # Memory Management Unit (MMU)

  The MMU is the bridge between the physical memory devices of the machine and the
  virtual memory space. It is a chip that sits between the CPU and the RAM and
  determines the physical location of the memory requested by the kernel as
  virtual memory.

  ![Virtual memory diagram](/img/virtual-memory-diagram.jpg)

  ## Pages

  We use the term **pages** to denote blocks of virtual memory and to distinguish
  them from **addresses** as physical blocks. The MMU possesses a **page table**
  which is registry logging which pages correspond to which physical blocks.

  ## Shared pages

  Virtual memory allows the sharing of files and memory by multiple processes.
  Crucially the shared data doesn't have to be within the address of each process,
  instead there is a reference in the page table that each process has access to
  the shared data.

  ## Page faults

  There are two kinds of error that can occur with relation to paged memory:

  - minor page faults
    - The desired page is in main memory but the MMU doesn't currently know where
      it is
  - major page faults
    - The desired page is not in main memory at all. Therefore the kernel must
      fetch it from disk

  Minor page faults are very common and are to be expected; they resolve quickly.
  On the other hand too many major page faults can slow the system down both
  because of the time-costly process of fetching data from disk and because it
  demands more kernel resources to locate the missing page, which puts other
  processes on hold.

raw_content:
  ---
  title: Memory_Management_Unit
  tags: [memory, Linux]
  created: Monday, July 08, 2024
  ---

  # Memory Management Unit (MMU)

  The MMU is the bridge between the physical memory devices of the machine and the
  virtual memory space. It is a chip that sits between the CPU and the RAM and
  determines the physical location of the memory requested by the kernel as
  virtual memory.

  ![Virtual memory diagram](/img/virtual-memory-diagram.jpg)

  ## Pages

  We use the term **pages** to denote blocks of virtual memory and to distinguish
  them from **addresses** as physical blocks. The MMU possesses a **page table**
  which is registry logging which pages correspond to which physical blocks.

  ## Shared pages

  Virtual memory allows the sharing of files and memory by multiple processes.
  Crucially the shared data doesn't have to be within the address of each process,
  instead there is a reference in the page table that each process has access to
  the shared data.

  ## Page faults

  There are two kinds of error that can occur with relation to paged memory:

  - minor page faults
    - The desired page is in main memory but the MMU doesn't currently know where
      it is
  - major page faults
    - The desired page is not in main memory at all. Therefore the kernel must
      fetch it from disk

  Minor page faults are very common and are to be expected; they resolve quickly.
  On the other hand too many major page faults can slow the system down both
  because of the time-costly process of fetching data from disk and because it
  demands more kernel resources to locate the missing page, which puts other
  processes on hold.


path: zk/VirtualMemory.md
sortable_path: zkVirtualMemory.md
title: Virtual memory
checksum: c6e2d4b86de8e0421423d58cb6d2cca2fd9b31d43c7b5ca3d64fa2fe0fa44e46
created: 2024-07-08 05:28:27.921902606+00:00
modified: 2024-07-16 06:14:27.871658561+00:00
metadata: {"tags":["memory","Linux","kernel"]}
lead:
  Virtual memory is an abstracted and idealised representation of the physical
  memory capacity of a machine that is presented to user space for its memory
  operations.

body:
  Virtual memory is an abstracted and idealised representation of the physical
  memory capacity of a machine that is presented to user space for its memory
  operations.

  When an OS implements virtual memory, [processes](./Processes.md) in
  [user space](./User_Space.md) cannot directly read or write to the physical
  memory. Instead they execute memory operations against virtual memory and the
  [kernel](./The_kernel.md) translates these into the actual operations against
  the memory hardware.

  The main benefits:

  - User mode processes do not have to be concerned with the physical memory
    management
  - There is a buffer between user mode processes and physical memory, meaning
    that memory cannot be accidentally corrupted by other processes in user space.

  Because the physical memory is abstracted, it can be the case that the physical
  [memory addresses](./Memory_addresses.md) are non-contiguous or even distributed
  accross different hardware components (such as the
  [cache](./Register_and_cache_memory.md) and [swap](./Swap_space.md)). Despite
  this, the memory addresses will appear contiguous in virtual memory. Each user
  space process is presented with the same range of available memory addresses and
  the same total capacity.

  It is also possible for the kernel to present user space with an available
  virtual memory capcacity that exceeds the current physical capacity of the
  machine:

  > _It's possible for the kernel and all running processes to request more bytes
  > of virtual memory than the total size of RAM. In that situation, the OS can
  > move move bytes of memory to secondary storage to make room in RAM for newly
  > requested memory._

  _How Computers Really Work_ (2021) p.206

  ## Virtual memory and the kernel

  The kernel itself utilises virtual memory.

  During the [boot_process](Boot_process.md), it runs in physical memory mode
  however it rapidly creates a 1:1 mapping of physical to virtual memory for
  itself. This direct mapping allows the kernel to easily access any physical
  memory location as needed, whilst retaining the benefits of virtual memory.

  With its own virtual memory established, it can then build additional virtual
  memory structures for itself and user processes.

  While the kernel uses virtual memory it can also access physical memory at any
  time wherease user space processes can only ever access virtual memory.

  The kernel virtual memory has a different range of virtual addresses to work
  with than user space virtual memory.

  Unlike user space virtual memory, the kernel has access to everything running in
  kernel address space whereas processes in user address space are partitioned
  from each other with separate address spaces that cannot interact.

  In addition to being able to access its own virtual memory (and physical memory,
  as required) the kernel can also access any user processes' virtual address.

  This said, the virtual memory space of the kernel and the virtual memory space
  of the user processes are distinct. It is not the case that the kernel is
  superset of all available virtual memory. It can access user space virtual
  memory because it sets up the tables and locations, not because it is a subset
  of its own virtual memory.

  ![Virtual memory diagram](/img/virtual-memory-diagram.jpg)

raw_content:
  ---
  tags:
    - memory
    - Linux
    - kernel
  ---

  # Virtual memory

  Virtual memory is an abstracted and idealised representation of the physical
  memory capacity of a machine that is presented to user space for its memory
  operations.

  When an OS implements virtual memory, [processes](./Processes.md) in
  [user space](./User_Space.md) cannot directly read or write to the physical
  memory. Instead they execute memory operations against virtual memory and the
  [kernel](./The_kernel.md) translates these into the actual operations against
  the memory hardware.

  The main benefits:

  - User mode processes do not have to be concerned with the physical memory
    management
  - There is a buffer between user mode processes and physical memory, meaning
    that memory cannot be accidentally corrupted by other processes in user space.

  Because the physical memory is abstracted, it can be the case that the physical
  [memory addresses](./Memory_addresses.md) are non-contiguous or even distributed
  accross different hardware components (such as the
  [cache](./Register_and_cache_memory.md) and [swap](./Swap_space.md)). Despite
  this, the memory addresses will appear contiguous in virtual memory. Each user
  space process is presented with the same range of available memory addresses and
  the same total capacity.

  It is also possible for the kernel to present user space with an available
  virtual memory capcacity that exceeds the current physical capacity of the
  machine:

  > _It's possible for the kernel and all running processes to request more bytes
  > of virtual memory than the total size of RAM. In that situation, the OS can
  > move move bytes of memory to secondary storage to make room in RAM for newly
  > requested memory._

  _How Computers Really Work_ (2021) p.206

  ## Virtual memory and the kernel

  The kernel itself utilises virtual memory.

  During the [boot_process](Boot_process.md), it runs in physical memory mode
  however it rapidly creates a 1:1 mapping of physical to virtual memory for
  itself. This direct mapping allows the kernel to easily access any physical
  memory location as needed, whilst retaining the benefits of virtual memory.

  With its own virtual memory established, it can then build additional virtual
  memory structures for itself and user processes.

  While the kernel uses virtual memory it can also access physical memory at any
  time wherease user space processes can only ever access virtual memory.

  The kernel virtual memory has a different range of virtual addresses to work
  with than user space virtual memory.

  Unlike user space virtual memory, the kernel has access to everything running in
  kernel address space whereas processes in user address space are partitioned
  from each other with separate address spaces that cannot interact.

  In addition to being able to access its own virtual memory (and physical memory,
  as required) the kernel can also access any user processes' virtual address.

  This said, the virtual memory space of the kernel and the virtual memory space
  of the user processes are distinct. It is not the case that the kernel is
  superset of all available virtual memory. It can access user space virtual
  memory because it sets up the tables and locations, not because it is a subset
  of its own virtual memory.

  ![Virtual memory diagram](/img/virtual-memory-diagram.jpg)


path: zk/Memory_addresses.md
sortable_path: zkMemory_addresses.md
title: Memory_addresses
lead: # Memory addresses
checksum: e36e1abbe4fb6b5e17f05aaa1e44942face29aa7ee6ff5e97937a734a16c1a39
created: 2024-07-12 05:50:46.67372147+00:00
modified: 2024-07-12 06:22:15.703266816+00:00
metadata: {"created":"Friday, July 12, 2024","tags":["memory"],"title":"Memory_addresses"}
body:
  # Memory addresses

  > Computers assign numeric addresses to bytes of memory and the CPU can read or
  > write to those addresses

  We can think of the internals of RAM as grids of memory cells.

  Each single-bit cell in the grid can be identified using two dimensional
  coordinates, like in a graph. The coordinates are the location of that cell in
  the grid.

  Handling one bit at a time isn't very efficient so RAM accesses **multiple
  grids** of 1-bit memory cells in parallel. This allows for reads or writes of
  multiple bits at once, such as a whole byte.

  The location of a set of bits in memory is known as a **memory address**.

  ### Demonstration

  Let's imagine we have a computer system that can address up to 64KB of memory
  and our system is byte addressable. This means there are
  $64 \cdot 1024 = 65,536$ bytes of memory because 1KB = 1024 bytes.

  We therefore have 65,536 addresses and each address can store one byte. So our
  addresses go from 0 to 65, 535.

  We now need to consider how many bits we need to uniquely represent an address
  on this system.

  What does this mean? Although there are approximately 64 thousand bytes of
  memory, to refer to each byte we can't just use 1, 2, 3... because computers use
  binary numbers. We need a binary number to refer to a given byte in the the 64KB
  of memory. The question we are asking is: how long does this binary number need
  to be to be able to represent each of the 64 thousand bytes?

  1 bit can represent two addresses: 0 and 1. 2 bits can represent four addresses:
  00, 01, 10, 11. The formula is as follows: number of addresses = $2^n$ where $n$
  is the number of bits.

  We need to reverse this formula to find out how many bits we need to represent a
  given number of addresses. We can do this with a [logarithm](Logarithms.md).

  We can reverse the formula as follows: number of bits = $\log_2$ (number of
  addresses).

  In our case we have 65,536 addresses so we need $\log_2(65,536)$ bits to
  represent each address. This is approximately 16 bits. Thus a 16 bit memory
  iaddress is needed to address 65, 546 bytes.

  Using memory addresses we end up with tables like the following:

  | Memory address   | Data             |
  | ---------------- | ---------------- |
  | 0000000000000000 | 1010101010101010 |
  | 0000000000000001 | 0010001001001011 |
  | 0000000000000010 | 0010001001001010 |

  This is hard to parse so we can instead use
  [hexadecimal numbers](Hexadecimal_number_system.md) to represent the addresses:

  | Memory address (as hex) | Data (as binary) |
  | ----------------------- | ---------------- |
  | 0x0000                  | 1010101010101010 |
  | 0x0001                  | 0010001001001011 |
  | 0x0002                  | 0010001001001010 |

  By itself, the the data is meaningless but we know from
  [binary encoding](Binary_encoding.md) that the binary data will correspond to
  some meaningful data, such as a character or a colour, depending on the encoding
  scheme used. The above table could correspond to the characters for 'A', 'B' and
  'C' in the ASCII encoding scheme:

  | Memory address (as hex) | Data (as binary) | Data (as ASCII) |
  | ----------------------- | ---------------- | --------------- |
  | 0x0000                  | 1010101010101010 | A               |
  | 0x0001                  | 0010001001001011 | B               |
  | 0x0002                  | 0010001001001010 | C               |

raw_content:
  ---
  title: Memory_addresses
  tags: [memory]
  created: Friday, July 12, 2024
  ---

  # Memory addresses

  > Computers assign numeric addresses to bytes of memory and the CPU can read or
  > write to those addresses

  We can think of the internals of RAM as grids of memory cells.

  Each single-bit cell in the grid can be identified using two dimensional
  coordinates, like in a graph. The coordinates are the location of that cell in
  the grid.

  Handling one bit at a time isn't very efficient so RAM accesses **multiple
  grids** of 1-bit memory cells in parallel. This allows for reads or writes of
  multiple bits at once, such as a whole byte.

  The location of a set of bits in memory is known as a **memory address**.

  ### Demonstration

  Let's imagine we have a computer system that can address up to 64KB of memory
  and our system is byte addressable. This means there are
  $64 \cdot 1024 = 65,536$ bytes of memory because 1KB = 1024 bytes.

  We therefore have 65,536 addresses and each address can store one byte. So our
  addresses go from 0 to 65, 535.

  We now need to consider how many bits we need to uniquely represent an address
  on this system.

  What does this mean? Although there are approximately 64 thousand bytes of
  memory, to refer to each byte we can't just use 1, 2, 3... because computers use
  binary numbers. We need a binary number to refer to a given byte in the the 64KB
  of memory. The question we are asking is: how long does this binary number need
  to be to be able to represent each of the 64 thousand bytes?

  1 bit can represent two addresses: 0 and 1. 2 bits can represent four addresses:
  00, 01, 10, 11. The formula is as follows: number of addresses = $2^n$ where $n$
  is the number of bits.

  We need to reverse this formula to find out how many bits we need to represent a
  given number of addresses. We can do this with a [logarithm](Logarithms.md).

  We can reverse the formula as follows: number of bits = $\log_2$ (number of
  addresses).

  In our case we have 65,536 addresses so we need $\log_2(65,536)$ bits to
  represent each address. This is approximately 16 bits. Thus a 16 bit memory
  iaddress is needed to address 65, 546 bytes.

  Using memory addresses we end up with tables like the following:

  | Memory address   | Data             |
  | ---------------- | ---------------- |
  | 0000000000000000 | 1010101010101010 |
  | 0000000000000001 | 0010001001001011 |
  | 0000000000000010 | 0010001001001010 |

  This is hard to parse so we can instead use
  [hexadecimal numbers](Hexadecimal_number_system.md) to represent the addresses:

  | Memory address (as hex) | Data (as binary) |
  | ----------------------- | ---------------- |
  | 0x0000                  | 1010101010101010 |
  | 0x0001                  | 0010001001001011 |
  | 0x0002                  | 0010001001001010 |

  By itself, the the data is meaningless but we know from
  [binary encoding](Binary_encoding.md) that the binary data will correspond to
  some meaningful data, such as a character or a colour, depending on the encoding
  scheme used. The above table could correspond to the characters for 'A', 'B' and
  'C' in the ASCII encoding scheme:

  | Memory address (as hex) | Data (as binary) | Data (as ASCII) |
  | ----------------------- | ---------------- | --------------- |
  | 0x0000                  | 1010101010101010 | A               |
  | 0x0001                  | 0010001001001011 | B               |
  | 0x0002                  | 0010001001001010 | C               |


path: zk/What_is_memory.md
sortable_path: zkWhat_is_memory.md
title: What_is_memory
lead: # What is memory ?
checksum: b606c535120fcd637dae315b5f58810136ea8851abfe705b4f2b8aee90016d18
created: 2024-07-12 06:24:41.184588602+00:00
modified: 2024-07-12 07:34:47.187378061+00:00
metadata: {"created":"Friday, July 12, 2024","tags":["memory"],"title":"What_is_memory"}
body:
  # What is memory ?

  > A CPU is just an operator on memory. It reads its instructions and data from
  > the memory and writes back out to the memory. (Ward 2021)

  When a [CPU](CPU_architecture.md) executes a program, it needs a place to store
  the program's **instructions** and **related data**. This is the role of memory.

  The data that comprises a program is a series of bits. The basic unit of memory
  storage is a **memory cell**: a circuit that can store a single bit.

  [RAM](./DRAM_and_SRAM_memory.md) memory is _volatile_ : the memory is only
  retained whilst the computer has a power supply and is wiped when the computer
  is rebooted. This contrasts with the memory of the harddisk which is
  non-volatile and is retained after a reboot.

  Programs that are executing are loaded into memory because the chips that
  comprise memory can read and store data much faster than the harddisk. It would
  be possible to run a program from the harddisk but it would be 500 - 1000 times
  slower than memory.

raw_content:
  ---
  title: What_is_memory
  tags: [memory]
  created: Friday, July 12, 2024
  ---

  # What is memory ?

  > A CPU is just an operator on memory. It reads its instructions and data from
  > the memory and writes back out to the memory. (Ward 2021)

  When a [CPU](CPU_architecture.md) executes a program, it needs a place to store
  the program's **instructions** and **related data**. This is the role of memory.

  The data that comprises a program is a series of bits. The basic unit of memory
  storage is a **memory cell**: a circuit that can store a single bit.

  [RAM](./DRAM_and_SRAM_memory.md) memory is _volatile_ : the memory is only
  retained whilst the computer has a power supply and is wiped when the computer
  is rebooted. This contrasts with the memory of the harddisk which is
  non-volatile and is retained after a reboot.

  Programs that are executing are loaded into memory because the chips that
  comprise memory can read and store data much faster than the harddisk. It would
  be possible to run a program from the harddisk but it would be 500 - 1000 times
  slower than memory.


path: zk/DRAM_and_SRAM_memory.md
sortable_path: zkDRAM_and_SRAM_memory.md
title: DRAM_and_SRAM_memory
lead: # DRAM_and_SRAM_memory
checksum: d9add1e54ecfb157315178815bcfef9596825282beb95cc7cfca7e8c16039f9c
created: 2024-07-12 06:25:44.482673223+00:00
modified: 2024-07-12 07:38:27.070728108+00:00
metadata: {"created":"Friday, July 12, 2024","tags":["memory","hardware"],"title":"DRAM_and_SRAM_memory"}
body:
  # DRAM_and_SRAM_memory

  There are two types of RAM memory: [SRAM and DRAM](./DRAM_and_SRAM_memory.md).

  ### DRAM

  DRAM uses capacitors to create the memory cell:

  > a **capacitor** is an electronic component that stores electrical energy in an
  > electrical field. A device which can accumulate and release electrical charge.

  In a DRAM cell, each bit of data is stored as a charge in a capacitor. The
  presence of charge represents a '1' bit and the absence of charge represents a
  '0' bit. Each of these cells is paired with a [transistor](Transistors.md) that
  controls the reading and writing of data.

  However capacitors lose [charge](Current.md) over time due to leaks. As a result
  DRAM is memory that needs to be refreshed (recharged) frequently. For this
  reason and because it only uses one transistor and capacitor per bit, DRAM is
  the less expensive form of volatile memory.

  ### SRAM

  SRAM (Static Random Access Memory) is also volatile memory but its electronical
  implementation is different. Unlike DRAM it doesn't use capacitors. Consequently
  the transistors do not leak or need to be refreshed, hence why SRAM is _static_
  and DRAM is _dynamic_.

  SRAM uses [flip flops](Flip_flops.md) to store the bits. It also uses multiple
  transistors per bit. This makes it faster than DRAM but more expensive. DRAM is
  at least ten times slower than SRAM.

raw_content:
  ---
  title: DRAM_and_SRAM_memory
  tags: [memory, hardware]
  created: Friday, July 12, 2024
  ---

  # DRAM_and_SRAM_memory

  There are two types of RAM memory: [SRAM and DRAM](./DRAM_and_SRAM_memory.md).

  ### DRAM

  DRAM uses capacitors to create the memory cell:

  > a **capacitor** is an electronic component that stores electrical energy in an
  > electrical field. A device which can accumulate and release electrical charge.

  In a DRAM cell, each bit of data is stored as a charge in a capacitor. The
  presence of charge represents a '1' bit and the absence of charge represents a
  '0' bit. Each of these cells is paired with a [transistor](Transistors.md) that
  controls the reading and writing of data.

  However capacitors lose [charge](Current.md) over time due to leaks. As a result
  DRAM is memory that needs to be refreshed (recharged) frequently. For this
  reason and because it only uses one transistor and capacitor per bit, DRAM is
  the less expensive form of volatile memory.

  ### SRAM

  SRAM (Static Random Access Memory) is also volatile memory but its electronical
  implementation is different. Unlike DRAM it doesn't use capacitors. Consequently
  the transistors do not leak or need to be refreshed, hence why SRAM is _static_
  and DRAM is _dynamic_.

  SRAM uses [flip flops](Flip_flops.md) to store the bits. It also uses multiple
  transistors per bit. This makes it faster than DRAM but more expensive. DRAM is
  at least ten times slower than SRAM.


path: zk/Role_of_memory_in_computation.md
sortable_path: zkRole_of_memory_in_computation.md
title: Role_of_memory_in_computation
lead: # The role of memory in computation
checksum: 4acd0a1948e7a47e9c71465aead421d563f2739a6e101bc54dcb68f23ae19455
created: 2024-07-12 07:36:37.987161729+00:00
modified: 2024-07-12 07:37:30.31914591+00:00
metadata: {"created":"Friday, July 12, 2024","tags":["memory"],"title":"Role_of_memory_in_computation"}
body:
  # The role of memory in computation

  The following steps outline the way in which memory interacts with the processor
  during computational cycles, once the [bootstrapping](Boot_process.md) process
  has completed and the OS kernel is itself loaded into memory.

  1. A file is loaded from the harddisk into memory.
  2. The instruction at the first address is sent to the CPU, travelling accross
     the data bus part of the [system bus](Bus.md).
  3. The CPU processes this instruction and then sends a request accross the
     address bus part of the system bus for the next instruction to the memory
     controller within the [chipset](Chipset_and_controllers.md).
  4. The chipset finds where this instruction is stored within the
     [DRAM](Memory.md#dram) and issues a request to have it read out and send to
     the CPU over the data bus.

  > This is a simplified account; it is not the case that only single requests are
  > passed back and forth. This would be inefficient and time-wasting. The kernel
  > sends to the CPU not just the first instruction in the requested file but also
  > a number of instructions that immediately follow it.

  ![Memory flow diagram](/img/memory-flow.svg)

  Every part of the above process - the journey accross the bus, the lookup in the
  controller, the operations on the DRAM, the journey back accross the bus - takes
  multiple CPU clock cycles.

raw_content:
  ---
  title: Role_of_memory_in_computation
  tags: [memory]
  created: Friday, July 12, 2024
  ---

  # The role of memory in computation

  The following steps outline the way in which memory interacts with the processor
  during computational cycles, once the [bootstrapping](Boot_process.md) process
  has completed and the OS kernel is itself loaded into memory.

  1. A file is loaded from the harddisk into memory.
  2. The instruction at the first address is sent to the CPU, travelling accross
     the data bus part of the [system bus](Bus.md).
  3. The CPU processes this instruction and then sends a request accross the
     address bus part of the system bus for the next instruction to the memory
     controller within the [chipset](Chipset_and_controllers.md).
  4. The chipset finds where this instruction is stored within the
     [DRAM](Memory.md#dram) and issues a request to have it read out and send to
     the CPU over the data bus.

  > This is a simplified account; it is not the case that only single requests are
  > passed back and forth. This would be inefficient and time-wasting. The kernel
  > sends to the CPU not just the first instruction in the requested file but also
  > a number of instructions that immediately follow it.

  ![Memory flow diagram](/img/memory-flow.svg)

  Every part of the above process - the journey accross the bus, the lookup in the
  controller, the operations on the DRAM, the journey back accross the bus - takes
  multiple CPU clock cycles.


path: zk/Register_and_cache_memory.md
sortable_path: zkRegister_and_cache_memory.md
title: Register_and_cache_memory
lead: # Register and cache memory
checksum: 8c24f5a1ba6fea0d3c7d7acaeeaafe4004c5ecd2940eee916479d53e3c758987
created: 2024-07-12 07:38:42.783507026+00:00
modified: 2024-07-12 07:43:28.307933469+00:00
metadata: {"created":"Friday, July 12, 2024","tags":["memory","CPU"],"title":"Register_and_cache_memory"}
body:
  # Register and cache memory

  The CPU has its own memory devices in the form of registers and cache memory.

  [Registers](./CPU_architecture.md#Registers) are a form of memory that are
  positioned on the same chip as the CPU. They are very fast but can only store a
  small amount of data. They are used to store the results of calculations and the
  addresses of the next instructions to be processed.

  The cache is SRAM memory that is separate from the DRAM memory which comprises
  the main memory. It exists in order to boost perfomance when executing the
  read/request cycles of the steps detailed above.

  There are two types of cache memory:

  - L1 cache
    - Situated on the CPU chip itself
    - Fastest to access but stores less
  - L2 cache
    - Situated outside of the CPU on its own chip
    - Slower to access than L1 but can store more data

  The L1 cache is the fastest since the data has less distance to travel when
  moving to and from the CPU. This said, the L2 cache is still very fast when
  compared to the main memory, both because it is SRAM rather than DRAM and
  because it is closer to the processor than the main memory.

  Cache controllers use complex algorithms to determine what should go into the
  cache to facilitate the best performance, but generally they work on the
  principle that what has been previously used by the CPU will be requested again
  soon. If the CPU has just asked for an instruction at memory location 555 it's
  very likely that it will next ask for the one at 556, and after that the one at
  557 and so on. The cache's controller circuits therefore go ahead and fetch
  these from slow [DRAM to fast SRAM](./DRAM_and_SRAM_memory.md).

raw_content:
  ---
  title: Register_and_cache_memory
  tags: [memory, CPU]
  created: Friday, July 12, 2024
  ---

  # Register and cache memory

  The CPU has its own memory devices in the form of registers and cache memory.

  [Registers](./CPU_architecture.md#Registers) are a form of memory that are
  positioned on the same chip as the CPU. They are very fast but can only store a
  small amount of data. They are used to store the results of calculations and the
  addresses of the next instructions to be processed.

  The cache is SRAM memory that is separate from the DRAM memory which comprises
  the main memory. It exists in order to boost perfomance when executing the
  read/request cycles of the steps detailed above.

  There are two types of cache memory:

  - L1 cache
    - Situated on the CPU chip itself
    - Fastest to access but stores less
  - L2 cache
    - Situated outside of the CPU on its own chip
    - Slower to access than L1 but can store more data

  The L1 cache is the fastest since the data has less distance to travel when
  moving to and from the CPU. This said, the L2 cache is still very fast when
  compared to the main memory, both because it is SRAM rather than DRAM and
  because it is closer to the processor than the main memory.

  Cache controllers use complex algorithms to determine what should go into the
  cache to facilitate the best performance, but generally they work on the
  principle that what has been previously used by the CPU will be requested again
  soon. If the CPU has just asked for an instruction at memory location 555 it's
  very likely that it will next ask for the one at 556, and after that the one at
  557 and so on. The cache's controller circuits therefore go ahead and fetch
  these from slow [DRAM to fast SRAM](./DRAM_and_SRAM_memory.md).


path: zk/The_memory_hierarchy.md
sortable_path: zkThe_memory_hierarchy.md
title: The_memory_hierarchy
lead: # The memory hierarchy
checksum: 9d6fff2067f9ad50c3adbeddfe1cb39491e6152c68b5fedcb9ea0449d99e8245
created: 2024-07-12 07:43:51.419849451+00:00
modified: 2024-07-12 07:44:31.205911868+00:00
metadata: {"created":"Friday, July 12, 2024","tags":["memory"],"title":"The_memory_hierarchy"}
body:
  # The memory hierarchy

  The diagram below compares the different forms of memory within a computing
  device in terms of speed, monetary cost and capacity:

  ![Memory hierarchy diagram](/img/Memory-Hierarchy.jpg)

raw_content:
  ---
  title: The_memory_hierarchy
  tags: [memory]
  created: Friday, July 12, 2024
  ---

  # The memory hierarchy

  The diagram below compares the different forms of memory within a computing
  device in terms of speed, monetary cost and capacity:

  ![Memory hierarchy diagram](/img/Memory-Hierarchy.jpg)


path: zk/Operating_system_API.md
sortable_path: zkOperating_system_API.md
title: Operating_system_API
lead: # Operating system API
checksum: 73c948438a3b19dcc71fd151540c37e209a25138f449ce2518e9bbbee7b86f4b
created: 2024-07-17 05:57:15.872985575+00:00
modified: 2024-07-20 10:22:53.307423218+00:00
metadata: {"created":"Wednesday, July 17, 2024","tags":["operating-systems"],"title":"Operating_system_API"}
body:
  # Operating system API

  A user can interact with the operating system either via the GUI or the
  [shell](Intro_to_the_Unix_shell.md). (Both ultimately access the OS via the
  shell since the UI is an abstraction layer of the shell.)

  When _programs_ interact with the operating system they use the **operating
  system's API** directly. The GUI and shell also invoke the OS API but the access
  is mediated.

  Every OS has an API. This specification details how a program should interact
  with the OS, exposing functions, properties and data types.

  Whether using the GUI or a terminal emulator, the shell translates the user's
  commands into API calls. The API the invokes internal operating system code to
  perform the action.

  ![OS API diagram](../img/OS-api.svg)

  ## Example: opening a file

  A user can use a file explorer and the mouse to open a file. Or they can open
  the file by invoking it from the terminal.

  A running application doesn't have to do this. If it's remit includes opening
  files, it can do so by calling the OS API.

  On a Unix-like system the applicaton would use the `open` API function to open
  the file:

  ```
  open("foo.txt", 0_WRONLY|0_CREAT)
  ```

  This command makes the action write-only and creates the file because it does
  not yet exist. When the user opens a file in their File Explorer, the
  application translates their graphical actions into this API call.

  ## Different OSs have different APIs

  Unix-like and Linux operating systems use variations on the **Portable Operating
  System Interface** (POSIX) standard. This defines the OS API and also the rules
  for the shell's behaviour and its included utilities.

  Variations on this include _Cocoa_, macOS's API and Android's _Android Platform_
  API.

  Windows has its own API called _The Windows API_. The original version was
  16-bit called _Win16_, then when it moved to 32-bits it became _Win32_. When it
  moved to an x86 [instruction_set_architecture](Instruction_set_architectures.md)
  it became _Win64_. Starting with Windows 10, it introduced the _Universal
  Windows Platform_ which aims to make app development consistent accross
  different devices that run Windows.

raw_content:
  ---
  title: Operating_system_API
  tags: [operating-systems]
  created: Wednesday, July 17, 2024
  ---

  # Operating system API

  A user can interact with the operating system either via the GUI or the
  [shell](Intro_to_the_Unix_shell.md). (Both ultimately access the OS via the
  shell since the UI is an abstraction layer of the shell.)

  When _programs_ interact with the operating system they use the **operating
  system's API** directly. The GUI and shell also invoke the OS API but the access
  is mediated.

  Every OS has an API. This specification details how a program should interact
  with the OS, exposing functions, properties and data types.

  Whether using the GUI or a terminal emulator, the shell translates the user's
  commands into API calls. The API the invokes internal operating system code to
  perform the action.

  ![OS API diagram](../img/OS-api.svg)

  ## Example: opening a file

  A user can use a file explorer and the mouse to open a file. Or they can open
  the file by invoking it from the terminal.

  A running application doesn't have to do this. If it's remit includes opening
  files, it can do so by calling the OS API.

  On a Unix-like system the applicaton would use the `open` API function to open
  the file:

  ```
  open("foo.txt", 0_WRONLY|0_CREAT)
  ```

  This command makes the action write-only and creates the file because it does
  not yet exist. When the user opens a file in their File Explorer, the
  application translates their graphical actions into this API call.

  ## Different OSs have different APIs

  Unix-like and Linux operating systems use variations on the **Portable Operating
  System Interface** (POSIX) standard. This defines the OS API and also the rules
  for the shell's behaviour and its included utilities.

  Variations on this include _Cocoa_, macOS's API and Android's _Android Platform_
  API.

  Windows has its own API called _The Windows API_. The original version was
  16-bit called _Win16_, then when it moved to 32-bits it became _Win32_. When it
  moved to an x86 [instruction_set_architecture](Instruction_set_architectures.md)
  it became _Win64_. Starting with Windows 10, it introduced the _Universal
  Windows Platform_ which aims to make app development consistent accross
  different devices that run Windows.


path: zk/Memory_versus_processor.md
sortable_path: zkMemory_versus_processor.md
title: Memory_versus_processor
lead: # Memory_versus_processor
checksum: f6019a3a6d127b96aeab6d08044e83cd194fd46934da16bd605e518811cb64ca
created: 2024-08-01 17:07:42.214100662+00:00
modified: 2024-08-03 06:03:19.733869594+00:00
metadata: {"created":"Thursday, August 01, 2024","tags":["question"],"title":"Memory_versus_processor"}
body:
  # Memory_versus_processor

  > Would a more powerful processor with average or reduced memory capacity
  > perform better than, equal to, or less than a less powerful processor with
  > more memory, in principle?

raw_content:
  ---
  title: Memory_versus_processor
  tags: ["question"]
  created: Thursday, August 01, 2024
  ---

  # Memory_versus_processor

  > Would a more powerful processor with average or reduced memory capacity
  > perform better than, equal to, or less than a less powerful processor with
  > more memory, in principle?


path: zk/Headless_Raspi_network_setup.md
sortable_path: zkHeadless_Raspi_network_setup.md
title: Headless network setup
lead: # Headless network setup
checksum: 45cab392fa277f763ea8e494149f7eefb56322460a63bc74c43e2c4cb0d72147
created: 2024-08-03 06:05:43.878164659+00:00
modified: 2024-08-03 06:04:15.37915733+00:00
body:
  # Headless network setup

  If you are using a headless system and you aren't using an ethernet connection,
  you face a chicken-egg issue of how to connect the device to WiFi via `ssh` when
  the device isn't yet on the network.

  You can get around this by saving a file called `wp_supplicant.conf` to the
  `/boot` directory of the device. This specifies the SSID name and password and
  will be used to auto-connect when the device boots.

  For example:

  ```
  country=GB
  ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
  update_config=1
  network={
       ssid="SKYVL7XP"
       scan_ssid=1
       psk="WIFI_PASSWORD"
       key_mgmt=WPA-PSK
  }
  ```

  ## Related notes

raw_content:
  ---
  id: hb10
  title: Headless network setup
  tags:
    - networks
    - Linux
    - raspberry-pi
    - procedural
  created: Wednesday, June 12, 2024
  ---

  # Headless network setup

  If you are using a headless system and you aren't using an ethernet connection,
  you face a chicken-egg issue of how to connect the device to WiFi via `ssh` when
  the device isn't yet on the network.

  You can get around this by saving a file called `wp_supplicant.conf` to the
  `/boot` directory of the device. This specifies the SSID name and password and
  will be used to auto-connect when the device boots.

  For example:

  ```
  country=GB
  ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
  update_config=1
  network={
       ssid="SKYVL7XP"
       scan_ssid=1
       psk="WIFI_PASSWORD"
       key_mgmt=WPA-PSK
  }
  ```

  ## Related notes

metadata:
  {"created":"Wednesday, June 12, 2024","id":"hb10","tags":["networks","Linux","raspberry-pi","procedural"],"title":"Headless network setup"}


path: zk/Network_fundamentals.md
sortable_path: zkNetwork_fundamentals.md
title: Network_fundamentals
lead: # Network fundamentals
checksum: 26b9d99bc3ef71e6235f901fe58c01eeb0c31f256b96047fed64bc4498dfdd42
created: 2024-08-03 06:06:05.867268631+00:00
modified: 2024-08-03 06:19:21.432457619+00:00
body:
  # Network fundamentals

  > A network is a system that allows computing devices to communicate and
  > exchange information with each other.

  In order for devices to be able to communicate they must share a common
  **communication protocol**.

  A protocol specifies **rules determining how information is to be exchanged**.
  (Simply connecting two devices is not sufficient for them to be able to
  communicate, they must have a shared language.)

  The nodes of a network are called [hosts](Network_hosts.md).

raw_content:
  ---
  title: Network_fundamentals
  tags: [networks, network-protocols]
  created: Saturday, August 03, 2024
  ---

  # Network fundamentals

  > A network is a system that allows computing devices to communicate and
  > exchange information with each other.

  In order for devices to be able to communicate they must share a common
  **communication protocol**.

  A protocol specifies **rules determining how information is to be exchanged**.
  (Simply connecting two devices is not sufficient for them to be able to
  communicate, they must have a shared language.)

  The nodes of a network are called [hosts](Network_hosts.md).

metadata:
  {"created":"Saturday, August 03, 2024","tags":["networks","network-protocols"],"title":"Network_fundamentals"}


path: zk/Torrenting.md
sortable_path: zkTorrenting.md
title: Torrenting
lead: # Torrenting
checksum: 67ad1ed9c215a955fa06c80d53b9b42f8ad1432c4993a8ecddcdf61c46bf5f30
created: 2024-08-03 06:10:21.423432559+00:00
modified: 2024-08-09 15:29:45.443865681+00:00
metadata: {"created":"Saturday, August 03, 2024","tags":["question","networks"],"title":"Torrenting"}
body:
  # Torrenting

  **What is torrenting? Is this the same thing as a peer-to-peer network? How do
  these technologies fit within the client-server paradigm?**

  Torrenting is a specific application of
  [peer_to_peer_networking](Peer_to_peer_network.md) applied to file-sharing.

  Each peer on the network both consumes resources and makes resources available
  to others.

  ## Procedure

  Large files are divided into small equal sized pieces and each piece is assigned
  a unique hash identifier. The `.torrent` file details the number of pieces,
  their sizes and hash values.

  Clients request different pieces from different peers simultaneously. As pieces
  are received, they are verified against their hash. Corrupted or incorrect
  pieces are discarded and re-downloaded.

  The client software reassembles the pieces in the correct order and the final
  file will be identical to the original which can be verified by comparing
  hashes.

  ## Key terms

  - _tracker_
    - a server that coordinates communication between peers (so there is a server
      but its role is more lightweight than client-server, coordinating the peers
      rather than hosting or distributing file content)
  - _swarm_
    - a group of peers sharing the same file
  - _seeding_
    - uploading file pieces to other peers
  - _leeching_
    - downloading file pieces from other peers

  > An alternative to a tracker server is to use trackerless methods such as
  > distributed hash tables

  ## Terminological clarification

  - _BitTorrent_ is the communications [protocols](Network_protocols.md) for P2P
    file sharing. It was also a client software developed by the company
    BitTorrent Inc.

  - _torrenting_ is the general terms for that act of sharing files using the
    BitTorrent protocol

  ## Napster

  Napster pre-dated the emergence of the BitTorrent protocol. It did not segment
  files and files were downloaded directly from peers. It was a centralised P2P
  network because there was a central server that indexed all available files
  which kept a database of all connected users and their files.

  When you searched Napster you would be querying this DB on the server. The
  server would facilitate a direct connection between the consumer and provider
  peer for the data exchange.

  When users share with BitTorrent the `.torrent` file is the source of truth and
  a swarm is created around the users who have possess the file.

raw_content:
  ---
  title: Torrenting
  tags: [question, networks]
  created: Saturday, August 03, 2024
  ---

  # Torrenting

  **What is torrenting? Is this the same thing as a peer-to-peer network? How do
  these technologies fit within the client-server paradigm?**

  Torrenting is a specific application of
  [peer_to_peer_networking](Peer_to_peer_network.md) applied to file-sharing.

  Each peer on the network both consumes resources and makes resources available
  to others.

  ## Procedure

  Large files are divided into small equal sized pieces and each piece is assigned
  a unique hash identifier. The `.torrent` file details the number of pieces,
  their sizes and hash values.

  Clients request different pieces from different peers simultaneously. As pieces
  are received, they are verified against their hash. Corrupted or incorrect
  pieces are discarded and re-downloaded.

  The client software reassembles the pieces in the correct order and the final
  file will be identical to the original which can be verified by comparing
  hashes.

  ## Key terms

  - _tracker_
    - a server that coordinates communication between peers (so there is a server
      but its role is more lightweight than client-server, coordinating the peers
      rather than hosting or distributing file content)
  - _swarm_
    - a group of peers sharing the same file
  - _seeding_
    - uploading file pieces to other peers
  - _leeching_
    - downloading file pieces from other peers

  > An alternative to a tracker server is to use trackerless methods such as
  > distributed hash tables

  ## Terminological clarification

  - _BitTorrent_ is the communications [protocols](Network_protocols.md) for P2P
    file sharing. It was also a client software developed by the company
    BitTorrent Inc.

  - _torrenting_ is the general terms for that act of sharing files using the
    BitTorrent protocol

  ## Napster

  Napster pre-dated the emergence of the BitTorrent protocol. It did not segment
  files and files were downloaded directly from peers. It was a centralised P2P
  network because there was a central server that indexed all available files
  which kept a database of all connected users and their files.

  When you searched Napster you would be querying this DB on the server. The
  server would facilitate a direct connection between the consumer and provider
  peer for the data exchange.

  When users share with BitTorrent the `.torrent` file is the source of truth and
  a swarm is created around the users who have possess the file.


path: zk/Network_hosts.md
sortable_path: zkNetwork_hosts.md
title: Network_hosts
lead: # Network_hosts
checksum: fb5bf5a38cec727e2227be1881423fe0a47a380352200135880c20451c292b43
created: 2024-08-03 06:14:23.069677144+00:00
modified: 2024-08-06 05:28:57.236805772+00:00
metadata: {"created":"Saturday, August 03, 2024","tags":["networks"],"title":"Network_hosts"}
body:
  # Network_hosts

  A network **host** or **node** is a single computing device attached to a
  ![network](Network_fundamentals.md)

  Hosts can act as servers or clients, or both.

  ## Server

  A network server is a host that listens for inbound network connections and
  provides services to other hosts. Examples include web and email servers.

  ## Client

  A network client is a host that makes outbound connections and requests servcies
  from network servers.

raw_content:
  ---
  title: Network_hosts
  tags: [networks]
  created: Saturday, August 03, 2024
  ---

  # Network_hosts

  A network **host** or **node** is a single computing device attached to a
  ![network](Network_fundamentals.md)

  Hosts can act as servers or clients, or both.

  ## Server

  A network server is a host that listens for inbound network connections and
  provides services to other hosts. Examples include web and email servers.

  ## Client

  A network client is a host that makes outbound connections and requests servcies
  from network servers.


path: zk/Internet_fundamentals.md
sortable_path: zkInternet_fundamentals.md
title: Internet_fundamentals
lead: # Internet fundamentals
checksum: e56ba5787cb40c2282d9feeccf42526179d816312d9d860a9fa40bd498ee79bb
created: 2024-08-03 06:51:56.584079758+00:00
modified: 2024-09-06 08:41:18.638862417+00:00
metadata: {"created":"Saturday, August 03, 2024","tags":["networks"],"title":"Internet_fundamentals"}
body:
  # Internet fundamentals

  The internet is a _[network](Network_fundamentals.md) of networks_. A globally
  connected set of computer networks that all use a suite of common
  [protocols](Network_protocols.md).

  The protocols governing the communication between networks on the internet are
  specified in the **Internet Protocol Suite**.

  This combines two protocols:

  - **TCP**
    - The Transmission Control Protocol
  - **IP**
    - The Internet Protocol

  Hence the Internet Protocol Suite is also known as "TCP/IP".

  The TCP/IP is a network stack comprising four layers, each with its own
  protocols. From the top:

  ![TCP/IP stack diagram](../img/TCP_IP.png)

  - The [Link Layer](Link_Layer_of_Internet_Protocol.md) concerns communication
    between devices on the same local network. The transfer of information once it
    has been received from the internet or its transfer between devices before it
    sent out into the internet. Ethernet and WiFi are two core protocols utilised
    at this layer.

  - The [Internet Layer](Internet_Layer_of_Internet_Protocol.md) is concerned with
    communication accross networks. It is responsible for identifying hosts on the
    inter-network by their addresses and routing data from network to network.

  - The [Transport Layer](Transport_Layer_of_Internet_Protocol.md) creates a
    commmunication channel between two hosts using ports and sockets.

  - The [Application Layer](Application_Layer_of_Internet_Protocol.md) concerned
    with software that utilises information received and sent over the internet,
    for instance web pages and emails. This is the highest level of abstraction
    for which all the lower levels serve as "plumbing". Examples of protocols
    operative at the Application Layer include HTTP and SSH.

  See:
  [Example_scenario_internet_data_transfer](Example_scenario_internet_data_transfer.md)

raw_content:
  ---
  title: Internet_fundamentals
  tags: [networks]
  created: Saturday, August 03, 2024
  ---

  # Internet fundamentals

  The internet is a _[network](Network_fundamentals.md) of networks_. A globally
  connected set of computer networks that all use a suite of common
  [protocols](Network_protocols.md).

  The protocols governing the communication between networks on the internet are
  specified in the **Internet Protocol Suite**.

  This combines two protocols:

  - **TCP**
    - The Transmission Control Protocol
  - **IP**
    - The Internet Protocol

  Hence the Internet Protocol Suite is also known as "TCP/IP".

  The TCP/IP is a network stack comprising four layers, each with its own
  protocols. From the top:

  ![TCP/IP stack diagram](../img/TCP_IP.png)

  - The [Link Layer](Link_Layer_of_Internet_Protocol.md) concerns communication
    between devices on the same local network. The transfer of information once it
    has been received from the internet or its transfer between devices before it
    sent out into the internet. Ethernet and WiFi are two core protocols utilised
    at this layer.

  - The [Internet Layer](Internet_Layer_of_Internet_Protocol.md) is concerned with
    communication accross networks. It is responsible for identifying hosts on the
    inter-network by their addresses and routing data from network to network.

  - The [Transport Layer](Transport_Layer_of_Internet_Protocol.md) creates a
    commmunication channel between two hosts using ports and sockets.

  - The [Application Layer](Application_Layer_of_Internet_Protocol.md) concerned
    with software that utilises information received and sent over the internet,
    for instance web pages and emails. This is the highest level of abstraction
    for which all the lower levels serve as "plumbing". Examples of protocols
    operative at the Application Layer include HTTP and SSH.

  See:
  [Example_scenario_internet_data_transfer](Example_scenario_internet_data_transfer.md)


path: zk/Network_protocols.md
sortable_path: zkNetwork_protocols.md
title: Network_protocols
lead: # Network protocols
checksum: fc8829025eea399e8a82613312ed8944207f67db3917b206a497a0453563072e
created: 2024-08-03 06:53:42.143909607+00:00
modified: 2024-08-03 07:04:01.327149245+00:00
metadata: {"created":"Saturday, August 03, 2024","tags":["networks"],"title":"Network_protocols"}
body:
  # Network protocols

  Protocols are agreed standards specifying and governing communication between
  nodes on a [network](Network_fundamentals.md).

  Network protocols are typically _layered_. The implmentation of a network layer
  is a _stack_.

  Stacks enshrine encapsulation: not every layer has to be directly concerned with
  every other layer. Only adjacent layers that interface with each other need to
  be in contact.

  A prime example of a layered network stack is the
  [internet_protocol_suite](Internet_fundamentals.md).

raw_content:
  ---
  title: Network_protocols
  tags: [networks]
  created: Saturday, August 03, 2024
  ---

  # Network protocols

  Protocols are agreed standards specifying and governing communication between
  nodes on a [network](Network_fundamentals.md).

  Network protocols are typically _layered_. The implmentation of a network layer
  is a _stack_.

  Stacks enshrine encapsulation: not every layer has to be directly concerned with
  every other layer. Only adjacent layers that interface with each other need to
  be in contact.

  A prime example of a layered network stack is the
  [internet_protocol_suite](Internet_fundamentals.md).


path: zk/Link_Layer_of_Internet_Protocol.md
sortable_path: zkLink_Layer_of_Internet_Protocol.md
title: Link_Layer_of_the_Internet_Protocol
lead: # Link Layer of the Internet Protocol Suite
checksum: 0cac905d60cd065bca5be51a0e40906fe3e43aa3ca908db15cee30bcf66a9b4c
created: 2024-08-06 05:27:00.733151391+00:00
modified: 2024-09-03 06:01:54.251843373+00:00
body:
  # Link Layer of the Internet Protocol Suite

  > The Link Layer is the lowest level of the network stack that makes up the
  > Internet Protocol Suite. It is concerned with the physical and logical
  > connections between [hosts](./Network_hosts.md) on the same local network.

  The physical and logical connections are known as **Links**.

  Link Layer protocols goven communication between devices. Examples include:

  - **WiFi**
    - For wireless communication between devices using radio waves
  - **Ethernet**
    - Wired communication

  All devices connected to the internet participate in the Link Layer since it is
  the base level and the gateway out of the local network to other networks. The
  same is not true for other layers. For example a device like a laptop
  participates in all four layers up to the Application Layer but a network router
  does not:

  ![Link Layer diagram](../img/link-layer-frame-revised.png)

  ## MAC addresses

  Each device on a link has a network address that uniquely identifies it. For
  most Link Layer Protocols this is a
  [Media Access Control (MAC) address](MAC_addresses.md).

  A device receives data addressed to its MAC address.

  MAC addresses are limited to local networks. Remote networks cannot send data to
  a specific MAC address. This is off-limits from the outside.

  ## Frames

  Link Layer data is divided into small units called "frames". The anatomy of a
  frame is as follows:

  ![Link Layer frame](../img/link-layer-frame-revised.png)

  The header contains the source and destination MAC address plus a descriptor of
  the type of data it contains. The data is the payload. The footer is used to
  detect errors.

raw_content:
  ---
  title: Link_Layer_of_the_Internet_Protocol
  tags: [networks, internet]
  created: Tuesday, August 06, 2024
  ---

  # Link Layer of the Internet Protocol Suite

  > The Link Layer is the lowest level of the network stack that makes up the
  > Internet Protocol Suite. It is concerned with the physical and logical
  > connections between [hosts](./Network_hosts.md) on the same local network.

  The physical and logical connections are known as **Links**.

  Link Layer protocols goven communication between devices. Examples include:

  - **WiFi**
    - For wireless communication between devices using radio waves
  - **Ethernet**
    - Wired communication

  All devices connected to the internet participate in the Link Layer since it is
  the base level and the gateway out of the local network to other networks. The
  same is not true for other layers. For example a device like a laptop
  participates in all four layers up to the Application Layer but a network router
  does not:

  ![Link Layer diagram](../img/link-layer-frame-revised.png)

  ## MAC addresses

  Each device on a link has a network address that uniquely identifies it. For
  most Link Layer Protocols this is a
  [Media Access Control (MAC) address](MAC_addresses.md).

  A device receives data addressed to its MAC address.

  MAC addresses are limited to local networks. Remote networks cannot send data to
  a specific MAC address. This is off-limits from the outside.

  ## Frames

  Link Layer data is divided into small units called "frames". The anatomy of a
  frame is as follows:

  ![Link Layer frame](../img/link-layer-frame-revised.png)

  The header contains the source and destination MAC address plus a descriptor of
  the type of data it contains. The data is the payload. The footer is used to
  detect errors.

metadata:
  {"created":"Tuesday, August 06, 2024","tags":["networks","internet"],"title":"Link_Layer_of_the_Internet_Protocol"}


path: zk/MAC_addresses.md
sortable_path: zkMAC_addresses.md
title: MAC_addresses
lead: # MAC addresses
checksum: cd77d1461280e63d68a7ed6a187b677a97c019b4ecf034e6995a006ace16f4c0
created: 2024-08-06 05:48:26.021725255+00:00
modified: 2024-08-15 05:35:16.087617339+00:00
metadata: {"created":"Tuesday, August 06, 2024","tags":["networks"],"title":"MAC_addresses"}
body:
  # MAC addresses

  A Media Access Control (MAC) address is a physical address in contrast to, say,
  an IP address which is logical.

  Each device on a [local network](./Link_Layer_of_Internet_Protocol.md) has a MAC
  as a unique identifier. It is a hardware-based, physical attribute of the
  device, typically permanently encoded onto a non-volatile memory chip attached
  to the ![network_card](Network_card.md).

  MAC addresses consist of 6 bytes (48-bits) represented as 12
  [hexadecimal_digits](Hexadecimal_number_system.md).

raw_content:
  ---
  title: MAC_addresses
  tags: [networks]
  created: Tuesday, August 06, 2024
  ---

  # MAC addresses

  A Media Access Control (MAC) address is a physical address in contrast to, say,
  an IP address which is logical.

  Each device on a [local network](./Link_Layer_of_Internet_Protocol.md) has a MAC
  as a unique identifier. It is a hardware-based, physical attribute of the
  device, typically permanently encoded onto a non-volatile memory chip attached
  to the ![network_card](Network_card.md).

  MAC addresses consist of 6 bytes (48-bits) represented as 12
  [hexadecimal_digits](Hexadecimal_number_system.md).


path: zk/Network_card.md
sortable_path: zkNetwork_card.md
title: Network_card
lead: # Network_card
checksum: 1e74d7ac5ec4028eafedd6b9251874559f564b989bbf4c905de11b480d1735f1
created: 2024-08-06 06:08:04.807273649+00:00
modified: 2024-08-15 05:38:29.647719886+00:00
metadata: {"created":"Tuesday, August 06, 2024","tags":["networks","hardware"],"title":"Network_card"}
body:
  # Network_card

  The network card (technically know as the Network Interface Card (NIC)) or
  _network adapter_ is the hardware device that allows a computer to connect to a
  network.

  They are the physical interface between the computer and the network
  [link](./Link_Layer_of_Internet_Protocol.md), e.g. WiFi or ethernet.

  They can be internal and built into the [motherboard](./Motherboard.md) or
  external as in the case of USB adaptors.

  Each network card has a unique [MAC address](./MAC_addresses.md) to identify the
  device on the network.

  Network cards on consumer-grade devices typically utilise the devices resources
  and the kernel manages its actions via a driver. More high-end devices
  (typically used in servers) have their own processors and memory.

raw_content:
  ---
  title: Network_card
  tags: [networks, hardware]
  created: Tuesday, August 06, 2024
  ---

  # Network_card

  The network card (technically know as the Network Interface Card (NIC)) or
  _network adapter_ is the hardware device that allows a computer to connect to a
  network.

  They are the physical interface between the computer and the network
  [link](./Link_Layer_of_Internet_Protocol.md), e.g. WiFi or ethernet.

  They can be internal and built into the [motherboard](./Motherboard.md) or
  external as in the case of USB adaptors.

  Each network card has a unique [MAC address](./MAC_addresses.md) to identify the
  device on the network.

  Network cards on consumer-grade devices typically utilise the devices resources
  and the kernel manages its actions via a driver. More high-end devices
  (typically used in servers) have their own processors and memory.


path: zk/Packet_switching_British_precursors.md
sortable_path: zkPacket_switching_British_precursors.md
title: Packet_switching_British_precursors
lead: # Packet switching British precursors
body: # Packet switching British precursors
checksum: 4fff3bbf79a59c9c514746d222bb729d30dfc7830571edd9da006db8f2759450
created: 2024-08-09 14:03:22.14006866+00:00
modified: 2024-08-09 14:03:32.717754775+00:00
metadata: {"created":"Friday, August 09, 2024","tags":[],"title":"Packet_switching_British_precursors"}
raw_content:
  ---
  title: Packet_switching_British_precursors
  tags: []
  created: Friday, August 09, 2024
  ---

  # Packet switching British precursors


path: zk/Peer_to_peer_network.md
sortable_path: zkPeer_to_peer_network.md
title: Peer_to_peer_network
lead: # Peer to peer network
checksum: aa0c1ff8482172c52b9f72d646259717b3a0f9029ea3eacd337074adf012b4c9
created: 2024-08-09 14:05:44.108407682+00:00
modified: 2024-08-09 14:19:35.879508791+00:00
metadata: {"created":"Friday, August 09, 2024","tags":["networks"],"title":"Peer_to_peer_network"}
body:
  # Peer to peer network

  A decentralised network model where each participant (peer) acts as both a
  client and a server.

  Resources are shared directly between peers rather than being coordinated via a
  server.

  Under a server architecture, multiple requests are made from different hosts for
  resources. The server manages this load and is in control of what is being
  shared, to whom, and when. With P2P there is no central authority equivalent to
  this. Each peer both shares and consumes resources and in this sense is both a
  client and a server.

  A practical example of this is [torrenting](Torrenting.md) - an applicaton of
  P2P technology to file sharing.

  ## Benefits

  - Decentralisation, no central authority, also means no single point of failure.
  - Scalable: the network capacity grows with the number of users (contrast
    servers)
  - Efficiency: idle resources of peers are put to use
  - Improved performance for popular content

  ## Drawbacks

  - Security: potential for malicious peers and content
  - Inconsistent availability of resources

raw_content:
  ---
  title: Peer_to_peer_network
  tags: [networks]
  created: Friday, August 09, 2024
  ---

  # Peer to peer network

  A decentralised network model where each participant (peer) acts as both a
  client and a server.

  Resources are shared directly between peers rather than being coordinated via a
  server.

  Under a server architecture, multiple requests are made from different hosts for
  resources. The server manages this load and is in control of what is being
  shared, to whom, and when. With P2P there is no central authority equivalent to
  this. Each peer both shares and consumes resources and in this sense is both a
  client and a server.

  A practical example of this is [torrenting](Torrenting.md) - an applicaton of
  P2P technology to file sharing.

  ## Benefits

  - Decentralisation, no central authority, also means no single point of failure.
  - Scalable: the network capacity grows with the number of users (contrast
    servers)
  - Efficiency: idle resources of peers are put to use
  - Improved performance for popular content

  ## Drawbacks

  - Security: potential for malicious peers and content
  - Inconsistent availability of resources


path: zk/Internet_Layer_of_Internet_Protocol.md
sortable_path: zkInternet_Layer_of_Internet_Protocol.md
title: Internet_Layer_of_Internet_Protocol
lead: # Internet Layer of the Internet Protocol Suite
checksum: 9ed8da78ee2cf8ebe743142cab77a49dd0221638aef99020d5a84e39f2c12b5c
created: 2024-08-15 05:53:06.099345649+00:00
modified: 2024-09-03 06:02:08.928958028+00:00
body:
  # Internet Layer of the Internet Protocol Suite

  The Internet Layer is concerned with establishing the basis for the connection
  and communication between hosts accross the internet.

  In order for the data to get from a machine on one local network to a machine on
  another local network, each machine must have a way of identifying and
  interfacing with the other.

  The Internet Layer utilises the **Internet Protocol** to determine how devices
  will be publicly identified to devices on other networks and how different
  devices on the same local network will be distinguished from outside of this
  network. This is achieved through
  [Internet Protocol (IP) addresses](./IP_addresses.md).

  ## Packets

  Data is sent accross the internet between hosts as _packets_. These packets,
  when sent and received within the
  [Link Layer](Link_Layer_of_Internet_Protocol.md) on the local network, are
  enclosed within a Link Layer frame (in its payload section):

  ![Internet Layer packet](../img/internet-layer-packet-revised.png)

  Like a frame, an IP packet has a header and a payload. The payload comprises the
  data between sent between hosts. The header contains a source IP address and a
  destination IP address and also specifies which version of the Internet Protocol
  is being used.

  // Need to find out more about packets work.

raw_content:
  ---
  title: Internet_Layer_of_Internet_Protocol
  tags:
    - internet
    - networks
  created: Thursday, August 15, 2024
  ---

  # Internet Layer of the Internet Protocol Suite

  The Internet Layer is concerned with establishing the basis for the connection
  and communication between hosts accross the internet.

  In order for the data to get from a machine on one local network to a machine on
  another local network, each machine must have a way of identifying and
  interfacing with the other.

  The Internet Layer utilises the **Internet Protocol** to determine how devices
  will be publicly identified to devices on other networks and how different
  devices on the same local network will be distinguished from outside of this
  network. This is achieved through
  [Internet Protocol (IP) addresses](./IP_addresses.md).

  ## Packets

  Data is sent accross the internet between hosts as _packets_. These packets,
  when sent and received within the
  [Link Layer](Link_Layer_of_Internet_Protocol.md) on the local network, are
  enclosed within a Link Layer frame (in its payload section):

  ![Internet Layer packet](../img/internet-layer-packet-revised.png)

  Like a frame, an IP packet has a header and a payload. The payload comprises the
  data between sent between hosts. The header contains a source IP address and a
  destination IP address and also specifies which version of the Internet Protocol
  is being used.

  // Need to find out more about packets work.

metadata:
  {"created":"Thursday, August 15, 2024","tags":["internet","networks"],"title":"Internet_Layer_of_Internet_Protocol"}


path: zk/IP_addresses.md
sortable_path: zkIP_addresses.md
title: IP_addresses
lead: # IP addresses
checksum: 2382804d6f979af470151bfe1e8bc7caa40b1bdedb75a142335011f0efc0e805
created: 2024-08-16 14:21:22.886184519+00:00
modified: 2024-09-01 12:41:55.943170057+00:00
metadata: {"created":"Friday, August 16, 2024","tags":["networks","internet"],"title":"IP_addresses"}
body:
  # IP addresses

  ## IPv4 vs IPv6

  IP addresses come in two formats and bit-lengths depending on which version of
  the [Internet Protocol](Internet_Layer_of_Internet_Protocol.md) is being used.

  - IPv4 addresses are 32-bits in length
  - IPv6 addresses are 128-bits in length

  The larger the bit-length, the more unique addresses that can be generated. Thus
  IPv6 can create more addresses that IPv4.

  This entry focuses on IPv4 addresses.

  ## Anatomy of an IPv4 address

  Each of the 32 bits are divided into four groups which are each 8 bits in
  length. Each group therefore comprises an octet and is separated by dots.

  The decimal number system, rather than binary is used to represent the values.
  This means each binary octet is three decimal digits in length.

  The 32 bits comprise two major groupings

  - the **network prefix**
  - the **host**

  ![Different sections of IP address](../img/ip-address-anatomy.png)

  Devices connected to the same network share the same network prefix. They are
  said to be on the same **subnet**. However each device will have a unique value
  for the host octet.

  (This account is somewhat idealised. The length of the prefix does not always
  align with the octet boundary. For instance, the prefix could be 25 bits in
  length, rather than 24. In this case it would "steal" one bit from the host
  section making it 23 bits in length.)

  Any changes to the length of the network prefix change the number of hosts
  available on the subnet. If the network prefix is longer, the number of unique
  hosts is reduced. If the network prefix is shorted, the number of unique hosts
  is increased.

  ## Identifying the network and host groupings

  Given that the network prefix and host identifier do not always stick to set
  boundaries, we require a way of distinguishing the two values. There are two
  dominant methods.

  ### CIDR notation

  This stands for _Classless Inter-Domain Routing_. An example:

  > 192.168.1.23/24

  The value after the forward slash is the number of bits that specify the network
  address. In the example, it is the standard 24-bits. Thus 192.168.1.23 stands
  for the network.

  ### Subnet masking

  The subnet mask is another number _in addition to_ the network address. It is a
  kind of superimposed map on top of the address.

  Here is an example of a subnet mask:

  > 11111111.11111111.11111111.00000000

  The denary form would be as follows (as 255 is the maximal decimal number that
  can be represented with a single 8-bit number and 0 is the smallest):

  > 255.255.255.0

  In the binary form, the 1 values represent the bits which designate the network
  address and the 0 values represent the bits that designate the host.

  In the example above this corresponds to the idealised 32-bit:8-bit ratio of
  CIDR /24.

  There is a clever consequence of the subnet mask: if you apply a bitwise AND
  operator against the IP address and mask (both in their binary form) you can
  determine whether two addresses are on the same network.

  To compare the IP address 192.168.1.23 against 192.168.1.100 to demonstrate:

  ```
  192.168.1.23:
  IP:     11000000.10101000.00000001.00010111
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000001.00000000
  ___________________________________

  192.168.1.100:
  IP:     11000000.10101000.00000001.00010111
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000001.00000000
  ```

  After applying the bitwise AND logic we see that the result is identical for
  both IPs indicating they are on the same network (share the same network
  prefix), whilst the host value is "masked".

  Here is a scenario where the subnet mask indicates that two IPs are not on the
  shared network:

  ```
  192.168.1.23:
  IP:     11000000.10101000.00000001.00010111
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000001.00000000
  ___________________________________

  192.168.2.1
  IP:     11000000.10101000.00000010.00000001
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000010.00000000

  ```

  255.255.255.0 is not the only possible subnet mask. There are masks
  corresponding to ratios other than CIDR /24. For example:

  - 255.0.0.0 (/8)
  - 255.255.0.0 (/8)
  - 255.255.255.192 (/26)
  - 255.255.255.240 (/28)

  As noted previously, the different ratios will obviously affect the number of
  unique hosts available on the subnet.

  ## Determining the range of hosts: practical example

  Let's say we have the following IP address expressed in CIDR: 192.168.0.133/27.
  In binary this would be:

  ```
  110000000.10101000.00000000.100--00101
  ```

  The `--` indicates the demarcation point between the network prefix and the bits
  designated for the host. 32 - 27 leaves us 5 bits for our range of hosts. This
  gives us 2^5 = 32 unique host values.

  In fact it will actually be 30 values since the first value (`00000`) will be
  used to identify the network itself and the last value (`11111`) will be the
  broadcast address - the address used when a message needs to be sent to all
  hosts on the network.

  An actual host address in this range is included in the example: `00101`.

raw_content:
  ---
  title: IP_addresses
  tags: [networks, internet]
  created: Friday, August 16, 2024
  ---

  # IP addresses

  ## IPv4 vs IPv6

  IP addresses come in two formats and bit-lengths depending on which version of
  the [Internet Protocol](Internet_Layer_of_Internet_Protocol.md) is being used.

  - IPv4 addresses are 32-bits in length
  - IPv6 addresses are 128-bits in length

  The larger the bit-length, the more unique addresses that can be generated. Thus
  IPv6 can create more addresses that IPv4.

  This entry focuses on IPv4 addresses.

  ## Anatomy of an IPv4 address

  Each of the 32 bits are divided into four groups which are each 8 bits in
  length. Each group therefore comprises an octet and is separated by dots.

  The decimal number system, rather than binary is used to represent the values.
  This means each binary octet is three decimal digits in length.

  The 32 bits comprise two major groupings

  - the **network prefix**
  - the **host**

  ![Different sections of IP address](../img/ip-address-anatomy.png)

  Devices connected to the same network share the same network prefix. They are
  said to be on the same **subnet**. However each device will have a unique value
  for the host octet.

  (This account is somewhat idealised. The length of the prefix does not always
  align with the octet boundary. For instance, the prefix could be 25 bits in
  length, rather than 24. In this case it would "steal" one bit from the host
  section making it 23 bits in length.)

  Any changes to the length of the network prefix change the number of hosts
  available on the subnet. If the network prefix is longer, the number of unique
  hosts is reduced. If the network prefix is shorted, the number of unique hosts
  is increased.

  ## Identifying the network and host groupings

  Given that the network prefix and host identifier do not always stick to set
  boundaries, we require a way of distinguishing the two values. There are two
  dominant methods.

  ### CIDR notation

  This stands for _Classless Inter-Domain Routing_. An example:

  > 192.168.1.23/24

  The value after the forward slash is the number of bits that specify the network
  address. In the example, it is the standard 24-bits. Thus 192.168.1.23 stands
  for the network.

  ### Subnet masking

  The subnet mask is another number _in addition to_ the network address. It is a
  kind of superimposed map on top of the address.

  Here is an example of a subnet mask:

  > 11111111.11111111.11111111.00000000

  The denary form would be as follows (as 255 is the maximal decimal number that
  can be represented with a single 8-bit number and 0 is the smallest):

  > 255.255.255.0

  In the binary form, the 1 values represent the bits which designate the network
  address and the 0 values represent the bits that designate the host.

  In the example above this corresponds to the idealised 32-bit:8-bit ratio of
  CIDR /24.

  There is a clever consequence of the subnet mask: if you apply a bitwise AND
  operator against the IP address and mask (both in their binary form) you can
  determine whether two addresses are on the same network.

  To compare the IP address 192.168.1.23 against 192.168.1.100 to demonstrate:

  ```
  192.168.1.23:
  IP:     11000000.10101000.00000001.00010111
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000001.00000000
  ___________________________________

  192.168.1.100:
  IP:     11000000.10101000.00000001.00010111
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000001.00000000
  ```

  After applying the bitwise AND logic we see that the result is identical for
  both IPs indicating they are on the same network (share the same network
  prefix), whilst the host value is "masked".

  Here is a scenario where the subnet mask indicates that two IPs are not on the
  shared network:

  ```
  192.168.1.23:
  IP:     11000000.10101000.00000001.00010111
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000001.00000000
  ___________________________________

  192.168.2.1
  IP:     11000000.10101000.00000010.00000001
  Mask:   11111111.11111111.11111111.00000000
  Result: 11000000.10101000.00000010.00000000

  ```

  255.255.255.0 is not the only possible subnet mask. There are masks
  corresponding to ratios other than CIDR /24. For example:

  - 255.0.0.0 (/8)
  - 255.255.0.0 (/8)
  - 255.255.255.192 (/26)
  - 255.255.255.240 (/28)

  As noted previously, the different ratios will obviously affect the number of
  unique hosts available on the subnet.

  ## Determining the range of hosts: practical example

  Let's say we have the following IP address expressed in CIDR: 192.168.0.133/27.
  In binary this would be:

  ```
  110000000.10101000.00000000.100--00101
  ```

  The `--` indicates the demarcation point between the network prefix and the bits
  designated for the host. 32 - 27 leaves us 5 bits for our range of hosts. This
  gives us 2^5 = 32 unique host values.

  In fact it will actually be 30 values since the first value (`00000`) will be
  used to identify the network itself and the last value (`11111`) will be the
  broadcast address - the address used when a message needs to be sent to all
  hosts on the network.

  An actual host address in this range is included in the example: `00101`.


path: zk/The_History_of_Computing_Swade.md
sortable_path: zkThe_History_of_Computing_Swade.md
title: The_History_of_Computing_Swade
checksum: 555622a2743056f86e528a25f04133abbad9d62a067bcb2926291a5a4a9826e8
created: 2024-08-23 07:46:48.602264538+00:00
modified: 2024-09-13 05:59:53.649571481+00:00
lead:
  | Title                                               | Author      | Publication date | Resource type |
  | --------------------------------------------------- | ----------- | ---------------- | ------------- |
  | The History of Computing: A Very Short Introduction | Doron Swade | 2022             | Book          |

body:
  | Title                                               | Author      | Publication date | Resource type |
  | --------------------------------------------------- | ----------- | ---------------- | ------------- |
  | The History of Computing: A Very Short Introduction | Doron Swade | 2022             | Book          |

  ## Timeline

  A typical timeline approach rooted in major innovations.

  - Ancient aids to counting: knotted cords and notched sticks
  - Ancient aids to calculation: counting boards and abacii
  - Early mechanical calculator devices in the 17th century (number wheels,
    Pascal, Leibniz)
  - Modern aids to calculation: slide rules following the discovery of
    [logarithms](Logarithms.md)
  - Mechanised, automated calculating engines of Babbage in the 19th century
  - Punched-card machines leading to IBM in the early 20th century
  - Analogue and electro-mechanical computers of the early 20th century inclusive
    of wartime computers
  - Early valve-based (vacuum-tubed) digital computers (again wartime and early
    Cold War)
  - The invention of the transistor and first fully-digital computers
  - The invention of [integrated_circuits](Integrated_circuits.md)
  - Supercomputers
  - Minicomputers
  - Consumer personal computers
  - Internet and later, Web
  - Smart phones

  First three phases of digital electronic computers:

  - Wartime up to 1950s vacuum-tube era
  - Transistor era up to 1963
  - First microchip era ending in early 1970s

  ## Mechanical calculating devices in the 17th century

  Focus was chiefly on creating a desktop calculator capable of four-function
  arithmetic.

  ![Photograph of Pascaline](../img/pascaline.jpg)

  ![Photograph of replica of Leibniz stepped drum machine](../img/leibniz-stepped-drum.jpg)

  The main contenders were the Pascaline of #Pascal (which only did cumulative
  addition) and the wheel or "stepped drum" calculator of #Leibniz that could do
  all operations (in theory).

  Subsequent designs were based on these artefacts. In practice, neither worked
  consistently well with the carriage of tens remaining a sticking point.

  ![Photograph of Arithmometer](../img/arithmometer.jpg)

  ![Photograph of Comptometer](../img/comptometer.png)

  The arithmometer (crank driven) and comptometer (key-driven) were descendents of
  the Leibniz design that became commercially viable by the 19th century along
  with other mechanical calculators. In the US, Burroughs dominated the market.

  ## Babbage: mechanized, automated calculation

  > I wish to God these calculations had been executed by Steam (#Babbage)

  With Babbage's machines we see an approach to computation that can only be
  understood against the backdrop of the Industrial Revolution in which they were
  conceived.

  The idea is that the machine is a factory and number is the product. In the same
  way as the mechanised looms created textiles. It is the extension of a model of
  industrial production from goods/commodities to information.

  Babbage conceived two machines: the Difference Engine (DE) and the Analytical
  Engine (AE). Neither were successfully built in his lifetime. The DE preceded
  the AE and was basically an advanced mechanical calculator whereas the AE
  approximated a general purpose computer.

  Despite this, with the Difference Engine, in contrast to preceding _aids to
  calculation_, the steps of the computational algorithm were no longer directed
  by human intelligence but by internal rules embodied in the mechanism and
  automatically generated.

  ### Difference Engine

  The DE's single purpose was to calculate and output mathematical tables such as
  the results of polynomial equations. The idea was that you would input the
  variables of the equation and activate the machine and it would output the
  results. Associated with this concept was the idea that once it arrived at the
  answer a bell would ring and the machine would _halt_. This influenced #Turing
  later. It was non-programmable and designed for a specific set of calculations.

  ### Analytical Engine

  Conceived as a general-purpose computing machine capable of perfoming a wide
  range of calculations, programmable using punched cards similar to those used
  with Jacquard looms.

  It more resembled modern computers in that Babbage used concepts that would
  later translate into the #vonNeumann architecture. There was a "mill" (CPU),
  "store" (memory) and input/output mechanisms. It also had a concept of looping
  and conditional branching.

  ### Lovelace's insight

  A central idea of Ada #Lovelace, expressed in her notes on the Analytical Engine
  is that **number can represent entities other than quantity**.

  If we assign meaning to number, results arrived at by operating on number
  according to rules can say things about the world when mapped back onto the
  world using the meanings assigned to them.

  Lovelace's insight was that the potential of computing lay in the power of
  machines to manipulate representations of the world contained in symbols.

  ## Analogue computers

  Both digital and analogue computers are automatic. They differ in _how they
  represent quantities_ and how their outputs are derived.

  With digital machines, quantity is represented as a string of discrete digits.

  With analogue machines, quantity is a physical property _in itself_ rather than
  a representation. This could be, for example: the lowering of a weight, the flow
  of a liquid, or an electrical charge. This physical behaviour is **analagous**
  to the system that is being modelled. Quantities are continuously variable
  values rather than discrete (discontinuous values).

  Digital machines produce results by _calculation_ whereas analogue machines
  produce results by _measurement_, e.g. the height of liquid in a tank or the
  time it taks for a tank to be emptied.

  ### Examples of analogue computers

  #### The Phillips Hydraulic Computer

  This used fluid to model the workings of the British economy. It consisted of a
  series of transparent plastic tanks and pipes which were fastened to a wooden
  board.

  ![Philips Hydraulic Computer](../img/hydraulic_computer.jpg)

  Each tank represented some aspect of the UK national economy and the flow of
  money around the economy was illustrated by coloured water. At the top of the
  board was a large tank called the treasury. Water (representing money) flowed
  from the treasury to other tanks representing the various ways in which a
  country could spend its money. For example, there were tanks for health and
  education. To increase spending on health care a tap could be opened to drain
  water from the treasury to the tank which represented health spending.

  #### Bush's Differential Analyser

  This was designed to solve differential equations by integration. In contrast to
  the Philips Computer it was general enough to be used to solve problems from
  different contexts. Examples of these contexts: heat flow, ballistics,
  mechanics, population growth, chemical interactions, astronomy.

  <img src="../img/differential_analyser_bush.jpg" />

  ![Differential Analyser](../img/differential_analyser_bush.jpg)

  It was about the size of a room and used shafts, motors, discs and wheels to
  work.

  ### Historiography

  There is a tendency in the history of computing to downplay or diminish the
  contribution of analogue computing devices and to present them as just an
  inferior precursor to the inevitable dominance of digital electronic computers.

  This is ahistorical and inaccurate.

  Analogue (and electromechanical devices) overlapped with and coexisted with
  digital devices for 40 years, spanning the first three generations of digital
  electronic devices. The term "analogue" itself only came about when the need
  arose to distinguish digital devices from other types of computer. They were not
  "rivals" before this.

  ### Electro-mechanical computers

  Electro-mechanical devices (also known as "electronic analogue computers") are a
  sort of midway between full digital devices and analogue computers, forming a
  bridge between the two eras.

  Their heyday was roughly 1935 - 1945.

  Their key components were:

  - [relays](Relays.md) for logic operations
  - rotating shafts and gears for performing calculations
  - punched cards or paper tape for input instructions and outputs

  They were slower than fully digital computers as they were limited by the speed

  - [ ] of moving components rather than the flow of electric charge. In addition
        the various mechanical parts were prone to wear and needed frequent
        replacement.

  ### Examples of electro-mechanical computers

  #### Harvard Mark 1 (1937-1944) - Aikin, IBM

  Designed by Howard Aiken and built by IBM (1937-1944). Also known as the
  _Automatic Sequence Controlled Calculator_ (ASCC). A general-purpose
  electro-mechanical computer it was most famously used at Los Alamos by
  #vonNeumann to calculate the blast yield of the atomic bomb.

  ![Harvard Mark One](../img/Harvard_Mark_1.jpg)

  It was more than 15m in length and weighed 5 tonnes comprising over 750,000
  parts. It used paper tape and punched cards for input/output.

  #### Complex Number Calculator (1940) - Bell Labs, Stibitz

  Not general purpose nor was it programmable. It was hardwired to perform a
  specific set of operations on numbers and nothing else. It used relays like the
  others. Its distinguishing feature was that it used a teletype for input rather
  than cards or paper tape.

  ![Calculating panel of the CMC](../img/complex-number-calculator-bell-labs.jpg)

  ![Teletype input of CMC](../img/stibitz-teletype.jpg)

  It comprised a panel (the calculating unit) and teletype (the input). One could
  remotely access the computer from the teletype in another location, providing it
  was connected to Bell Lab's telephone network.

raw_content:
  ---
  title: The_History_of_Computing_Swade
  tags: [literature, computer-history]
  created: Friday, August 23, 2024
  ---

  | Title                                               | Author      | Publication date | Resource type |
  | --------------------------------------------------- | ----------- | ---------------- | ------------- |
  | The History of Computing: A Very Short Introduction | Doron Swade | 2022             | Book          |

  ## Timeline

  A typical timeline approach rooted in major innovations.

  - Ancient aids to counting: knotted cords and notched sticks
  - Ancient aids to calculation: counting boards and abacii
  - Early mechanical calculator devices in the 17th century (number wheels,
    Pascal, Leibniz)
  - Modern aids to calculation: slide rules following the discovery of
    [logarithms](Logarithms.md)
  - Mechanised, automated calculating engines of Babbage in the 19th century
  - Punched-card machines leading to IBM in the early 20th century
  - Analogue and electro-mechanical computers of the early 20th century inclusive
    of wartime computers
  - Early valve-based (vacuum-tubed) digital computers (again wartime and early
    Cold War)
  - The invention of the transistor and first fully-digital computers
  - The invention of [integrated_circuits](Integrated_circuits.md)
  - Supercomputers
  - Minicomputers
  - Consumer personal computers
  - Internet and later, Web
  - Smart phones

  First three phases of digital electronic computers:

  - Wartime up to 1950s vacuum-tube era
  - Transistor era up to 1963
  - First microchip era ending in early 1970s

  ## Mechanical calculating devices in the 17th century

  Focus was chiefly on creating a desktop calculator capable of four-function
  arithmetic.

  ![Photograph of Pascaline](../img/pascaline.jpg)

  ![Photograph of replica of Leibniz stepped drum machine](../img/leibniz-stepped-drum.jpg)

  The main contenders were the Pascaline of #Pascal (which only did cumulative
  addition) and the wheel or "stepped drum" calculator of #Leibniz that could do
  all operations (in theory).

  Subsequent designs were based on these artefacts. In practice, neither worked
  consistently well with the carriage of tens remaining a sticking point.

  ![Photograph of Arithmometer](../img/arithmometer.jpg)

  ![Photograph of Comptometer](../img/comptometer.png)

  The arithmometer (crank driven) and comptometer (key-driven) were descendents of
  the Leibniz design that became commercially viable by the 19th century along
  with other mechanical calculators. In the US, Burroughs dominated the market.

  ## Babbage: mechanized, automated calculation

  > I wish to God these calculations had been executed by Steam (#Babbage)

  With Babbage's machines we see an approach to computation that can only be
  understood against the backdrop of the Industrial Revolution in which they were
  conceived.

  The idea is that the machine is a factory and number is the product. In the same
  way as the mechanised looms created textiles. It is the extension of a model of
  industrial production from goods/commodities to information.

  Babbage conceived two machines: the Difference Engine (DE) and the Analytical
  Engine (AE). Neither were successfully built in his lifetime. The DE preceded
  the AE and was basically an advanced mechanical calculator whereas the AE
  approximated a general purpose computer.

  Despite this, with the Difference Engine, in contrast to preceding _aids to
  calculation_, the steps of the computational algorithm were no longer directed
  by human intelligence but by internal rules embodied in the mechanism and
  automatically generated.

  ### Difference Engine

  The DE's single purpose was to calculate and output mathematical tables such as
  the results of polynomial equations. The idea was that you would input the
  variables of the equation and activate the machine and it would output the
  results. Associated with this concept was the idea that once it arrived at the
  answer a bell would ring and the machine would _halt_. This influenced #Turing
  later. It was non-programmable and designed for a specific set of calculations.

  ### Analytical Engine

  Conceived as a general-purpose computing machine capable of perfoming a wide
  range of calculations, programmable using punched cards similar to those used
  with Jacquard looms.

  It more resembled modern computers in that Babbage used concepts that would
  later translate into the #vonNeumann architecture. There was a "mill" (CPU),
  "store" (memory) and input/output mechanisms. It also had a concept of looping
  and conditional branching.

  ### Lovelace's insight

  A central idea of Ada #Lovelace, expressed in her notes on the Analytical Engine
  is that **number can represent entities other than quantity**.

  If we assign meaning to number, results arrived at by operating on number
  according to rules can say things about the world when mapped back onto the
  world using the meanings assigned to them.

  Lovelace's insight was that the potential of computing lay in the power of
  machines to manipulate representations of the world contained in symbols.

  ## Analogue computers

  Both digital and analogue computers are automatic. They differ in _how they
  represent quantities_ and how their outputs are derived.

  With digital machines, quantity is represented as a string of discrete digits.

  With analogue machines, quantity is a physical property _in itself_ rather than
  a representation. This could be, for example: the lowering of a weight, the flow
  of a liquid, or an electrical charge. This physical behaviour is **analagous**
  to the system that is being modelled. Quantities are continuously variable
  values rather than discrete (discontinuous values).

  Digital machines produce results by _calculation_ whereas analogue machines
  produce results by _measurement_, e.g. the height of liquid in a tank or the
  time it taks for a tank to be emptied.

  ### Examples of analogue computers

  #### The Phillips Hydraulic Computer

  This used fluid to model the workings of the British economy. It consisted of a
  series of transparent plastic tanks and pipes which were fastened to a wooden
  board.

  ![Philips Hydraulic Computer](../img/hydraulic_computer.jpg)

  Each tank represented some aspect of the UK national economy and the flow of
  money around the economy was illustrated by coloured water. At the top of the
  board was a large tank called the treasury. Water (representing money) flowed
  from the treasury to other tanks representing the various ways in which a
  country could spend its money. For example, there were tanks for health and
  education. To increase spending on health care a tap could be opened to drain
  water from the treasury to the tank which represented health spending.

  #### Bush's Differential Analyser

  This was designed to solve differential equations by integration. In contrast to
  the Philips Computer it was general enough to be used to solve problems from
  different contexts. Examples of these contexts: heat flow, ballistics,
  mechanics, population growth, chemical interactions, astronomy.

  <img src="../img/differential_analyser_bush.jpg" />

  ![Differential Analyser](../img/differential_analyser_bush.jpg)

  It was about the size of a room and used shafts, motors, discs and wheels to
  work.

  ### Historiography

  There is a tendency in the history of computing to downplay or diminish the
  contribution of analogue computing devices and to present them as just an
  inferior precursor to the inevitable dominance of digital electronic computers.

  This is ahistorical and inaccurate.

  Analogue (and electromechanical devices) overlapped with and coexisted with
  digital devices for 40 years, spanning the first three generations of digital
  electronic devices. The term "analogue" itself only came about when the need
  arose to distinguish digital devices from other types of computer. They were not
  "rivals" before this.

  ### Electro-mechanical computers

  Electro-mechanical devices (also known as "electronic analogue computers") are a
  sort of midway between full digital devices and analogue computers, forming a
  bridge between the two eras.

  Their heyday was roughly 1935 - 1945.

  Their key components were:

  - [relays](Relays.md) for logic operations
  - rotating shafts and gears for performing calculations
  - punched cards or paper tape for input instructions and outputs

  They were slower than fully digital computers as they were limited by the speed

  - [ ] of moving components rather than the flow of electric charge. In addition
        the various mechanical parts were prone to wear and needed frequent
        replacement.

  ### Examples of electro-mechanical computers

  #### Harvard Mark 1 (1937-1944) - Aikin, IBM

  Designed by Howard Aiken and built by IBM (1937-1944). Also known as the
  _Automatic Sequence Controlled Calculator_ (ASCC). A general-purpose
  electro-mechanical computer it was most famously used at Los Alamos by
  #vonNeumann to calculate the blast yield of the atomic bomb.

  ![Harvard Mark One](../img/Harvard_Mark_1.jpg)

  It was more than 15m in length and weighed 5 tonnes comprising over 750,000
  parts. It used paper tape and punched cards for input/output.

  #### Complex Number Calculator (1940) - Bell Labs, Stibitz

  Not general purpose nor was it programmable. It was hardwired to perform a
  specific set of operations on numbers and nothing else. It used relays like the
  others. Its distinguishing feature was that it used a teletype for input rather
  than cards or paper tape.

  ![Calculating panel of the CMC](../img/complex-number-calculator-bell-labs.jpg)

  ![Teletype input of CMC](../img/stibitz-teletype.jpg)

  It comprised a panel (the calculating unit) and teletype (the input). One could
  remotely access the computer from the teletype in another location, providing it
  was connected to Bell Lab's telephone network.

metadata:
  {"created":"Friday, August 23, 2024","tags":["literature","computer-history"],"title":"The_History_of_Computing_Swade"}


path: zk/Python_application_structure.md
sortable_path: zkPython_application_structure.md
title: Python_application_structure
lead: # Python application structure
checksum: fd40b46bcc9e8e55dd171f37d54d820668580a3c25d70821772b53a36d4c3f7e
created: 2024-08-27 14:33:04.042157825+00:00
modified: 2024-08-27 15:25:02.340435981+00:00
metadata: {"created":"Tuesday, August 27, 2024","tags":["python"],"title":"Python_application_structure"}
body:
  # Python application structure

  ## Directory layout

  ```
  src/
      cli.py OR app.py
      constants.py
      __init__.py
  tests/
  requirements.txt
  setup.py
  ```

  ## `requirements.txt`

  Dependencies used within the venv for the project, e.g:

  ```
  google-auth==1.35.0
  google-auth-httplib2==0.1.0
  google-auth-oauthlib==0.4.6
  google-api-python-client==2.26.1
  gspread==3.7.0
  requests==2.25.1
  oauth2client==4.1.3
  ```

  ## `setup.py`

  The declaration file for the application, equivalent to `package.json`, e.g:

  ```py
  from setuptools import setup, find_packages

  setup(
      name="eolas-db",
      version="0.1",
      packages=find_packages(where="src"),
      package_dir={"": "src"},
      install_requires=[
          # List your project dependencies here
      ],
      entry_points={
          "console_scripts": [
              "eolas-db=cli:main",
          ],
      },
  )
  ```

  The `entry_points` key is important. This is the first execution point of the
  application. Typically you would name your entry-point script as `app.py` or
  `cli.py` and then specify its `main` function.

  E.g. in `cli.py`:

  ```py
  def main():
      print('run the application')

  if __name__ == "__main__":
      main()

  ```

  ## `__init__.py`

  See
  [[Python_modules_and_imports#distinguishing-_modules_-from-_packages_]]|Python
  modules]]

raw_content:
  ---
  title: Python_application_structure
  tags: [python]
  created: Tuesday, August 27, 2024
  ---

  # Python application structure

  ## Directory layout

  ```
  src/
      cli.py OR app.py
      constants.py
      __init__.py
  tests/
  requirements.txt
  setup.py
  ```

  ## `requirements.txt`

  Dependencies used within the venv for the project, e.g:

  ```
  google-auth==1.35.0
  google-auth-httplib2==0.1.0
  google-auth-oauthlib==0.4.6
  google-api-python-client==2.26.1
  gspread==3.7.0
  requests==2.25.1
  oauth2client==4.1.3
  ```

  ## `setup.py`

  The declaration file for the application, equivalent to `package.json`, e.g:

  ```py
  from setuptools import setup, find_packages

  setup(
      name="eolas-db",
      version="0.1",
      packages=find_packages(where="src"),
      package_dir={"": "src"},
      install_requires=[
          # List your project dependencies here
      ],
      entry_points={
          "console_scripts": [
              "eolas-db=cli:main",
          ],
      },
  )
  ```

  The `entry_points` key is important. This is the first execution point of the
  application. Typically you would name your entry-point script as `app.py` or
  `cli.py` and then specify its `main` function.

  E.g. in `cli.py`:

  ```py
  def main():
      print('run the application')

  if __name__ == "__main__":
      main()

  ```

  ## `__init__.py`

  See
  [[Python_modules_and_imports#distinguishing-_modules_-from-_packages_]]|Python
  modules]]


path: zk/Transport_Layer_of_Internet_Protocol.md
sortable_path: zkTransport_Layer_of_Internet_Protocol.md
title: Transport_Layer_of_Internet_Protocol
lead: # Transport Layer of the Internet Protocol
checksum: 4f361b6f33035d4461f9e7e76d66951d1ab0351790ff79a6fb3505278900d3fa
created: 2024-09-03 05:37:32.871908625+00:00
modified: 2024-09-06 06:12:45.826227361+00:00
body:
  # Transport Layer of the Internet Protocol

  Transport Layer protocols provide a communications channel for data to be sent
  between hosts. The Transport Layer is dependent upon the
  [Internet Layer](Internet_Layer_of_Internet_Protocol.md) for addressing and
  routing the data and the
  [Link_Layer_of_Internet_Protocol](Link_Layer_of_Internet_Protocol.md) for
  transferring the data to a specific host once it has reached the local network.

  The protocol governing the Transport Layer is the **Transmission Control
  Protocol** (TCP).

  A TCP **fragment** fits within an IP packet's data section, giving us the
  following iteration of the network stack:

  ![Transport layer fragment](../img/transport-layer-segment.png)

  The segment header contains a destination network port number. The port number
  identifies the specific service or process on the host device which will receive
  the data.

  A useful analogy:

  _The IP address is like the street address of an office building. The port
  number is like the office number of a specific worker within the building. Once
  a letter arrives at the building, the mail processing office (like the OS) is
  not the intended recipient. Instead the mail worker passes on the letter to the
  intended recipient. Likewise, the OS examines the port number and delivers the
  inbound data to the process listening on the specified port._

  ## Ports

  A host with a single IP address can have multiple active ports, each used for
  performing a different type of activity on the network.

  Both servers and clients possess ports. A server will have a dedicated port
  which it listens for requests on (typically port 80 or port 443). A client will
  have an "ephemeral" port where it listens for responses from a server. They are
  ephemeral because they are open only temporarily to receive a response to a
  certain request and are assigned dynamically by the networking components of the
  OS.

  An IP address plus a port (e.g: 192.168.2.1:3000) constitutes an **endpoint**.
  An instance of an endpoint is a **socket**.

  If multiple clients connect to the same endpoint each has its own socket (for
  instance on a host that is a server). A socket can listen for new connections or
  it can represent an established, ongoing connection.

raw_content:
  ---
  title: Transport_Layer_of_Internet_Protocol
  tags: [internet, networks, ports]
  created: Tuesday, September 03, 2024
  ---

  # Transport Layer of the Internet Protocol

  Transport Layer protocols provide a communications channel for data to be sent
  between hosts. The Transport Layer is dependent upon the
  [Internet Layer](Internet_Layer_of_Internet_Protocol.md) for addressing and
  routing the data and the
  [Link_Layer_of_Internet_Protocol](Link_Layer_of_Internet_Protocol.md) for
  transferring the data to a specific host once it has reached the local network.

  The protocol governing the Transport Layer is the **Transmission Control
  Protocol** (TCP).

  A TCP **fragment** fits within an IP packet's data section, giving us the
  following iteration of the network stack:

  ![Transport layer fragment](../img/transport-layer-segment.png)

  The segment header contains a destination network port number. The port number
  identifies the specific service or process on the host device which will receive
  the data.

  A useful analogy:

  _The IP address is like the street address of an office building. The port
  number is like the office number of a specific worker within the building. Once
  a letter arrives at the building, the mail processing office (like the OS) is
  not the intended recipient. Instead the mail worker passes on the letter to the
  intended recipient. Likewise, the OS examines the port number and delivers the
  inbound data to the process listening on the specified port._

  ## Ports

  A host with a single IP address can have multiple active ports, each used for
  performing a different type of activity on the network.

  Both servers and clients possess ports. A server will have a dedicated port
  which it listens for requests on (typically port 80 or port 443). A client will
  have an "ephemeral" port where it listens for responses from a server. They are
  ephemeral because they are open only temporarily to receive a response to a
  certain request and are assigned dynamically by the networking components of the
  OS.

  An IP address plus a port (e.g: 192.168.2.1:3000) constitutes an **endpoint**.
  An instance of an endpoint is a **socket**.

  If multiple clients connect to the same endpoint each has its own socket (for
  instance on a host that is a server). A socket can listen for new connections or
  it can represent an established, ongoing connection.

metadata:
  {"created":"Tuesday, September 03, 2024","tags":["internet","networks","ports"],"title":"Transport_Layer_of_Internet_Protocol"}


path: zk/Application_Layer_of_Internet_Protocol.md
sortable_path: zkApplication_Layer_of_Internet_Protocol.md
title: Application_Layer_of_Internet_Protocol
lead: # Application Layer of the Internet Protocol
checksum: 522e2a9897de66268cf6fababf8422d4e07a5599efdeee7101accdc507c13b4b
created: 2024-09-06 06:11:59.753192243+00:00
modified: 2024-09-06 08:21:30.101780284+00:00
body:
  # Application Layer of the Internet Protocol

  The final, topmost layer of the Internet Protocol suite.

  ![Data at the Application Layer](../img/application-layer-data.png)

  The Application Layer is where we get to the protocols that describe the
  behaviour of applications. All the preceding lower levels are effectively
  **plumbing that enables applications to do the things they want accross the
  internet**.

  Whereas the lower levels facilitate communication accross the internet, the
  Application Layer is concerned with achieving a specific task once the ability
  to communicate is assured.

  Application Layer protocols include HTTP, the Simple Mail Transfer Protocol
  (SMTP), and FTP.

raw_content:
  ---
  title: Application_Layer_of_Internet_Protocol
  tags: [internet, networks]
  created: Friday, September 06, 2024
  ---

  # Application Layer of the Internet Protocol

  The final, topmost layer of the Internet Protocol suite.

  ![Data at the Application Layer](../img/application-layer-data.png)

  The Application Layer is where we get to the protocols that describe the
  behaviour of applications. All the preceding lower levels are effectively
  **plumbing that enables applications to do the things they want accross the
  internet**.

  Whereas the lower levels facilitate communication accross the internet, the
  Application Layer is concerned with achieving a specific task once the ability
  to communicate is assured.

  Application Layer protocols include HTTP, the Simple Mail Transfer Protocol
  (SMTP), and FTP.

metadata:
  {"created":"Friday, September 06, 2024","tags":["internet","networks"],"title":"Application_Layer_of_Internet_Protocol"}


path: zk/Example_scenario_internet_data_transfer.md
sortable_path: zkExample_scenario_internet_data_transfer.md
title: Example_scenario_internet_data_transfer
lead: # Example scenario of data transfer accross the internet
checksum: fca65071ee7496db70801ad96ae0361bfbdb2c964d77fa899c4c1fef90d7de0d
created: 2024-09-06 08:39:27.364238465+00:00
modified: 2024-09-06 09:27:47.210793036+00:00
body:
  # Example scenario of data transfer accross the internet

  ![Path of a network request accross the internet](../img/network-request-layers.png)

  ![All layers of the IP](../img/all-layers-of-IP.png)

  - A client device is connected to a wireless WiFi network.
  - This network is connected to the internet via a router.
  - Elsewhere on the internet is a server also connected to the internet via
    router.
  - A user of the client device opens a web browser and requests a web page hosted
    on the server (we will assume it already knows the IP address)
  - The web browser knows HTTP - a protocol of the
    [Application Layer](Application_Layer_of_Internet_Protocol.md) - and forms an
    HTTP request intended for the destination server.
  - The browser hands off the HTTP request to the TCP/IP software stack of the
    device's [operating system](Basic_model_of_the_operating_system.md), asking
    that the data be delivered to the server on its IP address and at port 80.
  - The TCP/IP software stack on the client OS encapsulates the HTTP payload in a
    TCP segment (at the
    [Transport Layer](Transport_Layer_of_Internet_Protocol.md)), setting the
    destination port to 80 in the segment header.
  - The [Internet Layer](Internet_Layer_of_Internet_Protocol.md) software on the
    client then wraps the TCP segment in an IP packet, which includes the
    destination IP address of the server in the packet header.
  - At the [Link Layer](Link_Layer_of_Internet_Protocol.md) of the client device,
    the IP packet is encapsulated in a frame with the
    [MAC_address](MAC_addresses.md) of the local router in its header. The frame
    is wirelessly transmitted by the client device's WiFi hardware.
  - The wireless access point receives the frame and passes to the router. The
    router examines the internet layer packet to determine the destination IP
    address.
  - To reach the server, the request needs to travel through multiple routers on
    the internet and the process is replicated (looking up MAC address of the
    device and determining IP target). Eventually, the request reaches the router
    on the network where the server is connected.
  - The last router puts the packet in a frame suitable for the server's local
    network and in the frame's header is the MAC address of the server.
  - The server receives the frame and the packet is passed to the TCP/IP software
    stack. This passes the HTTP data to the process listening on TCP port 80.
  - Web server software, listening on port 80 handles the request and replies to
    the client, start ing the process again in reverse order.

raw_content:
  ---
  title: Example_scenario_internet_data_transfer
  tags: [networks, internet]
  created: Friday, September 06, 2024
  ---

  # Example scenario of data transfer accross the internet

  ![Path of a network request accross the internet](../img/network-request-layers.png)

  ![All layers of the IP](../img/all-layers-of-IP.png)

  - A client device is connected to a wireless WiFi network.
  - This network is connected to the internet via a router.
  - Elsewhere on the internet is a server also connected to the internet via
    router.
  - A user of the client device opens a web browser and requests a web page hosted
    on the server (we will assume it already knows the IP address)
  - The web browser knows HTTP - a protocol of the
    [Application Layer](Application_Layer_of_Internet_Protocol.md) - and forms an
    HTTP request intended for the destination server.
  - The browser hands off the HTTP request to the TCP/IP software stack of the
    device's [operating system](Basic_model_of_the_operating_system.md), asking
    that the data be delivered to the server on its IP address and at port 80.
  - The TCP/IP software stack on the client OS encapsulates the HTTP payload in a
    TCP segment (at the
    [Transport Layer](Transport_Layer_of_Internet_Protocol.md)), setting the
    destination port to 80 in the segment header.
  - The [Internet Layer](Internet_Layer_of_Internet_Protocol.md) software on the
    client then wraps the TCP segment in an IP packet, which includes the
    destination IP address of the server in the packet header.
  - At the [Link Layer](Link_Layer_of_Internet_Protocol.md) of the client device,
    the IP packet is encapsulated in a frame with the
    [MAC_address](MAC_addresses.md) of the local router in its header. The frame
    is wirelessly transmitted by the client device's WiFi hardware.
  - The wireless access point receives the frame and passes to the router. The
    router examines the internet layer packet to determine the destination IP
    address.
  - To reach the server, the request needs to travel through multiple routers on
    the internet and the process is replicated (looking up MAC address of the
    device and determining IP target). Eventually, the request reaches the router
    on the network where the server is connected.
  - The last router puts the packet in a frame suitable for the server's local
    network and in the frame's header is the MAC address of the server.
  - The server receives the frame and the packet is passed to the TCP/IP software
    stack. This passes the HTTP data to the process listening on TCP port 80.
  - Web server software, listening on port 80 handles the request and replies to
    the client, start ing the process again in reverse order.

metadata:
  {"created":"Friday, September 06, 2024","tags":["networks","internet"],"title":"Example_scenario_internet_data_transfer"}


path: zk/Relays.md
sortable_path: zkRelays.md
title: Relays
lead: # Relays
checksum: 5e73d8ad6aed796c6235f97cbe700c9c1ed90ffee4f76023f9093d7c6e0098df
created: 2024-09-09 15:47:45.553399322+00:00
modified: 2024-09-09 16:02:42.31330339+00:00
metadata: {"created":"Monday, September 09, 2024","tags":["electronics"],"title":"Relays"}
body:
  # Relays

  A relay is an electrically operated switch.

  It uses an electromagnet to mechancially operate a switch. A switch in one
  circuit can be used the trigger the operation of another circuit.

  ## Core components

  - electromagnet
    - a coil of wire around an iron core
  - armature
    - a movable magnetic strip
  - contacts
    - electrical connections that can be open or closed
  - spring
    - returns the armature to its resting position when not energised

  ## Operation

  ![Operation of a relay](../img/relay.gif)

  Current flows through the electromagnet which creates a magnetic field. This
  field attracts the armature which then moves to open or close the contacts. When
  the current stops, this returns the armature to its original position.

  ## Usage

  During the
  [electro-mechanical era of computing](The_History_of_Computing_Swade.md) they
  were used as precursors to vacuum tubes (and later, transistors) to implement
  logic gates. While they were superseded in this domain, they are still used in
  industrial control systems and automobiles.

raw_content:
  ---
  title: Relays
  tags: [electronics]
  created: Monday, September 09, 2024
  ---

  # Relays

  A relay is an electrically operated switch.

  It uses an electromagnet to mechancially operate a switch. A switch in one
  circuit can be used the trigger the operation of another circuit.

  ## Core components

  - electromagnet
    - a coil of wire around an iron core
  - armature
    - a movable magnetic strip
  - contacts
    - electrical connections that can be open or closed
  - spring
    - returns the armature to its resting position when not energised

  ## Operation

  ![Operation of a relay](../img/relay.gif)

  Current flows through the electromagnet which creates a magnetic field. This
  field attracts the armature which then moves to open or close the contacts. When
  the current stops, this returns the armature to its original position.

  ## Usage

  During the
  [electro-mechanical era of computing](The_History_of_Computing_Swade.md) they
  were used as precursors to vacuum tubes (and later, transistors) to implement
  logic gates. While they were superseded in this domain, they are still used in
  industrial control systems and automobiles.


