# notes

path: JavaScript_module_types.md
sortable_path: JavaScript_module_types.md
title: JavaScript module types
checksum: 32ee0dc1c2405758a7981a08a63258512ef33f621d465370885dd3a057a40eff
created: 2024-11-24 10:32:03.592097816+00:00
modified: 2025-08-10 14:08:47.351597639+00:00
metadata: {"created":"Sunday, November 24, 2024","tags":["javascript"]}
lead:
  The following info is useful when you wish to use modern JavaScript tooling
  outside of a build system.

body:
  The following info is useful when you wish to use modern JavaScript tooling
  outside of a build system.

  For instance I wanted to use a bundled React application in a static website not
  created using NPM.

  Great account by
  [Julia Evans](https://jvns.ca/blog/2024/11/18/how-to-import-a-javascript-library/).

  In essence there are three main module formats:

  - CommonJS

    - Native Node JS module system using `require()` and `module.exports()`

  - ESModules

    - The defacto standard that works in both the browser and Node.js

  - AMD

    - A browser-friendly version of CommonJS since superseded by ESModules

  Then there is UMD, _Universal Module Definition_. This is a wrapper that
  combines compatibility for multiple module systems and works with AMD, CommonJS,
  as well as global variables.

raw_content:
  ---
  tags: [javascript]
  created: Sunday, November 24, 2024
  ---

  # JavaScript module types

  The following info is useful when you wish to use modern JavaScript tooling
  outside of a build system.

  For instance I wanted to use a bundled React application in a static website not
  created using NPM.

  Great account by
  [Julia Evans](https://jvns.ca/blog/2024/11/18/how-to-import-a-javascript-library/).

  In essence there are three main module formats:

  - CommonJS

    - Native Node JS module system using `require()` and `module.exports()`

  - ESModules

    - The defacto standard that works in both the browser and Node.js

  - AMD

    - A browser-friendly version of CommonJS since superseded by ESModules

  Then there is UMD, _Universal Module Definition_. This is a wrapper that
  combines compatibility for multiple module systems and works with AMD, CommonJS,
  as well as global variables.


path: AWS_SQS_SDK.md
sortable_path: AWS_SQS_SDK.md
title: AWS SQS SDK
lead: ## Initialising the SQS client
checksum: b721c137c34d710b811bc81e027c3913e17579210602c6cd6870bbbac5751496
created: 2024-11-25 06:33:56.480592735+00:00
modified: 2025-08-10 14:08:47.19414143+00:00
metadata: {"created":"Monday, November 25, 2024","tags":["AWS","SQS"]}
body:
  ## Initialising the SQS client

  ```ts
  import { SQSClient } from "@aws-sdk/client-sqs";

  const sqs = new SQSClient({ region: "us-east-1" });
  ```

  ## Sending a message to the queue

  ```ts
  import { SendMessageCommand } from "@aws-sdk/client-sqs";

  const command = new SendMessageCommand({
    QueueUrl: queueUrl,
    MessageBody: messageBody,
    // Optional parameters
    DelaySeconds: 0,
    MessageAttributes: {
      CustomAttribute: {
        DataType: "String",
        StringValue: "CustomValue",
      },
    },
  });

  return await sqs.send(command);
  ```

  ## Processing messages on the queue

  ```ts
  import { ReceiveMessageCommand } from "@aws-sdk/client-sqs";

  const command = new ReceiveMessageCommand({
    QueueUrl: queueUrl,
    MaxNumberOfMessages: 10,
    WaitTimeSeconds: 20, // Long polling
    VisibilityTimeout: 30, // Time other consumers won't see this message
  });

  const response = await sqs.send(command);

  // Then do something

  for (const message of response.Messages || []) {
    console.log(message);
  }
  ```

  ## Removing message from queue

  ```ts
  import { DeleteMessageCommand } from "@aws-sdk/client-sqs";

  for (const message of response.Messages || []) {
    await sqs.send(
      new DeleteMessageCommand({
        QueueUrl: queueUrl,
        ReceiptHandle: message.ReceiptHandle,
      }),
    );
  }
  ```

  ## Query a DLQ

  ```ts
  const checkDLQ = async (dlqUrl: string) => {
    const command = new GetQueueAttributesCommand({
      QueueUrl: dlqUrl,
      AttributeNames: ["ApproximateNumberOfMessages"],
    });

    const response = await sqs.send(command);
    return response.Attributes?.ApproximateNumberOfMessages;
  };
  ```

raw_content:
  ---
  tags: [AWS, SQS]
  created: Monday, November 25, 2024
  ---

  # AWS SQS SDK

  ## Initialising the SQS client

  ```ts
  import { SQSClient } from "@aws-sdk/client-sqs";

  const sqs = new SQSClient({ region: "us-east-1" });
  ```

  ## Sending a message to the queue

  ```ts
  import { SendMessageCommand } from "@aws-sdk/client-sqs";

  const command = new SendMessageCommand({
    QueueUrl: queueUrl,
    MessageBody: messageBody,
    // Optional parameters
    DelaySeconds: 0,
    MessageAttributes: {
      CustomAttribute: {
        DataType: "String",
        StringValue: "CustomValue",
      },
    },
  });

  return await sqs.send(command);
  ```

  ## Processing messages on the queue

  ```ts
  import { ReceiveMessageCommand } from "@aws-sdk/client-sqs";

  const command = new ReceiveMessageCommand({
    QueueUrl: queueUrl,
    MaxNumberOfMessages: 10,
    WaitTimeSeconds: 20, // Long polling
    VisibilityTimeout: 30, // Time other consumers won't see this message
  });

  const response = await sqs.send(command);

  // Then do something

  for (const message of response.Messages || []) {
    console.log(message);
  }
  ```

  ## Removing message from queue

  ```ts
  import { DeleteMessageCommand } from "@aws-sdk/client-sqs";

  for (const message of response.Messages || []) {
    await sqs.send(
      new DeleteMessageCommand({
        QueueUrl: queueUrl,
        ReceiptHandle: message.ReceiptHandle,
      }),
    );
  }
  ```

  ## Query a DLQ

  ```ts
  const checkDLQ = async (dlqUrl: string) => {
    const command = new GetQueueAttributesCommand({
      QueueUrl: dlqUrl,
      AttributeNames: ["ApproximateNumberOfMessages"],
    });

    const response = await sqs.send(command);
    return response.Attributes?.ApproximateNumberOfMessages;
  };
  ```


path: AWS_SNS.md
sortable_path: AWS_SNS.md
title: AWS SNS
checksum: f12c6513eaba9963248094d422b8c2e20f6c393909a60ba34a4a7721b52e9f7c
created: 2024-11-25 06:50:32.993088026+00:00
modified: 2025-08-10 14:08:47.017499569+00:00
metadata: {"created":"Monday, November 25, 2024","tags":["AWS","SNS","SQS"]}
lead:
  The Simple Notification Service is another form of messaging service in AWS,
  similar to [SQS](AWS_SQS.md). The key difference is that SNS pushes messages out
  to subscribers whereas SQS stores the messages until they are processed.

body:
  The Simple Notification Service is another form of messaging service in AWS,
  similar to [SQS](AWS_SQS.md). The key difference is that SNS pushes messages out
  to subscribers whereas SQS stores the messages until they are processed.

  Before a message can be sent, and SNS topic must exist. One service such as a
  lambda will _publish_ to this topic. Another service, will _subscribe_ to the
  topic and be alerted when a message is sent to it.

  With SNS, one message can be sent to multiple subscribers simultaneously. In
  addition there is message filtering at the subscription level so subscribers can
  choose to receive only the messages they are interested in.

  Many other AWS services can act as SNS subscribers. Some common examples:

  - SQS queues
  - Lambda functions (trigger on receipt of a message)
  - HTTP endpoints
  - Email/SMS/Push notifications to mobile apps

  ## Combining with SQS

  Utilising SNS with SQS is a common pattern in serverless applications.

  We use SNS to decouple publishers from subscribers and then use SQS to handle
  the different subscriber tasks. This is known as the "Fanout-Queue" pattern.

  ![SQS fanout pattern](../img/sns-example.png)

  For example, an order is placed which results in a publication event to an SNS
  topic.

  Multiple systems subscribe to this topic, e.g. an inventory system, a shipping
  system and an analytics system.

  Each system has its own queue for processing the message.

raw_content:
  ---
  tags: [AWS, SNS, SQS]
  created: Monday, November 25, 2024
  ---

  # AWS SNS

  The Simple Notification Service is another form of messaging service in AWS,
  similar to [SQS](AWS_SQS.md). The key difference is that SNS pushes messages out
  to subscribers whereas SQS stores the messages until they are processed.

  Before a message can be sent, and SNS topic must exist. One service such as a
  lambda will _publish_ to this topic. Another service, will _subscribe_ to the
  topic and be alerted when a message is sent to it.

  With SNS, one message can be sent to multiple subscribers simultaneously. In
  addition there is message filtering at the subscription level so subscribers can
  choose to receive only the messages they are interested in.

  Many other AWS services can act as SNS subscribers. Some common examples:

  - SQS queues
  - Lambda functions (trigger on receipt of a message)
  - HTTP endpoints
  - Email/SMS/Push notifications to mobile apps

  ## Combining with SQS

  Utilising SNS with SQS is a common pattern in serverless applications.

  We use SNS to decouple publishers from subscribers and then use SQS to handle
  the different subscriber tasks. This is known as the "Fanout-Queue" pattern.

  ![SQS fanout pattern](../img/sns-example.png)

  For example, an order is placed which results in a publication event to an SNS
  topic.

  Multiple systems subscribe to this topic, e.g. an inventory system, a shipping
  system and an analytics system.

  Each system has its own queue for processing the message.


path: Architecture_of_the_World_Wide_Web.md
sortable_path: Architecture_of_the_World_Wide_Web.md
title: Architecture of the World Wide Web
lead: ## Terms
checksum: 728a7f6d45ae4e59189ac0cb9ee63a5ee7909701f53784f6b2366793efcf8ccc
created: 2024-12-13 13:06:19.172815289+00:00
modified: 2025-08-10 14:08:47.108509009+00:00
metadata: {"created":"Friday, December 13, 2024","tags":["world-wide-web","internet"]}
body:
  ## Terms

  <dl>
  <dt>web</dt>
  <dd>a set of resources delivered over the Internet using the HTTP protocol</dd>

  <dt>web resource</dt>
  <dd>any information that can be accessed using the web (document, image, video)</dd>

  <dt>web server</dt>
  <dd>a software program on a computer that hosts web resources</dd>

  <dt>web browser</dt>
  <dd>an application used to acccess web resources on web servers</dd>

  </dl>

  ## The web is _distributed_, _addressable_, and _linked_

  ### Distributed

  There is no centralised organisation or system that decides the content that can
  be published. Any computer connected to the Internet can run a web server and
  the person running it can make available any content they wish.

  ### _Addressable_

  Every resource on the web has a unique address that includes both its location
  and how to access it (URL):

  ![](../img/web-url.png)

  - scheme
    - specifies the [Application Layer](Application_Layer_of_Internet_Protocol.md for
      accessing the resource. In the example this is HTTP but it could be FTP or another
      protocol, always bracketed with a colon
  - authority
    - comprises the [DNS](Transport_Layer_of_Internet_Protocol.md) hostname of the
      server containing the resource. This could also be an
      [IP address](IP_addresses.md) and could also include a port number
  - path
    - the location of the resource on the web server, organised hierarchically and
      analagous to a file path
  - query
    - a modifier of the resource returned to the client. Its usage varies accross
      sites but is typically deterimined by backend implementation details

  > Note a URL does not have to always contain _all_ this information. Consider an
  > index page.

raw_content:
  ---
  tags: [world-wide-web, internet]
  created: Friday, December 13, 2024
  ---

  # Architecture of the World Wide Web

  ## Terms

  <dl>
  <dt>web</dt>
  <dd>a set of resources delivered over the Internet using the HTTP protocol</dd>

  <dt>web resource</dt>
  <dd>any information that can be accessed using the web (document, image, video)</dd>

  <dt>web server</dt>
  <dd>a software program on a computer that hosts web resources</dd>

  <dt>web browser</dt>
  <dd>an application used to acccess web resources on web servers</dd>

  </dl>

  ## The web is _distributed_, _addressable_, and _linked_

  ### Distributed

  There is no centralised organisation or system that decides the content that can
  be published. Any computer connected to the Internet can run a web server and
  the person running it can make available any content they wish.

  ### _Addressable_

  Every resource on the web has a unique address that includes both its location
  and how to access it (URL):

  ![](../img/web-url.png)

  - scheme
    - specifies the [Application Layer](Application_Layer_of_Internet_Protocol.md for
      accessing the resource. In the example this is HTTP but it could be FTP or another
      protocol, always bracketed with a colon
  - authority
    - comprises the [DNS](Transport_Layer_of_Internet_Protocol.md) hostname of the
      server containing the resource. This could also be an
      [IP address](IP_addresses.md) and could also include a port number
  - path
    - the location of the resource on the web server, organised hierarchically and
      analagous to a file path
  - query
    - a modifier of the resource returned to the client. Its usage varies accross
      sites but is typically deterimined by backend implementation details

  > Note a URL does not have to always contain _all_ this information. Consider an
  > index page.


path: HTTPS.md
sortable_path: HTTPS.md
title: HTTPS
checksum: acc6bc1db39682003becdecd7d512d72398168b76cdbb1bc254e1c867012aa51
created: 2024-12-13 13:23:23.159398701+00:00
modified: 2025-08-10 14:08:47.308839955+00:00
metadata: {"created":"Friday, December 14, 2024","tags":["world-wide-web","internet","encryption","servers"]}
lead:
  The problem with standard, unencrypted HTTP requests is that the data can be
  captured in transit and be observed or modified by malicious actors.

body:
  The problem with standard, unencrypted HTTP requests is that the data can be
  captured in transit and be observed or modified by malicious actors.

  HTTPS is the solution. It encrypts the entirety of the HTTP request and response
  (URL, headers, body).

  It utilises two types of encryption to send messages securely:

  - symmetric encryption
    - a single shared key between client and server to encrypt the messages sent
      between them
  - asymmetric encryption
    - two keys are used: a [public key](./bbdcb54f_public_key_cryptography.md) to
      encrypt the data and a private key to decrypt it. The public key can be
      shared freely so anyone can encrypt and send data to a peer but only the
      peer can receive and decrypt it

  > Symmetric encryption applies once the server has been authenticated and its
  > public key has been shared with the client. Asymmetric encryption applies at
  > the initiation phase only.

  The encryption protocol used is TLS - the **Transport Layer Security Protocol**.
  Previously SSL (Secure Sockets Layer) was used but has been deprecated over
  security concerns.

  The initial handshake process takes place entirely within the Network Layer,
  over TCP. Only once encryption is secured is HTTPS activated at the
  [Application Layer](./Application_Layer_of_Internet_Protocol.md)

  ## How communication over HTTPs works

  ![Server handshake diagram](../img/https-handshake.png)

  - An initial handshake is shared between client and server, agreeing that
    communication will be encrypted. This is public. "Client hello" and "Server
    hello". Some random numbers are exchanged at this point
  - The server sends a digital [certificate](./Certificate_authorities.md) key The
    client checks that the certificate is valid
  - If valid, the client encrypts a string of bytes using the public key and sends
    it to the server. This is called the "premaster secret"
  - This is decrypted by the server
  - The premaster secret, along with the random bytes from the handshake is then
    used to compute a shared secret key (symmetric) which is used to encrypt all
    subsequent messages for the duration of the session

  ## Caching

  Once symmetric exchange is achieved the TLS handshake is complete and lasts
  typically until either terminates it.

  However, some caching does occur on both the server and browser side. Both use
  session IDs to enact an abbreviated handshake such as when the session cache
  expires. This way, certificate validation and the asymmetric exchange can be
  skipped. These tickets are shared as part of the original exchange and the
  client can present them to the server as proof they already authenticated when
  they come back later.

raw_content:
  ---
  tags: [world-wide-web, internet, encryption, servers]
  created: Friday, December 14, 2024
  ---

  # HTTPS

  The problem with standard, unencrypted HTTP requests is that the data can be
  captured in transit and be observed or modified by malicious actors.

  HTTPS is the solution. It encrypts the entirety of the HTTP request and response
  (URL, headers, body).

  It utilises two types of encryption to send messages securely:

  - symmetric encryption
    - a single shared key between client and server to encrypt the messages sent
      between them
  - asymmetric encryption
    - two keys are used: a [public key](./bbdcb54f_public_key_cryptography.md) to
      encrypt the data and a private key to decrypt it. The public key can be
      shared freely so anyone can encrypt and send data to a peer but only the
      peer can receive and decrypt it

  > Symmetric encryption applies once the server has been authenticated and its
  > public key has been shared with the client. Asymmetric encryption applies at
  > the initiation phase only.

  The encryption protocol used is TLS - the **Transport Layer Security Protocol**.
  Previously SSL (Secure Sockets Layer) was used but has been deprecated over
  security concerns.

  The initial handshake process takes place entirely within the Network Layer,
  over TCP. Only once encryption is secured is HTTPS activated at the
  [Application Layer](./Application_Layer_of_Internet_Protocol.md)

  ## How communication over HTTPs works

  ![Server handshake diagram](../img/https-handshake.png)

  - An initial handshake is shared between client and server, agreeing that
    communication will be encrypted. This is public. "Client hello" and "Server
    hello". Some random numbers are exchanged at this point
  - The server sends a digital [certificate](./Certificate_authorities.md) key The
    client checks that the certificate is valid
  - If valid, the client encrypts a string of bytes using the public key and sends
    it to the server. This is called the "premaster secret"
  - This is decrypted by the server
  - The premaster secret, along with the random bytes from the handshake is then
    used to compute a shared secret key (symmetric) which is used to encrypt all
    subsequent messages for the duration of the session

  ## Caching

  Once symmetric exchange is achieved the TLS handshake is complete and lasts
  typically until either terminates it.

  However, some caching does occur on both the server and browser side. Both use
  session IDs to enact an abbreviated handshake such as when the session cache
  expires. This way, certificate validation and the asymmetric exchange can be
  skipped. These tickets are shared as part of the original exchange and the
  client can present them to the server as proof they already authenticated when
  they come back later.


path: What_can_the_ISP_see.md
sortable_path: What_can_the_ISP_see.md
title: What can the ISP see?
lead: ## Plain HTTP
checksum: 992a0ae00256f2ee6644480b697656300e6b8b22ecb34c4a05dae060081b24f2
created: 2024-12-13 14:14:32.050396895+00:00
modified: 2025-08-10 14:08:47.079927942+00:00
metadata: {"created":"Friday, December 13, 2024","tags":["internet","encryption","privacy","Tor"]}
body:
  ## Plain HTTP

  The ISP can see all content exchanged: DNS lookups, IP address of the server you
  connect to, requests and body content.

  ## Just HTTPS

  With HTTPS alone, the content of the messages is encrypted but the ISP can see:

  - DNS queries
  - IP addresses you connect to
  - Traffic patterns and metadata (when, how often, amount of data exchanged etc)

  ## HTTPS and encrypted DNS server

  With HTTPS and an encrypted DNS server (e.g. Quad9) the DNS queries are hidden
  but the IP addresses are not. So they could still derive your DNS lookups from
  the IPs you end up connecting to.

  ## HTTPS and VPN

  The ISP only sees the encrypted traffic to the VPN server. Your real IP and
  destination IP is hidden.

  However the VPN provider can potentially see DNS queries, depending on their
  policy. Although a good VPN provider will encrypt DNS lookups like an encrypted
  DNS server.

  The VPN can also still see what IPs you are connecting to. You are basically
  swiching trust from the ISP to the VPN provider. However reputable VPNs do not
  have your billing and identity info (and payment can use cryptocurrencies) and
  don't keep logs. Many also have policies about refusal to cooporate with
  authorities in different juristictions. Also the VPN doesn't know your real IP.

  ## HTTPS, VPN, and encrypted DNS

  DNS queries hidded from both ISP and VPN. Traffic is fully encrypted and routed
  through VPN.

  ## Fingerprinting

  Note that none of the above stops fingerprinting. Even with full encryption
  there are methods of determining online behaviour through encrypted packet
  analysis and usage patterns.

  A way round this is to use SOCKS proxies which add hops or to use TOR. TOR
  effectively solves the "have to trust VPN provider" problem.

  With TOR:

  - Traffic routed through 3+ nodes
  - Each node only knows previous and next hop
  - No single node knows both source and destination. Even Tor nodes can't see the
    full path

  It also uses:

  - Fixed packet sizes
  - Timing obfuscation
  - Traffic padding

  Which makes pattern analysis and fingerprinting much harder than just a VPN.

  Furthermore there is also the following protection from VPN weaknesses:

  - No single provider to trust/compromise
  - Decentralized network
  - Free to use and no account needed
  - No permanent exit node like with VPN

raw_content:
  ---
  tags: [internet, encryption, privacy, Tor]
  created: Friday, December 13, 2024
  ---

  # What can the ISP see?

  ## Plain HTTP

  The ISP can see all content exchanged: DNS lookups, IP address of the server you
  connect to, requests and body content.

  ## Just HTTPS

  With HTTPS alone, the content of the messages is encrypted but the ISP can see:

  - DNS queries
  - IP addresses you connect to
  - Traffic patterns and metadata (when, how often, amount of data exchanged etc)

  ## HTTPS and encrypted DNS server

  With HTTPS and an encrypted DNS server (e.g. Quad9) the DNS queries are hidden
  but the IP addresses are not. So they could still derive your DNS lookups from
  the IPs you end up connecting to.

  ## HTTPS and VPN

  The ISP only sees the encrypted traffic to the VPN server. Your real IP and
  destination IP is hidden.

  However the VPN provider can potentially see DNS queries, depending on their
  policy. Although a good VPN provider will encrypt DNS lookups like an encrypted
  DNS server.

  The VPN can also still see what IPs you are connecting to. You are basically
  swiching trust from the ISP to the VPN provider. However reputable VPNs do not
  have your billing and identity info (and payment can use cryptocurrencies) and
  don't keep logs. Many also have policies about refusal to cooporate with
  authorities in different juristictions. Also the VPN doesn't know your real IP.

  ## HTTPS, VPN, and encrypted DNS

  DNS queries hidded from both ISP and VPN. Traffic is fully encrypted and routed
  through VPN.

  ## Fingerprinting

  Note that none of the above stops fingerprinting. Even with full encryption
  there are methods of determining online behaviour through encrypted packet
  analysis and usage patterns.

  A way round this is to use SOCKS proxies which add hops or to use TOR. TOR
  effectively solves the "have to trust VPN provider" problem.

  With TOR:

  - Traffic routed through 3+ nodes
  - Each node only knows previous and next hop
  - No single node knows both source and destination. Even Tor nodes can't see the
    full path

  It also uses:

  - Fixed packet sizes
  - Timing obfuscation
  - Traffic padding

  Which makes pattern analysis and fingerprinting much harder than just a VPN.

  Furthermore there is also the following protection from VPN weaknesses:

  - No single provider to trust/compromise
  - Decentralized network
  - Free to use and no account needed
  - No permanent exit node like with VPN


path: Network_packet_analysis_tools.md
sortable_path: Network_packet_analysis_tools.md
title: Network packet analysis tools
checksum: 6bffbdc731d985354637c3e58cefa82c0fce3db0c5514e06c091274a960afb4c
created: 2024-12-13 14:39:26.730227868+00:00
modified: 2025-08-10 14:08:47.075862617+00:00
metadata: {"created":"Friday, December 13, 2024","tags":["networks","procedural"]}
lead:
  - `tcpdump`: View raw packet data (`tcpdump -i wlan0 -XX` for full packet
    contents)
  - tshark: Wireshark's CLI for detailed packet analysis (`tshark -i wlan0 -V`)
  - `nethogs`: Monitor per-application network usage
  - `iftop`: Show bandwidth usage per connection

body:
  - `tcpdump`: View raw packet data (`tcpdump -i wlan0 -XX` for full packet
    contents)
  - tshark: Wireshark's CLI for detailed packet analysis (`tshark -i wlan0 -V`)
  - `nethogs`: Monitor per-application network usage
  - `iftop`: Show bandwidth usage per connection

raw_content:
  ---
  tags: [networks, procedural]
  created: Friday, December 13, 2024
  ---

  # Network packet analysis tools

  - `tcpdump`: View raw packet data (`tcpdump -i wlan0 -XX` for full packet
    contents)
  - tshark: Wireshark's CLI for detailed packet analysis (`tshark -i wlan0 -V`)
  - `nethogs`: Monitor per-application network usage
  - `iftop`: Show bandwidth usage per connection


path: User_agent.md
sortable_path: User_agent.md
title: User agent
checksum: 8ebaa79e8318ce51042596b2b34faeec781367002c93c8f5a8007a60e6c8a553
created: 2024-12-20 11:01:51.175194304+00:00
modified: 2025-08-10 14:08:47.149319651+00:00
metadata: {"created":"Friday, December 20, 2024","tags":["world-wide-web"]}
lead:
  _User agent_ is the technical term for a web browser. So named because it acts
  on behalf of the user.

body:
  _User agent_ is the technical term for a web browser. So named because it acts
  on behalf of the user.

  When a browser makes a request to a web server it includes a header value called
  `User-Agent` that describes the browser.

raw_content:
  ---
  tags: [world-wide-web]
  created: Friday, December 20, 2024
  ---

  # User agent

  _User agent_ is the technical term for a web browser. So named because it acts
  on behalf of the user.

  When a browser makes a request to a web server it includes a header value called
  `User-Agent` that describes the browser.


path: Dynamic_and_static_websites.md
sortable_path: Dynamic_and_static_websites.md
title: Dynamic and static websites
checksum: d823c51bb34e1fb4ebaf63240ab4f4196c1587b83e90ef6b71a80ce28ea9cdf1
created: 2024-12-20 11:06:21.197224512+00:00
modified: 2025-08-10 14:08:47.339503906+00:00
metadata: {"created":"Friday, December 20, 2024","tags":["world-wide-web"]}
lead:
  When we talk about dynamic and static websites we are talking about the role of
  the server, not anything to do with how the website behaves for the user.

body:
  When we talk about dynamic and static websites we are talking about the role of
  the server, not anything to do with how the website behaves for the user.

  A static site's content comes unchanged from the files on the server whereas a
  dynamic site's content is generated on the server.

  With a static website, the content of the site has been build before a request
  for it is made. It's HTML, CSS and JS are stored on the server and are simply
  returned to the client in this form when they are requested.

  When content is requested on a dynamic website the HTML is generated on-the-fly
  on the server-side and returned to the client. Typically a request will be made
  for an item of content, the data will be fetched on the backend from a DB and
  then formatted as HTML and sent back to the client.

  Static websites can be more performant because of the lack of BE processing
  however it means that any runtime processing or special rendering must be
  handled by the frontend JavaScript, without relying on server languages.

  Take the sourcing of remote data. With a dynamic site this can be handled in a
  single request because the backend can fetch the data and return to the client
  with the rendering applied. On the frontend, JS would have to make a request to
  an endpoint that calls a backend service that returns the data. Having received
  the raw data it would then need to render it.

  Nowadays there is generally a mixed approach. This distinction was more
  important in the early days of the web.

raw_content:
  ---
  tags: [world-wide-web]
  created: Friday, December 20, 2024
  ---

  # Dynamic and static websites

  When we talk about dynamic and static websites we are talking about the role of
  the server, not anything to do with how the website behaves for the user.

  A static site's content comes unchanged from the files on the server whereas a
  dynamic site's content is generated on the server.

  With a static website, the content of the site has been build before a request
  for it is made. It's HTML, CSS and JS are stored on the server and are simply
  returned to the client in this form when they are requested.

  When content is requested on a dynamic website the HTML is generated on-the-fly
  on the server-side and returned to the client. Typically a request will be made
  for an item of content, the data will be fetched on the backend from a DB and
  then formatted as HTML and sent back to the client.

  Static websites can be more performant because of the lack of BE processing
  however it means that any runtime processing or special rendering must be
  handled by the frontend JavaScript, without relying on server languages.

  Take the sourcing of remote data. With a dynamic site this can be handled in a
  single request because the backend can fetch the data and return to the client
  with the rendering applied. On the frontend, JS would have to make a request to
  an endpoint that calls a backend service that returns the data. Having received
  the raw data it would then need to render it.

  Nowadays there is generally a mixed approach. This distinction was more
  important in the early days of the web.


path: Binary_encoding_of_sound.md
sortable_path: Binary_encoding_of_sound.md
title: Binary encoding of sound
lead: Sound is a change in air pressure.
checksum: 49f4a2ec25f9431877890e0f6edfb7b9725f9b51382071c985b518d05793e144
created: 2024-12-27 12:29:02.492923417+00:00
modified: 2025-08-10 14:08:47.109657729+00:00
metadata: {"created":"Friday, December 27, 2024","tags":["sound","binary","analogue"]}
body:
  Sound is a change in air pressure.

  To record sound, we convert the change in air pressure into an electrical signal
  by means of a microphone. Effectively, the same natural waveform is converted
  from one medium (the vibration of air molecules) into another medium (electrical
  charge in a field) whilst retaining the same morphology. See
  [Recording and playback of sound](Recording_and_playback_of_sound.md).

  The resulting electrical signal is analogue: it has a continuous and variable
  frequency and amplitude and thus can have any value at any given moment.

  To encode this information digitally we need to make the continuous analogue
  single discrete, so that it can represented with a finite set of symbols (bits).

  This process is known as **quantization** and consists in sampling the analogue
  waveform. This means taking a measurement of it at different points in time -
  each second.

  CDs sample the audio source at a rate of 44.1 kHz: 44, 100 measurements of the
  signal per second.

  The measurements are then stored in a digital format: bits. The amount of bits
  used to record the measurement is the **bit-depth**. Each measurement is rounded
  to the nearest available digital value. CDs have a depth of 16 bits. Thus each
  measurement can be one of $16^2 = 65,536$ possible values.

  With sampling at a given bit-depth established, the **bit-rate** can be derived.

  This is the sampling rate (number of measurements per second) divided by the
  number of bits per sample (bit-depth), multiplied by the number of channels.

  Thus for stereo CDs this is:

  $$
  (44100 / 16 ) \cdot 2 = 1411200
  $$

  Or 1.4 Mb/s

  > Even with a high bit-rate (as with CDs) the process still necessarily involves
  > a loss of information when converting sound from analogue to digital. However,
  > given a high enough value for the sampling rate and bit-depth, the differences
  > are imperceptible to the human ear.

raw_content:
  ---
  tags: [sound, binary, analogue]
  created: Friday, December 27, 2024
  ---

  # Binary encoding of sound

  Sound is a change in air pressure.

  To record sound, we convert the change in air pressure into an electrical signal
  by means of a microphone. Effectively, the same natural waveform is converted
  from one medium (the vibration of air molecules) into another medium (electrical
  charge in a field) whilst retaining the same morphology. See
  [Recording and playback of sound](Recording_and_playback_of_sound.md).

  The resulting electrical signal is analogue: it has a continuous and variable
  frequency and amplitude and thus can have any value at any given moment.

  To encode this information digitally we need to make the continuous analogue
  single discrete, so that it can represented with a finite set of symbols (bits).

  This process is known as **quantization** and consists in sampling the analogue
  waveform. This means taking a measurement of it at different points in time -
  each second.

  CDs sample the audio source at a rate of 44.1 kHz: 44, 100 measurements of the
  signal per second.

  The measurements are then stored in a digital format: bits. The amount of bits
  used to record the measurement is the **bit-depth**. Each measurement is rounded
  to the nearest available digital value. CDs have a depth of 16 bits. Thus each
  measurement can be one of $16^2 = 65,536$ possible values.

  With sampling at a given bit-depth established, the **bit-rate** can be derived.

  This is the sampling rate (number of measurements per second) divided by the
  number of bits per sample (bit-depth), multiplied by the number of channels.

  Thus for stereo CDs this is:

  $$
  (44100 / 16 ) \cdot 2 = 1411200
  $$

  Or 1.4 Mb/s

  > Even with a high bit-rate (as with CDs) the process still necessarily involves
  > a loss of information when converting sound from analogue to digital. However,
  > given a high enough value for the sampling rate and bit-depth, the differences
  > are imperceptible to the human ear.


path: Audio_file_formats.md
sortable_path: Audio_file_formats.md
title: Audio file formats
lead: ## CD quality
checksum: 618daa1d68a637b54f26e3e16041039985606daa493d576509d141c4b5fd7c72
created: 2024-12-27 12:29:18.056498357+00:00
modified: 2025-08-10 14:08:47.225326969+00:00
metadata: {"created":"Friday, December 27, 2024","tags":["sound","binary"]}
body:
  ## CD quality

  We can use CD's as the digital gold standard as the best digital quality
  available (to consumers).

  CDs store audio in uncompressed PCM (Pulse Code Modulation) format.

  They have a sampling rate of 44.1kHz, which is done in two channels to enable
  stereo.

  ## What lossy formats entail

  A conversion to MP3 from, for example, a CD source is always a one-way process
  and is not reversible. Once information is discarded in the compression process,
  it cannot be retrieved.

  This is obviously in contrast to lossless methods like FLAC where the original
  CD audio can always be reconstructed.

  It follows from the above that if you repeatedly encode a CD source to MP3, it
  will deteriorate in quality since more data is being removed each time.

  ## Major audio formats

  ### WAV (Waveform Audio File Format)

  - CD-quality encoding with no compression
  - Bit-for-bit identical to the CD source
  - Historically developed for Windows machines but can play on all operating
    systems

  ### FLAC (Free Lossless Audio Codec)

  - Basically the same as WAV but in a (losslessly) compressed format
  - The difference between a novel in a text file (WAV) and as a zipped file

  ### MP3 (MPEG-1 Audio Layer MP3)

  - Lossy format.
  - When a WAV file (or other lossless format) is converted to MP3 a Fast Fourier
    Transform analysis is performed to determine the frequency of certain sounds
  - This is used by the encoder to decide which parts of the sound are
    imperceptible and thus which can be discarded to reduce the file size. This is
    done through the application of psycho-acoustic models.
  - The remaining data is then compressed
  - Examples of the data reduction applied:

    - Removing frequencies that humans cannot hear
    - Removing quieter sounds that are masked by louder sounds
    - Combining similar frequencies
    - Reducing stereo information where it is less noticeable

  ### OGG: Ogg Vorbis

  - An open-source alternative to MP3
  - Typically achieves better quality than MP3 at the same bit rate, especially at
    very high/ low frequencies
  - Also better stereo handling at low frequencies
  - Uses a more modern psycho-acoustic model

  ## Variable and constant bit rates

  For lossy formats like MP3, the amount of data that is being encoded from the
  uncompressed source is expressed via the unit of "bit rate": **how many
  thousands of bits are being used to represent each second of audio**.

  There are two methods of encoding that impact on the bit rate.

  With _variable bit rate_ encoding, the encoder dynamically adjusts the bit rate
  depending on what is happening in the music. During complex passages (e.g. a
  full orchestra) it uses a higher bit rate to capture the detail. During simpler
  passages (like a single instrument) it uses lower bit rates since less data is
  needed and during silence it can drop to very low bit rates.

  _Constant bit rate encoding_ uses the same bit rate throughout, it is
  consequently less efficient.

  When talking about the quality of MP3s there are generally two bit rates cited:

  - ~128kbps: acceptable but significantly reduced
  - 320kbps: the highest quality you can get whilst still using a lossy method
    like MP3

  With VBR, these are sometimes expressed as an average.

  Subjectively, A 128kbps MP3 might sound "underwater" or "swishy", while a
  320kbps version would preserve much more detail.

  Still, the bit-rate of a CD is 1411kbps! A 128kb/s MP3 is therefore only
  capturing about 9% of CD quality and a 320kbps MP3 is capturing about 23% of CD
  quality.

  ## Streaming services

  Spotify uses Ogg Vorbis throughout but uses different bit rates for its
  different tiers. The free tier has a range from 24-160kb/s at VBR with the
  option of 320kb/s on the premium tier.

  Other services offer FLAC or FLAC-equivalent quality at their most expensive
  tiers (Apple Music, Amazon Music, Tidal).

  Og Vorbis is particularly well-suited to streaming. It can seamlessly switch bit
  rates during the stream which is beneficial with changeable network conditions.
  Plus data is organised into independent packets so if a packet is lost there is
  no perceptible difference.

raw_content:
  ---
  tags: [sound, binary]
  created: Friday, December 27, 2024
  ---

  # Audio file formats

  ## CD quality

  We can use CD's as the digital gold standard as the best digital quality
  available (to consumers).

  CDs store audio in uncompressed PCM (Pulse Code Modulation) format.

  They have a sampling rate of 44.1kHz, which is done in two channels to enable
  stereo.

  ## What lossy formats entail

  A conversion to MP3 from, for example, a CD source is always a one-way process
  and is not reversible. Once information is discarded in the compression process,
  it cannot be retrieved.

  This is obviously in contrast to lossless methods like FLAC where the original
  CD audio can always be reconstructed.

  It follows from the above that if you repeatedly encode a CD source to MP3, it
  will deteriorate in quality since more data is being removed each time.

  ## Major audio formats

  ### WAV (Waveform Audio File Format)

  - CD-quality encoding with no compression
  - Bit-for-bit identical to the CD source
  - Historically developed for Windows machines but can play on all operating
    systems

  ### FLAC (Free Lossless Audio Codec)

  - Basically the same as WAV but in a (losslessly) compressed format
  - The difference between a novel in a text file (WAV) and as a zipped file

  ### MP3 (MPEG-1 Audio Layer MP3)

  - Lossy format.
  - When a WAV file (or other lossless format) is converted to MP3 a Fast Fourier
    Transform analysis is performed to determine the frequency of certain sounds
  - This is used by the encoder to decide which parts of the sound are
    imperceptible and thus which can be discarded to reduce the file size. This is
    done through the application of psycho-acoustic models.
  - The remaining data is then compressed
  - Examples of the data reduction applied:

    - Removing frequencies that humans cannot hear
    - Removing quieter sounds that are masked by louder sounds
    - Combining similar frequencies
    - Reducing stereo information where it is less noticeable

  ### OGG: Ogg Vorbis

  - An open-source alternative to MP3
  - Typically achieves better quality than MP3 at the same bit rate, especially at
    very high/ low frequencies
  - Also better stereo handling at low frequencies
  - Uses a more modern psycho-acoustic model

  ## Variable and constant bit rates

  For lossy formats like MP3, the amount of data that is being encoded from the
  uncompressed source is expressed via the unit of "bit rate": **how many
  thousands of bits are being used to represent each second of audio**.

  There are two methods of encoding that impact on the bit rate.

  With _variable bit rate_ encoding, the encoder dynamically adjusts the bit rate
  depending on what is happening in the music. During complex passages (e.g. a
  full orchestra) it uses a higher bit rate to capture the detail. During simpler
  passages (like a single instrument) it uses lower bit rates since less data is
  needed and during silence it can drop to very low bit rates.

  _Constant bit rate encoding_ uses the same bit rate throughout, it is
  consequently less efficient.

  When talking about the quality of MP3s there are generally two bit rates cited:

  - ~128kbps: acceptable but significantly reduced
  - 320kbps: the highest quality you can get whilst still using a lossy method
    like MP3

  With VBR, these are sometimes expressed as an average.

  Subjectively, A 128kbps MP3 might sound "underwater" or "swishy", while a
  320kbps version would preserve much more detail.

  Still, the bit-rate of a CD is 1411kbps! A 128kb/s MP3 is therefore only
  capturing about 9% of CD quality and a 320kbps MP3 is capturing about 23% of CD
  quality.

  ## Streaming services

  Spotify uses Ogg Vorbis throughout but uses different bit rates for its
  different tiers. The free tier has a range from 24-160kb/s at VBR with the
  option of 320kb/s on the premium tier.

  Other services offer FLAC or FLAC-equivalent quality at their most expensive
  tiers (Apple Music, Amazon Music, Tidal).

  Og Vorbis is particularly well-suited to streaming. It can seamlessly switch bit
  rates during the stream which is beneficial with changeable network conditions.
  Plus data is organised into independent packets so if a packet is lost there is
  no perceptible difference.


path: Recording_and_playback_of_sound.md
sortable_path: Recording_and_playback_of_sound.md
title: Recording and playback of sound
lead: ## Analogue
checksum: ba117d13d562a1cfffa839816844effd32d437afd558d0cb121f857c170ac5b1
created: 2025-01-02 16:47:21.217929078+00:00
modified: 2025-08-10 14:08:47.246942078+00:00
metadata: {"created":"Thursday, January 02, 2025","tags":["sound","analogue","physics"]}
body:
  ## Analogue

  ### Recording

  The microphone converts sound waves into a continuous electrical signal.

  The signal drives the recording equipment.

  In the case of a vinyl recording: the signal drives a cutting head that carves a
  groove into the master disk.

  In the case of a tape recording: the signal drives electro-magnets that align
  magnetic particles on the tape.

  In both cases, **the recording medium physically mirrors the shape of the sound
  wave**.

  ### Playback

  In the case of vinyl, the needle follows the groove, moving up and down to
  recreate the electrical signal.

  In the case of tape, the magnetic patterns induce electrical current in the
  playback head.

  In each case, the signal is transferred directly to amplifiers and speakers.

  ## Digital

  ### Recording

  Mostly covered in [Binary encoding of sound](Binary_encoding_of_sound.md).

  An important point to note is that the binary encoding that results from the
  sampling is lasered into the CD as microscopic impressions and depressions (pits
  and lands)

  ### Playback

  A laser in the reader reads the pits and lands and converts them back to binary
  numbers. A DAC then converts the numbers back into a continuous electrical
  signal which is then channelled to the amplifiers and speakers.

  ## From recording to production of storage medium

  With analogue production recordings would be made onto tape. Multiple takes are
  recorded, mixed and edited. This results in a master tape which is the final
  version of the recorded sound.

  This then goes to a mastering engineer for mastering which creates the master
  mix. This consists in optimising the sound for the given format (vinyl or tape).
  Also includes adjusting frequencies, levels and adding spacing between tracks.

  Then the master disk enters production. A recording is made onto lacquer disc
  which is very delicate. Often known as the "acetate". This is coated in metal to
  create the "metal master", after various processes this goes to the pressing
  plant who press vinyl copies from it.

  Each stage can introduce subtle variations, especially with re-pressings.
  Re-pressings require new stampers from the master which may have degraded in the
  interim. Plus there is a limit to the amount of pressings possible with a given
  stamper. The earlier pressings are therefore less degraded making them more
  valuable.

  ## Degradation

  With tape and vinyl each playback wears the medium, degrading the quality of the
  captured signal. With CDs, there is no degradation and every copy is a perfect
  copy of the original signal. However this capture, necessarily contains less
  information than the analogue version because of sampling.

raw_content:
  ---
  tags: [sound, analogue, physics]
  created: Thursday, January 02, 2025
  ---

  # Recording and playback of sound

  ## Analogue

  ### Recording

  The microphone converts sound waves into a continuous electrical signal.

  The signal drives the recording equipment.

  In the case of a vinyl recording: the signal drives a cutting head that carves a
  groove into the master disk.

  In the case of a tape recording: the signal drives electro-magnets that align
  magnetic particles on the tape.

  In both cases, **the recording medium physically mirrors the shape of the sound
  wave**.

  ### Playback

  In the case of vinyl, the needle follows the groove, moving up and down to
  recreate the electrical signal.

  In the case of tape, the magnetic patterns induce electrical current in the
  playback head.

  In each case, the signal is transferred directly to amplifiers and speakers.

  ## Digital

  ### Recording

  Mostly covered in [Binary encoding of sound](Binary_encoding_of_sound.md).

  An important point to note is that the binary encoding that results from the
  sampling is lasered into the CD as microscopic impressions and depressions (pits
  and lands)

  ### Playback

  A laser in the reader reads the pits and lands and converts them back to binary
  numbers. A DAC then converts the numbers back into a continuous electrical
  signal which is then channelled to the amplifiers and speakers.

  ## From recording to production of storage medium

  With analogue production recordings would be made onto tape. Multiple takes are
  recorded, mixed and edited. This results in a master tape which is the final
  version of the recorded sound.

  This then goes to a mastering engineer for mastering which creates the master
  mix. This consists in optimising the sound for the given format (vinyl or tape).
  Also includes adjusting frequencies, levels and adding spacing between tracks.

  Then the master disk enters production. A recording is made onto lacquer disc
  which is very delicate. Often known as the "acetate". This is coated in metal to
  create the "metal master", after various processes this goes to the pressing
  plant who press vinyl copies from it.

  Each stage can introduce subtle variations, especially with re-pressings.
  Re-pressings require new stampers from the master which may have degraded in the
  interim. Plus there is a limit to the amount of pressings possible with a given
  stamper. The earlier pressings are therefore less degraded making them more
  valuable.

  ## Degradation

  With tape and vinyl each playback wears the medium, degrading the quality of the
  captured signal. With CDs, there is no degradation and every copy is a perfect
  copy of the original signal. However this capture, necessarily contains less
  information than the analogue version because of sampling.


path: Monitor_DNS_resolution_and_internet_connectivity_script.md
sortable_path: Monitor_DNS_resolution_and_internet_connectivity_script.md
title: Monitor DNS resolution and internet connectivity script
checksum: 2783031db8f9eb0513eac607269b61e28142fb41b6b64f915b67878f8097e49b
created: 2025-01-04 13:27:13.572149706+00:00
modified: 2025-08-10 14:08:47.048881866+00:00
metadata: {"created":"Monday, January 13, 2025","tags":["networks","procedural"]}
lead:
  ```bash
   while true; do
      date
      echo "DNS check:"
      host google.com
      echo "Ping check:"
      ping -c 1 google.com | grep "time="
      echo "---"
      sleep 5
  done
  ```

body:
  ```bash
   while true; do
      date
      echo "DNS check:"
      host google.com
      echo "Ping check:"
      ping -c 1 google.com | grep "time="
      echo "---"
      sleep 5
  done
  ```

  See [ping](./Ping.md) for more details on what this is doing.

  The `host` command contacts a DNS server and does a reverse lookup for the
  IP-address of the human-readable domain name.

raw_content:
  ---
  tags: [networks, procedural]
  created: Monday, January 13, 2025
  ---

  # Monitor DNS resolution and internet connectivity script

  ```bash
   while true; do
      date
      echo "DNS check:"
      host google.com
      echo "Ping check:"
      ping -c 1 google.com | grep "time="
      echo "---"
      sleep 5
  done
  ```

  See [ping](./Ping.md) for more details on what this is doing.

  The `host` command contacts a DNS server and does a reverse lookup for the
  IP-address of the human-readable domain name.


path: View_IP_addresses.md
sortable_path: View_IP_addresses.md
title: View IP addresses
lead:
  Use `ip` package to view the [IP addresses](IP_addresses.md) associated with
  your current host:

checksum: eb10adeb1c2a55330524862e35dd667be3d04ac8a31d607e3bc2d3974587fcd1
created: 2025-01-10 16:51:50.329623342+00:00
modified: 2025-08-10 14:08:47.247542198+00:00
metadata: {"created":"Friday, January 10, 2025","tags":["networks","procedural","Linux"]}
body:
  Use `ip` package to view the [IP addresses](IP_addresses.md) associated with
  your current host:

  ```sh
  ip address show
  ```

  I am curently using ethernet on the computer I am writing this. This is listed
  as follows:

  ```
  enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  link/ether 90:2e:16:53:98:b9 brd ff:ff:ff:ff:ff:ffinet
  192.168.0.3/24 brd 192.168.0.255 scope global dynamic noprefixroute enp0s31f6
      valid_lft 83353sec preferred_lft 83353sec
  ```

  The most important line is the third. This indicates my IP address
  (`192.168.0.3/24`) including its subnet mask.

raw_content:
  ---
  tags: [networks, procedural, Linux]
  created: Friday, January 10, 2025
  ---

  # View IP addresses

  Use `ip` package to view the [IP addresses](IP_addresses.md) associated with
  your current host:

  ```sh
  ip address show
  ```

  I am curently using ethernet on the computer I am writing this. This is listed
  as follows:

  ```
  enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  link/ether 90:2e:16:53:98:b9 brd ff:ff:ff:ff:ff:ffinet
  192.168.0.3/24 brd 192.168.0.255 scope global dynamic noprefixroute enp0s31f6
      valid_lft 83353sec preferred_lft 83353sec
  ```

  The most important line is the third. This indicates my IP address
  (`192.168.0.3/24`) including its subnet mask.


path: Ping.md
sortable_path: Ping.md
title: Ping
checksum: 93c2e66315606664cbc568d8ba2309d8a5c0826627747e27188931b6314703bd
created: 2025-01-13 20:27:31.401736298+00:00
modified: 2025-08-10 14:08:47.276378959+00:00
metadata: {"created":"Monday, January 13, 2025","tags":["networks","procedural"]}
lead:
  `ping` is perhaps the most basic network debugging tool. It sends
  [ICMP](./ICMP_Protocol.md) echo request packets to a recipient host.

body:
  `ping` is perhaps the most basic network debugging tool. It sends
  [ICMP](./ICMP_Protocol.md) echo request packets to a recipient host.

  If the recipient receives the packet and is configured to reply, it sends an
  ICMP echo response packet in return.

  For example:

  ```
  $ ping google.com
  PING google.com (142.250.200.14) 56(84) bytes of data.
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=1 ttl=116 time=19.1 ms
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=2 ttl=116 time=19.2 ms
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=3 ttl=116 time=19.1 ms
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=4 ttl=116 time=21.1 ms
  ```

  The first indicates the IP address of the host you are pinging (once per second,
  by default). The subsequent lines are the response packets, echoed back from
  this host.

  The `icmp_seq` is the number of the request in the sequence. If there was to be
  a gap in the sequence, this would be the clearest indication of a connection
  problem.The packets should not arrive out of order since there is only one
  packet being sent per second.

  `time` is the round-trip time - from the sending of the packet to its return to
  the sender.

  If the packet cannot reach the destination host, the final router to see the
  packet retunes a "host unreachable" packet to `ping`.

raw_content:
  ---
  tags: [networks, procedural]
  created: Monday, January 13, 2025
  ---

  # Ping

  `ping` is perhaps the most basic network debugging tool. It sends
  [ICMP](./ICMP_Protocol.md) echo request packets to a recipient host.

  If the recipient receives the packet and is configured to reply, it sends an
  ICMP echo response packet in return.

  For example:

  ```
  $ ping google.com
  PING google.com (142.250.200.14) 56(84) bytes of data.
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=1 ttl=116 time=19.1 ms
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=2 ttl=116 time=19.2 ms
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=3 ttl=116 time=19.1 ms
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=4 ttl=116 time=21.1 ms
  ```

  The first indicates the IP address of the host you are pinging (once per second,
  by default). The subsequent lines are the response packets, echoed back from
  this host.

  The `icmp_seq` is the number of the request in the sequence. If there was to be
  a gap in the sequence, this would be the clearest indication of a connection
  problem.The packets should not arrive out of order since there is only one
  packet being sent per second.

  `time` is the round-trip time - from the sending of the packet to its return to
  the sender.

  If the packet cannot reach the destination host, the final router to see the
  packet retunes a "host unreachable" packet to `ping`.


path: ICMP_Protocol.md
sortable_path: ICMP_Protocol.md
title: ICMP Protocol
checksum: 81995cf8b107982fa597a658a7ae4ae2532b03dd8636940b6b4f9dd4da5d5693
created: 2025-01-13 20:53:19.536774731+00:00
modified: 2025-08-10 14:08:47.077148834+00:00
metadata: {"created":"Monday, January 13, 2025","tags":["networks"]}
lead:
  The Internet Control Message Protocol operates alongside the
  [Internet Protocol](Internet_Layer_of_Internet_Protocol.md) at the
  [Transport Layer](Transport_Layer_of_Internet_Protocol.md).

body:
  The Internet Control Message Protocol operates alongside the
  [Internet Protocol](Internet_Layer_of_Internet_Protocol.md) at the
  [Transport Layer](Transport_Layer_of_Internet_Protocol.md).

  It doesn't carry any actual user data and thus there is no
  [Application Layer](Application_Layer_of_Internet_Protocol.md) processes above
  it. It's usage is primarily for diagnostics and debugging. When network problems
  occur, ICMP generates messages to report errors back to the source.

  Probably the most familiar tools related to ICMP is [Ping](Ping.md).

raw_content:
  ---
  tags: [networks]
  created: Monday, January 13, 2025
  ---

  # ICMP Protocol

  The Internet Control Message Protocol operates alongside the
  [Internet Protocol](Internet_Layer_of_Internet_Protocol.md) at the
  [Transport Layer](Transport_Layer_of_Internet_Protocol.md).

  It doesn't carry any actual user data and thus there is no
  [Application Layer](Application_Layer_of_Internet_Protocol.md) processes above
  it. It's usage is primarily for diagnostics and debugging. When network problems
  occur, ICMP generates messages to report errors back to the source.

  Probably the most familiar tools related to ICMP is [Ping](Ping.md).


path: LineageOS_backup.md
sortable_path: LineageOS_backup.md
title: LineageOS major update procedure
lead: ## Backup
checksum: 8c8e95de8036787cb60edf83fb490b1b163818e6da4266b26ba8e7300305dcea
created: 2025-01-24 17:22:55.866259038+00:00
modified: 2025-08-10 14:08:47.144722088+00:00
metadata: {"created":"Friday, January 24, 2025","tags":["procedural","linux","android"]}
body:
  ## Backup

  - Enable developer mode by going to Settings > About Phone and tapping "Build
    Number" seven times

  - Enable USB debugging in Developer Options

  - Test by running `adb devices` in PC terminal

  - Then run `adb backup -apk -shared -all -system -f lineage_backup.ab`

  ## Download latest version of TWRP

  Make sure you get the right one for Google Pixel 2 (taimen), e.g:

  ```
  twrp-3.7.0_9-0-taimen.img
  ```

  ## Download ROM image and associated files

  Go to
  [https://wiki.lineageos.org/devices/taimen/](https://wiki.lineageos.org/devices/taimen/),
  download zip, viz.

  ```
  lineage-22.1-20250123-nightly-taimen-signed.zip
  ```

  ## Boot into fastboot

  Press Power + Vol Down

  Check connection with:

  ```
  fastboot devices
  ```

  ## Boot into TWRP

  ```
  fastboot boot  twrp-3.7.0_9-0-taimen.img
  ```

  ## Copy OS rom to Phone

  ```
  adb push lineage-22*.zip /sdcard/
  ```

  ## Install OS in TWRP

  In TWRP select Install > Select ROM zip

  Select flash

  Then reboot

  ```

  ```

raw_content:
  ---
  tags: [procedural, linux, android]
  created: Friday, January 24, 2025
  ---

  # LineageOS major update procedure

  ## Backup

  - Enable developer mode by going to Settings > About Phone and tapping "Build
    Number" seven times

  - Enable USB debugging in Developer Options

  - Test by running `adb devices` in PC terminal

  - Then run `adb backup -apk -shared -all -system -f lineage_backup.ab`

  ## Download latest version of TWRP

  Make sure you get the right one for Google Pixel 2 (taimen), e.g:

  ```
  twrp-3.7.0_9-0-taimen.img
  ```

  ## Download ROM image and associated files

  Go to
  [https://wiki.lineageos.org/devices/taimen/](https://wiki.lineageos.org/devices/taimen/),
  download zip, viz.

  ```
  lineage-22.1-20250123-nightly-taimen-signed.zip
  ```

  ## Boot into fastboot

  Press Power + Vol Down

  Check connection with:

  ```
  fastboot devices
  ```

  ## Boot into TWRP

  ```
  fastboot boot  twrp-3.7.0_9-0-taimen.img
  ```

  ## Copy OS rom to Phone

  ```
  adb push lineage-22*.zip /sdcard/
  ```

  ## Install OS in TWRP

  In TWRP select Install > Select ROM zip

  Select flash

  Then reboot

  ```

  ```


path: Proxies.md
sortable_path: Proxies.md
title: Proxies
checksum: c9e57817c83a63911f8dbce2a5d9d6b49013996b0a74b9880930cae767f789a7
created: 2025-02-03 12:33:32.607174396+00:00
modified: 2025-08-10 14:08:47.015535845+00:00
metadata: {"created":"Monday, February 03, 2025","tags":["networks","proxies"]}
lead:
  A standard proxy stands as a blocking layer between a client and a server. A
  client makes a request to a server but before it reaches it, it is intercepted
  by a proxy that reviews the request and decides whether to pass on to the
  requested location.

body:
  A standard proxy stands as a blocking layer between a client and a server. A
  client makes a request to a server but before it reaches it, it is intercepted
  by a proxy that reviews the request and decides whether to pass on to the
  requested location.

  As such, from the server's point of view, the request comes from the proxy not
  the client.

  An example application would be organisations who want to monitor or filter
  their employees web activities.

  A **reverse proxy** has the opposite orientation. It acts as buffer between a
  server and a client, filtering the requests that are sent to a server.

  This can also be for reasons of security, but most typically a reverse-proxy is
  used to manage requests to different services that a server is running.

  If a client requests `service-a.some-server/slug` the request is sent to the
  proxy operating at the root level, it siphons the request to its intended
  sub-service (`service-a`) and the port this service is running on.

  So, from the point of view of Service A, the request doesn't come from the
  external client but from the reverse proxy.

  In addition to directing traffic, reverse proxies can also implement load
  balancing: running procedures that ensure traffic is met at times of high
  capacity, ensuring that the server's resources are not exhausted.

raw_content:
  ---
  tags: [networks, proxies]
  created: Monday, February 03, 2025
  ---

  # Proxies

  A standard proxy stands as a blocking layer between a client and a server. A
  client makes a request to a server but before it reaches it, it is intercepted
  by a proxy that reviews the request and decides whether to pass on to the
  requested location.

  As such, from the server's point of view, the request comes from the proxy not
  the client.

  An example application would be organisations who want to monitor or filter
  their employees web activities.

  A **reverse proxy** has the opposite orientation. It acts as buffer between a
  server and a client, filtering the requests that are sent to a server.

  This can also be for reasons of security, but most typically a reverse-proxy is
  used to manage requests to different services that a server is running.

  If a client requests `service-a.some-server/slug` the request is sent to the
  proxy operating at the root level, it siphons the request to its intended
  sub-service (`service-a`) and the port this service is running on.

  So, from the point of view of Service A, the request doesn't come from the
  external client but from the reverse proxy.

  In addition to directing traffic, reverse proxies can also implement load
  balancing: running procedures that ensure traffic is met at times of high
  capacity, ensuring that the server's resources are not exhausted.


path: Certificate_authorities.md
sortable_path: Certificate_authorities.md
title: Certificates and certificate authorities
checksum: 77916e24993cca65e2475f8678807ce3ef38c5a17ae2943b3f06e7a979d2f9bd
created: 2025-02-09 12:18:13.284687081+00:00
modified: 2025-08-10 14:08:47.071568515+00:00
metadata: {"created":"Sunday, February 09, 2025","tags":["networks","servers"]}
lead:
  > HTTPS is a secure tunnel for data transmission, but the certificate is what
  > establishes trust and enables the secure tunnel in the first place.

body:
  > HTTPS is a secure tunnel for data transmission, but the certificate is what
  > establishes trust and enables the secure tunnel in the first place.

  Before secure communication over HTTPS can commence between a client and a
  server, the client must confirm that the server is who it says it is. This is in
  order to prevent man-in-the-middle attacks where malicious actors attempt to
  impersonate legitimate websites.

  Certificates exist to authenticate domains, prior to the exchange of encrypted
  data. To do this, a server generates a certificate with a reputable Certificate
  Authority, thereby verifying its identity with the CA. It sends this certificate
  (which includes its [public key](./bbdcb54f_public_key_cryptography.md)) when
  initiating a connection with a client.

  In addition to the public key, the certificate provides:

  - the domain name
  - the CA that certified the domain
  - the period of validity
  - a digital signature from the CA

  The client uses this data to validate the identity of the server it has sent a
  request to. Browsers come pre-installed with a list of trusted CAs.

  Certificates are called TLS/SSL certificates but the main protocol is TLS rather
  than SSL.

  Indications that a server is not to be trusted include:

  - expired certificates
  - certificates issued for the wrong domain
  - self-signed certificates not issued by a trusted CA

  ## Certification process

  Standard domain-validated authentication proceeds as follows:

  1. Server generates public/private key pair
  2. Server creates a certificate signing request containing its public key and
     domain name
  3. CA validates the domain and signs the certificate with its private key
  4. Server discloses its public key and CA signature
  5. Client knows CAs public key and uses this to verify the signature
  6. Server's public key is then trusted

  ### How validation executes between CA and server

  To validate the domain (step 3 above), the CA gives the server a "challenge" to
  prove they control the domain. This can include:

  - Putting a specific token as a DNS TXT record (DNS validation)
  - Putting a file at a certain URL (HTTP validation)
  - Email validation: sending an email to standard admin address for the server.
    Server owner clicks the link.

  ### Related

  See [Let's Encrypt](./Let's_Encrypt.md)

raw_content:
  ---
  tags: [networks, servers]
  created: Sunday, February 09, 2025
  ---

  # Certificates and certificate authorities

  > HTTPS is a secure tunnel for data transmission, but the certificate is what
  > establishes trust and enables the secure tunnel in the first place.

  Before secure communication over HTTPS can commence between a client and a
  server, the client must confirm that the server is who it says it is. This is in
  order to prevent man-in-the-middle attacks where malicious actors attempt to
  impersonate legitimate websites.

  Certificates exist to authenticate domains, prior to the exchange of encrypted
  data. To do this, a server generates a certificate with a reputable Certificate
  Authority, thereby verifying its identity with the CA. It sends this certificate
  (which includes its [public key](./bbdcb54f_public_key_cryptography.md)) when
  initiating a connection with a client.

  In addition to the public key, the certificate provides:

  - the domain name
  - the CA that certified the domain
  - the period of validity
  - a digital signature from the CA

  The client uses this data to validate the identity of the server it has sent a
  request to. Browsers come pre-installed with a list of trusted CAs.

  Certificates are called TLS/SSL certificates but the main protocol is TLS rather
  than SSL.

  Indications that a server is not to be trusted include:

  - expired certificates
  - certificates issued for the wrong domain
  - self-signed certificates not issued by a trusted CA

  ## Certification process

  Standard domain-validated authentication proceeds as follows:

  1. Server generates public/private key pair
  2. Server creates a certificate signing request containing its public key and
     domain name
  3. CA validates the domain and signs the certificate with its private key
  4. Server discloses its public key and CA signature
  5. Client knows CAs public key and uses this to verify the signature
  6. Server's public key is then trusted

  ### How validation executes between CA and server

  To validate the domain (step 3 above), the CA gives the server a "challenge" to
  prove they control the domain. This can include:

  - Putting a specific token as a DNS TXT record (DNS validation)
  - Putting a file at a certain URL (HTTP validation)
  - Email validation: sending an email to standard admin address for the server.
    Server owner clicks the link.

  ### Related

  See [Let's Encrypt](./Let's_Encrypt.md)


path: Firewalls.md
sortable_path: Firewalls.md
title: Firewalls
checksum: 223179f3ecf6b54694f9965cf4d922379173699550e035779384ef4197a1a1bf
created: 2025-02-09 12:53:38.517010875+00:00
modified: 2025-08-10 14:08:47.36037819+00:00
metadata: {"created":"Sunday, February 09, 2025","tags":["networks","servers","firewalls"]}
lead:
  A firewall is a server mechanism that filters incoming requests for resources
  and services that it hosts.

body:
  A firewall is a server mechanism that filters incoming requests for resources
  and services that it hosts.

  Based on data about the requester, derived from the
  [TLS](./Transport_Layer_of_Internet_Protocol.md) packet headers, the firewall
  decides whether or not to grant access to the requested resource.

  It will typically comprise three actions:

  - accept
  - drop
  - reject

  Accept is obvious. To "drop" means to block the traffic without sending a reply.
  To "reject" is to block the traffic but reply with an "unreachable" error.

  An example of a firewall rule would be to drop incoming traffic from IP
  addresses of non-technical employees to the SSH port (22) but allow technical
  staff to access it.

  Network traffic is matched agains a list of firewall rules in a sequence from
  first to last. As you cannot cover all cases, you must specify a default action
  when a rule is not matched.

  Whilst firewalls are focused on incoming traffic, you can also apply rules to
  outgoing traffic. A server will permit most outgoing traffic since, after all,
  it can trust itself however you may want to stop the server connecting to known
  malicious actors.

  ## Related software

  IPTables is the standard Linux software for managing firewalls. There is also
  Uncomplicated Firewall (UWF) which attaches to IPTables and simplifies the
  process.

  Another useful software is Fail2ban which can automatically configure your
  firewall to block brute force login attempts and DDOS attacks.

  ## Difference from reverse proxies

  Whilst both reverse proxies and firewalls filter incoming traffic to a server,
  there are important differences.

  The firewall operates at the
  [Transport Layer](./Transport_Layer_of_Internet_Protocol.md) wehreas proxies
  operate at the [Application Layer](./Application_Layer_of_Internet_Protocol.md).

  The firewall is primarily concerned with security and precedes anything that
  happens with the incoming packets at the Appplication Layer. The packets first
  have to get through the firewall before the request can be routed to a specific
  service on the server.

raw_content:
  ---
  tags: [networks, servers, firewalls]
  created: Sunday, February 09, 2025
  ---

  # Firewalls

  A firewall is a server mechanism that filters incoming requests for resources
  and services that it hosts.

  Based on data about the requester, derived from the
  [TLS](./Transport_Layer_of_Internet_Protocol.md) packet headers, the firewall
  decides whether or not to grant access to the requested resource.

  It will typically comprise three actions:

  - accept
  - drop
  - reject

  Accept is obvious. To "drop" means to block the traffic without sending a reply.
  To "reject" is to block the traffic but reply with an "unreachable" error.

  An example of a firewall rule would be to drop incoming traffic from IP
  addresses of non-technical employees to the SSH port (22) but allow technical
  staff to access it.

  Network traffic is matched agains a list of firewall rules in a sequence from
  first to last. As you cannot cover all cases, you must specify a default action
  when a rule is not matched.

  Whilst firewalls are focused on incoming traffic, you can also apply rules to
  outgoing traffic. A server will permit most outgoing traffic since, after all,
  it can trust itself however you may want to stop the server connecting to known
  malicious actors.

  ## Related software

  IPTables is the standard Linux software for managing firewalls. There is also
  Uncomplicated Firewall (UWF) which attaches to IPTables and simplifies the
  process.

  Another useful software is Fail2ban which can automatically configure your
  firewall to block brute force login attempts and DDOS attacks.

  ## Difference from reverse proxies

  Whilst both reverse proxies and firewalls filter incoming traffic to a server,
  there are important differences.

  The firewall operates at the
  [Transport Layer](./Transport_Layer_of_Internet_Protocol.md) wehreas proxies
  operate at the [Application Layer](./Application_Layer_of_Internet_Protocol.md).

  The firewall is primarily concerned with security and precedes anything that
  happens with the incoming packets at the Appplication Layer. The packets first
  have to get through the firewall before the request can be routed to a specific
  service on the server.


path: Let's_Encrypt.md
sortable_path: Let's_Encrypt.md
title: Let's Encrypt
checksum: 6dcd26eac7dd2f775254559524dd0f7d528405b8ebc1f87a8db080c3f34905ed
created: 2025-02-09 13:31:00.139717208+00:00
modified: 2025-08-10 14:08:47.170117767+00:00
metadata: {"created":"Sunday, February 09, 2025","tags":["networks","servers","encryption"]}
lead:
  ACME (Automatic Certificate Management Environment) is an open source protocol
  that automates the process of obtaining and renewing SSL/TLS certificates.

body:
  ACME (Automatic Certificate Management Environment) is an open source protocol
  that automates the process of obtaining and renewing SSL/TLS certificates.

  Let's Encrypt (LE) created this protocol and uses it to issue certificates.

  Certbot is a popular ACME client that communicates with the LE servers to

  - validate domain ownership
  - request certificates
  - install the certificates on the web server
  - handle automatic renewals

  Let's Encrypt was founded by the
  [Internet Security Research Group](https://www.abetterinternet.org/). Certbot
  was created independently by the EFF and is maintained by them.

raw_content:
  ---
  tags: [networks, servers, encryption]
  created: Sunday, February 09, 2025
  ---

  # Let's Encrypt

  ACME (Automatic Certificate Management Environment) is an open source protocol
  that automates the process of obtaining and renewing SSL/TLS certificates.

  Let's Encrypt (LE) created this protocol and uses it to issue certificates.

  Certbot is a popular ACME client that communicates with the LE servers to

  - validate domain ownership
  - request certificates
  - install the certificates on the web server
  - handle automatic renewals

  Let's Encrypt was founded by the
  [Internet Security Research Group](https://www.abetterinternet.org/). Certbot
  was created independently by the EFF and is maintained by them.


path: Setup encrypted harddrive.md
sortable_path: Setup encrypted harddrive.md
title: Setup an encrypted harddrive
checksum: 6eb802bbca6795f4feca4f0acf11b34b5e2cbe944e9c6e6ec815dfbfafdbd0ea
created: 2025-02-10 19:26:51.956385272+00:00
modified: 2025-02-11 18:11:41.796385106+00:00
metadata: {"created":"Monday, February 10, 2025","tags":["procedural","disks","encryption"]}
lead:
  Firstly [create a new partition](./Creating_a_Linux_partition_table.md) on the
  unencrypted drive with `fdisk`.

body:
  Firstly [create a new partition](./Creating_a_Linux_partition_table.md) on the
  unencrypted drive with `fdisk`.

  Create LUKs encryption:

  ```sh
  sudo cryptsetup luksFormat /dev/sdX
  ```

  Enter password when prompted.

  Open the encrypted drive with the name you want to use henceforth:

  ```sh
  sudo cryptsetup luksOpen /dev/sdX my_device_name
  ```

  Create filesystem:

  ```sh
  sudo mkfs.ext4 /dev/mapper/my_device_name
  ```

  Set mount point:

  ```sh
  sudo mkdir -p /media/my_device_name
  ```

  Mount:

  ```sh
  sudo mount /dev/mapper/my_device_name /media/my_device_name
  ```

  Add it to the crypttab:

  ```sh
  vim /etc/crypttab
  ```

  Get UUID for device with:

  ```
  sudo blkid /dev/sdx
  ```

  Add entry:

  ```txt
  my_device_name UUID="ae028027-20c5-40b6-9311-11f5a886f125" none noauto
  ```

  Then add it as a lint in the fstab for auto-mount:

  ```sh
  /dev/mapper/my_device_name /media/my_device_name ext4 defaults,nofail,noauto,user 0 2
  ```

  Give yourself full permissions (set once, then permanent):

  ```sh
  sudo chown thomas:thomas /meda/my_device_name
  sudo chmod 755 /media/my_device_name
  ```

  Make sure it shows up with its intended name:

  ```sh
  sudo e2label /dev/mapper/samsung-T3 samsung-T3
  ```

  (Where second value is the name we want it to show as.)

  Test auto-mount and loading from crypttab:

  First reload `systemd`:

  ```sh
  sudo systemctl daemon-reload
  ```

  Then, remount all disks.

  ```sh
  sudo mount -a
  ```

  Should be prompted for password and disk decrypts and mounts to specified mount
  point.

  ## Additional points

  Close LUKS (if you make a mistake):

  ```sh
  sudo cryptsetup luksClose device_name

  ```

  Remove vendor data to unlock more space:

  ```sh
  sudo tune2fs -m 0 /dev/mapper/samsung-T3
  ```

raw_content:
  ---
  tags: [procedural, disks, encryption]
  created: Monday, February 10, 2025
  ---

  # Setup an encrypted harddrive

  Firstly [create a new partition](./Creating_a_Linux_partition_table.md) on the
  unencrypted drive with `fdisk`.

  Create LUKs encryption:

  ```sh
  sudo cryptsetup luksFormat /dev/sdX
  ```

  Enter password when prompted.

  Open the encrypted drive with the name you want to use henceforth:

  ```sh
  sudo cryptsetup luksOpen /dev/sdX my_device_name
  ```

  Create filesystem:

  ```sh
  sudo mkfs.ext4 /dev/mapper/my_device_name
  ```

  Set mount point:

  ```sh
  sudo mkdir -p /media/my_device_name
  ```

  Mount:

  ```sh
  sudo mount /dev/mapper/my_device_name /media/my_device_name
  ```

  Add it to the crypttab:

  ```sh
  vim /etc/crypttab
  ```

  Get UUID for device with:

  ```
  sudo blkid /dev/sdx
  ```

  Add entry:

  ```txt
  my_device_name UUID="ae028027-20c5-40b6-9311-11f5a886f125" none noauto
  ```

  Then add it as a lint in the fstab for auto-mount:

  ```sh
  /dev/mapper/my_device_name /media/my_device_name ext4 defaults,nofail,noauto,user 0 2
  ```

  Give yourself full permissions (set once, then permanent):

  ```sh
  sudo chown thomas:thomas /meda/my_device_name
  sudo chmod 755 /media/my_device_name
  ```

  Make sure it shows up with its intended name:

  ```sh
  sudo e2label /dev/mapper/samsung-T3 samsung-T3
  ```

  (Where second value is the name we want it to show as.)

  Test auto-mount and loading from crypttab:

  First reload `systemd`:

  ```sh
  sudo systemctl daemon-reload
  ```

  Then, remount all disks.

  ```sh
  sudo mount -a
  ```

  Should be prompted for password and disk decrypts and mounts to specified mount
  point.

  ## Additional points

  Close LUKS (if you make a mistake):

  ```sh
  sudo cryptsetup luksClose device_name

  ```

  Remove vendor data to unlock more space:

  ```sh
  sudo tune2fs -m 0 /dev/mapper/samsung-T3
  ```


path: Disk_size_utilities.md
sortable_path: Disk_size_utilities.md
title: Disk size utilities
lead: ## Compare size of two directories in human-readable format
checksum: 6f798b9d6fa7cf7504b641f2898b711d22c1af378d7456371ba998871c3a408f
created: 2025-02-10 19:43:13.645798441+00:00
modified: 2025-08-10 14:08:47.070972921+00:00
metadata: {"created":"Monday, February 10, 2025","tags":["procedural","linux"]}
body:
  ## Compare size of two directories in human-readable format

  ```sh
  du -sh /dir_one /dir_two
  ```

  ## View total capacity and space used

  ```sh
  df -h /media/samsung-T7 # add more to compare if you like

  # Filesystem              Size  Used Avail Use% Mounted on
  # /dev/mapper/samsung-T7  916G   67G  850G   8% /media/samsung-T7

  ```

raw_content:
  ---
  tags: [procedural, linux]
  created: Monday, February 10, 2025
  ---

  # Disk size utilities

  ## Compare size of two directories in human-readable format

  ```sh
  du -sh /dir_one /dir_two
  ```

  ## View total capacity and space used

  ```sh
  df -h /media/samsung-T7 # add more to compare if you like

  # Filesystem              Size  Used Avail Use% Mounted on
  # /dev/mapper/samsung-T7  916G   67G  850G   8% /media/samsung-T7

  ```


path: Disable_non-root_ssh_access.md
sortable_path: Disable_non-root_ssh_access.md
title: Disable non-root SSH access
lead: Best practice is to:
checksum: 11bba760e6420a2b2ce3e862cc192eafda4190653cfaada60968c17b4753cad6
created: 2025-02-15 18:23:30.822342517+00:00
modified: 2025-08-10 14:08:47.253960076+00:00
metadata: {"created":"Saturday, February 15, 2025","tags":["servers","ssh","procedural","linux"]}
body:
  Best practice is to:

  - Disallow root login
  - Block password-based login
  - Allow only SSH-based login

  Do this by editing `/etc/ssh/sshd_config`:

  ```
  PasswordAuthentication no
  ChallengeResponseAuthentication no
  ```

  Then restart the SSH service:

  ```
  sudo systemctl restart sshd
  ```

  (Assumes you have already set-up SSH-based login for a non-root user!)

raw_content:
  ---
  tags: [servers, ssh, procedural, linux]
  created: Saturday, February 15, 2025
  ---

  # Disable non-root SSH access

  Best practice is to:

  - Disallow root login
  - Block password-based login
  - Allow only SSH-based login

  Do this by editing `/etc/ssh/sshd_config`:

  ```
  PasswordAuthentication no
  ChallengeResponseAuthentication no
  ```

  Then restart the SSH service:

  ```
  sudo systemctl restart sshd
  ```

  (Assumes you have already set-up SSH-based login for a non-root user!)


path: UFW_firewall_management.md
sortable_path: UFW_firewall_management.md
title: UFW firewall management
checksum: 6d9fc7c6c7fd064004870b60232cb4190c34bc94e8eea3cd855cb02dba64091d
created: 2025-02-16 15:00:32.757249884+00:00
modified: 2025-08-10 14:08:47.097161616+00:00
metadata: {"created":"Sunday, February 16, 2025","tags":["servers","procedural","linux","firewalls"]}
lead:
  `ufw` (Uncomplicated Firewall) simplifies the process of setting up firewalls on
  your server, without directly using `iptables`.

body:
  `ufw` (Uncomplicated Firewall) simplifies the process of setting up firewalls on
  your server, without directly using `iptables`.

  ```sh
  sudo apt install ufw
  ```

  First we deny all incoming requests as the default:

  ```sh
  ufw default deny incoming
  ```

  And allow all outgoing requests as the default:

  ```sh
  ufw default allow outgoing
  ```

  Next we need to allow certain requests based on their port:

  | Port | Service | Rule  |
  | ---- | ------- | ----- |
  | 22   | SSH     | allow |
  | 80   | HTTP    | allow |
  | 443  | HTTPS   | allow |

  ```sh
  sudo ufw allow 22/tcp
  sudo ufw allow 80/tcp
  sudo ufw allow 443/tcp
  ```

  Enable:

  ```sh
  ufw enable
  ```

  Verify rules:

  ```sh
  ufw status --verbose
  ```

  ```

  ```

raw_content:
  ---
  tags: [servers, procedural, linux, firewalls]
  created: Sunday, February 16, 2025
  ---

  # UFW firewall management

  `ufw` (Uncomplicated Firewall) simplifies the process of setting up firewalls on
  your server, without directly using `iptables`.

  ```sh
  sudo apt install ufw
  ```

  First we deny all incoming requests as the default:

  ```sh
  ufw default deny incoming
  ```

  And allow all outgoing requests as the default:

  ```sh
  ufw default allow outgoing
  ```

  Next we need to allow certain requests based on their port:

  | Port | Service | Rule  |
  | ---- | ------- | ----- |
  | 22   | SSH     | allow |
  | 80   | HTTP    | allow |
  | 443  | HTTPS   | allow |

  ```sh
  sudo ufw allow 22/tcp
  sudo ufw allow 80/tcp
  sudo ufw allow 443/tcp
  ```

  Enable:

  ```sh
  ufw enable
  ```

  Verify rules:

  ```sh
  ufw status --verbose
  ```

  ```

  ```


path: Check_if_value_null_in_Bash.md
sortable_path: Check_if_value_null_in_Bash.md
title: Check if value is null in Bash
lead:
  ```sh
  if [ -z $var ]; then
  # Value is null
  else
  # Value is not null
  fi
  ```

body:
  ```sh
  if [ -z $var ]; then
  # Value is null
  else
  # Value is not null
  fi
  ```

checksum: 5d91b8e1ee3fb3879a85ef3ec1f47b3e2bf7784b557dcb8e994f208c5765bc39
created: 2025-02-19 17:39:39.608947732+00:00
modified: 2025-08-10 14:08:47.352764356+00:00
metadata: {"created":"Wednesday, February 19, 2025","tags":["shell"]}
raw_content:
  ---
  tags: [shell]
  created: Wednesday, February 19, 2025
  ---

  # Check if value is null in Bash

  ```sh
  if [ -z $var ]; then
    # Value is null
  else
    # Value is not null
  fi
  ```


path: Associative_arrays_maps_in_Bash.md
sortable_path: Associative_arrays_maps_in_Bash.md
title: Associative arrays (lookup maps) in Bash
lead: ```sh
checksum: 7b90f376c8a02db6dd5db010baf77dc623396a94e0991d0006daa4b6c294cf33
created: 2025-02-19 17:55:15.069028167+00:00
modified: 2025-08-10 14:08:47.315767459+00:00
metadata: {"created":"Wednesday, February 19, 2025","tags":["shell"]}
body:
  ```sh

  declare -A CHANNEL_TO_WEBHOOK # Declare the map type

  # Specify key values
  CHANNEL_TO_WEBHOOK["test"]=$SLACK_WEBHOOK_TEST
  CHANNEL_TO_WEBHOOK["backups"]=$SLACK_WEBHOOK_BACKUPS
  CHANNEL_TO_WEBHOOK["eolas"]=$SLACK_WEBHOOK_EOLAS
  CHANNEL_TO_WEBHOOK["website"]=$SLACK_WEBHOOK_SYSTEMS_OBSCURE
  CHANNEL_TO_WEBHOOK["time-tracking"]=$SLACK_WEBHOOK_TIME_TRACKING

  # Invoke

  WEBHOOK=${CHANNEL_TO_WEBHOOK[$CHANNEL]}

  ```

raw_content:
  ---
  tags: [shell]
  created: Wednesday, February 19, 2025
  ---

  # Associative arrays (lookup maps) in Bash

  ```sh

  declare -A CHANNEL_TO_WEBHOOK # Declare the map type

  # Specify key values
  CHANNEL_TO_WEBHOOK["test"]=$SLACK_WEBHOOK_TEST
  CHANNEL_TO_WEBHOOK["backups"]=$SLACK_WEBHOOK_BACKUPS
  CHANNEL_TO_WEBHOOK["eolas"]=$SLACK_WEBHOOK_EOLAS
  CHANNEL_TO_WEBHOOK["website"]=$SLACK_WEBHOOK_SYSTEMS_OBSCURE
  CHANNEL_TO_WEBHOOK["time-tracking"]=$SLACK_WEBHOOK_TIME_TRACKING

  # Invoke

  WEBHOOK=${CHANNEL_TO_WEBHOOK[$CHANNEL]}

  ```


path: Capturing_program_error_in_Bash.md
sortable_path: Capturing_program_error_in_Bash.md
title: Capturing program error in Bash
lead:
  If you want to capture possible error immediately after running the command, use
  `!`:

checksum: 11fc1b98c4c7d34e395abe64e45451713fb88170480ee53fa1b40977e7f327d7
created: 2025-02-22 09:59:35.743392315+00:00
modified: 2025-08-10 14:08:47.032186574+00:00
metadata: {"created":"Saturday, February 22, 2025","tags":["shell"]}
body:
  If you want to capture possible error immediately after running the command, use
  `!`:

  ```sh
  if ! grep "pattern" file.txt; then
      echo "Pattern not found"
  fi
  ```

  This just checks the exit code. If you want the actual error too:

  ```sh
  if ! ERROR=$(grep "pattern" file.txt 2>&1); then
      echo "Pattern not found: $ERROR"
  fi
  ```

  If you want to store the outcome of a command and refer to it later, use `$?`.
  This built-in variable captures the exit status of the last executed command (0
  for success, non-zero for failure):

  ```sh
  ls -a
  status=$?  # Store immediately if needed later
  if [ $status -ne 0 ]; then
      # handle the error case
  else
      # command was successful (exit code = 0)
  fi
  ```

  Note: `$?` will be overwritten by any command that runs after the one you're
  interested in. If you need to check multiple commands' statuses, store $?
  immediately after each command:

  ```sh
  command1
  status1=$?
  command2
  status2=$?

  if [ $status1 -ne 0 ] || [ $status2 -ne 0 ]; then
      echo "One of the commands failed"
  fi
  ```

  Again, this just checks the exit code, to capture the error:

  ```sh
  error_msg=$(command 2>&1)
  status=$?
  if [ $status -ne 0 ]; then
      echo "Failed: $error_msg"
  fi
  ```

raw_content:
  ---
  tags: [shell]
  created: Saturday, February 22, 2025
  ---

  # Capturing program error in Bash

  If you want to capture possible error immediately after running the command, use
  `!`:

  ```sh
  if ! grep "pattern" file.txt; then
      echo "Pattern not found"
  fi
  ```

  This just checks the exit code. If you want the actual error too:

  ```sh
  if ! ERROR=$(grep "pattern" file.txt 2>&1); then
      echo "Pattern not found: $ERROR"
  fi
  ```

  If you want to store the outcome of a command and refer to it later, use `$?`.
  This built-in variable captures the exit status of the last executed command (0
  for success, non-zero for failure):

  ```sh
  ls -a
  status=$?  # Store immediately if needed later
  if [ $status -ne 0 ]; then
      # handle the error case
  else
      # command was successful (exit code = 0)
  fi
  ```

  Note: `$?` will be overwritten by any command that runs after the one you're
  interested in. If you need to check multiple commands' statuses, store $?
  immediately after each command:

  ```sh
  command1
  status1=$?
  command2
  status2=$?

  if [ $status1 -ne 0 ] || [ $status2 -ne 0 ]; then
      echo "One of the commands failed"
  fi
  ```

  Again, this just checks the exit code, to capture the error:

  ```sh
  error_msg=$(command 2>&1)
  status=$?
  if [ $status -ne 0 ]; then
      echo "Failed: $error_msg"
  fi
  ```


path: Network_tunnels.md
sortable_path: Network_tunnels.md
title: Network tunnels
checksum: 945df3a9470ef7167623f087f72fcd6c9a5ff8b17af60a89eafb3ff6211018bd
created: 2025-02-24 17:17:29.826117441+00:00
modified: 2025-08-10 14:08:47.236278835+00:00
metadata: {"created":"Monday, February 24, 2025","tags":["networks","VPN"]}
lead:
  A network tunnel is a
  [Transport Layer](./Transport_Layer_of_Internet_Protocol.md) mechanism whereby
  data packets can travel across a network they would otherwise not be able to
  access. Similarly how an underground physical tunnel can be use to travel across
  an inaccessible geographic region.

body:
  A network tunnel is a
  [Transport Layer](./Transport_Layer_of_Internet_Protocol.md) mechanism whereby
  data packets can travel across a network they would otherwise not be able to
  access. Similarly how an underground physical tunnel can be use to travel across
  an inaccessible geographic region.

  It is a broad concept of which there are various specific instances. However
  many involve creating a tunnel by wrapping a packet that cannot traverse a given
  network within the payload of a packet that can.

  For instance, an organisation may use [IPv6](./IP_addresses.md) addresses but
  want to send them over an IPv4 network. To achieve this they would enclose the
  packets within IPv4 packets.

  Similarly the scenario of a VPN. A user with a UK IP address wants to access a
  server in the US that has a firewall blocking non-US IPs. The user can bypass
  this restriction by connecting to a VPN running US servers. The UK packets are
  transferred under the aegis of the VPN's US IP.

  For more detail see
  [How tunnelling works with VPNs](./How_tunneling_works_with_VPNs.md).

raw_content:
  ---
  tags: [networks, VPN]
  created: Monday, February 24, 2025
  ---

  # Network tunnels

  A network tunnel is a
  [Transport Layer](./Transport_Layer_of_Internet_Protocol.md) mechanism whereby
  data packets can travel across a network they would otherwise not be able to
  access. Similarly how an underground physical tunnel can be use to travel across
  an inaccessible geographic region.

  It is a broad concept of which there are various specific instances. However
  many involve creating a tunnel by wrapping a packet that cannot traverse a given
  network within the payload of a packet that can.

  For instance, an organisation may use [IPv6](./IP_addresses.md) addresses but
  want to send them over an IPv4 network. To achieve this they would enclose the
  packets within IPv4 packets.

  Similarly the scenario of a VPN. A user with a UK IP address wants to access a
  server in the US that has a firewall blocking non-US IPs. The user can bypass
  this restriction by connecting to a VPN running US servers. The UK packets are
  transferred under the aegis of the VPN's US IP.

  For more detail see
  [How tunnelling works with VPNs](./How_tunneling_works_with_VPNs.md).


path: How_tunneling_works_with_VPNs.md
sortable_path: How_tunneling_works_with_VPNs.md
title: How tunnelling works with VPNs
checksum: 8edbd757700fba99b1345a687410ffe2df63fc3afd0e95052c89417e1feae7cd
created: 2025-02-24 17:46:00.628294938+00:00
modified: 2025-08-10 14:08:47.306949326+00:00
metadata: {"created":"Monday, February 24, 2025","tags":["networks","VPN"]}
lead:
  A VPN is a good example of a [network tunnel](./Network_tunnels.md) and the
  technique of packet wrapping / encapsulation to bypass network restrictions.

body:
  A VPN is a good example of a [network tunnel](./Network_tunnels.md) and the
  technique of packet wrapping / encapsulation to bypass network restrictions.

  When a user connects to a VPN, their device creates a connection to the VPN
  server. All subsequent traffic is routed first through the tunnel, rather than
  directly to the target host IP.

  When the data arrives at the VPN server, the VPN wraps the packets inside new
  packets that have the VPN server's addressing information. The nested packets
  are encrypted before they are encapsulated and the tunnel they arrive through is
  also encrypted end-to-end. Thus there is - in theory - no way to know where the
  original packets come from.

  When the data returns from the target host to the VPN server that requested it,
  the VPN server decrypts the data and forwards it back through the secure tunnel
  to the original user's device.

  Whilst connected to the VPN, the user's IP appears to be identical to the IP of
  the VPN server. This is because the outer packet header has the VPN's addressing
  information, not the source IP's (this is encrypted within the payload). As the
  target host only sees the VPN's headers, it appears as if the VPN user's IP is
  the same as the VPN's.

raw_content:
  ---
  tags: [networks, VPN]
  created: Monday, February 24, 2025
  ---

  # How tunnelling works with VPNs

  A VPN is a good example of a [network tunnel](./Network_tunnels.md) and the
  technique of packet wrapping / encapsulation to bypass network restrictions.

  When a user connects to a VPN, their device creates a connection to the VPN
  server. All subsequent traffic is routed first through the tunnel, rather than
  directly to the target host IP.

  When the data arrives at the VPN server, the VPN wraps the packets inside new
  packets that have the VPN server's addressing information. The nested packets
  are encrypted before they are encapsulated and the tunnel they arrive through is
  also encrypted end-to-end. Thus there is - in theory - no way to know where the
  original packets come from.

  When the data returns from the target host to the VPN server that requested it,
  the VPN server decrypts the data and forwards it back through the secure tunnel
  to the original user's device.

  Whilst connected to the VPN, the user's IP appears to be identical to the IP of
  the VPN server. This is because the outer packet header has the VPN's addressing
  information, not the source IP's (this is encrypted within the payload). As the
  target host only sees the VPN's headers, it appears as if the VPN user's IP is
  the same as the VPN's.


path: lsof.md
sortable_path: lsof.md
title: lsof
lead:
  `lsof` stands for _list open files_. It lists opened files and the processes
  using them.

checksum: da5c29f66229b34260b9bfbe95bdfc9821e7fac0f246e526f512e6660ca925cb
created: 2025-03-04 17:44:19.457824631+00:00
modified: 2025-08-10 14:08:47.277840678+00:00
metadata: {"created":"Tuesday, March 04, 2025","tags":["shell","procedural"]}
body:
  `lsof` stands for _list open files_. It lists opened files and the processes
  using them.

  ## Directory

  ```sh
  lsof /home/thomas
  ```

  ```
  COMMAND    PID   USER  FD   TYPE DEVICE SIZE/OFF    NODE NAME
  pipewire  1025 thomas cwd    DIR  253,0     4096 7340035 /home/thomas
  wireplumb 1026 thomas cwd    DIR  253,0     4096 7340035 /home/thomas
  ```

  ## Pass in a pid

  ```sh
  lsof -p 1175
  ```

  ```
  COMMAND    PID   USER  FD      TYPE             DEVICE SIZE/OFF    NODE NAME
  at-spi-bu 1175 thomas cwd       DIR              253,0     4096 7340035 /home/thomas
  at-spi-bu 1175 thomas rtd       DIR              259,2     4096       2 /
  at-spi-bu 1175 thomas txt       REG              259,2    26760 1225154 /usr/lib/at-spi-bus-launcher
  ```

  ## Networking

  ### List programs using or listening to ports

  ```sh
  lsof -i
  ```

  ```
  COMMAND    PID   USER  FD   TYPE DEVICE SIZE/OFF NODE NAME
  signal-de 3250 thomas  39u  IPv4 149863      0t0  TCP archlinux:44304->ac88393aca5853df7.awsglobalaccelerator.com:https (ESTABLISHED)
  slack     3602 thomas  18u  IPv4 138442      0t0  TCP archlinux:57950->ec2-18-169-55-40.eu-west-2.compute.amazonaws.com:https (ESTABLISHED)
  firefox   6177 thomas 147u  IPv4 138842      0t0  TCP archlinux:cresco-control->93.243.107.34.bc.googleusercontent.com:https (ESTABLISHED)
  ```

  ### Filter by specific port

  ```sh
  lsof -i:22
  lsof -i4:22 # limit to IPv4 addresses

  ```

  ```
  COMMAND    PID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
  sshd     45448   root    3u  IPv4  640237      0t0  TCP *:ssh (LISTEN)
  sshd     45448   root    4u  IPv6  640239      0t0  TCP *:ssh (LISTEN)
  sshd    185660   root    4u  IPv4 2554538      0t0  TCP static.18.113.203.116.clients.your-server.de:ssh->0542cb12.skybroadband.com:57744 (ESTABLISHED)

  ```

raw_content:
  ---
  tags: [shell, procedural]
  created: Tuesday, March 04, 2025
  ---

  # lsof

  `lsof` stands for _list open files_. It lists opened files and the processes
  using them.

  ## Directory

  ```sh
  lsof /home/thomas
  ```

  ```
  COMMAND    PID   USER  FD   TYPE DEVICE SIZE/OFF    NODE NAME
  pipewire  1025 thomas cwd    DIR  253,0     4096 7340035 /home/thomas
  wireplumb 1026 thomas cwd    DIR  253,0     4096 7340035 /home/thomas
  ```

  ## Pass in a pid

  ```sh
  lsof -p 1175
  ```

  ```
  COMMAND    PID   USER  FD      TYPE             DEVICE SIZE/OFF    NODE NAME
  at-spi-bu 1175 thomas cwd       DIR              253,0     4096 7340035 /home/thomas
  at-spi-bu 1175 thomas rtd       DIR              259,2     4096       2 /
  at-spi-bu 1175 thomas txt       REG              259,2    26760 1225154 /usr/lib/at-spi-bus-launcher
  ```

  ## Networking

  ### List programs using or listening to ports

  ```sh
  lsof -i
  ```

  ```
  COMMAND    PID   USER  FD   TYPE DEVICE SIZE/OFF NODE NAME
  signal-de 3250 thomas  39u  IPv4 149863      0t0  TCP archlinux:44304->ac88393aca5853df7.awsglobalaccelerator.com:https (ESTABLISHED)
  slack     3602 thomas  18u  IPv4 138442      0t0  TCP archlinux:57950->ec2-18-169-55-40.eu-west-2.compute.amazonaws.com:https (ESTABLISHED)
  firefox   6177 thomas 147u  IPv4 138842      0t0  TCP archlinux:cresco-control->93.243.107.34.bc.googleusercontent.com:https (ESTABLISHED)
  ```

  ### Filter by specific port

  ```sh
  lsof -i:22
  lsof -i4:22 # limit to IPv4 addresses

  ```

  ```
  COMMAND    PID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
  sshd     45448   root    3u  IPv4  640237      0t0  TCP *:ssh (LISTEN)
  sshd     45448   root    4u  IPv6  640239      0t0  TCP *:ssh (LISTEN)
  sshd    185660   root    4u  IPv4 2554538      0t0  TCP static.18.113.203.116.clients.your-server.de:ssh->0542cb12.skybroadband.com:57744 (ESTABLISHED)

  ```


path: SSH.md
sortable_path: SSH.md
title: SSH
lead: SSH is the de facto standard for remote access to a Unix machine.
checksum: 496db00e65369a5bc34427698d3d94edaa4f5c38d636031a6c1061485d8a8629
created: 2025-03-04 18:29:13.410802879+00:00
modified: 2025-08-10 14:08:47.088464634+00:00
metadata: {"created":"Tuesday, March 04, 2025","tags":["encryption","shell","servers"]}
body:
  SSH is the de facto standard for remote access to a Unix machine.

  `ssh` is the client which you use to connect to another machine.

  `sshd` is the server that manages incoming client requests for access.

  ## sshd

  Typically the SSH server will be turned off.

  To run at boot:

  ```sh
  sudo systemctl enable sshd
  ```

  To start immediately:

  ```sh
  sudo systemctl start sshd
  ```

  The `sshd` configuration is found in the directory `/etc/ssh`.

  The config file is `/etc/ssh/sshd_config`.

  Mostly you can leave this alone but the following is a useful property to set:

  ```
  PermitRootLogin no
  ```

  See [Disable non-root ssh access](./Disable_non-root_ssh_access.md) for more.

  ## Known hosts

  Within your home directory at `./ssh/known_hosts` you will find a record of all
  the public keys of the servers you have connected. This file exists for both
  servers and clients, e.g:

  ```sh
  cat ./ssh/known_hosts
  # systemsobscure.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKn6vyv9b+Nk5r
  # YVSAk5KvsTiC24K6uSpzCHzgLNoqt2
  ```

  This shows the public key of my server at `systemsobscure.net` along with
  others.

  ## Authorized keys

  On servers only, there is also an `authorized_keys` file which shows the
  server's own public keys that it presents to clients.

  If I go to my server I see that this key matches the one I have on my client
  computer `known_hosts`:

  ```sh
  sudo cat /etc/ssh/ssh_host_ed25519_key.pub
  # ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKn6vyv9b+Nk5rYVSAk5KvsTiC24K6uSpzCHzgLNoqt2 root@self-host-server

  ```

  Each user on a server will also have a `~/.ssh` directory also containing an
  `authorized_keys` file. This contains the public keys of clients who are allowed
  to connect to that user account.

  Hence I see the same public key of my desktop client machine in both places.

  ## How the tunnel is created

  ## Generating a key pair

  ```sh
  ssh-keygen
  ```

  ## Add a new public key to a server so that it can be accessed from client

  ```sh
  # On server

  vim .ssh/authorized_keys

  ```

  Then add public key of client

  Then, if you typically connect with the default SSH key, you need to specify the
  specific key when connecting:

  ```sh
  ssh -o "IdentitiesOnly=yes" -i ~/.ssh/public_key_file user@server
  ```

raw_content:
  ---
  tags: [encryption, shell, servers]
  created: Tuesday, March 04, 2025
  ---

  # SSH

  SSH is the de facto standard for remote access to a Unix machine.

  `ssh` is the client which you use to connect to another machine.

  `sshd` is the server that manages incoming client requests for access.

  ## sshd

  Typically the SSH server will be turned off.

  To run at boot:

  ```sh
  sudo systemctl enable sshd
  ```

  To start immediately:

  ```sh
  sudo systemctl start sshd
  ```

  The `sshd` configuration is found in the directory `/etc/ssh`.

  The config file is `/etc/ssh/sshd_config`.

  Mostly you can leave this alone but the following is a useful property to set:

  ```
  PermitRootLogin no
  ```

  See [Disable non-root ssh access](./Disable_non-root_ssh_access.md) for more.

  ## Known hosts

  Within your home directory at `./ssh/known_hosts` you will find a record of all
  the public keys of the servers you have connected. This file exists for both
  servers and clients, e.g:

  ```sh
  cat ./ssh/known_hosts
  # systemsobscure.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKn6vyv9b+Nk5r
  # YVSAk5KvsTiC24K6uSpzCHzgLNoqt2
  ```

  This shows the public key of my server at `systemsobscure.net` along with
  others.

  ## Authorized keys

  On servers only, there is also an `authorized_keys` file which shows the
  server's own public keys that it presents to clients.

  If I go to my server I see that this key matches the one I have on my client
  computer `known_hosts`:

  ```sh
  sudo cat /etc/ssh/ssh_host_ed25519_key.pub
  # ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKn6vyv9b+Nk5rYVSAk5KvsTiC24K6uSpzCHzgLNoqt2 root@self-host-server

  ```

  Each user on a server will also have a `~/.ssh` directory also containing an
  `authorized_keys` file. This contains the public keys of clients who are allowed
  to connect to that user account.

  Hence I see the same public key of my desktop client machine in both places.

  ## How the tunnel is created

  ## Generating a key pair

  ```sh
  ssh-keygen
  ```

  ## Add a new public key to a server so that it can be accessed from client

  ```sh
  # On server

  vim .ssh/authorized_keys

  ```

  Then add public key of client

  Then, if you typically connect with the default SSH key, you need to specify the
  specific key when connecting:

  ```sh
  ssh -o "IdentitiesOnly=yes" -i ~/.ssh/public_key_file user@server
  ```


path: USB-C.md
sortable_path: USB-C.md
title: USB-C
checksum: 9fd17c5cc79dec273849719ba1244026f36b2db8c7d5df229cd29e969721d067
created: 2025-03-12 18:03:27.850293965+00:00
modified: 2025-08-10 14:08:47.068602484+00:00
metadata: {"created":"Wednesday, March 12, 2025","tags":["hardware","usb"]}
lead:
  The "C" in the name refers to the shape of the _connector_ not the protocol that
  is used for the data transfer. (This is the case for all USB "types" which makes
  things confusing.)

body:
  The "C" in the name refers to the shape of the _connector_ not the protocol that
  is used for the data transfer. (This is the case for all USB "types" which makes
  things confusing.)

  Multiple protocols can use the same connector for input/output. For instance
  Thunderbolt can transfer over USB-C but not every port that can fit a USB-C is a
  Thunderbolt port. The device's internal hardware determines the protocols
  available.

  The USB-C connector can serve all of the following protocols (from slowest to
  fastest):

  - USB 1.0 (12mb/s)
  - USB 2.0 (480Mb/s)
  - USB 3.0 (5Gb/s)
  - USB 3.1 (10Gb/s)
  - USB 3.2 (20Gb/s)
  - USB 4 (40Gb/s)
  - Thunderbolt (40Gb/s)

  A USB-C connector is equipped to just charge a device as well as transfer data.

  The world is converging (often due to legislation) on USB-C as the global
  standard. It can be one connector for all existing protocols and future
  protocols (for about 15 years). Its physical design an form factor can support
  future advancements whilst using the same connector.

  Ultimately USB-C will be surpassed by more another yet more advanced connector
  and associated protocols but for the time being we can stop using the older
  models.

raw_content:
  ---
  tags: [hardware, usb]
  created: Wednesday, March 12, 2025
  ---

  # USB-C

  The "C" in the name refers to the shape of the _connector_ not the protocol that
  is used for the data transfer. (This is the case for all USB "types" which makes
  things confusing.)

  Multiple protocols can use the same connector for input/output. For instance
  Thunderbolt can transfer over USB-C but not every port that can fit a USB-C is a
  Thunderbolt port. The device's internal hardware determines the protocols
  available.

  The USB-C connector can serve all of the following protocols (from slowest to
  fastest):

  - USB 1.0 (12mb/s)
  - USB 2.0 (480Mb/s)
  - USB 3.0 (5Gb/s)
  - USB 3.1 (10Gb/s)
  - USB 3.2 (20Gb/s)
  - USB 4 (40Gb/s)
  - Thunderbolt (40Gb/s)

  A USB-C connector is equipped to just charge a device as well as transfer data.

  The world is converging (often due to legislation) on USB-C as the global
  standard. It can be one connector for all existing protocols and future
  protocols (for about 15 years). Its physical design an form factor can support
  future advancements whilst using the same connector.

  Ultimately USB-C will be surpassed by more another yet more advanced connector
  and associated protocols but for the time being we can stop using the older
  models.


path: Code_Craft_The_Practice_Of_Writing_Excellent_Code.md
sortable_path: Code_Craft_The_Practice_Of_Writing_Excellent_Code.md
title: Code Craft: The Practice Of Writing Excellent Code
lead: ## Clarity not brevity
checksum: 60cff63f07f116ece9b3e97c16a3bf9c62421aa6078869bb16cca691643a1a77
created: 2025-03-25 16:45:40.035843459+00:00
modified: 2025-08-10 14:08:47.043786609+00:00
metadata: {"created":"Tuesday, March 25, 2025","tags":["literature"]}
body:
  ## Clarity not brevity

  > Readability is the best single criterion of program quality: If a program is
  > easy to read, it is probably a good program; if it is hard to read, it
  > probably isn't good. -- _The Elements of Programming Style_ (Kernighan and
  > Plaugher, 1978)

  Whenever you can choose between concise (but potentially confusing) code and
  clear (but potentially tedious) code, use code that reads as intended, even if
  it’s less elegant.

  ## Initialise all variables at their points of declaration

  The uninitialized value may turn into a problem further down the line.

  ## Declare variables as late as possible

  Place the variable as close as possible to its use. This prevents it from
  confusing other parts of the code. It also clarifies the code using the
  variable. You don’t have to hunt around to find the variable’s type and
  initialization; a nearby declaration makes it obvious.

  ## Naming

  - Choose names from the perspective of an inexperienced reader, not from your
    internal, knowledgeable perspective.

  - If you can’t come up with a good name, then you might need to change your
    design. It’s an indication that something might be wrong.

  - Use clear names even if they are long. It doesn’t matter that a name is long
    if its meaning is unambiguous.

  - Avoid redundant words in names. Specifically, avoid these words in type names:
    _class_, _data_, _object_, and _type_.

  ## Self-documenting code

  This can be achieved with good naming alone if you are clever about it. It
  doesn't necessarily require automated tools or excessive commentary.

  > Don't document bad code - rewrite it. (Kernighan et al.)

  ### Scope

  - With smaller, more specific scope we can name things in a more intuitive and
    expressive way. For example `Tree.countApples()`, `Tree.hasApples` rather than
    `Tree.countApplesInTree()`.

  ### Good presentation

  - Make the “normal” path through your code obvious. Error cases should not
    confuse the normal flow of execution. Your if-then-else constructs should be
    ordered _consistently_ (i.e., always place the “normal” case before the
    “error” case, or vice versa).

  - _One function, one action_. Make that your mantra.

  - Public information should come first, since this is what the class user needs
    to see. Put the **private implementation details at the end**, since they are
    less important to most readers.

  ### Comments

  - Explain _why_ not _how_. Your comments should not describe how the program
    works. You can see that by reading the code. The code is the definitive
    description of how the code works. You should instead focus on describing why
    something is written the way it is or what the next block of statements
    ultimately achieves.

  ## Error handling

  - Handling errors well is challenging! It is not easy to write good error
    handling that doesn’t twist and warp the underlying program logic.

  - Try to throw errors close enough to the source to prevent obscure and
    out-of-context error handling _while_ being far enough away not to cloud
    normal code with roundabout paths.

    - _This is something I struggle with. The error is caught later than when it
      occurs and then all I have is a generic catch-all error that is hard to make
      sense of._

  ### Close to source - benefits

  - Managing each error near its source means that control passes through less
    code in an invalid state. By handling an error near to its cause, you retain
    important contextual information, making the error-handling code clearer.

  ### As late as possible - benefits

  - This recognizes that code detecting an error rarely knows what to do about it.
    It often depends on the context in which it is used. You can pass an exception
    through each level until you know how to deal with the error.

  ### Errors to check for

  - Function/method parameters
    - Ensure you have been given correct and consistent input. Consider using
      assertions for this.
    - Check that invariants are satisfied at interesting points in execution
    - Check all values from external sources for validity before you use them
    - Check the return status of all system and other subordinate function calls.

raw_content:
  ---
  tags: [literature]
  created: Tuesday, March 25, 2025
  ---

  # _Code Craft: The Practice Of Writing Excellent Code_

  ## Clarity not brevity

  > Readability is the best single criterion of program quality: If a program is
  > easy to read, it is probably a good program; if it is hard to read, it
  > probably isn't good. -- _The Elements of Programming Style_ (Kernighan and
  > Plaugher, 1978)

  Whenever you can choose between concise (but potentially confusing) code and
  clear (but potentially tedious) code, use code that reads as intended, even if
  it’s less elegant.

  ## Initialise all variables at their points of declaration

  The uninitialized value may turn into a problem further down the line.

  ## Declare variables as late as possible

  Place the variable as close as possible to its use. This prevents it from
  confusing other parts of the code. It also clarifies the code using the
  variable. You don’t have to hunt around to find the variable’s type and
  initialization; a nearby declaration makes it obvious.

  ## Naming

  - Choose names from the perspective of an inexperienced reader, not from your
    internal, knowledgeable perspective.

  - If you can’t come up with a good name, then you might need to change your
    design. It’s an indication that something might be wrong.

  - Use clear names even if they are long. It doesn’t matter that a name is long
    if its meaning is unambiguous.

  - Avoid redundant words in names. Specifically, avoid these words in type names:
    _class_, _data_, _object_, and _type_.

  ## Self-documenting code

  This can be achieved with good naming alone if you are clever about it. It
  doesn't necessarily require automated tools or excessive commentary.

  > Don't document bad code - rewrite it. (Kernighan et al.)

  ### Scope

  - With smaller, more specific scope we can name things in a more intuitive and
    expressive way. For example `Tree.countApples()`, `Tree.hasApples` rather than
    `Tree.countApplesInTree()`.

  ### Good presentation

  - Make the “normal” path through your code obvious. Error cases should not
    confuse the normal flow of execution. Your if-then-else constructs should be
    ordered _consistently_ (i.e., always place the “normal” case before the
    “error” case, or vice versa).

  - _One function, one action_. Make that your mantra.

  - Public information should come first, since this is what the class user needs
    to see. Put the **private implementation details at the end**, since they are
    less important to most readers.

  ### Comments

  - Explain _why_ not _how_. Your comments should not describe how the program
    works. You can see that by reading the code. The code is the definitive
    description of how the code works. You should instead focus on describing why
    something is written the way it is or what the next block of statements
    ultimately achieves.

  ## Error handling

  - Handling errors well is challenging! It is not easy to write good error
    handling that doesn’t twist and warp the underlying program logic.

  - Try to throw errors close enough to the source to prevent obscure and
    out-of-context error handling _while_ being far enough away not to cloud
    normal code with roundabout paths.

    - _This is something I struggle with. The error is caught later than when it
      occurs and then all I have is a generic catch-all error that is hard to make
      sense of._

  ### Close to source - benefits

  - Managing each error near its source means that control passes through less
    code in an invalid state. By handling an error near to its cause, you retain
    important contextual information, making the error-handling code clearer.

  ### As late as possible - benefits

  - This recognizes that code detecting an error rarely knows what to do about it.
    It often depends on the context in which it is used. You can pass an exception
    through each level until you know how to deal with the error.

  ### Errors to check for

  - Function/method parameters
    - Ensure you have been given correct and consistent input. Consider using
      assertions for this.
    - Check that invariants are satisfied at interesting points in execution
    - Check all values from external sources for validity before you use them
    - Check the return status of all system and other subordinate function calls.


path: Docker_cleanup.md
sortable_path: Docker_cleanup.md
title: Docker cleanup
lead: ## Identify _all_ running containers
checksum: e3b70f6c40418084ce4aaa90e471926bbf1f839cd1fb66ff8df00087632d8138
created: 2025-03-27 18:46:39.914761229+00:00
modified: 2025-08-10 14:08:47.312711124+00:00
metadata: {"created":"Thursday, March 27, 2025","tags":["docker","procedural"]}
body:
  ## Identify _all_ running containers

  ```sh
  docker ps -a
  ```

  ## Generic clean-up: free as many bytes as possible

  ```sh
  sudo docker system prune -a
  ```

  ## Check Docker storage usage

  ```sh
  sudo docker system df
  ```

  ## Unused volumes

  These are the biggest culprit when taking up server space.

  Use this command to clear them out:

  ```sh
  sudo docker volume rm $(sudo docker volume ls -q -f dangling=true)
  ```

  These are typically UUIDs and named anonymously which makes it hard to know what
  you are deleting.

  To avoid this, the best practice is to name the volumes in the Dockerfile and
  also label them to make it easier to tell what's doing what:

  ```yml
  volumes:
    nextcloud_data:
      name: nextcloud_data
      labels:
        app: nextcloud
        purpose: user_data

  services:
    nextcloud:
      volumes:
        - nextcloud_data:/var/www/html
  ```

  ### Locations to check (even after deleting volume)

  ```
  /var/lib/docker/image
  /var/lib/docker/containers
  /var/lib/docker/volumes
  /var/lib/docker/buildkit # build cache
  ```

  ## Unused images

  ### Identify

  ```sh
  sudo docker images --filter "dangling=true"
  ```

  ### Delete

  ```sh

  sudo docker rmi mariadb:latest
  ```

  ## Delete containers

  ```sh
  docker container rm <uuid>
  ```

raw_content:
  ---
  tags: [docker, procedural]
  created: Thursday, March 27, 2025
  ---

  # Docker cleanup

  ## Identify _all_ running containers

  ```sh
  docker ps -a
  ```

  ## Generic clean-up: free as many bytes as possible

  ```sh
  sudo docker system prune -a
  ```

  ## Check Docker storage usage

  ```sh
  sudo docker system df
  ```

  ## Unused volumes

  These are the biggest culprit when taking up server space.

  Use this command to clear them out:

  ```sh
  sudo docker volume rm $(sudo docker volume ls -q -f dangling=true)
  ```

  These are typically UUIDs and named anonymously which makes it hard to know what
  you are deleting.

  To avoid this, the best practice is to name the volumes in the Dockerfile and
  also label them to make it easier to tell what's doing what:

  ```yml
  volumes:
    nextcloud_data:
      name: nextcloud_data
      labels:
        app: nextcloud
        purpose: user_data

  services:
    nextcloud:
      volumes:
        - nextcloud_data:/var/www/html
  ```

  ### Locations to check (even after deleting volume)

  ```
  /var/lib/docker/image
  /var/lib/docker/containers
  /var/lib/docker/volumes
  /var/lib/docker/buildkit # build cache
  ```

  ## Unused images

  ### Identify

  ```sh
  sudo docker images --filter "dangling=true"
  ```

  ### Delete

  ```sh

  sudo docker rmi mariadb:latest
  ```

  ## Delete containers

  ```sh
  docker container rm <uuid>
  ```


path: Certbot.md
sortable_path: Certbot.md
title: Certbot
lead: ## Generate certificates for new subdomain
checksum: 4b1fc70930651700f9ba7aa0a1124af427761132f1ad444c463301980739a594
created: 2025-04-02 15:39:48.849541319+00:00
modified: 2025-08-10 14:08:47.121733463+00:00
metadata: {"created":"Wednesday, April 02, 2025","tags":["servers","procedural"]}
body:
  ## Generate certificates for new subdomain

  ```sh
  sudo certbot certonly --standalone -d rootdomain.net -d subdomainA.rootdomain.net -d subdomainB.rootdomain.net
  ```

  ## List certificates

  ```sh
  sudo certbot certificates
  ```

  ## Delete a certifcate

  ```sh
  sudo cerbot delete --cert-name rootdomain.net

  ```

raw_content:
  ---
  tags: [servers, procedural]
  created: Wednesday, April 02, 2025
  ---

  # Certbot

  ## Generate certificates for new subdomain

  ```sh
  sudo certbot certonly --standalone -d rootdomain.net -d subdomainA.rootdomain.net -d subdomainB.rootdomain.net
  ```

  ## List certificates

  ```sh
  sudo certbot certificates
  ```

  ## Delete a certifcate

  ```sh
  sudo cerbot delete --cert-name rootdomain.net

  ```


path: chmod.md
sortable_path: chmod.md
title: chmod
lead: Modify file permissions.
checksum: 4e19b65325eb68d2cf8b249bd6a45d56d00e455e6316b1dee6b1e260ab498884
created: 2025-04-11 17:15:07.926855321+00:00
modified: 2025-08-10 14:08:47.346600273+00:00
metadata: {"created":"Friday, April 11, 2025","tags":["permissions","Linux"]}
body:
  Modify file permissions.

  We can use symbols or numbers.

  For example:

  ```sh
  chmod +x filename
  ```

  To make a file executable.

  When we use numbers this is called an _absolute_ change, because all permission
  bits are being set at once in octal notation.

  Best just to memorise the most common sequences. Bear in mind that some only
  apply to files, rather than directories or executable programs.

  | Mode | Meaning                           | Applied to            |
  | ---- | --------------------------------- | --------------------- |
  | 644  | user: r/w, group/other: r         | files                 |
  | 600  | user: r/w, group/other: none      | files                 |
  | 755  | user: r/w/e, group/other: r/e     | directories, programs |
  | 700  | user: r/w/e, group/other: none    | directories, programs |
  | 711  | user: r/w/e, group/other: execute | directories           |

  ### Useful options

  `-v` → verbose: tell the user what `chmod` is doing

  `-r` → work recursively, i.e apply the action to directories as well as files

  > You can list the contents of a directory if it's readable but you can only
  > access a file in a directory if the directory is executable!

raw_content:
  ---
  tags: [permissions, Linux]
  created: Friday, April 11, 2025
  ---

  # chmod

  Modify file permissions.

  We can use symbols or numbers.

  For example:

  ```sh
  chmod +x filename
  ```

  To make a file executable.

  When we use numbers this is called an _absolute_ change, because all permission
  bits are being set at once in octal notation.

  Best just to memorise the most common sequences. Bear in mind that some only
  apply to files, rather than directories or executable programs.

  | Mode | Meaning                           | Applied to            |
  | ---- | --------------------------------- | --------------------- |
  | 644  | user: r/w, group/other: r         | files                 |
  | 600  | user: r/w, group/other: none      | files                 |
  | 755  | user: r/w/e, group/other: r/e     | directories, programs |
  | 700  | user: r/w/e, group/other: none    | directories, programs |
  | 711  | user: r/w/e, group/other: execute | directories           |

  ### Useful options

  `-v` → verbose: tell the user what `chmod` is doing

  `-r` → work recursively, i.e apply the action to directories as well as files

  > You can list the contents of a directory if it's readable but you can only
  > access a file in a directory if the directory is executable!


path: chown.md
sortable_path: chown.md
title: chown
lead: Change the owner of system files and directories.
checksum: bc23f326415f4488a594a19e8a733638e3357cbb13c17d7f68297a543f88c9da
created: 2025-04-11 17:41:00.475688525+00:00
modified: 2025-08-10 14:08:47.142996819+00:00
metadata: {"created":"Friday, April 11, 2025","tags":["permissions","Linux"]}
body:
  Change the owner of system files and directories.

  Basic syntax: `chown <owner:group> filename`.

  ```sh
  # Change file owner
  chown user1 doc.txt

  # Change owner and group
  chown user1:developers document.txt

  # Change only the group
  chown :staff shared_folder/

  # Do so recursively for dirs
  chown -R www-data:www-data /var/www/
  ```

  TODO: Partition into separate entry on groups:

  > When a user account in Linux is created the system also creates a group with
  > the same name as the user (known as the _primary group_ or _user private
  > group_).

  Because of the above, you will often change owndership to yourself with:

  ```sh
  chown thomas:thomas some_dir
  ```

  See groups:

  ```sh
  groups
  # thomas realtime docker input wheel adb plugdev
  ```

  When reassigning users and groups, it's safest to use the actual name. But each
  user/group also has a numeric representation, corresponding to UID:GID.

  The first regular, non-system user created on most Unix distributions is 1000
  but this isn't universal. Likewise his group will be 1000.

  See your UID/GID and the GIDs of the groups you're in:

  ```
  $ id
  uid=1000(thomas) gid=1000(thomas) groups=1000(thomas),959(realtime),966(docker),994(input),998(wheel),1001(adb),1002(plugdev)
  ```

raw_content:
  ---
  tags: [permissions, Linux]
  created: Friday, April 11, 2025
  ---

  # chown

  Change the owner of system files and directories.

  Basic syntax: `chown <owner:group> filename`.

  ```sh
  # Change file owner
  chown user1 doc.txt

  # Change owner and group
  chown user1:developers document.txt

  # Change only the group
  chown :staff shared_folder/

  # Do so recursively for dirs
  chown -R www-data:www-data /var/www/
  ```

  TODO: Partition into separate entry on groups:

  > When a user account in Linux is created the system also creates a group with
  > the same name as the user (known as the _primary group_ or _user private
  > group_).

  Because of the above, you will often change owndership to yourself with:

  ```sh
  chown thomas:thomas some_dir
  ```

  See groups:

  ```sh
  groups
  # thomas realtime docker input wheel adb plugdev
  ```

  When reassigning users and groups, it's safest to use the actual name. But each
  user/group also has a numeric representation, corresponding to UID:GID.

  The first regular, non-system user created on most Unix distributions is 1000
  but this isn't universal. Likewise his group will be 1000.

  See your UID/GID and the GIDs of the groups you're in:

  ```
  $ id
  uid=1000(thomas) gid=1000(thomas) groups=1000(thomas),959(realtime),966(docker),994(input),998(wheel),1001(adb),1002(plugdev)
  ```


path: Constructing_paths_in_Python.md
sortable_path: Constructing_paths_in_Python.md
title: Constructing paths in Python
lead: Rather than hard-coding paths it's better to use path construction operators.
checksum: da4bece8da8b45e6ec871105683cd04c9103a4b514199a9cd3ed2bce1a06727b
created: 2025-04-15 06:04:35.541270982+00:00
modified: 2025-08-10 14:08:47.293367938+00:00
metadata: {"created":"Sunday, March 16, 2025","tags":["python"]}
body:
  Rather than hard-coding paths it's better to use path construction operators.

  In the past this was done with `os.path` but a more modern library is `pathlib`.

  ```python
  import os
  import pathlib
  from pathlib import Path
  ```

  I'll show both, side by side.

  ## Get directory of the current script

  ```py
  SCRIPT_DIR = Path(__file__).parent.absolute()
  SCRIPT_DIR = os.path.dirname(os.path.abspath(__file))
  ```

  ## Get the user home directory

  ```py
  HOME_DIR = Path.home
  HOME_DIR = os.environ.get("HOME")
  ```

  ## Construct a path from $HOME

  ```py
  HOME_DIR = Path.home
  HOME_DIR = os.environ.get("HOME")

  SUB_DIR = HOME_DIR / "repos" / "systems-obscure" / "src"
  SUB_DIR = os.path.join(HOME_DIR, "repos", "systems-obscure", "src")
  ```

  ## Access root directories

  It's less straightforward to access directories that are not children of
  `$HOME`.

  For instance to access mounted drives:

  ```py
  EXT_HD = Path("/media/samsung-T3")
  EXT_HD_DIR = EXT_HD / "music"

  EXT_HD = "/media/samsung-T3"
  EXT_HD_DIR  = os.path.join(EXT_HD, "music")
  ```

raw_content:
  ---
  tags: [python]
  created: Sunday, March 16, 2025
  ---

  # Constructing paths in Python

  Rather than hard-coding paths it's better to use path construction operators.

  In the past this was done with `os.path` but a more modern library is `pathlib`.

  ```python
  import os
  import pathlib
  from pathlib import Path
  ```

  I'll show both, side by side.

  ## Get directory of the current script

  ```py
  SCRIPT_DIR = Path(__file__).parent.absolute()
  SCRIPT_DIR = os.path.dirname(os.path.abspath(__file))
  ```

  ## Get the user home directory

  ```py
  HOME_DIR = Path.home
  HOME_DIR = os.environ.get("HOME")
  ```

  ## Construct a path from $HOME

  ```py
  HOME_DIR = Path.home
  HOME_DIR = os.environ.get("HOME")

  SUB_DIR = HOME_DIR / "repos" / "systems-obscure" / "src"
  SUB_DIR = os.path.join(HOME_DIR, "repos", "systems-obscure", "src")
  ```

  ## Access root directories

  It's less straightforward to access directories that are not children of
  `$HOME`.

  For instance to access mounted drives:

  ```py
  EXT_HD = Path("/media/samsung-T3")
  EXT_HD_DIR = EXT_HD / "music"

  EXT_HD = "/media/samsung-T3"
  EXT_HD_DIR  = os.path.join(EXT_HD, "music")
  ```


path: File_permissions_in_Linux.md
sortable_path: File_permissions_in_Linux.md
title: File permissions in Linux
lead: ## View permissions
checksum: 8f03659b1506fd476c855eb4c88f0c0ca96c61b0422f138254da1492cbe1cb62
created: 2025-04-15 06:04:35.551074918+00:00
modified: 2025-08-10 14:08:47.181674789+00:00
metadata: {"created":"Friday, April 11, 2025","tags":["Linux","permissions"]}
body:
  ## View permissions

  ```bash
  ls -rfl
  ```

  ## What the letters mean

  ```bash
  drwxr-xr-x  2 thomas thomas  4096 Jan 21 18:00 dist
  drwxr-xr-x  2 thomas thomas  4096 Dec 29 12:50 out
  -rw-r--r--  1 thomas thomas  1108 Jan 21 17:42 README.md
  ```

  The first column of the permissions output is known as the file's _mode_. The
  sequence from left to right is as follows:

  ```
  -       - - -                 - - -                 - - -
  type    user permissions      group permissions     other permissions
  ```

  <dl>
    <dt>type</dt>
    <dd>The file type. A dash just means an ordinary file. `d` means directory </dd>
   
    <dt>user permissions</dt>
    <dd>read, write or execute. A dash means 'nothing': the permissions for that slot in the set have not be assigned</dd>

    <dt>group and other</dt>
    <dd>group is obviously what anyone belonging to the current file's user group can do. Everyone else (outside of the user and the group) is covered by the other permissions, sometimes known as 'world' permissions</dd>
  </dl>

raw_content:
  ---
  tags: [Linux, permissions]
  created: Friday, April 11, 2025
  ---

  # File permissions in Linux

  ## View permissions

  ```bash
  ls -rfl
  ```

  ## What the letters mean

  ```bash
  drwxr-xr-x  2 thomas thomas  4096 Jan 21 18:00 dist
  drwxr-xr-x  2 thomas thomas  4096 Dec 29 12:50 out
  -rw-r--r--  1 thomas thomas  1108 Jan 21 17:42 README.md
  ```

  The first column of the permissions output is known as the file's _mode_. The
  sequence from left to right is as follows:

  ```
  -       - - -                 - - -                 - - -
  type    user permissions      group permissions     other permissions
  ```

  <dl>
    <dt>type</dt>
    <dd>The file type. A dash just means an ordinary file. `d` means directory </dd>
   
    <dt>user permissions</dt>
    <dd>read, write or execute. A dash means 'nothing': the permissions for that slot in the set have not be assigned</dd>

    <dt>group and other</dt>
    <dd>group is obviously what anyone belonging to the current file's user group can do. Everyone else (outside of the user and the group) is covered by the other permissions, sometimes known as 'world' permissions</dd>
  </dl>


path: Git_access_over_HTTPS_with_credentials.md
sortable_path: Git_access_over_HTTPS_with_credentials.md
title: Git access with HTTPS and credentials
checksum: 219e5d816fb3f7440ee67faae49231968636ae225a29c9b5bfaccd2b4a18360c
created: 2025-04-15 06:04:35.559725483+00:00
modified: 2025-08-10 14:08:47.069780946+00:00
metadata: {"created":"Sunday, April 13, 2025","tags":["git"]}
lead:
  If not using SSH, use HTTPS. However in order to combine this with 2FA it is
  necessary to use an access token as the pass, not your actual password.

body:
  If not using SSH, use HTTPS. However in order to combine this with 2FA it is
  necessary to use an access token as the pass, not your actual password.

  Create an access token in your remote Git client (GitHub, Forgejo).

  Then, clear any existing credentials with:

  ```sh
  git config --global --unset credential.helper
  ```

  Set Git config to use store mode:

  ```sh
  git config --global credential.helper store
  ```

  Do a push or pull in a repo to prompt authentication. Put in username but
  credential as password. You should be able to login.

  Confirm the credential has been added by viewing the credential file at the root
  of your home directory:

  ```sh
  cat ${HOME}/.git-credentials
  ```

raw_content:
  ---
  tags: [git]
  created: Sunday, April 13, 2025
  ---

  # Git access with HTTPS and credentials

  If not using SSH, use HTTPS. However in order to combine this with 2FA it is
  necessary to use an access token as the pass, not your actual password.

  Create an access token in your remote Git client (GitHub, Forgejo).

  Then, clear any existing credentials with:

  ```sh
  git config --global --unset credential.helper
  ```

  Set Git config to use store mode:

  ```sh
  git config --global credential.helper store
  ```

  Do a push or pull in a repo to prompt authentication. Put in username but
  credential as password. You should be able to login.

  Confirm the credential has been added by viewing the credential file at the root
  of your home directory:

  ```sh
  cat ${HOME}/.git-credentials
  ```


path: USB_protocols_and_connectors.md
sortable_path: USB_protocols_and_connectors.md
title: USB protocols and connectors
checksum: 3ba7bc2c7eb1e116177febd1956dea73c7ee754dee929cde0a7fb473ec304f6d
created: 2025-04-15 06:04:35.569058179+00:00
modified: 2025-08-10 14:08:47.241267502+00:00
metadata: {"created":"Wednesday, March 12, 2025","tags":["protocols","USB"]}
lead:
  Different USB connectors can transfer according to different protocols. A
  connector is not always tied to a specific protocol. For instance each
  subsequent generation of connector is typically backwards-compatible with the
  protocol of the connector that it superseded.

body:
  Different USB connectors can transfer according to different protocols. A
  connector is not always tied to a specific protocol. For instance each
  subsequent generation of connector is typically backwards-compatible with the
  protocol of the connector that it superseded.

  Some connectors are limited to slower protocols but they have the example of
  being small which is a benefit for smaller devices. The USB-C connector is so
  good because it is both small and capable of supporting the most advanced
  protocols (USB-4 and Thunderbolt).

  The main types of USB connector:

  - USB-A/USB-B
  - USB-A/USB-B
  - Micro-USB ()
  - Mini-USB
  - USB-C

  The illustration below matches connector to the protocol supported:

  ![](../img/usb-illustration.png)

raw_content:
  ---
  tags: [protocols, USB]
  created: Wednesday, March 12, 2025
  ---

  # USB protocols and connectors

  Different USB connectors can transfer according to different protocols. A
  connector is not always tied to a specific protocol. For instance each
  subsequent generation of connector is typically backwards-compatible with the
  protocol of the connector that it superseded.

  Some connectors are limited to slower protocols but they have the example of
  being small which is a benefit for smaller devices. The USB-C connector is so
  good because it is both small and capable of supporting the most advanced
  protocols (USB-4 and Thunderbolt).

  The main types of USB connector:

  - USB-A/USB-B
  - USB-A/USB-B
  - Micro-USB ()
  - Mini-USB
  - USB-C

  The illustration below matches connector to the protocol supported:

  ![](../img/usb-illustration.png)


path: Using_a_context_manager_in_Python.md
sortable_path: Using_a_context_manager_in_Python.md
title: Using a context manager in Python
checksum: a58b2ac56703de9c1998f08af72bdce53d72146fa6809c27427143c7caa0ddac
created: 2025-04-15 06:04:35.576834332+00:00
modified: 2025-08-10 14:08:47.200704209+00:00
metadata: {"created":"Sunday, March 16, 2025","tags":["python"]}
lead:
  For certain process that have a start and an end state such as opening and
  closing a file or connecting and disconnecting from a database, instead of
  having dedicated open/close, connect/disconnect handling, you can use a Context
  Manager as a form of "syntactic sugar".

body:
  For certain process that have a start and an end state such as opening and
  closing a file or connecting and disconnecting from a database, instead of
  having dedicated open/close, connect/disconnect handling, you can use a Context
  Manager as a form of "syntactic sugar".

  By using a Context Manager you can use a more Pythonic construction of e.g:

  ```py
  with open('/file-path') as file:
      ... # handle file operation
  ```

  ```py
  with open DatabaseService() as connection:
      ... # do database stuff
  ```

  When you use this approach it is an abstraction over `try...finally`. Meaning
  the clean-up `finally` operation will run automatically without you having to
  explicitly handle it.

  Some common processes such as file handling have the Context Manager built-in
  and you don't have to explicitly provision it.

  Other processes may lend themselves to `with` syntax but you have to do the
  configuration yourself.

  One such example is creating a database accessor. Below I have done this by
  using `__enter__` and `__exit__` methods on a database class:

  ```py
  class DatabaseService:
      def __init__(self, db_name, db_path):
          self.db_name = db_name
          self.db_path = db_path
          self.connection: Optional[sqlite3.Connection] = None

      def connect(self) -> Optional[sqlite3.Connection]:
          if self.connection is not None:
              return self.connection

          try:
              if not os.path.exists(self.db_path):
                  os.makedirs(self.db_path)
                  print("INFO Created database directory")
              self.connection = sqlite3.connect(f"{self.db_path}/{self.db_name}.db")
              self.connection.execute("PRAGMA foreign_keys = ON")
              return self.connection

          except Exception as e:
              raise Exception(f"ERROR Problem connecting to database: {e}")

      def disconnect(self) -> None:
          try:
              if self.connection is not None:
                  self.connection.close()
                  self.connection = None
          except Exception as e:
              raise Exception(f"ERROR Problem disconnecting from database: {e}")

      def __enter__(self) -> sqlite3.Connection:
          connection = self.connect()
          if connection is None:
              raise RuntimeError("Failed to establish database connection")
          return connection

      def __exit__(self) -> None:
          self.disconnect()
  ```

  Then I can use it like so:

  ```py
  with DatabaseService() as connection:
      cursor = connection.cursor()
      cursor.execute(...)
  ```

raw_content:
  ---
  tags: [python]
  created: Sunday, March 16, 2025
  ---

  # Using a context manager in Python

  For certain process that have a start and an end state such as opening and
  closing a file or connecting and disconnecting from a database, instead of
  having dedicated open/close, connect/disconnect handling, you can use a Context
  Manager as a form of "syntactic sugar".

  By using a Context Manager you can use a more Pythonic construction of e.g:

  ```py
  with open('/file-path') as file:
      ... # handle file operation
  ```

  ```py
  with open DatabaseService() as connection:
      ... # do database stuff
  ```

  When you use this approach it is an abstraction over `try...finally`. Meaning
  the clean-up `finally` operation will run automatically without you having to
  explicitly handle it.

  Some common processes such as file handling have the Context Manager built-in
  and you don't have to explicitly provision it.

  Other processes may lend themselves to `with` syntax but you have to do the
  configuration yourself.

  One such example is creating a database accessor. Below I have done this by
  using `__enter__` and `__exit__` methods on a database class:

  ```py
  class DatabaseService:
      def __init__(self, db_name, db_path):
          self.db_name = db_name
          self.db_path = db_path
          self.connection: Optional[sqlite3.Connection] = None

      def connect(self) -> Optional[sqlite3.Connection]:
          if self.connection is not None:
              return self.connection

          try:
              if not os.path.exists(self.db_path):
                  os.makedirs(self.db_path)
                  print("INFO Created database directory")
              self.connection = sqlite3.connect(f"{self.db_path}/{self.db_name}.db")
              self.connection.execute("PRAGMA foreign_keys = ON")
              return self.connection

          except Exception as e:
              raise Exception(f"ERROR Problem connecting to database: {e}")

      def disconnect(self) -> None:
          try:
              if self.connection is not None:
                  self.connection.close()
                  self.connection = None
          except Exception as e:
              raise Exception(f"ERROR Problem disconnecting from database: {e}")

      def __enter__(self) -> sqlite3.Connection:
          connection = self.connect()
          if connection is None:
              raise RuntimeError("Failed to establish database connection")
          return connection

      def __exit__(self) -> None:
          self.disconnect()
  ```

  Then I can use it like so:

  ```py
  with DatabaseService() as connection:
      cursor = connection.cursor()
      cursor.execute(...)
  ```


path: Web_sockets.md
sortable_path: Web_sockets.md
title: Web sockets
checksum: 6327ad24c4c2eb8a289adf5f07f71aef87627fc2b63b47302f070e087df7d875
created: 2025-04-15 06:04:35.584484256+00:00
modified: 2025-08-10 14:08:47.255853364+00:00
metadata: {"created":"Monday, April 14, 2025","tags":["networks","network-protocols","WebSocket"]}
lead:
  The WebSocket protocol provides a mechanism whereby data can continuously be
  shared between a client and a server in both directions.

body:
  The WebSocket protocol provides a mechanism whereby data can continuously be
  shared between a client and a server in both directions.

  This contrasts with standard HTTP whereby a server can only send data when a
  client specifically requests it and the communication channel is closed until
  such a request is made.

  Whereas HTTP works on the basis of a client-server architecture, WebSocket is an
  event-driven architecture. Under an event-driven architecture, _events_ trigger
  data flows rather than client requests.

  Examples of its application include instant messaging platforms where messages
  are sent in real time, stock trading platforms, social media feeds and
  cloud-based collaboration tools (e.g. Google Sheets). Hence WebSocket is best
  suited to applications where the immediacy and concurrency of server resolution
  is a factor.

  WebSocket was borne out of the limitations of HTTP. Prior to the creation of the
  WebSocket protocol, event-driven communication was attempted over HTTP through
  "long polling". Basically, the server sends a request to the server and if the
  response is not available, the server holds the request until the response is
  available and then returns it. Then, after an interval, the client sends the
  same request again and the process repeats. This is obviously inefficient.

  WebSocket is an [Application Layer](./Application_Layer_of_Internet_Protocol.md)
  protocol just like HTTP. HTTP is used as the initial connection mechanism but
  the resulting TCP connection is kept alive after the HTTP request completes,
  establishing the web socket. The standard HTTP handshake takes place but the
  HTTP request will include an 'Upgrade' header that indicates the client wants to
  establish a socket. The presence of this header results in the protocol being
  upgraded from HTTP to WebSocket.

  > This is possible because both HTTP and WebSocket are Application Layer
  > protocols that run on top of the same TCP connection.

  ## Encryption

  In the case of moving from HTTPS to WebSocket, the encryption of the former
  protocol is maintained. Although there is a protocol switch, the data is still
  being transferred on the same 443 port (used for HTTPS). Although the protocol
  has changed at the Application Layer, the encryption established at the TCP
  Layer is still in place.

  The HTTPS analog for WebSocket is WebSocketSecure (WSS). Thus HTTPS/WSS is the
  secure version of the HTTP/WS protocols.

raw_content:
  ---
  tags: [networks, network-protocols, WebSocket]
  created: Monday, April 14, 2025
  ---

  # Web sockets

  The WebSocket protocol provides a mechanism whereby data can continuously be
  shared between a client and a server in both directions.

  This contrasts with standard HTTP whereby a server can only send data when a
  client specifically requests it and the communication channel is closed until
  such a request is made.

  Whereas HTTP works on the basis of a client-server architecture, WebSocket is an
  event-driven architecture. Under an event-driven architecture, _events_ trigger
  data flows rather than client requests.

  Examples of its application include instant messaging platforms where messages
  are sent in real time, stock trading platforms, social media feeds and
  cloud-based collaboration tools (e.g. Google Sheets). Hence WebSocket is best
  suited to applications where the immediacy and concurrency of server resolution
  is a factor.

  WebSocket was borne out of the limitations of HTTP. Prior to the creation of the
  WebSocket protocol, event-driven communication was attempted over HTTP through
  "long polling". Basically, the server sends a request to the server and if the
  response is not available, the server holds the request until the response is
  available and then returns it. Then, after an interval, the client sends the
  same request again and the process repeats. This is obviously inefficient.

  WebSocket is an [Application Layer](./Application_Layer_of_Internet_Protocol.md)
  protocol just like HTTP. HTTP is used as the initial connection mechanism but
  the resulting TCP connection is kept alive after the HTTP request completes,
  establishing the web socket. The standard HTTP handshake takes place but the
  HTTP request will include an 'Upgrade' header that indicates the client wants to
  establish a socket. The presence of this header results in the protocol being
  upgraded from HTTP to WebSocket.

  > This is possible because both HTTP and WebSocket are Application Layer
  > protocols that run on top of the same TCP connection.

  ## Encryption

  In the case of moving from HTTPS to WebSocket, the encryption of the former
  protocol is maintained. Although there is a protocol switch, the data is still
  being transferred on the same 443 port (used for HTTPS). Although the protocol
  has changed at the Application Layer, the encryption established at the TCP
  Layer is still in place.

  The HTTPS analog for WebSocket is WebSocketSecure (WSS). Thus HTTPS/WSS is the
  secure version of the HTTP/WS protocols.


path: Advanced_listing_in_the_shell.md
sortable_path: Advanced_listing_in_the_shell.md
title: Advanced listing in the shell
lead: ## List items by most recently modified
checksum: bc2c35f438b0121d53bd8c52d5a6ab76244b53d82b439446ff542cb6984b16f9
created: 2025-04-15 06:04:35.611679184+00:00
modified: 2025-08-10 14:08:47.164026277+00:00
metadata: {"created":"Tuesday, April 15, 2025","tags":["procedural","shell"]}
body:
  ## List items by most recently modified

  ```sh
  # -l = long format, t = time
  ls -lt
  ```

  To include the full date:

  ```sh
  ls -l --time-style=long-iso
  ```

  ## Limit the number of items returned

  ```sh
  ls -l | head -n
  ```

  For example, to list the first five items:

  ```sh
  ls -l | head -5
  ```

raw_content:
  ---
  tags: [procedural, shell]
  created: Tuesday, April 15, 2025
  ---

  # Advanced listing in the shell

  ## List items by most recently modified

  ```sh
  # -l = long format, t = time
  ls -lt
  ```

  To include the full date:

  ```sh
  ls -l --time-style=long-iso
  ```

  ## Limit the number of items returned

  ```sh
  ls -l | head -n
  ```

  For example, to list the first five items:

  ```sh
  ls -l | head -5
  ```


path: Docker_networks.md
sortable_path: Docker_networks.md
title: Docker networks
lead: ## Show networks
checksum: a38d12968774583643bbd8afb514caa880ac56e0bd8711e416fd9043802bae01
created: 2025-04-24 05:32:27.473315526+00:00
modified: 2025-08-10 14:08:47.224633662+00:00
metadata: {"created":"Thursday, April 24, 2025","tags":["docker","networks","procedural"]}
body:
  ## Show networks

  ```sh
  docker network list
  ```

  ## Check which containers are attached to a given network

  ```sh
  docker network inspect [container_name]
  ```

raw_content:
  ---
  tags: [docker, networks, procedural]
  created: Thursday, April 24, 2025
  ---

  # Docker networks

  ## Show networks

  ```sh
  docker network list
  ```

  ## Check which containers are attached to a given network

  ```sh
  docker network inspect [container_name]
  ```


path: Viewing_Docker_logs.md
sortable_path: Viewing_Docker_logs.md
title: Viewing Docker logs
lead: ## View logs from outside of container
checksum: e89d57428d7694c7f0036bcdc5bc281ca818928f424f16ebdaa177e3f25f80c6
created: 2025-04-24 05:37:01.993016305+00:00
modified: 2025-08-10 14:08:47.335029278+00:00
metadata: {"created":"Thursday, April 24, 2025","tags":["docker","procedural"]}
body:
  ## View logs from outside of container

  ```sh
  docker logs <container_name> OR <container_id>
  ```

  To view the logs in realtime, apend `-f` for `--follow`:

  ```sh
  docker logs -f <container_name> OR <container_id>
  ```

  ## View logs from within the container

  For example, to view nginx logs, first
  [enter into the container](./Enter_into_a_Docker_container.md) and then:

  ```sh
  cat /var/log/nginx/access.log
  cat /var/log/nginx/error.log
  ```

raw_content:
  ---
  tags: [docker, procedural]
  created: Thursday, April 24, 2025
  ---

  # Viewing Docker logs

  ## View logs from outside of container

  ```sh
  docker logs <container_name> OR <container_id>
  ```

  To view the logs in realtime, apend `-f` for `--follow`:

  ```sh
  docker logs -f <container_name> OR <container_id>
  ```

  ## View logs from within the container

  For example, to view nginx logs, first
  [enter into the container](./Enter_into_a_Docker_container.md) and then:

  ```sh
  cat /var/log/nginx/access.log
  cat /var/log/nginx/error.log
  ```


path: Enter_into_a_Docker_container.md
sortable_path: Enter_into_a_Docker_container.md
title: Enter into a Docker container
lead:
  ```sh
  docker exec -it <container_name_or_id> bash
  ```

checksum: 4dbed9ce2025ff81b1954971be8f82c63692a9dfbf6b95cf5add15a1788d15ae
created: 2025-04-24 05:46:21.30861715+00:00
modified: 2025-08-10 14:08:47.111479405+00:00
metadata: {"created":"Thursday, April 24, 2025","tags":["docker","procedural"]}
body:
  ```sh
  docker exec -it <container_name_or_id> bash
  ```

  This will will open up a Bash shell at the container root.

raw_content:
  ---
  tags: [docker, procedural]
  created: Thursday, April 24, 2025
  ---

  # Enter into a Docker container

  ```sh
  docker exec -it <container_name_or_id> bash
  ```

  This will will open up a Bash shell at the container root.


path: journalctl.md
sortable_path: journalctl.md
title: journalctl
checksum: aa883484277709c341c68a33f6aa23a6062587c42eddd7176d85921fb94d2f31
created: 2025-04-24 05:53:40.311554952+00:00
modified: 2025-08-10 14:08:47.084113606+00:00
metadata: {"created":"Thursday, April 24, 2025","tags":["Linux","procedural","logs"]}
lead:
  We use `journalctl` to access [journald](./journald.md) logs. The command by
  itself outputs the entire log which will be huge and hard to scroll through. We
  can refine the results with modifiers.

body:
  We use `journalctl` to access [journald](./journald.md) logs. The command by
  itself outputs the entire log which will be huge and hard to scroll through. We
  can refine the results with modifiers.

  ### View logs for a specific process with pid

  ```bash
  journalctl _PID=1234
  ```

  ### View logs for a specific time period

  This can be really helpful since you can bracket the most recent events which
  will be more memorable.

  ```bash
  journalctl -S -1h
  ```

  ### View logs for a specfic systemd unit

  ```bash
  journalctl -u [unit_name] -e
  ```

  ### View boot logs

  ```bash
  journalctl -b
  ```

  #### Identify specific boot

  ```bash
  journalctl --list-boots

  ```

  ### List only kernel entries to the journal

  ```bash
  journalctl -k
  ```

  ### View logs in realtime

  Use `-f` for `--follow`:

  ```sh
  journalctl -f
  ```

raw_content:
  ---
  tags: [Linux, procedural, logs]
  created: Thursday, April 24, 2025
  ---

  # journalctl

  We use `journalctl` to access [journald](./journald.md) logs. The command by
  itself outputs the entire log which will be huge and hard to scroll through. We
  can refine the results with modifiers.

  ### View logs for a specific process with pid

  ```bash
  journalctl _PID=1234
  ```

  ### View logs for a specific time period

  This can be really helpful since you can bracket the most recent events which
  will be more memorable.

  ```bash
  journalctl -S -1h
  ```

  ### View logs for a specfic systemd unit

  ```bash
  journalctl -u [unit_name] -e
  ```

  ### View boot logs

  ```bash
  journalctl -b
  ```

  #### Identify specific boot

  ```bash
  journalctl --list-boots

  ```

  ### List only kernel entries to the journal

  ```bash
  journalctl -k
  ```

  ### View logs in realtime

  Use `-f` for `--follow`:

  ```sh
  journalctl -f
  ```


path: Secure_disk_wipe.md
sortable_path: Secure_disk_wipe.md
title: Secure disk wipe
lead:
  ```sh
  sudo shred -v -n 2 -z /dev/sda1
  ```

checksum: 7546b1c37f691d847eadc93916c465451bfeed81a8f6f8c860bb496555ea38db
created: 2025-04-26 10:43:15.39186647+00:00
modified: 2025-08-10 14:08:47.162239394+00:00
metadata: {"created":"Saturday, April 26, 2025","tags":["disks","procedural"]}
body:
  ```sh
  sudo shred -v -n 2 -z /dev/sda1
  ```

  Overwrites the disk with random data multiple times (`-n 2`) and finishes with
  zeros (`-z`).

  The more overwrites you specify, the longer it takes. Three overwrites on a USB
  flash disk can take like three hours so use advisedly.

raw_content:
  ---
  tags: [disks, procedural]
  created: Saturday, April 26, 2025
  ---

  # Secure disk wipe

  ```sh
  sudo shred -v -n 2 -z /dev/sda1
  ```

  Overwrites the disk with random data multiple times (`-n 2`) and finishes with
  zeros (`-z`).

  The more overwrites you specify, the longer it takes. Three overwrites on a USB
  flash disk can take like three hours so use advisedly.


path: Repair_disks-with_fsck.md
sortable_path: Repair_disks-with_fsck.md
title: Repair disks with fsck
lead: Identify file system errors on a disk.
checksum: b12bc1b1a20a16980fbe76bcdcae9a8dae223d41be6c72ea39cbe28517584dab
created: 2025-04-26 10:59:33.398380192+00:00
modified: 2025-08-10 14:08:47.177252812+00:00
metadata: {"created":"Saturday, April 26, 2025","tags":["disks","procedural","Linux"]}
body:
  Identify file system errors on a disk.

  Be sure to unmount the device beforehand.

  ```sh
  sudo fsck.vfat -a /dev/sdf1
  ```

  This will automatically apply repairs. An example output:

  ```
  fsck.fat 4.2 (2021-01-31)
  There are differences between boot sector and its backup.
  This is mostly harmless. Differences: (offset:original/backup)
    65:01/00
    Not automatically fixing this.
  /.Trash-1000/files/retropie
   Start does point to root directory. Deleting dir.
  Reclaimed 312130 unused clusters (2556968960 bytes) in 1553 chains.
  Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
   Automatically removing dirty bit.
  Free cluster summary wrong (1516247 vs. really 1516241)
    Auto-correcting.

  *** Filesystem was changed ***
  Writing changes.
  /dev/sdf1: 1582 files, 396097/1912338 clusters

  ```

raw_content:
  ---
  tags: [disks, procedural, Linux]
  created: Saturday, April 26, 2025
  ---

  # Repair disks with fsck

  Identify file system errors on a disk.

  Be sure to unmount the device beforehand.

  ```sh
  sudo fsck.vfat -a /dev/sdf1
  ```

  This will automatically apply repairs. An example output:

  ```
  fsck.fat 4.2 (2021-01-31)
  There are differences between boot sector and its backup.
  This is mostly harmless. Differences: (offset:original/backup)
    65:01/00
    Not automatically fixing this.
  /.Trash-1000/files/retropie
   Start does point to root directory. Deleting dir.
  Reclaimed 312130 unused clusters (2556968960 bytes) in 1553 chains.
  Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
   Automatically removing dirty bit.
  Free cluster summary wrong (1516247 vs. really 1516241)
    Auto-correcting.

  *** Filesystem was changed ***
  Writing changes.
  /dev/sdf1: 1582 files, 396097/1912338 clusters

  ```


path: rsync.md
sortable_path: rsync.md
title: rsync
lead: ## Ordering
checksum: fc68ed7d91b56cb6a4ed0c5e05fd6954be5bb5213ea9895bb63648d96f2c897d
created: 2025-04-26 10:59:33.414154669+00:00
modified: 2025-08-10 14:08:47.1145547+00:00
metadata: {"created":"Saturday, April 26, 2025","tags":["file-transfer","Linux","procedural","disks"]}
body:
  ## Ordering

  ### Local to local

  ```
  SOURCE_DIR > TARGET_DIR
  ```

  ### Local to remote

  ```
  LOCAL_SOURC_DIR > REMOTE_TARGET_DIR
  ```

  ### Remote to local

  ```
  REMOTE_TARGET_DIR > LOCAL_SOURC_DIR
  ```

  ```sh
  rsync -a <dir_to_copy_to> <dir_to_copy_from>
  ```

  ## Expanding directories

  The following:

  ```sh
  rsync -a local_dir  target_dir
  ```

  Will create `/target_dir/local_dir` at the target. In other words it will nest
  the actual directory you are interested in within what you have named the
  target.

  To avoid this, add a slash to the source directory, viz:

  ```sh
  rysync -a local_dir/ target_dir
  ```

  Now, at the target, there will just be `local_dir`.

  ## Standard options I use

  ```sh
  rsync -vzP
  ```

  - verbose output

  - use compression (only really useful when running rysnc over a network)

  - display progress

  - preserve partially copied filed and resume if network connection interrupted

  ## Archive mode

  Use "archive mode" when specifically wanting to create a backup of a directory
  (i.e. for long term storage rather than immediate use).

  ```sh
  rsync -a
  ```

  Archive mode is an umbrella for the following flags:

  - `-r`: recursive

  - `l`: copy [symlinks](./Symlinks.md) as symlinks

  - `p`: preserve permissions

  - `t`: preserve times

  - `g`: preserve group

raw_content:
  ---
  tags: [file-transfer, Linux, procedural, disks]
  created: Saturday, April 26, 2025
  ---

  # rsync

  ## Ordering

  ### Local to local

  ```
  SOURCE_DIR > TARGET_DIR
  ```

  ### Local to remote

  ```
  LOCAL_SOURC_DIR > REMOTE_TARGET_DIR
  ```

  ### Remote to local

  ```
  REMOTE_TARGET_DIR > LOCAL_SOURC_DIR
  ```

  ```sh
  rsync -a <dir_to_copy_to> <dir_to_copy_from>
  ```

  ## Expanding directories

  The following:

  ```sh
  rsync -a local_dir  target_dir
  ```

  Will create `/target_dir/local_dir` at the target. In other words it will nest
  the actual directory you are interested in within what you have named the
  target.

  To avoid this, add a slash to the source directory, viz:

  ```sh
  rysync -a local_dir/ target_dir
  ```

  Now, at the target, there will just be `local_dir`.

  ## Standard options I use

  ```sh
  rsync -vzP
  ```

  - verbose output

  - use compression (only really useful when running rysnc over a network)

  - display progress

  - preserve partially copied filed and resume if network connection interrupted

  ## Archive mode

  Use "archive mode" when specifically wanting to create a backup of a directory
  (i.e. for long term storage rather than immediate use).

  ```sh
  rsync -a
  ```

  Archive mode is an umbrella for the following flags:

  - `-r`: recursive

  - `l`: copy [symlinks](./Symlinks.md) as symlinks

  - `p`: preserve permissions

  - `t`: preserve times

  - `g`: preserve group


path: scp.md
sortable_path: scp.md
title: scp
lead:
  ```sh
  scp host:/dir/*.sql.gz /local_machine/dir
  ```

checksum: 93d11d6905b9c936dbe59ef53c0c8e24c898e6bdc7a6a0c22da3241775e98bb5
created: 2025-04-27 11:43:49.944095321+00:00
modified: 2025-08-10 14:08:47.175155587+00:00
metadata: {"created":"Sunday, April 27, 2025","tags":["file-transfer","Linux","procedural","servers"]}
body:
  ```sh
  scp host:/dir/*.sql.gz /local_machine/dir
  ```

  > Obviously SSH from local to remote is a prerequisite

raw_content:
  ---
  tags: [file-transfer, Linux, procedural, servers]
  created: Sunday, April 27, 2025
  ---

  # scp

  ```sh
  scp host:/dir/*.sql.gz /local_machine/dir
  ```

  > Obviously SSH from local to remote is a prerequisite


path: Docker_volumes.md
sortable_path: Docker_volumes.md
title: Docker volumes
lead: > This is a WIP re-write of [Docker Storage](./Docker_storage.md).
checksum: ad9892b6983509262a46e58c5c388fd67503b798dd4503db46cdce6ea800c97b
created: 2025-05-04 17:06:36.576773567+00:00
modified: 2025-08-10 14:08:47.362864982+00:00
metadata: {"created":"Sunday, May 04, 2025","tags":["docker"]}
body:
  > This is a WIP re-write of [Docker Storage](./Docker_storage.md).

  ## _Bind mounts_ versus _named volumes_

  Mostly when I use Docker in production I am using bind mounts rather than named
  volumes.

  With bind mounts you explicitly map a path on the host to a path within the
  container. For example, in a Docker Compose:

  ```yml
  services:
    rocketchat:
      volumes:
        - /mnt/storagebox_alpha/rocketchat/mongo/db:/data/db
  ```

  Here, the DB data for the RocketChat container is mapped to a drive in the
  `/mnt` directory on the actual server.

  When you use bind mounts you do not need to include a `volumes` section in the
  Compose file.

  In contrast _named volumes_ let Docker manage the storage on the host. You
  specify a name, but Docker decides where the data is stored. (Typically, this
  will be in `/var/lib/docker/volumes`.)

  The equivalent to the previous example as a named volume:

  ```yml
  services:
    rocketchat:
      volumes: rocketchat-db:/data/db

  volumes:
    rocketchat-db:
  ```

raw_content:
  ---
  tags: [docker]
  created: Sunday, May 04, 2025
  ---

  # Docker volumes

  > This is a WIP re-write of [Docker Storage](./Docker_storage.md).

  ## _Bind mounts_ versus _named volumes_

  Mostly when I use Docker in production I am using bind mounts rather than named
  volumes.

  With bind mounts you explicitly map a path on the host to a path within the
  container. For example, in a Docker Compose:

  ```yml
  services:
    rocketchat:
      volumes:
        - /mnt/storagebox_alpha/rocketchat/mongo/db:/data/db
  ```

  Here, the DB data for the RocketChat container is mapped to a drive in the
  `/mnt` directory on the actual server.

  When you use bind mounts you do not need to include a `volumes` section in the
  Compose file.

  In contrast _named volumes_ let Docker manage the storage on the host. You
  specify a name, but Docker decides where the data is stored. (Typically, this
  will be in `/var/lib/docker/volumes`.)

  The equivalent to the previous example as a named volume:

  ```yml
  services:
    rocketchat:
      volumes: rocketchat-db:/data/db

  volumes:
    rocketchat-db:
  ```


path: Radio basics.md
sortable_path: Radio basics.md
title: Radio basics
lead:
  Radio is the technology of communication or exchanging information using radio
  waves.

checksum: 62ca4eb82590c6f77d213a17a30778b83bb7f42d13c3976bb0fab23cddfba3e4
created: 2025-07-06 17:44:30.612522047+00:00
modified: 2025-07-07 15:47:40.751706342+00:00
metadata: {"created":"Sunday, July 06, 2025","tags":["radio"]}
body:
  Radio is the technology of communication or exchanging information using radio
  waves.

  Radio waves are electromagnetic waves that transmit at a certain frequency of
  the [electromagnetic spectrum](./Electromagnetism.md): between 3Hz and 300GHz.

  ![](../img/em-spectrum-radio.jpg)

  This makes them the lowest in frequency and the longest in wavelength.

  Radiowaves are generated by an electronic device called a **transmitter**, which
  is connected to an **antenna** which radiates the waves out into space.

  These waves can be detected by another antenna connected to a radio
  **receiver**.

  Information is transmitted by adding information to the raw electromagnetic
  wave. The wave by itself doesn not contain any information.

  The wave form used is a sine wave that fluctuates smoothly and consistently.

  There are two main methods of adding information to the waveform: AM and FM.

  AM stands for **amplitude modulation**. Here, the amplitude of the wave is
  altered. The amplitude is the distance from peak to peak. For example, the sine
  wave produced by a person's voice is overlaid onto the transmitter's sine wave
  to vary its amplitude.

  FM stands for **frequency modulation**. Here, the frequency of the wave is
  altered. The frequency is the number of oscillations over a given time period
  which we measure in Hertz.

raw_content:
  ---
  tags: [radio]
  created: Sunday, July 06, 2025
  ---

  # Radio basics

  Radio is the technology of communication or exchanging information using radio
  waves.

  Radio waves are electromagnetic waves that transmit at a certain frequency of
  the [electromagnetic spectrum](./Electromagnetism.md): between 3Hz and 300GHz.

  ![](../img/em-spectrum-radio.jpg)

  This makes them the lowest in frequency and the longest in wavelength.

  Radiowaves are generated by an electronic device called a **transmitter**, which
  is connected to an **antenna** which radiates the waves out into space.

  These waves can be detected by another antenna connected to a radio
  **receiver**.

  Information is transmitted by adding information to the raw electromagnetic
  wave. The wave by itself doesn not contain any information.

  The wave form used is a sine wave that fluctuates smoothly and consistently.

  There are two main methods of adding information to the waveform: AM and FM.

  AM stands for **amplitude modulation**. Here, the amplitude of the wave is
  altered. The amplitude is the distance from peak to peak. For example, the sine
  wave produced by a person's voice is overlaid onto the transmitter's sine wave
  to vary its amplitude.

  FM stands for **frequency modulation**. Here, the frequency of the wave is
  altered. The frequency is the number of oscillations over a given time period
  which we measure in Hertz.


path: List_largest_files_bash.md
sortable_path: List_largest_files_bash.md
title: List_largest_files_bash
lead:
  ```sh
  ls -lSh
  ```

body:
  ```sh
  ls -lSh
  ```

  ```sh
  # Limited to n largest

  ls -lSh | tail -10

  ```

checksum: d21241f6fbf82c7446431c475c64f3e49d764c482e140e878fd4e620dc507548
created: 2025-07-07 15:47:49.657828546+00:00
modified: 2025-08-10 14:08:47.227906927+00:00
metadata: {"created":"Monday, July 07, 2025","tags":["shell"]}
raw_content:
  ---
  tags: [shell]
  created: Monday, July 07, 2025
  ---

  # List_largest_files_bash

  ```sh
  ls -lSh
  ```

  ```sh
  # Limited to n largest

  ls -lSh | tail -10

  ```


path: AOL_and_early_Internet_enclosure.md
sortable_path: AOL_and_early_Internet_enclosure.md
title: f6564af9_AOL_and_early_internet_enclosure
checksum: 8f2eefe769e6cd66fe8b5a4402b0a972e07f5cb2955ae3694dc7dca203fc5dda
created: 2025-08-09 13:41:46.69084947+00:00
modified: 2025-08-10 14:08:47.267106949+00:00
lead:
  We might view AOL, with its ubiquitous 'free trial' CD-ROMs, as the first
  attempt by a corporate entity to try and enclose the Internet/Web. To take the
  vast informational expanse and reduce it to a single, filtered and contained
  space controlled by a single company.

body:
  We might view AOL, with its ubiquitous 'free trial' CD-ROMs, as the first
  attempt by a corporate entity to try and enclose the Internet/Web. To take the
  vast informational expanse and reduce it to a single, filtered and contained
  space controlled by a single company.

  ![Early AOL interface](../img/aol-on-windows.jpg)

  _AOL interface from 1993_

  This would later be achieved or approximated more effectively by Google (under
  the guise of doing the opposite - people are "empowered" to access the
  information they otherwise would not find, granted a key to the Internet etc.
  When in fact, making everyone go through the same gate enables surveillance and
  the monetisation of the use of a public resource) and then of course by the
  social media companies.

  AOL in fact started this even before there was a Web, pioneering the concept of
  a web portal, with their [modem-based](cfbef1c4_web_precursors.md) private-BBS
  business model. This provided access to a circumscribed subset of the Internet
  at a time when accessing information was opaque and confusing to the average
  non-technical user. The advent of the web first threatened and then basically
  obviated this model, when access to information via browsers and HTTP servers
  made the process easier and more democratized. Hence why AOL then tellingly
  pivoted to the browser and search-engine market (purchasing
  [Netscape](c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md) in 1999), the next
  available mechanism of enclosure.

  ![AOL homepage](../img/aol-original.jpg)

  _AOL interface from 1999_

raw_content:
  ---
  tags: [world-wide-web, computer-history, internet, ecopolsoc, bulletin-boards]
  created: Monday, November 04, 2024
  ---

  # f6564af9_AOL_and_early_internet_enclosure

  We might view AOL, with its ubiquitous 'free trial' CD-ROMs, as the first
  attempt by a corporate entity to try and enclose the Internet/Web. To take the
  vast informational expanse and reduce it to a single, filtered and contained
  space controlled by a single company.

  ![Early AOL interface](../img/aol-on-windows.jpg)

  _AOL interface from 1993_

  This would later be achieved or approximated more effectively by Google (under
  the guise of doing the opposite - people are "empowered" to access the
  information they otherwise would not find, granted a key to the Internet etc.
  When in fact, making everyone go through the same gate enables surveillance and
  the monetisation of the use of a public resource) and then of course by the
  social media companies.

  AOL in fact started this even before there was a Web, pioneering the concept of
  a web portal, with their [modem-based](cfbef1c4_web_precursors.md) private-BBS
  business model. This provided access to a circumscribed subset of the Internet
  at a time when accessing information was opaque and confusing to the average
  non-technical user. The advent of the web first threatened and then basically
  obviated this model, when access to information via browsers and HTTP servers
  made the process easier and more democratized. Hence why AOL then tellingly
  pivoted to the browser and search-engine market (purchasing
  [Netscape](c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md) in 1999), the next
  available mechanism of enclosure.

  ![AOL homepage](../img/aol-original.jpg)

  _AOL interface from 1999_

metadata:
  {"created":"Monday, November 04, 2024","tags":["world-wide-web","computer-history","internet","ecopolsoc","bulletin-boards"]}


path: ARPANET_motivations.md
sortable_path: ARPANET_motivations.md
title: f0f70a3c_ARPANET_motivations
checksum: 40f8c1e073b574aba95598ac1905ef9b369eecac748284bac8c8785e583d949a
created: 2025-08-09 13:41:46.714747605+00:00
modified: 2025-08-10 14:08:47.064299722+00:00
metadata: {"created":"Friday, October 18, 2024","tags":["ARPA","ARPANET","networks","computer-history"]}
lead:
  It is inaccurate to say that the chief motivation in creating the ARPANET was to
  create a network that could withstand nuclear conflict, where mulitple nodes
  could be hit and the network could still go on functioning.

body:
  It is inaccurate to say that the chief motivation in creating the ARPANET was to
  create a network that could withstand nuclear conflict, where mulitple nodes
  could be hit and the network could still go on functioning.

  Ideas that originated from this concern _did_ influence the project (chiefly
  [Paul Baran's work on distributed networks](385af4b4_baran_distributed_networks.md)
  at RAND), but this was not the concern of the ARPANET.

  ARPA's objectives were focused more on reducing efficiency in the academic and
  military projects it sponsored.

  Different models of computer were used throughout the
  military-industrial-academic complex. Users required different logins and a
  knowledge of different operating systems and programming languages in order to
  be able to use them.

  At the same time, there was a duplication of hardware. Different projects would
  all require their own computer to do their research. It would be better if the
  same computer could be accessed by multiple teams through time-sharing (already
  in existene) and a network.

  Finally, results from different research projects were being duplicated.
  Different teams would be producing the same data independently of one another
  because the data could not be readily shared. If this redundancy could be
  removed, and the data accessed and shared between multiple projects, teams could
  focus on new research and the production of novel applications.

raw_content:
  ---
  tags: [ARPA, ARPANET, networks, computer-history]
  created: Friday, October 18, 2024
  ---

  # f0f70a3c_ARPANET_motivations

  It is inaccurate to say that the chief motivation in creating the ARPANET was to
  create a network that could withstand nuclear conflict, where mulitple nodes
  could be hit and the network could still go on functioning.

  Ideas that originated from this concern _did_ influence the project (chiefly
  [Paul Baran's work on distributed networks](385af4b4_baran_distributed_networks.md)
  at RAND), but this was not the concern of the ARPANET.

  ARPA's objectives were focused more on reducing efficiency in the academic and
  military projects it sponsored.

  Different models of computer were used throughout the
  military-industrial-academic complex. Users required different logins and a
  knowledge of different operating systems and programming languages in order to
  be able to use them.

  At the same time, there was a duplication of hardware. Different projects would
  all require their own computer to do their research. It would be better if the
  same computer could be accessed by multiple teams through time-sharing (already
  in existene) and a network.

  Finally, results from different research projects were being duplicated.
  Different teams would be producing the same data independently of one another
  because the data could not be readily shared. If this redundancy could be
  removed, and the data accessed and shared between multiple projects, teams could
  focus on new research and the production of novel applications.


path: ARPA_origins.md
sortable_path: ARPA_origins.md
title: 861cc26e_ARPA_origins
checksum: 7e12e88545f4b0811358fa463ec06c85aeda890751517aac1fe0e08e2e0d2058
created: 2025-08-09 13:41:46.738856335+00:00
modified: 2025-08-10 14:08:47.357266994+00:00
metadata: {"created":"Monday, October 14, 2024","tags":["fleeting","computer-history","ARPA"]}
lead:
  After Sputnik, Eisenhower wanted a dedicated, cross-departmental division within
  the US government to focus on advanced technological research so that they would
  not be caught out again. ARPA was the result.

body:
  After Sputnik, Eisenhower wanted a dedicated, cross-departmental division within
  the US government to focus on advanced technological research so that they would
  not be caught out again. ARPA was the result.

  Originally under the control of the Department of Defence (Pentagon).

  Once projects reached maturity, they would be handed over to the armed forces.

  It's name switched back and forth between ARPA/DARPA depending on congressional
  factors and funding. The switch to DARPA (and the associated re-focusing on
  military applications) indirectly precipitated a brain-drain in the late 1970s
  which contributed to the birth of the personal computer around the same time.

  Main technologies it contributed to or invented include the #ARPANET,
  time-sharing and GPS systems.

raw_content:
  ---
  tags: [fleeting, computer-history, ARPA]
  created: Monday, October 14, 2024
  ---

  # 861cc26e_ARPA_origins

  After Sputnik, Eisenhower wanted a dedicated, cross-departmental division within
  the US government to focus on advanced technological research so that they would
  not be caught out again. ARPA was the result.

  Originally under the control of the Department of Defence (Pentagon).

  Once projects reached maturity, they would be handed over to the armed forces.

  It's name switched back and forth between ARPA/DARPA depending on congressional
  factors and funding. The switch to DARPA (and the associated re-focusing on
  military applications) indirectly precipitated a brain-drain in the late 1970s
  which contributed to the birth of the personal computer around the same time.

  Main technologies it contributed to or invented include the #ARPANET,
  time-sharing and GPS systems.


path: Attributes_of_Internet_enclosure.md
sortable_path: Attributes_of_Internet_enclosure.md
title: a771a6d9_defining_internet_enclosure
lead: ## Defining _enclosure_
checksum: 18744f42130d70d76167fc467e6607c602071cb923634b0c0a74ce43b3421edb
created: 2025-08-09 13:41:46.747854733+00:00
modified: 2025-08-10 14:08:47.359746634+00:00
metadata: {"created":"Friday, November 22, 2024","tags":["ecopolsoc","internet"]}
body:
  ## Defining _enclosure_

  The attempt by capital interests to encircle, control and contain networked
  digital communication with the object of extracting financial value from the
  exchanges and the community from which they arise.

  This constitutes the commodification of a human-constituted resource, like
  labour rather than a natural resource, like oil.

  ## Main characteristics

  ### Strategic denial of alternative communication media

  Where there is a _multiplicity_ of media and within media, a multiplicity of
  channels, capital steers communication toward a _single_ entity.

  In a "democratic free-market society", capital cannot simply ban alternative
  media, it can however work to diminish the influence of alternatives through the
  strategic denial of interoperability and the aquisition and subsumption of
  rivals into its system.

  ### Tyrannical control of the rules of communication

  Where the standards of communication are elective and established organically
  and collaboratively between peers (possibly even democratically), capital
  enforces rules derived externally and applied in an authoritarian manner. These
  rules are mutable but changes are enacted unilaterally and arbitrarily by the
  owners of the medium, not its users. Hence, tyranny in its technical political
  sense.

  ### Total control of information and content

  Where individuals share information freely in collaborative enterprise with
  diminished concern for ownership (typically favouring citation and iteration as
  models), capital takes full possession of the information generated such that it
  can be removed and destroyed at will without consultation.

  ### Surveillance

  Where communication is anonymised (loosly or strictly (i.e. through
  cryptographic means)), capital _surveils_ all exchanges and the behaviour of
  community members.

  Typically this is not _personal_ in the sense of being interested in the
  specific user's identity, rather it is abstracted into a composite of the
  person's preferences which may or may not disclose their legal identity (i.e
  metadata).

  This characteristic is both an _attribute_ of internet enclosure and its
  ultimate reason for existence. It is critical to the mechanism of extraction
  since it provides both data that can be sold to other capital interests and
  content that can be used to sell to community members as consumers.

  ### Artificial promotion of commercial content and prolongation of engagement

  Where engagement with media is idiosyncratic and organic, capital circumscribes
  the field. This is done to prolong engagement.

  At one pole this is achieved through the excitation of visceral responses by
  artificially favouring content that titillates, disgusts or angers the user. At
  the other, it is achieved through mollification and the coddling of the user.
  The user encounters only like-minded fellows who affirm and repeat his existing
  beliefs and values, creating community and a sense of safety through the
  othering of contrary viewpoints. Both techniques are demagogic in nature
  familiar from the control mechanisms of cults and extreme political ideologies.

raw_content:
  ---
  tags: [ecopolsoc, internet]
  created: Friday, November 22, 2024
  ---

  # a771a6d9_defining_internet_enclosure

  ## Defining _enclosure_

  The attempt by capital interests to encircle, control and contain networked
  digital communication with the object of extracting financial value from the
  exchanges and the community from which they arise.

  This constitutes the commodification of a human-constituted resource, like
  labour rather than a natural resource, like oil.

  ## Main characteristics

  ### Strategic denial of alternative communication media

  Where there is a _multiplicity_ of media and within media, a multiplicity of
  channels, capital steers communication toward a _single_ entity.

  In a "democratic free-market society", capital cannot simply ban alternative
  media, it can however work to diminish the influence of alternatives through the
  strategic denial of interoperability and the aquisition and subsumption of
  rivals into its system.

  ### Tyrannical control of the rules of communication

  Where the standards of communication are elective and established organically
  and collaboratively between peers (possibly even democratically), capital
  enforces rules derived externally and applied in an authoritarian manner. These
  rules are mutable but changes are enacted unilaterally and arbitrarily by the
  owners of the medium, not its users. Hence, tyranny in its technical political
  sense.

  ### Total control of information and content

  Where individuals share information freely in collaborative enterprise with
  diminished concern for ownership (typically favouring citation and iteration as
  models), capital takes full possession of the information generated such that it
  can be removed and destroyed at will without consultation.

  ### Surveillance

  Where communication is anonymised (loosly or strictly (i.e. through
  cryptographic means)), capital _surveils_ all exchanges and the behaviour of
  community members.

  Typically this is not _personal_ in the sense of being interested in the
  specific user's identity, rather it is abstracted into a composite of the
  person's preferences which may or may not disclose their legal identity (i.e
  metadata).

  This characteristic is both an _attribute_ of internet enclosure and its
  ultimate reason for existence. It is critical to the mechanism of extraction
  since it provides both data that can be sold to other capital interests and
  content that can be used to sell to community members as consumers.

  ### Artificial promotion of commercial content and prolongation of engagement

  Where engagement with media is idiosyncratic and organic, capital circumscribes
  the field. This is done to prolong engagement.

  At one pole this is achieved through the excitation of visceral responses by
  artificially favouring content that titillates, disgusts or angers the user. At
  the other, it is achieved through mollification and the coddling of the user.
  The user encounters only like-minded fellows who affirm and repeat his existing
  beliefs and values, creating community and a sense of safety through the
  othering of contrary viewpoints. Both techniques are demagogic in nature
  familiar from the control mechanisms of cults and extreme political ideologies.


path: Baran_distributed_networks.md
sortable_path: Baran_distributed_networks.md
title: 385af4b4_Baran_distributed_networks
lead: ## Paul Baran's concept of a distributed network and influence on architecture of the ARPANET
checksum: 5137724945b6cbe0f9574feae0b76780c6a3f2a0aa0ada179ce335c4dc3d1c74
created: 2025-08-09 13:41:46.754645421+00:00
modified: 2025-08-10 14:08:47.133438683+00:00
metadata: {"created":"Tuesday, October 22, 2024","tags":["fleeting","networks","ARPANET","packet-switching"]}
body:
  ## Paul Baran's concept of a distributed network and influence on architecture of the ARPANET

  Baran worked for RAND not ARPA and his work predated the commencement of the
  ARPANET. However his model of a distributed network was highly influential on
  the design of the ARPANET and served as a consultant during the initial
  construction.

  Unlike ARPA, Baran was concerned with command and control and how a
  communication network could be resiliant enough to withstand nuclear conflict.
  He analysed the weaknesses in AT&T's telephone network and proposed a more
  robust and resiliant network model.

  If we think of the nodes of the phone network as being loci for the transferral
  of data, the phone network was highly centralised. The message would go from its
  transmitter to a central switch which would route it to the intended receiver:

  ![Centralised network](../img/centralised.png)

  The problem is obvious: if the central switch went down, all data transferance
  stop.

  In fact this is an over-simplification. The phone network at the time was
  actually decentralised:

  ![Decentralised network](../img/decentralised.png)

  This system is an improvement because each region has a switch and not all nodes
  are dependent on the central switch at all times. If the central switch goes
  down, the peripheral switches can still manage connections within their regions.
  It just means that trans-regional messages are not longer possible. Conversely,
  if one regional switch goes down, the other regions are not affected and can
  communicate between themselves and each other. So there is greater resiliance
  but still points of failure - at the central and regional switches.

  Baran conceived a _distributed_ network as his model for the most resiliant
  communications network. Instead of centralised or decentralised switches, nodes
  would connect to eachother in aggregate:

  ![Distributed network](../img/distributed.png)

  Under this system, every node can eventually access every other node through
  intermediary nodes, without going through a main switch that establishes the
  connection.

  This is far more resiliant than the previous two network types because if a
  single node or multiple nodes go down, the message can be re-routed through
  others.

  Furthermore, by distributing the connections between nodes, the overall transfer
  of data can be more efficient as there isn't a bottleneck of centralised
  switches that can block up at times of high network activity. If one pathway is
  blocked, another route can be taken.

  The distributed model also opens up the opportunity to optimise transmission.
  For a given node in the network, it becomes possible to find the shortest route
  to another node. This would be very important in designing the IMP nodes of the
  ARPANET.

raw_content:
  ---
  tags: [fleeting, networks, ARPANET, packet-switching]
  created: Tuesday, October 22, 2024
  ---

  # 385af4b4_Baran_distributed_networks

  ## Paul Baran's concept of a distributed network and influence on architecture of the ARPANET

  Baran worked for RAND not ARPA and his work predated the commencement of the
  ARPANET. However his model of a distributed network was highly influential on
  the design of the ARPANET and served as a consultant during the initial
  construction.

  Unlike ARPA, Baran was concerned with command and control and how a
  communication network could be resiliant enough to withstand nuclear conflict.
  He analysed the weaknesses in AT&T's telephone network and proposed a more
  robust and resiliant network model.

  If we think of the nodes of the phone network as being loci for the transferral
  of data, the phone network was highly centralised. The message would go from its
  transmitter to a central switch which would route it to the intended receiver:

  ![Centralised network](../img/centralised.png)

  The problem is obvious: if the central switch went down, all data transferance
  stop.

  In fact this is an over-simplification. The phone network at the time was
  actually decentralised:

  ![Decentralised network](../img/decentralised.png)

  This system is an improvement because each region has a switch and not all nodes
  are dependent on the central switch at all times. If the central switch goes
  down, the peripheral switches can still manage connections within their regions.
  It just means that trans-regional messages are not longer possible. Conversely,
  if one regional switch goes down, the other regions are not affected and can
  communicate between themselves and each other. So there is greater resiliance
  but still points of failure - at the central and regional switches.

  Baran conceived a _distributed_ network as his model for the most resiliant
  communications network. Instead of centralised or decentralised switches, nodes
  would connect to eachother in aggregate:

  ![Distributed network](../img/distributed.png)

  Under this system, every node can eventually access every other node through
  intermediary nodes, without going through a main switch that establishes the
  connection.

  This is far more resiliant than the previous two network types because if a
  single node or multiple nodes go down, the message can be re-routed through
  others.

  Furthermore, by distributing the connections between nodes, the overall transfer
  of data can be more efficient as there isn't a bottleneck of centralised
  switches that can block up at times of high network activity. If one pathway is
  blocked, another route can be taken.

  The distributed model also opens up the opportunity to optimise transmission.
  For a given node in the network, it becomes possible to find the shortest route
  to another node. This would be very important in designing the IMP nodes of the
  ARPANET.


path: Berners_Lee_Motivation.md
sortable_path: Berners_Lee_Motivation.md
title: 3f1b9fe5_motivation_Berners_Lee
checksum: 18f3ace9839c6f27213b3ec1e7513adc9c4addb1bc0c71000f94872daab49de1
created: 2025-08-09 13:41:46.761342458+00:00
modified: 2025-08-10 14:08:47.106701218+00:00
metadata: {"created":"Monday, October 28, 2024","tags":["world-wide-web","internet"]}
lead:
  TBL describes his motivation in creating the WWW as the desire to marry two
  separate concepts/technologies: hypertext and the Internet, after both had come
  of age.

body:
  TBL describes his motivation in creating the WWW as the desire to marry two
  separate concepts/technologies: hypertext and the Internet, after both had come
  of age.

raw_content:
  ---
  tags: [world-wide-web, internet]
  created: Monday, October 28, 2024
  ---

  # 3f1b9fe5_motivation_Berners_Lee

  TBL describes his motivation in creating the WWW as the desire to marry two
  separate concepts/technologies: hypertext and the Internet, after both had come
  of age.


path: Dissolution_of_ARPANET.md
sortable_path: Dissolution_of_ARPANET.md
title: 82f9c228_dissolution_of_ARPANET
checksum: 1213ce22718441dcb3e35d33c411d9e4fd3318a07cf2302548756d0cda3eeac4
created: 2025-08-09 13:41:46.768711575+00:00
modified: 2025-08-10 14:08:47.083537562+00:00
metadata: {"created":"Sunday, October 27, 2024","tags":["internet","ARPANET"]}
lead:
  By 1984, the US military became concerned about the expansion of the ARPANET and
  its growing non-military applications (see the film _War Games_ released in
  1983!). They split the network into the _ARPANET_ for academic research and the
  _MILNET_ for non-civilian users where transmission was encrypted and protected.

body:
  By 1984, the US military became concerned about the expansion of the ARPANET and
  its growing non-military applications (see the film _War Games_ released in
  1983!). They split the network into the _ARPANET_ for academic research and the
  _MILNET_ for non-civilian users where transmission was encrypted and protected.

  Throughout the 1980s, the US National Science Foundation began promoting private
  networks that used the ARPANET as their _backbone_. (Private in the sense of not
  publicly-funded.)

  Later, these became commercial and
  [subscription-based](cfbef1c4_web_precursors.md). A plethora of ..._NETS_ grew
  up (NSFNET, USENET, CSNET, TELENET etc.), all based on #packet-switching, the
  ARPANET nodes as the backbone and [TCP/IP](Link_Layer_of_Internet_Protocol.md)
  host communication.

  These networks, although accessible to the public through subscriptions,
  remained unwieldy and inconsistent to access until the Web.

raw_content:
  ---
  tags: [internet, ARPANET]
  created: Sunday, October 27, 2024
  ---

  # 82f9c228_dissolution_of_ARPANET

  By 1984, the US military became concerned about the expansion of the ARPANET and
  its growing non-military applications (see the film _War Games_ released in
  1983!). They split the network into the _ARPANET_ for academic research and the
  _MILNET_ for non-civilian users where transmission was encrypted and protected.

  Throughout the 1980s, the US National Science Foundation began promoting private
  networks that used the ARPANET as their _backbone_. (Private in the sense of not
  publicly-funded.)

  Later, these became commercial and
  [subscription-based](cfbef1c4_web_precursors.md). A plethora of ..._NETS_ grew
  up (NSFNET, USENET, CSNET, TELENET etc.), all based on #packet-switching, the
  ARPANET nodes as the backbone and [TCP/IP](Link_Layer_of_Internet_Protocol.md)
  host communication.

  These networks, although accessible to the public through subscriptions,
  remained unwieldy and inconsistent to access until the Web.


path: Effects_of_privacy_violation.md
sortable_path: Effects_of_privacy_violation.md
title: 747589ad_effects_of_privacy_violation
checksum: c2ddc920f3ad949a4524a43d94ece19d3de464e8709332e389b91bf1fe07f8f5
created: 2025-08-09 13:41:46.790716436+00:00
modified: 2025-08-10 14:08:47.263973389+00:00
metadata: {"created":"Thursday, January 09, 2025","tags":["ecopolsoc","privacy","surveillance-capitalism"]}
lead:
  _What surveillance really is, at its root, is a highly effective form of social
  control. The knowledge of always being watched changes our behavior and stifles
  dissent. The inability to associate secretly means there is no longer any
  possibility for free association. The inability to whisper means there is no
  longer any speech that is truly free of coercion, real or implied. Most
  profoundly, pervasive surveillance threatens to eliminate the most vital element
  of both democracy and social movements: the mental space for people to form
  dissenting and unpopular views._

body:
  _What surveillance really is, at its root, is a highly effective form of social
  control. The knowledge of always being watched changes our behavior and stifles
  dissent. The inability to associate secretly means there is no longer any
  possibility for free association. The inability to whisper means there is no
  longer any speech that is truly free of coercion, real or implied. Most
  profoundly, pervasive surveillance threatens to eliminate the most vital element
  of both democracy and social movements: the mental space for people to form
  dissenting and unpopular views._

  - https://riseup.net/en/about-us/newsletter/2013/08#riseup-and-the-recent-email-provider-closures

  _If you live in a surveillance state for long enough, you create a censor in
  your head. When video cameras are routine fixtures in urban landscapes; when
  corporate Internet giants store records of online navigation and communication;
  and when managers and bosses maintain capabilities to measure and monitor
  employees as never before, society at large pays the price. These different
  vectors of surveillance aggregate, exerting a pressure for us to blend in, to
  think twice before speaking out, to, in essence, follow a narrow set of
  prescribed norms. Social conformity encourages quiet resignation and discourages
  the experimental - and necessarily risky - acts of speaking, thinking, and doing
  required for healthy democratic dissent._

  Gabriella Coleman. _Hacker, Hoaxer, Whistleblower, Spy_ (2014): 380

raw_content:
  ---
  tags: [ecopolsoc, privacy, surveillance-capitalism]
  created: Thursday, January 09, 2025
  ---

  # 747589ad_effects_of_privacy_violation

  _What surveillance really is, at its root, is a highly effective form of social
  control. The knowledge of always being watched changes our behavior and stifles
  dissent. The inability to associate secretly means there is no longer any
  possibility for free association. The inability to whisper means there is no
  longer any speech that is truly free of coercion, real or implied. Most
  profoundly, pervasive surveillance threatens to eliminate the most vital element
  of both democracy and social movements: the mental space for people to form
  dissenting and unpopular views._

  - https://riseup.net/en/about-us/newsletter/2013/08#riseup-and-the-recent-email-provider-closures

  _If you live in a surveillance state for long enough, you create a censor in
  your head. When video cameras are routine fixtures in urban landscapes; when
  corporate Internet giants store records of online navigation and communication;
  and when managers and bosses maintain capabilities to measure and monitor
  employees as never before, society at large pays the price. These different
  vectors of surveillance aggregate, exerting a pressure for us to blend in, to
  think twice before speaking out, to, in essence, follow a narrow set of
  prescribed norms. Social conformity encourages quiet resignation and discourages
  the experimental - and necessarily risky - acts of speaking, thinking, and doing
  required for healthy democratic dissent._

  Gabriella Coleman. _Hacker, Hoaxer, Whistleblower, Spy_ (2014): 380


path: From_ARPANET_to_Internet.md
sortable_path: From_ARPANET_to_Internet.md
title: 4da22434_from_ARPANET_to_Internet
lead:
  By the mid-1970s many additional computer networks had emerged which also used
  #packet-switching.

checksum: 7e52f41b102cdf98731fa8105feb6c0d0d6384401194b8b0264211a08f8959b2
created: 2025-08-09 13:41:46.796647063+00:00
modified: 2025-08-10 14:08:47.047000162+00:00
metadata: {"created":"Sunday, October 27, 2024","tags":["internet","ARPANET"]}
body:
  By the mid-1970s many additional computer networks had emerged which also used
  #packet-switching.

  It was the success of ARPANET as a "proof of concept" for packet-switching that
  led to these additional networks being created in other countries and contexts.
  Some used different modes of transmission than the #ARPANET's phone-wire based
  system such as radio waves and satellites. However this created a problem: these
  networks were siloed from each other and a bridge was needed to connect them.

  Those working on the ARPANET steering committee (the Information Processing
  Techniques Office (IPTO) within ARPA/DARPA) began to discuss what they called a
  "concatenated network" or "CATENET" that would connect these different networks
  together. Thus emerged the idea of a _network of networks_ or _inter-network_.

  For this to work it would be necessary to construct "gateways": routing
  computers that would hand-off messages from one network to another.

  This required a new protocol because the Network Control Protocol was only
  designed to work _within_ the ARPANET with its IMPs. Unlike the IMPs, the
  gateways would not be responsible for reconstituting packets or ensuring
  reliability, this would be shifted to the hosts. The protocol was the
  [Transmission Control Protocol/Internet Protocol](Internet_Layer_of_Internet_Protocol.md).
  TCP would be responsible for packet assembly and disassembly and IP for routing.

  The "common language" of TCP/IP and the growth in networks in addition to the
  ARPANET is what formed the Internet. Different commercial groups and research
  enterprises inaugurated their own networks and, one-by-one, they adopted TCP and
  became accessible to each other.

  As the ARPANET was retired, its individual sites either became nodes within the
  Internet or they split off into sub-networks such as MILNET (which was part of
  the ARPANET retained for classified military communication). The IMPs were
  retired in favour of TCP/IP routers.

  _The ARPANET in 1973_

  ![](../img/Arpanet_map_1973.jpg)

  _The ARPANET as part of an inter-network 1977_

  ![](../img/internetworking_1977.jpg)

  _The ARPANET in 1977_

  ![](../img/Arpanet_logical_map,_march_1977.png)

  _The Internet in 1984_

  ![](../img/internet_1984.jpg)

raw_content:
  ---
  tags: [internet, ARPANET]
  created: Sunday, October 27, 2024
  ---

  # 4da22434_from_ARPANET_to_Internet

  By the mid-1970s many additional computer networks had emerged which also used
  #packet-switching.

  It was the success of ARPANET as a "proof of concept" for packet-switching that
  led to these additional networks being created in other countries and contexts.
  Some used different modes of transmission than the #ARPANET's phone-wire based
  system such as radio waves and satellites. However this created a problem: these
  networks were siloed from each other and a bridge was needed to connect them.

  Those working on the ARPANET steering committee (the Information Processing
  Techniques Office (IPTO) within ARPA/DARPA) began to discuss what they called a
  "concatenated network" or "CATENET" that would connect these different networks
  together. Thus emerged the idea of a _network of networks_ or _inter-network_.

  For this to work it would be necessary to construct "gateways": routing
  computers that would hand-off messages from one network to another.

  This required a new protocol because the Network Control Protocol was only
  designed to work _within_ the ARPANET with its IMPs. Unlike the IMPs, the
  gateways would not be responsible for reconstituting packets or ensuring
  reliability, this would be shifted to the hosts. The protocol was the
  [Transmission Control Protocol/Internet Protocol](Internet_Layer_of_Internet_Protocol.md).
  TCP would be responsible for packet assembly and disassembly and IP for routing.

  The "common language" of TCP/IP and the growth in networks in addition to the
  ARPANET is what formed the Internet. Different commercial groups and research
  enterprises inaugurated their own networks and, one-by-one, they adopted TCP and
  became accessible to each other.

  As the ARPANET was retired, its individual sites either became nodes within the
  Internet or they split off into sub-networks such as MILNET (which was part of
  the ARPANET retained for classified military communication). The IMPs were
  retired in favour of TCP/IP routers.

  _The ARPANET in 1973_

  ![](../img/Arpanet_map_1973.jpg)

  _The ARPANET as part of an inter-network 1977_

  ![](../img/internetworking_1977.jpg)

  _The ARPANET in 1977_

  ![](../img/Arpanet_logical_map,_march_1977.png)

  _The Internet in 1984_

  ![](../img/internet_1984.jpg)


path: From_free_software_to_open_source.md
sortable_path: From_free_software_to_open_source.md
title: d212acdb_from_free-software_to_open_source
checksum: 0368b86ffeff2b8dcb00806f86aea6a1d910c7356d218a7f5af2dbac9eb89344
created: 2025-08-09 13:41:46.811774345+00:00
modified: 2025-08-10 14:08:47.072723861+00:00
metadata: {"created":"Thursday, March 27, 2025","tags":["history","ecopolsoc","FOSS"]}
lead:
  For the decade or so after Stallman's "Initial Announcement" of GNU in 1983,
  free software and the philosophy behind it remained the preserve of the
  technical underground. During these years, the phrase "open source" did not
  exist.

body:
  For the decade or so after Stallman's "Initial Announcement" of GNU in 1983,
  free software and the philosophy behind it remained the preserve of the
  technical underground. During these years, the phrase "open source" did not
  exist.

  Many engineers were using and contributing to free software in their free-time
  and surreptitiously using it in the corporate context but there was little
  awareness of it amongst the business class. If they did know about GNU-Linux
  they did not see it as a viable alternative to proprietary software, due to the
  lack of licensing, customer support and the misapprehension that the product
  would be inferior.

  The turning point came when
  [Netscape](./c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md), in a last ditch
  effort to resist Microsoft's obliteration of their market, made their
  source-code open. This caught the attention of the corporate class and the
  additional media attention led to a greater interest amongst capitalists
  generally.

  They quickly surmised that this could be an avenue of profit maximisation:
  free-labour, no licensing costs etc. As part of this process, free software was
  rebranded as "open source", all the better to calm investors' worries about its
  socialistic flavour having misread "free" as meaning "for no cost".

  Interestingly Microsoft publicly disparaged Linux and free software generally in
  public but an employee leaked internal memoranda which showed them to be most
  concerned. They led a vigorous PR campaign of disinformation about free
  software, tying it to the emerging phenomenon of "illegal file sharing". This
  was propitious timing, as it coincided with the passing of the DMCA (Digital
  Millennium Copyright Act) in the United States which introduced punitive
  punishment for those caught distributing proprietary media and software.

  So this is really the origin of hackers' antipathy to Microsoft and it's
  interesting to reflect on this today where Microsoft has very effectively
  subsumed "open source" into itself, realising it couldn't beat it on its own
  terms. Most notably with their acquisition of GitHub.

  ## Source

  Gabriella Coleman, _Coding Freedom: The Ethics and Aesthetics of Hacking_ (2013)

raw_content:
  ---
  tags: [history, ecopolsoc, FOSS]
  created: Thursday, March 27, 2025
  ---

  # d212acdb_from_free-software_to_open_source

  For the decade or so after Stallman's "Initial Announcement" of GNU in 1983,
  free software and the philosophy behind it remained the preserve of the
  technical underground. During these years, the phrase "open source" did not
  exist.

  Many engineers were using and contributing to free software in their free-time
  and surreptitiously using it in the corporate context but there was little
  awareness of it amongst the business class. If they did know about GNU-Linux
  they did not see it as a viable alternative to proprietary software, due to the
  lack of licensing, customer support and the misapprehension that the product
  would be inferior.

  The turning point came when
  [Netscape](./c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md), in a last ditch
  effort to resist Microsoft's obliteration of their market, made their
  source-code open. This caught the attention of the corporate class and the
  additional media attention led to a greater interest amongst capitalists
  generally.

  They quickly surmised that this could be an avenue of profit maximisation:
  free-labour, no licensing costs etc. As part of this process, free software was
  rebranded as "open source", all the better to calm investors' worries about its
  socialistic flavour having misread "free" as meaning "for no cost".

  Interestingly Microsoft publicly disparaged Linux and free software generally in
  public but an employee leaked internal memoranda which showed them to be most
  concerned. They led a vigorous PR campaign of disinformation about free
  software, tying it to the emerging phenomenon of "illegal file sharing". This
  was propitious timing, as it coincided with the passing of the DMCA (Digital
  Millennium Copyright Act) in the United States which introduced punitive
  punishment for those caught distributing proprietary media and software.

  So this is really the origin of hackers' antipathy to Microsoft and it's
  interesting to reflect on this today where Microsoft has very effectively
  subsumed "open source" into itself, realising it couldn't beat it on its own
  terms. Most notably with their acquisition of GitHub.

  ## Source

  Gabriella Coleman, _Coding Freedom: The Ethics and Aesthetics of Hacking_ (2013)


path: Host_protocols_of_ARPANET.md
sortable_path: Host_protocols_of_ARPANET.md
title: 653c0b1d_host_protocols_of_ARPANET
checksum: b73ca9d5f96d7c34a742f9769cd75e1c91208f05f5a176784e63462be7724e76
created: 2025-08-09 13:41:46.826292529+00:00
modified: 2025-08-10 14:08:47.025502149+00:00
metadata: {"created":"Wednesday, October 23, 2024","tags":["ARPANET","computer-history","networks"]}
lead:
  The responsibility for enabling communication between nodes fell to the hosts,
  not the [IMPs](e470bf3d_IMPs_in_the_ARPANET.md). At the lowest level (equivalent
  to the [link layer](Link_Layer_of_Internet_Protocol.md) in the subsequent
  internet?), this consisted in moving raw bits between hosts regardless of what
  specific type of data they encoded.

body:
  The responsibility for enabling communication between nodes fell to the hosts,
  not the [IMPs](e470bf3d_IMPs_in_the_ARPANET.md). At the lowest level (equivalent
  to the [link layer](Link_Layer_of_Internet_Protocol.md) in the subsequent
  internet?), this consisted in moving raw bits between hosts regardless of what
  specific type of data they encoded.

  In the book (_Where Wizards Stay Up Late_), there is a great description by
  analogy:

  > Some water out of the tap is used for making coffee, some for washing dishes,
  > some for bathing. The pipe and the faucet do not care about this; they convey
  > the water regardless.

raw_content:
  ---
  tags: [ARPANET, computer-history, networks]
  created: Wednesday, October 23, 2024
  ---

  # 653c0b1d_host_protocols_of_ARPANET

  The responsibility for enabling communication between nodes fell to the hosts,
  not the [IMPs](e470bf3d_IMPs_in_the_ARPANET.md). At the lowest level (equivalent
  to the [link layer](Link_Layer_of_Internet_Protocol.md) in the subsequent
  internet?), this consisted in moving raw bits between hosts regardless of what
  specific type of data they encoded.

  In the book (_Where Wizards Stay Up Late_), there is a great description by
  analogy:

  > Some water out of the tap is used for making coffee, some for washing dishes,
  > some for bathing. The pipe and the faucet do not care about this; they convey
  > the water regardless.


path: IMPs_in_the_ARPANET.md
sortable_path: IMPs_in_the_ARPANET.md
title: e470bf3d_IMPs_in_the_ARPANET
checksum: 8e2cc9c8714a45239d8cb1752f4f5ed10ded442706db964d1a40286f7fafed8f
created: 2025-08-09 13:41:46.84782443+00:00
modified: 2025-08-10 14:08:47.344611217+00:00
metadata: {"created":"Friday, October 18, 2024","tags":["ARPANET","networks","computer-history"]}
lead:
  Within the #ARPANET, IMPs (Information Message Processors) were nodes
  distinguished from [hosts](Network_hosts.md). They were computers designed to
  receive and route the the packets from one host to another, described as
  "adaptive routing".

body:
  Within the #ARPANET, IMPs (Information Message Processors) were nodes
  distinguished from [hosts](Network_hosts.md). They were computers designed to
  receive and route the the packets from one host to another, described as
  "adaptive routing".

  Their job was to:

  - packetize or reassemble the data from the host
  - find the most efficient pathway from one host to another
  - store packets in their buffers whilst attending to other requests
  - retry the transmission a set number of times in the case of error or data
    corruption
  - confirm successful transmission

  These responsibilities required the IMP to "know" the location and addresses of
  other IMPs and hosts on the [network](Network_fundamentals.md) and their
  proximity to them.

  From the point of view of the host user, the existence of the IMPs would be
  hidden. Hosts would connect to eachother transparently - it would appear as if
  you were directly connecting to the other host rather than an IMP.

  Thus an IMP was a subnetwork within the broader ARPANET: the IMP would be the
  core of this subnetwork and the links to its direct hosts, the periperhy.
  Initially, in its full extent, the ARPANET comprised a network of these
  subnetworks.

  ![IMP diagram](../img/IMP-network.png)

  IMPs were later known as 'gateways' and were the precursors to what we today
  call routers.

raw_content:
  ---
  tags: [ARPANET, networks, computer-history]
  created: Friday, October 18, 2024
  ---

  # e470bf3d_IMPs_in_the_ARPANET

  Within the #ARPANET, IMPs (Information Message Processors) were nodes
  distinguished from [hosts](Network_hosts.md). They were computers designed to
  receive and route the the packets from one host to another, described as
  "adaptive routing".

  Their job was to:

  - packetize or reassemble the data from the host
  - find the most efficient pathway from one host to another
  - store packets in their buffers whilst attending to other requests
  - retry the transmission a set number of times in the case of error or data
    corruption
  - confirm successful transmission

  These responsibilities required the IMP to "know" the location and addresses of
  other IMPs and hosts on the [network](Network_fundamentals.md) and their
  proximity to them.

  From the point of view of the host user, the existence of the IMPs would be
  hidden. Hosts would connect to eachother transparently - it would appear as if
  you were directly connecting to the other host rather than an IMP.

  Thus an IMP was a subnetwork within the broader ARPANET: the IMP would be the
  core of this subnetwork and the links to its direct hosts, the periperhy.
  Initially, in its full extent, the ARPANET comprised a network of these
  subnetworks.

  ![IMP diagram](../img/IMP-network.png)

  IMPs were later known as 'gateways' and were the precursors to what we today
  call routers.


path: Key_figures_ARPANET.md
sortable_path: Key_figures_ARPANET.md
title: 4a3dc316_key_figures_ARPANET
checksum: 0b781afb66f4731f64577a49ea652afc95660de8a0054f8c9728ef2baaf05c61
created: 2025-08-09 13:41:46.867458574+00:00
modified: 2025-08-10 14:08:47.112108822+00:00
metadata: {"created":"Monday, October 28, 2024","tags":["internet","ARPANET","ARPA"]}
lead:
  - [Paul Baran](385af4b4_Baran_distributed_networks.md) for his work at RAND on
    distributed networks as an alternative to the extant
    centralised/de-centralised phone networks.

body:
  - [Paul Baran](385af4b4_Baran_distributed_networks.md) for his work at RAND on
    distributed networks as an alternative to the extant
    centralised/de-centralised phone networks.

  - Donald Davies at the UK Physical Laboratory for developing ideas on
    #packet-switching, similar to but derived independently of Baran's work.

  - Bob Kahn as the principle designer of the ARPANET, who with Vint Cerf, wrote
    the TCP/IP protocols. Became director of the Information Processing Techniques
    Office within ARPA/DARPA.

  - Vint Cerf was head of the International Networking Group and worked on the
    Network Control Protocol (the first protocol for packet switching using the
    IMPs). Later joined up with Kahn on TCP/IP and the network of packet-switching
    networks (early internet).

raw_content:
  ---
  tags: [internet, ARPANET, ARPA]
  created: Monday, October 28, 2024
  ---

  # 4a3dc316_key_figures_ARPANET

  - [Paul Baran](385af4b4_Baran_distributed_networks.md) for his work at RAND on
    distributed networks as an alternative to the extant
    centralised/de-centralised phone networks.

  - Donald Davies at the UK Physical Laboratory for developing ideas on
    #packet-switching, similar to but derived independently of Baran's work.

  - Bob Kahn as the principle designer of the ARPANET, who with Vint Cerf, wrote
    the TCP/IP protocols. Became director of the Information Processing Techniques
    Office within ARPA/DARPA.

  - Vint Cerf was head of the International Networking Group and worked on the
    Network Control Protocol (the first protocol for packet switching using the
    IMPs). Later joined up with Kahn on TCP/IP and the network of packet-switching
    networks (early internet).


path: Mosaic_Netscape_and_Browser_Wars.md
sortable_path: Mosaic_Netscape_and_Browser_Wars.md
title: c301a0b3_Mosaic_Netscape_and_Browser_Wars
checksum: 2932435c9c734455a7996da2c686501f21609773035248164eb5f8b8cc59abf8
created: 2025-08-09 13:41:46.892365925+00:00
modified: 2025-08-10 14:08:47.318408295+00:00
metadata: {"created":"Sunday, November 03, 2024","tags":["world-wide-web","computer-history","Microsoft"]}
lead:
  The Mosaic browser was conceived and produced at the University of Illinois
  during the very early growth of the Web, when TBL was still at CERN.

body:
  The Mosaic browser was conceived and produced at the University of Illinois
  during the very early growth of the Web, when TBL was still at CERN.

  _Mosaic_:

  ![Mosaic browser](../img/mosaic-browser.jpg)

  It became the best of the early browsers (a group that included ViolaWWW and
  Erwise) with the notable advantage of being able to embed graphics alongside
  text (instead of inside a separate window). It also supported multiple
  [application layer](Application_Layer_of_Internet_Protocol.md) protocols - FTP,
  the Network News Transfer Protocol, Gopher - not just HTTP.

  At this point there were nascent browsers for different operating systems and
  which browser you used was mostly determined by your OS. Being the product of a
  US university, Mosaic was originally written for Unix's X-Window system but was
  ported to multiple OSs later, a key factor in its success and uptake.

  Mosaic's creators formed their own company outside of the university: Netscape.
  The browser they created, Netscape Navigator became the heir to Mosaic, released
  in 1994.

  _Netscape Navigator_:

  ![Netscape Navigator](../img/Navigator_Netscape.png)

  It was made free of charge for non-commercial use which assured rapid uptake.
  This was done to outrun Microsoft's Internet Explorer (released two months after
  Netscape went public) which they intended to bundle with all new versions of
  Windows 95. Netscape became the de facto standard browser during the early Web,
  including on Windows. It took a few years for IE to overtake it.

  _Internet Explorer 1 with licensing info_:

  ![Internet Explorer](../img/internet_explorer.png)

  Netscape also birthed JavaScript, which was written rapidly specifically for the
  browser. Prior to this, dynamic client-side functionality was provided by Java
  'applets' which never really took off.

  IE in fact shares a genealogy with Mosaic since Microsoft licensed a version of
  the Mosaic code and used it to construct IE.

  By the end of the 1990s, Netscape's usage had declined precipitously as IE
  captured the market share.

raw_content:
  ---
  tags: [world-wide-web, computer-history, Microsoft]
  created: Sunday, November 03, 2024
  ---

  # c301a0b3_Mosaic_Netscape_and_Browser_Wars

  The Mosaic browser was conceived and produced at the University of Illinois
  during the very early growth of the Web, when TBL was still at CERN.

  _Mosaic_:

  ![Mosaic browser](../img/mosaic-browser.jpg)

  It became the best of the early browsers (a group that included ViolaWWW and
  Erwise) with the notable advantage of being able to embed graphics alongside
  text (instead of inside a separate window). It also supported multiple
  [application layer](Application_Layer_of_Internet_Protocol.md) protocols - FTP,
  the Network News Transfer Protocol, Gopher - not just HTTP.

  At this point there were nascent browsers for different operating systems and
  which browser you used was mostly determined by your OS. Being the product of a
  US university, Mosaic was originally written for Unix's X-Window system but was
  ported to multiple OSs later, a key factor in its success and uptake.

  Mosaic's creators formed their own company outside of the university: Netscape.
  The browser they created, Netscape Navigator became the heir to Mosaic, released
  in 1994.

  _Netscape Navigator_:

  ![Netscape Navigator](../img/Navigator_Netscape.png)

  It was made free of charge for non-commercial use which assured rapid uptake.
  This was done to outrun Microsoft's Internet Explorer (released two months after
  Netscape went public) which they intended to bundle with all new versions of
  Windows 95. Netscape became the de facto standard browser during the early Web,
  including on Windows. It took a few years for IE to overtake it.

  _Internet Explorer 1 with licensing info_:

  ![Internet Explorer](../img/internet_explorer.png)

  Netscape also birthed JavaScript, which was written rapidly specifically for the
  browser. Prior to this, dynamic client-side functionality was provided by Java
  'applets' which never really took off.

  IE in fact shares a genealogy with Mosaic since Microsoft licensed a version of
  the Mosaic code and used it to construct IE.

  By the end of the 1990s, Netscape's usage had declined precipitously as IE
  captured the market share.


path: Network_Layer_clarification.md
sortable_path: Network_Layer_clarification.md
title: a0ab0bfb_network_layer_clarification
checksum: 6fe734cba1763497e75b107efeb4f7ae8e97f179505adc48e73803a77e7c5ed3
created: 2025-08-09 13:41:46.906240601+00:00
modified: 2025-08-10 14:08:47.206367443+00:00
metadata: {"created":"Sunday, March 02, 2025","tags":["networks","internet"]}
lead:
  I have difficulty understanding the places at which the different layers of the
  TCP/IP model of the Internet reside and their spatiality in terms of the overall
  system. These are some clarifying remarks.

body:
  I have difficulty understanding the places at which the different layers of the
  TCP/IP model of the Internet reside and their spatiality in terms of the overall
  system. These are some clarifying remarks.

  All layers exist on every device involved in the communication process. But not
  all layers are accessed by every device in the overall transaction.

  > While the entire encapsulated packet travels across the internet, different
  > devices interact with different layers of that packet based on their role in
  > the network.

  The server and client access data at all layers:

  - [Application](./Application_Layer_of_Internet_Protocol.md)
  - [Transport](./Transport_Layer_of_Internet_Protocol.md)
  - [Internet](./Internet_Layer_of_Internet_Protocol.md)
  - [Link](./Link_Layer_of_Internet_Protocol.md)

  This is because the data must be composed on the device, a route planned and
  then sent accross the network where it is processed at the local network layer.

  Routers however only care about routing the data, forwarding packets based on
  their IP addresses, they only process data up to the Internet Layer.

raw_content:
  ---
  tags: [networks, internet]
  created: Sunday, March 02, 2025
  ---

  # a0ab0bfb_network_layer_clarification

  I have difficulty understanding the places at which the different layers of the
  TCP/IP model of the Internet reside and their spatiality in terms of the overall
  system. These are some clarifying remarks.

  All layers exist on every device involved in the communication process. But not
  all layers are accessed by every device in the overall transaction.

  > While the entire encapsulated packet travels across the internet, different
  > devices interact with different layers of that packet based on their role in
  > the network.

  The server and client access data at all layers:

  - [Application](./Application_Layer_of_Internet_Protocol.md)
  - [Transport](./Transport_Layer_of_Internet_Protocol.md)
  - [Internet](./Internet_Layer_of_Internet_Protocol.md)
  - [Link](./Link_Layer_of_Internet_Protocol.md)

  This is because the data must be composed on the device, a route planned and
  then sent accross the network where it is processed at the local network layer.

  Routers however only care about routing the data, forwarding packets based on
  their IP addresses, they only process data up to the Internet Layer.


path: Nginx_vs_traefik.md
sortable_path: Nginx_vs_traefik.md
title: e383b8b3_nginx_vs_traefik
checksum: 92f468f8a952b0f9c395c853d6659984d6cb450b94215e09386f20eb510313b9
created: 2025-08-09 13:41:46.945184542+00:00
modified: 2025-08-10 14:08:47.169557342+00:00
metadata: {"created":"Monday, February 03, 2025","tags":["networks","proxies","nginx"]}
lead:
  Both nginx and traefik are server technologies (along with Apache), nginx has a
  longer pedigree but traefik is calibrated for more modern microservice-based
  architectures (i.e Docker containers). nginx is better suited to traditional
  static web-based hosting.

body:
  Both nginx and traefik are server technologies (along with Apache), nginx has a
  longer pedigree but traefik is calibrated for more modern microservice-based
  architectures (i.e Docker containers). nginx is better suited to traditional
  static web-based hosting.

  traefik has support for [HTTPS](./HTTPS.md) natively whereas nginx requires a
  plugin.

raw_content:
  ---
  tags: [networks, proxies, nginx]
  created: Monday, February 03, 2025
  ---

  # e383b8b3_nginx_vs_traefik

  Both nginx and traefik are server technologies (along with Apache), nginx has a
  longer pedigree but traefik is calibrated for more modern microservice-based
  architectures (i.e Docker containers). nginx is better suited to traditional
  static web-based hosting.

  traefik has support for [HTTPS](./HTTPS.md) natively whereas nginx requires a
  plugin.


path: Political_accomodation_inveigelment_surveillance_capitalism.md
sortable_path: Political_accomodation_inveigelment_surveillance_capitalism.md
title: fbbfbc32-political-accommodation-inveigelment-surveillance-capitalism
checksum: 68352d1a57dcb358ab8a805550c10d8022f50445c03e634f6c38c7cc423fcff6
created: 2025-08-09 13:41:46.958197295+00:00
modified: 2025-08-10 14:08:47.328609121+00:00
metadata: {"created":"Sunday, December 01, 2024","tags":["ecopolsoc","internet","surveillance-capitalism"]}
lead:
  The reason why surveillance capitalism was able to take root when it did was not
  merely down to technological forces. Google, as the originator of the form, was
  able to exploit the political currents of the time to further innoculate itself
  against scrutiny and oversight.

body:
  The reason why surveillance capitalism was able to take root when it did was not
  merely down to technological forces. Google, as the originator of the form, was
  able to exploit the political currents of the time to further innoculate itself
  against scrutiny and oversight.

  The discovery of behavioural surplus coincided with September 11 and the
  subsequent "global war on terror". The quest for domestic "security" led to the
  sweeping erosion of previous generations' privacy rights.

  Prior to 911, the FCC had been arranging a series of legal challenges to Google
  on the question of privacy. This went out the window in the post-911 era where
  Google's unique ability to track and surveil online behaviour became a coveted
  commodity for the intelligence apparatus. Google began working closely with both
  the CIA and the NSA, with the CIA establishing an office and forward base in
  Silicon Valley.

  There was an deep interdependency between the Pentagon, intelligence agencies
  and Google:

  _The US intelligence community's incubation of Google from inception occurred
  through a combination of direct sponsorship and informal networks of financial
  influence, themselves closely aligned with Pentagon interests._

  _The Age of Surveillance Capitalism_ (2019): 118

  Another key aspect was cosying up to politicians during the electoral cycle.
  Most notably during Obama's elections:

  _Google demonstrated that the same predictive knowledge derived from behavioural
  surplus that had made the surveillance capitalists wealthy could also help
  candidates win elections_

  _Ibid_: 122

  Eric Schmidt took an active role in making Google's dataset and technologies
  available to the Obama campaign in both of his elections in order to target
  voters more accurately. He then was swiftly appointed to advisory roles within
  the administration throughout both terms.

raw_content:
  ---
  tags: [ecopolsoc, internet, surveillance-capitalism]
  created: Sunday, December 01, 2024
  ---

  # fbbfbc32-political-accommodation-inveigelment-surveillance-capitalism

  The reason why surveillance capitalism was able to take root when it did was not
  merely down to technological forces. Google, as the originator of the form, was
  able to exploit the political currents of the time to further innoculate itself
  against scrutiny and oversight.

  The discovery of behavioural surplus coincided with September 11 and the
  subsequent "global war on terror". The quest for domestic "security" led to the
  sweeping erosion of previous generations' privacy rights.

  Prior to 911, the FCC had been arranging a series of legal challenges to Google
  on the question of privacy. This went out the window in the post-911 era where
  Google's unique ability to track and surveil online behaviour became a coveted
  commodity for the intelligence apparatus. Google began working closely with both
  the CIA and the NSA, with the CIA establishing an office and forward base in
  Silicon Valley.

  There was an deep interdependency between the Pentagon, intelligence agencies
  and Google:

  _The US intelligence community's incubation of Google from inception occurred
  through a combination of direct sponsorship and informal networks of financial
  influence, themselves closely aligned with Pentagon interests._

  _The Age of Surveillance Capitalism_ (2019): 118

  Another key aspect was cosying up to politicians during the electoral cycle.
  Most notably during Obama's elections:

  _Google demonstrated that the same predictive knowledge derived from behavioural
  surplus that had made the surveillance capitalists wealthy could also help
  candidates win elections_

  _Ibid_: 122

  Eric Schmidt took an active role in making Google's dataset and technologies
  available to the Obama campaign in both of his elections in order to target
  voters more accurately. He then was swiftly appointed to advisory roles within
  the administration throughout both terms.


path: Precursors_to_search_engines.md
sortable_path: Precursors_to_search_engines.md
title: ecec56e6_precursors_to_search_engines
checksum: f160dea20cd9f987f808c75db9e5931b5357007d35513bd3a30476ea42945ed3
created: 2025-08-09 13:41:46.962650192+00:00
modified: 2025-08-10 14:08:47.105509076+00:00
metadata: {"created":"Monday, November 04, 2024","tags":["world-wide-web","computer-history"]}
lead:
  Before search engines as searchable indices of all content on the web, the
  custom was for ISPs, browsers and websites (e.g. Yahoo!) to provide directories
  of links to other sites. They functioned as sort of global web-portals.

body:
  Before search engines as searchable indices of all content on the web, the
  custom was for ISPs, browsers and websites (e.g. Yahoo!) to provide directories
  of links to other sites. They functioned as sort of global web-portals.

  ![Yahoo's original homepage](../img/yahoo-original-homepage.png)

  _The Yahoo! homepage from 1999._

  ![AltaVista homepage](../img/alta-vista.png)

  _The AltaVista homepage from 1999. It has a primitive search engine but also
  directories of sites._

raw_content:
  ---
  tags: [world-wide-web, computer-history]
  created: Monday, November 04, 2024
  ---

  # ecec56e6_precursors_to_search_engines

  Before search engines as searchable indices of all content on the web, the
  custom was for ISPs, browsers and websites (e.g. Yahoo!) to provide directories
  of links to other sites. They functioned as sort of global web-portals.

  ![Yahoo's original homepage](../img/yahoo-original-homepage.png)

  _The Yahoo! homepage from 1999._

  ![AltaVista homepage](../img/alta-vista.png)

  _The AltaVista homepage from 1999. It has a primitive search engine but also
  directories of sites._


path: Public_key_cryptography.md
sortable_path: Public_key_cryptography.md
title: bbdcb54f_public_key_cryptography
checksum: c03f7d29fb3b86739ddab99d4d1e7c9f10bab31461cc9b6c3fe2f90f27c281fb
created: 2025-08-09 13:41:46.969040114+00:00
modified: 2025-08-10 14:08:47.010508372+00:00
metadata: {"created":"Friday, December 20, 2024","tags":["encryption","cryptography"]}
lead:
  _A_ wants to send a message to _B_. A wants to ensure that only _B_ can read the
  message and that it is not intercepted or altered during transmission to _B_.

body:
  _A_ wants to send a message to _B_. A wants to ensure that only _B_ can read the
  message and that it is not intercepted or altered during transmission to _B_.

  Think of _B_ as being in possession of a letterbox. The letterbox contains
  messages that only he can read. Anyone can put letters in it for _B_ to read but
  _B_ alone can read them.

  This is achieved by encyrpting the messages with two keys: a public key and a
  private key. The public key can be shared with others and anyone in possession
  of the public key can encrypt a message for _B_. Once encrypted, only _B_ can
  decrypt it using his private key. Once receiving the message, B uses his private
  secret key - that he doesn't share with anyone else - to decrypt. This is the
  equivalent to opening up the letterbox and taking out the letters.

  Another useful analogy is to think of invisible ink. Anyone can write a letter
  using the invisible ink but only the recipient has the liquid that can render it
  visible again. Anyone intercepting the message (without the liquid) will see
  nothing, equivalent to the encrypted data.

  This is asymmetric encryption because the two communicating parties do not have
  equal access to the messages exchanged. _A_ can only send messages - he can't
  decrypt them, only _B_ can. So _A_ is unequal relative to _B_. With symmetric
  encryption _A_ and _B_ would both share a single private key which they would
  each use to encrypt and decrypt their messages.

raw_content:
  ---
  tags: [encryption, cryptography]
  created: Friday, December 20, 2024
  ---

  # bbdcb54f_public_key_cryptography

  _A_ wants to send a message to _B_. A wants to ensure that only _B_ can read the
  message and that it is not intercepted or altered during transmission to _B_.

  Think of _B_ as being in possession of a letterbox. The letterbox contains
  messages that only he can read. Anyone can put letters in it for _B_ to read but
  _B_ alone can read them.

  This is achieved by encyrpting the messages with two keys: a public key and a
  private key. The public key can be shared with others and anyone in possession
  of the public key can encrypt a message for _B_. Once encrypted, only _B_ can
  decrypt it using his private key. Once receiving the message, B uses his private
  secret key - that he doesn't share with anyone else - to decrypt. This is the
  equivalent to opening up the letterbox and taking out the letters.

  Another useful analogy is to think of invisible ink. Anyone can write a letter
  using the invisible ink but only the recipient has the liquid that can render it
  visible again. Anyone intercepting the message (without the liquid) will see
  nothing, equivalent to the encrypted data.

  This is asymmetric encryption because the two communicating parties do not have
  equal access to the messages exchanged. _A_ can only send messages - he can't
  decrypt them, only _B_ can. So _A_ is unequal relative to _B_. With symmetric
  encryption _A_ and _B_ would both share a single private key which they would
  each use to encrypt and decrypt their messages.


path: Requerimiento.md
sortable_path: Requerimiento.md
title: c9d7492f_requerimiento
checksum: d6deba34cb598c5af3990c218985e07d74483368d97e451556392cb118e5e990
created: 2025-08-09 13:41:46.977487779+00:00
modified: 2025-08-10 14:08:47.21391477+00:00
metadata: {"created":"Monday, January 20, 2025","tags":["ecopolsoc","internet","surveillance-capitalism"]}
lead:
  Zuboff likens the expropriation of private information by surveillance
  capitalists to the behaviour of the Conquistadores when they first arrived in
  South America.

body:
  Zuboff likens the expropriation of private information by surveillance
  capitalists to the behaviour of the Conquistadores when they first arrived in
  South America.

  In order to legitimise their actions they were obliged to inform the native
  inhabitants of their intentions and tell them that they were now subordinate to
  the Spanish crown.

  This provided supposed justification for the violence and brutality that would
  follow if the natives did not comply. Sometimes the declaration would just be
  muttered moments before attack.

  "Conquest by declaration" is also the technique of Google, she claims. The
  equivalent of the _requerimiento_ being the opaque and incomprehensible terms of
  of service, _designed_ not to be read.

  _These twenty-first-century invaders do not ask permission; the forge ahead,
  papering the scorched earth with faux-legitimation practices. Instead of
  cynically conveyed monarchical edicts, they offer cynically conveyed
  terms-of-service agreements whose stipulations are just as obscured and
  incomprehensible. They build their fortifications, fiercely defending their
  claimed territories, while gathering strength for the next incursion.
  Eventually, they build their towns in intricate ecosystems of commerce, politics
  and culture that declare the legitimacy and inevitability of all that they have
  accomplished._

  _The Age of Surveillance Capitalism_ (2019): 179

raw_content:
  ---
  tags: [ecopolsoc, internet, surveillance-capitalism]
  created: Monday, January 20, 2025
  ---

  # c9d7492f_requerimiento

  Zuboff likens the expropriation of private information by surveillance
  capitalists to the behaviour of the Conquistadores when they first arrived in
  South America.

  In order to legitimise their actions they were obliged to inform the native
  inhabitants of their intentions and tell them that they were now subordinate to
  the Spanish crown.

  This provided supposed justification for the violence and brutality that would
  follow if the natives did not comply. Sometimes the declaration would just be
  muttered moments before attack.

  "Conquest by declaration" is also the technique of Google, she claims. The
  equivalent of the _requerimiento_ being the opaque and incomprehensible terms of
  of service, _designed_ not to be read.

  _These twenty-first-century invaders do not ask permission; the forge ahead,
  papering the scorched earth with faux-legitimation practices. Instead of
  cynically conveyed monarchical edicts, they offer cynically conveyed
  terms-of-service agreements whose stipulations are just as obscured and
  incomprehensible. They build their fortifications, fiercely defending their
  claimed territories, while gathering strength for the next incursion.
  Eventually, they build their towns in intricate ecosystems of commerce, politics
  and culture that declare the legitimacy and inevitability of all that they have
  accomplished._

  _The Age of Surveillance Capitalism_ (2019): 179


path: Rewilding_the_internet.md
sortable_path: Rewilding_the_internet.md
title: 0716531c_rewilding_the_internet
checksum: bf356c0a1ef11f4c0f98e42151962507661eb1e1b464a2a2fb1f085aa2ec5999
created: 2025-08-09 13:41:46.987592234+00:00
modified: 2025-08-10 14:08:47.348631233+00:00
metadata: {"created":"Thursday, November 07, 2024","tags":["ecopolsoc","internet"]}
lead:
  The following
  [analogy](https://www.noemamag.com/we-need-to-rewild-the-internet/) can be
  useful in understanding how the internet has become such a negative environment.
  How it has transmogrified from its original promise of a humanistic space that
  would facilitate the connection of individuals based on shared interests and
  values to an environment that is entirely mediated by capital, subject to
  control, censorship and the violation of privacy.

body:
  The following
  [analogy](https://www.noemamag.com/we-need-to-rewild-the-internet/) can be
  useful in understanding how the internet has become such a negative environment.
  How it has transmogrified from its original promise of a humanistic space that
  would facilitate the connection of individuals based on shared interests and
  values to an environment that is entirely mediated by capital, subject to
  control, censorship and the violation of privacy.

  In the 18th-century in the northern German states, officials decided to
  rationalise the production of timber to enable more systematic extraction.
  (Clearly a product of Enlightenment principles of applying reason and science to
  agriculture similar to Jethro Tull's invention of the seed drill and the
  suspension of three-field rotation to four-field all-round harvesting.)

  Instead of a sprawling forest ecosystem that had developed naturally over
  hundreds of thousands of years, which allowed for felling without upsetting the
  continuance of the system, the forests were stripped and trees were planted in
  rows and grids with the arrangement designed to maximise yield and access. This
  would be broadly analogous to the factory-farming of chickens compared to
  free-range rearing.

  For the first cycle it appeared as if this restructuring was a huge success.
  Yields were high and the timber was of excellent quality. This was a false dawn,
  however. The subsequent yields were a complete failure compared to the past.
  Fewer trees grew and those that did had a reduced yield and were subject to
  disease. The overall health of the forests declined precipitously.

  What explains the initial success? It was made possible by the preceding
  centuries of free-range growth and husbandry. The seeming success of the new
  system was a product of the generations of investment in the natural ecosystem.
  This was exhausted in one cycle of the new regime.

  Taking this back to the Web, the analogy is that the web was initially a natural
  ecosystem of different communities and technologies developing organically out
  of the original open and decentralised protocols. From this diversity emerged a
  formidably rich informational space.

  Since then, the Web has become highly centralised and homogenised with a tiny
  cadre of corporations determining how individuals communicate, share and access
  information (Google for search, Android or Apple for phone, Twitter or Facebook
  etc.).

  The initial fruits of this centralisation (just like the forest 'optimisation')
  appeared to be beneficial in the initial 'heyday' of the 2010s when social-media
  was new and full of prospect (the political benefits of Twitter and enriched
  public sphere, the 'Arab Spring', WikiLeaks etc). But this actually proved to be
  a nadir. The vitality on display was a product of the preceding era of the
  natural growth of the Web ecosystem.

  This decline in quality should be apparent to any reflective user:

  - the saturation of advertising to the point where many websites are unusable
  - the confection of low-quality content to game 'organic' search rankings
  - the surveillance of users across sites and devices to enable the extraction of
    profitable marketing data
  - the censorship of political content for the 'violation' of bogus terms of use
    on social media platforms if it does not the serve the broader interests of
    capital and western imperialism
  - the saturation of pornography and dehumanising media more generally

  Thbs has also involved the reshaping of Internet users themselves, from active
  and independent _agents_, to drone-like passive _consumers_ of drivel engineered
  to maximise 'engagement' (the monopolisation of attention), where users are like
  the viewers of _Infinite Jest_. See, e.g TikTok and its clones.

  There are many further points that can be made and ideas that can be
  extrapolated from this. Two examples:

  - the emergence of supposed 'AI' is going to lead to greater content
    homogenisation and the intensification of informational entropy as the
    technological mechanism consists in feeding the regurgitated content of the
    internet back into itself

  - reality itself has become infected by the disease that killed the Web
    (ecocide). The distinction between offline and online is becoming increasingly
    hard to draw (politics - the normalisation and ascendancy of the far right and
    subsequent destabilisation of western states)

raw_content:
  ---
  tags: [ecopolsoc, internet]
  created: Thursday, November 07, 2024
  ---

  # 0716531c_rewilding_the_internet

  The following
  [analogy](https://www.noemamag.com/we-need-to-rewild-the-internet/) can be
  useful in understanding how the internet has become such a negative environment.
  How it has transmogrified from its original promise of a humanistic space that
  would facilitate the connection of individuals based on shared interests and
  values to an environment that is entirely mediated by capital, subject to
  control, censorship and the violation of privacy.

  In the 18th-century in the northern German states, officials decided to
  rationalise the production of timber to enable more systematic extraction.
  (Clearly a product of Enlightenment principles of applying reason and science to
  agriculture similar to Jethro Tull's invention of the seed drill and the
  suspension of three-field rotation to four-field all-round harvesting.)

  Instead of a sprawling forest ecosystem that had developed naturally over
  hundreds of thousands of years, which allowed for felling without upsetting the
  continuance of the system, the forests were stripped and trees were planted in
  rows and grids with the arrangement designed to maximise yield and access. This
  would be broadly analogous to the factory-farming of chickens compared to
  free-range rearing.

  For the first cycle it appeared as if this restructuring was a huge success.
  Yields were high and the timber was of excellent quality. This was a false dawn,
  however. The subsequent yields were a complete failure compared to the past.
  Fewer trees grew and those that did had a reduced yield and were subject to
  disease. The overall health of the forests declined precipitously.

  What explains the initial success? It was made possible by the preceding
  centuries of free-range growth and husbandry. The seeming success of the new
  system was a product of the generations of investment in the natural ecosystem.
  This was exhausted in one cycle of the new regime.

  Taking this back to the Web, the analogy is that the web was initially a natural
  ecosystem of different communities and technologies developing organically out
  of the original open and decentralised protocols. From this diversity emerged a
  formidably rich informational space.

  Since then, the Web has become highly centralised and homogenised with a tiny
  cadre of corporations determining how individuals communicate, share and access
  information (Google for search, Android or Apple for phone, Twitter or Facebook
  etc.).

  The initial fruits of this centralisation (just like the forest 'optimisation')
  appeared to be beneficial in the initial 'heyday' of the 2010s when social-media
  was new and full of prospect (the political benefits of Twitter and enriched
  public sphere, the 'Arab Spring', WikiLeaks etc). But this actually proved to be
  a nadir. The vitality on display was a product of the preceding era of the
  natural growth of the Web ecosystem.

  This decline in quality should be apparent to any reflective user:

  - the saturation of advertising to the point where many websites are unusable
  - the confection of low-quality content to game 'organic' search rankings
  - the surveillance of users across sites and devices to enable the extraction of
    profitable marketing data
  - the censorship of political content for the 'violation' of bogus terms of use
    on social media platforms if it does not the serve the broader interests of
    capital and western imperialism
  - the saturation of pornography and dehumanising media more generally

  Thbs has also involved the reshaping of Internet users themselves, from active
  and independent _agents_, to drone-like passive _consumers_ of drivel engineered
  to maximise 'engagement' (the monopolisation of attention), where users are like
  the viewers of _Infinite Jest_. See, e.g TikTok and its clones.

  There are many further points that can be made and ideas that can be
  extrapolated from this. Two examples:

  - the emergence of supposed 'AI' is going to lead to greater content
    homogenisation and the intensification of informational entropy as the
    technological mechanism consists in feeding the regurgitated content of the
    internet back into itself

  - reality itself has become infected by the disease that killed the Web
    (ecocide). The distinction between offline and online is becoming increasingly
    hard to draw (politics - the normalisation and ascendancy of the far right and
    subsequent destabilisation of western states)


path: Sound_recording_as_wave_phenomena.md
sortable_path: Sound_recording_as_wave_phenomena.md
title: 738d0481_sound_recording_as_wave_phenomena
checksum: a9dc1aead9226467ca2ae21b8f7233ae0d3b326c9e2b6048e2d122ff0141b496
created: 2025-08-09 13:41:47.005926137+00:00
modified: 2025-08-10 14:08:47.023116624+00:00
metadata: {"created":"Thursday, January 02, 2025","tags":["sound","analogue","physics"]}
lead:
  When the microphone captures alterations in sound pressure and converts it into
  an electric signal, we have a scenario where the same formal or mathematical
  relationships in the phenomena (amplitude, frequency, wavelength) are being
  expressed in two different modes of matter.

body:
  When the microphone captures alterations in sound pressure and converts it into
  an electric signal, we have a scenario where the same formal or mathematical
  relationships in the phenomena (amplitude, frequency, wavelength) are being
  expressed in two different modes of matter.

  This is quite profound especially as the resulting electrical waveform is
  _identical_ to the original sound wave. This is effectively what enables
  accurate playback. We can convert between the two mediums seamlessly providing
  we captured their initial expression accurately.

  It's like two different people doing the same dance:

  In air:

  - The dancers are air molecules
  - They move back and forth in waves of compression and rarefaction
  - Their "dance pattern" is the sound wave

  In wire:

  - The dancers are electrons
  - They move back and forth creating waves of voltage
  - They perform the exact same "dance pattern"

raw_content:
  ---
  tags: [sound, analogue, physics]
  created: Thursday, January 02, 2025
  ---

  # 738d0481_sound_recording_as_wave_phenomena

  When the microphone captures alterations in sound pressure and converts it into
  an electric signal, we have a scenario where the same formal or mathematical
  relationships in the phenomena (amplitude, frequency, wavelength) are being
  expressed in two different modes of matter.

  This is quite profound especially as the resulting electrical waveform is
  _identical_ to the original sound wave. This is effectively what enables
  accurate playback. We can convert between the two mediums seamlessly providing
  we captured their initial expression accurately.

  It's like two different people doing the same dance:

  In air:

  - The dancers are air molecules
  - They move back and forth in waves of compression and rarefaction
  - Their "dance pattern" is the sound wave

  In wire:

  - The dancers are electrons
  - They move back and forth creating waves of voltage
  - They perform the exact same "dance pattern"


path: Surveillance_capitalism_not_necessary.md
sortable_path: Surveillance_capitalism_not_necessary.md
title: a247fa9b_surveillance_capitalism_not_necessary
checksum: 6982996ba7fd461a4139427f331faafde8db2ffe30532126b850a052b8f777d5
created: 2025-08-09 13:41:47.011835747+00:00
modified: 2025-08-10 14:08:47.26585822+00:00
metadata: {"created":"Sunday, December 01, 2024","tags":["ecopolsoc","internet","surveillance-capitalism"]}
lead:
  _Surveillance capitalism is neither a necessary development of information
  capitalism nor a necessary product of digital technology or the internet. It is
  a specifically constructed human choice, an unprecedented market form, an
  original solution to an emergency, and the underlying mechanism through which a
  new asset class is created on the cheap and converted to revenue._

body:
  _Surveillance capitalism is neither a necessary development of information
  capitalism nor a necessary product of digital technology or the internet. It is
  a specifically constructed human choice, an unprecedented market form, an
  original solution to an emergency, and the underlying mechanism through which a
  new asset class is created on the cheap and converted to revenue._

  _The Age of Surveillance Capitalism_ (2019): 91

raw_content:
  ---
  tags: [ecopolsoc, internet, surveillance-capitalism]
  created: Sunday, December 01, 2024
  ---

  # a247fa9b_surveillance_capitalism_not_necessary

  _Surveillance capitalism is neither a necessary development of information
  capitalism nor a necessary product of digital technology or the internet. It is
  a specifically constructed human choice, an unprecedented market form, an
  original solution to an emergency, and the underlying mechanism through which a
  new asset class is created on the cheap and converted to revenue._

  _The Age of Surveillance Capitalism_ (2019): 91


path: TIMPs.md
sortable_path: TIMPs.md
title: c8173d17_TIMPs
checksum: 8229d083e090ae2c63d332d19e6c4c8f093943706d8e20948dbdc7310a98fbef
created: 2025-08-09 13:41:47.016275951+00:00
modified: 2025-08-10 14:08:47.296088235+00:00
metadata: {"created":"Friday, October 25, 2024","tags":["ARPANET","networks","computer-history"]}
lead:
  After the initial ARPANET was complete, the next major milestone was to enable
  access to the network regardless of one's proximity to a host node with an IMP
  connection.

body:
  After the initial ARPANET was complete, the next major milestone was to enable
  access to the network regardless of one's proximity to a host node with an IMP
  connection.

  The idea was to allow a computer to access resources on another computer
  directly without connecting to a host first. This connective computer would
  connect to an IMP directly (but still transparently) as a 'dumb terminal' as
  with time-sharing and would not be a fully-fledged computing device. These
  devices would be called _Terminal_ Information Processors (TIMPs) for this
  reason.

  The development of TIMPs makes it clearer that the host machines on the ARPANET
  were not just connection and transmission nodes for their own purpose, they were
  loci for other non-host computers to gain access to an IMP, and thereby, the
  broader network. In other words computers would connect to a host which
  sustained a connection to an IMP.

raw_content:
  ---
  tags: [ARPANET, networks, computer-history]
  created: Friday, October 25, 2024
  ---

  # c8173d17_TIMPs

  After the initial ARPANET was complete, the next major milestone was to enable
  access to the network regardless of one's proximity to a host node with an IMP
  connection.

  The idea was to allow a computer to access resources on another computer
  directly without connecting to a host first. This connective computer would
  connect to an IMP directly (but still transparently) as a 'dumb terminal' as
  with time-sharing and would not be a fully-fledged computing device. These
  devices would be called _Terminal_ Information Processors (TIMPs) for this
  reason.

  The development of TIMPs makes it clearer that the host machines on the ARPANET
  were not just connection and transmission nodes for their own purpose, they were
  loci for other non-host computers to gain access to an IMP, and thereby, the
  broader network. In other words computers would connect to a host which
  sustained a connection to an IMP.


path: The_Shadow_Text.md
sortable_path: The_Shadow_Text.md
title: 4dec6fe0_shadow_text
checksum: 4285ac86bb54e3db9fde04613ea1b8ebaa3870262e8a7fd582492a1e59f86832
created: 2025-08-09 13:41:47.026554147+00:00
modified: 2025-08-10 14:08:47.112679749+00:00
metadata: {"created":"Monday, January 20, 2025","tags":["ecopolsoc","internet","surveillance-capitalism"]}
lead:
  Surveillance capitalism creates two "electronic texts". The first is the
  public-facing text which consists in the benefits conferred by the surveillance
  capitalists' services, expressed in their self-regarding mission statements. The
  "universe of information and connection [they] bring to our fingertips" (posts,
  blogs, likes, photos, spreadsheets, music, tweets etc).

body:
  Surveillance capitalism creates two "electronic texts". The first is the
  public-facing text which consists in the benefits conferred by the surveillance
  capitalists' services, expressed in their self-regarding mission statements. The
  "universe of information and connection [they] bring to our fingertips" (posts,
  blogs, likes, photos, spreadsheets, music, tweets etc).

  The first text is a sort of Trojan Horse, serving as the "supply operation" for
  the second, _shadow_ text. Everything we contribute to the first text becomes a
  target for surplus extraction via the shadow text. This is the ultimate purpose
  of the services and is hidden from view and only for the purview of the
  surveillance capitalists.

  _In this text, our experience is dragooned as raw material to be accumulated and
  analyzed as means to others' market ends. The shadow text is a burgeoning
  accumulation of behavioural surplus and its analyses, and it says more about us
  than we can know about ourselves. Worse still, it becomes increasingly
  difficult, and perhaps impossible, to refrain from contributing to the shadow
  text. It automatically feeds on our experience as we engage in the normal and
  necessary routines of social participation._

  _The Age of Surveillance Capitalism_ (2019): 185

raw_content:
  ---
  tags: [ecopolsoc, internet, surveillance-capitalism]
  created: Monday, January 20, 2025
  ---

  # 4dec6fe0_shadow_text

  Surveillance capitalism creates two "electronic texts". The first is the
  public-facing text which consists in the benefits conferred by the surveillance
  capitalists' services, expressed in their self-regarding mission statements. The
  "universe of information and connection [they] bring to our fingertips" (posts,
  blogs, likes, photos, spreadsheets, music, tweets etc).

  The first text is a sort of Trojan Horse, serving as the "supply operation" for
  the second, _shadow_ text. Everything we contribute to the first text becomes a
  target for surplus extraction via the shadow text. This is the ultimate purpose
  of the services and is hidden from view and only for the purview of the
  surveillance capitalists.

  _In this text, our experience is dragooned as raw material to be accumulated and
  analyzed as means to others' market ends. The shadow text is a burgeoning
  accumulation of behavioural surplus and its analyses, and it says more about us
  than we can know about ourselves. Worse still, it becomes increasingly
  difficult, and perhaps impossible, to refrain from contributing to the shadow
  text. It automatically feeds on our experience as we engage in the normal and
  necessary routines of social participation._

  _The Age of Surveillance Capitalism_ (2019): 185


path: The_Web_versus_modem_BBSs.md
sortable_path: The_Web_versus_modem_BBSs.md
title: 241fe1a3_the_Web_versus_modem_BBSs
checksum: 84911f03e729d16f43038f10b328821c150b1b362115c555864dea1e500d54e4
created: 2025-08-09 13:41:47.034951185+00:00
modified: 2025-08-10 14:08:47.17453399+00:00
lead:
  As access to the Web and the demand for public access grew, services like AOL,
  which provided dial-up access to a subset of the broader Internet (along with
  email and chat-rooms) became increasingly redundant.

body:
  As access to the Web and the demand for public access grew, services like AOL,
  which provided dial-up access to a subset of the broader Internet (along with
  email and chat-rooms) became increasingly redundant.

  In response to this, they pivoted in their marketing to presenting their service
  as a safe, curated means of accessing the information: the Web was confusing, it
  was hard to know what was out there and how to find useful information.

  By using AOL, you would know which sites to visit for different tasks and which
  of these were reputable. They adopted a kind of friendly gatekeeper persona in
  the era of unhampered and universal access to information that the Web heralded
  and embodied.

  (This isn't so strange if we consider that outside of work (and the communities
  of computer enthusiasts), personal computers were still relatively new and
  computers themselves still rather forbidding entities.)

  This function of a gateway to the overwhelming content of the web would be taken
  up first by browsers
  ([Mosaic's](c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md) landing page would
  list websites for specific sources of information and user tasks and links to
  other directories) and then of course search engines which would allow users to
  search for what they needed against an index (usually a subset) of the Web's
  total content. (Interestingly in this regard, AOL would eventually buy
  Netscape.)

  AOL et al made sense when users wanted access to the internet and the web didn't
  yet exist. Different networks had different access criteria and were often
  poorly signposted, accessible only to those in the know. Private commercial
  networks were set up to service typical consumer needs (news, banking, weather
  reports etc) but the whole point of the Web as conceived by TBL was to make
  everything available to anyone who wanted it through a streamlined and universal
  set of protocols.

raw_content:
  ---
  tags: [world-wide-web, computer-history, internet, bulletin-boards]
  created: Monday, November 04, 2024
  ---

  # 241fe1a3_the_Web_versus_modem_BBSs

  As access to the Web and the demand for public access grew, services like AOL,
  which provided dial-up access to a subset of the broader Internet (along with
  email and chat-rooms) became increasingly redundant.

  In response to this, they pivoted in their marketing to presenting their service
  as a safe, curated means of accessing the information: the Web was confusing, it
  was hard to know what was out there and how to find useful information.

  By using AOL, you would know which sites to visit for different tasks and which
  of these were reputable. They adopted a kind of friendly gatekeeper persona in
  the era of unhampered and universal access to information that the Web heralded
  and embodied.

  (This isn't so strange if we consider that outside of work (and the communities
  of computer enthusiasts), personal computers were still relatively new and
  computers themselves still rather forbidding entities.)

  This function of a gateway to the overwhelming content of the web would be taken
  up first by browsers
  ([Mosaic's](c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md) landing page would
  list websites for specific sources of information and user tasks and links to
  other directories) and then of course search engines which would allow users to
  search for what they needed against an index (usually a subset) of the Web's
  total content. (Interestingly in this regard, AOL would eventually buy
  Netscape.)

  AOL et al made sense when users wanted access to the internet and the web didn't
  yet exist. Different networks had different access criteria and were often
  poorly signposted, accessible only to those in the know. Private commercial
  networks were set up to service typical consumer needs (news, banking, weather
  reports etc) but the whole point of the Web as conceived by TBL was to make
  everything available to anyone who wanted it through a streamlined and universal
  set of protocols.

metadata:
  {"created":"Monday, November 04, 2024","tags":["world-wide-web","computer-history","internet","bulletin-boards"]}


path: Tor_prioritising_uptake.md
sortable_path: Tor_prioritising_uptake.md
title: c8820974_Tor_prioritising_uptake
lead:
  When formulating the initial architecture, the Tor engineers were faced with the
  following dilemma.

checksum: 489e3cb853474db12b6e0c16dd1d6d5a0557de4d8dcbc1640e23a4a7afd44cfd
created: 2025-08-09 13:41:47.052010634+00:00
modified: 2025-08-10 14:08:47.014890343+00:00
metadata: {"created":"Wednesday, March 26, 2025","tags":["Tor"]}
body:
  When formulating the initial architecture, the Tor engineers were faced with the
  following dilemma.

  - Greater security meant greater complexity for the user and therefore reduced
    adoption.

  - Less adoption means less security because there are less users on the network
    and thus individual nodes become easier to track and identify.

  Factors that would result in greater security include more cryptographic rigour
  or more hops. Also additional methods of obfuscation such as mixing and padding.

  Both could involve more overhead for the user, which introduces friction and the
  likelihood of continued use. More hops increases latency which makes everyday
  use less viable.

  Broad uptake and a rich ecosystem including users who were not the main or
  target beneficiaries (spies, totalitarian subjects) was necessary in order to
  obscure the activities of the target users.

  ## Source

  Ben Collier, _Tor: From the Dark Web to the Future of Privacy_ (2024)

raw_content:
  ---
  tags: [Tor]
  created: Wednesday, March 26, 2025
  ---

  # c8820974_Tor_prioritising_uptake

  When formulating the initial architecture, the Tor engineers were faced with the
  following dilemma.

  - Greater security meant greater complexity for the user and therefore reduced
    adoption.

  - Less adoption means less security because there are less users on the network
    and thus individual nodes become easier to track and identify.

  Factors that would result in greater security include more cryptographic rigour
  or more hops. Also additional methods of obfuscation such as mixing and padding.

  Both could involve more overhead for the user, which introduces friction and the
  likelihood of continued use. More hops increases latency which makes everyday
  use less viable.

  Broad uptake and a rich ecosystem including users who were not the main or
  target beneficiaries (spies, totalitarian subjects) was necessary in order to
  obscure the activities of the target users.

  ## Source

  Ben Collier, _Tor: From the Dark Web to the Future of Privacy_ (2024)


path: Transport_Layer_clarification.md
sortable_path: Transport_Layer_clarification.md
title: b01fd836_Transport_Layer_clarification
checksum: 7892a77009c267ad84909fb00fc355bbd66657c0b8573f6ee7ff633487c66751
created: 2025-08-09 13:41:47.059282302+00:00
modified: 2025-08-10 14:08:47.148022609+00:00
metadata: {"created":"Sunday, March 02, 2025","tags":["networks","internet"]}
lead:
  The Transport Layer is a _logical_ representation of the connection between two
  hosts on the Internet. It does not include the entire path (through various
  routers) from source to destination. This detail is assumed and is the concern
  of the Internet Layer - the next layer down.

body:
  The Transport Layer is a _logical_ representation of the connection between two
  hosts on the Internet. It does not include the entire path (through various
  routers) from source to destination. This detail is assumed and is the concern
  of the Internet Layer - the next layer down.

  When applications communicate accross via TCP they "perceive" a simple direct
  connection however this abstracts the fact that the data may:

  - be split into dozens of packets
  - travel through different physical pathways and different routers
  - face congestion, packet loss and network failure.

raw_content:
  ---
  tags: [networks, internet]
  created: Sunday, March 02, 2025
  ---

  # b01fd836_Transport_Layer_clarification

  The Transport Layer is a _logical_ representation of the connection between two
  hosts on the Internet. It does not include the entire path (through various
  routers) from source to destination. This detail is assumed and is the concern
  of the Internet Layer - the next layer down.

  When applications communicate accross via TCP they "perceive" a simple direct
  connection however this abstracts the fact that the data may:

  - be split into dozens of packets
  - travel through different physical pathways and different routers
  - face congestion, packet loss and network failure.


path: Trapdoor_functions.md
sortable_path: Trapdoor_functions.md
title: a4601796_trapdoor_functions
checksum: 663700b03beee5d3d47e4d43afabc07b52f25db9a0844c1b033528bed50eced8
created: 2025-08-09 13:41:47.065036643+00:00
modified: 2025-08-10 14:08:47.301563157+00:00
metadata: {"created":"Friday, December 20, 2024","tags":["cryptography","encryption"]}
lead:
  How does [public key cryptography](./bbdcb54f_public_key_cryptography.md)
  achieve the feat of making the encyrpted data only decipherable in one
  direction. In other words how come the same data can be encrypted with one key
  but only decryptable by another?

body:
  How does [public key cryptography](./bbdcb54f_public_key_cryptography.md)
  achieve the feat of making the encyrpted data only decipherable in one
  direction. In other words how come the same data can be encrypted with one key
  but only decryptable by another?

  It is acheived by the use of **trapdoor functions**. These are functions that
  are easy to compute in one direction but extremely difficult to reverse. But
  they also contain trapdoors which, if known, make the functions trivially easy
  to compute in both directions.

  The public key holder can compute the encryption but only the private key holder
  has the trapdoor that makes reversing the encryption trivial.

  As an example we can look at a common method for generating trapdoor functions
  using prime numbers.

  First we need to convert the message into numerical form. One way would be to
  convert the characters to their ASCII numbers. These numbers then become input
  into the mathematical encryption function.

  Next we would pick two very large prime numbers. Using RSA-like encryption they
  would be need to be 2048+ bits in length. For the example we will use 17 and 23.

  For simplicity let's say that our message is one character equivalent to `5` in
  ASCII. We multiply our two primes to get 391 (17 x 23 = 391) and then find the
  remainder of: our message cubed and divided by the prime sum. This is `125`
  (`5^3 / 391`) which is now our encrypted message which we send.

  Unless you know the two primes which were used, it is extremely difficult and
  time consuming to reverse the calculation and find the factors that would get
  you back to the starting point (remember the numbers will be much larger in
  reality). The owner of the private key knows these primes however, so can do
  this.

raw_content:
  ---
  tags: [cryptography, encryption]
  created: Friday, December 20, 2024
  ---

  # a4601796_trapdoor_functions

  How does [public key cryptography](./bbdcb54f_public_key_cryptography.md)
  achieve the feat of making the encyrpted data only decipherable in one
  direction. In other words how come the same data can be encrypted with one key
  but only decryptable by another?

  It is acheived by the use of **trapdoor functions**. These are functions that
  are easy to compute in one direction but extremely difficult to reverse. But
  they also contain trapdoors which, if known, make the functions trivially easy
  to compute in both directions.

  The public key holder can compute the encryption but only the private key holder
  has the trapdoor that makes reversing the encryption trivial.

  As an example we can look at a common method for generating trapdoor functions
  using prime numbers.

  First we need to convert the message into numerical form. One way would be to
  convert the characters to their ASCII numbers. These numbers then become input
  into the mathematical encryption function.

  Next we would pick two very large prime numbers. Using RSA-like encryption they
  would be need to be 2048+ bits in length. For the example we will use 17 and 23.

  For simplicity let's say that our message is one character equivalent to `5` in
  ASCII. We multiply our two primes to get 391 (17 x 23 = 391) and then find the
  remainder of: our message cubed and divided by the prime sum. This is `125`
  (`5^3 / 391`) which is now our encrypted message which we send.

  Unless you know the two primes which were used, it is extremely difficult and
  time consuming to reverse the calculation and find the factors that would get
  you back to the starting point (remember the numbers will be much larger in
  reality). The owner of the private key knows these primes however, so can do
  this.


path: Web_precursors.md
sortable_path: Web_precursors.md
title: cfbef1c4_web_precursors
lead: ## Early precursors to the commercial internet
checksum: f5f842b9d92e189c4cfc0cab9ff8cf03050a627d4e9daceee266e21473d8bf2c
created: 2025-08-09 13:41:47.080781086+00:00
modified: 2025-08-10 14:08:47.251646656+00:00
body:
  ## Early precursors to the commercial internet

  Prior to the commercial internet and the advent of the web, there were services
  in the 1980s which allowed home computers to connect to limited public networks.
  Example services: checking one's bank balance, paying bills, transferring money,
  booking holidays.

  (Presumably, the services would be hosted on a providers'
  [mainframe](Mainframe_computers.md) or mini-computer, which the consumer would
  access through something like time-sharing?)

  ![BT telephone modem](../img/modem-bt.png)

  The mechanism of connection was via a modem. A device that connected to your
  phone's cradle. Digital signals from your home computer would be converted into
  sound signals ("tones") that would travel through the phone transmitter to a
  receiver at the service provider which you would "dial-up" beforehand.

  In the US the main provider was Micronet in the 1970s, which became Compuserve
  in the 1980s and eventually AOL in the 1990s. In the UK there was Prestel
  provided by BT, first introduced in 1971.

  These services had an expensive subscription model and consequently uptake was
  limited.

  In addition there were bulletin board systems (BBS), typically administrated by
  hobbyists. Their personal computers acted as servers that other users would
  dial-up to access. They permitted the sharing of news and messages and later
  chat-rooms for realtime communication. Some of the biggest BBSs later became the
  first ISPs.

raw_content:
  ---
  tags: [fleeting, modems, computer-history, internet, bulletin-boards]
  created: Monday, October 14, 2024
  ---

  # cfbef1c4_web_precursors

  ## Early precursors to the commercial internet

  Prior to the commercial internet and the advent of the web, there were services
  in the 1980s which allowed home computers to connect to limited public networks.
  Example services: checking one's bank balance, paying bills, transferring money,
  booking holidays.

  (Presumably, the services would be hosted on a providers'
  [mainframe](Mainframe_computers.md) or mini-computer, which the consumer would
  access through something like time-sharing?)

  ![BT telephone modem](../img/modem-bt.png)

  The mechanism of connection was via a modem. A device that connected to your
  phone's cradle. Digital signals from your home computer would be converted into
  sound signals ("tones") that would travel through the phone transmitter to a
  receiver at the service provider which you would "dial-up" beforehand.

  In the US the main provider was Micronet in the 1970s, which became Compuserve
  in the 1980s and eventually AOL in the 1990s. In the UK there was Prestel
  provided by BT, first introduced in 1971.

  These services had an expensive subscription model and consequently uptake was
  limited.

  In addition there were bulletin board systems (BBS), typically administrated by
  hobbyists. Their personal computers acted as servers that other users would
  dial-up to access. They permitted the sharing of news and messages and later
  chat-rooms for realtime communication. Some of the biggest BBSs later became the
  first ISPs.

metadata:
  {"created":"Monday, October 14, 2024","tags":["fleeting","modems","computer-history","internet","bulletin-boards"]}


path: Add_API_key_authorization_Express.md
sortable_path: Add_API_key_authorization_Express.md
title: Add API key authorization Express
lead: Basically you add it as a piece of [middlewear](./Middleware_in_NodeJS.md):
checksum: e82daa6509391cafb009764c5852f256a72888f4d092453d9b47501e271a41b4
created: 2025-08-09 13:41:56.016661945+00:00
modified: 2025-08-10 14:08:47.32977445+00:00
metadata: {"created":"Saturday, August 09, 2025","tags":["node-js"]}
body:
  Basically you add it as a piece of [middlewear](./Middleware_in_NodeJS.md):

  ```js
  // auth.js
  const validateApiKey = (req, res, next) => {
    const apiKey = req.headers["x-api-key"];

    if (!apiKey) {
      return res.status(401).json({
        error: "API key is required.",
      });
    }

    if (apiKey !== process.env.API_KEY) {
      return res.status(403).json({ error: "Invalid API key" });
    }

    next();
  };

  export { validateApiKey };
  ```

  And then apply it to all routes, e.g.:

  ```js
  import { validateApiKey } from "./middlewear/auth.js";

  app.use("/", validateApiKey);
  ```

raw_content:
  ---
  tags: ["node-js"]
  created: Saturday, August 09, 2025
  ---

  # Add API key authorization Express

  Basically you add it as a piece of [middlewear](./Middleware_in_NodeJS.md):

  ```js
  // auth.js
  const validateApiKey = (req, res, next) => {
    const apiKey = req.headers["x-api-key"];

    if (!apiKey) {
      return res.status(401).json({
        error: "API key is required.",
      });
    }

    if (apiKey !== process.env.API_KEY) {
      return res.status(403).json({ error: "Invalid API key" });
    }

    next();
  };

  export { validateApiKey };
  ```

  And then apply it to all routes, e.g.:

  ```js
  import { validateApiKey } from "./middlewear/auth.js";

  app.use("/", validateApiKey);
  ```


path: Forgejo_runner_failure_after_VPS_reboot.md
sortable_path: Forgejo_runner_failure_after_VPS_reboot.md
title: Forgejo runner failure after VPS reboot
checksum: 63faf3ea088df08b177ce3f4f2a52a3138bc6c4b1b2672c177d0601a7458924d
created: 2025-08-09 14:40:30.774283241+00:00
modified: 2025-08-10 14:08:47.110862627+00:00
metadata: {"created":"Saturday, August 09, 2025","tags":["docker","servers"]}
lead:
  If you get stuck on "Waiting" when running an automated deployment in Forgejo,
  this can be because you've rebooted the VPS and the Docker permissions have
  reset.

body:
  If you get stuck on "Waiting" when running an automated deployment in Forgejo,
  this can be because you've rebooted the VPS and the Docker permissions have
  reset.

  Error in Docker logs typically looks like:

  ```
  Error: cannot ping the docker daemon, does it running? permission denied
  while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock:
  Get "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock:
  connect: permission denied
  ```

  Fix with:

  ```sh
  sudo chmod 666 /var/run/docker.sock
  docker restart forgejo-runner
  ```

raw_content:
  ---
  tags: [docker, servers]
  created: Saturday, August 09, 2025
  ---

  # Forgejo runner failure after VPS reboot

  If you get stuck on "Waiting" when running an automated deployment in Forgejo,
  this can be because you've rebooted the VPS and the Docker permissions have
  reset.

  Error in Docker logs typically looks like:

  ```
  Error: cannot ping the docker daemon, does it running? permission denied
  while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock:
  Get "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock:
  connect: permission denied
  ```

  Fix with:

  ```sh
  sudo chmod 666 /var/run/docker.sock
  docker restart forgejo-runner
  ```


