# notes

path: Weaving_the_Web_Berners_Lee.md
sortable_path: Weaving_the_Web_Berners_Lee.md
title: Weaving the Web (Berners-Lee, 1999)
checksum: 3761dfc6b81bf13c1e89e99bd3610c27737224fb4f5a3d5e66e13fd1ce071eb6
created: 2024-11-04 15:20:40.673485872+00:00
modified: 2025-01-09 18:51:19.315768345+00:00
metadata: {"created":"Monday, November 04, 2024","tags":["literature","ecopolsoc","world-wide-web"]}
lead:
  Tim did not really conceive of the possibility of the centralising and
  totalising effects of social media:

body:
  Tim did not really conceive of the possibility of the centralising and
  totalising effects of social media:

  _The Web's universality leads to a thriving richness and diversity. If a company
  claims to give access to the world of information, then presents a filtered
  view, the Web loses its credibility._ (p.143)

  _Happily, the Web is so huge that there's no way any one company can dominate
  it. All the human effort people and organisations have put in all over the world
  to create Web sites and home pages is astoundingly large, and most of the effort
  has to do with what's in the Web, not the software used to browse it. The Web's
  content, and thus value, will continue despite any one company's actions._
  (p.144)

  Apropos the "alt-right" etc:

  _The analogy of a global brain is tempting, because Web and brain both involve
  huge numbers of elements - neurons and Web pages - and a mixture of structure
  and apparent randomness. However, a brain has an intelligence that emerges on
  quite a different level from anything that a neuron could be aware
  of...[W]riters have contemplated an 'emergent property' arising from the mass of
  humanity and computers. But remember that such a phenomenon would have its own
  agenda. We would not as individuals be ware of it, let alone control it, any
  more than the neuron controls the brain._ (p.222)

raw_content:
  ---
  tags: [literature, ecopolsoc, world-wide-web]
  created: Monday, November 04, 2024
  ---

  # Weaving the Web (Berners-Lee, 1999)

  Tim did not really conceive of the possibility of the centralising and
  totalising effects of social media:

  _The Web's universality leads to a thriving richness and diversity. If a company
  claims to give access to the world of information, then presents a filtered
  view, the Web loses its credibility._ (p.143)

  _Happily, the Web is so huge that there's no way any one company can dominate
  it. All the human effort people and organisations have put in all over the world
  to create Web sites and home pages is astoundingly large, and most of the effort
  has to do with what's in the Web, not the software used to browse it. The Web's
  content, and thus value, will continue despite any one company's actions._
  (p.144)

  Apropos the "alt-right" etc:

  _The analogy of a global brain is tempting, because Web and brain both involve
  huge numbers of elements - neurons and Web pages - and a mixture of structure
  and apparent randomness. However, a brain has an intelligence that emerges on
  quite a different level from anything that a neuron could be aware
  of...[W]riters have contemplated an 'emergent property' arising from the mass of
  humanity and computers. But remember that such a phenomenon would have its own
  agenda. We would not as individuals be ware of it, let alone control it, any
  more than the neuron controls the brain._ (p.222)


path: c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md
sortable_path: c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md
title: c301a0b3_Mosaic_Netscape_and_Browser_Wars
checksum: 2932435c9c734455a7996da2c686501f21609773035248164eb5f8b8cc59abf8
created: 2024-11-04 15:35:45.011665284+00:00
modified: 2024-11-04 16:40:18.394456919+00:00
metadata: {"created":"Sunday, November 03, 2024","tags":["world-wide-web","computer-history","Microsoft"]}
lead:
  The Mosaic browser was conceived and produced at the University of Illinois
  during the very early growth of the Web, when TBL was still at CERN.

body:
  The Mosaic browser was conceived and produced at the University of Illinois
  during the very early growth of the Web, when TBL was still at CERN.

  _Mosaic_:

  ![Mosaic browser](../img/mosaic-browser.jpg)

  It became the best of the early browsers (a group that included ViolaWWW and
  Erwise) with the notable advantage of being able to embed graphics alongside
  text (instead of inside a separate window). It also supported multiple
  [application layer](Application_Layer_of_Internet_Protocol.md) protocols - FTP,
  the Network News Transfer Protocol, Gopher - not just HTTP.

  At this point there were nascent browsers for different operating systems and
  which browser you used was mostly determined by your OS. Being the product of a
  US university, Mosaic was originally written for Unix's X-Window system but was
  ported to multiple OSs later, a key factor in its success and uptake.

  Mosaic's creators formed their own company outside of the university: Netscape.
  The browser they created, Netscape Navigator became the heir to Mosaic, released
  in 1994.

  _Netscape Navigator_:

  ![Netscape Navigator](../img/Navigator_Netscape.png)

  It was made free of charge for non-commercial use which assured rapid uptake.
  This was done to outrun Microsoft's Internet Explorer (released two months after
  Netscape went public) which they intended to bundle with all new versions of
  Windows 95. Netscape became the de facto standard browser during the early Web,
  including on Windows. It took a few years for IE to overtake it.

  _Internet Explorer 1 with licensing info_:

  ![Internet Explorer](../img/internet_explorer.png)

  Netscape also birthed JavaScript, which was written rapidly specifically for the
  browser. Prior to this, dynamic client-side functionality was provided by Java
  'applets' which never really took off.

  IE in fact shares a genealogy with Mosaic since Microsoft licensed a version of
  the Mosaic code and used it to construct IE.

  By the end of the 1990s, Netscape's usage had declined precipitously as IE
  captured the market share.

raw_content:
  ---
  tags: [world-wide-web, computer-history, Microsoft]
  created: Sunday, November 03, 2024
  ---

  # c301a0b3_Mosaic_Netscape_and_Browser_Wars

  The Mosaic browser was conceived and produced at the University of Illinois
  during the very early growth of the Web, when TBL was still at CERN.

  _Mosaic_:

  ![Mosaic browser](../img/mosaic-browser.jpg)

  It became the best of the early browsers (a group that included ViolaWWW and
  Erwise) with the notable advantage of being able to embed graphics alongside
  text (instead of inside a separate window). It also supported multiple
  [application layer](Application_Layer_of_Internet_Protocol.md) protocols - FTP,
  the Network News Transfer Protocol, Gopher - not just HTTP.

  At this point there were nascent browsers for different operating systems and
  which browser you used was mostly determined by your OS. Being the product of a
  US university, Mosaic was originally written for Unix's X-Window system but was
  ported to multiple OSs later, a key factor in its success and uptake.

  Mosaic's creators formed their own company outside of the university: Netscape.
  The browser they created, Netscape Navigator became the heir to Mosaic, released
  in 1994.

  _Netscape Navigator_:

  ![Netscape Navigator](../img/Navigator_Netscape.png)

  It was made free of charge for non-commercial use which assured rapid uptake.
  This was done to outrun Microsoft's Internet Explorer (released two months after
  Netscape went public) which they intended to bundle with all new versions of
  Windows 95. Netscape became the de facto standard browser during the early Web,
  including on Windows. It took a few years for IE to overtake it.

  _Internet Explorer 1 with licensing info_:

  ![Internet Explorer](../img/internet_explorer.png)

  Netscape also birthed JavaScript, which was written rapidly specifically for the
  browser. Prior to this, dynamic client-side functionality was provided by Java
  'applets' which never really took off.

  IE in fact shares a genealogy with Mosaic since Microsoft licensed a version of
  the Mosaic code and used it to construct IE.

  By the end of the 1990s, Netscape's usage had declined precipitously as IE
  captured the market share.


path: ecec56e6_precursors_to_search_engines.md
sortable_path: ecec56e6_precursors_to_search_engines.md
title: ecec56e6_precursors_to_search_engines
checksum: f160dea20cd9f987f808c75db9e5931b5357007d35513bd3a30476ea42945ed3
created: 2024-11-04 15:59:31.8979808+00:00
modified: 2024-11-04 16:07:38.069712579+00:00
metadata: {"created":"Monday, November 04, 2024","tags":["world-wide-web","computer-history"]}
lead:
  Before search engines as searchable indices of all content on the web, the
  custom was for ISPs, browsers and websites (e.g. Yahoo!) to provide directories
  of links to other sites. They functioned as sort of global web-portals.

body:
  Before search engines as searchable indices of all content on the web, the
  custom was for ISPs, browsers and websites (e.g. Yahoo!) to provide directories
  of links to other sites. They functioned as sort of global web-portals.

  ![Yahoo's original homepage](../img/yahoo-original-homepage.png)

  _The Yahoo! homepage from 1999._

  ![AltaVista homepage](../img/alta-vista.png)

  _The AltaVista homepage from 1999. It has a primitive search engine but also
  directories of sites._

raw_content:
  ---
  tags: [world-wide-web, computer-history]
  created: Monday, November 04, 2024
  ---

  # ecec56e6_precursors_to_search_engines

  Before search engines as searchable indices of all content on the web, the
  custom was for ISPs, browsers and websites (e.g. Yahoo!) to provide directories
  of links to other sites. They functioned as sort of global web-portals.

  ![Yahoo's original homepage](../img/yahoo-original-homepage.png)

  _The Yahoo! homepage from 1999._

  ![AltaVista homepage](../img/alta-vista.png)

  _The AltaVista homepage from 1999. It has a primitive search engine but also
  directories of sites._


path: f6564af9_AOL_and_early_internet_enclosure.md
sortable_path: f6564af9_AOL_and_early_internet_enclosure.md
title: f6564af9_AOL_and_early_internet_enclosure
checksum: 8f2eefe769e6cd66fe8b5a4402b0a972e07f5cb2955ae3694dc7dca203fc5dda
created: 2024-11-04 16:08:34.73788458+00:00
modified: 2025-01-09 18:51:36.342729966+00:00
lead:
  We might view AOL, with its ubiquitous 'free trial' CD-ROMs, as the first
  attempt by a corporate entity to try and enclose the Internet/Web. To take the
  vast informational expanse and reduce it to a single, filtered and contained
  space controlled by a single company.

body:
  We might view AOL, with its ubiquitous 'free trial' CD-ROMs, as the first
  attempt by a corporate entity to try and enclose the Internet/Web. To take the
  vast informational expanse and reduce it to a single, filtered and contained
  space controlled by a single company.

  ![Early AOL interface](../img/aol-on-windows.jpg)

  _AOL interface from 1993_

  This would later be achieved or approximated more effectively by Google (under
  the guise of doing the opposite - people are "empowered" to access the
  information they otherwise would not find, granted a key to the Internet etc.
  When in fact, making everyone go through the same gate enables surveillance and
  the monetisation of the use of a public resource) and then of course by the
  social media companies.

  AOL in fact started this even before there was a Web, pioneering the concept of
  a web portal, with their [modem-based](cfbef1c4_web_precursors.md) private-BBS
  business model. This provided access to a circumscribed subset of the Internet
  at a time when accessing information was opaque and confusing to the average
  non-technical user. The advent of the web first threatened and then basically
  obviated this model, when access to information via browsers and HTTP servers
  made the process easier and more democratized. Hence why AOL then tellingly
  pivoted to the browser and search-engine market (purchasing
  [Netscape](c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md) in 1999), the next
  available mechanism of enclosure.

  ![AOL homepage](../img/aol-original.jpg)

  _AOL interface from 1999_

raw_content:
  ---
  tags: [world-wide-web, computer-history, internet, ecopolsoc, bulletin-boards]
  created: Monday, November 04, 2024
  ---

  # f6564af9_AOL_and_early_internet_enclosure

  We might view AOL, with its ubiquitous 'free trial' CD-ROMs, as the first
  attempt by a corporate entity to try and enclose the Internet/Web. To take the
  vast informational expanse and reduce it to a single, filtered and contained
  space controlled by a single company.

  ![Early AOL interface](../img/aol-on-windows.jpg)

  _AOL interface from 1993_

  This would later be achieved or approximated more effectively by Google (under
  the guise of doing the opposite - people are "empowered" to access the
  information they otherwise would not find, granted a key to the Internet etc.
  When in fact, making everyone go through the same gate enables surveillance and
  the monetisation of the use of a public resource) and then of course by the
  social media companies.

  AOL in fact started this even before there was a Web, pioneering the concept of
  a web portal, with their [modem-based](cfbef1c4_web_precursors.md) private-BBS
  business model. This provided access to a circumscribed subset of the Internet
  at a time when accessing information was opaque and confusing to the average
  non-technical user. The advent of the web first threatened and then basically
  obviated this model, when access to information via browsers and HTTP servers
  made the process easier and more democratized. Hence why AOL then tellingly
  pivoted to the browser and search-engine market (purchasing
  [Netscape](c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md) in 1999), the next
  available mechanism of enclosure.

  ![AOL homepage](../img/aol-original.jpg)

  _AOL interface from 1999_

metadata:
  {"created":"Monday, November 04, 2024","tags":["world-wide-web","computer-history","internet","ecopolsoc","bulletin-boards"]}


path: 241fe1a3_the_Web_versus_modem_BBSs.md
sortable_path: 241fe1a3_the_Web_versus_modem_BBSs.md
title: 241fe1a3_the_Web_versus_modem_BBSs
checksum: 84911f03e729d16f43038f10b328821c150b1b362115c555864dea1e500d54e4
created: 2024-11-04 16:43:52.707927075+00:00
modified: 2024-11-04 18:33:45.584497495+00:00
lead:
  As access to the Web and the demand for public access grew, services like AOL,
  which provided dial-up access to a subset of the broader Internet (along with
  email and chat-rooms) became increasingly redundant.

body:
  As access to the Web and the demand for public access grew, services like AOL,
  which provided dial-up access to a subset of the broader Internet (along with
  email and chat-rooms) became increasingly redundant.

  In response to this, they pivoted in their marketing to presenting their service
  as a safe, curated means of accessing the information: the Web was confusing, it
  was hard to know what was out there and how to find useful information.

  By using AOL, you would know which sites to visit for different tasks and which
  of these were reputable. They adopted a kind of friendly gatekeeper persona in
  the era of unhampered and universal access to information that the Web heralded
  and embodied.

  (This isn't so strange if we consider that outside of work (and the communities
  of computer enthusiasts), personal computers were still relatively new and
  computers themselves still rather forbidding entities.)

  This function of a gateway to the overwhelming content of the web would be taken
  up first by browsers
  ([Mosaic's](c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md) landing page would
  list websites for specific sources of information and user tasks and links to
  other directories) and then of course search engines which would allow users to
  search for what they needed against an index (usually a subset) of the Web's
  total content. (Interestingly in this regard, AOL would eventually buy
  Netscape.)

  AOL et al made sense when users wanted access to the internet and the web didn't
  yet exist. Different networks had different access criteria and were often
  poorly signposted, accessible only to those in the know. Private commercial
  networks were set up to service typical consumer needs (news, banking, weather
  reports etc) but the whole point of the Web as conceived by TBL was to make
  everything available to anyone who wanted it through a streamlined and universal
  set of protocols.

raw_content:
  ---
  tags: [world-wide-web, computer-history, internet, bulletin-boards]
  created: Monday, November 04, 2024
  ---

  # 241fe1a3_the_Web_versus_modem_BBSs

  As access to the Web and the demand for public access grew, services like AOL,
  which provided dial-up access to a subset of the broader Internet (along with
  email and chat-rooms) became increasingly redundant.

  In response to this, they pivoted in their marketing to presenting their service
  as a safe, curated means of accessing the information: the Web was confusing, it
  was hard to know what was out there and how to find useful information.

  By using AOL, you would know which sites to visit for different tasks and which
  of these were reputable. They adopted a kind of friendly gatekeeper persona in
  the era of unhampered and universal access to information that the Web heralded
  and embodied.

  (This isn't so strange if we consider that outside of work (and the communities
  of computer enthusiasts), personal computers were still relatively new and
  computers themselves still rather forbidding entities.)

  This function of a gateway to the overwhelming content of the web would be taken
  up first by browsers
  ([Mosaic's](c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md) landing page would
  list websites for specific sources of information and user tasks and links to
  other directories) and then of course search engines which would allow users to
  search for what they needed against an index (usually a subset) of the Web's
  total content. (Interestingly in this regard, AOL would eventually buy
  Netscape.)

  AOL et al made sense when users wanted access to the internet and the web didn't
  yet exist. Different networks had different access criteria and were often
  poorly signposted, accessible only to those in the know. Private commercial
  networks were set up to service typical consumer needs (news, banking, weather
  reports etc) but the whole point of the Web as conceived by TBL was to make
  everything available to anyone who wanted it through a streamlined and universal
  set of protocols.

metadata:
  {"created":"Monday, November 04, 2024","tags":["world-wide-web","computer-history","internet","bulletin-boards"]}


path: Using_SQLite_with_Python.md
sortable_path: Using_SQLite_with_Python.md
title: Using SQLite with Python
lead: SQLite is part of the core library, you do not require a separate package
checksum: 03ce3deea72df42fd5f2a60f72b37b791107c9ddc87f97b87fa48674925c6750
created: 2024-11-07 07:27:38.831355238+00:00
modified: 2024-11-14 14:43:27.013210641+00:00
metadata: {"created":"Thursday, November 07, 2024","tags":["python","databases","SQL","SQLite"]}
body:
  SQLite is part of the core library, you do not require a separate package

  ```py
  import sqlite3
  ```

  ## Connect to database

  Connect to an existing SQLite `.db` file or create a new database with the same
  command:

  ```py
  connection = sqlite3.connect("my_database.db")
  ```

  ## Disconnect from database

  ```python
  connection.close()
  ```

  ## Querying

  `connect` exposes a `cursor` entity. This is used for executing changes against
  the database file.

  ### Example: creating a table

  ```python
  CREATE_TAGS_TABLE = """
  CREATE TABLE IF NOT EXISTS tags (
      name TEXT PRIMARY KEY
  )
  """

  cursor = connection.cursor()
  cursor.execute(CREATE_TAGS_TABLE)
  cursor.commit()

  # Confirm changes

  cursor.execute("SELECT name from sqlite_master WHERE type='table'")
  print(cursor.fetchall())
  # (tags)
  ```

  Note that it is necessary to `commit` the changes. The cursor object just stores
  the operation in memory and does not execute until you specifically commit them.

  ## Prepared statements and parameterised queries

  ### Parameterised queries

  ```py
  username = "thomas"
  password = "123"
  cursor.execute("SELECT * FROM users WHERE username = ? AND password = ?",
              (username, password))
  ```

  Alternatively, pass the [tuple](Tuples_in_Python.md) directly:

  ```python
  login_data = ("thomas", "123")
  cursor.execute("SELECT * FROM users WHERE username = ? AND password = ?",
  login_data)
  ```

  ### Prepared statements

  > With prepared statements you have to pass in the parameters as a dictionary

  ```python
  user_data = {
      "username": "thomas",
      "email": "thomas@email.com"
  }

  cursor.execute("INSERT INTO users (username, email) VALUES (:username, :email)",
                 user_data)
  ```

  In the example above it so happens that we already have a dictionary we are
  working with. If you want to use a prepared statement and you don't have a
  dictionary already, you will need to construct one as part of the query, e.g:

  ```python
  cursor.execute("INSERT INTO table_name (value_one, value_two) VALUES (:value_one, :value_two)", { value_one: 'foo', value_two: 'bar'})
  ```

raw_content:
  ---
  tags: [python, databases, SQL, SQLite]
  created: Thursday, November 07, 2024
  ---

  # Using SQLite with Python

  SQLite is part of the core library, you do not require a separate package

  ```py
  import sqlite3
  ```

  ## Connect to database

  Connect to an existing SQLite `.db` file or create a new database with the same
  command:

  ```py
  connection = sqlite3.connect("my_database.db")
  ```

  ## Disconnect from database

  ```python
  connection.close()
  ```

  ## Querying

  `connect` exposes a `cursor` entity. This is used for executing changes against
  the database file.

  ### Example: creating a table

  ```python
  CREATE_TAGS_TABLE = """
  CREATE TABLE IF NOT EXISTS tags (
      name TEXT PRIMARY KEY
  )
  """

  cursor = connection.cursor()
  cursor.execute(CREATE_TAGS_TABLE)
  cursor.commit()

  # Confirm changes

  cursor.execute("SELECT name from sqlite_master WHERE type='table'")
  print(cursor.fetchall())
  # (tags)
  ```

  Note that it is necessary to `commit` the changes. The cursor object just stores
  the operation in memory and does not execute until you specifically commit them.

  ## Prepared statements and parameterised queries

  ### Parameterised queries

  ```py
  username = "thomas"
  password = "123"
  cursor.execute("SELECT * FROM users WHERE username = ? AND password = ?",
              (username, password))
  ```

  Alternatively, pass the [tuple](Tuples_in_Python.md) directly:

  ```python
  login_data = ("thomas", "123")
  cursor.execute("SELECT * FROM users WHERE username = ? AND password = ?",
  login_data)
  ```

  ### Prepared statements

  > With prepared statements you have to pass in the parameters as a dictionary

  ```python
  user_data = {
      "username": "thomas",
      "email": "thomas@email.com"
  }

  cursor.execute("INSERT INTO users (username, email) VALUES (:username, :email)",
                 user_data)
  ```

  In the example above it so happens that we already have a dictionary we are
  working with. If you want to use a prepared statement and you don't have a
  dictionary already, you will need to construct one as part of the query, e.g:

  ```python
  cursor.execute("INSERT INTO table_name (value_one, value_two) VALUES (:value_one, :value_two)", { value_one: 'foo', value_two: 'bar'})
  ```


path: 0716531c_rewilding_the_internet.md
sortable_path: 0716531c_rewilding_the_internet.md
title: 0716531c_rewilding_the_internet
checksum: bf356c0a1ef11f4c0f98e42151962507661eb1e1b464a2a2fb1f085aa2ec5999
created: 2024-11-07 09:45:05.17798276+00:00
modified: 2024-11-19 17:43:28.175211723+00:00
metadata: {"created":"Thursday, November 07, 2024","tags":["ecopolsoc","internet"]}
lead:
  The following
  [analogy](https://www.noemamag.com/we-need-to-rewild-the-internet/) can be
  useful in understanding how the internet has become such a negative environment.
  How it has transmogrified from its original promise of a humanistic space that
  would facilitate the connection of individuals based on shared interests and
  values to an environment that is entirely mediated by capital, subject to
  control, censorship and the violation of privacy.

body:
  The following
  [analogy](https://www.noemamag.com/we-need-to-rewild-the-internet/) can be
  useful in understanding how the internet has become such a negative environment.
  How it has transmogrified from its original promise of a humanistic space that
  would facilitate the connection of individuals based on shared interests and
  values to an environment that is entirely mediated by capital, subject to
  control, censorship and the violation of privacy.

  In the 18th-century in the northern German states, officials decided to
  rationalise the production of timber to enable more systematic extraction.
  (Clearly a product of Enlightenment principles of applying reason and science to
  agriculture similar to Jethro Tull's invention of the seed drill and the
  suspension of three-field rotation to four-field all-round harvesting.)

  Instead of a sprawling forest ecosystem that had developed naturally over
  hundreds of thousands of years, which allowed for felling without upsetting the
  continuance of the system, the forests were stripped and trees were planted in
  rows and grids with the arrangement designed to maximise yield and access. This
  would be broadly analogous to the factory-farming of chickens compared to
  free-range rearing.

  For the first cycle it appeared as if this restructuring was a huge success.
  Yields were high and the timber was of excellent quality. This was a false dawn,
  however. The subsequent yields were a complete failure compared to the past.
  Fewer trees grew and those that did had a reduced yield and were subject to
  disease. The overall health of the forests declined precipitously.

  What explains the initial success? It was made possible by the preceding
  centuries of free-range growth and husbandry. The seeming success of the new
  system was a product of the generations of investment in the natural ecosystem.
  This was exhausted in one cycle of the new regime.

  Taking this back to the Web, the analogy is that the web was initially a natural
  ecosystem of different communities and technologies developing organically out
  of the original open and decentralised protocols. From this diversity emerged a
  formidably rich informational space.

  Since then, the Web has become highly centralised and homogenised with a tiny
  cadre of corporations determining how individuals communicate, share and access
  information (Google for search, Android or Apple for phone, Twitter or Facebook
  etc.).

  The initial fruits of this centralisation (just like the forest 'optimisation')
  appeared to be beneficial in the initial 'heyday' of the 2010s when social-media
  was new and full of prospect (the political benefits of Twitter and enriched
  public sphere, the 'Arab Spring', WikiLeaks etc). But this actually proved to be
  a nadir. The vitality on display was a product of the preceding era of the
  natural growth of the Web ecosystem.

  This decline in quality should be apparent to any reflective user:

  - the saturation of advertising to the point where many websites are unusable
  - the confection of low-quality content to game 'organic' search rankings
  - the surveillance of users across sites and devices to enable the extraction of
    profitable marketing data
  - the censorship of political content for the 'violation' of bogus terms of use
    on social media platforms if it does not the serve the broader interests of
    capital and western imperialism
  - the saturation of pornography and dehumanising media more generally

  Thbs has also involved the reshaping of Internet users themselves, from active
  and independent _agents_, to drone-like passive _consumers_ of drivel engineered
  to maximise 'engagement' (the monopolisation of attention), where users are like
  the viewers of _Infinite Jest_. See, e.g TikTok and its clones.

  There are many further points that can be made and ideas that can be
  extrapolated from this. Two examples:

  - the emergence of supposed 'AI' is going to lead to greater content
    homogenisation and the intensification of informational entropy as the
    technological mechanism consists in feeding the regurgitated content of the
    internet back into itself

  - reality itself has become infected by the disease that killed the Web
    (ecocide). The distinction between offline and online is becoming increasingly
    hard to draw (politics - the normalisation and ascendancy of the far right and
    subsequent destabilisation of western states)

raw_content:
  ---
  tags: [ecopolsoc, internet]
  created: Thursday, November 07, 2024
  ---

  # 0716531c_rewilding_the_internet

  The following
  [analogy](https://www.noemamag.com/we-need-to-rewild-the-internet/) can be
  useful in understanding how the internet has become such a negative environment.
  How it has transmogrified from its original promise of a humanistic space that
  would facilitate the connection of individuals based on shared interests and
  values to an environment that is entirely mediated by capital, subject to
  control, censorship and the violation of privacy.

  In the 18th-century in the northern German states, officials decided to
  rationalise the production of timber to enable more systematic extraction.
  (Clearly a product of Enlightenment principles of applying reason and science to
  agriculture similar to Jethro Tull's invention of the seed drill and the
  suspension of three-field rotation to four-field all-round harvesting.)

  Instead of a sprawling forest ecosystem that had developed naturally over
  hundreds of thousands of years, which allowed for felling without upsetting the
  continuance of the system, the forests were stripped and trees were planted in
  rows and grids with the arrangement designed to maximise yield and access. This
  would be broadly analogous to the factory-farming of chickens compared to
  free-range rearing.

  For the first cycle it appeared as if this restructuring was a huge success.
  Yields were high and the timber was of excellent quality. This was a false dawn,
  however. The subsequent yields were a complete failure compared to the past.
  Fewer trees grew and those that did had a reduced yield and were subject to
  disease. The overall health of the forests declined precipitously.

  What explains the initial success? It was made possible by the preceding
  centuries of free-range growth and husbandry. The seeming success of the new
  system was a product of the generations of investment in the natural ecosystem.
  This was exhausted in one cycle of the new regime.

  Taking this back to the Web, the analogy is that the web was initially a natural
  ecosystem of different communities and technologies developing organically out
  of the original open and decentralised protocols. From this diversity emerged a
  formidably rich informational space.

  Since then, the Web has become highly centralised and homogenised with a tiny
  cadre of corporations determining how individuals communicate, share and access
  information (Google for search, Android or Apple for phone, Twitter or Facebook
  etc.).

  The initial fruits of this centralisation (just like the forest 'optimisation')
  appeared to be beneficial in the initial 'heyday' of the 2010s when social-media
  was new and full of prospect (the political benefits of Twitter and enriched
  public sphere, the 'Arab Spring', WikiLeaks etc). But this actually proved to be
  a nadir. The vitality on display was a product of the preceding era of the
  natural growth of the Web ecosystem.

  This decline in quality should be apparent to any reflective user:

  - the saturation of advertising to the point where many websites are unusable
  - the confection of low-quality content to game 'organic' search rankings
  - the surveillance of users across sites and devices to enable the extraction of
    profitable marketing data
  - the censorship of political content for the 'violation' of bogus terms of use
    on social media platforms if it does not the serve the broader interests of
    capital and western imperialism
  - the saturation of pornography and dehumanising media more generally

  Thbs has also involved the reshaping of Internet users themselves, from active
  and independent _agents_, to drone-like passive _consumers_ of drivel engineered
  to maximise 'engagement' (the monopolisation of attention), where users are like
  the viewers of _Infinite Jest_. See, e.g TikTok and its clones.

  There are many further points that can be made and ideas that can be
  extrapolated from this. Two examples:

  - the emergence of supposed 'AI' is going to lead to greater content
    homogenisation and the intensification of informational entropy as the
    technological mechanism consists in feeding the regurgitated content of the
    internet back into itself

  - reality itself has become infected by the disease that killed the Web
    (ecocide). The distinction between offline and online is becoming increasingly
    hard to draw (politics - the normalisation and ascendancy of the far right and
    subsequent destabilisation of western states)


path: Batch_processing.md
sortable_path: Batch_processing.md
title: Batch processing
checksum: 9b37ad2702ca729fa1f71b0ece5693e8711ef2f10cd46601aa23242ae92dd22d
created: 2024-11-12 07:13:27.501226554+00:00
modified: 2024-11-12 07:34:48.457649692+00:00
metadata: {"created":"Tuesday, November 12, 2024","tags":["computer-history"]}
lead:
  With modern computer systems programs can be executed continuously and as
  needed. With earlier computers (e.g. mainframes or mini-computers that were
  shared by multiple users throughout an organisation) operating on this basis
  would be expensive and would squander resources. It would also be inefficient
  because computers needed time to warm up and cool down between operations.

body:
  With modern computer systems programs can be executed continuously and as
  needed. With earlier computers (e.g. mainframes or mini-computers that were
  shared by multiple users throughout an organisation) operating on this basis
  would be expensive and would squander resources. It would also be inefficient
  because computers needed time to warm up and cool down between operations.

  Batch processing meant waiting until the required data for the operation met a
  certain volume or threshold and then running the job. This way all the computer
  resources were used at once instead repeatedly over an extended period.

  Running operations in batch rather than in real-time also meant you could time
  when you used the computer, running your jobs during off-peak hours when more
  resources would be available. Particularly beneficial during the era of
  time-sharing.

raw_content:
  ---
  tags: [computer-history]
  created: Tuesday, November 12, 2024
  ---

  # Batch processing

  With modern computer systems programs can be executed continuously and as
  needed. With earlier computers (e.g. mainframes or mini-computers that were
  shared by multiple users throughout an organisation) operating on this basis
  would be expensive and would squander resources. It would also be inefficient
  because computers needed time to warm up and cool down between operations.

  Batch processing meant waiting until the required data for the operation met a
  certain volume or threshold and then running the job. This way all the computer
  resources were used at once instead repeatedly over an extended period.

  Running operations in batch rather than in real-time also meant you could time
  when you used the computer, running your jobs during off-peak hours when more
  resources would be available. Particularly beneficial during the era of
  time-sharing.


path: Usenet.md
sortable_path: Usenet.md
title: Usenet
checksum: 2445b5be1ea783e6fd2700023528ec68df8054271bc568eedd5f0601ba5f3c69
created: 2024-11-12 07:36:26.966774102+00:00
modified: 2024-11-12 08:26:14.724821538+00:00
metadata: {"created":"Tuesday, November 12, 2024","tags":["networks","internet"]}
lead:
  A decentralised system of computer servers that allowed users to post messages
  to topic-based discussion groups called "newsgroups" which other users could
  read and reply to.

body:
  A decentralised system of computer servers that allowed users to post messages
  to topic-based discussion groups called "newsgroups" which other users could
  read and reply to.

  Arguably the first social network and the oldest internet network that is still
  operational.

  It was designed by staff at two separate universities. Both used Unix systems
  and wanted a way to communicate with each other.

  ## Technology

  Usenet comprised a [distributed network](385af4b4_Baran_distributed_networks.md)
  of dedicated servers that exchanged messages over the **Network News Transfer
  Protocol** (NNTP).

  A user would post a message to their local news server. The server would assign
  it a UUID and then share it with other peer servers that it was connected to,
  who would also share it, repeating the process. This created a ripple effect
  that would propagate the message over the global network. Because of the
  distributed nature of this network it meant that messages would have a high
  longevity and retention rate and that it was effectively impossible to delete a
  message once posted, since it would be stored across a multitude of servers.

  ## Relation to the Internet

  Usenet developed separately and in parallel with the Internet. Originally it did
  not operate over the Internet: servers would connect to each other over phone
  lines using the Unix-to-Unix Copy Protocol. In the late 1980s when the Internet
  became more widespread and not just limited to DARPA-adjacent universities,
  Usenet transitioned to running over
  [TCP/IP](Transport_Layer_of_Internet_Protocol.md) networks.

  ## Newsgroup hierarchy

  Usenet used a structured, dot-separated naming system for newsgroups that got
  more specific with each subdivision, e.g:

  - comp.sys.mac
  - sci.physics.particle
  - alt.music.nirvana

  Users would subscribe to the newsgroups that interested them and their
  newsreader (which connected to the servers) would only download messages from
  these groups.

  ## Influence

  - Created the concept of threaded discussions in its reply structure
  - Reddit appropriated the concept of topic-based communities in its sub-reddits
    and also used threading
  - RSS adopted Usenet's model of subscribing to specific information sources and
    getting updates: the concept of "pulling" new content from feeds mirrors how
    newsreaders worked
  - Distributed social networks like Mastodon/ActivityPub use a similar
    server-to-server propagation model that Usenet pioneered
  - [BitTorrent's](Torrenting.md) distributed file-sharing model has similarities
    to how binary files were distributed across Usenet servers

  ## "Eternal September"

  For most of its early history Usenet was limited to university students whose
  university was part of the network. Each September new students would join the
  university and this led to an influx of inexperienced Usenet users who didn't
  yet know the etiquette and culture. They would have to be schooled and inducted.

  In the mid-1980s Usenet became accessible to the general public as ISPs started
  granting access. In 1993-1994, AOL granted access to subscribers. This changed
  the nature of discussions from primarily academic and technical topics to
  include more general interest and entertainment content, leading to the creation
  of the "alt." hierarchy which allowed anyone to create new newsgroups without
  formal approval processes that were required in the traditional hierarchies.
  Hence it was "always September".

  This phrase is often used as a shorthand for the way in which the Internet and
  Web has moved from a niche community of technical users to a demotic public
  space with an attendant decline in perceived value.

raw_content:
  ---
  tags: [networks, internet]
  created: Tuesday, November 12, 2024
  ---

  # Usenet

  A decentralised system of computer servers that allowed users to post messages
  to topic-based discussion groups called "newsgroups" which other users could
  read and reply to.

  Arguably the first social network and the oldest internet network that is still
  operational.

  It was designed by staff at two separate universities. Both used Unix systems
  and wanted a way to communicate with each other.

  ## Technology

  Usenet comprised a [distributed network](385af4b4_Baran_distributed_networks.md)
  of dedicated servers that exchanged messages over the **Network News Transfer
  Protocol** (NNTP).

  A user would post a message to their local news server. The server would assign
  it a UUID and then share it with other peer servers that it was connected to,
  who would also share it, repeating the process. This created a ripple effect
  that would propagate the message over the global network. Because of the
  distributed nature of this network it meant that messages would have a high
  longevity and retention rate and that it was effectively impossible to delete a
  message once posted, since it would be stored across a multitude of servers.

  ## Relation to the Internet

  Usenet developed separately and in parallel with the Internet. Originally it did
  not operate over the Internet: servers would connect to each other over phone
  lines using the Unix-to-Unix Copy Protocol. In the late 1980s when the Internet
  became more widespread and not just limited to DARPA-adjacent universities,
  Usenet transitioned to running over
  [TCP/IP](Transport_Layer_of_Internet_Protocol.md) networks.

  ## Newsgroup hierarchy

  Usenet used a structured, dot-separated naming system for newsgroups that got
  more specific with each subdivision, e.g:

  - comp.sys.mac
  - sci.physics.particle
  - alt.music.nirvana

  Users would subscribe to the newsgroups that interested them and their
  newsreader (which connected to the servers) would only download messages from
  these groups.

  ## Influence

  - Created the concept of threaded discussions in its reply structure
  - Reddit appropriated the concept of topic-based communities in its sub-reddits
    and also used threading
  - RSS adopted Usenet's model of subscribing to specific information sources and
    getting updates: the concept of "pulling" new content from feeds mirrors how
    newsreaders worked
  - Distributed social networks like Mastodon/ActivityPub use a similar
    server-to-server propagation model that Usenet pioneered
  - [BitTorrent's](Torrenting.md) distributed file-sharing model has similarities
    to how binary files were distributed across Usenet servers

  ## "Eternal September"

  For most of its early history Usenet was limited to university students whose
  university was part of the network. Each September new students would join the
  university and this led to an influx of inexperienced Usenet users who didn't
  yet know the etiquette and culture. They would have to be schooled and inducted.

  In the mid-1980s Usenet became accessible to the general public as ISPs started
  granting access. In 1993-1994, AOL granted access to subscribers. This changed
  the nature of discussions from primarily academic and technical topics to
  include more general interest and entertainment content, leading to the creation
  of the "alt." hierarchy which allowed anyone to create new newsgroups without
  formal approval processes that were required in the traditional hierarchies.
  Hence it was "always September".

  This phrase is often used as a shorthand for the way in which the Internet and
  Web has moved from a niche community of technical users to a demotic public
  space with an attendant decline in perceived value.


path: SQLite.md
sortable_path: SQLite.md
title: SQLite
lead: ## Enabling foreign key constraints
checksum: 2eb67715c75bdedf756b1e5b91c55501863d3698d1c868a22f736eda6d9055ec
created: 2024-11-13 18:12:28.941320219+00:00
modified: 2024-12-27 15:08:29.205514805+00:00
metadata: {"created":"Wednesday, November 13, 2024","tags":["databases","SQL","SQLite"]}
body:
  ## Enabling foreign key constraints

  Foreign key constraints are not enabled by default - it is very permissive. You
  must manually turn them on, as part of the connection process:

  ```py
  connection = sqlite3.connect("my_database.db")
  connection.execute("PRAGMA foreign_keys = ON")
  ```

  ## INSERT OR IGNORE INTO

  If table has a `UNIQUE` constraint on a field, insert if it is unique otherwise
  skip without throwing a constraint error.

  ```sql
  INSERT OR IGNORE INTO table_name (value) VALUES (:value)
  ```

  ## exec and fetchall

  For `CREATE`, `INSERT`, `UPDATE`, `DELETE` operations it is sufficient to simply
  use `execute` to carry out the operations:

  ```py
  cursor = connection.cursor()
  cursor.execute(sql)
  ```

  For `SELECT` operations, you must add a return statement that uses `fetchall`:

  ```py
  sql = "SELECT * FROM table"
  cursor = connection.cursor()
  cursor.execute(sql)
  return cursor.fetchall()
  ```

  ## SELECT response data type

  A `SELECT` operation returns a [list](Lists_in_Python.md) of
  [tuples](Tuples_in_Python.md) where each entry in the tuple is a returned field.

  For example:

  ```py
  entries_sql = "SELECT title, size FROM entries"

  entries_response = cursor.execute(entries_sql)
  print(cursor.fetchall())
  # [('Lorem ipsum', 127), ('Dolar sit', 4231)]
  ```

raw_content:
  ---
  tags: [databases, SQL, SQLite]
  created: Wednesday, November 13, 2024
  ---

  # SQLite

  ## Enabling foreign key constraints

  Foreign key constraints are not enabled by default - it is very permissive. You
  must manually turn them on, as part of the connection process:

  ```py
  connection = sqlite3.connect("my_database.db")
  connection.execute("PRAGMA foreign_keys = ON")
  ```

  ## INSERT OR IGNORE INTO

  If table has a `UNIQUE` constraint on a field, insert if it is unique otherwise
  skip without throwing a constraint error.

  ```sql
  INSERT OR IGNORE INTO table_name (value) VALUES (:value)
  ```

  ## exec and fetchall

  For `CREATE`, `INSERT`, `UPDATE`, `DELETE` operations it is sufficient to simply
  use `execute` to carry out the operations:

  ```py
  cursor = connection.cursor()
  cursor.execute(sql)
  ```

  For `SELECT` operations, you must add a return statement that uses `fetchall`:

  ```py
  sql = "SELECT * FROM table"
  cursor = connection.cursor()
  cursor.execute(sql)
  return cursor.fetchall()
  ```

  ## SELECT response data type

  A `SELECT` operation returns a [list](Lists_in_Python.md) of
  [tuples](Tuples_in_Python.md) where each entry in the tuple is a returned field.

  For example:

  ```py
  entries_sql = "SELECT title, size FROM entries"

  entries_response = cursor.execute(entries_sql)
  print(cursor.fetchall())
  # [('Lorem ipsum', 127), ('Dolar sit', 4231)]
  ```


path: Terraform.md
sortable_path: Terraform.md
title: Terraform
lead: ## Authentication
checksum: 389c99f29af9b45f5cfbb202a791cf8718bb9492927bac18062032aaf822e9fe
created: 2024-11-16 15:05:11.481499315+00:00
modified: 2024-11-17 11:45:30.497508673+00:00
metadata: {"created":"Saturday, November 16, 2024","tags":["AWS","IaC"]}
body:
  ## Authentication

  Terraform will default to the currently active AWS user providing you have your
  [AWS credentials](/zk/AWS_CLI.md) set-up.

  ## Creating resources

  > Resources are specified in the `main.tf` file.

  Create a new S3 bucket:

  ```hcl
  resource "aws_s3_bucket" "my_new_bucket" {
      bucket = "my-new-bucket"
  }
  ```

  At this point the resource doesn't exist so there is no ARN or ID. When the
  resource is created (through `terraform apply`) Terraform receives this info
  from AWS and store it in a state file.

  This info can be retrieved with `terraform show`, e.g:

  ```
  resource "aws_s3_bucket" "my_new_bucket" {
      id                    = "my-new-bucket"
      arn                   = "arn:aws:s3:::my-new-bucket"
      bucket                = "my-new-bucket"
      bucket_domain_name    = "my-new-bucket.s3.amazonaws.com"
   }

  ```

  ## Standard sequence

  Make your `terraform` directory, then `cd` and:

  ### Initialise project

  ```
  terraform init
  ```

  This will create the file `.terraform.lock.hcl` which should be committed. Stuff
  in `.terraform/` can be git-ignored.

  ### Check changes

  ```
  terraform plan
  ```

  Providing you have written a `main.tf`. The command will list the proposed
  changes to your resources.

  ### Apply changes

  ```
  terraform apply
  ```

  ### Destroy the resource

  Typically when you want to do a clean recreate.

  ```
  terraform destoy
  ```

  ## Run with debug/logger

  ```
  TF_LOG=DEBUG terraform apply
  ```

  Or use with other Terraform command, e.g. `destroy`, `refresh` - whatever is
  causing the problem.

raw_content:
  ---
  tags: [AWS, IaC]
  created: Saturday, November 16, 2024
  ---

  # Terraform

  ## Authentication

  Terraform will default to the currently active AWS user providing you have your
  [AWS credentials](/zk/AWS_CLI.md) set-up.

  ## Creating resources

  > Resources are specified in the `main.tf` file.

  Create a new S3 bucket:

  ```hcl
  resource "aws_s3_bucket" "my_new_bucket" {
      bucket = "my-new-bucket"
  }
  ```

  At this point the resource doesn't exist so there is no ARN or ID. When the
  resource is created (through `terraform apply`) Terraform receives this info
  from AWS and store it in a state file.

  This info can be retrieved with `terraform show`, e.g:

  ```
  resource "aws_s3_bucket" "my_new_bucket" {
      id                    = "my-new-bucket"
      arn                   = "arn:aws:s3:::my-new-bucket"
      bucket                = "my-new-bucket"
      bucket_domain_name    = "my-new-bucket.s3.amazonaws.com"
   }

  ```

  ## Standard sequence

  Make your `terraform` directory, then `cd` and:

  ### Initialise project

  ```
  terraform init
  ```

  This will create the file `.terraform.lock.hcl` which should be committed. Stuff
  in `.terraform/` can be git-ignored.

  ### Check changes

  ```
  terraform plan
  ```

  Providing you have written a `main.tf`. The command will list the proposed
  changes to your resources.

  ### Apply changes

  ```
  terraform apply
  ```

  ### Destroy the resource

  Typically when you want to do a clean recreate.

  ```
  terraform destoy
  ```

  ## Run with debug/logger

  ```
  TF_LOG=DEBUG terraform apply
  ```

  Or use with other Terraform command, e.g. `destroy`, `refresh` - whatever is
  causing the problem.


path: Bash_file_exists.md
sortable_path: Bash_file_exists.md
title: Bash file exists
checksum: 0e5b993b84d6938d458194bd0f7727110ddcdf0c9c2bae79156a67527ec41d9d
created: 2024-11-17 11:46:48.437505596+00:00
modified: 2024-11-17 15:07:20.621890261+00:00
metadata: {"created":"Sunday, November 17, 2024","tags":["shell"]}
lead:
  ```bash
  FILE_PATH="$PWD/foo/bar.json"
  if [ ! -f "$FILE_PATH" ]; then
  	echo -e "ERROR Graph file ( $FILE_PATH ) does not exist. Exiting."
  	exit 0
  fi
  ```

body:
  ```bash
  FILE_PATH="$PWD/foo/bar.json"
  if [ ! -f "$FILE_PATH" ]; then
  	echo -e "ERROR Graph file ( $FILE_PATH ) does not exist. Exiting."
  	exit 0
  fi
  ```

raw_content:
  ---
  tags: [shell]
  created: Sunday, November 17, 2024
  ---

  # Bash file exists

  ```bash
  FILE_PATH="$PWD/foo/bar.json"
  if [ ! -f "$FILE_PATH" ]; then
  	echo -e "ERROR Graph file ( $FILE_PATH ) does not exist. Exiting."
  	exit 0
  fi
  ```


path: Bash_colour_output.md
sortable_path: Bash_colour_output.md
title: Bash colour output
lead: Define colours:
checksum: 4eb8a64d547a1eff1e99e0ff1fa35449d6abb963a2610540a25d576d0662ceac
created: 2024-11-17 11:49:04.660212914+00:00
modified: 2025-02-08 20:08:44.878468586+00:00
metadata: {"created":"Sunday, November 17, 2024","tags":["shell"]}
body:
  Define colours:

  ```bash
  RED='\033[0;31m'
  GREEN='\033[0;32m'
  BLUE='\033[0;34m'
  NO_COLOR='\033[0m'
  ```

  Interpolate in string:

  ```bash
  echo -e "${BLUE}INFO Uploading graph file...${NO_COLOR}"
  ```

  Note: must end with the blank colour otherwise everything subsequent will be in
  blue.

raw_content:
  ---
  tags: [shell]
  created: Sunday, November 17, 2024
  ---

  # Bash colour output

  Define colours:

  ```bash
  RED='\033[0;31m'
  GREEN='\033[0;32m'
  BLUE='\033[0;34m'
  NO_COLOR='\033[0m'
  ```

  Interpolate in string:

  ```bash
  echo -e "${BLUE}INFO Uploading graph file...${NO_COLOR}"
  ```

  Note: must end with the blank colour otherwise everything subsequent will be in
  blue.


path: Bash_store_status_of_command_execution.md
sortable_path: Bash_store_status_of_command_execution.md
title: Bash store status of command execution
checksum: 224d99ee81dd46f5dd86f7d3909c5af9e79564dad5bbc32bff669259ea70a6f7
created: 2024-11-17 11:52:00.945353278+00:00
modified: 2024-11-17 15:07:07.868344074+00:00
metadata: {"created":"Sunday, November 17, 2024","tags":["shell","AWS","S3"]}
lead:
  We can store the outcome of a command in a variable, then check this variable to
  derive the status and then provide custom outputs or other branching logic.

body:
  We can store the outcome of a command in a variable, then check this variable to
  derive the status and then provide custom outputs or other branching logic.

  Here is an example of adding an object to a bucket in S3:

  ```bash
  aws s3api put-object --bucket "${BUCKET_NAME}" --key "${FILE}" --body "${FILE_PATH}" >/dev/null 2>&1
  STATUS=$?

  if [ $STATUS -eq 0 ]; then
  	echo -e "SUCCESS File uploaded."
  else
  	echo -e "Upload failed with status $STATUS."
  fi
  ```

raw_content:
  ---
  tags: [shell, AWS, S3]
  created: Sunday, November 17, 2024
  ---

  # Bash store status of command execution

  We can store the outcome of a command in a variable, then check this variable to
  derive the status and then provide custom outputs or other branching logic.

  Here is an example of adding an object to a bucket in S3:

  ```bash
  aws s3api put-object --bucket "${BUCKET_NAME}" --key "${FILE}" --body "${FILE_PATH}" >/dev/null 2>&1
  STATUS=$?

  if [ $STATUS -eq 0 ]; then
  	echo -e "SUCCESS File uploaded."
  else
  	echo -e "Upload failed with status $STATUS."
  fi
  ```


path: Bauman_quote.md
sortable_path: Bauman_quote.md
title: Bauman quote
checksum: 6f4b7b7158946e4696ece1fe7fe94e28c690fef353fed8fb54ab7abd91ad24d4
created: 2024-11-19 17:35:01.869241943+00:00
modified: 2024-12-01 19:45:51.094616521+00:00
metadata: {"created":"Tuesday, November 19, 2024","tags":["ecopolsoc","surveillance-capitalism"]}
lead:
  _The yawning gap between the right of self-assertion and the capacity to control
  the social settings which render such self-assertion feasible. It is from that
  abysmal gap that the most poisonous effluvia contaminating the lives of
  contemporary individuals emanate_.

body:
  _The yawning gap between the right of self-assertion and the capacity to control
  the social settings which render such self-assertion feasible. It is from that
  abysmal gap that the most poisonous effluvia contaminating the lives of
  contemporary individuals emanate_.

  Quoted in _The Age of Surveillance Capitalism_ (2019): 45

raw_content:
  ---
  tags: [ecopolsoc, surveillance-capitalism]
  created: Tuesday, November 19, 2024
  ---

  # Bauman quote

  _The yawning gap between the right of self-assertion and the capacity to control
  the social settings which render such self-assertion feasible. It is from that
  abysmal gap that the most poisonous effluvia contaminating the lives of
  contemporary individuals emanate_.

  Quoted in _The Age of Surveillance Capitalism_ (2019): 45


path: AWS_SQS.md
sortable_path: AWS_SQS.md
title: AWS SQS
checksum: 8fc99f527cb7d66f20b0bed1b5c9df452740c9d316d7c9c356990b0755abf72d
created: 2024-11-22 17:22:10.666427159+00:00
modified: 2024-11-25 06:50:57.193178069+00:00
metadata: {"tags":["AWS","SQS"]}
lead:
  SQS ("Simple Queue Service") is a service that allows you to send, store and
  receive messages between apps and software components built in AWS. It helps
  with decoupling and scaling.

body:
  SQS ("Simple Queue Service") is a service that allows you to send, store and
  receive messages between apps and software components built in AWS. It helps
  with decoupling and scaling.

  > Amazon SQS is a distributed queue system that enables web service applications
  > to quickly and reliably queue messages that one component in the application
  > generates to be consumed by another component where a queue is a temporary
  > repository for messages that are awaiting processing.

  As the name indicates, its operating mode is that of a [queue](Queue.md) data
  structure offering first-in, first-out and other queue implementations.

  ## Example use case

  ![SQS example diagram](../img/SQS-example.png)

  A request is made to an [API Gateway](API_Gateway.md) endpoint with a body. The
  body is then parsed and inserted into a database.

  The benefit of adding SQS as a buffer between the request and the updating of
  the database:

  - It can better handle spikes in activity, buffering requests to the database
    until it is ready to handle them. This prevents the messages getting lost if
    the server is overloaded

  - There is a retry mechanism built into SQS. If the database write fails, the
    message stays in the queue allowing for retries

  - It facilitates decoupling. Without SQS as the middleman the responsibilites of
    the lambda would be compounded - it would receive requests and update the DB,
    plus any additional processes such sending a message to SNS. In the solution
    we have two lambdas co-ordinating actions in a decouped manner via SQS.

  ## Dead letters

  As SQS allows for multiple retries we could end up in a situation where a
  malformed message is continually processed in a loop. To avoid this you can set
  a maxiumum retry limit and, when this is exceeded, shift the problematic message
  to the dead letter queue (DLQ) and remove it from the main queue.

  ![SQS deadletter example](../img/SQS-deadletter.png)

  If the DLQ reaches a certain threshold this can trigger additional handling such
  as raising an Alert in [CloudWatch](./AWS_CloudWatch.md) or other monitoring
  tool.

  Note that a DLQ is not a distinct entity within the SDK, it is just another SQS
  queue that is designated to store failures.

  See [AWS SDK Syntax](AWS_SQS_SDK.md).

raw_content:
  ---
  tags: [AWS, SQS]
  ---

  # AWS SQS

  SQS ("Simple Queue Service") is a service that allows you to send, store and
  receive messages between apps and software components built in AWS. It helps
  with decoupling and scaling.

  > Amazon SQS is a distributed queue system that enables web service applications
  > to quickly and reliably queue messages that one component in the application
  > generates to be consumed by another component where a queue is a temporary
  > repository for messages that are awaiting processing.

  As the name indicates, its operating mode is that of a [queue](Queue.md) data
  structure offering first-in, first-out and other queue implementations.

  ## Example use case

  ![SQS example diagram](../img/SQS-example.png)

  A request is made to an [API Gateway](API_Gateway.md) endpoint with a body. The
  body is then parsed and inserted into a database.

  The benefit of adding SQS as a buffer between the request and the updating of
  the database:

  - It can better handle spikes in activity, buffering requests to the database
    until it is ready to handle them. This prevents the messages getting lost if
    the server is overloaded

  - There is a retry mechanism built into SQS. If the database write fails, the
    message stays in the queue allowing for retries

  - It facilitates decoupling. Without SQS as the middleman the responsibilites of
    the lambda would be compounded - it would receive requests and update the DB,
    plus any additional processes such sending a message to SNS. In the solution
    we have two lambdas co-ordinating actions in a decouped manner via SQS.

  ## Dead letters

  As SQS allows for multiple retries we could end up in a situation where a
  malformed message is continually processed in a loop. To avoid this you can set
  a maxiumum retry limit and, when this is exceeded, shift the problematic message
  to the dead letter queue (DLQ) and remove it from the main queue.

  ![SQS deadletter example](../img/SQS-deadletter.png)

  If the DLQ reaches a certain threshold this can trigger additional handling such
  as raising an Alert in [CloudWatch](./AWS_CloudWatch.md) or other monitoring
  tool.

  Note that a DLQ is not a distinct entity within the SDK, it is just another SQS
  queue that is designated to store failures.

  See [AWS SDK Syntax](AWS_SQS_SDK.md).


path: JavaScript_module_types.md
sortable_path: JavaScript_module_types.md
title: JavaScript module types
checksum: 32ee0dc1c2405758a7981a08a63258512ef33f621d465370885dd3a057a40eff
created: 2024-11-24 10:32:03.592097816+00:00
modified: 2024-11-24 10:46:32.193244857+00:00
metadata: {"created":"Sunday, November 24, 2024","tags":["javascript"]}
lead:
  The following info is useful when you wish to use modern JavaScript tooling
  outside of a build system.

body:
  The following info is useful when you wish to use modern JavaScript tooling
  outside of a build system.

  For instance I wanted to use a bundled React application in a static website not
  created using NPM.

  Great account by
  [Julia Evans](https://jvns.ca/blog/2024/11/18/how-to-import-a-javascript-library/).

  In essence there are three main module formats:

  - CommonJS

    - Native Node JS module system using `require()` and `module.exports()`

  - ESModules

    - The defacto standard that works in both the browser and Node.js

  - AMD

    - A browser-friendly version of CommonJS since superseded by ESModules

  Then there is UMD, _Universal Module Definition_. This is a wrapper that
  combines compatibility for multiple module systems and works with AMD, CommonJS,
  as well as global variables.

raw_content:
  ---
  tags: [javascript]
  created: Sunday, November 24, 2024
  ---

  # JavaScript module types

  The following info is useful when you wish to use modern JavaScript tooling
  outside of a build system.

  For instance I wanted to use a bundled React application in a static website not
  created using NPM.

  Great account by
  [Julia Evans](https://jvns.ca/blog/2024/11/18/how-to-import-a-javascript-library/).

  In essence there are three main module formats:

  - CommonJS

    - Native Node JS module system using `require()` and `module.exports()`

  - ESModules

    - The defacto standard that works in both the browser and Node.js

  - AMD

    - A browser-friendly version of CommonJS since superseded by ESModules

  Then there is UMD, _Universal Module Definition_. This is a wrapper that
  combines compatibility for multiple module systems and works with AMD, CommonJS,
  as well as global variables.


path: a771a6d9_attributes_of_internet_enclosure.md
sortable_path: a771a6d9_attributes_of_internet_enclosure.md
title: a771a6d9_defining_internet_enclosure
lead: ## Defining _enclosure_
checksum: 18744f42130d70d76167fc467e6607c602071cb923634b0c0a74ce43b3421edb
created: 2024-11-25 06:33:56.472756619+00:00
modified: 2024-11-24 10:52:28.496405398+00:00
metadata: {"created":"Friday, November 22, 2024","tags":["ecopolsoc","internet"]}
body:
  ## Defining _enclosure_

  The attempt by capital interests to encircle, control and contain networked
  digital communication with the object of extracting financial value from the
  exchanges and the community from which they arise.

  This constitutes the commodification of a human-constituted resource, like
  labour rather than a natural resource, like oil.

  ## Main characteristics

  ### Strategic denial of alternative communication media

  Where there is a _multiplicity_ of media and within media, a multiplicity of
  channels, capital steers communication toward a _single_ entity.

  In a "democratic free-market society", capital cannot simply ban alternative
  media, it can however work to diminish the influence of alternatives through the
  strategic denial of interoperability and the aquisition and subsumption of
  rivals into its system.

  ### Tyrannical control of the rules of communication

  Where the standards of communication are elective and established organically
  and collaboratively between peers (possibly even democratically), capital
  enforces rules derived externally and applied in an authoritarian manner. These
  rules are mutable but changes are enacted unilaterally and arbitrarily by the
  owners of the medium, not its users. Hence, tyranny in its technical political
  sense.

  ### Total control of information and content

  Where individuals share information freely in collaborative enterprise with
  diminished concern for ownership (typically favouring citation and iteration as
  models), capital takes full possession of the information generated such that it
  can be removed and destroyed at will without consultation.

  ### Surveillance

  Where communication is anonymised (loosly or strictly (i.e. through
  cryptographic means)), capital _surveils_ all exchanges and the behaviour of
  community members.

  Typically this is not _personal_ in the sense of being interested in the
  specific user's identity, rather it is abstracted into a composite of the
  person's preferences which may or may not disclose their legal identity (i.e
  metadata).

  This characteristic is both an _attribute_ of internet enclosure and its
  ultimate reason for existence. It is critical to the mechanism of extraction
  since it provides both data that can be sold to other capital interests and
  content that can be used to sell to community members as consumers.

  ### Artificial promotion of commercial content and prolongation of engagement

  Where engagement with media is idiosyncratic and organic, capital circumscribes
  the field. This is done to prolong engagement.

  At one pole this is achieved through the excitation of visceral responses by
  artificially favouring content that titillates, disgusts or angers the user. At
  the other, it is achieved through mollification and the coddling of the user.
  The user encounters only like-minded fellows who affirm and repeat his existing
  beliefs and values, creating community and a sense of safety through the
  othering of contrary viewpoints. Both techniques are demagogic in nature
  familiar from the control mechanisms of cults and extreme political ideologies.

raw_content:
  ---
  tags: [ecopolsoc, internet]
  created: Friday, November 22, 2024
  ---

  # a771a6d9_defining_internet_enclosure

  ## Defining _enclosure_

  The attempt by capital interests to encircle, control and contain networked
  digital communication with the object of extracting financial value from the
  exchanges and the community from which they arise.

  This constitutes the commodification of a human-constituted resource, like
  labour rather than a natural resource, like oil.

  ## Main characteristics

  ### Strategic denial of alternative communication media

  Where there is a _multiplicity_ of media and within media, a multiplicity of
  channels, capital steers communication toward a _single_ entity.

  In a "democratic free-market society", capital cannot simply ban alternative
  media, it can however work to diminish the influence of alternatives through the
  strategic denial of interoperability and the aquisition and subsumption of
  rivals into its system.

  ### Tyrannical control of the rules of communication

  Where the standards of communication are elective and established organically
  and collaboratively between peers (possibly even democratically), capital
  enforces rules derived externally and applied in an authoritarian manner. These
  rules are mutable but changes are enacted unilaterally and arbitrarily by the
  owners of the medium, not its users. Hence, tyranny in its technical political
  sense.

  ### Total control of information and content

  Where individuals share information freely in collaborative enterprise with
  diminished concern for ownership (typically favouring citation and iteration as
  models), capital takes full possession of the information generated such that it
  can be removed and destroyed at will without consultation.

  ### Surveillance

  Where communication is anonymised (loosly or strictly (i.e. through
  cryptographic means)), capital _surveils_ all exchanges and the behaviour of
  community members.

  Typically this is not _personal_ in the sense of being interested in the
  specific user's identity, rather it is abstracted into a composite of the
  person's preferences which may or may not disclose their legal identity (i.e
  metadata).

  This characteristic is both an _attribute_ of internet enclosure and its
  ultimate reason for existence. It is critical to the mechanism of extraction
  since it provides both data that can be sold to other capital interests and
  content that can be used to sell to community members as consumers.

  ### Artificial promotion of commercial content and prolongation of engagement

  Where engagement with media is idiosyncratic and organic, capital circumscribes
  the field. This is done to prolong engagement.

  At one pole this is achieved through the excitation of visceral responses by
  artificially favouring content that titillates, disgusts or angers the user. At
  the other, it is achieved through mollification and the coddling of the user.
  The user encounters only like-minded fellows who affirm and repeat his existing
  beliefs and values, creating community and a sense of safety through the
  othering of contrary viewpoints. Both techniques are demagogic in nature
  familiar from the control mechanisms of cults and extreme political ideologies.


path: AWS_SQS_SDK.md
sortable_path: AWS_SQS_SDK.md
title: AWS SQS SDK
lead: ## Initialising the SQS client
checksum: b721c137c34d710b811bc81e027c3913e17579210602c6cd6870bbbac5751496
created: 2024-11-25 06:33:56.480592735+00:00
modified: 2024-11-25 06:50:09.432367712+00:00
metadata: {"created":"Monday, November 25, 2024","tags":["AWS","SQS"]}
body:
  ## Initialising the SQS client

  ```ts
  import { SQSClient } from "@aws-sdk/client-sqs";

  const sqs = new SQSClient({ region: "us-east-1" });
  ```

  ## Sending a message to the queue

  ```ts
  import { SendMessageCommand } from "@aws-sdk/client-sqs";

  const command = new SendMessageCommand({
    QueueUrl: queueUrl,
    MessageBody: messageBody,
    // Optional parameters
    DelaySeconds: 0,
    MessageAttributes: {
      CustomAttribute: {
        DataType: "String",
        StringValue: "CustomValue",
      },
    },
  });

  return await sqs.send(command);
  ```

  ## Processing messages on the queue

  ```ts
  import { ReceiveMessageCommand } from "@aws-sdk/client-sqs";

  const command = new ReceiveMessageCommand({
    QueueUrl: queueUrl,
    MaxNumberOfMessages: 10,
    WaitTimeSeconds: 20, // Long polling
    VisibilityTimeout: 30, // Time other consumers won't see this message
  });

  const response = await sqs.send(command);

  // Then do something

  for (const message of response.Messages || []) {
    console.log(message);
  }
  ```

  ## Removing message from queue

  ```ts
  import { DeleteMessageCommand } from "@aws-sdk/client-sqs";

  for (const message of response.Messages || []) {
    await sqs.send(
      new DeleteMessageCommand({
        QueueUrl: queueUrl,
        ReceiptHandle: message.ReceiptHandle,
      }),
    );
  }
  ```

  ## Query a DLQ

  ```ts
  const checkDLQ = async (dlqUrl: string) => {
    const command = new GetQueueAttributesCommand({
      QueueUrl: dlqUrl,
      AttributeNames: ["ApproximateNumberOfMessages"],
    });

    const response = await sqs.send(command);
    return response.Attributes?.ApproximateNumberOfMessages;
  };
  ```

raw_content:
  ---
  tags: [AWS, SQS]
  created: Monday, November 25, 2024
  ---

  # AWS SQS SDK

  ## Initialising the SQS client

  ```ts
  import { SQSClient } from "@aws-sdk/client-sqs";

  const sqs = new SQSClient({ region: "us-east-1" });
  ```

  ## Sending a message to the queue

  ```ts
  import { SendMessageCommand } from "@aws-sdk/client-sqs";

  const command = new SendMessageCommand({
    QueueUrl: queueUrl,
    MessageBody: messageBody,
    // Optional parameters
    DelaySeconds: 0,
    MessageAttributes: {
      CustomAttribute: {
        DataType: "String",
        StringValue: "CustomValue",
      },
    },
  });

  return await sqs.send(command);
  ```

  ## Processing messages on the queue

  ```ts
  import { ReceiveMessageCommand } from "@aws-sdk/client-sqs";

  const command = new ReceiveMessageCommand({
    QueueUrl: queueUrl,
    MaxNumberOfMessages: 10,
    WaitTimeSeconds: 20, // Long polling
    VisibilityTimeout: 30, // Time other consumers won't see this message
  });

  const response = await sqs.send(command);

  // Then do something

  for (const message of response.Messages || []) {
    console.log(message);
  }
  ```

  ## Removing message from queue

  ```ts
  import { DeleteMessageCommand } from "@aws-sdk/client-sqs";

  for (const message of response.Messages || []) {
    await sqs.send(
      new DeleteMessageCommand({
        QueueUrl: queueUrl,
        ReceiptHandle: message.ReceiptHandle,
      }),
    );
  }
  ```

  ## Query a DLQ

  ```ts
  const checkDLQ = async (dlqUrl: string) => {
    const command = new GetQueueAttributesCommand({
      QueueUrl: dlqUrl,
      AttributeNames: ["ApproximateNumberOfMessages"],
    });

    const response = await sqs.send(command);
    return response.Attributes?.ApproximateNumberOfMessages;
  };
  ```


path: AWS_SNS.md
sortable_path: AWS_SNS.md
title: AWS SNS
checksum: f12c6513eaba9963248094d422b8c2e20f6c393909a60ba34a4a7721b52e9f7c
created: 2024-11-25 06:50:32.993088026+00:00
modified: 2024-11-25 07:19:10.056124501+00:00
metadata: {"created":"Monday, November 25, 2024","tags":["AWS","SNS","SQS"]}
lead:
  The Simple Notification Service is another form of messaging service in AWS,
  similar to [SQS](AWS_SQS.md). The key difference is that SNS pushes messages out
  to subscribers whereas SQS stores the messages until they are processed.

body:
  The Simple Notification Service is another form of messaging service in AWS,
  similar to [SQS](AWS_SQS.md). The key difference is that SNS pushes messages out
  to subscribers whereas SQS stores the messages until they are processed.

  Before a message can be sent, and SNS topic must exist. One service such as a
  lambda will _publish_ to this topic. Another service, will _subscribe_ to the
  topic and be alerted when a message is sent to it.

  With SNS, one message can be sent to multiple subscribers simultaneously. In
  addition there is message filtering at the subscription level so subscribers can
  choose to receive only the messages they are interested in.

  Many other AWS services can act as SNS subscribers. Some common examples:

  - SQS queues
  - Lambda functions (trigger on receipt of a message)
  - HTTP endpoints
  - Email/SMS/Push notifications to mobile apps

  ## Combining with SQS

  Utilising SNS with SQS is a common pattern in serverless applications.

  We use SNS to decouple publishers from subscribers and then use SQS to handle
  the different subscriber tasks. This is known as the "Fanout-Queue" pattern.

  ![SQS fanout pattern](../img/sns-example.png)

  For example, an order is placed which results in a publication event to an SNS
  topic.

  Multiple systems subscribe to this topic, e.g. an inventory system, a shipping
  system and an analytics system.

  Each system has its own queue for processing the message.

raw_content:
  ---
  tags: [AWS, SNS, SQS]
  created: Monday, November 25, 2024
  ---

  # AWS SNS

  The Simple Notification Service is another form of messaging service in AWS,
  similar to [SQS](AWS_SQS.md). The key difference is that SNS pushes messages out
  to subscribers whereas SQS stores the messages until they are processed.

  Before a message can be sent, and SNS topic must exist. One service such as a
  lambda will _publish_ to this topic. Another service, will _subscribe_ to the
  topic and be alerted when a message is sent to it.

  With SNS, one message can be sent to multiple subscribers simultaneously. In
  addition there is message filtering at the subscription level so subscribers can
  choose to receive only the messages they are interested in.

  Many other AWS services can act as SNS subscribers. Some common examples:

  - SQS queues
  - Lambda functions (trigger on receipt of a message)
  - HTTP endpoints
  - Email/SMS/Push notifications to mobile apps

  ## Combining with SQS

  Utilising SNS with SQS is a common pattern in serverless applications.

  We use SNS to decouple publishers from subscribers and then use SQS to handle
  the different subscriber tasks. This is known as the "Fanout-Queue" pattern.

  ![SQS fanout pattern](../img/sns-example.png)

  For example, an order is placed which results in a publication event to an SNS
  topic.

  Multiple systems subscribe to this topic, e.g. an inventory system, a shipping
  system and an analytics system.

  Each system has its own queue for processing the message.


path: a247fa9b_surveillance_capitalism_not_necessary.md
sortable_path: a247fa9b_surveillance_capitalism_not_necessary.md
title: a247fa9b_surveillance_capitalism_not_necessary
checksum: 6982996ba7fd461a4139427f331faafde8db2ffe30532126b850a052b8f777d5
created: 2024-12-01 19:09:52.691742335+00:00
modified: 2024-12-01 19:34:56.319205514+00:00
metadata: {"created":"Sunday, December 01, 2024","tags":["ecopolsoc","internet","surveillance-capitalism"]}
lead:
  _Surveillance capitalism is neither a necessary development of information
  capitalism nor a necessary product of digital technology or the internet. It is
  a specifically constructed human choice, an unprecedented market form, an
  original solution to an emergency, and the underlying mechanism through which a
  new asset class is created on the cheap and converted to revenue._

body:
  _Surveillance capitalism is neither a necessary development of information
  capitalism nor a necessary product of digital technology or the internet. It is
  a specifically constructed human choice, an unprecedented market form, an
  original solution to an emergency, and the underlying mechanism through which a
  new asset class is created on the cheap and converted to revenue._

  _The Age of Surveillance Capitalism_ (2019): 91

raw_content:
  ---
  tags: [ecopolsoc, internet, surveillance-capitalism]
  created: Sunday, December 01, 2024
  ---

  # a247fa9b_surveillance_capitalism_not_necessary

  _Surveillance capitalism is neither a necessary development of information
  capitalism nor a necessary product of digital technology or the internet. It is
  a specifically constructed human choice, an unprecedented market form, an
  original solution to an emergency, and the underlying mechanism through which a
  new asset class is created on the cheap and converted to revenue._

  _The Age of Surveillance Capitalism_ (2019): 91


path: fbbfbc32-political-accommodation-inveigelment-surveillance-capitalism.md
sortable_path: fbbfbc32-political-accommodation-inveigelment-surveillance-capitalism.md
title: fbbfbc32-political-accommodation-inveigelment-surveillance-capitalism
checksum: 68352d1a57dcb358ab8a805550c10d8022f50445c03e634f6c38c7cc423fcff6
created: 2024-12-01 19:15:54.90583713+00:00
modified: 2025-01-09 19:03:40.155661978+00:00
metadata: {"created":"Sunday, December 01, 2024","tags":["ecopolsoc","internet","surveillance-capitalism"]}
lead:
  The reason why surveillance capitalism was able to take root when it did was not
  merely down to technological forces. Google, as the originator of the form, was
  able to exploit the political currents of the time to further innoculate itself
  against scrutiny and oversight.

body:
  The reason why surveillance capitalism was able to take root when it did was not
  merely down to technological forces. Google, as the originator of the form, was
  able to exploit the political currents of the time to further innoculate itself
  against scrutiny and oversight.

  The discovery of behavioural surplus coincided with September 11 and the
  subsequent "global war on terror". The quest for domestic "security" led to the
  sweeping erosion of previous generations' privacy rights.

  Prior to 911, the FCC had been arranging a series of legal challenges to Google
  on the question of privacy. This went out the window in the post-911 era where
  Google's unique ability to track and surveil online behaviour became a coveted
  commodity for the intelligence apparatus. Google began working closely with both
  the CIA and the NSA, with the CIA establishing an office and forward base in
  Silicon Valley.

  There was an deep interdependency between the Pentagon, intelligence agencies
  and Google:

  _The US intelligence community's incubation of Google from inception occurred
  through a combination of direct sponsorship and informal networks of financial
  influence, themselves closely aligned with Pentagon interests._

  _The Age of Surveillance Capitalism_ (2019): 118

  Another key aspect was cosying up to politicians during the electoral cycle.
  Most notably during Obama's elections:

  _Google demonstrated that the same predictive knowledge derived from behavioural
  surplus that had made the surveillance capitalists wealthy could also help
  candidates win elections_

  _Ibid_: 122

  Eric Schmidt took an active role in making Google's dataset and technologies
  available to the Obama campaign in both of his elections in order to target
  voters more accurately. He then was swiftly appointed to advisory roles within
  the administration throughout both terms.

raw_content:
  ---
  tags: [ecopolsoc, internet, surveillance-capitalism]
  created: Sunday, December 01, 2024
  ---

  # fbbfbc32-political-accommodation-inveigelment-surveillance-capitalism

  The reason why surveillance capitalism was able to take root when it did was not
  merely down to technological forces. Google, as the originator of the form, was
  able to exploit the political currents of the time to further innoculate itself
  against scrutiny and oversight.

  The discovery of behavioural surplus coincided with September 11 and the
  subsequent "global war on terror". The quest for domestic "security" led to the
  sweeping erosion of previous generations' privacy rights.

  Prior to 911, the FCC had been arranging a series of legal challenges to Google
  on the question of privacy. This went out the window in the post-911 era where
  Google's unique ability to track and surveil online behaviour became a coveted
  commodity for the intelligence apparatus. Google began working closely with both
  the CIA and the NSA, with the CIA establishing an office and forward base in
  Silicon Valley.

  There was an deep interdependency between the Pentagon, intelligence agencies
  and Google:

  _The US intelligence community's incubation of Google from inception occurred
  through a combination of direct sponsorship and informal networks of financial
  influence, themselves closely aligned with Pentagon interests._

  _The Age of Surveillance Capitalism_ (2019): 118

  Another key aspect was cosying up to politicians during the electoral cycle.
  Most notably during Obama's elections:

  _Google demonstrated that the same predictive knowledge derived from behavioural
  surplus that had made the surveillance capitalists wealthy could also help
  candidates win elections_

  _Ibid_: 122

  Eric Schmidt took an active role in making Google's dataset and technologies
  available to the Obama campaign in both of his elections in order to target
  voters more accurately. He then was swiftly appointed to advisory roles within
  the administration throughout both terms.


path: Architecture_of_the_World_Wide_Web.md
sortable_path: Architecture_of_the_World_Wide_Web.md
title: Architecture of the World Wide Web
lead: ## Terms
checksum: 728a7f6d45ae4e59189ac0cb9ee63a5ee7909701f53784f6b2366793efcf8ccc
created: 2024-12-13 13:06:19.172815289+00:00
modified: 2024-12-13 15:16:12.762244333+00:00
metadata: {"created":"Friday, December 13, 2024","tags":["world-wide-web","internet"]}
body:
  ## Terms

  <dl>
  <dt>web</dt>
  <dd>a set of resources delivered over the Internet using the HTTP protocol</dd>

  <dt>web resource</dt>
  <dd>any information that can be accessed using the web (document, image, video)</dd>

  <dt>web server</dt>
  <dd>a software program on a computer that hosts web resources</dd>

  <dt>web browser</dt>
  <dd>an application used to acccess web resources on web servers</dd>

  </dl>

  ## The web is _distributed_, _addressable_, and _linked_

  ### Distributed

  There is no centralised organisation or system that decides the content that can
  be published. Any computer connected to the Internet can run a web server and
  the person running it can make available any content they wish.

  ### _Addressable_

  Every resource on the web has a unique address that includes both its location
  and how to access it (URL):

  ![](../img/web-url.png)

  - scheme
    - specifies the [Application Layer](Application_Layer_of_Internet_Protocol.md for
      accessing the resource. In the example this is HTTP but it could be FTP or another
      protocol, always bracketed with a colon
  - authority
    - comprises the [DNS](Transport_Layer_of_Internet_Protocol.md) hostname of the
      server containing the resource. This could also be an
      [IP address](IP_addresses.md) and could also include a port number
  - path
    - the location of the resource on the web server, organised hierarchically and
      analagous to a file path
  - query
    - a modifier of the resource returned to the client. Its usage varies accross
      sites but is typically deterimined by backend implementation details

  > Note a URL does not have to always contain _all_ this information. Consider an
  > index page.

raw_content:
  ---
  tags: [world-wide-web, internet]
  created: Friday, December 13, 2024
  ---

  # Architecture of the World Wide Web

  ## Terms

  <dl>
  <dt>web</dt>
  <dd>a set of resources delivered over the Internet using the HTTP protocol</dd>

  <dt>web resource</dt>
  <dd>any information that can be accessed using the web (document, image, video)</dd>

  <dt>web server</dt>
  <dd>a software program on a computer that hosts web resources</dd>

  <dt>web browser</dt>
  <dd>an application used to acccess web resources on web servers</dd>

  </dl>

  ## The web is _distributed_, _addressable_, and _linked_

  ### Distributed

  There is no centralised organisation or system that decides the content that can
  be published. Any computer connected to the Internet can run a web server and
  the person running it can make available any content they wish.

  ### _Addressable_

  Every resource on the web has a unique address that includes both its location
  and how to access it (URL):

  ![](../img/web-url.png)

  - scheme
    - specifies the [Application Layer](Application_Layer_of_Internet_Protocol.md for
      accessing the resource. In the example this is HTTP but it could be FTP or another
      protocol, always bracketed with a colon
  - authority
    - comprises the [DNS](Transport_Layer_of_Internet_Protocol.md) hostname of the
      server containing the resource. This could also be an
      [IP address](IP_addresses.md) and could also include a port number
  - path
    - the location of the resource on the web server, organised hierarchically and
      analagous to a file path
  - query
    - a modifier of the resource returned to the client. Its usage varies accross
      sites but is typically deterimined by backend implementation details

  > Note a URL does not have to always contain _all_ this information. Consider an
  > index page.


path: HTTPS.md
sortable_path: HTTPS.md
title: HTTPS
checksum: 03b0118a6d3a1324b31f6ad140a0035d47d0b0e8f53ded9577ba806a7b982152
created: 2024-12-13 13:23:23.159398701+00:00
modified: 2025-02-27 18:27:38.874430064+00:00
lead:
  The problem with standard, unencrypted HTTP requests is that the data can be
  captured in transit and be observed or modified by malicious actors.

body:
  The problem with standard, unencrypted HTTP requests is that the data can be
  captured in transit and be observed or modified by malicious actors.

  HTTPS is the solution. It encrypts the entirety of the HTTP request and response
  (URL, headers, body).

  It utilises two types of encryption to send messages securely:

  - symmetric encryption
    - a single shared key between client and server to encrypt the messages sent
      between them
  - asymmetric encryption
    - two keys are used: a [public key](./bbdcb54f_public_key_cryptography.md) to
      encrypt the data and a private key to decrypt it. The public key can be
      shared freely so anyone can encrypt and send data to a peer but only the
      peer can receive and decrypt it

  > Symmetric encryption applies once the server has been authenticated and its
  > public key has been shared with the client. Asymmetric encryption applies at
  > the initiation phase only.

  The encryption protocol used is TLS - the **Transport Layer Security Protocol**.
  Previously SSL (Secure Sockets Layer) was used but has been deprecated over
  security concerns.

  The initial handshake process takes place entirely within the Network Layer,
  over TCP. Only once encryption is secured is HTTPS activated at the
  [Application Layer](./Application_Layer_of_Internet_Protocol.md)

  ## How communication over HTTPs works

  ![Server handshake diagram](../img/https-handshake.png)

  - An initial handshake is shared between client and server, agreeing that
    communication will be encrypted. This is public. "Client hello" and "Server
    hello". Some random numbers are exchanged at this point
  - The server sends a digital [certificate](./Certificate_authorities.md) key The
    client checks that the certificate is valid
  - If valid, the client encrypts a string of bytes using the public key and sends
    it to the server. This is called the "premaster secret"
  - This is decrypted by the server
  - The premaster secret, along with the random bytes from the handshake is then
    used to compute a shared secret key (symmetric) which is used to encrypt all
    subsequent messages for the duration of the session

  ## Caching

  Once symmetric exchange is achieved the TLS handshake is complete and lasts
  typically until either terminates it.

  However, some caching does occur on both the server and browser side. Both use
  session IDs to enact an abbreviated handshake such as when the session cache
  expires. This way, certificate validation and the asymmetric exchange can be
  skipped. These tickets are shared as part of the original exchange and the
  client can present them to the server as proof they already authenticated when
  they come back later.

raw_content:
  ---
  tags: [world-wide-web, internet, encryption, server-management]
  created: Friday, December 14, 2024
  ---

  # HTTPS

  The problem with standard, unencrypted HTTP requests is that the data can be
  captured in transit and be observed or modified by malicious actors.

  HTTPS is the solution. It encrypts the entirety of the HTTP request and response
  (URL, headers, body).

  It utilises two types of encryption to send messages securely:

  - symmetric encryption
    - a single shared key between client and server to encrypt the messages sent
      between them
  - asymmetric encryption
    - two keys are used: a [public key](./bbdcb54f_public_key_cryptography.md) to
      encrypt the data and a private key to decrypt it. The public key can be
      shared freely so anyone can encrypt and send data to a peer but only the
      peer can receive and decrypt it

  > Symmetric encryption applies once the server has been authenticated and its
  > public key has been shared with the client. Asymmetric encryption applies at
  > the initiation phase only.

  The encryption protocol used is TLS - the **Transport Layer Security Protocol**.
  Previously SSL (Secure Sockets Layer) was used but has been deprecated over
  security concerns.

  The initial handshake process takes place entirely within the Network Layer,
  over TCP. Only once encryption is secured is HTTPS activated at the
  [Application Layer](./Application_Layer_of_Internet_Protocol.md)

  ## How communication over HTTPs works

  ![Server handshake diagram](../img/https-handshake.png)

  - An initial handshake is shared between client and server, agreeing that
    communication will be encrypted. This is public. "Client hello" and "Server
    hello". Some random numbers are exchanged at this point
  - The server sends a digital [certificate](./Certificate_authorities.md) key The
    client checks that the certificate is valid
  - If valid, the client encrypts a string of bytes using the public key and sends
    it to the server. This is called the "premaster secret"
  - This is decrypted by the server
  - The premaster secret, along with the random bytes from the handshake is then
    used to compute a shared secret key (symmetric) which is used to encrypt all
    subsequent messages for the duration of the session

  ## Caching

  Once symmetric exchange is achieved the TLS handshake is complete and lasts
  typically until either terminates it.

  However, some caching does occur on both the server and browser side. Both use
  session IDs to enact an abbreviated handshake such as when the session cache
  expires. This way, certificate validation and the asymmetric exchange can be
  skipped. These tickets are shared as part of the original exchange and the
  client can present them to the server as proof they already authenticated when
  they come back later.

metadata:
  {"created":"Friday, December 14, 2024","tags":["world-wide-web","internet","encryption","server-management"]}


path: What_can_the_ISP_see.md
sortable_path: What_can_the_ISP_see.md
title: What can the ISP see?
lead: ## Plain HTTP
checksum: 992a0ae00256f2ee6644480b697656300e6b8b22ecb34c4a05dae060081b24f2
created: 2024-12-13 14:14:32.050396895+00:00
modified: 2025-03-26 18:39:10.552530261+00:00
metadata: {"created":"Friday, December 13, 2024","tags":["internet","encryption","privacy","Tor"]}
body:
  ## Plain HTTP

  The ISP can see all content exchanged: DNS lookups, IP address of the server you
  connect to, requests and body content.

  ## Just HTTPS

  With HTTPS alone, the content of the messages is encrypted but the ISP can see:

  - DNS queries
  - IP addresses you connect to
  - Traffic patterns and metadata (when, how often, amount of data exchanged etc)

  ## HTTPS and encrypted DNS server

  With HTTPS and an encrypted DNS server (e.g. Quad9) the DNS queries are hidden
  but the IP addresses are not. So they could still derive your DNS lookups from
  the IPs you end up connecting to.

  ## HTTPS and VPN

  The ISP only sees the encrypted traffic to the VPN server. Your real IP and
  destination IP is hidden.

  However the VPN provider can potentially see DNS queries, depending on their
  policy. Although a good VPN provider will encrypt DNS lookups like an encrypted
  DNS server.

  The VPN can also still see what IPs you are connecting to. You are basically
  swiching trust from the ISP to the VPN provider. However reputable VPNs do not
  have your billing and identity info (and payment can use cryptocurrencies) and
  don't keep logs. Many also have policies about refusal to cooporate with
  authorities in different juristictions. Also the VPN doesn't know your real IP.

  ## HTTPS, VPN, and encrypted DNS

  DNS queries hidded from both ISP and VPN. Traffic is fully encrypted and routed
  through VPN.

  ## Fingerprinting

  Note that none of the above stops fingerprinting. Even with full encryption
  there are methods of determining online behaviour through encrypted packet
  analysis and usage patterns.

  A way round this is to use SOCKS proxies which add hops or to use TOR. TOR
  effectively solves the "have to trust VPN provider" problem.

  With TOR:

  - Traffic routed through 3+ nodes
  - Each node only knows previous and next hop
  - No single node knows both source and destination. Even Tor nodes can't see the
    full path

  It also uses:

  - Fixed packet sizes
  - Timing obfuscation
  - Traffic padding

  Which makes pattern analysis and fingerprinting much harder than just a VPN.

  Furthermore there is also the following protection from VPN weaknesses:

  - No single provider to trust/compromise
  - Decentralized network
  - Free to use and no account needed
  - No permanent exit node like with VPN

raw_content:
  ---
  tags: [internet, encryption, privacy, Tor]
  created: Friday, December 13, 2024
  ---

  # What can the ISP see?

  ## Plain HTTP

  The ISP can see all content exchanged: DNS lookups, IP address of the server you
  connect to, requests and body content.

  ## Just HTTPS

  With HTTPS alone, the content of the messages is encrypted but the ISP can see:

  - DNS queries
  - IP addresses you connect to
  - Traffic patterns and metadata (when, how often, amount of data exchanged etc)

  ## HTTPS and encrypted DNS server

  With HTTPS and an encrypted DNS server (e.g. Quad9) the DNS queries are hidden
  but the IP addresses are not. So they could still derive your DNS lookups from
  the IPs you end up connecting to.

  ## HTTPS and VPN

  The ISP only sees the encrypted traffic to the VPN server. Your real IP and
  destination IP is hidden.

  However the VPN provider can potentially see DNS queries, depending on their
  policy. Although a good VPN provider will encrypt DNS lookups like an encrypted
  DNS server.

  The VPN can also still see what IPs you are connecting to. You are basically
  swiching trust from the ISP to the VPN provider. However reputable VPNs do not
  have your billing and identity info (and payment can use cryptocurrencies) and
  don't keep logs. Many also have policies about refusal to cooporate with
  authorities in different juristictions. Also the VPN doesn't know your real IP.

  ## HTTPS, VPN, and encrypted DNS

  DNS queries hidded from both ISP and VPN. Traffic is fully encrypted and routed
  through VPN.

  ## Fingerprinting

  Note that none of the above stops fingerprinting. Even with full encryption
  there are methods of determining online behaviour through encrypted packet
  analysis and usage patterns.

  A way round this is to use SOCKS proxies which add hops or to use TOR. TOR
  effectively solves the "have to trust VPN provider" problem.

  With TOR:

  - Traffic routed through 3+ nodes
  - Each node only knows previous and next hop
  - No single node knows both source and destination. Even Tor nodes can't see the
    full path

  It also uses:

  - Fixed packet sizes
  - Timing obfuscation
  - Traffic padding

  Which makes pattern analysis and fingerprinting much harder than just a VPN.

  Furthermore there is also the following protection from VPN weaknesses:

  - No single provider to trust/compromise
  - Decentralized network
  - Free to use and no account needed
  - No permanent exit node like with VPN


path: Network_packet_analysis_tools.md
sortable_path: Network_packet_analysis_tools.md
title: Network packet analysis tools
checksum: 6bffbdc731d985354637c3e58cefa82c0fce3db0c5514e06c091274a960afb4c
created: 2024-12-13 14:39:26.730227868+00:00
modified: 2024-12-13 14:41:45.257768788+00:00
metadata: {"created":"Friday, December 13, 2024","tags":["networks","procedural"]}
lead:
  - `tcpdump`: View raw packet data (`tcpdump -i wlan0 -XX` for full packet
    contents)
  - tshark: Wireshark's CLI for detailed packet analysis (`tshark -i wlan0 -V`)
  - `nethogs`: Monitor per-application network usage
  - `iftop`: Show bandwidth usage per connection

body:
  - `tcpdump`: View raw packet data (`tcpdump -i wlan0 -XX` for full packet
    contents)
  - tshark: Wireshark's CLI for detailed packet analysis (`tshark -i wlan0 -V`)
  - `nethogs`: Monitor per-application network usage
  - `iftop`: Show bandwidth usage per connection

raw_content:
  ---
  tags: [networks, procedural]
  created: Friday, December 13, 2024
  ---

  # Network packet analysis tools

  - `tcpdump`: View raw packet data (`tcpdump -i wlan0 -XX` for full packet
    contents)
  - tshark: Wireshark's CLI for detailed packet analysis (`tshark -i wlan0 -V`)
  - `nethogs`: Monitor per-application network usage
  - `iftop`: Show bandwidth usage per connection


path: User_agent.md
sortable_path: User_agent.md
title: User agent
checksum: 8ebaa79e8318ce51042596b2b34faeec781367002c93c8f5a8007a60e6c8a553
created: 2024-12-20 11:01:51.175194304+00:00
modified: 2024-12-20 11:05:59.817461091+00:00
metadata: {"created":"Friday, December 20, 2024","tags":["world-wide-web"]}
lead:
  _User agent_ is the technical term for a web browser. So named because it acts
  on behalf of the user.

body:
  _User agent_ is the technical term for a web browser. So named because it acts
  on behalf of the user.

  When a browser makes a request to a web server it includes a header value called
  `User-Agent` that describes the browser.

raw_content:
  ---
  tags: [world-wide-web]
  created: Friday, December 20, 2024
  ---

  # User agent

  _User agent_ is the technical term for a web browser. So named because it acts
  on behalf of the user.

  When a browser makes a request to a web server it includes a header value called
  `User-Agent` that describes the browser.


path: Dynamic_and_static_websites.md
sortable_path: Dynamic_and_static_websites.md
title: Dynamic and static websites
checksum: d823c51bb34e1fb4ebaf63240ab4f4196c1587b83e90ef6b71a80ce28ea9cdf1
created: 2024-12-20 11:06:21.197224512+00:00
modified: 2024-12-20 11:32:47.747935335+00:00
metadata: {"created":"Friday, December 20, 2024","tags":["world-wide-web"]}
lead:
  When we talk about dynamic and static websites we are talking about the role of
  the server, not anything to do with how the website behaves for the user.

body:
  When we talk about dynamic and static websites we are talking about the role of
  the server, not anything to do with how the website behaves for the user.

  A static site's content comes unchanged from the files on the server whereas a
  dynamic site's content is generated on the server.

  With a static website, the content of the site has been build before a request
  for it is made. It's HTML, CSS and JS are stored on the server and are simply
  returned to the client in this form when they are requested.

  When content is requested on a dynamic website the HTML is generated on-the-fly
  on the server-side and returned to the client. Typically a request will be made
  for an item of content, the data will be fetched on the backend from a DB and
  then formatted as HTML and sent back to the client.

  Static websites can be more performant because of the lack of BE processing
  however it means that any runtime processing or special rendering must be
  handled by the frontend JavaScript, without relying on server languages.

  Take the sourcing of remote data. With a dynamic site this can be handled in a
  single request because the backend can fetch the data and return to the client
  with the rendering applied. On the frontend, JS would have to make a request to
  an endpoint that calls a backend service that returns the data. Having received
  the raw data it would then need to render it.

  Nowadays there is generally a mixed approach. This distinction was more
  important in the early days of the web.

raw_content:
  ---
  tags: [world-wide-web]
  created: Friday, December 20, 2024
  ---

  # Dynamic and static websites

  When we talk about dynamic and static websites we are talking about the role of
  the server, not anything to do with how the website behaves for the user.

  A static site's content comes unchanged from the files on the server whereas a
  dynamic site's content is generated on the server.

  With a static website, the content of the site has been build before a request
  for it is made. It's HTML, CSS and JS are stored on the server and are simply
  returned to the client in this form when they are requested.

  When content is requested on a dynamic website the HTML is generated on-the-fly
  on the server-side and returned to the client. Typically a request will be made
  for an item of content, the data will be fetched on the backend from a DB and
  then formatted as HTML and sent back to the client.

  Static websites can be more performant because of the lack of BE processing
  however it means that any runtime processing or special rendering must be
  handled by the frontend JavaScript, without relying on server languages.

  Take the sourcing of remote data. With a dynamic site this can be handled in a
  single request because the backend can fetch the data and return to the client
  with the rendering applied. On the frontend, JS would have to make a request to
  an endpoint that calls a backend service that returns the data. Having received
  the raw data it would then need to render it.

  Nowadays there is generally a mixed approach. This distinction was more
  important in the early days of the web.


path: bbdcb54f_public_key_cryptography.md
sortable_path: bbdcb54f_public_key_cryptography.md
title: bbdcb54f_public_key_cryptography
checksum: c03f7d29fb3b86739ddab99d4d1e7c9f10bab31461cc9b6c3fe2f90f27c281fb
created: 2024-12-20 11:49:13.122588284+00:00
modified: 2024-12-20 13:12:22.104945119+00:00
metadata: {"created":"Friday, December 20, 2024","tags":["encryption","cryptography"]}
lead:
  _A_ wants to send a message to _B_. A wants to ensure that only _B_ can read the
  message and that it is not intercepted or altered during transmission to _B_.

body:
  _A_ wants to send a message to _B_. A wants to ensure that only _B_ can read the
  message and that it is not intercepted or altered during transmission to _B_.

  Think of _B_ as being in possession of a letterbox. The letterbox contains
  messages that only he can read. Anyone can put letters in it for _B_ to read but
  _B_ alone can read them.

  This is achieved by encyrpting the messages with two keys: a public key and a
  private key. The public key can be shared with others and anyone in possession
  of the public key can encrypt a message for _B_. Once encrypted, only _B_ can
  decrypt it using his private key. Once receiving the message, B uses his private
  secret key - that he doesn't share with anyone else - to decrypt. This is the
  equivalent to opening up the letterbox and taking out the letters.

  Another useful analogy is to think of invisible ink. Anyone can write a letter
  using the invisible ink but only the recipient has the liquid that can render it
  visible again. Anyone intercepting the message (without the liquid) will see
  nothing, equivalent to the encrypted data.

  This is asymmetric encryption because the two communicating parties do not have
  equal access to the messages exchanged. _A_ can only send messages - he can't
  decrypt them, only _B_ can. So _A_ is unequal relative to _B_. With symmetric
  encryption _A_ and _B_ would both share a single private key which they would
  each use to encrypt and decrypt their messages.

raw_content:
  ---
  tags: [encryption, cryptography]
  created: Friday, December 20, 2024
  ---

  # bbdcb54f_public_key_cryptography

  _A_ wants to send a message to _B_. A wants to ensure that only _B_ can read the
  message and that it is not intercepted or altered during transmission to _B_.

  Think of _B_ as being in possession of a letterbox. The letterbox contains
  messages that only he can read. Anyone can put letters in it for _B_ to read but
  _B_ alone can read them.

  This is achieved by encyrpting the messages with two keys: a public key and a
  private key. The public key can be shared with others and anyone in possession
  of the public key can encrypt a message for _B_. Once encrypted, only _B_ can
  decrypt it using his private key. Once receiving the message, B uses his private
  secret key - that he doesn't share with anyone else - to decrypt. This is the
  equivalent to opening up the letterbox and taking out the letters.

  Another useful analogy is to think of invisible ink. Anyone can write a letter
  using the invisible ink but only the recipient has the liquid that can render it
  visible again. Anyone intercepting the message (without the liquid) will see
  nothing, equivalent to the encrypted data.

  This is asymmetric encryption because the two communicating parties do not have
  equal access to the messages exchanged. _A_ can only send messages - he can't
  decrypt them, only _B_ can. So _A_ is unequal relative to _B_. With symmetric
  encryption _A_ and _B_ would both share a single private key which they would
  each use to encrypt and decrypt their messages.


path: a4601796_trapdoor_functions.md
sortable_path: a4601796_trapdoor_functions.md
title: a4601796_trapdoor_functions
checksum: 663700b03beee5d3d47e4d43afabc07b52f25db9a0844c1b033528bed50eced8
created: 2024-12-20 12:03:04.183937845+00:00
modified: 2024-12-20 13:12:24.194980394+00:00
metadata: {"created":"Friday, December 20, 2024","tags":["cryptography","encryption"]}
lead:
  How does [public key cryptography](./bbdcb54f_public_key_cryptography.md)
  achieve the feat of making the encyrpted data only decipherable in one
  direction. In other words how come the same data can be encrypted with one key
  but only decryptable by another?

body:
  How does [public key cryptography](./bbdcb54f_public_key_cryptography.md)
  achieve the feat of making the encyrpted data only decipherable in one
  direction. In other words how come the same data can be encrypted with one key
  but only decryptable by another?

  It is acheived by the use of **trapdoor functions**. These are functions that
  are easy to compute in one direction but extremely difficult to reverse. But
  they also contain trapdoors which, if known, make the functions trivially easy
  to compute in both directions.

  The public key holder can compute the encryption but only the private key holder
  has the trapdoor that makes reversing the encryption trivial.

  As an example we can look at a common method for generating trapdoor functions
  using prime numbers.

  First we need to convert the message into numerical form. One way would be to
  convert the characters to their ASCII numbers. These numbers then become input
  into the mathematical encryption function.

  Next we would pick two very large prime numbers. Using RSA-like encryption they
  would be need to be 2048+ bits in length. For the example we will use 17 and 23.

  For simplicity let's say that our message is one character equivalent to `5` in
  ASCII. We multiply our two primes to get 391 (17 x 23 = 391) and then find the
  remainder of: our message cubed and divided by the prime sum. This is `125`
  (`5^3 / 391`) which is now our encrypted message which we send.

  Unless you know the two primes which were used, it is extremely difficult and
  time consuming to reverse the calculation and find the factors that would get
  you back to the starting point (remember the numbers will be much larger in
  reality). The owner of the private key knows these primes however, so can do
  this.

raw_content:
  ---
  tags: [cryptography, encryption]
  created: Friday, December 20, 2024
  ---

  # a4601796_trapdoor_functions

  How does [public key cryptography](./bbdcb54f_public_key_cryptography.md)
  achieve the feat of making the encyrpted data only decipherable in one
  direction. In other words how come the same data can be encrypted with one key
  but only decryptable by another?

  It is acheived by the use of **trapdoor functions**. These are functions that
  are easy to compute in one direction but extremely difficult to reverse. But
  they also contain trapdoors which, if known, make the functions trivially easy
  to compute in both directions.

  The public key holder can compute the encryption but only the private key holder
  has the trapdoor that makes reversing the encryption trivial.

  As an example we can look at a common method for generating trapdoor functions
  using prime numbers.

  First we need to convert the message into numerical form. One way would be to
  convert the characters to their ASCII numbers. These numbers then become input
  into the mathematical encryption function.

  Next we would pick two very large prime numbers. Using RSA-like encryption they
  would be need to be 2048+ bits in length. For the example we will use 17 and 23.

  For simplicity let's say that our message is one character equivalent to `5` in
  ASCII. We multiply our two primes to get 391 (17 x 23 = 391) and then find the
  remainder of: our message cubed and divided by the prime sum. This is `125`
  (`5^3 / 391`) which is now our encrypted message which we send.

  Unless you know the two primes which were used, it is extremely difficult and
  time consuming to reverse the calculation and find the factors that would get
  you back to the starting point (remember the numbers will be much larger in
  reality). The owner of the private key knows these primes however, so can do
  this.


path: Binary_encoding_of_sound.md
sortable_path: Binary_encoding_of_sound.md
title: Binary encoding of sound
lead: Sound is a change in air pressure.
checksum: 49f4a2ec25f9431877890e0f6edfb7b9725f9b51382071c985b518d05793e144
created: 2024-12-27 12:29:02.492923417+00:00
modified: 2025-01-02 17:22:27.562253496+00:00
metadata: {"created":"Friday, December 27, 2024","tags":["sound","binary","analogue"]}
body:
  Sound is a change in air pressure.

  To record sound, we convert the change in air pressure into an electrical signal
  by means of a microphone. Effectively, the same natural waveform is converted
  from one medium (the vibration of air molecules) into another medium (electrical
  charge in a field) whilst retaining the same morphology. See
  [Recording and playback of sound](Recording_and_playback_of_sound.md).

  The resulting electrical signal is analogue: it has a continuous and variable
  frequency and amplitude and thus can have any value at any given moment.

  To encode this information digitally we need to make the continuous analogue
  single discrete, so that it can represented with a finite set of symbols (bits).

  This process is known as **quantization** and consists in sampling the analogue
  waveform. This means taking a measurement of it at different points in time -
  each second.

  CDs sample the audio source at a rate of 44.1 kHz: 44, 100 measurements of the
  signal per second.

  The measurements are then stored in a digital format: bits. The amount of bits
  used to record the measurement is the **bit-depth**. Each measurement is rounded
  to the nearest available digital value. CDs have a depth of 16 bits. Thus each
  measurement can be one of $16^2 = 65,536$ possible values.

  With sampling at a given bit-depth established, the **bit-rate** can be derived.

  This is the sampling rate (number of measurements per second) divided by the
  number of bits per sample (bit-depth), multiplied by the number of channels.

  Thus for stereo CDs this is:

  $$
  (44100 / 16 ) \cdot 2 = 1411200
  $$

  Or 1.4 Mb/s

  > Even with a high bit-rate (as with CDs) the process still necessarily involves
  > a loss of information when converting sound from analogue to digital. However,
  > given a high enough value for the sampling rate and bit-depth, the differences
  > are imperceptible to the human ear.

raw_content:
  ---
  tags: [sound, binary, analogue]
  created: Friday, December 27, 2024
  ---

  # Binary encoding of sound

  Sound is a change in air pressure.

  To record sound, we convert the change in air pressure into an electrical signal
  by means of a microphone. Effectively, the same natural waveform is converted
  from one medium (the vibration of air molecules) into another medium (electrical
  charge in a field) whilst retaining the same morphology. See
  [Recording and playback of sound](Recording_and_playback_of_sound.md).

  The resulting electrical signal is analogue: it has a continuous and variable
  frequency and amplitude and thus can have any value at any given moment.

  To encode this information digitally we need to make the continuous analogue
  single discrete, so that it can represented with a finite set of symbols (bits).

  This process is known as **quantization** and consists in sampling the analogue
  waveform. This means taking a measurement of it at different points in time -
  each second.

  CDs sample the audio source at a rate of 44.1 kHz: 44, 100 measurements of the
  signal per second.

  The measurements are then stored in a digital format: bits. The amount of bits
  used to record the measurement is the **bit-depth**. Each measurement is rounded
  to the nearest available digital value. CDs have a depth of 16 bits. Thus each
  measurement can be one of $16^2 = 65,536$ possible values.

  With sampling at a given bit-depth established, the **bit-rate** can be derived.

  This is the sampling rate (number of measurements per second) divided by the
  number of bits per sample (bit-depth), multiplied by the number of channels.

  Thus for stereo CDs this is:

  $$
  (44100 / 16 ) \cdot 2 = 1411200
  $$

  Or 1.4 Mb/s

  > Even with a high bit-rate (as with CDs) the process still necessarily involves
  > a loss of information when converting sound from analogue to digital. However,
  > given a high enough value for the sampling rate and bit-depth, the differences
  > are imperceptible to the human ear.


path: Audio_file_formats.md
sortable_path: Audio_file_formats.md
title: Audio file formats
lead: ## CD quality
checksum: 618daa1d68a637b54f26e3e16041039985606daa493d576509d141c4b5fd7c72
created: 2024-12-27 12:29:18.056498357+00:00
modified: 2025-01-02 17:31:38.478953252+00:00
metadata: {"created":"Friday, December 27, 2024","tags":["sound","binary"]}
body:
  ## CD quality

  We can use CD's as the digital gold standard as the best digital quality
  available (to consumers).

  CDs store audio in uncompressed PCM (Pulse Code Modulation) format.

  They have a sampling rate of 44.1kHz, which is done in two channels to enable
  stereo.

  ## What lossy formats entail

  A conversion to MP3 from, for example, a CD source is always a one-way process
  and is not reversible. Once information is discarded in the compression process,
  it cannot be retrieved.

  This is obviously in contrast to lossless methods like FLAC where the original
  CD audio can always be reconstructed.

  It follows from the above that if you repeatedly encode a CD source to MP3, it
  will deteriorate in quality since more data is being removed each time.

  ## Major audio formats

  ### WAV (Waveform Audio File Format)

  - CD-quality encoding with no compression
  - Bit-for-bit identical to the CD source
  - Historically developed for Windows machines but can play on all operating
    systems

  ### FLAC (Free Lossless Audio Codec)

  - Basically the same as WAV but in a (losslessly) compressed format
  - The difference between a novel in a text file (WAV) and as a zipped file

  ### MP3 (MPEG-1 Audio Layer MP3)

  - Lossy format.
  - When a WAV file (or other lossless format) is converted to MP3 a Fast Fourier
    Transform analysis is performed to determine the frequency of certain sounds
  - This is used by the encoder to decide which parts of the sound are
    imperceptible and thus which can be discarded to reduce the file size. This is
    done through the application of psycho-acoustic models.
  - The remaining data is then compressed
  - Examples of the data reduction applied:

    - Removing frequencies that humans cannot hear
    - Removing quieter sounds that are masked by louder sounds
    - Combining similar frequencies
    - Reducing stereo information where it is less noticeable

  ### OGG: Ogg Vorbis

  - An open-source alternative to MP3
  - Typically achieves better quality than MP3 at the same bit rate, especially at
    very high/ low frequencies
  - Also better stereo handling at low frequencies
  - Uses a more modern psycho-acoustic model

  ## Variable and constant bit rates

  For lossy formats like MP3, the amount of data that is being encoded from the
  uncompressed source is expressed via the unit of "bit rate": **how many
  thousands of bits are being used to represent each second of audio**.

  There are two methods of encoding that impact on the bit rate.

  With _variable bit rate_ encoding, the encoder dynamically adjusts the bit rate
  depending on what is happening in the music. During complex passages (e.g. a
  full orchestra) it uses a higher bit rate to capture the detail. During simpler
  passages (like a single instrument) it uses lower bit rates since less data is
  needed and during silence it can drop to very low bit rates.

  _Constant bit rate encoding_ uses the same bit rate throughout, it is
  consequently less efficient.

  When talking about the quality of MP3s there are generally two bit rates cited:

  - ~128kbps: acceptable but significantly reduced
  - 320kbps: the highest quality you can get whilst still using a lossy method
    like MP3

  With VBR, these are sometimes expressed as an average.

  Subjectively, A 128kbps MP3 might sound "underwater" or "swishy", while a
  320kbps version would preserve much more detail.

  Still, the bit-rate of a CD is 1411kbps! A 128kb/s MP3 is therefore only
  capturing about 9% of CD quality and a 320kbps MP3 is capturing about 23% of CD
  quality.

  ## Streaming services

  Spotify uses Ogg Vorbis throughout but uses different bit rates for its
  different tiers. The free tier has a range from 24-160kb/s at VBR with the
  option of 320kb/s on the premium tier.

  Other services offer FLAC or FLAC-equivalent quality at their most expensive
  tiers (Apple Music, Amazon Music, Tidal).

  Og Vorbis is particularly well-suited to streaming. It can seamlessly switch bit
  rates during the stream which is beneficial with changeable network conditions.
  Plus data is organised into independent packets so if a packet is lost there is
  no perceptible difference.

raw_content:
  ---
  tags: [sound, binary]
  created: Friday, December 27, 2024
  ---

  # Audio file formats

  ## CD quality

  We can use CD's as the digital gold standard as the best digital quality
  available (to consumers).

  CDs store audio in uncompressed PCM (Pulse Code Modulation) format.

  They have a sampling rate of 44.1kHz, which is done in two channels to enable
  stereo.

  ## What lossy formats entail

  A conversion to MP3 from, for example, a CD source is always a one-way process
  and is not reversible. Once information is discarded in the compression process,
  it cannot be retrieved.

  This is obviously in contrast to lossless methods like FLAC where the original
  CD audio can always be reconstructed.

  It follows from the above that if you repeatedly encode a CD source to MP3, it
  will deteriorate in quality since more data is being removed each time.

  ## Major audio formats

  ### WAV (Waveform Audio File Format)

  - CD-quality encoding with no compression
  - Bit-for-bit identical to the CD source
  - Historically developed for Windows machines but can play on all operating
    systems

  ### FLAC (Free Lossless Audio Codec)

  - Basically the same as WAV but in a (losslessly) compressed format
  - The difference between a novel in a text file (WAV) and as a zipped file

  ### MP3 (MPEG-1 Audio Layer MP3)

  - Lossy format.
  - When a WAV file (or other lossless format) is converted to MP3 a Fast Fourier
    Transform analysis is performed to determine the frequency of certain sounds
  - This is used by the encoder to decide which parts of the sound are
    imperceptible and thus which can be discarded to reduce the file size. This is
    done through the application of psycho-acoustic models.
  - The remaining data is then compressed
  - Examples of the data reduction applied:

    - Removing frequencies that humans cannot hear
    - Removing quieter sounds that are masked by louder sounds
    - Combining similar frequencies
    - Reducing stereo information where it is less noticeable

  ### OGG: Ogg Vorbis

  - An open-source alternative to MP3
  - Typically achieves better quality than MP3 at the same bit rate, especially at
    very high/ low frequencies
  - Also better stereo handling at low frequencies
  - Uses a more modern psycho-acoustic model

  ## Variable and constant bit rates

  For lossy formats like MP3, the amount of data that is being encoded from the
  uncompressed source is expressed via the unit of "bit rate": **how many
  thousands of bits are being used to represent each second of audio**.

  There are two methods of encoding that impact on the bit rate.

  With _variable bit rate_ encoding, the encoder dynamically adjusts the bit rate
  depending on what is happening in the music. During complex passages (e.g. a
  full orchestra) it uses a higher bit rate to capture the detail. During simpler
  passages (like a single instrument) it uses lower bit rates since less data is
  needed and during silence it can drop to very low bit rates.

  _Constant bit rate encoding_ uses the same bit rate throughout, it is
  consequently less efficient.

  When talking about the quality of MP3s there are generally two bit rates cited:

  - ~128kbps: acceptable but significantly reduced
  - 320kbps: the highest quality you can get whilst still using a lossy method
    like MP3

  With VBR, these are sometimes expressed as an average.

  Subjectively, A 128kbps MP3 might sound "underwater" or "swishy", while a
  320kbps version would preserve much more detail.

  Still, the bit-rate of a CD is 1411kbps! A 128kb/s MP3 is therefore only
  capturing about 9% of CD quality and a 320kbps MP3 is capturing about 23% of CD
  quality.

  ## Streaming services

  Spotify uses Ogg Vorbis throughout but uses different bit rates for its
  different tiers. The free tier has a range from 24-160kb/s at VBR with the
  option of 320kb/s on the premium tier.

  Other services offer FLAC or FLAC-equivalent quality at their most expensive
  tiers (Apple Music, Amazon Music, Tidal).

  Og Vorbis is particularly well-suited to streaming. It can seamlessly switch bit
  rates during the stream which is beneficial with changeable network conditions.
  Plus data is organised into independent packets so if a packet is lost there is
  no perceptible difference.


path: Recording_and_playback_of_sound.md
sortable_path: Recording_and_playback_of_sound.md
title: Recording and playback of sound
lead: ## Analogue
checksum: ba117d13d562a1cfffa839816844effd32d437afd558d0cb121f857c170ac5b1
created: 2025-01-02 16:47:21.217929078+00:00
modified: 2025-01-02 17:22:03.138219579+00:00
metadata: {"created":"Thursday, January 02, 2025","tags":["sound","analogue","physics"]}
body:
  ## Analogue

  ### Recording

  The microphone converts sound waves into a continuous electrical signal.

  The signal drives the recording equipment.

  In the case of a vinyl recording: the signal drives a cutting head that carves a
  groove into the master disk.

  In the case of a tape recording: the signal drives electro-magnets that align
  magnetic particles on the tape.

  In both cases, **the recording medium physically mirrors the shape of the sound
  wave**.

  ### Playback

  In the case of vinyl, the needle follows the groove, moving up and down to
  recreate the electrical signal.

  In the case of tape, the magnetic patterns induce electrical current in the
  playback head.

  In each case, the signal is transferred directly to amplifiers and speakers.

  ## Digital

  ### Recording

  Mostly covered in [Binary encoding of sound](Binary_encoding_of_sound.md).

  An important point to note is that the binary encoding that results from the
  sampling is lasered into the CD as microscopic impressions and depressions (pits
  and lands)

  ### Playback

  A laser in the reader reads the pits and lands and converts them back to binary
  numbers. A DAC then converts the numbers back into a continuous electrical
  signal which is then channelled to the amplifiers and speakers.

  ## From recording to production of storage medium

  With analogue production recordings would be made onto tape. Multiple takes are
  recorded, mixed and edited. This results in a master tape which is the final
  version of the recorded sound.

  This then goes to a mastering engineer for mastering which creates the master
  mix. This consists in optimising the sound for the given format (vinyl or tape).
  Also includes adjusting frequencies, levels and adding spacing between tracks.

  Then the master disk enters production. A recording is made onto lacquer disc
  which is very delicate. Often known as the "acetate". This is coated in metal to
  create the "metal master", after various processes this goes to the pressing
  plant who press vinyl copies from it.

  Each stage can introduce subtle variations, especially with re-pressings.
  Re-pressings require new stampers from the master which may have degraded in the
  interim. Plus there is a limit to the amount of pressings possible with a given
  stamper. The earlier pressings are therefore less degraded making them more
  valuable.

  ## Degradation

  With tape and vinyl each playback wears the medium, degrading the quality of the
  captured signal. With CDs, there is no degradation and every copy is a perfect
  copy of the original signal. However this capture, necessarily contains less
  information than the analogue version because of sampling.

raw_content:
  ---
  tags: [sound, analogue, physics]
  created: Thursday, January 02, 2025
  ---

  # Recording and playback of sound

  ## Analogue

  ### Recording

  The microphone converts sound waves into a continuous electrical signal.

  The signal drives the recording equipment.

  In the case of a vinyl recording: the signal drives a cutting head that carves a
  groove into the master disk.

  In the case of a tape recording: the signal drives electro-magnets that align
  magnetic particles on the tape.

  In both cases, **the recording medium physically mirrors the shape of the sound
  wave**.

  ### Playback

  In the case of vinyl, the needle follows the groove, moving up and down to
  recreate the electrical signal.

  In the case of tape, the magnetic patterns induce electrical current in the
  playback head.

  In each case, the signal is transferred directly to amplifiers and speakers.

  ## Digital

  ### Recording

  Mostly covered in [Binary encoding of sound](Binary_encoding_of_sound.md).

  An important point to note is that the binary encoding that results from the
  sampling is lasered into the CD as microscopic impressions and depressions (pits
  and lands)

  ### Playback

  A laser in the reader reads the pits and lands and converts them back to binary
  numbers. A DAC then converts the numbers back into a continuous electrical
  signal which is then channelled to the amplifiers and speakers.

  ## From recording to production of storage medium

  With analogue production recordings would be made onto tape. Multiple takes are
  recorded, mixed and edited. This results in a master tape which is the final
  version of the recorded sound.

  This then goes to a mastering engineer for mastering which creates the master
  mix. This consists in optimising the sound for the given format (vinyl or tape).
  Also includes adjusting frequencies, levels and adding spacing between tracks.

  Then the master disk enters production. A recording is made onto lacquer disc
  which is very delicate. Often known as the "acetate". This is coated in metal to
  create the "metal master", after various processes this goes to the pressing
  plant who press vinyl copies from it.

  Each stage can introduce subtle variations, especially with re-pressings.
  Re-pressings require new stampers from the master which may have degraded in the
  interim. Plus there is a limit to the amount of pressings possible with a given
  stamper. The earlier pressings are therefore less degraded making them more
  valuable.

  ## Degradation

  With tape and vinyl each playback wears the medium, degrading the quality of the
  captured signal. With CDs, there is no degradation and every copy is a perfect
  copy of the original signal. However this capture, necessarily contains less
  information than the analogue version because of sampling.


path: 738d0481_sound_recording_as_wave_phenomena.md
sortable_path: 738d0481_sound_recording_as_wave_phenomena.md
title: 738d0481_sound_recording_as_wave_phenomena
checksum: a9dc1aead9226467ca2ae21b8f7233ae0d3b326c9e2b6048e2d122ff0141b496
created: 2025-01-02 17:16:02.337376164+00:00
modified: 2025-01-02 18:02:28.77666568+00:00
metadata: {"created":"Thursday, January 02, 2025","tags":["sound","analogue","physics"]}
lead:
  When the microphone captures alterations in sound pressure and converts it into
  an electric signal, we have a scenario where the same formal or mathematical
  relationships in the phenomena (amplitude, frequency, wavelength) are being
  expressed in two different modes of matter.

body:
  When the microphone captures alterations in sound pressure and converts it into
  an electric signal, we have a scenario where the same formal or mathematical
  relationships in the phenomena (amplitude, frequency, wavelength) are being
  expressed in two different modes of matter.

  This is quite profound especially as the resulting electrical waveform is
  _identical_ to the original sound wave. This is effectively what enables
  accurate playback. We can convert between the two mediums seamlessly providing
  we captured their initial expression accurately.

  It's like two different people doing the same dance:

  In air:

  - The dancers are air molecules
  - They move back and forth in waves of compression and rarefaction
  - Their "dance pattern" is the sound wave

  In wire:

  - The dancers are electrons
  - They move back and forth creating waves of voltage
  - They perform the exact same "dance pattern"

raw_content:
  ---
  tags: [sound, analogue, physics]
  created: Thursday, January 02, 2025
  ---

  # 738d0481_sound_recording_as_wave_phenomena

  When the microphone captures alterations in sound pressure and converts it into
  an electric signal, we have a scenario where the same formal or mathematical
  relationships in the phenomena (amplitude, frequency, wavelength) are being
  expressed in two different modes of matter.

  This is quite profound especially as the resulting electrical waveform is
  _identical_ to the original sound wave. This is effectively what enables
  accurate playback. We can convert between the two mediums seamlessly providing
  we captured their initial expression accurately.

  It's like two different people doing the same dance:

  In air:

  - The dancers are air molecules
  - They move back and forth in waves of compression and rarefaction
  - Their "dance pattern" is the sound wave

  In wire:

  - The dancers are electrons
  - They move back and forth creating waves of voltage
  - They perform the exact same "dance pattern"


path: Monitor_DNS_resolution_and_internet_connectivity_script.md
sortable_path: Monitor_DNS_resolution_and_internet_connectivity_script.md
title: Monitor DNS resolution and internet connectivity script
checksum: 2783031db8f9eb0513eac607269b61e28142fb41b6b64f915b67878f8097e49b
created: 2025-01-04 13:27:13.572149706+00:00
modified: 2025-01-13 20:54:38.228432527+00:00
metadata: {"created":"Monday, January 13, 2025","tags":["networks","procedural"]}
lead:
  ```bash
   while true; do
      date
      echo "DNS check:"
      host google.com
      echo "Ping check:"
      ping -c 1 google.com | grep "time="
      echo "---"
      sleep 5
  done
  ```

body:
  ```bash
   while true; do
      date
      echo "DNS check:"
      host google.com
      echo "Ping check:"
      ping -c 1 google.com | grep "time="
      echo "---"
      sleep 5
  done
  ```

  See [ping](./Ping.md) for more details on what this is doing.

  The `host` command contacts a DNS server and does a reverse lookup for the
  IP-address of the human-readable domain name.

raw_content:
  ---
  tags: [networks, procedural]
  created: Monday, January 13, 2025
  ---

  # Monitor DNS resolution and internet connectivity script

  ```bash
   while true; do
      date
      echo "DNS check:"
      host google.com
      echo "Ping check:"
      ping -c 1 google.com | grep "time="
      echo "---"
      sleep 5
  done
  ```

  See [ping](./Ping.md) for more details on what this is doing.

  The `host` command contacts a DNS server and does a reverse lookup for the
  IP-address of the human-readable domain name.


path: 747589ad_effects_of_privacy_violation.md
sortable_path: 747589ad_effects_of_privacy_violation.md
title: 747589ad_effects_of_privacy_violation
checksum: c2ddc920f3ad949a4524a43d94ece19d3de464e8709332e389b91bf1fe07f8f5
created: 2025-01-09 18:50:38.180558945+00:00
modified: 2025-01-09 19:03:41.565691729+00:00
metadata: {"created":"Thursday, January 09, 2025","tags":["ecopolsoc","privacy","surveillance-capitalism"]}
lead:
  _What surveillance really is, at its root, is a highly effective form of social
  control. The knowledge of always being watched changes our behavior and stifles
  dissent. The inability to associate secretly means there is no longer any
  possibility for free association. The inability to whisper means there is no
  longer any speech that is truly free of coercion, real or implied. Most
  profoundly, pervasive surveillance threatens to eliminate the most vital element
  of both democracy and social movements: the mental space for people to form
  dissenting and unpopular views._

body:
  _What surveillance really is, at its root, is a highly effective form of social
  control. The knowledge of always being watched changes our behavior and stifles
  dissent. The inability to associate secretly means there is no longer any
  possibility for free association. The inability to whisper means there is no
  longer any speech that is truly free of coercion, real or implied. Most
  profoundly, pervasive surveillance threatens to eliminate the most vital element
  of both democracy and social movements: the mental space for people to form
  dissenting and unpopular views._

  - https://riseup.net/en/about-us/newsletter/2013/08#riseup-and-the-recent-email-provider-closures

  _If you live in a surveillance state for long enough, you create a censor in
  your head. When video cameras are routine fixtures in urban landscapes; when
  corporate Internet giants store records of online navigation and communication;
  and when managers and bosses maintain capabilities to measure and monitor
  employees as never before, society at large pays the price. These different
  vectors of surveillance aggregate, exerting a pressure for us to blend in, to
  think twice before speaking out, to, in essence, follow a narrow set of
  prescribed norms. Social conformity encourages quiet resignation and discourages
  the experimental - and necessarily risky - acts of speaking, thinking, and doing
  required for healthy democratic dissent._

  Gabriella Coleman. _Hacker, Hoaxer, Whistleblower, Spy_ (2014): 380

raw_content:
  ---
  tags: [ecopolsoc, privacy, surveillance-capitalism]
  created: Thursday, January 09, 2025
  ---

  # 747589ad_effects_of_privacy_violation

  _What surveillance really is, at its root, is a highly effective form of social
  control. The knowledge of always being watched changes our behavior and stifles
  dissent. The inability to associate secretly means there is no longer any
  possibility for free association. The inability to whisper means there is no
  longer any speech that is truly free of coercion, real or implied. Most
  profoundly, pervasive surveillance threatens to eliminate the most vital element
  of both democracy and social movements: the mental space for people to form
  dissenting and unpopular views._

  - https://riseup.net/en/about-us/newsletter/2013/08#riseup-and-the-recent-email-provider-closures

  _If you live in a surveillance state for long enough, you create a censor in
  your head. When video cameras are routine fixtures in urban landscapes; when
  corporate Internet giants store records of online navigation and communication;
  and when managers and bosses maintain capabilities to measure and monitor
  employees as never before, society at large pays the price. These different
  vectors of surveillance aggregate, exerting a pressure for us to blend in, to
  think twice before speaking out, to, in essence, follow a narrow set of
  prescribed norms. Social conformity encourages quiet resignation and discourages
  the experimental - and necessarily risky - acts of speaking, thinking, and doing
  required for healthy democratic dissent._

  Gabriella Coleman. _Hacker, Hoaxer, Whistleblower, Spy_ (2014): 380


path: View_IP_addresses.md
sortable_path: View_IP_addresses.md
title: View IP addresses
lead:
  Use `ip` package to view the [IP addresses](IP_addresses.md) associated with
  your current host:

checksum: eb10adeb1c2a55330524862e35dd667be3d04ac8a31d607e3bc2d3974587fcd1
created: 2025-01-10 16:51:50.329623342+00:00
modified: 2025-01-10 17:09:27.835728168+00:00
metadata: {"created":"Friday, January 10, 2025","tags":["networks","procedural","Linux"]}
body:
  Use `ip` package to view the [IP addresses](IP_addresses.md) associated with
  your current host:

  ```sh
  ip address show
  ```

  I am curently using ethernet on the computer I am writing this. This is listed
  as follows:

  ```
  enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  link/ether 90:2e:16:53:98:b9 brd ff:ff:ff:ff:ff:ffinet
  192.168.0.3/24 brd 192.168.0.255 scope global dynamic noprefixroute enp0s31f6
      valid_lft 83353sec preferred_lft 83353sec
  ```

  The most important line is the third. This indicates my IP address
  (`192.168.0.3/24`) including its subnet mask.

raw_content:
  ---
  tags: [networks, procedural, Linux]
  created: Friday, January 10, 2025
  ---

  # View IP addresses

  Use `ip` package to view the [IP addresses](IP_addresses.md) associated with
  your current host:

  ```sh
  ip address show
  ```

  I am curently using ethernet on the computer I am writing this. This is listed
  as follows:

  ```
  enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  link/ether 90:2e:16:53:98:b9 brd ff:ff:ff:ff:ff:ffinet
  192.168.0.3/24 brd 192.168.0.255 scope global dynamic noprefixroute enp0s31f6
      valid_lft 83353sec preferred_lft 83353sec
  ```

  The most important line is the third. This indicates my IP address
  (`192.168.0.3/24`) including its subnet mask.


path: Ping.md
sortable_path: Ping.md
title: Ping
checksum: 93c2e66315606664cbc568d8ba2309d8a5c0826627747e27188931b6314703bd
created: 2025-01-13 20:27:31.401736298+00:00
modified: 2025-01-13 20:54:42.518506698+00:00
metadata: {"created":"Monday, January 13, 2025","tags":["networks","procedural"]}
lead:
  `ping` is perhaps the most basic network debugging tool. It sends
  [ICMP](./ICMP_Protocol.md) echo request packets to a recipient host.

body:
  `ping` is perhaps the most basic network debugging tool. It sends
  [ICMP](./ICMP_Protocol.md) echo request packets to a recipient host.

  If the recipient receives the packet and is configured to reply, it sends an
  ICMP echo response packet in return.

  For example:

  ```
  $ ping google.com
  PING google.com (142.250.200.14) 56(84) bytes of data.
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=1 ttl=116 time=19.1 ms
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=2 ttl=116 time=19.2 ms
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=3 ttl=116 time=19.1 ms
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=4 ttl=116 time=21.1 ms
  ```

  The first indicates the IP address of the host you are pinging (once per second,
  by default). The subsequent lines are the response packets, echoed back from
  this host.

  The `icmp_seq` is the number of the request in the sequence. If there was to be
  a gap in the sequence, this would be the clearest indication of a connection
  problem.The packets should not arrive out of order since there is only one
  packet being sent per second.

  `time` is the round-trip time - from the sending of the packet to its return to
  the sender.

  If the packet cannot reach the destination host, the final router to see the
  packet retunes a "host unreachable" packet to `ping`.

raw_content:
  ---
  tags: [networks, procedural]
  created: Monday, January 13, 2025
  ---

  # Ping

  `ping` is perhaps the most basic network debugging tool. It sends
  [ICMP](./ICMP_Protocol.md) echo request packets to a recipient host.

  If the recipient receives the packet and is configured to reply, it sends an
  ICMP echo response packet in return.

  For example:

  ```
  $ ping google.com
  PING google.com (142.250.200.14) 56(84) bytes of data.
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=1 ttl=116 time=19.1 ms
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=2 ttl=116 time=19.2 ms
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=3 ttl=116 time=19.1 ms
  64 bytes from lhr48s29-in-f14.1e100.net (142.250.200.14): icmp_seq=4 ttl=116 time=21.1 ms
  ```

  The first indicates the IP address of the host you are pinging (once per second,
  by default). The subsequent lines are the response packets, echoed back from
  this host.

  The `icmp_seq` is the number of the request in the sequence. If there was to be
  a gap in the sequence, this would be the clearest indication of a connection
  problem.The packets should not arrive out of order since there is only one
  packet being sent per second.

  `time` is the round-trip time - from the sending of the packet to its return to
  the sender.

  If the packet cannot reach the destination host, the final router to see the
  packet retunes a "host unreachable" packet to `ping`.


path: ICMP_Protocol.md
sortable_path: ICMP_Protocol.md
title: ICMP Protocol
checksum: 81995cf8b107982fa597a658a7ae4ae2532b03dd8636940b6b4f9dd4da5d5693
created: 2025-01-13 20:53:19.536774731+00:00
modified: 2025-01-13 20:40:58.180917574+00:00
metadata: {"created":"Monday, January 13, 2025","tags":["networks"]}
lead:
  The Internet Control Message Protocol operates alongside the
  [Internet Protocol](Internet_Layer_of_Internet_Protocol.md) at the
  [Transport Layer](Transport_Layer_of_Internet_Protocol.md).

body:
  The Internet Control Message Protocol operates alongside the
  [Internet Protocol](Internet_Layer_of_Internet_Protocol.md) at the
  [Transport Layer](Transport_Layer_of_Internet_Protocol.md).

  It doesn't carry any actual user data and thus there is no
  [Application Layer](Application_Layer_of_Internet_Protocol.md) processes above
  it. It's usage is primarily for diagnostics and debugging. When network problems
  occur, ICMP generates messages to report errors back to the source.

  Probably the most familiar tools related to ICMP is [Ping](Ping.md).

raw_content:
  ---
  tags: [networks]
  created: Monday, January 13, 2025
  ---

  # ICMP Protocol

  The Internet Control Message Protocol operates alongside the
  [Internet Protocol](Internet_Layer_of_Internet_Protocol.md) at the
  [Transport Layer](Transport_Layer_of_Internet_Protocol.md).

  It doesn't carry any actual user data and thus there is no
  [Application Layer](Application_Layer_of_Internet_Protocol.md) processes above
  it. It's usage is primarily for diagnostics and debugging. When network problems
  occur, ICMP generates messages to report errors back to the source.

  Probably the most familiar tools related to ICMP is [Ping](Ping.md).


path: c9d7492f_requerimiento.md
sortable_path: c9d7492f_requerimiento.md
title: c9d7492f_requerimiento
checksum: d6deba34cb598c5af3990c218985e07d74483368d97e451556392cb118e5e990
created: 2025-01-20 18:03:46.146452477+00:00
modified: 2025-01-21 16:29:03.411943638+00:00
metadata: {"created":"Monday, January 20, 2025","tags":["ecopolsoc","internet","surveillance-capitalism"]}
lead:
  Zuboff likens the expropriation of private information by surveillance
  capitalists to the behaviour of the Conquistadores when they first arrived in
  South America.

body:
  Zuboff likens the expropriation of private information by surveillance
  capitalists to the behaviour of the Conquistadores when they first arrived in
  South America.

  In order to legitimise their actions they were obliged to inform the native
  inhabitants of their intentions and tell them that they were now subordinate to
  the Spanish crown.

  This provided supposed justification for the violence and brutality that would
  follow if the natives did not comply. Sometimes the declaration would just be
  muttered moments before attack.

  "Conquest by declaration" is also the technique of Google, she claims. The
  equivalent of the _requerimiento_ being the opaque and incomprehensible terms of
  of service, _designed_ not to be read.

  _These twenty-first-century invaders do not ask permission; the forge ahead,
  papering the scorched earth with faux-legitimation practices. Instead of
  cynically conveyed monarchical edicts, they offer cynically conveyed
  terms-of-service agreements whose stipulations are just as obscured and
  incomprehensible. They build their fortifications, fiercely defending their
  claimed territories, while gathering strength for the next incursion.
  Eventually, they build their towns in intricate ecosystems of commerce, politics
  and culture that declare the legitimacy and inevitability of all that they have
  accomplished._

  _The Age of Surveillance Capitalism_ (2019): 179

raw_content:
  ---
  tags: [ecopolsoc, internet, surveillance-capitalism]
  created: Monday, January 20, 2025
  ---

  # c9d7492f_requerimiento

  Zuboff likens the expropriation of private information by surveillance
  capitalists to the behaviour of the Conquistadores when they first arrived in
  South America.

  In order to legitimise their actions they were obliged to inform the native
  inhabitants of their intentions and tell them that they were now subordinate to
  the Spanish crown.

  This provided supposed justification for the violence and brutality that would
  follow if the natives did not comply. Sometimes the declaration would just be
  muttered moments before attack.

  "Conquest by declaration" is also the technique of Google, she claims. The
  equivalent of the _requerimiento_ being the opaque and incomprehensible terms of
  of service, _designed_ not to be read.

  _These twenty-first-century invaders do not ask permission; the forge ahead,
  papering the scorched earth with faux-legitimation practices. Instead of
  cynically conveyed monarchical edicts, they offer cynically conveyed
  terms-of-service agreements whose stipulations are just as obscured and
  incomprehensible. They build their fortifications, fiercely defending their
  claimed territories, while gathering strength for the next incursion.
  Eventually, they build their towns in intricate ecosystems of commerce, politics
  and culture that declare the legitimacy and inevitability of all that they have
  accomplished._

  _The Age of Surveillance Capitalism_ (2019): 179


path: 4dec6fe0_shadow_text.md
sortable_path: 4dec6fe0_shadow_text.md
title: 4dec6fe0_shadow_text
checksum: 4285ac86bb54e3db9fde04613ea1b8ebaa3870262e8a7fd582492a1e59f86832
created: 2025-01-20 18:13:35.230592075+00:00
modified: 2025-01-21 16:29:26.828989586+00:00
metadata: {"created":"Monday, January 20, 2025","tags":["ecopolsoc","internet","surveillance-capitalism"]}
lead:
  Surveillance capitalism creates two "electronic texts". The first is the
  public-facing text which consists in the benefits conferred by the surveillance
  capitalists' services, expressed in their self-regarding mission statements. The
  "universe of information and connection [they] bring to our fingertips" (posts,
  blogs, likes, photos, spreadsheets, music, tweets etc).

body:
  Surveillance capitalism creates two "electronic texts". The first is the
  public-facing text which consists in the benefits conferred by the surveillance
  capitalists' services, expressed in their self-regarding mission statements. The
  "universe of information and connection [they] bring to our fingertips" (posts,
  blogs, likes, photos, spreadsheets, music, tweets etc).

  The first text is a sort of Trojan Horse, serving as the "supply operation" for
  the second, _shadow_ text. Everything we contribute to the first text becomes a
  target for surplus extraction via the shadow text. This is the ultimate purpose
  of the services and is hidden from view and only for the purview of the
  surveillance capitalists.

  _In this text, our experience is dragooned as raw material to be accumulated and
  analyzed as means to others' market ends. The shadow text is a burgeoning
  accumulation of behavioural surplus and its analyses, and it says more about us
  than we can know about ourselves. Worse still, it becomes increasingly
  difficult, and perhaps impossible, to refrain from contributing to the shadow
  text. It automatically feeds on our experience as we engage in the normal and
  necessary routines of social participation._

  _The Age of Surveillance Capitalism_ (2019): 185

raw_content:
  ---
  tags: [ecopolsoc, internet, surveillance-capitalism]
  created: Monday, January 20, 2025
  ---

  # 4dec6fe0_shadow_text

  Surveillance capitalism creates two "electronic texts". The first is the
  public-facing text which consists in the benefits conferred by the surveillance
  capitalists' services, expressed in their self-regarding mission statements. The
  "universe of information and connection [they] bring to our fingertips" (posts,
  blogs, likes, photos, spreadsheets, music, tweets etc).

  The first text is a sort of Trojan Horse, serving as the "supply operation" for
  the second, _shadow_ text. Everything we contribute to the first text becomes a
  target for surplus extraction via the shadow text. This is the ultimate purpose
  of the services and is hidden from view and only for the purview of the
  surveillance capitalists.

  _In this text, our experience is dragooned as raw material to be accumulated and
  analyzed as means to others' market ends. The shadow text is a burgeoning
  accumulation of behavioural surplus and its analyses, and it says more about us
  than we can know about ourselves. Worse still, it becomes increasingly
  difficult, and perhaps impossible, to refrain from contributing to the shadow
  text. It automatically feeds on our experience as we engage in the normal and
  necessary routines of social participation._

  _The Age of Surveillance Capitalism_ (2019): 185


path: LineageOS_backup.md
sortable_path: LineageOS_backup.md
title: LineageOS major update procedure
lead: ## Backup
checksum: 8c8e95de8036787cb60edf83fb490b1b163818e6da4266b26ba8e7300305dcea
created: 2025-01-24 17:22:55.866259038+00:00
modified: 2025-01-24 19:35:54.296081399+00:00
metadata: {"created":"Friday, January 24, 2025","tags":["procedural","linux","android"]}
body:
  ## Backup

  - Enable developer mode by going to Settings > About Phone and tapping "Build
    Number" seven times

  - Enable USB debugging in Developer Options

  - Test by running `adb devices` in PC terminal

  - Then run `adb backup -apk -shared -all -system -f lineage_backup.ab`

  ## Download latest version of TWRP

  Make sure you get the right one for Google Pixel 2 (taimen), e.g:

  ```
  twrp-3.7.0_9-0-taimen.img
  ```

  ## Download ROM image and associated files

  Go to
  [https://wiki.lineageos.org/devices/taimen/](https://wiki.lineageos.org/devices/taimen/),
  download zip, viz.

  ```
  lineage-22.1-20250123-nightly-taimen-signed.zip
  ```

  ## Boot into fastboot

  Press Power + Vol Down

  Check connection with:

  ```
  fastboot devices
  ```

  ## Boot into TWRP

  ```
  fastboot boot  twrp-3.7.0_9-0-taimen.img
  ```

  ## Copy OS rom to Phone

  ```
  adb push lineage-22*.zip /sdcard/
  ```

  ## Install OS in TWRP

  In TWRP select Install > Select ROM zip

  Select flash

  Then reboot

  ```

  ```

raw_content:
  ---
  tags: [procedural, linux, android]
  created: Friday, January 24, 2025
  ---

  # LineageOS major update procedure

  ## Backup

  - Enable developer mode by going to Settings > About Phone and tapping "Build
    Number" seven times

  - Enable USB debugging in Developer Options

  - Test by running `adb devices` in PC terminal

  - Then run `adb backup -apk -shared -all -system -f lineage_backup.ab`

  ## Download latest version of TWRP

  Make sure you get the right one for Google Pixel 2 (taimen), e.g:

  ```
  twrp-3.7.0_9-0-taimen.img
  ```

  ## Download ROM image and associated files

  Go to
  [https://wiki.lineageos.org/devices/taimen/](https://wiki.lineageos.org/devices/taimen/),
  download zip, viz.

  ```
  lineage-22.1-20250123-nightly-taimen-signed.zip
  ```

  ## Boot into fastboot

  Press Power + Vol Down

  Check connection with:

  ```
  fastboot devices
  ```

  ## Boot into TWRP

  ```
  fastboot boot  twrp-3.7.0_9-0-taimen.img
  ```

  ## Copy OS rom to Phone

  ```
  adb push lineage-22*.zip /sdcard/
  ```

  ## Install OS in TWRP

  In TWRP select Install > Select ROM zip

  Select flash

  Then reboot

  ```

  ```


path: Proxies.md
sortable_path: Proxies.md
title: Proxies
checksum: c9e57817c83a63911f8dbce2a5d9d6b49013996b0a74b9880930cae767f789a7
created: 2025-02-03 12:33:32.607174396+00:00
modified: 2025-02-03 12:45:57.62435577+00:00
metadata: {"created":"Monday, February 03, 2025","tags":["networks","proxies"]}
lead:
  A standard proxy stands as a blocking layer between a client and a server. A
  client makes a request to a server but before it reaches it, it is intercepted
  by a proxy that reviews the request and decides whether to pass on to the
  requested location.

body:
  A standard proxy stands as a blocking layer between a client and a server. A
  client makes a request to a server but before it reaches it, it is intercepted
  by a proxy that reviews the request and decides whether to pass on to the
  requested location.

  As such, from the server's point of view, the request comes from the proxy not
  the client.

  An example application would be organisations who want to monitor or filter
  their employees web activities.

  A **reverse proxy** has the opposite orientation. It acts as buffer between a
  server and a client, filtering the requests that are sent to a server.

  This can also be for reasons of security, but most typically a reverse-proxy is
  used to manage requests to different services that a server is running.

  If a client requests `service-a.some-server/slug` the request is sent to the
  proxy operating at the root level, it siphons the request to its intended
  sub-service (`service-a`) and the port this service is running on.

  So, from the point of view of Service A, the request doesn't come from the
  external client but from the reverse proxy.

  In addition to directing traffic, reverse proxies can also implement load
  balancing: running procedures that ensure traffic is met at times of high
  capacity, ensuring that the server's resources are not exhausted.

raw_content:
  ---
  tags: [networks, proxies]
  created: Monday, February 03, 2025
  ---

  # Proxies

  A standard proxy stands as a blocking layer between a client and a server. A
  client makes a request to a server but before it reaches it, it is intercepted
  by a proxy that reviews the request and decides whether to pass on to the
  requested location.

  As such, from the server's point of view, the request comes from the proxy not
  the client.

  An example application would be organisations who want to monitor or filter
  their employees web activities.

  A **reverse proxy** has the opposite orientation. It acts as buffer between a
  server and a client, filtering the requests that are sent to a server.

  This can also be for reasons of security, but most typically a reverse-proxy is
  used to manage requests to different services that a server is running.

  If a client requests `service-a.some-server/slug` the request is sent to the
  proxy operating at the root level, it siphons the request to its intended
  sub-service (`service-a`) and the port this service is running on.

  So, from the point of view of Service A, the request doesn't come from the
  external client but from the reverse proxy.

  In addition to directing traffic, reverse proxies can also implement load
  balancing: running procedures that ensure traffic is met at times of high
  capacity, ensuring that the server's resources are not exhausted.


path: e383b8b3_nginx_vs_traefik.md
sortable_path: e383b8b3_nginx_vs_traefik.md
title: e383b8b3_nginx_vs_traefik
checksum: 670bc2639951745e4940a6ad5aab4c8de9543aea604ae4d97270e799826db92f
created: 2025-02-03 12:45:42.187390835+00:00
modified: 2025-02-03 12:48:36.34026334+00:00
metadata: {"created":"Monday, February 03, 2025","tags":["networks","proxies"]}
lead:
  Both nginx and traefik are server technologies (along with Apache), nginx has a
  longer pedigree but traefik is calibrated for more modern microservice-based
  architectures (i.e Docker containers). nginx is better suited to traditional
  static web-based hosting.

body:
  Both nginx and traefik are server technologies (along with Apache), nginx has a
  longer pedigree but traefik is calibrated for more modern microservice-based
  architectures (i.e Docker containers). nginx is better suited to traditional
  static web-based hosting.

  traefik has support for [HTTPS](./HTTPS.md) natively whereas nginx requires a
  plugin.

raw_content:
  ---
  tags: [networks, proxies]
  created: Monday, February 03, 2025
  ---

  # e383b8b3_nginx_vs_traefik

  Both nginx and traefik are server technologies (along with Apache), nginx has a
  longer pedigree but traefik is calibrated for more modern microservice-based
  architectures (i.e Docker containers). nginx is better suited to traditional
  static web-based hosting.

  traefik has support for [HTTPS](./HTTPS.md) natively whereas nginx requires a
  plugin.


path: Certificate_authorities.md
sortable_path: Certificate_authorities.md
title: Certificates and certificate authorities
checksum: b0a00de4c20b1170e84393af3c84c436fc3cdfdb64ba50629f3ff5eb4e770b3e
created: 2025-02-09 12:18:13.284687081+00:00
modified: 2025-02-09 13:31:23.258812929+00:00
metadata: {"created":"Sunday, February 09, 2025","tags":["networks","server-management"]}
lead:
  > HTTPS is a secure tunnel for data transmission, but the certificate is what
  > establishes trust and enables the secure tunnel in the first place.

body:
  > HTTPS is a secure tunnel for data transmission, but the certificate is what
  > establishes trust and enables the secure tunnel in the first place.

  Before secure communication over HTTPS can commence between a client and a
  server, the client must confirm that the server is who it says it is. This is in
  order to prevent man-in-the-middle attacks where malicious actors attempt to
  impersonate legitimate websites.

  Certificates exist to authenticate domains, prior to the exchange of encrypted
  data. To do this, a server generates a certificate with a reputable Certificate
  Authority, thereby verifying its identity with the CA. It sends this certificate
  (which includes its [public key](./bbdcb54f_public_key_cryptography.md)) when
  initiating a connection with a client.

  In addition to the public key, the certificate provides:

  - the domain name
  - the CA that certified the domain
  - the period of validity
  - a digital signature from the CA

  The client uses this data to validate the identity of the server it has sent a
  request to. Browsers come pre-installed with a list of trusted CAs.

  Certificates are called TLS/SSL certificates but the main protocol is TLS rather
  than SSL.

  Indications that a server is not to be trusted include:

  - expired certificates
  - certificates issued for the wrong domain
  - self-signed certificates not issued by a trusted CA

  ## Certification process

  Standard domain-validated authentication proceeds as follows:

  1. Server generates public/private key pair
  2. Server creates a certificate signing request containing its public key and
     domain name
  3. CA validates the domain and signs the certificate with its private key
  4. Server discloses its public key and CA signature
  5. Client knows CAs public key and uses this to verify the signature
  6. Server's public key is then trusted

  ### How validation executes between CA and server

  To validate the domain (step 3 above), the CA gives the server a "challenge" to
  prove they control the domain. This can include:

  - Putting a specific token as a DNS TXT record (DNS validation)
  - Putting a file at a certain URL (HTTP validation)
  - Email validation: sending an email to standard admin address for the server.
    Server owner clicks the link.

  ### Related

  See [Let's Encrypt](./Let's_Encrypt.md)

raw_content:
  ---
  tags: [networks, server-management]
  created: Sunday, February 09, 2025
  ---

  # Certificates and certificate authorities

  > HTTPS is a secure tunnel for data transmission, but the certificate is what
  > establishes trust and enables the secure tunnel in the first place.

  Before secure communication over HTTPS can commence between a client and a
  server, the client must confirm that the server is who it says it is. This is in
  order to prevent man-in-the-middle attacks where malicious actors attempt to
  impersonate legitimate websites.

  Certificates exist to authenticate domains, prior to the exchange of encrypted
  data. To do this, a server generates a certificate with a reputable Certificate
  Authority, thereby verifying its identity with the CA. It sends this certificate
  (which includes its [public key](./bbdcb54f_public_key_cryptography.md)) when
  initiating a connection with a client.

  In addition to the public key, the certificate provides:

  - the domain name
  - the CA that certified the domain
  - the period of validity
  - a digital signature from the CA

  The client uses this data to validate the identity of the server it has sent a
  request to. Browsers come pre-installed with a list of trusted CAs.

  Certificates are called TLS/SSL certificates but the main protocol is TLS rather
  than SSL.

  Indications that a server is not to be trusted include:

  - expired certificates
  - certificates issued for the wrong domain
  - self-signed certificates not issued by a trusted CA

  ## Certification process

  Standard domain-validated authentication proceeds as follows:

  1. Server generates public/private key pair
  2. Server creates a certificate signing request containing its public key and
     domain name
  3. CA validates the domain and signs the certificate with its private key
  4. Server discloses its public key and CA signature
  5. Client knows CAs public key and uses this to verify the signature
  6. Server's public key is then trusted

  ### How validation executes between CA and server

  To validate the domain (step 3 above), the CA gives the server a "challenge" to
  prove they control the domain. This can include:

  - Putting a specific token as a DNS TXT record (DNS validation)
  - Putting a file at a certain URL (HTTP validation)
  - Email validation: sending an email to standard admin address for the server.
    Server owner clicks the link.

  ### Related

  See [Let's Encrypt](./Let's_Encrypt.md)


path: Firewalls.md
sortable_path: Firewalls.md
title: Firewalls
checksum: c5ffcb02a9eb88062982d8fabc611468f51901d2470b16cb0ec811e8d84290d4
created: 2025-02-09 12:53:38.517010875+00:00
modified: 2025-02-16 15:17:14.646142747+00:00
metadata: {"created":"Sunday, February 09, 2025","tags":["networks","server-management","firewalls"]}
lead:
  A firewall is a server mechanism that filters incoming requests for resources
  and services that it hosts.

body:
  A firewall is a server mechanism that filters incoming requests for resources
  and services that it hosts.

  Based on data about the requester, derived from the
  [TLS](./Transport_Layer_of_Internet_Protocol.md) packet headers, the firewall
  decides whether or not to grant access to the requested resource.

  It will typically comprise three actions:

  - accept
  - drop
  - reject

  Accept is obvious. To "drop" means to block the traffic without sending a reply.
  To "reject" is to block the traffic but reply with an "unreachable" error.

  An example of a firewall rule would be to drop incoming traffic from IP
  addresses of non-technical employees to the SSH port (22) but allow technical
  staff to access it.

  Network traffic is matched agains a list of firewall rules in a sequence from
  first to last. As you cannot cover all cases, you must specify a default action
  when a rule is not matched.

  Whilst firewalls are focused on incoming traffic, you can also apply rules to
  outgoing traffic. A server will permit most outgoing traffic since, after all,
  it can trust itself however you may want to stop the server connecting to known
  malicious actors.

  ## Related software

  IPTables is the standard Linux software for managing firewalls. There is also
  Uncomplicated Firewall (UWF) which attaches to IPTables and simplifies the
  process.

  Another useful software is Fail2ban which can automatically configure your
  firewall to block brute force login attempts and DDOS attacks.

  ## Difference from reverse proxies

  Whilst both reverse proxies and firewalls filter incoming traffic to a server,
  there are important differences.

  The firewall operates at the
  [Transport Layer](./Transport_Layer_of_Internet_Protocol.md) wehreas proxies
  operate at the [Application Layer](./Application_Layer_of_Internet_Protocol.md).

  The firewall is primarily concerned with security and precedes anything that
  happens with the incoming packets at the Appplication Layer. The packets first
  have to get through the firewall before the request can be routed to a specific
  service on the server.

raw_content:
  ---
  tags: [networks, server-management, firewalls]
  created: Sunday, February 09, 2025
  ---

  # Firewalls

  A firewall is a server mechanism that filters incoming requests for resources
  and services that it hosts.

  Based on data about the requester, derived from the
  [TLS](./Transport_Layer_of_Internet_Protocol.md) packet headers, the firewall
  decides whether or not to grant access to the requested resource.

  It will typically comprise three actions:

  - accept
  - drop
  - reject

  Accept is obvious. To "drop" means to block the traffic without sending a reply.
  To "reject" is to block the traffic but reply with an "unreachable" error.

  An example of a firewall rule would be to drop incoming traffic from IP
  addresses of non-technical employees to the SSH port (22) but allow technical
  staff to access it.

  Network traffic is matched agains a list of firewall rules in a sequence from
  first to last. As you cannot cover all cases, you must specify a default action
  when a rule is not matched.

  Whilst firewalls are focused on incoming traffic, you can also apply rules to
  outgoing traffic. A server will permit most outgoing traffic since, after all,
  it can trust itself however you may want to stop the server connecting to known
  malicious actors.

  ## Related software

  IPTables is the standard Linux software for managing firewalls. There is also
  Uncomplicated Firewall (UWF) which attaches to IPTables and simplifies the
  process.

  Another useful software is Fail2ban which can automatically configure your
  firewall to block brute force login attempts and DDOS attacks.

  ## Difference from reverse proxies

  Whilst both reverse proxies and firewalls filter incoming traffic to a server,
  there are important differences.

  The firewall operates at the
  [Transport Layer](./Transport_Layer_of_Internet_Protocol.md) wehreas proxies
  operate at the [Application Layer](./Application_Layer_of_Internet_Protocol.md).

  The firewall is primarily concerned with security and precedes anything that
  happens with the incoming packets at the Appplication Layer. The packets first
  have to get through the firewall before the request can be routed to a specific
  service on the server.


path: Let's_Encrypt.md
sortable_path: Let's_Encrypt.md
title: Let's Encrypt
checksum: 6b3391e3f1de6e9ebe9e6b4be1cf01d7dd2fd03f33f9c52ec43db5cf0b80c746
created: 2025-02-09 13:31:00.139717208+00:00
modified: 2025-02-10 17:14:02.242801267+00:00
metadata: {"created":"Sunday, February 09, 2025","tags":["networks","server-management","encryption"]}
lead:
  ACME (Automatic Certificate Management Environment) is an open source protocol
  that automates the process of obtaining and renewing SSL/TLS certificates.

body:
  ACME (Automatic Certificate Management Environment) is an open source protocol
  that automates the process of obtaining and renewing SSL/TLS certificates.

  Let's Encrypt (LE) created this protocol and uses it to issue certificates.

  Certbot is a popular ACME client that communicates with the LE servers to

  - validate domain ownership
  - request certificates
  - install the certificates on the web server
  - handle automatic renewals

  Let's Encrypt was founded by the
  [Internet Security Research Group](https://www.abetterinternet.org/). Certbot
  was created independently by the EFF and is maintained by them.

raw_content:
  ---
  tags: [networks, server-management, encryption]
  created: Sunday, February 09, 2025
  ---

  # Let's Encrypt

  ACME (Automatic Certificate Management Environment) is an open source protocol
  that automates the process of obtaining and renewing SSL/TLS certificates.

  Let's Encrypt (LE) created this protocol and uses it to issue certificates.

  Certbot is a popular ACME client that communicates with the LE servers to

  - validate domain ownership
  - request certificates
  - install the certificates on the web server
  - handle automatic renewals

  Let's Encrypt was founded by the
  [Internet Security Research Group](https://www.abetterinternet.org/). Certbot
  was created independently by the EFF and is maintained by them.


path: Setup encrypted harddrive.md
sortable_path: Setup encrypted harddrive.md
title: Setup an encrypted harddrive
checksum: 6eb802bbca6795f4feca4f0acf11b34b5e2cbe944e9c6e6ec815dfbfafdbd0ea
created: 2025-02-10 19:26:51.956385272+00:00
modified: 2025-02-11 18:11:41.796385106+00:00
metadata: {"created":"Monday, February 10, 2025","tags":["procedural","disks","encryption"]}
lead:
  Firstly [create a new partition](./Creating_a_Linux_partition_table.md) on the
  unencrypted drive with `fdisk`.

body:
  Firstly [create a new partition](./Creating_a_Linux_partition_table.md) on the
  unencrypted drive with `fdisk`.

  Create LUKs encryption:

  ```sh
  sudo cryptsetup luksFormat /dev/sdX
  ```

  Enter password when prompted.

  Open the encrypted drive with the name you want to use henceforth:

  ```sh
  sudo cryptsetup luksOpen /dev/sdX my_device_name
  ```

  Create filesystem:

  ```sh
  sudo mkfs.ext4 /dev/mapper/my_device_name
  ```

  Set mount point:

  ```sh
  sudo mkdir -p /media/my_device_name
  ```

  Mount:

  ```sh
  sudo mount /dev/mapper/my_device_name /media/my_device_name
  ```

  Add it to the crypttab:

  ```sh
  vim /etc/crypttab
  ```

  Get UUID for device with:

  ```
  sudo blkid /dev/sdx
  ```

  Add entry:

  ```txt
  my_device_name UUID="ae028027-20c5-40b6-9311-11f5a886f125" none noauto
  ```

  Then add it as a lint in the fstab for auto-mount:

  ```sh
  /dev/mapper/my_device_name /media/my_device_name ext4 defaults,nofail,noauto,user 0 2
  ```

  Give yourself full permissions (set once, then permanent):

  ```sh
  sudo chown thomas:thomas /meda/my_device_name
  sudo chmod 755 /media/my_device_name
  ```

  Make sure it shows up with its intended name:

  ```sh
  sudo e2label /dev/mapper/samsung-T3 samsung-T3
  ```

  (Where second value is the name we want it to show as.)

  Test auto-mount and loading from crypttab:

  First reload `systemd`:

  ```sh
  sudo systemctl daemon-reload
  ```

  Then, remount all disks.

  ```sh
  sudo mount -a
  ```

  Should be prompted for password and disk decrypts and mounts to specified mount
  point.

  ## Additional points

  Close LUKS (if you make a mistake):

  ```sh
  sudo cryptsetup luksClose device_name

  ```

  Remove vendor data to unlock more space:

  ```sh
  sudo tune2fs -m 0 /dev/mapper/samsung-T3
  ```

raw_content:
  ---
  tags: [procedural, disks, encryption]
  created: Monday, February 10, 2025
  ---

  # Setup an encrypted harddrive

  Firstly [create a new partition](./Creating_a_Linux_partition_table.md) on the
  unencrypted drive with `fdisk`.

  Create LUKs encryption:

  ```sh
  sudo cryptsetup luksFormat /dev/sdX
  ```

  Enter password when prompted.

  Open the encrypted drive with the name you want to use henceforth:

  ```sh
  sudo cryptsetup luksOpen /dev/sdX my_device_name
  ```

  Create filesystem:

  ```sh
  sudo mkfs.ext4 /dev/mapper/my_device_name
  ```

  Set mount point:

  ```sh
  sudo mkdir -p /media/my_device_name
  ```

  Mount:

  ```sh
  sudo mount /dev/mapper/my_device_name /media/my_device_name
  ```

  Add it to the crypttab:

  ```sh
  vim /etc/crypttab
  ```

  Get UUID for device with:

  ```
  sudo blkid /dev/sdx
  ```

  Add entry:

  ```txt
  my_device_name UUID="ae028027-20c5-40b6-9311-11f5a886f125" none noauto
  ```

  Then add it as a lint in the fstab for auto-mount:

  ```sh
  /dev/mapper/my_device_name /media/my_device_name ext4 defaults,nofail,noauto,user 0 2
  ```

  Give yourself full permissions (set once, then permanent):

  ```sh
  sudo chown thomas:thomas /meda/my_device_name
  sudo chmod 755 /media/my_device_name
  ```

  Make sure it shows up with its intended name:

  ```sh
  sudo e2label /dev/mapper/samsung-T3 samsung-T3
  ```

  (Where second value is the name we want it to show as.)

  Test auto-mount and loading from crypttab:

  First reload `systemd`:

  ```sh
  sudo systemctl daemon-reload
  ```

  Then, remount all disks.

  ```sh
  sudo mount -a
  ```

  Should be prompted for password and disk decrypts and mounts to specified mount
  point.

  ## Additional points

  Close LUKS (if you make a mistake):

  ```sh
  sudo cryptsetup luksClose device_name

  ```

  Remove vendor data to unlock more space:

  ```sh
  sudo tune2fs -m 0 /dev/mapper/samsung-T3
  ```


path: Disk_size_utilities.md
sortable_path: Disk_size_utilities.md
title: Disk size utilities
lead: ## Compare size of two directories in human-readable format
checksum: 6f798b9d6fa7cf7504b641f2898b711d22c1af378d7456371ba998871c3a408f
created: 2025-02-10 19:43:13.645798441+00:00
modified: 2025-02-10 20:44:43.44075998+00:00
metadata: {"created":"Monday, February 10, 2025","tags":["procedural","linux"]}
body:
  ## Compare size of two directories in human-readable format

  ```sh
  du -sh /dir_one /dir_two
  ```

  ## View total capacity and space used

  ```sh
  df -h /media/samsung-T7 # add more to compare if you like

  # Filesystem              Size  Used Avail Use% Mounted on
  # /dev/mapper/samsung-T7  916G   67G  850G   8% /media/samsung-T7

  ```

raw_content:
  ---
  tags: [procedural, linux]
  created: Monday, February 10, 2025
  ---

  # Disk size utilities

  ## Compare size of two directories in human-readable format

  ```sh
  du -sh /dir_one /dir_two
  ```

  ## View total capacity and space used

  ```sh
  df -h /media/samsung-T7 # add more to compare if you like

  # Filesystem              Size  Used Avail Use% Mounted on
  # /dev/mapper/samsung-T7  916G   67G  850G   8% /media/samsung-T7

  ```


path: Disable_non-root_ssh_access.md
sortable_path: Disable_non-root_ssh_access.md
title: Disable non-root SSH access
lead: Best practice is to:
checksum: 99d0d5102eabc289abb38aa04e83cabfab2a7acbe18832697766a8321e717852
created: 2025-02-15 18:23:30.822342517+00:00
modified: 2025-02-16 15:03:17.548955122+00:00
metadata: {"created":"Saturday, February 15, 2025","tags":["server-management","ssh","procedural","linux"]}
body:
  Best practice is to:

  - Disallow root login
  - Block password-based login
  - Allow only SSH-based login

  Do this by editing `/etc/ssh/sshd_config`:

  ```
  PasswordAuthentication no
  ChallengeResponseAuthentication no
  ```

  Then restart the SSH service:

  ```
  sudo systemctl restart sshd
  ```

  (Assumes you have already set-up SSH-based login for a non-root user!)

raw_content:
  ---
  tags: [server-management, ssh, procedural, linux]
  created: Saturday, February 15, 2025
  ---

  # Disable non-root SSH access

  Best practice is to:

  - Disallow root login
  - Block password-based login
  - Allow only SSH-based login

  Do this by editing `/etc/ssh/sshd_config`:

  ```
  PasswordAuthentication no
  ChallengeResponseAuthentication no
  ```

  Then restart the SSH service:

  ```
  sudo systemctl restart sshd
  ```

  (Assumes you have already set-up SSH-based login for a non-root user!)


path: UFW_firewall_management.md
sortable_path: UFW_firewall_management.md
title: UFW firewall management
checksum: 3ba2e1703b0c627d7d2828f6ab090e84d172376ed29a4e5d9ea2fc8573e5c649
created: 2025-02-16 15:00:32.757249884+00:00
modified: 2025-02-16 15:17:16.806900496+00:00
lead:
  `ufw` (Uncomplicated Firewall) simplifies the process of setting up firewalls on
  your server, without directly using `iptables`.

body:
  `ufw` (Uncomplicated Firewall) simplifies the process of setting up firewalls on
  your server, without directly using `iptables`.

  ```sh
  sudo apt install ufw
  ```

  First we deny all incoming requests as the default:

  ```sh
  ufw default deny incoming
  ```

  And allow all outgoing requests as the default:

  ```sh
  ufw default allow outgoing
  ```

  Next we need to allow certain requests based on their port:

  | Port | Service | Rule  |
  | ---- | ------- | ----- |
  | 22   | SSH     | allow |
  | 80   | HTTP    | allow |
  | 443  | HTTPS   | allow |

  ```sh
  sudo ufw allow 22/tcp
  sudo ufw allow 80/tcp
  sudo ufw allow 443/tcp
  ```

  Enable:

  ```sh
  ufw enable
  ```

  Verify rules:

  ```sh
  ufw status --verbose
  ```

  ```

  ```

raw_content:
  ---
  tags: [server-management, procedural, linux, firewalls]
  created: Sunday, February 16, 2025
  ---

  # UFW firewall management

  `ufw` (Uncomplicated Firewall) simplifies the process of setting up firewalls on
  your server, without directly using `iptables`.

  ```sh
  sudo apt install ufw
  ```

  First we deny all incoming requests as the default:

  ```sh
  ufw default deny incoming
  ```

  And allow all outgoing requests as the default:

  ```sh
  ufw default allow outgoing
  ```

  Next we need to allow certain requests based on their port:

  | Port | Service | Rule  |
  | ---- | ------- | ----- |
  | 22   | SSH     | allow |
  | 80   | HTTP    | allow |
  | 443  | HTTPS   | allow |

  ```sh
  sudo ufw allow 22/tcp
  sudo ufw allow 80/tcp
  sudo ufw allow 443/tcp
  ```

  Enable:

  ```sh
  ufw enable
  ```

  Verify rules:

  ```sh
  ufw status --verbose
  ```

  ```

  ```

metadata:
  {"created":"Sunday, February 16, 2025","tags":["server-management","procedural","linux","firewalls"]}


path: Check_if_value_null_in_Bash.md
sortable_path: Check_if_value_null_in_Bash.md
title: Check if value is null in Bash
lead:
  ```sh
  if [ -z $var ]; then
  # Value is null
  else
  # Value is not null
  fi
  ```

body:
  ```sh
  if [ -z $var ]; then
  # Value is null
  else
  # Value is not null
  fi
  ```

checksum: 5d91b8e1ee3fb3879a85ef3ec1f47b3e2bf7784b557dcb8e994f208c5765bc39
created: 2025-02-19 17:39:39.608947732+00:00
modified: 2025-02-22 10:03:09.759792142+00:00
metadata: {"created":"Wednesday, February 19, 2025","tags":["shell"]}
raw_content:
  ---
  tags: [shell]
  created: Wednesday, February 19, 2025
  ---

  # Check if value is null in Bash

  ```sh
  if [ -z $var ]; then
    # Value is null
  else
    # Value is not null
  fi
  ```


path: Associative_arrays_maps_in_Bash.md
sortable_path: Associative_arrays_maps_in_Bash.md
title: Associative arrays (lookup maps) in Bash
lead: ```sh
checksum: 7b90f376c8a02db6dd5db010baf77dc623396a94e0991d0006daa4b6c294cf33
created: 2025-02-19 17:55:15.069028167+00:00
modified: 2025-02-19 17:56:34.992942174+00:00
metadata: {"created":"Wednesday, February 19, 2025","tags":["shell"]}
body:
  ```sh

  declare -A CHANNEL_TO_WEBHOOK # Declare the map type

  # Specify key values
  CHANNEL_TO_WEBHOOK["test"]=$SLACK_WEBHOOK_TEST
  CHANNEL_TO_WEBHOOK["backups"]=$SLACK_WEBHOOK_BACKUPS
  CHANNEL_TO_WEBHOOK["eolas"]=$SLACK_WEBHOOK_EOLAS
  CHANNEL_TO_WEBHOOK["website"]=$SLACK_WEBHOOK_SYSTEMS_OBSCURE
  CHANNEL_TO_WEBHOOK["time-tracking"]=$SLACK_WEBHOOK_TIME_TRACKING

  # Invoke

  WEBHOOK=${CHANNEL_TO_WEBHOOK[$CHANNEL]}

  ```

raw_content:
  ---
  tags: [shell]
  created: Wednesday, February 19, 2025
  ---

  # Associative arrays (lookup maps) in Bash

  ```sh

  declare -A CHANNEL_TO_WEBHOOK # Declare the map type

  # Specify key values
  CHANNEL_TO_WEBHOOK["test"]=$SLACK_WEBHOOK_TEST
  CHANNEL_TO_WEBHOOK["backups"]=$SLACK_WEBHOOK_BACKUPS
  CHANNEL_TO_WEBHOOK["eolas"]=$SLACK_WEBHOOK_EOLAS
  CHANNEL_TO_WEBHOOK["website"]=$SLACK_WEBHOOK_SYSTEMS_OBSCURE
  CHANNEL_TO_WEBHOOK["time-tracking"]=$SLACK_WEBHOOK_TIME_TRACKING

  # Invoke

  WEBHOOK=${CHANNEL_TO_WEBHOOK[$CHANNEL]}

  ```


path: Capturing_program_error_in_Bash.md
sortable_path: Capturing_program_error_in_Bash.md
title: Capturing program error in Bash
lead:
  If you want to capture possible error immediately after running the command, use
  `!`:

checksum: 11fc1b98c4c7d34e395abe64e45451713fb88170480ee53fa1b40977e7f327d7
created: 2025-02-22 09:59:35.743392315+00:00
modified: 2025-02-22 11:28:30.496715716+00:00
metadata: {"created":"Saturday, February 22, 2025","tags":["shell"]}
body:
  If you want to capture possible error immediately after running the command, use
  `!`:

  ```sh
  if ! grep "pattern" file.txt; then
      echo "Pattern not found"
  fi
  ```

  This just checks the exit code. If you want the actual error too:

  ```sh
  if ! ERROR=$(grep "pattern" file.txt 2>&1); then
      echo "Pattern not found: $ERROR"
  fi
  ```

  If you want to store the outcome of a command and refer to it later, use `$?`.
  This built-in variable captures the exit status of the last executed command (0
  for success, non-zero for failure):

  ```sh
  ls -a
  status=$?  # Store immediately if needed later
  if [ $status -ne 0 ]; then
      # handle the error case
  else
      # command was successful (exit code = 0)
  fi
  ```

  Note: `$?` will be overwritten by any command that runs after the one you're
  interested in. If you need to check multiple commands' statuses, store $?
  immediately after each command:

  ```sh
  command1
  status1=$?
  command2
  status2=$?

  if [ $status1 -ne 0 ] || [ $status2 -ne 0 ]; then
      echo "One of the commands failed"
  fi
  ```

  Again, this just checks the exit code, to capture the error:

  ```sh
  error_msg=$(command 2>&1)
  status=$?
  if [ $status -ne 0 ]; then
      echo "Failed: $error_msg"
  fi
  ```

raw_content:
  ---
  tags: [shell]
  created: Saturday, February 22, 2025
  ---

  # Capturing program error in Bash

  If you want to capture possible error immediately after running the command, use
  `!`:

  ```sh
  if ! grep "pattern" file.txt; then
      echo "Pattern not found"
  fi
  ```

  This just checks the exit code. If you want the actual error too:

  ```sh
  if ! ERROR=$(grep "pattern" file.txt 2>&1); then
      echo "Pattern not found: $ERROR"
  fi
  ```

  If you want to store the outcome of a command and refer to it later, use `$?`.
  This built-in variable captures the exit status of the last executed command (0
  for success, non-zero for failure):

  ```sh
  ls -a
  status=$?  # Store immediately if needed later
  if [ $status -ne 0 ]; then
      # handle the error case
  else
      # command was successful (exit code = 0)
  fi
  ```

  Note: `$?` will be overwritten by any command that runs after the one you're
  interested in. If you need to check multiple commands' statuses, store $?
  immediately after each command:

  ```sh
  command1
  status1=$?
  command2
  status2=$?

  if [ $status1 -ne 0 ] || [ $status2 -ne 0 ]; then
      echo "One of the commands failed"
  fi
  ```

  Again, this just checks the exit code, to capture the error:

  ```sh
  error_msg=$(command 2>&1)
  status=$?
  if [ $status -ne 0 ]; then
      echo "Failed: $error_msg"
  fi
  ```


path: Network_tunnels.md
sortable_path: Network_tunnels.md
title: Network tunnels
checksum: 945df3a9470ef7167623f087f72fcd6c9a5ff8b17af60a89eafb3ff6211018bd
created: 2025-02-24 17:17:29.826117441+00:00
modified: 2025-02-24 18:04:48.885806292+00:00
metadata: {"created":"Monday, February 24, 2025","tags":["networks","VPN"]}
lead:
  A network tunnel is a
  [Transport Layer](./Transport_Layer_of_Internet_Protocol.md) mechanism whereby
  data packets can travel across a network they would otherwise not be able to
  access. Similarly how an underground physical tunnel can be use to travel across
  an inaccessible geographic region.

body:
  A network tunnel is a
  [Transport Layer](./Transport_Layer_of_Internet_Protocol.md) mechanism whereby
  data packets can travel across a network they would otherwise not be able to
  access. Similarly how an underground physical tunnel can be use to travel across
  an inaccessible geographic region.

  It is a broad concept of which there are various specific instances. However
  many involve creating a tunnel by wrapping a packet that cannot traverse a given
  network within the payload of a packet that can.

  For instance, an organisation may use [IPv6](./IP_addresses.md) addresses but
  want to send them over an IPv4 network. To achieve this they would enclose the
  packets within IPv4 packets.

  Similarly the scenario of a VPN. A user with a UK IP address wants to access a
  server in the US that has a firewall blocking non-US IPs. The user can bypass
  this restriction by connecting to a VPN running US servers. The UK packets are
  transferred under the aegis of the VPN's US IP.

  For more detail see
  [How tunnelling works with VPNs](./How_tunneling_works_with_VPNs.md).

raw_content:
  ---
  tags: [networks, VPN]
  created: Monday, February 24, 2025
  ---

  # Network tunnels

  A network tunnel is a
  [Transport Layer](./Transport_Layer_of_Internet_Protocol.md) mechanism whereby
  data packets can travel across a network they would otherwise not be able to
  access. Similarly how an underground physical tunnel can be use to travel across
  an inaccessible geographic region.

  It is a broad concept of which there are various specific instances. However
  many involve creating a tunnel by wrapping a packet that cannot traverse a given
  network within the payload of a packet that can.

  For instance, an organisation may use [IPv6](./IP_addresses.md) addresses but
  want to send them over an IPv4 network. To achieve this they would enclose the
  packets within IPv4 packets.

  Similarly the scenario of a VPN. A user with a UK IP address wants to access a
  server in the US that has a firewall blocking non-US IPs. The user can bypass
  this restriction by connecting to a VPN running US servers. The UK packets are
  transferred under the aegis of the VPN's US IP.

  For more detail see
  [How tunnelling works with VPNs](./How_tunneling_works_with_VPNs.md).


path: How_tunneling_works_with_VPNs.md
sortable_path: How_tunneling_works_with_VPNs.md
title: How tunnelling works with VPNs
checksum: 8edbd757700fba99b1345a687410ffe2df63fc3afd0e95052c89417e1feae7cd
created: 2025-02-24 17:46:00.628294938+00:00
modified: 2025-02-24 18:04:50.797199481+00:00
metadata: {"created":"Monday, February 24, 2025","tags":["networks","VPN"]}
lead:
  A VPN is a good example of a [network tunnel](./Network_tunnels.md) and the
  technique of packet wrapping / encapsulation to bypass network restrictions.

body:
  A VPN is a good example of a [network tunnel](./Network_tunnels.md) and the
  technique of packet wrapping / encapsulation to bypass network restrictions.

  When a user connects to a VPN, their device creates a connection to the VPN
  server. All subsequent traffic is routed first through the tunnel, rather than
  directly to the target host IP.

  When the data arrives at the VPN server, the VPN wraps the packets inside new
  packets that have the VPN server's addressing information. The nested packets
  are encrypted before they are encapsulated and the tunnel they arrive through is
  also encrypted end-to-end. Thus there is - in theory - no way to know where the
  original packets come from.

  When the data returns from the target host to the VPN server that requested it,
  the VPN server decrypts the data and forwards it back through the secure tunnel
  to the original user's device.

  Whilst connected to the VPN, the user's IP appears to be identical to the IP of
  the VPN server. This is because the outer packet header has the VPN's addressing
  information, not the source IP's (this is encrypted within the payload). As the
  target host only sees the VPN's headers, it appears as if the VPN user's IP is
  the same as the VPN's.

raw_content:
  ---
  tags: [networks, VPN]
  created: Monday, February 24, 2025
  ---

  # How tunnelling works with VPNs

  A VPN is a good example of a [network tunnel](./Network_tunnels.md) and the
  technique of packet wrapping / encapsulation to bypass network restrictions.

  When a user connects to a VPN, their device creates a connection to the VPN
  server. All subsequent traffic is routed first through the tunnel, rather than
  directly to the target host IP.

  When the data arrives at the VPN server, the VPN wraps the packets inside new
  packets that have the VPN server's addressing information. The nested packets
  are encrypted before they are encapsulated and the tunnel they arrive through is
  also encrypted end-to-end. Thus there is - in theory - no way to know where the
  original packets come from.

  When the data returns from the target host to the VPN server that requested it,
  the VPN server decrypts the data and forwards it back through the secure tunnel
  to the original user's device.

  Whilst connected to the VPN, the user's IP appears to be identical to the IP of
  the VPN server. This is because the outer packet header has the VPN's addressing
  information, not the source IP's (this is encrypted within the payload). As the
  target host only sees the VPN's headers, it appears as if the VPN user's IP is
  the same as the VPN's.


path: a0ab0bfb_network_layer_clarification.md
sortable_path: a0ab0bfb_network_layer_clarification.md
title: a0ab0bfb_network_layer_clarification
checksum: 6fe734cba1763497e75b107efeb4f7ae8e97f179505adc48e73803a77e7c5ed3
created: 2025-03-02 13:36:40.295981896+00:00
modified: 2025-03-02 15:03:34.679830885+00:00
metadata: {"created":"Sunday, March 02, 2025","tags":["networks","internet"]}
lead:
  I have difficulty understanding the places at which the different layers of the
  TCP/IP model of the Internet reside and their spatiality in terms of the overall
  system. These are some clarifying remarks.

body:
  I have difficulty understanding the places at which the different layers of the
  TCP/IP model of the Internet reside and their spatiality in terms of the overall
  system. These are some clarifying remarks.

  All layers exist on every device involved in the communication process. But not
  all layers are accessed by every device in the overall transaction.

  > While the entire encapsulated packet travels across the internet, different
  > devices interact with different layers of that packet based on their role in
  > the network.

  The server and client access data at all layers:

  - [Application](./Application_Layer_of_Internet_Protocol.md)
  - [Transport](./Transport_Layer_of_Internet_Protocol.md)
  - [Internet](./Internet_Layer_of_Internet_Protocol.md)
  - [Link](./Link_Layer_of_Internet_Protocol.md)

  This is because the data must be composed on the device, a route planned and
  then sent accross the network where it is processed at the local network layer.

  Routers however only care about routing the data, forwarding packets based on
  their IP addresses, they only process data up to the Internet Layer.

raw_content:
  ---
  tags: [networks, internet]
  created: Sunday, March 02, 2025
  ---

  # a0ab0bfb_network_layer_clarification

  I have difficulty understanding the places at which the different layers of the
  TCP/IP model of the Internet reside and their spatiality in terms of the overall
  system. These are some clarifying remarks.

  All layers exist on every device involved in the communication process. But not
  all layers are accessed by every device in the overall transaction.

  > While the entire encapsulated packet travels across the internet, different
  > devices interact with different layers of that packet based on their role in
  > the network.

  The server and client access data at all layers:

  - [Application](./Application_Layer_of_Internet_Protocol.md)
  - [Transport](./Transport_Layer_of_Internet_Protocol.md)
  - [Internet](./Internet_Layer_of_Internet_Protocol.md)
  - [Link](./Link_Layer_of_Internet_Protocol.md)

  This is because the data must be composed on the device, a route planned and
  then sent accross the network where it is processed at the local network layer.

  Routers however only care about routing the data, forwarding packets based on
  their IP addresses, they only process data up to the Internet Layer.


path: b01fd836_Transport_Layer_clarification.md
sortable_path: b01fd836_Transport_Layer_clarification.md
title: b01fd836_Transport_Layer_clarification
checksum: 7892a77009c267ad84909fb00fc355bbd66657c0b8573f6ee7ff633487c66751
created: 2025-03-02 14:04:30.423222319+00:00
modified: 2025-03-02 15:03:32.910984968+00:00
metadata: {"created":"Sunday, March 02, 2025","tags":["networks","internet"]}
lead:
  The Transport Layer is a _logical_ representation of the connection between two
  hosts on the Internet. It does not include the entire path (through various
  routers) from source to destination. This detail is assumed and is the concern
  of the Internet Layer - the next layer down.

body:
  The Transport Layer is a _logical_ representation of the connection between two
  hosts on the Internet. It does not include the entire path (through various
  routers) from source to destination. This detail is assumed and is the concern
  of the Internet Layer - the next layer down.

  When applications communicate accross via TCP they "perceive" a simple direct
  connection however this abstracts the fact that the data may:

  - be split into dozens of packets
  - travel through different physical pathways and different routers
  - face congestion, packet loss and network failure.

raw_content:
  ---
  tags: [networks, internet]
  created: Sunday, March 02, 2025
  ---

  # b01fd836_Transport_Layer_clarification

  The Transport Layer is a _logical_ representation of the connection between two
  hosts on the Internet. It does not include the entire path (through various
  routers) from source to destination. This detail is assumed and is the concern
  of the Internet Layer - the next layer down.

  When applications communicate accross via TCP they "perceive" a simple direct
  connection however this abstracts the fact that the data may:

  - be split into dozens of packets
  - travel through different physical pathways and different routers
  - face congestion, packet loss and network failure.


path: lsof.md
sortable_path: lsof.md
title: lsof
lead:
  `lsof` stands for _list open files_. It lists opened files and the processes
  using them.

checksum: da5c29f66229b34260b9bfbe95bdfc9821e7fac0f246e526f512e6660ca925cb
created: 2025-03-04 17:44:19.457824631+00:00
modified: 2025-03-04 18:19:37.915035321+00:00
metadata: {"created":"Tuesday, March 04, 2025","tags":["shell","procedural"]}
body:
  `lsof` stands for _list open files_. It lists opened files and the processes
  using them.

  ## Directory

  ```sh
  lsof /home/thomas
  ```

  ```
  COMMAND    PID   USER  FD   TYPE DEVICE SIZE/OFF    NODE NAME
  pipewire  1025 thomas cwd    DIR  253,0     4096 7340035 /home/thomas
  wireplumb 1026 thomas cwd    DIR  253,0     4096 7340035 /home/thomas
  ```

  ## Pass in a pid

  ```sh
  lsof -p 1175
  ```

  ```
  COMMAND    PID   USER  FD      TYPE             DEVICE SIZE/OFF    NODE NAME
  at-spi-bu 1175 thomas cwd       DIR              253,0     4096 7340035 /home/thomas
  at-spi-bu 1175 thomas rtd       DIR              259,2     4096       2 /
  at-spi-bu 1175 thomas txt       REG              259,2    26760 1225154 /usr/lib/at-spi-bus-launcher
  ```

  ## Networking

  ### List programs using or listening to ports

  ```sh
  lsof -i
  ```

  ```
  COMMAND    PID   USER  FD   TYPE DEVICE SIZE/OFF NODE NAME
  signal-de 3250 thomas  39u  IPv4 149863      0t0  TCP archlinux:44304->ac88393aca5853df7.awsglobalaccelerator.com:https (ESTABLISHED)
  slack     3602 thomas  18u  IPv4 138442      0t0  TCP archlinux:57950->ec2-18-169-55-40.eu-west-2.compute.amazonaws.com:https (ESTABLISHED)
  firefox   6177 thomas 147u  IPv4 138842      0t0  TCP archlinux:cresco-control->93.243.107.34.bc.googleusercontent.com:https (ESTABLISHED)
  ```

  ### Filter by specific port

  ```sh
  lsof -i:22
  lsof -i4:22 # limit to IPv4 addresses

  ```

  ```
  COMMAND    PID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
  sshd     45448   root    3u  IPv4  640237      0t0  TCP *:ssh (LISTEN)
  sshd     45448   root    4u  IPv6  640239      0t0  TCP *:ssh (LISTEN)
  sshd    185660   root    4u  IPv4 2554538      0t0  TCP static.18.113.203.116.clients.your-server.de:ssh->0542cb12.skybroadband.com:57744 (ESTABLISHED)

  ```

raw_content:
  ---
  tags: [shell, procedural]
  created: Tuesday, March 04, 2025
  ---

  # lsof

  `lsof` stands for _list open files_. It lists opened files and the processes
  using them.

  ## Directory

  ```sh
  lsof /home/thomas
  ```

  ```
  COMMAND    PID   USER  FD   TYPE DEVICE SIZE/OFF    NODE NAME
  pipewire  1025 thomas cwd    DIR  253,0     4096 7340035 /home/thomas
  wireplumb 1026 thomas cwd    DIR  253,0     4096 7340035 /home/thomas
  ```

  ## Pass in a pid

  ```sh
  lsof -p 1175
  ```

  ```
  COMMAND    PID   USER  FD      TYPE             DEVICE SIZE/OFF    NODE NAME
  at-spi-bu 1175 thomas cwd       DIR              253,0     4096 7340035 /home/thomas
  at-spi-bu 1175 thomas rtd       DIR              259,2     4096       2 /
  at-spi-bu 1175 thomas txt       REG              259,2    26760 1225154 /usr/lib/at-spi-bus-launcher
  ```

  ## Networking

  ### List programs using or listening to ports

  ```sh
  lsof -i
  ```

  ```
  COMMAND    PID   USER  FD   TYPE DEVICE SIZE/OFF NODE NAME
  signal-de 3250 thomas  39u  IPv4 149863      0t0  TCP archlinux:44304->ac88393aca5853df7.awsglobalaccelerator.com:https (ESTABLISHED)
  slack     3602 thomas  18u  IPv4 138442      0t0  TCP archlinux:57950->ec2-18-169-55-40.eu-west-2.compute.amazonaws.com:https (ESTABLISHED)
  firefox   6177 thomas 147u  IPv4 138842      0t0  TCP archlinux:cresco-control->93.243.107.34.bc.googleusercontent.com:https (ESTABLISHED)
  ```

  ### Filter by specific port

  ```sh
  lsof -i:22
  lsof -i4:22 # limit to IPv4 addresses

  ```

  ```
  COMMAND    PID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
  sshd     45448   root    3u  IPv4  640237      0t0  TCP *:ssh (LISTEN)
  sshd     45448   root    4u  IPv6  640239      0t0  TCP *:ssh (LISTEN)
  sshd    185660   root    4u  IPv4 2554538      0t0  TCP static.18.113.203.116.clients.your-server.de:ssh->0542cb12.skybroadband.com:57744 (ESTABLISHED)

  ```


path: SSH.md
sortable_path: SSH.md
title: SSH
lead: SSH is the de facto standard for remote access to a Unix machine.
checksum: fe8d7acb8084decad0be0b7c4303f2c4933595158d794dab7a8b3e9a4e4faed9
created: 2025-03-04 18:29:13.410802879+00:00
modified: 2025-03-04 19:03:24.485191272+00:00
metadata: {"created":"Tuesday, March 04, 2025","tags":["encryption","shell","server-management"]}
body:
  SSH is the de facto standard for remote access to a Unix machine.

  `ssh` is the client which you use to connect to another machine.

  `sshd` is the server that manages incoming client requests for access.

  ## sshd

  Typically the SSH server will be turned off.

  To run at boot:

  ```sh
  sudo systemctl enable sshd
  ```

  To start immediately:

  ```sh
  sudo systemctl start sshd
  ```

  The `sshd` configuration is found in the directory `/etc/ssh`.

  The config file is `/etc/ssh/sshd_config`.

  Mostly you can leave this alone but the following is a useful property to set:

  ```
  PermitRootLogin no
  ```

  See [Disable non-root ssh access](./Disable_non-root_ssh_access.md) for more.

  ## Known hosts

  Within your home directory at `./ssh/known_hosts` you will find a record of all
  the public keys of the servers you have connected. This file exists for both
  servers and clients, e.g:

  ```sh
  cat ./ssh/known_hosts
  # systemsobscure.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKn6vyv9b+Nk5r
  # YVSAk5KvsTiC24K6uSpzCHzgLNoqt2
  ```

  This shows the public key of my server at `systemsobscure.net` along with
  others.

  ## Authorized keys

  On servers only, there is also an `authorized_keys` file which shows the
  server's own public keys that it presents to clients.

  If I go to my server I see that this key matches the one I have on my client
  computer `known_hosts`:

  ```sh
  sudo cat /etc/ssh/ssh_host_ed25519_key.pub
  # ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKn6vyv9b+Nk5rYVSAk5KvsTiC24K6uSpzCHzgLNoqt2 root@self-host-server

  ```

  Each user on a server will also have a `~/.ssh` directory also containing an
  `authorized_keys` file. This contains the public keys of clients who are allowed
  to connect to that user account.

  Hence I see the same public key of my desktop client machine in both places.

  ## How the tunnel is created

raw_content:
  ---
  tags: [encryption, shell, server-management]
  created: Tuesday, March 04, 2025
  ---

  # SSH

  SSH is the de facto standard for remote access to a Unix machine.

  `ssh` is the client which you use to connect to another machine.

  `sshd` is the server that manages incoming client requests for access.

  ## sshd

  Typically the SSH server will be turned off.

  To run at boot:

  ```sh
  sudo systemctl enable sshd
  ```

  To start immediately:

  ```sh
  sudo systemctl start sshd
  ```

  The `sshd` configuration is found in the directory `/etc/ssh`.

  The config file is `/etc/ssh/sshd_config`.

  Mostly you can leave this alone but the following is a useful property to set:

  ```
  PermitRootLogin no
  ```

  See [Disable non-root ssh access](./Disable_non-root_ssh_access.md) for more.

  ## Known hosts

  Within your home directory at `./ssh/known_hosts` you will find a record of all
  the public keys of the servers you have connected. This file exists for both
  servers and clients, e.g:

  ```sh
  cat ./ssh/known_hosts
  # systemsobscure.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKn6vyv9b+Nk5r
  # YVSAk5KvsTiC24K6uSpzCHzgLNoqt2
  ```

  This shows the public key of my server at `systemsobscure.net` along with
  others.

  ## Authorized keys

  On servers only, there is also an `authorized_keys` file which shows the
  server's own public keys that it presents to clients.

  If I go to my server I see that this key matches the one I have on my client
  computer `known_hosts`:

  ```sh
  sudo cat /etc/ssh/ssh_host_ed25519_key.pub
  # ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKn6vyv9b+Nk5rYVSAk5KvsTiC24K6uSpzCHzgLNoqt2 root@self-host-server

  ```

  Each user on a server will also have a `~/.ssh` directory also containing an
  `authorized_keys` file. This contains the public keys of clients who are allowed
  to connect to that user account.

  Hence I see the same public key of my desktop client machine in both places.

  ## How the tunnel is created


path: USB-C.md
sortable_path: USB-C.md
title: USB-C
checksum: 9fd17c5cc79dec273849719ba1244026f36b2db8c7d5df229cd29e969721d067
created: 2025-03-12 18:03:27.850293965+00:00
modified: 2025-03-14 12:47:49.932554447+00:00
metadata: {"created":"Wednesday, March 12, 2025","tags":["hardware","usb"]}
lead:
  The "C" in the name refers to the shape of the _connector_ not the protocol that
  is used for the data transfer. (This is the case for all USB "types" which makes
  things confusing.)

body:
  The "C" in the name refers to the shape of the _connector_ not the protocol that
  is used for the data transfer. (This is the case for all USB "types" which makes
  things confusing.)

  Multiple protocols can use the same connector for input/output. For instance
  Thunderbolt can transfer over USB-C but not every port that can fit a USB-C is a
  Thunderbolt port. The device's internal hardware determines the protocols
  available.

  The USB-C connector can serve all of the following protocols (from slowest to
  fastest):

  - USB 1.0 (12mb/s)
  - USB 2.0 (480Mb/s)
  - USB 3.0 (5Gb/s)
  - USB 3.1 (10Gb/s)
  - USB 3.2 (20Gb/s)
  - USB 4 (40Gb/s)
  - Thunderbolt (40Gb/s)

  A USB-C connector is equipped to just charge a device as well as transfer data.

  The world is converging (often due to legislation) on USB-C as the global
  standard. It can be one connector for all existing protocols and future
  protocols (for about 15 years). Its physical design an form factor can support
  future advancements whilst using the same connector.

  Ultimately USB-C will be surpassed by more another yet more advanced connector
  and associated protocols but for the time being we can stop using the older
  models.

raw_content:
  ---
  tags: [hardware, usb]
  created: Wednesday, March 12, 2025
  ---

  # USB-C

  The "C" in the name refers to the shape of the _connector_ not the protocol that
  is used for the data transfer. (This is the case for all USB "types" which makes
  things confusing.)

  Multiple protocols can use the same connector for input/output. For instance
  Thunderbolt can transfer over USB-C but not every port that can fit a USB-C is a
  Thunderbolt port. The device's internal hardware determines the protocols
  available.

  The USB-C connector can serve all of the following protocols (from slowest to
  fastest):

  - USB 1.0 (12mb/s)
  - USB 2.0 (480Mb/s)
  - USB 3.0 (5Gb/s)
  - USB 3.1 (10Gb/s)
  - USB 3.2 (20Gb/s)
  - USB 4 (40Gb/s)
  - Thunderbolt (40Gb/s)

  A USB-C connector is equipped to just charge a device as well as transfer data.

  The world is converging (often due to legislation) on USB-C as the global
  standard. It can be one connector for all existing protocols and future
  protocols (for about 15 years). Its physical design an form factor can support
  future advancements whilst using the same connector.

  Ultimately USB-C will be surpassed by more another yet more advanced connector
  and associated protocols but for the time being we can stop using the older
  models.


path: USB protocols and connectors.md
sortable_path: USB protocols and connectors.md
title: USB protocols and connectors
checksum: 3ba7bc2c7eb1e116177febd1956dea73c7ee754dee929cde0a7fb473ec304f6d
created: 2025-03-12 18:14:05.472720924+00:00
modified: 2025-03-14 12:39:46.931001811+00:00
metadata: {"created":"Wednesday, March 12, 2025","tags":["protocols","USB"]}
lead:
  Different USB connectors can transfer according to different protocols. A
  connector is not always tied to a specific protocol. For instance each
  subsequent generation of connector is typically backwards-compatible with the
  protocol of the connector that it superseded.

body:
  Different USB connectors can transfer according to different protocols. A
  connector is not always tied to a specific protocol. For instance each
  subsequent generation of connector is typically backwards-compatible with the
  protocol of the connector that it superseded.

  Some connectors are limited to slower protocols but they have the example of
  being small which is a benefit for smaller devices. The USB-C connector is so
  good because it is both small and capable of supporting the most advanced
  protocols (USB-4 and Thunderbolt).

  The main types of USB connector:

  - USB-A/USB-B
  - USB-A/USB-B
  - Micro-USB ()
  - Mini-USB
  - USB-C

  The illustration below matches connector to the protocol supported:

  ![](../img/usb-illustration.png)

raw_content:
  ---
  tags: [protocols, USB]
  created: Wednesday, March 12, 2025
  ---

  # USB protocols and connectors

  Different USB connectors can transfer according to different protocols. A
  connector is not always tied to a specific protocol. For instance each
  subsequent generation of connector is typically backwards-compatible with the
  protocol of the connector that it superseded.

  Some connectors are limited to slower protocols but they have the example of
  being small which is a benefit for smaller devices. The USB-C connector is so
  good because it is both small and capable of supporting the most advanced
  protocols (USB-4 and Thunderbolt).

  The main types of USB connector:

  - USB-A/USB-B
  - USB-A/USB-B
  - Micro-USB ()
  - Mini-USB
  - USB-C

  The illustration below matches connector to the protocol supported:

  ![](../img/usb-illustration.png)


path: Using a context manager in Python.md
sortable_path: Using a context manager in Python.md
title: Using a context manager in Python
checksum: a58b2ac56703de9c1998f08af72bdce53d72146fa6809c27427143c7caa0ddac
created: 2025-03-16 13:19:42.465941843+00:00
modified: 2025-03-16 13:40:18.065864953+00:00
metadata: {"created":"Sunday, March 16, 2025","tags":["python"]}
lead:
  For certain process that have a start and an end state such as opening and
  closing a file or connecting and disconnecting from a database, instead of
  having dedicated open/close, connect/disconnect handling, you can use a Context
  Manager as a form of "syntactic sugar".

body:
  For certain process that have a start and an end state such as opening and
  closing a file or connecting and disconnecting from a database, instead of
  having dedicated open/close, connect/disconnect handling, you can use a Context
  Manager as a form of "syntactic sugar".

  By using a Context Manager you can use a more Pythonic construction of e.g:

  ```py
  with open('/file-path') as file:
      ... # handle file operation
  ```

  ```py
  with open DatabaseService() as connection:
      ... # do database stuff
  ```

  When you use this approach it is an abstraction over `try...finally`. Meaning
  the clean-up `finally` operation will run automatically without you having to
  explicitly handle it.

  Some common processes such as file handling have the Context Manager built-in
  and you don't have to explicitly provision it.

  Other processes may lend themselves to `with` syntax but you have to do the
  configuration yourself.

  One such example is creating a database accessor. Below I have done this by
  using `__enter__` and `__exit__` methods on a database class:

  ```py
  class DatabaseService:
      def __init__(self, db_name, db_path):
          self.db_name = db_name
          self.db_path = db_path
          self.connection: Optional[sqlite3.Connection] = None

      def connect(self) -> Optional[sqlite3.Connection]:
          if self.connection is not None:
              return self.connection

          try:
              if not os.path.exists(self.db_path):
                  os.makedirs(self.db_path)
                  print("INFO Created database directory")
              self.connection = sqlite3.connect(f"{self.db_path}/{self.db_name}.db")
              self.connection.execute("PRAGMA foreign_keys = ON")
              return self.connection

          except Exception as e:
              raise Exception(f"ERROR Problem connecting to database: {e}")

      def disconnect(self) -> None:
          try:
              if self.connection is not None:
                  self.connection.close()
                  self.connection = None
          except Exception as e:
              raise Exception(f"ERROR Problem disconnecting from database: {e}")

      def __enter__(self) -> sqlite3.Connection:
          connection = self.connect()
          if connection is None:
              raise RuntimeError("Failed to establish database connection")
          return connection

      def __exit__(self) -> None:
          self.disconnect()
  ```

  Then I can use it like so:

  ```py
  with DatabaseService() as connection:
      cursor = connection.cursor()
      cursor.execute(...)
  ```

raw_content:
  ---
  tags: [python]
  created: Sunday, March 16, 2025
  ---

  # Using a context manager in Python

  For certain process that have a start and an end state such as opening and
  closing a file or connecting and disconnecting from a database, instead of
  having dedicated open/close, connect/disconnect handling, you can use a Context
  Manager as a form of "syntactic sugar".

  By using a Context Manager you can use a more Pythonic construction of e.g:

  ```py
  with open('/file-path') as file:
      ... # handle file operation
  ```

  ```py
  with open DatabaseService() as connection:
      ... # do database stuff
  ```

  When you use this approach it is an abstraction over `try...finally`. Meaning
  the clean-up `finally` operation will run automatically without you having to
  explicitly handle it.

  Some common processes such as file handling have the Context Manager built-in
  and you don't have to explicitly provision it.

  Other processes may lend themselves to `with` syntax but you have to do the
  configuration yourself.

  One such example is creating a database accessor. Below I have done this by
  using `__enter__` and `__exit__` methods on a database class:

  ```py
  class DatabaseService:
      def __init__(self, db_name, db_path):
          self.db_name = db_name
          self.db_path = db_path
          self.connection: Optional[sqlite3.Connection] = None

      def connect(self) -> Optional[sqlite3.Connection]:
          if self.connection is not None:
              return self.connection

          try:
              if not os.path.exists(self.db_path):
                  os.makedirs(self.db_path)
                  print("INFO Created database directory")
              self.connection = sqlite3.connect(f"{self.db_path}/{self.db_name}.db")
              self.connection.execute("PRAGMA foreign_keys = ON")
              return self.connection

          except Exception as e:
              raise Exception(f"ERROR Problem connecting to database: {e}")

      def disconnect(self) -> None:
          try:
              if self.connection is not None:
                  self.connection.close()
                  self.connection = None
          except Exception as e:
              raise Exception(f"ERROR Problem disconnecting from database: {e}")

      def __enter__(self) -> sqlite3.Connection:
          connection = self.connect()
          if connection is None:
              raise RuntimeError("Failed to establish database connection")
          return connection

      def __exit__(self) -> None:
          self.disconnect()
  ```

  Then I can use it like so:

  ```py
  with DatabaseService() as connection:
      cursor = connection.cursor()
      cursor.execute(...)
  ```


path: Constructing paths in Python.md
sortable_path: Constructing paths in Python.md
title: Constructing paths in Python
lead: Rather than hard-coding paths it's better to use path construction operators.
checksum: da4bece8da8b45e6ec871105683cd04c9103a4b514199a9cd3ed2bce1a06727b
created: 2025-03-16 13:41:38.84967532+00:00
modified: 2025-03-16 13:59:56.186979064+00:00
metadata: {"created":"Sunday, March 16, 2025","tags":["python"]}
body:
  Rather than hard-coding paths it's better to use path construction operators.

  In the past this was done with `os.path` but a more modern library is `pathlib`.

  ```python
  import os
  import pathlib
  from pathlib import Path
  ```

  I'll show both, side by side.

  ## Get directory of the current script

  ```py
  SCRIPT_DIR = Path(__file__).parent.absolute()
  SCRIPT_DIR = os.path.dirname(os.path.abspath(__file))
  ```

  ## Get the user home directory

  ```py
  HOME_DIR = Path.home
  HOME_DIR = os.environ.get("HOME")
  ```

  ## Construct a path from $HOME

  ```py
  HOME_DIR = Path.home
  HOME_DIR = os.environ.get("HOME")

  SUB_DIR = HOME_DIR / "repos" / "systems-obscure" / "src"
  SUB_DIR = os.path.join(HOME_DIR, "repos", "systems-obscure", "src")
  ```

  ## Access root directories

  It's less straightforward to access directories that are not children of
  `$HOME`.

  For instance to access mounted drives:

  ```py
  EXT_HD = Path("/media/samsung-T3")
  EXT_HD_DIR = EXT_HD / "music"

  EXT_HD = "/media/samsung-T3"
  EXT_HD_DIR  = os.path.join(EXT_HD, "music")
  ```

raw_content:
  ---
  tags: [python]
  created: Sunday, March 16, 2025
  ---

  # Constructing paths in Python

  Rather than hard-coding paths it's better to use path construction operators.

  In the past this was done with `os.path` but a more modern library is `pathlib`.

  ```python
  import os
  import pathlib
  from pathlib import Path
  ```

  I'll show both, side by side.

  ## Get directory of the current script

  ```py
  SCRIPT_DIR = Path(__file__).parent.absolute()
  SCRIPT_DIR = os.path.dirname(os.path.abspath(__file))
  ```

  ## Get the user home directory

  ```py
  HOME_DIR = Path.home
  HOME_DIR = os.environ.get("HOME")
  ```

  ## Construct a path from $HOME

  ```py
  HOME_DIR = Path.home
  HOME_DIR = os.environ.get("HOME")

  SUB_DIR = HOME_DIR / "repos" / "systems-obscure" / "src"
  SUB_DIR = os.path.join(HOME_DIR, "repos", "systems-obscure", "src")
  ```

  ## Access root directories

  It's less straightforward to access directories that are not children of
  `$HOME`.

  For instance to access mounted drives:

  ```py
  EXT_HD = Path("/media/samsung-T3")
  EXT_HD_DIR = EXT_HD / "music"

  EXT_HD = "/media/samsung-T3"
  EXT_HD_DIR  = os.path.join(EXT_HD, "music")
  ```


path: Code_Craft_The_Practice_Of_Writing_Excellent_Code.md
sortable_path: Code_Craft_The_Practice_Of_Writing_Excellent_Code.md
title: Code Craft: The Practice Of Writing Excellent Code
lead: ## Clarity not brevity
checksum: a809f093468f99692d5b7296cd3a8a387db815d9029992db01abd690d28eb7fc
created: 2025-03-25 16:45:40.035843459+00:00
modified: 2025-03-26 17:50:37.477728627+00:00
metadata: {"created":"Tuesday, March 25, 2025","tags":["literature"]}
body:
  ## Clarity not brevity

  > Readability is the best single criterion of program quality: If a program is
  > easy to read, it is probably a good program; if it is hard to read, it
  > probably isn't good. -- _The Elements of Programming Style_ (Kernighan and
  > Plaugher, 1978)

  Whenever you can choose between concise (but potentially confusing) code and
  clear (but potentially tedious) code, use code that reads as intended, even if
  it’s less elegant.

  ## Initialise all variables at their points of declaration

  The uninitialized value may turn into a problem further down the line.

  ## Declare variables as late as possible

  Place the variable as close as possible to its use. This prevents it from
  confusing other parts of the code. It also clarifies the code using the
  variable. You don’t have to hunt around to find the variable’s type and
  initialization; a nearby declaration makes it obvious.

  ## Naming

  - Choose names from the perspective of an inexperienced reader, not from your
    internal, knowledgeable perspective.

  - If you can’t come up with a good name, then you might need to change your
    design. It’s an indication that something might be wrong.

  - Use clear names even if they are long. It doesn’t matter that a name is long
    if its meaning is unambiguous.

  - Avoid redundant words in names. Specifically, avoid these words in type names:
    _class_, _data_, _object_, and _type_.

  ## Self-documenting code

  This can be achieved with good naming alone if you are clever about it. It
  doesn't necessarily require automated tools or excessive commentary.

  ### Scope

  - With smaller, more specific scope we can name things in a more intuitive and
    expressive way. For example `Tree.countApples()`, `Tree.hasApples` rather than
    `Tree.countApplesInTree()`.

  ### Good presentation

  - Make the “normal” path through your code obvious. Error cases should not
    confuse the normal flow of execution. Your if-then-else constructs should be
    ordered _consistently_ (i.e., always place the “normal” case before the
    “error” case, or vice versa).

  - _One function, one action_. Make that your mantra.

  - Public information should come first, since this is what the class user needs
    to see. Put the **private implementation details at the end**, since they are
    less important to most readers.

  ### Comments

  - Explain _why_ not _how_. Your comments should not describe how the program
    works. You can see that by reading the code. The code is the definitive
    description of how the code works. You should instead focus on describing why
    something is written the way it is or what the next block of statements
    ultimately achieves.

raw_content:
  ---
  tags: [literature]
  created: Tuesday, March 25, 2025
  ---

  # _Code Craft: The Practice Of Writing Excellent Code_

  ## Clarity not brevity

  > Readability is the best single criterion of program quality: If a program is
  > easy to read, it is probably a good program; if it is hard to read, it
  > probably isn't good. -- _The Elements of Programming Style_ (Kernighan and
  > Plaugher, 1978)

  Whenever you can choose between concise (but potentially confusing) code and
  clear (but potentially tedious) code, use code that reads as intended, even if
  it’s less elegant.

  ## Initialise all variables at their points of declaration

  The uninitialized value may turn into a problem further down the line.

  ## Declare variables as late as possible

  Place the variable as close as possible to its use. This prevents it from
  confusing other parts of the code. It also clarifies the code using the
  variable. You don’t have to hunt around to find the variable’s type and
  initialization; a nearby declaration makes it obvious.

  ## Naming

  - Choose names from the perspective of an inexperienced reader, not from your
    internal, knowledgeable perspective.

  - If you can’t come up with a good name, then you might need to change your
    design. It’s an indication that something might be wrong.

  - Use clear names even if they are long. It doesn’t matter that a name is long
    if its meaning is unambiguous.

  - Avoid redundant words in names. Specifically, avoid these words in type names:
    _class_, _data_, _object_, and _type_.

  ## Self-documenting code

  This can be achieved with good naming alone if you are clever about it. It
  doesn't necessarily require automated tools or excessive commentary.

  ### Scope

  - With smaller, more specific scope we can name things in a more intuitive and
    expressive way. For example `Tree.countApples()`, `Tree.hasApples` rather than
    `Tree.countApplesInTree()`.

  ### Good presentation

  - Make the “normal” path through your code obvious. Error cases should not
    confuse the normal flow of execution. Your if-then-else constructs should be
    ordered _consistently_ (i.e., always place the “normal” case before the
    “error” case, or vice versa).

  - _One function, one action_. Make that your mantra.

  - Public information should come first, since this is what the class user needs
    to see. Put the **private implementation details at the end**, since they are
    less important to most readers.

  ### Comments

  - Explain _why_ not _how_. Your comments should not describe how the program
    works. You can see that by reading the code. The code is the definitive
    description of how the code works. You should instead focus on describing why
    something is written the way it is or what the next block of statements
    ultimately achieves.


path: c8820974_Tor_prioritising_uptake.md
sortable_path: c8820974_Tor_prioritising_uptake.md
title: c8820974_Tor_prioritising_uptake
lead:
  When formulating the initial architecture, the Tor engineers were faced with the
  following dilemma.

checksum: 489e3cb853474db12b6e0c16dd1d6d5a0557de4d8dcbc1640e23a4a7afd44cfd
created: 2025-03-26 18:37:52.043976724+00:00
modified: 2025-03-27 17:56:22.498213401+00:00
metadata: {"created":"Wednesday, March 26, 2025","tags":["Tor"]}
body:
  When formulating the initial architecture, the Tor engineers were faced with the
  following dilemma.

  - Greater security meant greater complexity for the user and therefore reduced
    adoption.

  - Less adoption means less security because there are less users on the network
    and thus individual nodes become easier to track and identify.

  Factors that would result in greater security include more cryptographic rigour
  or more hops. Also additional methods of obfuscation such as mixing and padding.

  Both could involve more overhead for the user, which introduces friction and the
  likelihood of continued use. More hops increases latency which makes everyday
  use less viable.

  Broad uptake and a rich ecosystem including users who were not the main or
  target beneficiaries (spies, totalitarian subjects) was necessary in order to
  obscure the activities of the target users.

  ## Source

  Ben Collier, _Tor: From the Dark Web to the Future of Privacy_ (2024)

raw_content:
  ---
  tags: [Tor]
  created: Wednesday, March 26, 2025
  ---

  # c8820974_Tor_prioritising_uptake

  When formulating the initial architecture, the Tor engineers were faced with the
  following dilemma.

  - Greater security meant greater complexity for the user and therefore reduced
    adoption.

  - Less adoption means less security because there are less users on the network
    and thus individual nodes become easier to track and identify.

  Factors that would result in greater security include more cryptographic rigour
  or more hops. Also additional methods of obfuscation such as mixing and padding.

  Both could involve more overhead for the user, which introduces friction and the
  likelihood of continued use. More hops increases latency which makes everyday
  use less viable.

  Broad uptake and a rich ecosystem including users who were not the main or
  target beneficiaries (spies, totalitarian subjects) was necessary in order to
  obscure the activities of the target users.

  ## Source

  Ben Collier, _Tor: From the Dark Web to the Future of Privacy_ (2024)


path: d212acdb_from_free-software_to_open_source.md
sortable_path: d212acdb_from_free-software_to_open_source.md
title: d212acdb_from_free-software_to_open_source
checksum: 0368b86ffeff2b8dcb00806f86aea6a1d910c7356d218a7f5af2dbac9eb89344
created: 2025-03-27 17:31:35.362366492+00:00
modified: 2025-03-27 17:56:23.695788135+00:00
metadata: {"created":"Thursday, March 27, 2025","tags":["history","ecopolsoc","FOSS"]}
lead:
  For the decade or so after Stallman's "Initial Announcement" of GNU in 1983,
  free software and the philosophy behind it remained the preserve of the
  technical underground. During these years, the phrase "open source" did not
  exist.

body:
  For the decade or so after Stallman's "Initial Announcement" of GNU in 1983,
  free software and the philosophy behind it remained the preserve of the
  technical underground. During these years, the phrase "open source" did not
  exist.

  Many engineers were using and contributing to free software in their free-time
  and surreptitiously using it in the corporate context but there was little
  awareness of it amongst the business class. If they did know about GNU-Linux
  they did not see it as a viable alternative to proprietary software, due to the
  lack of licensing, customer support and the misapprehension that the product
  would be inferior.

  The turning point came when
  [Netscape](./c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md), in a last ditch
  effort to resist Microsoft's obliteration of their market, made their
  source-code open. This caught the attention of the corporate class and the
  additional media attention led to a greater interest amongst capitalists
  generally.

  They quickly surmised that this could be an avenue of profit maximisation:
  free-labour, no licensing costs etc. As part of this process, free software was
  rebranded as "open source", all the better to calm investors' worries about its
  socialistic flavour having misread "free" as meaning "for no cost".

  Interestingly Microsoft publicly disparaged Linux and free software generally in
  public but an employee leaked internal memoranda which showed them to be most
  concerned. They led a vigorous PR campaign of disinformation about free
  software, tying it to the emerging phenomenon of "illegal file sharing". This
  was propitious timing, as it coincided with the passing of the DMCA (Digital
  Millennium Copyright Act) in the United States which introduced punitive
  punishment for those caught distributing proprietary media and software.

  So this is really the origin of hackers' antipathy to Microsoft and it's
  interesting to reflect on this today where Microsoft has very effectively
  subsumed "open source" into itself, realising it couldn't beat it on its own
  terms. Most notably with their acquisition of GitHub.

  ## Source

  Gabriella Coleman, _Coding Freedom: The Ethics and Aesthetics of Hacking_ (2013)

raw_content:
  ---
  tags: [history, ecopolsoc, FOSS]
  created: Thursday, March 27, 2025
  ---

  # d212acdb_from_free-software_to_open_source

  For the decade or so after Stallman's "Initial Announcement" of GNU in 1983,
  free software and the philosophy behind it remained the preserve of the
  technical underground. During these years, the phrase "open source" did not
  exist.

  Many engineers were using and contributing to free software in their free-time
  and surreptitiously using it in the corporate context but there was little
  awareness of it amongst the business class. If they did know about GNU-Linux
  they did not see it as a viable alternative to proprietary software, due to the
  lack of licensing, customer support and the misapprehension that the product
  would be inferior.

  The turning point came when
  [Netscape](./c301a0b3-1d8_Mosaic_Netscape_and_Browser_Wars.md), in a last ditch
  effort to resist Microsoft's obliteration of their market, made their
  source-code open. This caught the attention of the corporate class and the
  additional media attention led to a greater interest amongst capitalists
  generally.

  They quickly surmised that this could be an avenue of profit maximisation:
  free-labour, no licensing costs etc. As part of this process, free software was
  rebranded as "open source", all the better to calm investors' worries about its
  socialistic flavour having misread "free" as meaning "for no cost".

  Interestingly Microsoft publicly disparaged Linux and free software generally in
  public but an employee leaked internal memoranda which showed them to be most
  concerned. They led a vigorous PR campaign of disinformation about free
  software, tying it to the emerging phenomenon of "illegal file sharing". This
  was propitious timing, as it coincided with the passing of the DMCA (Digital
  Millennium Copyright Act) in the United States which introduced punitive
  punishment for those caught distributing proprietary media and software.

  So this is really the origin of hackers' antipathy to Microsoft and it's
  interesting to reflect on this today where Microsoft has very effectively
  subsumed "open source" into itself, realising it couldn't beat it on its own
  terms. Most notably with their acquisition of GitHub.

  ## Source

  Gabriella Coleman, _Coding Freedom: The Ethics and Aesthetics of Hacking_ (2013)


path: Docker_cleanup.md
sortable_path: Docker_cleanup.md
title: Docker cleanup
lead: ## Check Docker storage usage
checksum: 63a76b3fd6564f4e03433c5ec546e78b2bbedb80804057b37c05720979da3c81
created: 2025-03-27 18:46:39.914761229+00:00
modified: 2025-03-27 19:05:56.512295599+00:00
metadata: {"created":"Thursday, March 27, 2025","tags":["docker","procedural"]}
body:
  ## Check Docker storage usage

  ```sh
  sudo docker system df
  ```

  ## Unused volumes

  These are the biggest culprit when taking up server space.

  Use this command to clear them out:

  ```sh
  sudo docker volume rm $(sudo docker volume ls -q -f dangling=true)
  ```

  These are typically UUIDs and named anonymously which makes it hard to know what
  you are deleting.

  To avoid this, the best practice is to name the volumes in the Dockerfile and
  also label them to make it easier to tell what's doing what:

  ```yml
  volumes:
    nextcloud_data:
      name: nextcloud_data
      labels:
        app: nextcloud
        purpose: user_data

  services:
    nextcloud:
      volumes:
        - nextcloud_data:/var/www/html
  ```

  ### Locations to check (even after deleting volume)

  ```
  /var/lib/docker/image
  /var/lib/docker/containers
  /var/lib/docker/volumes
  /var/lib/docker/buildkit # build cache
  ```

  ## Unused images

  ### Identify

  ```sh
  sudo docker images --filter "dangling=tre"
  ```

  ### Delete

  ```sh

  sudo docker rmi mariadb:latest
  ```

raw_content:
  ---
  tags: [docker, procedural]
  created: Thursday, March 27, 2025
  ---

  # Docker cleanup

  ## Check Docker storage usage

  ```sh
  sudo docker system df
  ```

  ## Unused volumes

  These are the biggest culprit when taking up server space.

  Use this command to clear them out:

  ```sh
  sudo docker volume rm $(sudo docker volume ls -q -f dangling=true)
  ```

  These are typically UUIDs and named anonymously which makes it hard to know what
  you are deleting.

  To avoid this, the best practice is to name the volumes in the Dockerfile and
  also label them to make it easier to tell what's doing what:

  ```yml
  volumes:
    nextcloud_data:
      name: nextcloud_data
      labels:
        app: nextcloud
        purpose: user_data

  services:
    nextcloud:
      volumes:
        - nextcloud_data:/var/www/html
  ```

  ### Locations to check (even after deleting volume)

  ```
  /var/lib/docker/image
  /var/lib/docker/containers
  /var/lib/docker/volumes
  /var/lib/docker/buildkit # build cache
  ```

  ## Unused images

  ### Identify

  ```sh
  sudo docker images --filter "dangling=tre"
  ```

  ### Delete

  ```sh

  sudo docker rmi mariadb:latest
  ```


path: Certbot.md
sortable_path: Certbot.md
title: Certbot
lead: ## Generate certificates for new subdomain
checksum: f6205a0b6206b012e4708fb79229d69cc2cd799408b425f04f39630e32f705f4
created: 2025-04-02 15:39:48.849541319+00:00
modified: 2025-04-02 15:42:14.615041197+00:00
metadata: {"created":"Wednesday, April 02, 2025","tags":["server-management","procedural"]}
body:
  ## Generate certificates for new subdomain

  ```sh
  sudo certbot certonly --standalone -d rootdomain.net -d subdomainA.rootdomain.net -d subdomainB.rootdomain.net
  ```

  ## List certificates

  ```sh
  sudo certbot certificates
  ```

  ## Delete a certifcate

  ```sh
  sudo cerbot delete --certname rootdomain.net

  ```

raw_content:
  ---
  tags: [server-management, procedural]
  created: Wednesday, April 02, 2025
  ---

  # Certbot

  ## Generate certificates for new subdomain

  ```sh
  sudo certbot certonly --standalone -d rootdomain.net -d subdomainA.rootdomain.net -d subdomainB.rootdomain.net
  ```

  ## List certificates

  ```sh
  sudo certbot certificates
  ```

  ## Delete a certifcate

  ```sh
  sudo cerbot delete --certname rootdomain.net

  ```


