Differences

This shows you the differences between two versions of the page.

--- infrastructure:servers:chaosknoten [2023-07-29 18:12 UTC] – [Konfiguration] stb
+++ infrastructure:servers:chaosknoten [2026-02-24 13:44 UTC] (current) – fix netbox domain lilly
@@ Line 2: / Line 2: @@
 ---- dataentry server ----
-hostname:     chaosknoten
+hostname    : chaosknoten
-location:     IRZ42
+location    : IRZ42
-maintainers:  @@Maintainers@@
+maintainers : Infra-Team
-netbox_url:   https://netbox.ccchh.net/dcim/devices/40/
+netbox_url  : https://netbox.hamburg.ccc.de/dcim/devices/40/
 ----
 <WRAP column half>
@@ Line 33: / Line 34: @@
 ===== Gast-VMs =====
-Die VMs sind in Netbox dokumentiert: [[https://netbox.ccchh.net/virtualization/virtual-machines/?cluster_id=2|Virtual Machines Cluster=chaosknoten]]
+Die VMs sind in Netbox dokumentiert: [[https://netbox.hamburg.ccc.de/virtualization/virtual-machines/?cluster_id=2|Virtual Machines Cluster=chaosknoten]]
 ===== Firewall-Freischaltung =====
@@ Line 41: / Line 42: @@
   * stb: 213.240.180.39 und 2a01:170:118b::1
   * haegar: 136.243.3.21, 136.243.3.60, 2a01:4f8:211:1c94::2, 82.66.166.90
 ===== Netzwerkanbindung =====
@@ Line 56: / Line 58: @@
     * von hinten gesehen rechter Port
     * rotes Patchkabel zu Switchport 18
-  * IPMI-iDRAC-Interface:
-    * 1G Kupfer
-    * verbunden mit IRZ42-Netz VLAN 512 (aber bei uns ungetaggt)
-    * Switchport 17, aber kein Patchkabel gesteckt, weil (noch) keine Firewall
 ''eno3'' und ''eno4'' sind jeweils auf ''vmbr3'' und ''vmbr4'' gebridged.
@@ Line 65: / Line 63: @@
 ''vmbr4'' hat die Adresse ''212.12.48.126/24'' für SSH und Management-Webinterface von Proxmox.
-DIe VMs haben Adressen aus verschiedenen Netzen. Siehe [[https://netbox.ccchh.net/ipam/prefixes/?site_id=2|Netbox Prefixes IRZ42]]
+Die VMs haben Adressen aus verschiedenen Netzen. Siehe [[https://netbox.ccchh.net/ipam/prefixes/?site_id=2|Netbox Prefixes IRZ42]]
+==== IPMI-iDRAC-Interface ====
+  * 1G Kupfer
+  * verbunden mit ''ipmi-vpn.zs64.net'', ein EdgeRouter X mit OpenWrt, der 6 HE höher hängt (hinter lokschuppen und jachthafen)
+  * Wireguard-Config liegt in Pass unter ''noc/server/chaosknoten-ipmi/wireguard''
+  * Zugang zum Router liegt in Pass unter ''noc/server/zs64-ipmi/root''
+  * Hostname ''chaosknoten-ipmi.hamburg.ccc.de'' / 44.128.124.4
+==== Network Design ====
+Currently only some traffic is going through the old router VM (turing) and other VMs are routed by Wieske.
+We intend to move over to a setup where all traffic is going through a new router VM ([[https://netbox.hamburg.ccc.de/virtualization/virtual-machines/77/|router.hamburg.ccc.de]]).
+There are requirements for 3 main networks:
+  * v4-NAT: VMs without public IPv4 address behind the public reverse proxy
+  * public: VMs with public IPv4 address
+  * VMs without public interface at all, e.g. CI runners
+Internal networks are all running on one linux bridge interface ''vmbr0'' using VLANs to separate them.
+See [[https://netbox.hamburg.ccc.de/ipam/vlans/?group_id=3|NetBox VLANs IRZ42]] for available VLAN IDs and [[https://netbox.hamburg.ccc.de/virtualization/interfaces/80/|NetBox router net0]] for those assigned to ''vmbr0''/''net0'' of the router VM.
+Internal IPv4 addresses shall use ''10.32.0.0/16'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/43/|NetBox-Link]]).
+Each VLAN shall use it's own ''/24''-network with the third byte matching the VLAN ID, if private IPs are needed.
+The router will have additional network interfaces for the uplink bridge devices ''vmbr3'' ([[https://netbox.hamburg.ccc.de/virtualization/interfaces/81/|net1]]) and ''vmbr4'' ([[https://netbox.hamburg.ccc.de/virtualization/interfaces/83/|net2]]) which match the physical interfaces.
+=== How to add VMs ===
+To add a new VM:
+  * Generate new public IPv6 and add to Netbox, e.g. in [[https://netbox.hamburg.ccc.de/ipam/prefixes/47/ip-addresses/|v4-NAT network]]
+  * Add records to DNS server
+    * AAAA record on ''$hostname.hosts.hamburg.ccc.de''
+    * A and AAAA records for ''$service.hamburg.ccc.de'' as needed, e.g. via public-reverse-proxy
+  * Create VM in Proxmox and note MAC address
+  * Add VM in [[https://netbox.hamburg.ccc.de/virtualization/virtual-machines/?cluster_id=2|NetBox]]
+    * Create interface in ''net0'' with the MAC address from Proxmox and the suitable VLAN
+    * Configure the chosen IPv6 address in cloud-init, IPv4 shall be DHCP
+  * Setup firewall in Proxmox as needed
+  * When in v4-NAT network: Re-deploy public-reverse-proxy
+==== Legacy IPv4 Networks ====
+=== Public IPv4s ===
+We have the following public IPv4 subnets/ranges available:
+  * ''212.12.50.208/29'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/13/|NetBox-Link]])
+  * ''212.12.51.128/28'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/15/|NetBox-Link]])
+  * ''212.12.48.122-126/24'' ([[https://netbox.hamburg.ccc.de/ipam/ip-ranges/7/|NetBox-Link]]), part of ''212.12.48.0/24'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/12/|NetBox-Link]])
+=== Private IPv4s ===
+We use the following private IPv4 ranges:
+  * ''172.31.17.128/25'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/37/|NetBox-Link]])
+  * ''172.31.17.0/25'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/16/|NetBox-Link]])
+==== Legacy IPv6 Networks ====
+The new prefix for IPv6 connectivity is ''2a00:14b0:42:100::/56'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/46/|NetBox-Link]]).
+It is routed through our new router VM (which shall have ''2a00:14b0:4200:3500::130:2/112'') and the uplink router on ''2a00:14b0:4200:3500::130:1''.
+Please see the NetBox which sub-prefixes are used for what!
+Legacy: We have 2 IPv6-64-Prefixes, which map to corresponding IPv4-Prefixes/-Ranges.
+=== 2a00:14b0:4200:3000::/64 ===
+''2a00:14b0:4200:3000::/64'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/36/|NetBox-Link]])\\
+This subnet corresponds to the following IPv4-Subnets:
+  * ''212.12.48.0/24'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/12/|NetBox-Link]])
+To generate an IPv6 corresponding to an IPv4, we use the following convention: Take the last octet of the IPv4-address in decimal and use it for the first two bytes of the localpart, but with the digits as hex.\\
+So e.g.: ''212.12.48.125'' -> ''2a00:14b0:4200:3000:125::1''
+=== 2a00:14b0:f000:23::/64 ===
+''2a00:14b0:f000:23::/64'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/35/|NetBox-Link]])\\
+This subnet corresponds to the following IPv4-Subnets:
+  * ''212.12.50.208/29'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/13/|NetBox-Link]])
+  * ''212.12.51.128/28'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/15/|NetBox-Link]])
+To generate an IPv6 corresponding to an IPv4 from either ''212.12.50.208/29'' or ''212.12.51.128/28'', we use the following convention: Take the 3rd octet of the IPv4-address in decimal and use it for the first two bytes of the localpart, but with the digits in hex. Then take the last octet of the IPv4-address in decimal and use it for the third and fourth bytes of the localpart, but with the digits as hex.\\
+So e.g.:
+  * ''212.12.50.212'' -> ''2a00:14b0:f000:23:50:212:0:1''
+  * ''212.12.51.133'' -> ''2a00:14b0:f000:23:51:133:0:1''
+To generate an IPv6 corresponding to an IPv4 from ''172.31.17.0/25'', we use the following convention: Take the last octet of the IPv4-address in decimal and use it for the last two bytes of the localpart, but with the digits in hex.\\
+So e.g.:
+  * ''172.31.17.53'' -> ''2a00:14b0:f000:23::53''
 ===== Konfiguration =====
@@ Line 71: / Line 161: @@
 ===== Zugriff auf VMs =====
-SSH auf allen VMs läuft auf nicht-Standard-Ports, normalerweise 42666.
+SSH auf alten VMs läuft auf nicht-Standard-Ports, normalerweise 42666.
-Alle VMs, die eine RFC1918-Adresse haben, können über turing-router oder turing-main als Jumphost erreicht werden. Als Beispiel hier ein Snippet für ''.ssh/config''.
+VMs sollten per IPv6 direkt erreichbar sein.
+Falls Zugriff aus einem Netz ohne IPv6 erforderlich ist, kann die Router-VM als Jumphost verwendet werden.
+Alte VMs, die eine RFC1918-Adresse haben, können über turing-router oder turing-main als Jumphost erreicht werden. Als Beispiel hier ein Snippet für ''.ssh/config''.
 <code>
+Host ccchh-router
+    User chaos
+    Hostname router.hamburg.ccc.de
+Host *.host.hamburg.ccc.de
+    User chaos
+    ProxyJump ccchh-router
+## legacy
+Host ccchh-jumphost
+    User chaos
+    Hostname public-reverse-proxy.hamburg.ccc.de
+Host *-intern.hamburg.ccc.de
+    User chaos
+    ProxyJump ccchh-jumphost
 Host turing
     HostName turing.hamburg.ccc.de
     Port 4222
     User chaos
-    IdentitiesOnly yes
 Host turing-main
@@ Line 86: / Line 196: @@
     Port 42666
     User chaos
-    IdentitiesOnly yes
+</code>
-Host ns-intern
-    HostName ns-intern.hamburg.ccc.de
-    User chaos
-    IdentitiesOnly yes
-    ProxyJump turing
-Host rproxy-intern
+===== HOWTO Chaosknoten-Reboot =====
-    HostName rproxy-intern.hamburg.ccc.de
-    User chaos
+==== Vor dem Reboot ====
-    IdentitiesOnly yes
-    ProxyJump turing
+=== VMs hibernaten ===
-</code>
+Eine Reihe von VMs brauchen beim Booten ein Secret über die Konsole, z. B. für LUKS. Wenn man das nicht machen will, kann mann die betreffenden VMs in den Winterschlaf schicken. Wir haben alle VMs, für die das notwendig ist, mit dem Tag "luks" markiert.
+  # ./suspend-luks-vms.sh
+==== Nach dem Reboot ====
+Was nach einem reboot alles passieren muss, damit alle services wieder hochkommen.
+=== ZFS encrypted Dataset entsperren ===
+Key liegt im pass unter ''noc/server/chaosknoten/zfs/rust0-encrypted-passphrase''
+  zfs load-key rust0/encrypted
+===== Assigned Services =====
+---- datatable ----
+headers : Service, Service-URLs, Host-FQDN
+cols    : %pageid%, service-urls_urls, host-fqdn
+filter  : %class%=service
+and     : %pageid%!=infrastructure:services:template
+and     : server_page==infrastructure:servers:chaosknoten
+----