====== Chaosknoten ======
---- dataentry server ----
hostname : chaosknoten
location : IRZ42
maintainers : Infra-Team
netbox_url : https://netbox.hamburg.ccc.de/dcim/devices/40/
----
===== Hardware =====
* Dell PowerEdge R730
* 2x Xeon E5-2640 v4 (10c 20t)
* 512GB RAM (32x 16GB)
{{:infrastructure:servers:cropped_meow_on_dell_poweredge_photo_2023-07-17_22-30-15.jpg?nolink&200|}}
===== Zweck =====
Öffentliche Dienste des CCCHH und von Freunden
{{:infrastructure:chaosknoten_oben_worker_unten_vorne.jpeg?400|}}
{{:infrastructure:chaosknoten_oben_worker_unten_hinten.jpeg?400|}}
===== Gast-VMs =====
Die VMs sind in Netbox dokumentiert: [[https://netbox.ccchh.net/virtualization/virtual-machines/?cluster_id=2|Virtual Machines Cluster=chaosknoten]]
===== Firewall-Freischaltung =====
Der Zugriff auf das [[https://chaosknoten.hamburg.ccc.de:8006|Proxmox-Webinterface]] ist nur von ausgewählten IP-Adressen aus möglich:
* CCCHH-Z9: 185.161.129.132 und 2a07:c480:103:2::2
* stb: 213.240.180.39 und 2a01:170:118b::1
* haegar: 136.243.3.21, 136.243.3.60, 2a01:4f8:211:1c94::2, 82.66.166.90
===== Netzwerkanbindung =====
Chaosknoten hat vier physikalische Interfaces (plus IPMI):
* ''eno1'': SFP+-Slot, dz. nicht benutzt
* ''eno2'': SFP+-Slot, dz. nicht benutzt
* ''eno3'':
* 1G Kupfer
* verbunden mit IRZ42-Netz VLAN 48 (aber bei uns ungetaggt)
* von hinten gesehen linker Port
* grünes Patchkabel zu Switchport 15
* ''eno4'':
* 1G Kupfer
* verbunden mit IRZ42-Netz VLAN 512 (aber bei uns ungetaggt)
* von hinten gesehen rechter Port
* rotes Patchkabel zu Switchport 18
''eno3'' und ''eno4'' sind jeweils auf ''vmbr3'' und ''vmbr4'' gebridged.
''vmbr4'' hat die Adresse ''212.12.48.126/24'' für SSH und Management-Webinterface von Proxmox.
Die VMs haben Adressen aus verschiedenen Netzen. Siehe [[https://netbox.ccchh.net/ipam/prefixes/?site_id=2|Netbox Prefixes IRZ42]]
==== IPMI-iDRAC-Interface ====
* 1G Kupfer
* verbunden mit ''ipmi-vpn.zs64.net'', ein EdgeRouter X mit OpenWrt, der 6 HE höher hängt (hinter lokschuppen und jachthafen)
* Wireguard-Config liegt in Pass unter ''noc/server/chaosknoten-ipmi/wireguard''
* Zugang zum Router liegt in Pass unter ''noc/server/zs64-ipmi/root''
* Hostname ''chaosknoten-ipmi.hamburg.ccc.de'' / 44.128.124.4
==== Network Design ====
Currently only some traffic is going through the old router VM (turing) and other VMs are routed by Wieske.
We intend to move over to a setup where all traffic is going through a new router VM ([[https://netbox.hamburg.ccc.de/virtualization/virtual-machines/77/|router.hamburg.ccc.de]]).
There are requirements for 3 main networks:
* v4-NAT: VMs without public IPv4 address behind the public reverse proxy
* public: VMs with public IPv4 address
* VMs without public interface at all, e.g. CI runners
Internal networks are all running on one linux bridge interface ''vmbr0'' using VLANs to separate them.
See [[https://netbox.hamburg.ccc.de/ipam/vlans/?group_id=3|NetBox VLANs IRZ42]] for available VLAN IDs and [[https://netbox.hamburg.ccc.de/virtualization/interfaces/80/|NetBox router net0]] for those assigned to ''vmbr0''/''net0'' of the router VM.
Internal IPv4 addresses shall use ''10.32.0.0/16'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/43/|NetBox-Link]]).
Each VLAN shall use it's own ''/24''-network with the third byte matching the VLAN ID, if private IPs are needed.
The router will have additional network interfaces for the uplink bridge devices ''vmbr3'' ([[https://netbox.hamburg.ccc.de/virtualization/interfaces/81/|net1]]) and ''vmbr4'' ([[https://netbox.hamburg.ccc.de/virtualization/interfaces/83/|net2]]) which match the physical interfaces.
=== How to add VMs ===
To add a new VM:
* Generate new public IPv6 and add to Netbox, e.g. in [[https://netbox.hamburg.ccc.de/ipam/prefixes/47/ip-addresses/|v4-NAT network]]
* Add records to DNS server
* AAAA record on ''$hostname.hosts.hamburg.ccc.de''
* A and AAAA records for ''$service.hamburg.ccc.de'' as needed, e.g. via public-reverse-proxy
* Create VM in Proxmox and note MAC address
* Add VM in [[https://netbox.hamburg.ccc.de/virtualization/virtual-machines/?cluster_id=2|NetBox]]
* Create interface in ''net0'' with the MAC address from Proxmox and the suitable VLAN
* Configure the chosen IPv6 address in cloud-init, IPv4 shall be DHCP
* Setup firewall in Proxmox as needed
* When in v4-NAT network: Re-deploy public-reverse-proxy
==== Legacy IPv4 Networks ====
=== Public IPv4s ===
We have the following public IPv4 subnets/ranges available:
* ''212.12.50.208/29'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/13/|NetBox-Link]])
* ''212.12.51.128/28'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/15/|NetBox-Link]])
* ''212.12.48.122-126/24'' ([[https://netbox.hamburg.ccc.de/ipam/ip-ranges/7/|NetBox-Link]]), part of ''212.12.48.0/24'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/12/|NetBox-Link]])
=== Private IPv4s ===
We use the following private IPv4 ranges:
* ''172.31.17.128/25'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/37/|NetBox-Link]])
* ''172.31.17.0/25'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/16/|NetBox-Link]])
==== Legacy IPv6 Networks ====
The new prefix for IPv6 connectivity is ''2a00:14b0:42:100::/56'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/46/|NetBox-Link]]).
It is routed through our new router VM (which shall have ''2a00:14b0:4200:3500::130:2/112'') and the uplink router on ''2a00:14b0:4200:3500::130:1''.
Please see the NetBox which sub-prefixes are used for what!
Legacy: We have 2 IPv6-64-Prefixes, which map to corresponding IPv4-Prefixes/-Ranges.
=== 2a00:14b0:4200:3000::/64 ===
''2a00:14b0:4200:3000::/64'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/36/|NetBox-Link]])\\
This subnet corresponds to the following IPv4-Subnets:
* ''212.12.48.0/24'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/12/|NetBox-Link]])
To generate an IPv6 corresponding to an IPv4, we use the following convention: Take the last octet of the IPv4-address in decimal and use it for the first two bytes of the localpart, but with the digits as hex.\\
So e.g.: ''212.12.48.125'' -> ''2a00:14b0:4200:3000:125::1''
=== 2a00:14b0:f000:23::/64 ===
''2a00:14b0:f000:23::/64'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/35/|NetBox-Link]])\\
This subnet corresponds to the following IPv4-Subnets:
* ''212.12.50.208/29'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/13/|NetBox-Link]])
* ''212.12.51.128/28'' ([[https://netbox.hamburg.ccc.de/ipam/prefixes/15/|NetBox-Link]])
To generate an IPv6 corresponding to an IPv4 from either ''212.12.50.208/29'' or ''212.12.51.128/28'', we use the following convention: Take the 3rd octet of the IPv4-address in decimal and use it for the first two bytes of the localpart, but with the digits in hex. Then take the last octet of the IPv4-address in decimal and use it for the third and fourth bytes of the localpart, but with the digits as hex.\\
So e.g.:
* ''212.12.50.212'' -> ''2a00:14b0:f000:23:50:212:0:1''
* ''212.12.51.133'' -> ''2a00:14b0:f000:23:51:133:0:1''
To generate an IPv6 corresponding to an IPv4 from ''172.31.17.0/25'', we use the following convention: Take the last octet of the IPv4-address in decimal and use it for the last two bytes of the localpart, but with the digits in hex.\\
So e.g.:
* ''172.31.17.53'' -> ''2a00:14b0:f000:23::53''
===== Konfiguration =====
===== Zugriff auf VMs =====
SSH auf alten VMs läuft auf nicht-Standard-Ports, normalerweise 42666.
VMs sollten per IPv6 direkt erreichbar sein.
Falls Zugriff aus einem Netz ohne IPv6 erforderlich ist, kann die Router-VM als Jumphost verwendet werden.
Alte VMs, die eine RFC1918-Adresse haben, können über turing-router oder turing-main als Jumphost erreicht werden. Als Beispiel hier ein Snippet für ''.ssh/config''.
Host ccchh-router
User chaos
Hostname router.hamburg.ccc.de
Host *.host.hamburg.ccc.de
User chaos
ProxyJump ccchh-router
## legacy
Host ccchh-jumphost
User chaos
Hostname public-reverse-proxy.hamburg.ccc.de
Host *-intern.hamburg.ccc.de
User chaos
ProxyJump ccchh-jumphost
Host turing
HostName turing.hamburg.ccc.de
Port 4222
User chaos
Host turing-main
HostName turing.hamburg.ccc.de
Port 42666
User chaos
===== HOWTO Chaosknoten-Reboot =====
==== Vor dem Reboot ====
=== VMs hibernaten ===
Eine Reihe von VMs brauchen beim Booten ein Secret über die Konsole, z. B. für LUKS. Wenn man das nicht machen will, kann mann die betreffenden VMs in den Winterschlaf schicken. Wir haben alle VMs, für die das notwendig ist, mit dem Tag "luks" markiert.
# ./suspend-luks-vms.sh
==== Nach dem Reboot ====
Was nach einem reboot alles passieren muss, damit alle services wieder hochkommen.
=== ZFS encrypted Dataset entsperren ===
Key liegt im pass unter ''noc/server/chaosknoten/zfs/rust0-encrypted-passphrase''
zfs load-key rust0/encrypted
===== Assigned Services =====
---- datatable ----
headers : Service, Service-URLs, Host-FQDN
cols : %pageid%, service-urls_urls, host-fqdn
filter : %class%=service
and : %pageid%!=infrastructure:services:template
and : server_page==infrastructure:servers:chaosknoten
----