docs/k8s: update docs (#11307)

Update docs for static Tailscale deployments on kube
to always use firewall mode autodection when in non-userspace.
Also add a note about running multiple replicas and a few suggestions how folks could do that.

Updates#cleanup

Signed-off-by: Irbe Krumina <irbe@tailscale.com>
Co-authored-by: Anton Tolchanov <1687799+knyar@users.noreply.github.com>

docs/k8s/README.md

@@ -1,6 +1,10 @@
 # Overview
 
-There are quite a few ways of running Tailscale inside a Kubernetes Cluster, some of the common ones are covered in this doc.
+There are quite a few ways of running Tailscale inside a Kubernetes Cluster.
+This doc covers creating and managing your own Tailscale node deployments in cluster.
+If you want a higher level of automation, easier configuration, automated cleanup of stopped Tailscale devices, or a mechanism for exposing the [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/) server to the tailnet, take a look at [Tailscale Kubernetes operator](https://tailscale.com/kb/1236/kubernetes-operator).
+
+:warning: Note that the manifests generated by the following commands are not intended for production use, and you will need to tweak them based on your environment and use case. For example, the commands to generate a standalone proxy manifest, will create a standalone `Pod`- this will not persist across cluster upgrades etc. :warning:
 
 ## Instructions
 
@@ -153,3 +157,74 @@ the entire Kubernetes cluster network (assuming NetworkPolicies allow) over Tail
    INTERNAL_PORT=8080
    curl http://$INTERNAL_IP:$INTERNAL_PORT
    ```
+
+## Multiple replicas
+
+Note that if you want to use the `Pod` manifests generated by the commands above in a multi-replica setup (i.e a multi-replica `StatefulSet`) you will need to change the mechanism for storing tailscale state to ensure that multiple replicas are not attemting to use a single Kubernetes `Secret` to store their individual states.
+
+To avoid proxy state clashes you could either store the state in memory or an `emptyDir` volume, or you could change the provided state `Secret` name to ensure that a unique name gets generated for each replica.
+
+### Option 1: storing in an `emptyDir`
+
+You can mount an [`emptyDir` volume](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) and configure the mount as the tailscale state store via `TS_STATE_DIR` env var. 
+You must also set `TS_KUBE_SECRET` to an empty string.
+
+An example:
+
+```yaml
+kind: StatefulSet
+metadata:
+  name: subnetrouter
+spec:
+  replicas: 2
+  ...
+  template:
+    ...
+    spec:
+      ...
+      volumes:
+      - name: tsstate
+        emptyDir: {}
+      containers:
+      - name: tailscale
+        env:
+        - name: TS_STATE_DIR
+          value: /tsstate
+        - name: TS_KUBE_SECRET
+          value: ""
+        volumeMounts:
+        - name: tsstate
+          mountPath: /tsstate
+```
+
+The downside of this approach is that the state will be lost when a `Pod` is
+deleted. In practice this means that when you, for example, upgrade proxy
+versions you will get a new set of Tailscale devices with different hostnames.
+
+### Option 2: dynamically generating unique `Secret` names
+
+If you run the proxy as a `StatefulSet`, the `Pod`s get [stable identifiers](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id).
+You can use that to pass an individual, static state `Secret` name to each proxy:
+
+```yaml
+kind: StatefulSet
+metadata:
+  name: subnetrouter
+spec:
+  replicas: 2
+  ...
+  template:
+    ...
+    spec:
+      ...
+       containers:
+      - name: tailscale
+        env:
+        - name: TS_KUBE_SECRET
+          valueFrom:
+            fieldRef:
+              apiVersion: v1
+              fieldPath: metadata.name
+```
+
+In this case, each replica will store its state in a `Secret` named the same as the `Pod` and as `Pod` names for a `StatefulSet` do not change if `Pod`s get recreated, proxy state will persist across cluster and proxy version updates etc.

docs/k8s/proxy.yaml

@@ -32,6 +32,8 @@ spec:
       value: "{{TS_KUBE_SECRET}}"
     - name: TS_USERSPACE
       value: "false"
+    - name: TS_DEBUG_FIREWALL_MODE
+      value: auto
     - name: TS_AUTHKEY
       valueFrom:
         secretKeyRef:

docs/k8s/sidecar.yaml

@@ -18,6 +18,8 @@ spec:
       value: "{{TS_KUBE_SECRET}}"
     - name: TS_USERSPACE
       value: "false"
+    - name: TS_DEBUG_FIREWALL_MODE
+      value: auto
     - name: TS_AUTHKEY
       valueFrom:
         secretKeyRef: