tailscaled's ipn package writes a collection of keys to state after
authenticating to control, but one at a time. If containerboot happens
to send a SIGTERM signal to tailscaled in the middle of writing those
keys, it may shut down with an inconsistent state Secret and never
recover. While we can't durably fix this with our current single-use
auth keys (no atomic operation to auth + write state), we can reduce
the window for this race condition by checking for partial state
before sending SIGTERM to tailscaled. Best effort only.
Updates #14080
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
Every so often, the ProxyGroup and other controllers lose an optimistic locking race
with other controllers that update the objects they create. Stop treating
this as an error event, and instead just log an info level log line for it.
Fixes#14072
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
This provides an interface for a user to force a preferred DERP outcome
for all future netchecks that will take precedence unless the forced
region is unreachable.
The option does not persist and will be lost when the daemon restarts.
Updates tailscale/corp#18997
Updates tailscale/corp#24755
Signed-off-by: James Tucker <james@tailscale.com>
cmd/containerboot,kube/kubetypes,cmd/k8s-operator: detect if Ingress is created in a tailnet that has no HTTPS
This attempts to make Kubernetes Operator L7 Ingress setup failures more explicit:
- the Ingress resource now only advertises HTTPS endpoint via status.ingress.loadBalancer.hostname when/if the proxy has succesfully loaded serve config
- the proxy attempts to catch cases where HTTPS is disabled for the tailnet and logs a warning
Updates tailscale/tailscale#12079
Updates tailscale/tailscale#10407
Signed-off-by: Irbe Krumina <irbe@tailscale.com>
cmd/k8s-operator/deploy/chart: allow reading OAuth creds from a CSI driver's volume and annotating operator's Service account
Updates #14264
Signed-off-by: Oliver Rahner <o.rahner@dke-data.com>
When the operator enables metrics on a proxy, it uses the port 9001,
and in the near future it will start using 9002 for the debug endpoint
as well. Make sure we don't choose ports from a range that includes
9001 so that we never clash. Setting TS_SOCKS5_SERVER, TS_HEALTHCHECK_ADDR_PORT,
TS_OUTBOUND_HTTP_PROXY_LISTEN, and PORT could also open arbitrary ports,
so we will need to document that users should not choose ports from the
10000-11000 range for those settings.
Updates #13406
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
* cmd/k8s-operator,k8s-operator,go.mod: optionally create ServiceMonitor
Adds a new spec.metrics.serviceMonitor field to ProxyClass.
If that's set to true (and metrics are enabled), the operator
will create a Prometheus ServiceMonitor for each proxy to which
the ProxyClass applies.
Additionally, create a metrics Service for each proxy that has
metrics enabled.
Updates tailscale/tailscale#11292
Signed-off-by: Irbe Krumina <irbe@tailscale.com>
We were previously relying on unintended behaviour by runc where
all containers where by default given read/write/mknod permissions
for tun devices.
This behaviour was removed in https://github.com/opencontainers/runc/pull/3468
and released in runc 1.2.
Containerd container runtime, used by Docker and majority of Kubernetes distributions
bumped runc to 1.2 in 1.7.24 https://github.com/containerd/containerd/releases/tag/v1.7.24
thus breaking our reference tun mode Tailscale Kubernetes manifests and Kubernetes
operator proxies.
This PR changes the all Kubernetes container configs that run Tailscale in tun mode
to privileged. This should not be a breaking change because all these containers would
run in a Pod that already has a privileged init container.
Updates tailscale/tailscale#14256
Updates tailscale/tailscale#10814
Signed-off-by: Irbe Krumina <irbe@tailscale.com>
This commit updates ServeConfig to allow configuration to Services (VIPServices for now) via Serve.
The scope of this commit is only adding the Services field to ServeConfig. The field doesn't actually
allow packet flowing yet. The purpose of this commit is to unblock other work on k8s end.
Updates #22953
Signed-off-by: KevinLiang10 <37811973+KevinLiang10@users.noreply.github.com>
* cmd/containerboot: serve health on local endpoint
We introduced stable (user) metrics in #14035, and `TS_LOCAL_ADDR_PORT`
with it. Rather than requiring users to specify a new addr/port
combination for each new local endpoint they want the container to
serve, this combines the health check endpoint onto the local addr/port
used by metrics if `TS_ENABLE_HEALTH_CHECK` is used instead of
`TS_HEALTHCHECK_ADDR_PORT`.
`TS_LOCAL_ADDR_PORT` now defaults to binding to all interfaces on 9002
so that it works more seamlessly and with less configuration in
environments other than Kubernetes, where the operator always overrides
the default anyway. In particular, listening on localhost would not be
accessible from outside the container, and many scripted container
environments do not know the IP address of the container before it's
started. Listening on all interfaces allows users to just set one env
var (`TS_ENABLE_METRICS` or `TS_ENABLE_HEALTH_CHECK`) to get a fully
functioning local endpoint they can query from outside the container.
Updates #14035, #12898
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
This commit adds a command to validate that all the metrics that
are registring in the client are also present in a path or url.
It is intended to be ran from the KB against the latest version of
tailscale.
Updates tailscale/corp#24066
Updates tailscale/corp#22075
Co-Authored-By: Brad Fitzpatrick <bradfitz@tailscale.com>
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Ensure that the ExternalName Service port names are always synced to the
ClusterIP Service, to fix a bug where if users created a Service with
a single unnamed port and later changed to 1+ named ports, the operator
attempted to apply an invalid multi-port Service with an unnamed port.
Also, fixes a small internal issue where not-yet Service status conditons
were lost on a spec update.
Updates tailscale/tailscale#10102
Signed-off-by: Irbe Krumina <irbe@tailscale.com>
this commit reduced the amount of data sent in the metrics
data integration test from 10MB to 1MB.
On various machines 10MB was quite flaky, while 1MB has not failed
once on 10000 runs.
Updates #13420
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Re-use a pre-allocated bytes.Buffer struct and
shallow the copy the result of bytes.NewBuffer into it
to avoid allocating the struct.
Note that we're only reusing the bytes.Buffer struct itself
and not the underling []byte temporarily stored within it.
Updates #cleanup
Updates tailscale/corp#18514
Updates golang/go#67004
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
In https://github.com/tailscale/tailscale/pull/13726 we added logic to
`checkExitNodePrefsLocked` to error out on platforms where using an
exit node is unsupported in order to give users more obvious feedback
than having this silently fail downstream.
The above change neglected to properly check whether the device in
question was actually trying to use an exit node when doing the check
and was incorrectly returning an error on any calls to
`checkExitNodePrefsLocked` on platforms where using an exit node is not
supported as a result.
This change remedies this by adding a check to see whether the device is
attempting to use an exit node before doing the `CanUseExitNode` check.
Updates https://github.com/tailscale/corp/issues/24835
Signed-off-by: Mario Minardi <mario@tailscale.com>
I was hoping we'd catch an example input quickly, but the reporter had
rebooted their machine and it is no longer exhibiting the behavior. As
such this code may be sticking around quite a bit longer and we might
encounter other errors, so include the panic in the log entry.
Updates #14201
Updates #14202
Updates golang/go#70528
Signed-off-by: James Tucker <james@tailscale.com>
We add a policy definition for the AllowedSuggestedExitNodes syspolicy setting, allowing admins
to configure a list of exit node IDs to be used as a pool for automatic suggested exit node selection.
We update definitions for policy settings configurable on both a per-user and per-machine basis,
such as UI customizations, to specify class="Both".
Lastly, we update the help text for existing policy definitions to include a link to the KB article
as the last line instead of in the first paragraph.
Updates #12687
Updates tailscale/corp#19681
Signed-off-by: Nick Khyl <nickk@tailscale.com>
In this PR, we update LocalBackend to rebuild the set of allowed suggested exit nodes whenever
the AllowedSuggestedExitNodes syspolicy setting changes. Additionally, we request a new suggested
exit node when this occurs, enabling its use if the ExitNodeID syspolicy setting is set to auto:any.
Updates #12687
Signed-off-by: Nick Khyl <nickk@tailscale.com>
This PR removes the sync.Once wrapper around retrieving the MachineCertificateSubject policy
setting value, ensuring the most recent version is always used if it changes after the service starts.
Although this policy setting is used by a very limited number of customers, recent support escalations have highlighted issues caused by outdated or incorrect policy values being applied.
Updates #12687
Signed-off-by: Nick Khyl <nickk@tailscale.com>
In this PR, we update ipnlocal.NewLocalBackend to subscribe to policy change notifications
and reapply syspolicy settings to the current profile's ipn.Prefs whenever a change occurs.
Updates #12687
Signed-off-by: Nick Khyl <nickk@tailscale.com>
This moves code that handles ExitNodeID/ExitNodeIP syspolicy settings
from (*LocalBackend).setExitNodeID to applySysPolicy.
Updates #12687
Signed-off-by: Nick Khyl <nickk@tailscale.com>
This updates the syspolicy.LogSCMInteractions check to run at the time of an interaction,
just before logging a message, instead of during service startup. This ensures the most
recent policy setting is used if it has changed since the service started.
Updates #12687
Signed-off-by: Nick Khyl <nickk@tailscale.com>
In this PR, we move the syspolicy.FlushDNSOnSessionUnlock check from service startup
to when a session change notification is received. This ensures that the most recent policy
setting value is used if it has changed since the service started.
We also plan to handle session change notifications for unrelated reasons
and need to decouple notification subscriptions from DNS anyway.
Updates #12687
Updates tailscale/corp#18342
Signed-off-by: Nick Khyl <nickk@tailscale.com>
These delays determine how soon syspolicy change callbacks are invoked after a policy setting is updated
in a policy source. For tests, we shorten these delays to minimize unnecessary wait times. This adjustment
only affects tests that subscribe to policy change notifications and modify policy settings after they have
already been set. Initial policy settings are always available immediately without delay.
Updates #12687
Signed-off-by: Nick Khyl <nickk@tailscale.com>
We have several places where LocalBackend instances are created for testing, but they are rarely shut down
when the tests that created them exit.
In this PR, we update newTestLocalBackend and similar functions to use testing.TB.Cleanup(lb.Shutdown)
to ensure LocalBackend instances are properly shut down during test cleanup.
Updates #12687
Signed-off-by: Nick Khyl <nickk@tailscale.com>
containerboot:
Adds 3 new environment variables for containerboot, `TS_LOCAL_ADDR_PORT` (default
`"${POD_IP}:9002"`), `TS_METRICS_ENABLED` (default `false`), and `TS_DEBUG_ADDR_PORT`
(default `""`), to configure metrics and debug endpoints. In a follow-up PR, the
health check endpoint will be updated to use the `TS_LOCAL_ADDR_PORT` if
`TS_HEALTHCHECK_ADDR_PORT` hasn't been set.
Users previously only had access to internal debug metrics (which are unstable
and not recommended) via passing the `--debug` flag to tailscaled, but can now
set `TS_METRICS_ENABLED=true` to expose the stable metrics documented at
https://tailscale.com/kb/1482/client-metrics at `/metrics` on the addr/port
specified by `TS_LOCAL_ADDR_PORT`.
Users can also now configure a debug endpoint more directly via the
`TS_DEBUG_ADDR_PORT` environment variable. This is not recommended for production
use, but exposes an internal set of debug metrics and pprof endpoints.
operator:
The `ProxyClass` CRD's `.spec.metrics.enable` field now enables serving the
stable user metrics documented at https://tailscale.com/kb/1482/client-metrics
at `/metrics` on the same "metrics" container port that debug metrics were
previously served on. To smooth the transition for anyone relying on the way the
operator previously consumed this field, we also _temporarily_ serve tailscaled's
internal debug metrics on the same `/debug/metrics` path as before, until 1.82.0
when debug metrics will be turned off by default even if `.spec.metrics.enable`
is set. At that point, anyone who wishes to continue using the internal debug
metrics (not recommended) will need to set the new `ProxyClass` field
`.spec.statefulSet.pod.tailscaleContainer.debug.enable`.
Users who wish to opt out of the transitional behaviour, where enabling
`.spec.metrics.enable` also enables debug metrics, can set
`.spec.statefulSet.pod.tailscaleContainer.debug.enable` to false (recommended).
Separately but related, the operator will no longer specify a host port for the
"metrics" container port definition. This caused scheduling conflicts when k8s
needs to schedule more than one proxy per node, and was not necessary for allowing
the pod's port to be exposed to prometheus scrapers.
Updates #11292
---------
Co-authored-by: Kristoffer Dalby <kristoffer@tailscale.com>
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
A small follow-up to #14112- ensures that the operator itself can emit
Events for its kube state store changes.
Updates tailscale/tailscale#14080
Signed-off-by: Irbe Krumina <irbe@tailscale.com>
https://go.dev/wiki/CodeReviewComments#useful-test-failures
(Previously it was using subtests with names including the input, but
once those went away, there was no context left)
Updates #14169
Change-Id: Ib217028183a3d001fe4aee58f2edb746b7b3aa88
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Otherwise we'll see a panic if we hit the dnsfallback code and try to
call NewDialer with a nil NetMon.
Updates #14161
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: I81c6e72376599b341cb58c37134c2a948b97cf5f
So we can locate them in logs more easily.
Updates tailscale/corp#24721
Change-Id: Ia766c75608050dde7edc99835979a6e9bb328df2
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Extracts tsaddr.IsTailscaleIPv4 out of tsaddr.IsTailscaleIP.
This will allow for checking valid Tailscale assigned IPv4 addresses
without checking IPv6 addresses.
Updates #14168
Updates tailscale/corp#24620
Signed-off-by: James Scott <jim@tailscale.com>
Tom Proctor
James Tucker
Brad Fitzpatrick
Irbe Krumina
Oliver Rahner
KevinLiang10
Kristoffer Dalby
Joe Tsai
Anton Tolchanov
Andrew Dunham
Nick Khyl
Mario Minardi
Andrea Gottardo
James Scott