The RusagePrefixLog is rarely useful, hasn't been useful in a long
time, is rarely the measurement we need, and is pretty spammy (and
syscall-heavy). Disable it by default. We can enable it when we're
debugging memory.
At startup the client doesn't yet have the DERP map so can't do STUN
queries against DERP servers, so it only knows it local interface
addresses, not its STUN-mapped addresses.
We were reporting the interface-local addresses to control, getting
the DERP map, and then immediately reporting the full set of
updates. That was an extra HTTP request to control, but worse: it was
an extra broadcast from control out to all the peers in the network.
Now, skip the initial update if there are no stun results and we don't
have a DERP map.
More work remains optimizing start-up requests/map updates, but this
is a start.
Updates tailscale/corp#557
There was a bug with the lazy wireguard config code where, if the
minimum set of peers to tell wireguard didn't change, we skipped
calling userspaceEngine.updateActivityMapsLocked which updated
the various data structures that matched incoming traffic to later
reconfigure the minimum config.
That meant if an idle peer restarted and changed discovery keys, we
skipped updating our maps of disco keys/IPs that would caused us to
lazily inflate the config for that peer later if/when it did send
traffic.
Use golang.zx2c4.com/wireguard/windows/tunnel/winipcfg
instead of github.com/tailscale/winipcfg-go package.
Updates #760
Signed-off-by: Alex Brainman <alex.brainman@gmail.com>
Due to a copy/paste-o, we were monitoring address changes twice, and
not monitoring route changes at all.
Verified with 'tailscale debug --monitor' that this actually works now (while
running 'route add 10.3.0.0 mask 255.255.0.0 10.0.0.1' and 'route delete (same)'
back and forth in cmd.exe)
In practice route changes are accompanied by address changes and this
doesn't fix any known issues. I just noticed this while reading this
code again. But at least the code does what it was trying to do now.
This function is only called in fake mode, which won't do anything more
with the packet after we respond to it anyway, so dropping it in the
prefilter is not necessary. And it's kinda semantically wrong: we did
not reject it, so telling the upper layer that it was rejected produces
an ugly error message.
Signed-off-by: Avery Pennarun <apenwarr@tailscale.com>
At some point faketun got implemented as a loopback (put a packet in
from wireguard, the same packet goes back to wireguard) which is not
useful. It's supposed to be an interface that just sinks all packets,
and then wgengine adds *only* and ICMP Echo responder as a layer on
top.
This caused extremely odd bugs on darwin, where the special case that
reinjects packets from local->local was filling the loopback channel
and creating an infinite loop (which became jammed since the reader and
writer were in the same goroutine).
Signed-off-by: Avery Pennarun <apenwarr@tailscale.com>
If no interfaces are up, calm down and stop spamming so much. It was
noticed as especially bad on Windows, but probably was bad
everywhere. I just have the best network conditions testing on a
Windows VM.
Updates #604
Gracefully skips touching the v6 NAT table on systems that don't have
it, and doesn't configure IPv6 at all if IPv6 is globally disabled.
Signed-off-by: David Anderson <danderson@tailscale.com>
When the network link changes, existing UDP sockets fail immediately
and permanently on macOS.
The forwarder set up a single UDP conn and never changed it.
As a result, any time there was a network link change,
all forwarded DNS queries failed.
To fix this, create a new connection when send requests
fail because of network unreachability.
This change is darwin-only, although extended it to other platforms
should be straightforward.
Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
While we're here, parseQuery into a plain function.
This is helpful for fuzzing. (Which I did a bit of. Didn't find anything.)
And clean up a few minor things.
Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
Part of unforking our winipcfg-go and using upstream (#760), move our
additions into our repo. (We might upstream them later if upstream has
interest)
Originally these were:
@apenwarr: "Add ifc.SyncAddresses() and SyncRoutes()."
609dcf2df5
@bradfitz: "winipcfg: make Interface.AddRoutes do as much as possible, return combined error"
e9f93d53f3
@bradfitz: "prevent unnecessary Interface.SyncAddresses work; normalize IPNets in deltaNets"
decb9ee8e1
Might fix it. I've spent too much time failing to reproduce the issue. This doesn't
seem to make it worse, though (it still runs for me), so I'll include this and
see if it helps others while I still work on a reliable way to reproduce it.
Updates tailscale/corp#474
Otherwise when PAC server is down, we log, and each log entry is a new
HTTP request (from logtail) and a new GetProxyForURL call, which again
logs, non-stop. This is also nicer to the WinHTTP service.
Then also hook up link change notifications to the cache to reset it
if there's a chance the network might work sooner.
This change is to restore /etc/resolv.conf after tailscale down is called. This is done by setting the dns.Manager before errors occur. Error collection is also added.
Fixes#723
* wgengine/router/router_linux.go: Switched `cidrDiff("addr")` and `cidrDiff("route")` order
Signed-off-by: Christina Wen <christina@tailscale.com>
Co-authored-by: Christina Wen <christina@tailscale.com>
Also remove rebinding logic from the windows router. Magicsock will
instead rebind based on link change signals.
Signed-off-by: David Anderson <danderson@tailscale.com>
Updates #654. See that issue for a discussion of why
this timeout reduces flakiness, and what next steps are.
Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
For now. Get it working again so it's not stuck on 0.98.
Subnet relay can come later.
Updates #451
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Start of making the IPN state machine react to link changes and down
its DNS & routes if necessary to unblock proxy resolution (e.g. for
transitioning from public to corp networks where the corp network has
mandatory proxies and WPAD PAC files that can't be resolved while
using the DNS/routes configured previously)
This change should be a no-op. Just some callback plumbing.
It turns out that otherwise we don't know what exactly was set.
Also remove the now unused RootDomain config option.
Signed-off-by: Dmytro Shynkevych <dmytro@tailscale.com>
Previously, a type AAAA query would be answered with an A record
if only an IPv4 address was available. This is irrelevant for us
while we only use IPv4, but it will be a bug one day,
so it's worth being precise about semantics.
Signed-off-by: Dmytro Shynkevych <dmytro@tailscale.com>
Rather than consider bigs jumps in last-received-from activity as a
signal to possibly reconfigure the set of wireguard peers to have
configured, instead just track the set of peers that are currently
excluded from the configuration. Easier to reason about.
Also adds a bit more logging.
This might fix an error we saw on a machine running a recent unstable
build:
2020-08-26 17:54:11.528033751 +0000 UTC: 8.6M/92.6M magicsock: [unexpected] lazy endpoint not created for [UcppE], d:42a770f678357249
2020-08-26 17:54:13.691305296 +0000 UTC: 8.7M/92.6M magicsock: DERP packet received from idle peer [UcppE]; created=false
2020-08-26 17:54:13.691383687 +0000 UTC: 8.7M/92.6M magicsock: DERP packet from unknown key: [UcppE]
If it does happen again, though, we'll have more logs.
Seems to break linux CI builder. Cannot reproduce locally,
so attempting a rollback.
This reverts commit cd7bc02ab1.
Signed-off-by: David Crawshaw <crawshaw@tailscale.com>
Without this, a freshly started ipn client will be stuck in the
"Starting" state until something triggers a call to RequestStatus.
Usually a UI does this, but until then we can sit in this state
until poked by an external event, as is evidenced by our e2e tests
locking up when DERP is attached.
(This only recently became a problem when we enabled lazy handshaking
everywhere, otherwise the wireugard tunnel creation would also
trigger a RequestStatus.)
Signed-off-by: David Crawshaw <crawshaw@tailscale.com>
The previous approach modifies name in-place in the request slice to avoid an allocation.
This is incorrect: the question section of a DNS request
must be copied verbatim, without any such modification.
Software may rely on it (we rely on other resolvers doing it it in tsdns/forwarder).
Signed-off-by: Dmytro Shynkevych <dmytro@tailscale.com>
control/controlclient, wgengine/filter: extract parsePacketFilter to new constructor in wgengine/filter
Signed-off-by: chungdaniel <daniel@tailscale.com>
Consider:
Hard NAT (A) <---> Hard NAT w/ mapped port (B)
If A sends a packet to B's mapped port, A can disco ping B directly,
with low latency, without DERP.
But B couldn't establish a path back to A and needed to use DERP,
despite already logging about A's endpoint and adding a mapping to it
for other purposes (the wireguard conn.Endpoint lookup also needed
it).
This adds the tracking to discoEndpoint too so it'll be used for
finding a path back.
Fixestailscale/corp#556
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
For example:
$ tailscale ping -h
USAGE
ping <hostname-or-IP>
FLAGS
-c 10 max number of pings to send
-stop-once-direct true stop once a direct path is established
-verbose false verbose output
$ tailscale ping mon.ts.tailscale.com
pong from monitoring (100.88.178.64) via DERP(sfo) in 65ms
pong from monitoring (100.88.178.64) via DERP(sfo) in 252ms
pong from monitoring (100.88.178.64) via [2604:a880:2:d1::36:d001]:41641 in 33ms
Fixes#661
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
1) we weren't waking up a discoEndpoint that once existed and
went idle for 5 minutes and then got a disco message again.
2) userspaceEngine.noteReceiveActivity had a buggy check; fixed
and added a test
This removes the atomic bool that tried to track whether we needed to acquire
the lock on a future recursive call back into magicsock. Unfortunately that
hack doesn't work because we also had a lock ordering issue between magicsock
and userspaceEngine (see issue). This documents that too.
Fixes#644
If a node is behind a hard NAT and is using an explicit local port
number, assume they might've mapped a port and add their public IPv4
address with the local tailscaled's port number as a candidate endpoint.
A comparison operator was backwards.
The bad case went:
* device A send packet to B at t=1s
* B gets added to A's wireguard config
* B gets packet
(5 minutes pass)
* some other activity happens, causing B to expire
to be removed from A's network map, since it's
been over 5 minutes since sent or received activity
* device A sends packet to B at t=5m1s
* normally, B would get added back, but the old send
time was not zero (we sent earlier!) and the time
comparison was backwards, so we never regenerated
the wireguard config.
This also refactors the code for legibility and moves constants up
top, with comments.
The OS (tries) to send these but we drop them. No need to worry the
user with spam that we're dropping it.
Fixes#402
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Starting with fe68841dc7, some e2e tests
got flaky. Rather than debug them (they're gnarly), just revert to the old
behavior as far as those tests are concerned. The tests were somehow
using magicsock without a private key and expecting it to do ... something.
My goal with fe68841dc7 was to stop log spam
and unnecessary work I saw on the iOS app when when stopping the app.
Instead, only stop doing that work on any transition from
once-had-a-private-key to no-longer-have-a-private-key. That fixes
what I wanted to fix while still making the mysterious e2e tests
happy.
There is a race in natlab where we might start shutdown while natlab is still running
a goroutine or two to deliver packets. This adds a small grace period to try and receive
it before continuing shutdown.
Signed-off-by: David Anderson <danderson@tailscale.com>
The first packet to transit may take several seconds to do so, because
setup rates in wgengine may result in the initial WireGuard handshake
init to get dropped. So, we have to wait at least long enough for a
retransmit to correct the fault.
Signed-off-by: David Anderson <danderson@tailscale.com>
Active discovery lets us introspect the state of the network stack precisely
enough that it's unnecessary, and dropping the initial DERP packets greatly
slows down tests. Additionally, it's unrealistic since our production network
will never deliver _only_ discovery packets, it'll be all or nothing.
Signed-off-by: David Anderson <danderson@tailscale.com>
Uses natlab only, because the point of this active discovery test is going to be
that it should get through a lot of obstacles.
Signed-off-by: David Anderson <danderson@tailscale.com>
The deadlock was:
* Conn.Close was called, which acquired c.mu
* Then this goroutine scheduled:
if firstDerp {
startGate = c.derpStarted
go func() {
dc.Connect(ctx)
close(c.derpStarted)
}()
}
* The getRegion hook for that derphttp.Client then ran, which also
tries to acquire c.mu.
This change makes that hook first see if we're already in a closing
state and then it can pretend that region doesn't exist.
wireguard-go uses 3 goroutines per peer (with reasonably large stacks
& buffers).
Rather than tell wireguard-go about all our peers, only tell it about
peers we're actively communicating with. That means we need hooks into
magicsock's packet receiving path and tstun's packet sending path to
lazily create a wireguard peer on demand from the network map.
This frees up lots of memory for iOS (where we have almost nothing
left for larger domains with many users).
We should ideally do this in wireguard-go itself one day, but that'd
be a pretty big change.
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
sync.Pools should almost always be packate globals, even though in this
case we only have exactly 1 TUN device anyway, so it matters less.
Still, it's unusual to see a Pool that's not a package global, so move it.
We originally picked those numbers somewhat at random, but with the idea
that 8 is a traditionally lucky number in Chinese culture. Unfortunately,
"88" is also neo-nazi shorthand language.
Use 52 instead, because those are the digits above the letters
"TS" (tailscale) on a qwerty keyboard, so we're unlikely to collide with
other users. 5, 2 and 52 are also pleasantly culturally meaningless.
Signed-off-by: David Anderson <danderson@tailscale.com>
Nameserver IP 10.11.12.13 would otherwise get written to resolv.conf as 13.12.11.10, as was happening on my client.
Signed-off-by: Eduardo Kienetz <eduardo@kienetz.com>
Before this patch, the 250ms sleep would not be interrupted by context cancellation,
which would result in the goroutine sometimes lingering in tests (100ms grace period).
Signed-off-by: Dmytro Shynkevych <dmytro@tailscale.com>
Very rarely, cancellation occurs between a successful send on derpRecvCh
and a call to copyBuf on the receiving side.
Without this patch, this situation results in <-copyBuf blocking indefinitely.
Signed-off-by: Dmytro Shynkevych <dmytro@tailscale.com>
Peers advertising a discovery key know how to speak the discovery
protocol and do their own heartbeats to get through NATs and keep NATs
open. No need for the pinger except for with legacy peers.
The new interface lets implementors more precisely distinguish
local traffic from forwarded traffic, and applies different
forwarding logic within Machines for each type. This allows
Machines to be packet forwarders, which didn't quite work
with the implementation of Inject.
Signed-off-by: David Anderson <danderson@tailscale.com>
This log line looks buggy, even though lacking a filter is expected during bringup.
We already know if we forget to SetFilter: it breaks the magicsock test,
so no useful information is lost.
Resolves#559.
Signed-off-by: Dmytro Shynkevych <dmytro@tailscale.com>
The test demonstrates that magicsock can traverse two stateful
firewalls facing each other, that each require localhost to
initiate connections.
Signed-off-by: David Anderson <danderson@tailscale.com>
This change adds to tsdns the ability to delegate lookups to upstream nameservers.
This is crucial for setting Magic DNS as the system resolver.
Signed-off-by: Dmytro Shynkevych <dmytro@tailscale.com>
Never return "nil, nil" anymore. The caller expected a usable
interface now. I missed some of these earlier.
Also, handle address deletion now.
Updates #532
There's a lot of confusion around what tailscale status shows, so make it better:
show region names, last write time, and put stars around DERP too if active.
Now stars are always present if activity, and always somewhere.
* fix tailscale status for peers using discovery
* as part of that, pull out disco address selection into reusable
and testable discoEndpoint.addrForSendLocked
* truncate ping/pong logged hex txids in half to eliminate noise
* move a bunch of random time constants into named constants
with docs
* track a history of per-endpoint pong replies for future use &
status display
* add "send" and " got" prefix to discovery message logging
immediately before the frame type so it's easier to read than
searching for the "<-" or "->" arrows earlier in the line; but keep
those as the more reasily machine readable part for later.
Updates #483
Update the mapping from ip:port to discokey, so when we retrieve a
packet from the network, we can find the same conn.Endpoint that we
gave to wireguard-go previously, without making it think we've
roamed. (We did, but we're not using its roaming.)
Updates #483
Ping messages now go out somewhat regularly, pong replies are sent,
and pong replies are now partially handled enough to upgrade off DERP
to LAN.
CallMeMaybe packets are sent & received over DERP, but aren't yet
handled. That's next (and regular maintenance timers), and then WAN
should work.
Updates #483
Starting at yesterday's e96f22e560 (convering some UDPAddrs to
IPPorts), Conn.ReceiveIPv4 could return a nil addr, which would make
its way through wireguard-go and blow up later. The DERP read path
wasn't initializing the addr result parameter any more, and wgRecvAddr
wasn't checking it either.
Fixes#515
And while plumbing, a bit of discovery work I'll need: the
endpointOfAddr map to map from validated paths to the discoEndpoint.
Not being populated yet.
Updates #483
This adds a new magicsock endpoint type only used when both sides
support discovery (that is, are advertising a discovery
key). Otherwise the old code is used.
So far the new code only communicates over DERP as proof that the new
code paths are wired up. None of the actually discovery messaging is
implemented yet.
Support for discovery (generating and advertising a key) are still
behind an environment variable for now.
Updates #483
The new deepprint package just walks a Go data structure and writes to
an io.Writer. It's not pretty like go-spew, etc.
We then use it to replace the use of UAPI (which we have a TODO to
remove) to generate signatures of data structures to detect whether
anything changed (without retaining the old copy).
This was necessary because the UAPI conversion ends up trying to do
DNS lookups which an upcoming change depends on not happening.
And track known peers.
Doesn't yet do anything with the messages. (nor does it send any yet)
Start of docs on the message format. More will come in subsequent changes.
Updates #483
Brad Fitzpatrick
David Anderson
Alex Brainman
Avery Pennarun
Josh Bleecher Snyder
Christina Wen
Josh Bleecher Snyder
Dmytro Shynkevych
halulu
David Crawshaw
Disconnect3d
chungdaniel
Mike Kramlich
Eduardo Kienetz
Wendi Yu