There is a good chance this is actually caused by the uplink stalling or throttling long enough for the pf queues to fill up and start dropping packets; the queues take a long time to drain once the connection resumes full throughput

So far, I suspect it's a PF issue.

  • It's happened on two different machines with different hardware (xl(4) versus rl(4))
  • It usually manifests more frequently the longer the machine has been running
  • It only happens with PF under moderate (DSL) network load
  • It happens regardless of what secondary rules (if any) are loaded
  • It only affects packets that are not part of an established session (ICMP, UDP, TCP SYN)
  • It manifests as a huge delay, and when it stops happening there's a burst of activity
64 bytes from 66.219.31.21: icmp_seq=40050 ttl=55 time=42.371 ms
64 bytes from 66.219.31.21: icmp_seq=40051 ttl=55 time=56.848 ms
64 bytes from 66.219.31.21: icmp_seq=40052 ttl=55 time=57.587 ms
64 bytes from 66.219.31.21: icmp_seq=40053 ttl=55 time=43.788 ms
64 bytes from 66.219.31.21: icmp_seq=40054 ttl=55 time=75.965 ms
64 bytes from 66.219.31.21: icmp_seq=40055 ttl=55 time=67.637 ms
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
ping: sendto: No buffer space available
64 bytes from 66.219.31.21: icmp_seq=40056 ttl=55 time=22934.168 ms
64 bytes from 66.219.31.21: icmp_seq=40057 ttl=55 time=21959.228 ms
64 bytes from 66.219.31.21: icmp_seq=40079 ttl=55 time=45.744 ms
64 bytes from 66.219.31.21: icmp_seq=40080 ttl=55 time=48.900 ms
64 bytes from 66.219.31.21: icmp_seq=40081 ttl=55 time=33.343 ms
64 bytes from 66.219.31.21: icmp_seq=40082 ttl=55 time=35.842 ms
64 bytes from 66.219.31.21: icmp_seq=40083 ttl=55 time=32.758 ms
  • Bouncing the interface (ifconfig down / ifconfig up) does not help

  • Bouncing PF (pfctl -d / pfctl -e) does not work

    • When the firewall is disabled, packets flow fine, but without NAT
    • When the firewall is reenabled, symptoms return immediately
  • Reloading the ruleset and queue descriptors (pfctl -f /etc/pf/base.pf) instantly resolves the issue
  • Commenting out the altq queue definition appears to eliminate the problem, but the network latency is crap