* [Cake] [PATCH net-next v12 3/4] net/sched: cap the accounted backlog before it can wrap
2026-11-21 19:21 [Cake] [PATCH net-next v12 0/4] net/sched: cap the accounted backlog before it can wrap Jamal Hadi Salim
2026-11-21 19:21 ` [Cake] [PATCH net-next v12 1/4] net/sched: cake: return the classifier result to the caller Jamal Hadi Salim
2026-11-21 19:21 ` [Cake] [PATCH net-next v12 2/4] net/sched: cake: use a dedicated split-GSO list iterator Jamal Hadi Salim
@ 2026-11-21 19:21 ` Jamal Hadi Salim
2026-11-21 19:21 ` [Cake] [PATCH net-next v12 4/4] net/sched: cake: hash before segmenting and reject an overlong split Jamal Hadi Salim
3 siblings, 0 replies; 5+ messages in thread
From: Jamal Hadi Salim @ 2026-11-21 19:21 UTC (permalink / raw)
To: netdev
Cc: Jamal Hadi Salim, Jiri Pirko, David S . Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Simon Horman, Stephen Hemminger,
Vinicius Costa Gomes, Victor Nogueira,
Toke Høiland-Jørgensen, Björn Töpel, cake,
Sashiko
This is a followup of commit 709f34f7c28d ("net/sched: fq: add overflow
bounds to quantum and initial quantum") as reported by Sashiko.
The issue: fq_codel, cake, codel, pie, fq_pie, dualpi2, RED, pfifo,
plug and (non-WRED) gred account every enqueued packet's stab-adjusted
length into the 32-bit sch->qstats.backlog. A crafted tc size table
inflates qdisc_pkt_len() up to QDISC_PKT_LEN_MAX (1 MiB), and a
supported BIG TCP aggregate can account more than that through the
headers qdisc_pkt_len_segs_init() replicates per segment, so a few
thousand packets wrap the counter mod 2^32.
The AQM then reads a small backlog and makes the wrong drop decision,
and the dequeue-side subtractions leave it corrupt (for non-WRED gred
it is the reported stats and the dequeue-side subtractions that go
wrong instead). In fq_codel a wrapped per-flow backlog can also make
fq_codel_drop()'s scan pick an empty flow, dereferencing a NULL
flow->head.
The Fix: drop at enqueue once the stored backlog cannot represent the
incoming packet's accounted length.
Ordinary packets (qdisc_pkt_len() at most QDISC_PKT_LEN_MAX) are
covered by a standing ceiling: drop once the backlog is within
QDISC_MAX_BACKLOG (U32_MAX - QDISC_PKT_LEN_MAX) of the wrap point, the
largest backlog one more 1 MiB packet cannot wrap, so there is no
per-enqueue 64-bit add.
A larger wire estimate is checked against the space that actually
remains (len > U32_MAX - backlog): qdisc_pkt_len_segs_init() adds the
headers replicated on every output segment, so a supported small-MSS
BIG TCP aggregate can carry a wire estimate above 1 MiB while its raw
length stays within GSO_MAX_SIZE, and such traffic must pass on an
empty queue.
qdisc_pkt_len() itself stays truthful for every input: it is not
clamped and the initializer keeps succeeding, so BPF
__sk_buff.wire_len, act_police and the HTB/TBF shapers read the real
length, and sch_handle_ingress() -- which ignores the initializer's
return value and reads pkt_segs for packet statistics -- keeps
complete metadata.
Cover sch_plug, pfifo, non-WRED sch_gred, the leaves fq, sfq, hhf,
skbprio, netem, choke, etf and taprio, the classful parents prio, htb,
drr, ets, qfq, hfsc, cbs and sfb, and tbf's direct enqueue path as well,
since each adds the packet's own length to a 32-bit backlog. In RED,
account the packet before handing it to the child, bound the child
backlog too, and re-derive the aggregate from pre-reservation snapshots
on failure.
The guard bounds each qdisc's own enqueue path: a child that segments a
GSO skb (tbf, netem) can still grow a parent's aggregate through
qdisc_tree_reduce_backlog()'s unclamped byte delta, which is left to a
separate change.
CAKE's split-GSO branch is the other exception; 4/4 sums and checks those
segments before linking any of them.
Conditions to recreate the bug:
ip tuntap add tun0 mode tun
ip link set tun0 txqueuelen 32 up
ip addr add 10.99.0.1/24 dev tun0
tc qdisc add dev tun0 root handle 1: stab overhead 2000000000 \
fq_codel flows 1 limit 20000 ecn
# hold the tun fd open without reading so the backlog persists, then
# send 6000 packets: the counter wraps and the unfixed kernel reports
# a wrapped backlog, while the fixed one drops at the ceiling.
Reported-by: Sashiko (nipa) <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/netdev/20260818101130.16203-1-jhs@mojatatu.com/
Link: https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20260818101130.16203-1-jhs@mojatatu.com
Link: https://lore.kernel.org/netdev/CANn89iLfMJV7ancKH1Gjzzm7ZUG-gKcrczkJjJEWN5sCuWd-ug@mail.gmail.com/
Link: https://lore.kernel.org/netdev/CANn89i+GOFH_8g+vSV0jUj0aBqLYVWuFOrHuDggZu9kPC_m6tQ@mail.gmail.com/
Link: https://lore.kernel.org/netdev/87a4oottcj.fsf@toke.dk/
Link: https://lore.kernel.org/netdev/87pkxks68f.fsf@toke.dk/
Link: https://lore.kernel.org/netdev/179231932946.3135.7497478990186385688@kernel.org/
Link: https://lore.kernel.org/netdev/179269577131.3135.3967915390403138015@kernel.org/
Signed-off-by: Jamal Hadi Salim <jhs@mojatatu.com>
---
v11 -> v12:
- taprio: read the child's backlog and queue length before the handoff
and mirror the net change afterwards, instead of booking the packet when
the child returns NET_XMIT_SUCCESS. Some children free the skb and still
report success (tbf_segment(), cake's split path), so the v11 code read
the packet after it was gone; and because taprio_graft() sets
TCQ_F_NOPARENT, a child that evicts on NET_XMIT_CN never reaches taprio
through qdisc_tree_reduce_backlog(), so booking only on success let the
dequeue subtraction run ahead of the booking and the counter wrapped.
The segmented path reads the segment length before the handoff and no
longer counts a segment drop twice.
(Sashiko nipa and gemini reviews of v11)
- RED: book the packet on success again, as the tree did before v11. The
v11 delta arithmetic re-derived the aggregate from the snapshots and so
dropped any offload-driver update that landed while the child ran, and it
passed that same amount on to the ancestors. With the child and the
netem duplicate propagating their own changes, the plain accounting is
correct and cannot wrap.
(Sashiko nipa review of v11)
- qdisc_qstats_backlog_dec() stops at zero. An offload driver updates
sch->qstats.backlog from the dump path while holding only RTNL, so a
datapath increment can be lost; the matching decrement then wrapped the
counter and the new enqueue guard turned the wrap into a permanent drop.
(Sashiko nipa review of v11)
- leave room for two full-size packets in the standing ceiling, because
netem books a duplicate before it re-enters the tree and one enqueue can
then add twice the packet length.
- pfifo's backlog drops report QDISC_DROP_OVERLIMIT and count an
overlimit, like the other guards added here.
(Sashiko nipa review of v11)
- drop the Reviewed-by; the taprio and RED accounting it covered was
replaced in this version.
---
include/net/pkt_sched.h | 1 -
include/net/sch_generic.h | 33 +++++++++++++++++++++++++++++++-
net/sched/sch_cake.c | 7 +++++++
net/sched/sch_cbs.c | 6 ++++++
net/sched/sch_choke.c | 6 ++++++
net/sched/sch_codel.c | 16 +++++++++-------
net/sched/sch_drr.c | 6 ++++++
net/sched/sch_dualpi2.c | 1 +
net/sched/sch_etf.c | 6 ++++++
net/sched/sch_ets.c | 6 ++++++
net/sched/sch_fifo.c | 12 ++++++++++++
net/sched/sch_fq.c | 6 ++++++
net/sched/sch_fq_codel.c | 9 +++++++++
net/sched/sch_fq_pie.c | 3 ++-
net/sched/sch_gred.c | 9 +++++++++
net/sched/sch_hfsc.c | 6 ++++++
net/sched/sch_hhf.c | 18 ++++++++++++++++--
net/sched/sch_htb.c | 14 +++++++++++---
net/sched/sch_netem.c | 18 ++++++++++++++++++
net/sched/sch_pie.c | 3 ++-
net/sched/sch_plug.c | 3 ++-
net/sched/sch_prio.c | 6 ++++++
net/sched/sch_qfq.c | 6 ++++++
net/sched/sch_red.c | 10 ++++++++++
net/sched/sch_sfb.c | 6 ++++++
net/sched/sch_sfq.c | 6 ++++++
net/sched/sch_skbprio.c | 17 +++++++++++++++--
net/sched/sch_taprio.c | 40 ++++++++++++++++++++++++++++++---------
net/sched/sch_tbf.c | 6 ++++++
29 files changed, 258 insertions(+), 28 deletions(-)
diff --git a/include/net/pkt_sched.h b/include/net/pkt_sched.h
index 90d3e7943b19..18a419cd9d94 100644
--- a/include/net/pkt_sched.h
+++ b/include/net/pkt_sched.h
@@ -12,7 +12,6 @@
#define DEFAULT_TX_QUEUE_LEN 1000
#define STAB_SIZE_LOG_MAX 30
-#define QDISC_PKT_LEN_MAX (1 << 20) /* 1 MiB */
struct qdisc_walker {
int stop;
diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
index 1acaadb3e2cf..bff07ce8c130 100644
--- a/include/net/sch_generic.h
+++ b/include/net/sch_generic.h
@@ -915,6 +915,29 @@ static inline unsigned int qdisc_pkt_len(const struct sk_buff *skb)
return qdisc_skb_cb(skb)->pkt_len;
}
+#define QDISC_PKT_LEN_MAX (1 << 20) /* 1 MiB */
+
+/* Largest backlog that still has room for a 1 MiB packet and a netem
+ * duplicate of it, so that a single enqueue can never run out of
+ * headroom before it is accounted.
+ */
+#define QDISC_MAX_BACKLOG (U32_MAX - 2 * QDISC_PKT_LEN_MAX)
+
+/* Would this packet's accounted length wrap sch->qstats.backlog?
+ * Packets up to QDISC_PKT_LEN_MAX fit below QDISC_MAX_BACKLOG; GSO
+ * can make the length larger, so compare against the real headroom.
+ */
+static inline bool qdisc_backlog_overflow(const struct Qdisc *sch,
+ const struct sk_buff *skb)
+{
+ u32 backlog = READ_ONCE(sch->qstats.backlog);
+ u32 len = qdisc_pkt_len(skb);
+
+ if (likely(len <= QDISC_PKT_LEN_MAX))
+ return backlog > QDISC_MAX_BACKLOG;
+ return (u64)backlog + len + QDISC_PKT_LEN_MAX > U32_MAX;
+}
+
static inline unsigned int qdisc_pkt_segs(const struct sk_buff *skb)
{
u32 pkt_segs = qdisc_skb_cb(skb)->pkt_segs;
@@ -997,7 +1020,15 @@ static inline void qstats_backlog_sub(struct Qdisc *sch, u32 val)
static inline void qdisc_qstats_backlog_dec(struct Qdisc *sch,
const struct sk_buff *skb)
{
- qstats_backlog_sub(sch, qdisc_pkt_len(skb));
+ u32 backlog = READ_ONCE(sch->qstats.backlog);
+ u32 len = qdisc_pkt_len(skb);
+
+ /* An offload driver can update the counter from the dump path
+ * while the packet is in flight, so a lost increment can leave
+ * the backlog below the length being removed. Do not let that
+ * wrap: the enqueue guard would then drop every packet for good.
+ */
+ WRITE_ONCE(sch->qstats.backlog, backlog > len ? backlog - len : 0);
}
static inline void qdisc_qstats_cpu_backlog_dec(struct Qdisc *sch,
diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
index 88ff3e4f6754..803ead499ae9 100644
--- a/net/sched/sch_cake.c
+++ b/net/sched/sch_cake.c
@@ -1781,6 +1781,13 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
return ret;
}
+ /* Drop before touching the hash or shaper state. */
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
/* choose flow to insert into */
b = cake_select_tin(sch, skb);
idx = cake_hash(b, skb, q->config->flow_mode, flow_override,
diff --git a/net/sched/sch_cbs.c b/net/sched/sch_cbs.c
index 1c93469c56e3..3c018ef70e23 100644
--- a/net/sched/sch_cbs.c
+++ b/net/sched/sch_cbs.c
@@ -92,6 +92,12 @@ static int cbs_child_enqueue(struct sk_buff *skb, struct Qdisc *sch,
unsigned int len = qdisc_pkt_len(skb);
int err;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
err = child->ops->enqueue(skb, child, to_free);
if (err != NET_XMIT_SUCCESS)
return err;
diff --git a/net/sched/sch_choke.c b/net/sched/sch_choke.c
index 73d3e673dc7b..9ce02273b624 100644
--- a/net/sched/sch_choke.c
+++ b/net/sched/sch_choke.c
@@ -215,6 +215,12 @@ static int choke_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct choke_sched_data *q = qdisc_priv(sch);
const struct red_parms *p = &q->parms;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
choke_skb_cb(skb)->keys_valid = 0;
/* Compute average queue usage (see RED) */
q->vars.qavg = red_calc_qavg(p, &q->vars, sch->q.qlen);
diff --git a/net/sched/sch_codel.c b/net/sched/sch_codel.c
index a1ba2e355f2d..78c5d72d6406 100644
--- a/net/sched/sch_codel.c
+++ b/net/sched/sch_codel.c
@@ -116,15 +116,17 @@ static struct sk_buff *codel_peek(struct Qdisc *sch)
static int codel_qdisc_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct sk_buff **to_free)
{
- struct codel_sched_data *q;
+ struct codel_sched_data *q = qdisc_priv(sch);
- if (likely(qdisc_qlen(sch) < sch->limit)) {
- codel_set_enqueue_time(skb);
- return qdisc_enqueue_tail(skb, sch);
+ if (unlikely(qdisc_qlen(sch) >= sch->limit) ||
+ unlikely(qdisc_backlog_overflow(sch, skb))) {
+ WRITE_ONCE(q->drop_overlimit, q->drop_overlimit + 1);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
}
- q = qdisc_priv(sch);
- WRITE_ONCE(q->drop_overlimit, q->drop_overlimit + 1);
- return qdisc_drop_reason(skb, sch, to_free, QDISC_DROP_OVERLIMIT);
+
+ codel_set_enqueue_time(skb);
+ return qdisc_enqueue_tail(skb, sch);
}
static const struct nla_policy codel_policy[TCA_CODEL_MAX + 1] = {
diff --git a/net/sched/sch_drr.c b/net/sched/sch_drr.c
index 8621d057edd9..eeef304f76a7 100644
--- a/net/sched/sch_drr.c
+++ b/net/sched/sch_drr.c
@@ -342,6 +342,12 @@ static int drr_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct drr_class *cl;
int err = 0;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
cl = drr_classify(skb, sch, &err);
if (cl == NULL) {
if (err & __NET_XMIT_BYPASS)
diff --git a/net/sched/sch_dualpi2.c b/net/sched/sch_dualpi2.c
index 4947def7c49e..7331f0ec1c8f 100644
--- a/net/sched/sch_dualpi2.c
+++ b/net/sched/sch_dualpi2.c
@@ -392,6 +392,7 @@ static int dualpi2_enqueue_skb(struct sk_buff *skb, struct Qdisc *sch,
struct dualpi2_skb_cb *cb;
if (unlikely(qdisc_qlen(sch) >= sch->limit) ||
+ unlikely(qdisc_backlog_overflow(sch, skb)) ||
unlikely((u64)q->memory_used + skb->truesize > q->memory_limit)) {
qdisc_qstats_overlimit(sch);
if (skb_in_l_queue(skb))
diff --git a/net/sched/sch_etf.c b/net/sched/sch_etf.c
index aaafcf640bc7..a534b11eadd0 100644
--- a/net/sched/sch_etf.c
+++ b/net/sched/sch_etf.c
@@ -168,6 +168,12 @@ static int etf_enqueue_timesortedlist(struct sk_buff *nskb, struct Qdisc *sch,
ktime_t txtime = nskb->tstamp;
bool leftmost = true;
+ if (unlikely(qdisc_backlog_overflow(sch, nskb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(nskb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
if (!is_packet_valid(sch, nskb)) {
report_sock_error(nskb, EINVAL,
SO_EE_CODE_TXTIME_INVALID_PARAM);
diff --git a/net/sched/sch_ets.c b/net/sched/sch_ets.c
index 6cc902a03838..7f74f79ea6b6 100644
--- a/net/sched/sch_ets.c
+++ b/net/sched/sch_ets.c
@@ -420,6 +420,12 @@ static int ets_qdisc_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct ets_class *cl;
int err = 0;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
cl = ets_classify(skb, sch, &err);
if (!cl) {
if (err & __NET_XMIT_BYPASS)
diff --git a/net/sched/sch_fifo.c b/net/sched/sch_fifo.c
index 1b6388d50967..53f159d5b723 100644
--- a/net/sched/sch_fifo.c
+++ b/net/sched/sch_fifo.c
@@ -29,6 +29,12 @@ static int bfifo_enqueue(struct sk_buff *skb, struct Qdisc *sch,
static int pfifo_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct sk_buff **to_free)
{
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
if (likely(sch->q.qlen < READ_ONCE(sch->limit)))
return qdisc_enqueue_tail(skb, sch);
@@ -43,6 +49,12 @@ static int pfifo_tail_enqueue(struct sk_buff *skb, struct Qdisc *sch,
if (unlikely(READ_ONCE(sch->limit) == 0))
return qdisc_drop(skb, sch, to_free);
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
if (likely(sch->q.qlen < READ_ONCE(sch->limit)))
return qdisc_enqueue_tail(skb, sch);
diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c
index a282812c192e..03d06b65ed03 100644
--- a/net/sched/sch_fq.c
+++ b/net/sched/sch_fq.c
@@ -628,6 +628,12 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch,
u64 now;
u8 band;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
band = fq_prio2band(q->prio2band, skb->priority & TC_PRIO_MAX);
if (unlikely(q->band_pkt_count[band] >= sch->limit)) {
q->stat_band_drops[band]++;
diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c
index e0fcda26ff62..5feb50db41e8 100644
--- a/net/sched/sch_fq_codel.c
+++ b/net/sched/sch_fq_codel.c
@@ -201,6 +201,15 @@ static int fq_codel_enqueue(struct sk_buff *skb, struct Qdisc *sch,
}
idx--;
+ /* Drop before the flow is touched, so a dropped packet cannot
+ * create or update flow state.
+ */
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ q->drop_overlimit++;
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
codel_set_enqueue_time(skb);
flow = &q->flows[idx];
flow_queue_add(flow, skb);
diff --git a/net/sched/sch_fq_pie.c b/net/sched/sch_fq_pie.c
index 5982847df8f8..9f927a08b7bc 100644
--- a/net/sched/sch_fq_pie.c
+++ b/net/sched/sch_fq_pie.c
@@ -155,7 +155,8 @@ static int fq_pie_qdisc_enqueue(struct sk_buff *skb, struct Qdisc *sch,
memory_limited = q->memory_usage > q->memory_limit + skb->truesize;
/* Checks if the qdisc is full */
- if (unlikely(qdisc_qlen(sch) >= sch->limit)) {
+ if (unlikely(qdisc_qlen(sch) >= sch->limit) ||
+ unlikely(qdisc_backlog_overflow(sch, skb))) {
q->stats.overlimit++;
goto out;
} else if (unlikely(memory_limited)) {
diff --git a/net/sched/sch_gred.c b/net/sched/sch_gred.c
index f3d9a7252949..de96badff654 100644
--- a/net/sched/sch_gred.c
+++ b/net/sched/sch_gred.c
@@ -170,6 +170,15 @@ static int gred_enqueue(struct sk_buff *skb, struct Qdisc *sch,
unsigned long qavg = 0;
u16 dp = tc_index_to_dp(skb);
+ /* Per-VQ limits don't bound the total: every VQ adds into the
+ * same u32 backlog, so check it before picking a VQ.
+ */
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
if (dp >= t->DPs || (q = t->tab[dp]) == NULL) {
dp = t->def;
diff --git a/net/sched/sch_hfsc.c b/net/sched/sch_hfsc.c
index 284490fd6ca9..c11457e6cc24 100644
--- a/net/sched/sch_hfsc.c
+++ b/net/sched/sch_hfsc.c
@@ -1564,6 +1564,12 @@ hfsc_enqueue(struct sk_buff *skb, struct Qdisc *sch, struct sk_buff **to_free)
int err;
bool first;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
cl = hfsc_classify(skb, sch, &err);
if (cl == NULL) {
if (err & __NET_XMIT_BYPASS)
diff --git a/net/sched/sch_hhf.c b/net/sched/sch_hhf.c
index fca20c50c97c..a3d1e7c850e6 100644
--- a/net/sched/sch_hhf.c
+++ b/net/sched/sch_hhf.c
@@ -373,9 +373,16 @@ static int hhf_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct sk_buff **to_free)
{
struct hhf_sched_data *q = qdisc_priv(sch);
- enum wdrr_bucket_idx idx;
+ unsigned int len = qdisc_pkt_len(skb);
struct wdrr_bucket *bucket;
unsigned int prev_backlog;
+ enum wdrr_bucket_idx idx;
+
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
idx = hhf_classify(skb, sch);
@@ -409,8 +416,15 @@ static int hhf_enqueue(struct sk_buff *skb, struct Qdisc *sch,
/* Return Congestion Notification only if we dropped a packet from this
* bucket.
*/
- if (hhf_drop(sch, to_free) == idx)
+ if (hhf_drop(sch, to_free) == idx) {
+ /* The parent does not book this packet on CN, so pass the net
+ * change on instead.
+ */
+ qdisc_tree_reduce_backlog(sch, 0,
+ prev_backlog - sch->qstats.backlog -
+ len);
return NET_XMIT_CN;
+ }
/* As we dropped a packet, better let upper stack know this. */
qdisc_tree_reduce_backlog(sch, 1, prev_backlog - sch->qstats.backlog);
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
index 1ba67b121de4..37c6f28b1d26 100644
--- a/net/sched/sch_htb.c
+++ b/net/sched/sch_htb.c
@@ -625,10 +625,18 @@ static inline void htb_deactivate(struct htb_sched *q, struct htb_class *cl)
static int htb_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct sk_buff **to_free)
{
- int ret;
- unsigned int len = qdisc_pkt_len(skb);
struct htb_sched *q = qdisc_priv(sch);
- struct htb_class *cl = htb_classify(skb, sch, &ret);
+ unsigned int len = qdisc_pkt_len(skb);
+ struct htb_class *cl;
+ int ret;
+
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
+ cl = htb_classify(skb, sch, &ret);
if (cl == HTB_DIRECT) {
/* enqueue to helper queue */
diff --git a/net/sched/sch_netem.c b/net/sched/sch_netem.c
index 1dc2924890eb..c25ce4c3175e 100644
--- a/net/sched/sch_netem.c
+++ b/net/sched/sch_netem.c
@@ -467,6 +467,12 @@ static int netem_enqueue(struct sk_buff *skb, struct Qdisc *sch,
unsigned int prev_len = qdisc_pkt_len(skb);
int count = 1;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
/* Do not fool qdisc_drop_all() */
skb->prev = NULL;
@@ -567,7 +573,19 @@ static int netem_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct Qdisc *rootq = qdisc_root_bh(sch);
skb2->tc_depth++; /* prevent duplicating a dup... */
+ /* The clone re-enters the tree before this packet is booked,
+ * so every guard it reaches would compare against a backlog
+ * that does not hold it yet and admit two packets where it
+ * has room for one. Account it here and undo, because the
+ * normal path below books it again.
+ */
+ qstats_backlog_add(sch, prev_len);
+ qdisc_qlen_inc(sch);
+ qdisc_tree_reduce_backlog(sch, 0, -(int)prev_len);
rootq->enqueue(skb2, rootq, to_free);
+ qdisc_tree_reduce_backlog(sch, 0, prev_len);
+ qstats_backlog_sub(sch, prev_len);
+ qdisc_qlen_dec(sch);
skb2 = NULL;
}
diff --git a/net/sched/sch_pie.c b/net/sched/sch_pie.c
index 3b7863ffd284..2a5fc493a7e5 100644
--- a/net/sched/sch_pie.c
+++ b/net/sched/sch_pie.c
@@ -89,7 +89,8 @@ static int pie_qdisc_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct pie_sched_data *q = qdisc_priv(sch);
bool enqueue = false;
- if (unlikely(qdisc_qlen(sch) >= sch->limit)) {
+ if (unlikely(qdisc_qlen(sch) >= sch->limit) ||
+ unlikely(qdisc_backlog_overflow(sch, skb))) {
WRITE_ONCE(q->stats.overlimit, q->stats.overlimit + 1);
goto out;
}
diff --git a/net/sched/sch_plug.c b/net/sched/sch_plug.c
index b60ddfee6a68..5654e5310fb5 100644
--- a/net/sched/sch_plug.c
+++ b/net/sched/sch_plug.c
@@ -89,7 +89,8 @@ static int plug_enqueue(struct sk_buff *skb, struct Qdisc *sch,
{
struct plug_sched_data *q = qdisc_priv(sch);
- if (likely((u64)sch->qstats.backlog + skb->len <= q->limit)) {
+ if (likely((u64)sch->qstats.backlog + qdisc_pkt_len(skb) <=
+ q->limit)) {
if (!q->unplug_indefinite)
q->pkts_current_epoch++;
return qdisc_enqueue_tail(skb, sch);
diff --git a/net/sched/sch_prio.c b/net/sched/sch_prio.c
index 79437c587e7e..11f9354f57c2 100644
--- a/net/sched/sch_prio.c
+++ b/net/sched/sch_prio.c
@@ -72,6 +72,12 @@ prio_enqueue(struct sk_buff *skb, struct Qdisc *sch, struct sk_buff **to_free)
struct Qdisc *qdisc;
int ret;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
qdisc = prio_classify(skb, sch, &ret);
#ifdef CONFIG_NET_CLS_ACT
if (qdisc == NULL) {
diff --git a/net/sched/sch_qfq.c b/net/sched/sch_qfq.c
index e900890e9c54..8e0b2702669e 100644
--- a/net/sched/sch_qfq.c
+++ b/net/sched/sch_qfq.c
@@ -1236,6 +1236,12 @@ static int qfq_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct qfq_aggregate *agg;
int err = 0;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
cl = qfq_classify(skb, sch, &err);
if (cl == NULL) {
if (err & __NET_XMIT_BYPASS)
diff --git a/net/sched/sch_red.c b/net/sched/sch_red.c
index d7598214270b..738868f08a2e 100644
--- a/net/sched/sch_red.c
+++ b/net/sched/sch_red.c
@@ -76,6 +76,16 @@ static int red_enqueue(struct sk_buff *skb, struct Qdisc *sch,
unsigned int len;
int ret;
+ /* RED feeds the child backlog into red_calc_qavg() and books the
+ * packet on its own counter, so both must have room for it.
+ */
+ if (unlikely(qdisc_backlog_overflow(sch, skb) ||
+ qdisc_backlog_overflow(child, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
q->vars.qavg = red_calc_qavg(&q->parms,
&q->vars,
child->qstats.backlog);
diff --git a/net/sched/sch_sfb.c b/net/sched/sch_sfb.c
index ed39869199c0..7e8edb481edc 100644
--- a/net/sched/sch_sfb.c
+++ b/net/sched/sch_sfb.c
@@ -296,6 +296,12 @@ static int sfb_enqueue(struct sk_buff *skb, struct Qdisc *sch,
u32 slot = q->slot;
int ret = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
if (unlikely(sch->q.qlen >= q->limit)) {
qdisc_qstats_overlimit(sch);
WRITE_ONCE(q->stats.queuedrop,
diff --git a/net/sched/sch_sfq.c b/net/sched/sch_sfq.c
index 8bbcfc9e85d9..67d797f50755 100644
--- a/net/sched/sch_sfq.c
+++ b/net/sched/sch_sfq.c
@@ -351,6 +351,12 @@ sfq_enqueue(struct sk_buff *skb, struct Qdisc *sch, struct sk_buff **to_free)
struct sk_buff *head;
int delta;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
hash = sfq_classify(skb, sch, &ret);
if (hash == 0) {
if (ret & __NET_XMIT_BYPASS)
diff --git a/net/sched/sch_skbprio.c b/net/sched/sch_skbprio.c
index 52abfb4015a3..688a786364a9 100644
--- a/net/sched/sch_skbprio.c
+++ b/net/sched/sch_skbprio.c
@@ -70,11 +70,18 @@ static int skbprio_enqueue(struct sk_buff *skb, struct Qdisc *sch,
{
const unsigned int max_priority = SKBPRIO_MAX_PRIORITY - 1;
struct skbprio_sched_data *q = qdisc_priv(sch);
- struct sk_buff_head *qdisc;
struct sk_buff_head *lp_qdisc;
+ struct sk_buff_head *qdisc;
+ unsigned int to_drop_len;
struct sk_buff *to_drop;
u16 prio, lp;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
/* Obtain the priority of @skb. */
prio = min(skb->priority, max_priority);
@@ -113,10 +120,11 @@ static int skbprio_enqueue(struct sk_buff *skb, struct Qdisc *sch,
lp_qdisc = &q->qdiscs[lp];
to_drop = __skb_dequeue_tail(lp_qdisc);
BUG_ON(!to_drop);
+ to_drop_len = qdisc_pkt_len(to_drop);
qdisc_qstats_backlog_dec(sch, to_drop);
qdisc_drop(to_drop, sch, to_free);
- q->qstats[lp].backlog -= qdisc_pkt_len(to_drop);
+ q->qstats[lp].backlog -= to_drop_len;
q->qstats[lp].drops++;
q->qstats[lp].overlimits++;
@@ -133,6 +141,11 @@ static int skbprio_enqueue(struct sk_buff *skb, struct Qdisc *sch,
if (prio > q->highest_prio)
q->highest_prio = prio;
+ /* The parent does not book this packet on CN, so pass the net
+ * change on instead.
+ */
+ qdisc_tree_reduce_backlog(sch, 0, to_drop_len - qdisc_pkt_len(skb));
+
return NET_XMIT_CN;
}
diff --git a/net/sched/sch_taprio.c b/net/sched/sch_taprio.c
index 5a1dca9d5e9d..2f199541e004 100644
--- a/net/sched/sch_taprio.c
+++ b/net/sched/sch_taprio.c
@@ -589,6 +589,15 @@ static int taprio_enqueue_one(struct sk_buff *skb, struct Qdisc *sch,
struct Qdisc *child, struct sk_buff **to_free)
{
struct taprio_sched *q = qdisc_priv(sch);
+ u32 backlog;
+ int qlen;
+ int ret;
+
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
/* sk_flags are only safe to use on full sockets. */
if (skb->sk && sk_fullsock(skb->sk) && sock_flag(skb->sk, SOCK_TXTIME)) {
@@ -600,10 +609,20 @@ static int taprio_enqueue_one(struct sk_buff *skb, struct Qdisc *sch,
return qdisc_drop(skb, sch, to_free);
}
- qdisc_qstats_backlog_inc(sch, skb);
- qdisc_qlen_inc(sch);
+ /* A child may free skb (tbf_segment(), cake's split path) or evict
+ * another packet before returning, so take everything this function
+ * needs before the handoff and mirror the child's net change. A
+ * child that evicts cannot reach us through
+ * qdisc_tree_reduce_backlog() because taprio_graft() marks it
+ * TCQ_F_NOPARENT.
+ */
+ backlog = READ_ONCE(child->qstats.backlog);
+ qlen = qdisc_qlen(child);
+ ret = qdisc_enqueue(skb, child, to_free);
+ qstats_backlog_add(sch, READ_ONCE(child->qstats.backlog) - backlog);
+ WRITE_ONCE(sch->q.qlen, sch->q.qlen + qdisc_qlen(child) - qlen);
- return qdisc_enqueue(skb, child, to_free);
+ return ret;
}
static int taprio_enqueue_segmented(struct sk_buff *skb, struct Qdisc *sch,
@@ -620,10 +639,11 @@ static int taprio_enqueue_segmented(struct sk_buff *skb, struct Qdisc *sch,
return qdisc_drop(skb, sch, to_free);
skb_list_walk_safe(segs, segs, nskb) {
+ unsigned int seg_len = segs->len;
+
skb_mark_not_on_list(segs);
- qdisc_skb_cb(segs)->pkt_len = segs->len;
+ qdisc_skb_cb(segs)->pkt_len = seg_len;
qdisc_skb_cb(segs)->pkt_segs = 1;
- slen += segs->len;
/* FIXME: we should be segmenting to a smaller size
* rather than dropping these
@@ -633,10 +653,12 @@ static int taprio_enqueue_segmented(struct sk_buff *skb, struct Qdisc *sch,
else
ret = taprio_enqueue_one(segs, sch, child, to_free);
- if (ret != NET_XMIT_SUCCESS) {
- if (net_xmit_drop_count(ret))
- qdisc_qstats_drop(sch);
- } else {
+ /* Every drop above is already counted by the site that
+ * performs it; only keep track of the admitted segments
+ * here.
+ */
+ if (ret == NET_XMIT_SUCCESS) {
+ slen += seg_len;
numsegs++;
}
}
diff --git a/net/sched/sch_tbf.c b/net/sched/sch_tbf.c
index 58c9687efb0f..5a23b6749758 100644
--- a/net/sched/sch_tbf.c
+++ b/net/sched/sch_tbf.c
@@ -255,6 +255,12 @@ static int tbf_enqueue(struct sk_buff *skb, struct Qdisc *sch,
unsigned int len = qdisc_pkt_len(skb);
int ret;
+ if (unlikely(qdisc_backlog_overflow(sch, skb))) {
+ qdisc_qstats_overlimit(sch);
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_OVERLIMIT);
+ }
+
if (qdisc_pkt_len(skb) > q->max_size) {
if (skb_is_gso(skb) &&
skb_gso_validate_mac_len(skb, q->max_size))
--
2.43.0
^ permalink raw reply related [flat|nested] 5+ messages in thread* [Cake] [PATCH net-next v12 4/4] net/sched: cake: hash before segmenting and reject an overlong split
2026-11-21 19:21 [Cake] [PATCH net-next v12 0/4] net/sched: cap the accounted backlog before it can wrap Jamal Hadi Salim
` (2 preceding siblings ...)
2026-11-21 19:21 ` [Cake] [PATCH net-next v12 3/4] net/sched: cap the accounted backlog before it can wrap Jamal Hadi Salim
@ 2026-11-21 19:21 ` Jamal Hadi Salim
3 siblings, 0 replies; 5+ messages in thread
From: Jamal Hadi Salim @ 2026-11-21 19:21 UTC (permalink / raw)
To: netdev
Cc: Jamal Hadi Salim, Jiri Pirko, David S . Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Simon Horman, Stephen Hemminger,
Vinicius Costa Gomes, Victor Nogueira,
Toke Høiland-Jørgensen, Björn Töpel, cake,
Sashiko
The Fix:
with CAKE_FLAG_SPLIT_GSO, cake computes the flow/host hash keys, then
splits a GSO skb into segments it accounts individually. Key extraction
must run on the unsegmented skb -- skb_gso_segment() can rewrite an
encapsulated packet's headers to the inner ones -- and the resolution
must wait until the packet is admitted, since it commits the
set-associative tags, host indices and way counters that a later
rejection would leave mutated (a GSO skb whose segmentation fails was
dropped with that state already committed).
Split cake_hash() into a side-effect-free cake_hash_keys(), run on the
unsegmented skb, and cake_hash_resolve(), run only once the packet is
admitted. Move the split-GSO path ahead of the resolution: sum the
segment list, test the total against the ceiling before linking any
segment, and drop the whole list on rejection so no persistent state is
committed. The list is deferred to to_free so the tail segment's
sock_wfree() never runs under the qdisc lock.
Tag every deferred segment with QDISC_DROP_OVERLIMIT: __skb_gso_segment()
keeps SKB_GSO_CB at cb offset 32, aliasing tc_skb_cb.drop_reason, and
__copy_skb_header() copies it into every segment, so tagging only the
head left the rest with GSO scratch (a headroom-sized value) as their
drop reason, which __tcf_kfree_skb_list() would report as an unrelated
core reason. The segmentation-failure return has the same aliasing and
uses qdisc_drop_reason(..., QDISC_DROP_GENERIC) so its drop is still
reported as a qdisc drop.
Read q->config->flow_mode once with READ_ONCE() in cake_enqueue() and
pass that value to both cake_hash_keys() and cake_hash_resolve(), and to
the later flowchain calls (cake_get_flow_quantum(),
cake_inc_srchost_bulk_flow_count(), cake_inc_dsthost_bulk_flow_count()).
cake_mq_change() updates the shared cake_config outside the qdisc lock, so
a reconfiguration could extract and resolve keys in different modes.
Conditions to recreate the bug:
ip tuntap add tun0 mode tun
ip link set tun0 txqueuelen 32 up
ip addr add 10.99.0.1/24 dev tun0
# hold the tun fd open without reading so the backlog persists; a stab
# overhead of 1048468 accounts each 92-byte packet at 1048560 bytes, so
# 4095 queued packets leave 65519 bytes below QDISC_MAX_BACKLOG. A
# 65000-byte UDP_SEGMENT payload with a 1400-byte segment size then has
# a segment sum past the remaining headroom, and cake rejects the whole
# split. Setup needs CAP_NET_ADMIN; the payload itself does not.
tc qdisc add dev tun0 root handle 1: stab overhead 1048468 \
cake split-gso memlimit 512Mb
Link: https://lore.kernel.org/netdev/179231933068.3135.80340792633077986@kernel.org/
Link: https://lore.kernel.org/netdev/179269577270.3135.16072065138845192450@kernel.org/
Reviewed-by: Victor Nogueira <victor@mojatatu.com>
Acked-by: Toke Høiland-Jørgensen <toke@toke.dk>
Signed-off-by: Jamal Hadi Salim <jhs@mojatatu.com>
---
v11 -> v12:
- compare the split's segment sum against U32_MAX, the wrap boundary that
qdisc_backlog_overflow() uses, instead of the post-add ceiling, which
rejected a split up to two packet lengths before it could wrap.
(Sashiko nipa review of v11)
---
net/sched/sch_cake.c | 133 +++++++++++++++++++++++++++++++------------
1 file changed, 97 insertions(+), 36 deletions(-)
diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
index 803ead499ae9..24fff15f8f7d 100644
--- a/net/sched/sch_cake.c
+++ b/net/sched/sch_cake.c
@@ -709,19 +709,28 @@ static u16 cake_get_flow_quantum(struct cake_tin_data *q,
get_random_u16()) >> 16;
}
-static u32 cake_hash(struct cake_tin_data *q, const struct sk_buff *skb,
- int flow_mode, u16 flow_override, u16 host_override)
+struct cake_hash_keys {
+ u32 srchost_hash;
+ u32 dsthost_hash;
+ u32 flow_hash;
+};
+
+static void cake_hash_keys(const struct sk_buff *skb, int flow_mode,
+ u16 flow_override, u16 host_override,
+ struct cake_hash_keys *out)
{
bool hash_flows = (!flow_override && !!(flow_mode & CAKE_FLOW_FLOWS));
bool hash_hosts = (!host_override && !!(flow_mode & CAKE_FLOW_HOSTS));
bool nat_enabled = !!(flow_mode & CAKE_FLOW_NAT_FLAG);
- u32 flow_hash = 0, srchost_hash = 0, dsthost_hash = 0;
- u16 reduced_hash, srchost_idx, dsthost_idx;
struct flow_keys keys, host_keys;
bool use_skbhash = skb->l4_hash;
+ out->flow_hash = 0;
+ out->srchost_hash = 0;
+ out->dsthost_hash = 0;
+
if (unlikely(flow_mode == CAKE_FLOW_NONE))
- return 0;
+ return;
/* If both overrides are set, or we can use the SKB hash and nat mode is
* disabled, we can skip packet dissection entirely. If nat mode is
@@ -756,50 +765,64 @@ static u32 cake_hash(struct cake_tin_data *q, const struct sk_buff *skb,
switch (host_keys.control.addr_type) {
case FLOW_DISSECTOR_KEY_IPV4_ADDRS:
host_keys.addrs.v4addrs.src = 0;
- dsthost_hash = flow_hash_from_keys(&host_keys);
+ out->dsthost_hash = flow_hash_from_keys(&host_keys);
host_keys.addrs.v4addrs.src = keys.addrs.v4addrs.src;
host_keys.addrs.v4addrs.dst = 0;
- srchost_hash = flow_hash_from_keys(&host_keys);
+ out->srchost_hash = flow_hash_from_keys(&host_keys);
break;
case FLOW_DISSECTOR_KEY_IPV6_ADDRS:
memset(&host_keys.addrs.v6addrs.src, 0,
sizeof(host_keys.addrs.v6addrs.src));
- dsthost_hash = flow_hash_from_keys(&host_keys);
+ out->dsthost_hash = flow_hash_from_keys(&host_keys);
host_keys.addrs.v6addrs.src = keys.addrs.v6addrs.src;
memset(&host_keys.addrs.v6addrs.dst, 0,
sizeof(host_keys.addrs.v6addrs.dst));
- srchost_hash = flow_hash_from_keys(&host_keys);
+ out->srchost_hash = flow_hash_from_keys(&host_keys);
break;
default:
- dsthost_hash = 0;
- srchost_hash = 0;
+ out->dsthost_hash = 0;
+ out->srchost_hash = 0;
}
/* This *must* be after the above switch, since as a
* side-effect it sorts the src and dst addresses.
*/
if (hash_flows && !use_skbhash)
- flow_hash = flow_hash_from_keys(&keys);
+ out->flow_hash = flow_hash_from_keys(&keys);
skip_hash:
if (flow_override)
- flow_hash = flow_override - 1;
+ out->flow_hash = flow_override - 1;
else if (use_skbhash && (flow_mode & CAKE_FLOW_FLOWS))
- flow_hash = skb->hash;
+ out->flow_hash = skb->hash;
if (host_override) {
- dsthost_hash = host_override - 1;
- srchost_hash = host_override - 1;
+ out->dsthost_hash = host_override - 1;
+ out->srchost_hash = host_override - 1;
}
if (!(flow_mode & CAKE_FLOW_FLOWS)) {
if (flow_mode & CAKE_FLOW_SRC_IP)
- flow_hash ^= srchost_hash;
+ out->flow_hash ^= out->srchost_hash;
if (flow_mode & CAKE_FLOW_DST_IP)
- flow_hash ^= dsthost_hash;
+ out->flow_hash ^= out->dsthost_hash;
}
+}
+
+/* Commit the flow into the set-associative table (q->tags[], host
+ * indices, bulk counters). Only after the packet is admitted.
+ */
+static u32 cake_hash_resolve(struct cake_tin_data *q,
+ const struct cake_hash_keys *keys, int flow_mode)
+{
+ u32 flow_hash = keys->flow_hash, srchost_hash = keys->srchost_hash,
+ dsthost_hash = keys->dsthost_hash;
+ u16 reduced_hash, srchost_idx, dsthost_idx;
+
+ if (unlikely(flow_mode == CAKE_FLOW_NONE))
+ return 0;
reduced_hash = flow_hash % CAKE_QUEUES;
@@ -1765,13 +1788,17 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
{
u32 idx, tin, prev_qlen, prev_backlog, drop_id;
struct cake_sched_data *q = qdisc_priv(sch);
+ int flow_mode = READ_ONCE(q->config->flow_mode);
int len = qdisc_pkt_len(skb), ret;
u16 flow_override, host_override;
+ struct sk_buff *segs = NULL;
+ struct cake_hash_keys keys;
struct sk_buff *ack = NULL;
ktime_t now = ktime_get();
struct cake_tin_data *b;
struct cake_flow *flow;
bool same_flow = false;
+ u64 slen = 0;
/* Classify first: a terminal verdict means do not enqueue. */
if (cake_tcf_classify(sch, skb, &flow_override, &host_override, &ret)) {
@@ -1788,12 +1815,52 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
QDISC_DROP_OVERLIMIT);
}
- /* choose flow to insert into */
+ /* Select the tin and extract the keys before segmenting:
+ * skb_gso_segment() rewrites encapsulated headers, which
+ * changes the keys.
+ */
b = cake_select_tin(sch, skb);
- idx = cake_hash(b, skb, q->config->flow_mode, flow_override,
- host_override) + 1;
+ cake_hash_keys(skb, flow_mode, flow_override, host_override, &keys);
+
+ if (qdisc_pkt_segs(skb) > 1 &&
+ q->config->rate_flags & CAKE_FLAG_SPLIT_GSO) {
+ netdev_features_t features = netif_skb_features(skb);
+ struct sk_buff *nskb, *seg;
+
+ segs = skb_gso_segment(skb, features & ~NETIF_F_GSO_MASK);
+ if (IS_ERR_OR_NULL(segs))
+ return qdisc_drop_reason(skb, sch, to_free,
+ QDISC_DROP_GENERIC);
+
+ /* we never use the original skb after splitting */
+ consume_skb(skb);
+ skb = NULL;
+
+ /* The split path accounts the segment sum, not the
+ * stab-adjusted qdisc_pkt_len(), so check that sum
+ * against the same wrap boundary as the enqueue guard.
+ */
+ skb_list_walk_safe(segs, seg, nskb)
+ slen += seg->len;
+
+ if (unlikely((u64)sch->qstats.backlog + slen > U32_MAX)) {
+ qdisc_qstats_overlimit(sch);
+ /* __skb_gso_segment() puts SKB_GSO_CB at cb
+ * offset 32, over tc_skb_cb.drop_reason; every
+ * segment inherits it. Tag each one with the
+ * qdisc drop reason.
+ */
+ skb_list_walk_safe(segs, seg, nskb)
+ tcf_set_qdisc_drop_reason(seg,
+ QDISC_DROP_OVERLIMIT);
+
+ return qdisc_drop_all(segs, sch, to_free);
+ }
+ }
+
+ /* Admitted: resolve the flow. */
+ idx = cake_hash_resolve(b, &keys, flow_mode);
tin = (u32)(b - q->tins);
- idx--;
flow = &b->flows[idx];
/* ensure shaper state isn't stale */
@@ -1820,14 +1887,9 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
if (unlikely(len > b->max_skblen))
WRITE_ONCE(b->max_skblen, len);
- if (qdisc_pkt_segs(skb) > 1 && q->config->rate_flags & CAKE_FLAG_SPLIT_GSO) {
- struct sk_buff *segs, *nskb, *seg;
- netdev_features_t features = netif_skb_features(skb);
- unsigned int slen = 0, numsegs = 0;
-
- segs = skb_gso_segment(skb, features & ~NETIF_F_GSO_MASK);
- if (IS_ERR_OR_NULL(segs))
- return qdisc_drop(skb, sch, to_free);
+ if (segs) {
+ struct sk_buff *nskb, *seg;
+ unsigned int numsegs = 0;
skb_list_walk_safe(segs, seg, nskb) {
skb_mark_not_on_list(seg);
@@ -1841,7 +1903,6 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
qdisc_qlen_inc(sch);
numsegs++;
q->buffer_used += seg->truesize;
- slen += seg->len;
WRITE_ONCE(b->packets, b->packets + 1);
}
@@ -1853,9 +1914,8 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
WRITE_ONCE(b->backlogs[idx], b->backlogs[idx] + slen);
qdisc_tree_reduce_backlog(sch, 1-numsegs, len-slen);
- consume_skb(skb);
} else {
- /* not splitting */
+ /* not splitting - skb pointer still valid */
int ack_pkt_len = 0;
cobalt_set_enqueue_time(skb, now);
@@ -1949,7 +2009,8 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
flow->set = CAKE_SET_SPARSE;
WRITE_ONCE(b->sparse_flow_count, b->sparse_flow_count + 1);
- WRITE_ONCE(flow->deficit, cake_get_flow_quantum(b, flow, q->config->flow_mode));
+ WRITE_ONCE(flow->deficit,
+ cake_get_flow_quantum(b, flow, flow_mode));
} else if (flow->set == CAKE_SET_SPARSE_WAIT) {
/* this flow was empty, accounted as a sparse flow, but actually
* in the bulk rotation.
@@ -1958,8 +2019,8 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
WRITE_ONCE(b->sparse_flow_count, b->sparse_flow_count - 1);
WRITE_ONCE(b->bulk_flow_count, b->bulk_flow_count + 1);
- cake_inc_srchost_bulk_flow_count(b, flow, q->config->flow_mode);
- cake_inc_dsthost_bulk_flow_count(b, flow, q->config->flow_mode);
+ cake_inc_srchost_bulk_flow_count(b, flow, flow_mode);
+ cake_inc_dsthost_bulk_flow_count(b, flow, flow_mode);
}
if (q->buffer_used > u64_stats_read(&q->buffer_max_used)) {
--
2.43.0
^ permalink raw reply related [flat|nested] 5+ messages in thread