[Cake] total download rate with many flows

[David Lang]

Ingres and Egress are fundamentally different due to the fact that in ingress 
mode you are having to throw away data that has successfully traversed the 
bottleneck (deliberatly wasting your limited resource now to avoid having 
senders bottleneck the queue on the far side of the link)

in Egress mode, you aren't doing that, and so you can get much closer to the 
actual bandwidth.

In addition, in Ingress mode, you are always working via second-order effects, 
you can't slow a transmission directly like you can in egress mode, all you can 
do is drop packets and wait until the sender notices, retransmits and slows 
down. Egress has full control of the queues and can send things in any order, 
and may be able to continually fill the pipe and avoid any timeouts and 
retransmissions (if the flow is short enough)

David Lang