On Thu, 19 Dec 2024 11:31:08 -0500
David Collier-Brown via Bloat <bloat@lists.bufferbloat.net> wrote:

On 12/18/24 17:17, David Lang via Bloat wrote:

so, what happens when a standardized test is mandated and then it's 
found that that test isn't as good as others?

I'm leery of any government mandates.  

Governments in general are good a "policing" things*, such as 
deficiencies in specifications and persons trying to weasel around them.

At the same time, good specifiers write in "or better" clauses so that 
subsequent standards can be a few lines added to the original work.

We can tell that is broken in Canada when the CRTC does a request for 
comments ... but then rejects all the comments and proposed amendments. 
Oh, and resists publishing them (:-))

Have you seen that in the US?

--dave

And it will just create benchmark cheating...
Look at any of the standardized database benchmarks as an example.
The benchmark starts out trying to an express a workload; then the vendors
discover new and creative ways to get higher numbers.
_______________________________________________

Hmmn, and that's for non-mandated performance standards!

I'll speculate here that we have a "whack-a-mole" situation. No matter how many holes you fix, you can't make a benchmark fair.

The interesting question is if you can make a "not less than" rule system monotonically reduce the attack surface, instead of leaving it the same size or worse (:-))

I know we do that with case law (I'm a former Quicklaw nerd) but it can be arbitrarily hard...

--dave

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net           |              -- Mark Twain