[Cerowrt-devel] Had to disable dnssec today

[Cerowrt-devel] Had to disable dnssec today

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

Just too many sites aren't working correctly with dnsmasq and using
Google's DNS servers.

- Bank of America (sso-fi.bankofamerica.com)
- Weather Underground (cdnjs.cloudflare.com)
- Akamai (e3191.dscc.akamaiedge.net.0.1.cn.akamaiedge.net)

And I'm not getting any traction with reporting the errors to those sites,
so it's frustrating in getting it properly fixed.

While Akamai and cloudflare appear to be issues with their entries in
google dns, or with dnsmasq's validation of them being insecure domains,
the BofA issue appears to be an outright bad key.  And BofA isn't being
helpful (just a continual "we use ssl" sort of quasi-automated response).

So I'm disabling it for now, or rather, falling back to using my ISP's dns
servers, which don't support DNSSEC at this time.  I'll be periodically
turning it back on, but too much is broken (mainly due to the cdns) to be
able to rely on it at this time.

-Aaron

[-- Attachment #2: Type: text/html, Size: 1328 bytes --]

Re: [Cerowrt-devel] Had to disable dnssec today

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 1572 bytes --]


Is this just a dnsmasq issue or is the DNSSEC mechanism broken at these sites?   If it is the latter, I can get attention from executives at some of these companies (Heartbleed has sensitized all kinds of companies to the need to strengthen security infrastructure).
 
If the former, the change process is going to be more tricky, because dnsmasq is easily dismissed as too small a proportion of the market to care.  (wish it were not so).


On Saturday, April 26, 2014 7:38am, "Aaron Wood" <woody77@gmail.com> said:



Just too many sites aren't working correctly with dnsmasq and using Google's DNS servers.
- Bank of America ([http://sso-fi.bankofamerica.com] sso-fi.bankofamerica.com)
- Weather Underground ([http://cdnjs.cloudflare.com] cdnjs.cloudflare.com)
- Akamai ([http://e3191.dscc.akamaiedge.net.0.1.cn.akamaiedge.net] e3191.dscc.akamaiedge.net.0.1.cn.akamaiedge.net)
And I'm not getting any traction with reporting the errors to those sites, so it's frustrating in getting it properly fixed.
While Akamai and cloudflare appear to be issues with their entries in google dns, or with dnsmasq's validation of them being insecure domains, the BofA issue appears to be an outright bad key.  And BofA isn't being helpful (just a continual "we use ssl" sort of quasi-automated response).
So I'm disabling it for now, or rather, falling back to using my ISP's dns servers, which don't support DNSSEC at this time.  I'll be periodically turning it back on, but too much is broken (mainly due to the cdns) to be able to rely on it at this time.
-Aaron

[-- Attachment #2: Type: text/html, Size: 2033 bytes --]

Re: [Cerowrt-devel] Had to disable dnssec today

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 2337 bytes --]

David,

With two of them (akamai and cloudflare), I _think_ it's a dnsmasq issue
with the DS records for proving insecure domains are insecure.  But Simon
Kelley would know that better than I.

With BofA, I'm nearly certain it's them, or an issue with one of their
partners (since the domain that fails isn't BofA, but something else):

(with dnssec turned off):

;; QUESTION SECTION:
;sso-fi.bankofamerica.com. IN A

;; ANSWER SECTION:
sso-fi.bankofamerica.com. 3599 IN CNAME saml-bac.onefiserv.com.
saml-bac.onefiserv.com. 299 IN CNAME saml-bac.gslb.onefiserv.com.
saml-bac.gslb.onefiserv.com. 119 IN A 208.235.248.157

And it's the saml-bac.gslb.onefiserv.com host that's failing (see here for
debug info):

http://dnssec-debugger.verisignlabs.com/sso-fi.bankofamerica.com

-Aaron


On Sat, Apr 26, 2014 at 6:00 PM, <dpreed@reed.com> wrote:

> Is this just a dnsmasq issue or is the DNSSEC mechanism broken at these
> sites?   If it is the latter, I can get attention from executives at some
> of these companies (Heartbleed has sensitized all kinds of companies to the
> need to strengthen security infrastructure).
>
>
>
> If the former, the change process is going to be more tricky, because
> dnsmasq is easily dismissed as too small a proportion of the market to
> care.  (wish it were not so).
>
>
>
> On Saturday, April 26, 2014 7:38am, "Aaron Wood" <woody77@gmail.com> said:
>
>  Just too many sites aren't working correctly with dnsmasq and using
> Google's DNS servers.
> - Bank of America (sso-fi.bankofamerica.com)
> - Weather Underground (cdnjs.cloudflare.com)
> - Akamai (e3191.dscc.akamaiedge.net.0.1.cn.akamaiedge.net)
> And I'm not getting any traction with reporting the errors to those sites,
> so it's frustrating in getting it properly fixed.
> While Akamai and cloudflare appear to be issues with their entries in
> google dns, or with dnsmasq's validation of them being insecure domains,
> the BofA issue appears to be an outright bad key.  And BofA isn't being
> helpful (just a continual "we use ssl" sort of quasi-automated response).
> So I'm disabling it for now, or rather, falling back to using my ISP's dns
> servers, which don't support DNSSEC at this time.  I'll be periodically
> turning it back on, but too much is broken (mainly due to the cdns) to be
> able to rely on it at this time.
> -Aaron
>

[-- Attachment #2: Type: text/html, Size: 4445 bytes --]

Re: [Cerowrt-devel] [Dnsmasq-discuss] Had to disable dnssec today

[Simon Kelley]

On 26/04/14 17:20, Aaron Wood wrote:
> David,
> 
> With two of them (akamai and cloudflare), I _think_ it's a dnsmasq 
> issue with the DS records for proving insecure domains are insecure. 
> But Simon Kelley would know that better than I.
> 


The result of the analysis of the akamai domain was that there's a
problem with the domain (ie it's an akamai problem) See the post in the
Cerowrt list by Evan Hunt for the origin of this conclusion.

There's a dnsmasq issue to the extent that dnsmasq uses a different
strategy for proving that a name should not be signed than other
nameservers (dnsmasq works bottom-up, the others can work top-down,
since they are recursive servers, not forwarders.) This means that
dnsmasq sees the akamai problem, whilst eg unbound happens not to. I
plan to see if dnsmasq can be modified to improve this.

I'm not sure of cloudflare has been looked at in detail, but my
impression was that it's the same as akamai.

> With BofA, I'm nearly certain it's them, or an issue with one of 
> their partners (since the domain that fails isn't BofA, but
> something else):
> 
> (with dnssec turned off):
> 
> ;; QUESTION SECTION: ;sso-fi.bankofamerica.com. IN A
> 
> ;; ANSWER SECTION: sso-fi.bankofamerica.com. 3599 IN CNAME 
> saml-bac.onefiserv.com. saml-bac.onefiserv.com. 299 IN CNAME 
> saml-bac.gslb.onefiserv.com. saml-bac.gslb.onefiserv.com. 119 IN A 
> 208.235.248.157
> 
> And it's the saml-bac.gslb.onefiserv.com host that's failing (see 
> here for debug info):
> 
> http://dnssec-debugger.verisignlabs.com/sso-fi.bankofamerica.com
> 
> -Aaron
> 
> 
> On Sat, Apr 26, 2014 at 6:00 PM, <dpreed@reed.com> wrote:
> 
>> Is this just a dnsmasq issue or is the DNSSEC mechanism broken at 
>> these sites?   If it is the latter, I can get attention from 
>> executives at some of these companies (Heartbleed has sensitized 
>> all kinds of companies to the need to strengthen security 
>> infrastructure).
>> 
>> 
>> 
>> If the former, the change process is going to be more tricky, 
>> because dnsmasq is easily dismissed as too small a proportion of 
>> the market to care.  (wish it were not so).
>> 


Given it's less than a month since the first DNSSEC-capable dnsmasq
release, anything other than small market share would be fairly miraculous!

Cheers,

Simon.

Re: [Cerowrt-devel] [Dnsmasq-discuss] Had to disable dnssec today

[Simon Kelley]

On 26/04/14 20:44, Simon Kelley wrote:
> I plan to see if dnsmasq can be modified to improve this.

In the git repo now, the change allows the akamai domain to resolve
successfully.


Simon.

Re: [Cerowrt-devel] [Dnsmasq-discuss] Had to disable dnssec today

[Dave Taht]

On Sat, Apr 26, 2014 at 12:44 PM, Simon Kelley <simon@thekelleys.org.uk> wrote:
> On 26/04/14 17:20, Aaron Wood wrote:
>> David,
>>
>> With two of them (akamai and cloudflare), I _think_ it's a dnsmasq
>> issue with the DS records for proving insecure domains are insecure.
>> But Simon Kelley would know that better than I.
>>
>
>
> The result of the analysis of the akamai domain was that there's a
> problem with the domain (ie it's an akamai problem) See the post in the
> Cerowrt list by Evan Hunt for the origin of this conclusion.
>
> There's a dnsmasq issue to the extent that dnsmasq uses a different
> strategy for proving that a name should not be signed than other
> nameservers (dnsmasq works bottom-up, the others can work top-down,
> since they are recursive servers, not forwarders.) This means that
> dnsmasq sees the akamai problem, whilst eg unbound happens not to. I
> plan to see if dnsmasq can be modified to improve this.

If it's not a violation of the specification, the bottom-up method might
be good to add to a dnssec validation tool.

>
> I'm not sure of cloudflare has been looked at in detail, but my
> impression was that it's the same as akamai.
>
>> With BofA, I'm nearly certain it's them, or an issue with one of
>> their partners (since the domain that fails isn't BofA, but
>> something else):
>>
>> (with dnssec turned off):
>>
>> ;; QUESTION SECTION: ;sso-fi.bankofamerica.com. IN A
>>
>> ;; ANSWER SECTION: sso-fi.bankofamerica.com. 3599 IN CNAME
>> saml-bac.onefiserv.com. saml-bac.onefiserv.com. 299 IN CNAME
>> saml-bac.gslb.onefiserv.com. saml-bac.gslb.onefiserv.com. 119 IN A
>> 208.235.248.157
>>
>> And it's the saml-bac.gslb.onefiserv.com host that's failing (see
>> here for debug info):
>>
>> http://dnssec-debugger.verisignlabs.com/sso-fi.bankofamerica.com
>>
>> -Aaron
>>
>>
>> On Sat, Apr 26, 2014 at 6:00 PM, <dpreed@reed.com> wrote:
>>
>>> Is this just a dnsmasq issue or is the DNSSEC mechanism broken at
>>> these sites?   If it is the latter, I can get attention from
>>> executives at some of these companies (Heartbleed has sensitized
>>> all kinds of companies to the need to strengthen security
>>> infrastructure).
>>>
>>>
>>>
>>> If the former, the change process is going to be more tricky,
>>> because dnsmasq is easily dismissed as too small a proportion of
>>> the market to care.  (wish it were not so).
>>>
>
>
> Given it's less than a month since the first DNSSEC-capable dnsmasq
> release, anything other than small market share would be fairly miraculous!
>
> Cheers,
>
> Simon.
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

Re: [Cerowrt-devel] Had to disable dnssec today

[Dave Taht]

On Sat, Apr 26, 2014 at 4:38 AM, Aaron Wood <woody77@gmail.com> wrote:
> Just too many sites aren't working correctly with dnsmasq and using Google's
> DNS servers.

After 4 days of uptime, I too ended up with a wedged cerowrt 3.10.36-6 on wifi.

The symptoms
were dissimilar from what has been described here - I was seeing odhcpd
trying to and failing to answer requests on the wifi interfaces, which I'd never
seen in operation before (and could have been a self-induced failure by
fiddling with hnetd)

I have merged with openwrt head, which has some hostapd and routing fixes,
as well as dnsmasq head which has some dnssec lookup fixes...
and put out cerowrt-3.10.36-7. On first boot, it had problems getting anything
on wifi to do dhcp. A reboot later (with multicast 9000 also disabled),
a kindle that was failing to get online did. This box has also never got
upstream dns servers right from the isp. I'll fiddle with the multicast thing
later, to see if that or the reboot fixed it.

With this dnssec with dnssec-check-unsigned, once time is correct:

> - Bank of America (sso-fi.bankofamerica.com)

still fails. It ain't our fault it's broke.

> - Weather Underground (cdnjs.cloudflare.com)

succeeds.

> - Akamai (e3191.dscc.akamaiedge.net.0.1.cn.akamaiedge.net)

succeeds.

> http://test-ipv6.com/

don't have ipv6 capability at this location, so this succeeds. I did see
it fail once on the first boot but haven't repeated it.

>
> And I'm not getting any traction with reporting the errors to those sites,
> so it's frustrating in getting it properly fixed.

There needs to be constant network wide scanning service of some kind
to detect dnssec configuration errors.

>
> While Akamai and cloudflare appear to be issues with their entries in google
> dns, or with dnsmasq's validation of them being insecure domains, the BofA
> issue appears to be an outright bad key.  And BofA isn't being helpful (just
> a continual "we use ssl" sort of quasi-automated response).

Cluebats are needed.

> So I'm disabling it for now, or rather, falling back to using my ISP's dns
> servers, which don't support DNSSEC at this time.  I'll be periodically
> turning it back on, but too much is broken (mainly due to the cdns) to be
> able to rely on it at this time.

don't blame you, but if we weren't beating it up, nobody would be.

>
> -Aaron
>
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

Re: [Cerowrt-devel] Had to disable dnssec today

[Stephen Hemminger]

On Sat, 26 Apr 2014 13:38:08 +0200
Aaron Wood <woody77@gmail.com> wrote:

> Just too many sites aren't working correctly with dnsmasq and using
> Google's DNS servers.
> 
> - Bank of America (sso-fi.bankofamerica.com)
> - Weather Underground (cdnjs.cloudflare.com)
> - Akamai (e3191.dscc.akamaiedge.net.0.1.cn.akamaiedge.net)
> 
> And I'm not getting any traction with reporting the errors to those sites,
> so it's frustrating in getting it properly fixed.
> 
> While Akamai and cloudflare appear to be issues with their entries in
> google dns, or with dnsmasq's validation of them being insecure domains,
> the BofA issue appears to be an outright bad key.  And BofA isn't being
> helpful (just a continual "we use ssl" sort of quasi-automated response).
> 
> So I'm disabling it for now, or rather, falling back to using my ISP's dns
> servers, which don't support DNSSEC at this time.  I'll be periodically
> turning it back on, but too much is broken (mainly due to the cdns) to be
> able to rely on it at this time.
> 
> -Aaron

Ditto. I was holding out, but performance was much worse, many websites
would load poorly and got complaints from many errors from my customers (family).

Re: [Cerowrt-devel] Had to disable dnssec today

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1396 bytes --]

Now that I'm on Comcast, I'm going to try it again.

-Aaron


On Fri, May 16, 2014 at 8:25 PM, Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Sat, 26 Apr 2014 13:38:08 +0200
> Aaron Wood <woody77@gmail.com> wrote:
>
> > Just too many sites aren't working correctly with dnsmasq and using
> > Google's DNS servers.
> >
> > - Bank of America (sso-fi.bankofamerica.com)
> > - Weather Underground (cdnjs.cloudflare.com)
> > - Akamai (e3191.dscc.akamaiedge.net.0.1.cn.akamaiedge.net)
> >
> > And I'm not getting any traction with reporting the errors to those
> sites,
> > so it's frustrating in getting it properly fixed.
> >
> > While Akamai and cloudflare appear to be issues with their entries in
> > google dns, or with dnsmasq's validation of them being insecure domains,
> > the BofA issue appears to be an outright bad key.  And BofA isn't being
> > helpful (just a continual "we use ssl" sort of quasi-automated response).
> >
> > So I'm disabling it for now, or rather, falling back to using my ISP's
> dns
> > servers, which don't support DNSSEC at this time.  I'll be periodically
> > turning it back on, but too much is broken (mainly due to the cdns) to be
> > able to rely on it at this time.
> >
> > -Aaron
>
> Ditto. I was holding out, but performance was much worse, many websites
> would load poorly and got complaints from many errors from my customers
> (family).
>

[-- Attachment #2: Type: text/html, Size: 2221 bytes --]