[Cerowrt-devel] 10GigE nics and SFP+ modules?

[Cerowrt-devel] 10GigE nics and SFP+ modules?

[Dave Taht]

Given that the rangeley series of processors apparently has support in
openwrt already, I picked up one last week.

http://www.amazon.com/Supermicro-Atom-C2758-Motherboards-MBD-A1SRI-2758F-O/dp/B00FM4M7TQ/ref=sr_1_1?ie=UTF8&qid=undefined&sr=8-1&keywords=C2758

I'm still looking for a good case for it - the first rack-mount case I
got had mini-itx mounts but a non-ATX power supply, suggestions? Only
need 50 watts or less of power supply...

Where my brain falls off a cliff is sorting through the SFP+ options.
I've been told to seek out the intel chipset cards as the best
supported under linux, so would this be good? Are there other options?

http://www.amazon.com/Intel-Gigabit-Dual-Server-Adapter/dp/B001AGFXTQ/ref=sr_1_10?ie=UTF8&qid=1410027344&sr=8-10&keywords=10GigE+nic

Or these?

http://www.amazon.com/Intel-Ethernet-X520-SR2-Server-Adapter/dp/B002I9JCQY/ref=pd_cp_pc_0

I'd like single mode (20km+) fiber support, but also to try whatever
mode is more common in DCs... ?

http://www.amazon.com/Intel-Ethernet-ETHERNET-MODULE-10GBase-SR/dp/B009KZNWE2/ref=sr_1_1?s=electronics&ie=UTF8&qid=1410028448&sr=1-1&keywords=Intel+SFP%2B

These are the most expensive items (with the exception of snapon) I've
ever bought on behalf of bufferbloat.net, and it feels weird to be
doing this after fighting all month with 100mbit issues... but I've
been dying to get some 10GigE data, so...

-- 
Dave Täht

https://www.bufferbloat.net/projects/make-wifi-fast

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 3270 bytes --]


I have been happy with the PicoPSU power supplies which are tiny ATX PSU's that go up to 160 W.
They take 12V input, which can be supplied by an external brick or a small 12V power supply of the sort used to supply power to lighting circuits (I use a Meanwell NES-350-12 to power 3 boards with individual PicoPSU's.
 
For home experiments I have been using the less expensive X520-DA2 Dual copper Cat6 10 GigE boards from Intel, at about $500 per card. They are well supported in Linux.  SFP+ is the way to go for fiber.
 
Also, Netgear sells a small 8 port copper interface 10 GigE switch for < $1000. http://www.netgear.com/business/products/switches/unmanaged-plus/10g-plus-switch.aspx It's an 8-port "desktop" switch, which means unmanaged, etc.  I haven't tried that.  For various reasons my home machines are connected directly with each other, without a switch, but you want a switch, and this is far less pricey.
 
I think the copper option might get you 100m of distance, but I'm not sure.
 
All that said, if you want high end stuff, you might want to go for the X540 fiber product line and switches that seem to be $5,000 or more, plus the cost of the SFP+ cables, ...  At least a factor of 10 more expensive, for better latency in the switch, fancy management features in the switch and more reach in the cabling for less electrical power.  Depends on what you want.
 
We use the higher end stuff at work.
 
 
 
 


On Saturday, September 6, 2014 2:36pm, "Dave Taht" <dave.taht@gmail.com> said:



> Given that the rangeley series of processors apparently has support in
> openwrt already, I picked up one last week.
> 
> http://www.amazon.com/Supermicro-Atom-C2758-Motherboards-MBD-A1SRI-2758F-O/dp/B00FM4M7TQ/ref=sr_1_1?ie=UTF8&qid=undefined&sr=8-1&keywords=C2758
> 
> I'm still looking for a good case for it - the first rack-mount case I
> got had mini-itx mounts but a non-ATX power supply, suggestions? Only
> need 50 watts or less of power supply...
> 
> Where my brain falls off a cliff is sorting through the SFP+ options.
> I've been told to seek out the intel chipset cards as the best
> supported under linux, so would this be good? Are there other options?
> 
> http://www.amazon.com/Intel-Gigabit-Dual-Server-Adapter/dp/B001AGFXTQ/ref=sr_1_10?ie=UTF8&qid=1410027344&sr=8-10&keywords=10GigE+nic
> 
> Or these?
> 
> http://www.amazon.com/Intel-Ethernet-X520-SR2-Server-Adapter/dp/B002I9JCQY/ref=pd_cp_pc_0
> 
> I'd like single mode (20km+) fiber support, but also to try whatever
> mode is more common in DCs... ?
> 
> http://www.amazon.com/Intel-Ethernet-ETHERNET-MODULE-10GBase-SR/dp/B009KZNWE2/ref=sr_1_1?s=electronics&ie=UTF8&qid=1410028448&sr=1-1&keywords=Intel+SFP%2B
> 
> These are the most expensive items (with the exception of snapon) I've
> ever bought on behalf of bufferbloat.net, and it feels weird to be
> doing this after fighting all month with 100mbit issues... but I've
> been dying to get some 10GigE data, so...
> 
> --
> Dave Täht
> 
> https://www.bufferbloat.net/projects/make-wifi-fast
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 

[-- Attachment #2: Type: text/html, Size: 5517 bytes --]

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Dave Taht]

On Sat, Sep 6, 2014 at 12:05 PM,  <dpreed@reed.com> wrote:
> I have been happy with the PicoPSU power supplies which are tiny ATX PSU's
> that go up to 160 W.

I love those too, it is really remarkable how they have shrunk the electronics,
but in this case having an internal power supply seems best. Also, as it
appears the 10GigE card has a large heat sink, and the rangely only rated
to 40C, having more cooling in a rackmount case seems to be a good idea.

I haven't a clue what to do with 8 cores (20W TDP), my original intent was
to get the lower end 4 core box (9.5W TDP), but was unable to find one.

>
> They take 12V input, which can be supplied by an external brick or a small
> 12V power supply of the sort used to supply power to lighting circuits (I
> use a Meanwell NES-350-12 to power 3 boards with individual PicoPSU's.

that's clever.

>
>
>
> For home experiments I have been using the less expensive X520-DA2 Dual
> copper Cat6 10 GigE boards from Intel, at about $500 per card. They are well
> supported in Linux.  SFP+ is the way to go for fiber.
>
>
>
> Also, Netgear sells a small 8 port copper interface 10 GigE switch for <
> $1000.
> http://www.netgear.com/business/products/switches/unmanaged-plus/10g-plus-switch.aspx
> It's an 8-port "desktop" switch, which means unmanaged, etc.  I haven't
> tried that.  For various reasons my home machines are connected directly
> with each other, without a switch, but you want a switch, and this is far
> less pricey.

Well, while measuring what goes on in a 10GigE switch nowadays is on
my todo list, I don't know to what extent I can drive 10GigE with the gear
I have available in the first place, and my budget is already busted
as it is. First up is getting a useful build of openwrt for it.

(getting a basic build of openwrt head (chaos calmer) was a snap)

I've also
long desired to have an x86 build in general to run in a virtualized
environment to do gui work better.

> I think the copper option might get you 100m of distance, but I'm not sure.

I may have a shot at some dark fiber over a 20km distance, so my hope
is to connect two DCs together ultimately. Or I might run a few hundred
meters of fiber around the campus.

> All that said, if you want high end stuff, you might want to go for the X540
> fiber product line and switches that seem to be $5,000 or more, plus the

The X540 card is not that much more expensive than the X520.

> cost of the SFP+ cables,

Longer runs are very expensive and it's not clear what sort of
equipment is required to cut and connect bulk cable.

>...  At least a factor of 10 more expensive, for
> better latency in the switch, fancy management features in the switch and
> more reach in the cabling for less electrical power.  Depends on what you
> want.
>
>
>
> We use the higher end stuff at work.
>
>
>
>
>
>
>
>
>
>
>
> On Saturday, September 6, 2014 2:36pm, "Dave Taht" <dave.taht@gmail.com>
> said:
>
>> Given that the rangeley series of processors apparently has support in
>> openwrt already, I picked up one last week.
>>
>>
>> http://www.amazon.com/Supermicro-Atom-C2758-Motherboards-MBD-A1SRI-2758F-O/dp/B00FM4M7TQ/ref=sr_1_1?ie=UTF8&qid=undefined&sr=8-1&keywords=C2758
>>
>> I'm still looking for a good case for it - the first rack-mount case I
>> got had mini-itx mounts but a non-ATX power supply, suggestions? Only
>> need 50 watts or less of power supply...
>>
>> Where my brain falls off a cliff is sorting through the SFP+ options.
>> I've been told to seek out the intel chipset cards as the best
>> supported under linux, so would this be good? Are there other options?
>>
>>
>> http://www.amazon.com/Intel-Gigabit-Dual-Server-Adapter/dp/B001AGFXTQ/ref=sr_1_10?ie=UTF8&qid=1410027344&sr=8-10&keywords=10GigE+nic
>>
>> Or these?
>>
>>
>> http://www.amazon.com/Intel-Ethernet-X520-SR2-Server-Adapter/dp/B002I9JCQY/ref=pd_cp_pc_0
>>
>> I'd like single mode (20km+) fiber support, but also to try whatever
>> mode is more common in DCs... ?
>>
>>
>> http://www.amazon.com/Intel-Ethernet-ETHERNET-MODULE-10GBase-SR/dp/B009KZNWE2/ref=sr_1_1?s=electronics&ie=UTF8&qid=1410028448&sr=1-1&keywords=Intel+SFP%2B
>>
>> These are the most expensive items (with the exception of snapon) I've
>> ever bought on behalf of bufferbloat.net, and it feels weird to be
>> doing this after fighting all month with 100mbit issues... but I've
>> been dying to get some 10GigE data, so...
>>
>> --
>> Dave Täht
>>
>> https://www.bufferbloat.net/projects/make-wifi-fast
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>



-- 
Dave Täht

https://www.bufferbloat.net/projects/make-wifi-fast

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Wes Felter]

The Intel X710 just came out today; it's a little cheaper. (Note that 
the X710 is 10G and the XL710 is 40G because XL is 40 in Roman numerals.)

http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2227&idcategory=0

(I've never bought from Colfax, but they carry a lot of "enterprise" 
equipment that isn't available on NewEgg/Amazon.)

There are super-cheap optics and twinax available from China. You may 
have to hack the Intel driver since they're not "supported".

http://www.fiberstore.com/c/10g-sfp+_63
http://www.fiberstore.com/c/10g-sfp+-cables_1115

Or you could just get an account at the Snabb Lab.

-- 
Wes Felter
IBM Research - Austin

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Joel Wirāmu Pauling]

Just a head's up I have had issues with the X5xx intel SFP+ optics
interoperating with other vendors. This may not be an issue for your
deployment.

If you want good interop then Broadcom based optics seem to be the
best bet at the moment.

On 9 September 2014 12:09, Wes Felter <wmf@felter.org> wrote:
> The Intel X710 just came out today; it's a little cheaper. (Note that the
> X710 is 10G and the XL710 is 40G because XL is 40 in Roman numerals.)
>
> http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2227&idcategory=0
>
> (I've never bought from Colfax, but they carry a lot of "enterprise"
> equipment that isn't available on NewEgg/Amazon.)
>
> There are super-cheap optics and twinax available from China. You may have
> to hack the Intel driver since they're not "supported".
>
> http://www.fiberstore.com/c/10g-sfp+_63
> http://www.fiberstore.com/c/10g-sfp+-cables_1115
>
> Or you could just get an account at the Snabb Lab.
>
> --
> Wes Felter
> IBM Research - Austin
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Joel Wirāmu Pauling]

Note I know whilst you are talking copper. IME 10Gbit copper is
nothing but a hassle. You are lucky to get 30Metres out of a run
without Negotiation issues.

I encourage you strongly to look to using Optical LX style fibre if
you are moving to 10G. The cost of structured cabling and the length
of runs is a lot cheaper than Cat6a/7 runs needed for 10gbit copper.

On 9 September 2014 16:03, Joel Wirāmu Pauling <joel@aenertia.net> wrote:
> Just a head's up I have had issues with the X5xx intel SFP+ optics
> interoperating with other vendors. This may not be an issue for your
> deployment.
>
> If you want good interop then Broadcom based optics seem to be the
> best bet at the moment.
>
> On 9 September 2014 12:09, Wes Felter <wmf@felter.org> wrote:
>> The Intel X710 just came out today; it's a little cheaper. (Note that the
>> X710 is 10G and the XL710 is 40G because XL is 40 in Roman numerals.)
>>
>> http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2227&idcategory=0
>>
>> (I've never bought from Colfax, but they carry a lot of "enterprise"
>> equipment that isn't available on NewEgg/Amazon.)
>>
>> There are super-cheap optics and twinax available from China. You may have
>> to hack the Intel driver since they're not "supported".
>>
>> http://www.fiberstore.com/c/10g-sfp+_63
>> http://www.fiberstore.com/c/10g-sfp+-cables_1115
>>
>> Or you could just get an account at the Snabb Lab.
>>
>> --
>> Wes Felter
>> IBM Research - Austin
>>
>>
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 2542 bytes --]


I agree with you if you have to install Cat6a and Cat7 structured wiring in a building! What a nightmare trying to find contractors who can meet spec on junction boxes, etc. and do the right testing.  Every connector on the path is problematic.

But for "casual" home networking use or research lab experiments, copper is fine.  That was my point.  I think in a "rack-scale" datascenter networking situation. probably better to go with fiber, but copper costs are attractive.
 
 


On Tuesday, September 9, 2014 12:06am, "Joel Wirāmu Pauling" <joel@aenertia.net> said:



> Note I know whilst you are talking copper. IME 10Gbit copper is
> nothing but a hassle. You are lucky to get 30Metres out of a run
> without Negotiation issues.
> 
> I encourage you strongly to look to using Optical LX style fibre if
> you are moving to 10G. The cost of structured cabling and the length
> of runs is a lot cheaper than Cat6a/7 runs needed for 10gbit copper.
> 
> On 9 September 2014 16:03, Joel Wirāmu Pauling <joel@aenertia.net>
> wrote:
> > Just a head's up I have had issues with the X5xx intel SFP+ optics
> > interoperating with other vendors. This may not be an issue for your
> > deployment.
> >
> > If you want good interop then Broadcom based optics seem to be the
> > best bet at the moment.
> >
> > On 9 September 2014 12:09, Wes Felter <wmf@felter.org> wrote:
> >> The Intel X710 just came out today; it's a little cheaper. (Note that the
> >> X710 is 10G and the XL710 is 40G because XL is 40 in Roman numerals.)
> >>
> >>
> http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2227&idcategory=0
> >>
> >> (I've never bought from Colfax, but they carry a lot of "enterprise"
> >> equipment that isn't available on NewEgg/Amazon.)
> >>
> >> There are super-cheap optics and twinax available from China. You may
> have
> >> to hack the Intel driver since they're not "supported".
> >>
> >> http://www.fiberstore.com/c/10g-sfp+_63
> >> http://www.fiberstore.com/c/10g-sfp+-cables_1115
> >>
> >> Or you could just get an account at the Snabb Lab.
> >>
> >> --
> >> Wes Felter
> >> IBM Research - Austin
> >>
> >>
> >> _______________________________________________
> >> Cerowrt-devel mailing list
> >> Cerowrt-devel@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 

[-- Attachment #2: Type: text/html, Size: 3741 bytes --]

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Dave Taht]

On Tue, Sep 9, 2014 at 9:26 AM,  <dpreed@reed.com> wrote:
> I agree with you if you have to install Cat6a and Cat7 structured wiring in
> a building! What a nightmare trying to find contractors who can meet spec on
> junction boxes, etc. and do the right testing.  Every connector on the path
> is problematic.
>
> But for "casual" home networking use or research lab experiments, copper is
> fine.  That was my point.  I think in a "rack-scale" datascenter networking
> situation. probably better to go with fiber, but copper costs are
> attractive.
>

I was absolutely delighted to see all the tutorials on this site, example:

http://www.fiberstore.com/How-to-install-aerial-fiber-optic-cables-aid-386.html

Here I was thinking I'd have to go write another
http://www.rage.net/wireless/wireless-howto.html ...

I have been a big advocate of municipal fiber, but have not had a good
handle on the costs involved. I have generally heard it's 80% labor,
20% materials, and the cost of fiber cabling alone, at least at retail
is kind of intimidating.

I see a lot of tradeoffs being made with GPON that for example, single
mode fiber deployments such as amsterdam's didn't make.
http://arstechnica.com/tech-policy/2010/03/how-amsterdam-was-wired-for-open-access-fiber/
- and kind of concerned about the "bumps in the wire" there.

and several costs there have dropped significantly, routerboard is
making a SFP+ capable
5 port switch for like 50 dollars, and things like this
router-you-can-drop-off-a-truck, developed in africa...

http://shop.brck.com/brcks/brck-v1.html

I'm not so much interested in 10GigE in the home, as 10GigE or 1GigE
on a small campus, like  what I have here, which is presently a dozen+
directional APs spread over runs of 126-380 meters, outdoors. If I
were to pursue FTTY (Fiber To The Yurt!), what would the costs and
tradeoffs be? What would a reasonably redundant network look like?
Could you go with a meshier topology? How would you repair things
after an earthquake?

It's an interesting set of theoretical questions. Unlike some, I'd not
have certain ISPs lobbying to stop me from trying, the telephone poles
are locally owned, ground burial is not a problem, either, and the
labor is cheap.

Admittedly, now that bug 442 is seemingly fixed, I can finally bring
the reliability of the wifi here back up to a saner level, and deploy
multiple exit nodes to comcast and elsewhere via hnetd, so upgrading
to fiber throughout the campus is purely speculative at the moment, a
nice distraction from thinking further on fixing wifi.

(my original intent remains: just fiddling with 10GigE on a small test
setup, to see packet patterns at high rates)




>
>
>
>
>
>
> On Tuesday, September 9, 2014 12:06am, "Joel Wirāmu Pauling"
> <joel@aenertia.net> said:
>
>> Note I know whilst you are talking copper. IME 10Gbit copper is
>> nothing but a hassle. You are lucky to get 30Metres out of a run
>> without Negotiation issues.
>>
>> I encourage you strongly to look to using Optical LX style fibre if
>> you are moving to 10G. The cost of structured cabling and the length
>> of runs is a lot cheaper than Cat6a/7 runs needed for 10gbit copper.
>>
>> On 9 September 2014 16:03, Joel Wirāmu Pauling <joel@aenertia.net>
>> wrote:
>> > Just a head's up I have had issues with the X5xx intel SFP+ optics
>> > interoperating with other vendors. This may not be an issue for your
>> > deployment.
>> >
>> > If you want good interop then Broadcom based optics seem to be the
>> > best bet at the moment.
>> >
>> > On 9 September 2014 12:09, Wes Felter <wmf@felter.org> wrote:
>> >> The Intel X710 just came out today; it's a little cheaper. (Note that
>> >> the
>> >> X710 is 10G and the XL710 is 40G because XL is 40 in Roman numerals.)
>> >>
>> >>
>>
>> http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2227&idcategory=0
>> >>
>> >> (I've never bought from Colfax, but they carry a lot of "enterprise"
>> >> equipment that isn't available on NewEgg/Amazon.)
>> >>
>> >> There are super-cheap optics and twinax available from China. You may
>> have
>> >> to hack the Intel driver since they're not "supported".
>> >>
>> >> http://www.fiberstore.com/c/10g-sfp+_63
>> >> http://www.fiberstore.com/c/10g-sfp+-cables_1115
>> >>
>> >> Or you could just get an account at the Snabb Lab.
>> >>
>> >> --
>> >> Wes Felter
>> >> IBM Research - Austin
>> >>
>> >>
>> >> _______________________________________________
>> >> Cerowrt-devel mailing list
>> >> Cerowrt-devel@lists.bufferbloat.net
>> >> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht

https://www.bufferbloat.net/projects/make-wifi-fast

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Wes Felter]

On 9/8/14, 11:06 PM, Joel Wirāmu Pauling wrote:

> I encourage you strongly to look to using Optical LX style fibre if
> you are moving to 10G. The cost of structured cabling and the length
> of runs is a lot cheaper than Cat6a/7 runs needed for 10gbit copper.

Twinax is cheaper than optical or base-T.

-- 
Wes Felter
IBM Research - Austin

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Dave Taht]

In this space also is:

http://linuxgizmos.com/quad-core-cortex-a15-soc-features-6mb-on-chip-ram/

I like that TI has also (finally) adopted an upstream-first kernel
policy. It is,
of course, unclear how their offloads work.


On Tue, Sep 9, 2014 at 12:06 PM, Wes Felter <wmf@felter.org> wrote:
> On 9/8/14, 11:06 PM, Joel Wirāmu Pauling wrote:
>
>> I encourage you strongly to look to using Optical LX style fibre if
>> you are moving to 10G. The cost of structured cabling and the length
>> of runs is a lot cheaper than Cat6a/7 runs needed for 10gbit copper.
>
>
> Twinax is cheaper than optical or base-T.
>
>
> --
> Wes Felter
> IBM Research - Austin
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht

https://www.bufferbloat.net/projects/make-wifi-fast

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 529 bytes --]

On Tue, 9 Sep 2014, Joel Wirāmu Pauling wrote:

> Just a head's up I have had issues with the X5xx intel SFP+ optics
> interoperating with other vendors. This may not be an issue for your
> deployment.

As far as I know, you also have to make sure to buy intel optics to put in 
intel NICs. This vendor lock-in is something we (as customers) have been 
able to beat out of Cisco, Juniper and the lot, but it seems Intel is 
still able to get their customer base to accept this.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Mikael Abrahamsson]

On Tue, 9 Sep 2014, Dave Taht wrote:

> I have been a big advocate of municipal fiber, but have not had a good
> handle on the costs involved. I have generally heard it's 80% labor,
> 20% materials, and the cost of fiber cabling alone, at least at retail
> is kind of intimidating.

I'd say this depends on the market, considering such things as "how much 
does it cost to dig up the street", labor cost per hour etc.

If you're looking at last-mile in dense high-rises, I'd say fiber cabling 
doesn't cost that much more than copper, but what costs is that for fiber, 
the operator has to provide a CPE and make sure there is power nearby. I 
did a procurement of this for my condo with 750 apartments and compared 
FTTH and CTTH and CTTH is cheaper, but also more bulky and requires to 
have switches all over the place, whereas FTTH means you can aggregate all 
apartments in a single place.

Cost of indoor copper cabling alone here in Sweden was around 100USD per 
apartment. Fiber was more expensive both in materials and labor, but not 
hugely so. What drives the cost of fiber was the cost of the media 
converters and the power cabling to the media converters or the 
alternative, cabling within apartments to get the fiber to where there was 
power and at the same time convenient for the user to have the media 
converter. All in all, in highrises, I'd say it's approximately 50% cost 
of labor+cabling for the physical cabling needed (power+fiber), and then 
another 50% for equipment (media converter and aggregation equipment). 
Total cost of install was around 150 USD per apartment for copper and 
300-400 USD per apartment for fiber.

For houses, this is of course very different, since digging is involved. 
Digging is very expensive. If you ever dig, put in ducts. Large ducts, and 
lots of them. Typical cost of FTTH in Sweden to houses, (and now we're 
talking fairly dense developments with 500-1000sqm lots next to each other 
with one house on each), is around 2500 USD per house.

> I see a lot of tradeoffs being made with GPON that for example, single
> mode fiber deployments such as amsterdam's didn't make.
> http://arstechnica.com/tech-policy/2010/03/how-amsterdam-was-wired-for-open-access-fiber/
> - and kind of concerned about the "bumps in the wire" there.

I don't like when people create their cable plant to match what GPON 
needs. It's done because of the illusion that long-haul fiber is 
expensive. It isn't, if you have to dig anyway. The difference in cost of 
a 12 fiber cable, and a 1000 fiber cable, isn't huge compared to the 
digging costs. Splicing a 1000 fiber cable isn't huge either. 
Point-to-point fiber cabling is the way to go. If you then decide to light 
it up using PON of some kind, fine, that's up to you, at least you have 
the flexibility to change technology in the future.

> and several costs there have dropped significantly, routerboard is
> making a SFP+ capable
> 5 port switch for like 50 dollars

URL?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Chuck Anderson]

> thank you all for sharing your knowledge!
> 
> In my life I've punched holes in coax, run arcnet, climbed mountains
> and tall buildings to run wifi P2P links and cut and crimped more
> twisted pair than I care to remember, but my exposure to fiber's
> methods has been sadly limited, til now.
> 
> I still don't quite have a grip on how and when to use various forms
> of cwdm or what wavelengths make the most sense, when...

"Normal" optics (sometimes called "grey") run at the standard
wavelength defined for whatever standard they are made for, for
example 1000BASE-SX runs at 850nm over multimode fiber, and
1000BASE-LX runs at 1310nm over singlemode fiber.  Similarly,
10GBASE-SR is 850nm and 10GBASE-LR is 1310nm.  These are the commonly
available wavelengths and would be what you would normally use unless
you have a specific reason to do otherwise.

You use CWDM or DWDM "colored" optics when you have a scarcity of
fiber strands and you need/want to run multiple signals or links
across them.  For example, for metro fiber or long haul fiber, you get
charged by how many strands you lease and how long the run is.  With
cheap passive CWDM or DWDM multiplexers, you can run multiple 1gig or
10gig Ethernet links across just 2 strands of metro fiber, or even a
single strand with "bidirectional" optical modules which use a
different wavelength for each direction.  As another example, if you
want to run Fibre Channel alongside Ethernet over the same strands you
can also do that with WDM.

Longer haul links may require more sophisticated active WDM gear and
amplifiers and can get very expensive.  Historically, these active WDM
systems also had (and may still have) "transponders" to convert from
the standard "grey" wavelengths 850nm or 1310nm installed in the
routers/switches to the WDM wavelengths used on the WDM links.  It is
becoming more common though to forego the expense of the transponders
and instead install the WDM optics directly into the routers &
switches via removeable SFP (1gig - 4gig) or SFP+ (8gig, 10gig)
modules.

The C in CWDM stands for Coarse, and the spectrum is divided into 4,
8, or less commonly 16 wavelengths, from 1270nm to 1610nm.  Here is a
good pictorial:

http://a2net.eu/images/dwdm.gif

The D in DWDM means Dense, and there are 40, 80, or more wavelengths
much more tightly packed together, usually wholly within the 1550nm
band.

More here:

http://a2net.eu/en/solutions-fiberoptics/

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Dave Taht]

On Wed, Sep 10, 2014 at 12:31 PM,  <dpreed@reed.com> wrote:
> I'm confused....
>
>
>
> SFP is not SFP+.   SFP carries at most 4.25 Gb/sec.   SFP+ works at >10
> Gb/sec.   So, it's not clear that the MikroTik is very useful in a 10 Gig
> world.  It's not immediately clear but VERY likely, that this is an "edge
> switch" that is intended for collapsing the GigE copper traffic onto a
> potentially bottlenecked GigE "local backbone".

Yes. And as such it's the cheapest I've ever seen. You can find slightly larger,
rack mount SFP capable switches, but nothing in this form factor that I know
of exists.

> Of course if you want to go from GigE fiber to GigE copper, that board might
> be useful.

In my case, I ended up trying to figure out what the costs would be
for 1G FTTY, which is a run of about 200 meters. This could be a
welcome change from the current 6 hop wifi mesh network in place to
get to the internet from where I sit. As fond as I am of wifi, even
with having high hopes to improve it further, sometimes a cable is
just better.

This was a wonderfully wide-ranging conversation overall, along the
way I learned about the differences between SFP, SFP+, and GSFTP, how
to crimp and run cables, and many of the relevant costs involved at
every layer in doing it...

thank you all for sharing your knowledge!

In my life I've punched holes in coax, run arcnet, climbed mountains
and tall buildings to run wifi P2P links and cut and crimped more
twisted pair than I care to remember, but my exposure to fiber's
methods has been sadly limited, til now.

I still don't quite have a grip on how and when to use various forms
of cwdm or what wavelengths make the most sense, when...



-- 
Dave Täht

https://www.bufferbloat.net/projects/make-wifi-fast

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Joel Wirāmu Pauling]

I have been heavily involved with the UFB (Ultrafast Broadband) PON
deployment here in New Zealand.

I am not sure how the regulated environment is playing out in Canada
(I am moving there in a month so I guess I will find out). But here
the GPON architecture is METH based and Layer2 only. Providers (RSP's)
are the ones responsible for asking for Handoffer buffer tweaks to the
LFC(local fibre companies; the layer 0-2 outfits-) which have mandated
targets for Latency (at most 4.5ms) accross their PON Access networks
to the Handover port.

Most of the time this has been to 'fix' Speedtest.net TCP based
results to report whatever Marketed service (100/30 For example) is in
everyones favourite site speedtest.net.

This has meant at least for the Chorus LFC regions where they use
Alcatel-Lucent 7450's as the handover/aggregation switches we have
deliberately introduced buffer bloat to please the RSP's - who
otherwise get whingy about customers whinging about speedtest not
showing 100/30mbit. Of course user education is 'too hard' .



-Joel





On 11 September 2014 08:03, Michael Richardson <mcr@sandelman.ca> wrote:
>
>     >> I don't like when people create their cable plant to match what GPON
>     >> needs.  It's done because of the illusion that long-haul fiber is
>     >> expensive. It isn't, if you have to dig anyway. The difference in cost
>     >> of a 12 fiber cable, and a 1000 fiber cable, isn't huge compared to
>     >> the digging costs.  Splicing a 1000 fiber cable isn't huge
>     >> either. Point-to-point fiber cabling is the way to go. If you then
>     >> decide to light it up using PON of some kind, fine, that's up to you,
>     >> at least you have the flexibility to change technology in the future.
>
> I went through a GPON install.  It started as a PtP install.
> The problem is that while we were installing for 90% of the way, there were
> a number of places where we could not: conduits under highways, etc.
>
> In *Canada* at least, fiber construction is a regulated activity, and
> people who own fiber are required to lease to others who want it.
>
> So, the GPON because of the 10% of places where we had to lease fiber,
> and leasing 2-3 strands is much easier than 1000.  In one case, my
> understanding was there was only a dozen strands installed, period, under,
> for instance, the Trans-Canada highway (quebec hwy 40/25 interchange was
> involved).
>
> I'm not otherwise very fond of the GPON stuff.  The "terminals"
> are too smart, and not flexible enough, and yes, they had hidden bufferbloat.
>
> --
> ]               Never tell me the odds!                 | ipv6 mesh networks [
> ]   Michael Richardson, Sandelman Software Works        | network architect  [
> ]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [
>
>
>

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Michael Richardson]

    >> I don't like when people create their cable plant to match what GPON
    >> needs.  It's done because of the illusion that long-haul fiber is
    >> expensive. It isn't, if you have to dig anyway. The difference in cost
    >> of a 12 fiber cable, and a 1000 fiber cable, isn't huge compared to
    >> the digging costs.  Splicing a 1000 fiber cable isn't huge
    >> either. Point-to-point fiber cabling is the way to go. If you then
    >> decide to light it up using PON of some kind, fine, that's up to you,
    >> at least you have the flexibility to change technology in the future.

I went through a GPON install.  It started as a PtP install.
The problem is that while we were installing for 90% of the way, there were
a number of places where we could not: conduits under highways, etc.

In *Canada* at least, fiber construction is a regulated activity, and 
people who own fiber are required to lease to others who want it.

So, the GPON because of the 10% of places where we had to lease fiber,
and leasing 2-3 strands is much easier than 1000.  In one case, my
understanding was there was only a dozen strands installed, period, under,
for instance, the Trans-Canada highway (quebec hwy 40/25 interchange was
involved). 

I'm not otherwise very fond of the GPON stuff.  The "terminals"
are too smart, and not flexible enough, and yes, they had hidden bufferbloat.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [ 
	

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 3116 bytes --]


I'm confused....
 
SFP is not SFP+.   SFP carries at most 4.25 Gb/sec.   SFP+ works at >10 Gb/sec.   So, it's not clear that the MikroTik is very useful in a 10 Gig world.  It's not immediately clear but VERY likely, that this is an "edge switch" that is intended for collapsing the GigE copper traffic onto a potentially bottlenecked GigE "local backbone".
 
Of course if you want to go from GigE fiber to GigE copper, that board might be useful.


On Wednesday, September 10, 2014 2:05pm, "Dave Taht" <dave.taht@gmail.com> said:



> On Wed, Sep 10, 2014 at 12:01 AM, Mikael Abrahamsson <swmike@swm.pp.se>
> wrote:
> 
> > I don't like when people create their cable plant to match what GPON needs.
> > It's done because of the illusion that long-haul fiber is expensive. It
> > isn't, if you have to dig anyway. The difference in cost of a 12 fiber
> > cable, and a 1000 fiber cable, isn't huge compared to the digging costs.
> > Splicing a 1000 fiber cable isn't huge either. Point-to-point fiber cabling
> > is the way to go. If you then decide to light it up using PON of some kind,
> > fine, that's up to you, at least you have the flexibility to change
> > technology in the future.
> >
> >> and several costs there have dropped significantly, routerboard is
> >> making a SFP+ capable
> >> 5 port switch for like 50 dollars
> >
> >
> > URL?
> 
> Nick weaver (of ICSI) just turned me onto them -
> 
> http://www.cloudrouterswitches.com/RB260GS.asp?gclid=Cj0KEQjw7b-gBRC45uLY_avSrdgBEiQAD3Olx8_iFXJ_xKjZInc2T54XEu5VyMsTe42Rla3GTRKrkwwaAu2M8P8HAQ
> 
> He also steered me to a nifty port mirroring POE passthrough device:
> 
> http://www.dual-comm.com/gigabit_port-mirroring-LAN_switch.htm
> 
> Haven't tried either yet, personally.
> 
> It turns out that both he and I are using the nearly same model nucs
> for load testing, with a standard 2.5inch sata 3 slot, and 2 mini pcie
> slots for a half length and full length wifi device.
> 
> "The D54250WYK1, which is a dual core, dual thread/core i5 based with
> 16 GB Ram and a 120 GB SSD). The only disadvantage is it doesn't
> include the "Vpro" lights out management suite present in the 3rd gen
> model."
> 
> I am using the i3 versions. I'd written up a review of the one without
> the 2.5 inch slot here:
> 
> http://snapon.lab.bufferbloat.net/~cero2/nuc-to-puck/results.html
> 
> and later upgraded to the one with the slot, as sata is faster than
> msata, and the best wifi (atheros ath9k and ath10k) cards are all full
> length).
> 
> These have e1000e cards in them, which support BQL under linux 3.6 and
> later, and although the (i3 at least) can't drive gigE to saturation
> without TSO, they have been quite nice and quite quiet so far, and
> have been giving solid results. They are a really good desktop, too,
> under linux, and mounting them on the back of the monitor (or, as I
> do, on a pegboard), is helpful too.
> 
> 
> >
> > --
> > Mikael Abrahamsson email: swmike@swm.pp.se
> 
> 
> 
> --
> Dave Täht
> 
> https://www.bufferbloat.net/projects/make-wifi-fast
> 

[-- Attachment #2: Type: text/html, Size: 4494 bytes --]

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Nicholas Weaver]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]


On Sep 10, 2014, at 11:05 AM, Dave Taht <dave.taht@gmail.com> wrote:
> http://www.cloudrouterswitches.com/RB260GS.asp?gclid=Cj0KEQjw7b-gBRC45uLY_avSrdgBEiQAD3Olx8_iFXJ_xKjZInc2T54XEu5VyMsTe42Rla3GTRKrkwwaAu2M8P8HAQ

I haven't powered mine on to play with in details yet, this is sitting on my desk.

> 
> He also steered me to a nifty port mirroring POE passthrough device:
> 
> http://www.dual-comm.com/gigabit_port-mirroring-LAN_switch.htm

This I use all the time, I've got my desktop traffic passing through one that is hooked up to my NUC/Bro Box.

--
Nicholas Weaver                  it is a tale, told by an idiot,
nweaver@icsi.berkeley.edu                full of sound and fury,
510-666-2903                                 .signifying nothing
PGP: http://www1.icsi.berkeley.edu/~nweaver/data/nweaver_pub.asc


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: [Cerowrt-devel] 10GigE nics and SFP+ modules?

[Dave Taht]

On Wed, Sep 10, 2014 at 12:01 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> I don't like when people create their cable plant to match what GPON needs.
> It's done because of the illusion that long-haul fiber is expensive. It
> isn't, if you have to dig anyway. The difference in cost of a 12 fiber
> cable, and a 1000 fiber cable, isn't huge compared to the digging costs.
> Splicing a 1000 fiber cable isn't huge either. Point-to-point fiber cabling
> is the way to go. If you then decide to light it up using PON of some kind,
> fine, that's up to you, at least you have the flexibility to change
> technology in the future.
>
>> and several costs there have dropped significantly, routerboard is
>> making a SFP+ capable
>> 5 port switch for like 50 dollars
>
>
> URL?

Nick weaver (of ICSI) just turned me onto them -

http://www.cloudrouterswitches.com/RB260GS.asp?gclid=Cj0KEQjw7b-gBRC45uLY_avSrdgBEiQAD3Olx8_iFXJ_xKjZInc2T54XEu5VyMsTe42Rla3GTRKrkwwaAu2M8P8HAQ

He also steered me to a nifty port mirroring POE passthrough device:

http://www.dual-comm.com/gigabit_port-mirroring-LAN_switch.htm

Haven't tried either yet, personally.

It turns out that both he and I are using the nearly same model nucs
for load testing, with a standard 2.5inch sata 3 slot, and 2 mini pcie
slots for a half length and full length wifi device.

"The D54250WYK1, which is a dual core, dual thread/core i5 based with
16 GB Ram and a 120 GB SSD).  The only disadvantage is it doesn't
include the "Vpro" lights out management suite present in the 3rd gen
model."

I am using the i3 versions. I'd written up a review of the one without
the 2.5 inch slot here:

http://snapon.lab.bufferbloat.net/~cero2/nuc-to-puck/results.html

and later upgraded to the one with the slot, as sata is faster than
msata, and the best wifi (atheros ath9k and ath10k) cards are all full
length).

These have e1000e cards in them, which support BQL under linux 3.6 and
later, and although the (i3 at least) can't drive gigE to saturation
without TSO, they have been quite nice and quite quiet so far, and
have been giving solid results. They are a really good desktop, too,
under linux, and mounting them on the back of the monitor (or, as I
do, on a pegboard), is helpful too.


>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 
Dave Täht

https://www.bufferbloat.net/projects/make-wifi-fast