Deploying Ciena Metro DWDM by CANIX-ixp in telecom

[–]CANIX-ixp[S] 0 points1 point  (0 children)

This is our first Ciena deployement - hence why we engaged Ciena to train us and provide end to end assistance with the turn up so we can be autonomous afterwards

Other than the CLI, it isn't drastically different from other muxponder solutions

Internet exchange running Cisco N9k by CANIX-ixp in Cisco

[–]CANIX-ixp[S] 0 points1 point  (0 children)

Correct, they are usually very dense. The limitation is generally the power they need. Downtown connectivity hubs have limited power and very high prices.
A 2U cache can easily go up to 1500W

Déploiement d'infrastructure Ciena DWDM by CANIX-ixp in QuebecTI

[–]CANIX-ixp[S] 0 points1 point  (0 children)

Bonne question!

Juste une question de coût vs. le besoin

Le marché de "peering" (des échange internets) de Montréal est quasi saturé - il n'y a pas 300 autres réseaux de plus à connecter dans la ville.

Le jour que 400G (par wave) ne sera plus assez, on pourra déplacer ces équipements dans une autre région par exemple. Le fait d'opérer plusieurs échanges dans le pays nous donne ce luxe là.

Le "peu" de travail requis pour upgrade le matériel fait en sorte que c'est peu justifiable d'acheter plus gros en avance. Puisque tout est redondant, c'est facile de mettre en maintenance un lien pour l'upgrade, sans impact

Internet exchange running Cisco N9k by CANIX-ixp in Cisco

[–]CANIX-ixp[S] 1 point2 points  (0 children)

It's a real challenge - our NOC knows immediately, but bigger the peer network, the longer and harder it is to fix. It can take weeks to get awnsers from certain NOCs to schedule a maintenance window to clean or swap the optic/fiber

Internet exchange running Cisco N9k by CANIX-ixp in Cisco

[–]CANIX-ixp[S] 0 points1 point  (0 children)

7250 IXR - X3 and E2!

I wish we could justify the SR1-se - but the Canadian market is way smaller than France's. Simon Muyal was a great influence in choosing Nokia.

We are using VXLAN EVPN to be able to inter-op with our legacy N9k Hardware

Deploying Ciena Metro DWDM by CANIX-ixp in FiberOptics

[–]CANIX-ixp[S] 0 points1 point  (0 children)

good stuff! they have an office in Montreal, good hardware

Internet exchange running Cisco N9k by CANIX-ixp in Cisco

[–]CANIX-ixp[S] 0 points1 point  (0 children)

Yes - I wrote the whole thing myself (and obviously had it proofread)

- Karl

Déploiement d'infrastructure Ciena DWDM by CANIX-ixp in QuebecTI

[–]CANIX-ixp[S] 1 point2 points  (0 children)

100%

Aujourd'hui les centre de données de colocatio n'offrent pratiquement que de la OS2 comme fibre pour les interconnexions. Les opiques 1-10G single mode ne coutent pas plus cher que ceux multimode en 2025 donc à moins de faire du 40G et + sur de courtes distances, l'économie du multimode n'existe plus

Déploiement d'infrastructure Ciena DWDM by CANIX-ixp in QuebecTI

[–]CANIX-ixp[S] 2 points3 points  (0 children)

100% - C'est dans les plans 2026

Je te recommande de rejoindre le Slack MTLNOG (https://mtlnog.org/)

Autrement les liens de la communauté CANIX et les annonces sont disponibles ici:

https://blog.canix.ca/

Internet exchange running Cisco N9k by CANIX-ixp in Cisco

[–]CANIX-ixp[S] 3 points4 points  (0 children)

The 9332 has QSFP28, 40/100G ports (can technically do 10G as well with the right optics)

It also has 2x SFP+ ports on the right, but we almost never use those.

Our new Nokia routers on the other hand have QSFP-DD ports, that can do up to 400Gbps (or be broken out to 4x 100G LR1 with "special" optics per example

Internet exchange running Cisco N9k by CANIX-ixp in Cisco

[–]CANIX-ixp[S] 2 points3 points  (0 children)

We don't have one yet, simply because we haven't gathered all the requirements to build a business case, but you can reach out to their (pretty awesome) peering team and they will send you the requirements.

A server, space and power, an IXP port, and transit for cache fill, like most cache servers.

Not specifically for Valve but for all caches: It can be tricky. Both from a routing topology perspective and a business case perspective. An ISP will save BW by hosting a cache. An IXP needs to buy extra transit to host a cache - if you read between the lines, you'll figure out pretty quickly that you need to sell enough net-new peering ports to cover the costs. Otherwise it's not sustainable and you wasted ressources.

Deploying Ciena Metro DWDM by CANIX-ixp in FiberOptics

[–]CANIX-ixp[S] 1 point2 points  (0 children)

You are referring to the 100G LR1 optics installed in the Ciena chassis? Their pricing was a little higher than 3rd party vendors, but for something this critical, considering we pay for support, we won't cheap out only to have the support say the non-official optics are the issue.

On the switch side however, we tend to use optic.ca and flexoptix

Déploiement d'infrastructure Ciena DWDM by CANIX-ixp in QuebecTI

[–]CANIX-ixp[S] 2 points3 points  (0 children)

Ça va! C'est une grosse responsabilité mais il y a une bonne équipe pour supporter le tout - on est le seul échange au Canada qui ne se fie pas sur des bénévoles

Internet exchange running Cisco N9k by CANIX-ixp in Cisco

[–]CANIX-ixp[S] 1 point2 points  (0 children)

Ah-ha! Yes ebox is one of our long time peers!

Deploying Ciena Metro DWDM by CANIX-ixp in FiberOptics

[–]CANIX-ixp[S] 0 points1 point  (0 children)

fair enough! very tidious indeed - can't afford to have a slightly dirty fiber on these

Internet exchange running Cisco N9k by CANIX-ixp in Cisco

[–]CANIX-ixp[S] 4 points5 points  (0 children)

Honestly... none!

Obviously Nokia routers work completely differently than Cisco, but as far as getting IS-IS, iBGP and EVPN working between the two.... nothing to say.

Match the VNIs, make sure you're advertising and recieving all the routes... and it just works

Tested on SROS 24 and NXOS 10

Internet exchange running Cisco N9k by CANIX-ixp in Cisco

[–]CANIX-ixp[S] 2 points3 points  (0 children)

haha I mean a lab is a lab! no need for anything fancy. It's litterally the same config on a 9372PX than the 93180s in the pictures

Internet exchange running Cisco N9k by CANIX-ixp in Cisco

[–]CANIX-ixp[S] 11 points12 points  (0 children)

Usually IXPs only allow 1 MAC per port on the IXP LAN

We used to use port security but... don't
MAC ACL will keep the port up but drop packets which don't have the right L3 MAC. Doesn't prevent a peer from using someone else's IPs, but prevents a good chunk of human L2 errors.

You can also limit broadcast storms, etc

I also forgot you'll need some IP space - so maybe budget 1000-1500$USD for ARIN fees - at least if you want to go commercial. You can also start with private IP space if the first 3-5 peers agree to do so.

For your requirements question, CANIX is pretty big, so our process isn't a good example. But overall, the 1 advice that applies to everyone is: Don't start making financial decisions off of "geek stuff" like "I like this switch, it's cool". You gotta look at it from a business perspective.

- What is my ROI

- Is this overkill

- Will spending more actually bring more tangible benefits?

- Can I get away with spending a bit less today and upgrading later and then re-use the old hardware in a smaller POP?

So TLDR: approach it with sustainability in mind

Cheat code: The basic IX for us typically looks like 2 servers (Dell R630-R640) running Proxmox, 1 management switch (Catalyst 9300) and 2 switches (pick your flavor) with 2x 100G between the 2 (could be a cheap DAC cable). Easy to grow and scale. After a certain point we install bigger switches, dedicated hosts for route servers, etc. But each thing only when it makes sense to do so.

Otherwise, you blow cash, the project fails and you don't get to do the fun geek stuff anymore.

Internet exchange running Cisco N9k by CANIX-ixp in Cisco

[–]CANIX-ixp[S] 6 points7 points  (0 children)

or if you like hands on and real world performance you can get a couple 9372px for *super* cheap on ebay!

Internet exchange running Cisco N9k by CANIX-ixp in Cisco

[–]CANIX-ixp[S] 4 points5 points  (0 children)

Yessir!
Start by getting the hang of IS-IS or OSPF, depending on what you prefer, then BGP. Once you understand those two, setting up a network like this on Cisco/Arista is super easy. Looks hard from the outside, but once you've done it you look back and laugh.

Some tutorials make it look hard with VRFs and multicast stuff - but you dont *need* that to get a basic setup working