Servers reused from a previous life carried old volume-manager metadata that made the installer abort halfway. Erasing every disk is now a step in the standard install sequence, which fixes that — and means a machine that enters service carries nothing from whatever ran on it before.
— Cloud platform
New hardware is discovered, inventoried, wiped and installed without anyone touching it, driven over the management controller. It shortens the time between a server arriving and capacity being available, and it makes every machine come up identically instead of however the last person did it by hand.
— Cloud platform
A second availability zone is being built. Its network foundation is now in service: two independent 100 Gbit/s uplinks to the border router, each carrying traffic at the same time through BGP multipath. If one link, optic or cable fails, the other keeps forwarding and no session is interrupted. The zone also has its own public address block, so customers who take resources in both zones get addresses that route between them.
— Cloud platform
Direct peering in Zurich over the exchange's route servers, which shortens the path to Swiss networks that peer there instead of sending the traffic through a transit carrier abroad. Brings us to seven exchanges and 260 Gbit/s of exchange port capacity.
— Network — AS43141
Any site can now show current, accurate facts about the network — carriers, internet exchanges, installed capacity, address space — without anyone keeping them up to date by hand. Two lines of HTML for the ready-made panel, or read the JSON and lay it out yourself. It refreshes hourly and states when it was built, so a number on someone else\s page is never older than the hour.
— Network — AS43141
It listed only the carriers that had an active measurement probe attached, which quietly understated the network to anyone reading it. All four now appear, ordered by installed capacity, and the list is derived from the same table as the capacity total so the two cannot drift apart.
— Network — AS43141
The numbers came from a single router probing a single anycast address. "1 ms" therefore said that destination was close from that one point, not that the carrier was fast, and placed side by side the figures invited a comparison the measurement cannot support. They also said nothing about the sites where most of our capacity sits. We would rather publish less than publish a number that does not survive being checked. The measurement stays where it belongs, choosing routes.
— Network — AS43141
The endpoints ran through session handling, so every response carried a cookie — and many shared caches and CDNs refuse to store a response with a cookie no matter how cacheable it claims to be. They now set none. Sites embedding our data serve it from their own cache, and keep serving it for up to 24 hours if our host is unreachable.
— Network — AS43141
They were measured from a single router towards an anycast address, so they described that path and not the carrier. Removed rather than dressed up. If you want to know the path to you, ask us for a trace from the site you care about.
— Network — AS43141
The institute built a live feed of the AS43141 data — capacity, exchanges, carriers, address space — so the numbers on this site are read rather than typed. Three versions in one day, each one more precise than the last.
— Network — AS43141
The limit was counted against the shared token the site uses, not against each visitor, so a busy minute would have turned into refusals for everyone — sign-ups included. Found during a check after launch, before anyone hit it.
A mailbox for vulnerability reports, first in our security.txt ahead of the contact form. For a CSIRT at three in the morning, a form is not a channel.
The Security page now states the schedule, the retention, the second copy — and the limit: both copies are in the same datacentre, so it does not yet cover losing the site. An off-site copy is on the list.
Capacity, exchanges, carriers and peers, read every hour from the public looking glass of the network we are part of, with the time of the last reading next to them. If they drop, you will watch them drop.
Turn it on under Account: your password, then a six-digit code from any authenticator app. Eight single-use recovery codes for the day you lose the phone. Not SMS — a phone number can be moved to someone else with a phone call, which is the attack this is meant to stop.
— Cloud platform
It now waits instead of refusing.
— Cloud platform
Inclusion depended on a flag someone had to set by hand, which is a design that eventually forgets something. It no longer does.
— Cloud platform
— Cloud platform
The node was advertising itself as healthy for work it could not do. It is now taken out of rotation when that check fails.
— Cloud platform
Cluster API endpoints now live under a name that belongs to this service, instead of an internal platform domain.
— Kubernetes platform
The platform team traced it to a missing internal endpoint record and to the management tunnel pointing at an address that was not reachable. Both fixed within a day of being reported.
— Kubernetes platform
It used to answer "Unexpected server response: 500", which is a way of saying nothing. Now it says which part is not ready.
A single entry point for alerts, from us or from anyone who monitors something that touches us, fanned out to the people on call. If you see something wrong before we do, we want it to be easy to tell us.
It arrives when the cluster answers, not when the order is accepted, and the button takes you to the cluster rather than to a login page.
Clusters and public addresses are charged for what you have allocated, by the hour, on the same wallet as everything else. What you see on the card is what is being counted.
Groups of workers with their own size, labels and taints, so a memory-hungry job and a small web front end can live in the same cluster without paying for the larger machine twice.
Useful when the same customer keeps environments in separate projects and still wants them to talk without going out to the internet.
— Cloud platform
Every machine moved onto a private range with room to grow. Planned, rehearsed and done with the old addresses kept alive until the new ones carried everything.
kubectl against your own cluster from the portal, with a session that lives as long as the tab. Useful precisely on the day something is broken and you do not have your laptop.
Nodes, pods that are not happy, deployments short of replicas, recent warnings. Read-only, and the paths are chosen by us: your secrets are not reachable through this, because there is no call that asks for them.
Three separate reasons stacked on top of each other, which is why it looked like one stubborn bug. It now returns a token, the CA hash and the endpoint, ready to paste.
Add or remove workers after creation. While a cluster is being built the card shows the actual steps the platform is going through, not a spinner: three minutes of "Creating…" is three minutes of not knowing whether it is working.
Your own private network for your instances, a router when you need to reach out, and a public address when you need to be reached.
Social crawlers do not run JavaScript, so every shared link showed the home page title whatever page you had shared. Each public route is now also a static page with its own title and description.
Confirming the second password field cleared the acceptance you had just given. Reported by the first person to try the full sign-up.
The proxy in front of the application was cutting them. It now allows what the application allows.
It happened, and it is the reason the second copy of customer files exists. Fixed, and those files are now pulled to another machine where nothing on the application server can delete them.
A logical dump every night, verified as a readable archive immediately after it is written and deleted if it cannot be read. A second copy on another machine, with an alarm if the chain stops.
The fonts are served from our own machines. A page that explains privacy while asking a third party for a typeface is telling on itself.
Version, control plane replicas, worker size and count up to 32, shared or dedicated API endpoint. Three minutes to a cluster, most of it spent copying the golden image.
We probe only the sessions that are genuinely online, and a session still held open on an address that has not answered for months is now reported as its own state rather than as a fault. Real faults stand out instead of sitting in a list of things that were never going to answer.
— Network — AS43141
For every domain our relay sends for, we now verify on a schedule that the records deliverability rests on — SPF, DKIM, DMARC and MTA-STS — are really published and still current. A record dropped from a zone gets caught here, rather than by someone's message bouncing.
— Network — AS43141
Every block we hold now has a registry entry with correct contact details and a monitored abuse address. Abuse reports reach somebody instead of bouncing, and nothing we announce is missing from the registry.
— Network — AS43141
A published file in the standard format (RFC 8805) gives the real location of each of our address blocks, and our registry entries point at it. Geolocation providers collect it on their own, so a wrong city or country now corrects itself instead of needing a separate support request to every provider that got it wrong.
— Network — AS43141
Receiving mail servers judge a sender on its reverse DNS, so moving a block's delegation is an efficient way to have mail rejected. We enumerated the existing records first, carried all of them over, and verified each one against the new nameservers before and after the switch. No interruption to mail.
— Network — AS43141
Re-provisioning a zone could write a version number lower than the one the secondaries already held. They correctly ignored it, so every later edit to that zone stopped propagating — with no error raised anywhere. Fixed, and held in place by a test so it cannot come back quietly.
— Network — AS43141
Our address blocks were registered with a country but no location, so geolocation databases fell back to the middle of Switzerland. Every block now carries its real coordinates, and we publish a machine-readable geofeed those databases can read. Services that pick a region from an IP address — streaming, speed tests, regional content — now place these addresses where they actually are.
— Network — AS43141
Nothing is created until a card is verified — it is what keeps the platform from being free compute for whoever finds it. The welcome credit lands when you add the card, not when you register, for the same reason.
Worker prices set against a survey of what comparable Swiss and European providers charge, not against a margin target. Shared control plane free, dedicated endpoint 25 CHF a month. You pay for what you allocated, by the hour.
— Cloud platform
So you can measure the bandwidth to our network yourself instead of taking a number off a page.
— Network — AS43141
It does now, which is what makes maintenance on a compute node invisible to whoever is running on it.
— Cloud platform
— Cloud platform
Losing a compute node no longer means losing what was on it.
— Cloud platform
Polling once a minute missed interruptions of six and thirty-four seconds: real drops that left no trace anywhere. We now read each interface's own last-change counter, so a flap is recorded with the second it happened however brief it was. A fault on a link can be shown rather than argued about.
— Network — AS43141
A second tier-1 carrier at the Milan site: if one upstream has a bad day, traffic continues on the other instead of waiting for it to come back.
— Network — AS43141
On the route servers at both, so paths to the Italian and international networks present in Milan are local and direct rather than long trips through transit.
— Network — AS43141
— Network — AS43141
The backups rotate, and one is restored every week to prove it can be. A backup nobody has restored is a hope, not a backup.
— Cloud platform
What each project consumed, measured and priced, which is what makes per-hour billing possible upstream of us.
— Cloud platform
Virtual machine memory can be encrypted by the CPU itself, so the contents are not readable from the host. Bandwidth limits can be set per network.
— Cloud platform
A filesystem several machines can mount at once, and the ability to book resources for a date instead of hoping they are free.
— Cloud platform
The default volume type was not visible to projects, so the most obvious way to ask for a disk was the one that did not work.
— Cloud platform
— Cloud platform
Nine thousand bytes per packet between virtual machines, which means more of each packet is your data and less is header.
— Cloud platform
Each project can hold its own zones and edit them through the API, instead of asking someone to add a record.
— Cloud platform
Verification, welcome, password changes and security notices, sent through a European relay. Two silent failures found and fixed while testing them.
The internal network offers 9000-byte frames, the path to the internet carries 1500. Anything in between vanished without an error — the failure mode that looks like "the internet is slow today". Now the route carries the right size, set by a service at boot instead of a config file that gets overwritten.
One machine per job — site, application, database, cache — each tier allowed to talk only to the next one it actually needs. HTTPS from the first day, with the web server configuration owned by Ansible rather than edited by hand.
The image catalogue and the backup service no longer depend on one service machine being alive.
— Cloud platform
Two incompatible settings on the network nodes dropped the traffic of every public address, with no error anywhere — the kind of failure you find out about from customers. Fixed, and a check now runs after every gateway change, because a silent fault deserves a loud test.
— Cloud platform
Create a load balancer from the API or the console and it gets a public address. Operations that take a while now resume if the service running them restarts, so a balancer does not end up half-built.
— Cloud platform
The GPU is passed through to the virtual machine with nothing in between, so the performance is the card's. Available on a subset of the compute nodes.
— Cloud platform
Virtual machine disks are written to two synchronous copies plus an arbiter that holds no data. If one node goes down the volume stays writable, and nobody has to intervene.
— Cloud platform
The OpenStack API and both web consoles answer on public addresses over TLS, with certificates that renew themselves, across two network nodes rather than one.
— Cloud platform
— Network — AS43141
Traffic arriving at one site could fail to reach destinations served by another. Routing corrected so every prefix is reachable from every entry point.
— Network — AS43141
Direct peering with Init7 at CH-IX: traffic towards their users takes a direct route rather than going through our upstreams.
— Network — AS43141
More capacity, and no single link whose failure takes a site off the map.
— Network — AS43141
Volumetric attack traffic aimed at one address is discarded upstream, before it can saturate the links. The point is that everything else stays reachable while an attack is in progress.
— Network — AS43141
Routes whose origin fails RPKI validation are discarded instead of accepted. It is the difference between noticing a route hijack and carrying it: traffic to your services cannot be pulled towards a network with no right to announce the address.
— Network — AS43141
In some failover scenarios the IPv6 prefix stopped being originated, which makes every IPv6 service unreachable while everything still looks healthy from inside. It is now always originated by the network.
— Network — AS43141
Shorter paths to the hundreds of networks present there, instead of going through transit.
— Network — AS43141