Gigakube changelog

Fixed: A new cluster's API address could be published a moment before its certificate covered it

The settings that publish a new cluster's API endpoint are now written in one go. Before, a write that failed halfway through could leave the address published while the serving certificate did not yet carry that name, and kubectl would refuse the connection until the next pass — seconds in practice, longer if the failure persisted. Either the whole configuration is in place now, or none of it is and it is retried.

— Kubernetes platform

Improved: A cluster order that has to wait now tells you why, and starts on its own

When the platform has no room for another cluster, your order waits instead of failing, and it says which wait it is: no room on the nodes right now, or too many clusters starting at the same time. It begins by itself as soon as room appears — a cluster removed, a node added — and there is nothing to re-submit. Measured on the platform this week: an order left the queue 37 seconds after room appeared, and its control plane answered 12 seconds later. How much room there is is now read from the cluster itself instead of a number we keep up to date by hand, so adding a node raises the limit the moment the node is there.

— Kubernetes platform

New: Our second region is being built as a full-L3 fabric

In gk02 routing goes all the way down to the individual machine: each host has its own loopback address and speaks BGP with the switches it is wired to, instead of sharing one large layer-2 network with every other server.

Three consequences. A fault stays where it starts — a broadcast storm or a flapping MAC address is felt by every machine in a shared layer-2 domain, and by none when each host is routed. Failover is decided by routing timers instead of by how long a switch takes to forget a MAC address: seconds rather than tens of seconds. And both uplinks on a server carry traffic as genuine equal-cost paths towards two different switches, with no link aggregation, so nothing requires the two ports to land on the same box.

It also grows by adding a switch and its routing sessions rather than by stretching an existing network wider. The part that makes this work for an OpenStack cloud — announcing floating addresses over BGP from whichever host currently holds them — is not shipped ready to use by the upstream projects: we integrate it ourselves and prove it in the lab before it carries anyone's traffic.

Today gk02 routes down to the leaf switch, and host-level BGP is being introduced step by step. Existing regions are unaffected.

— Cloud platform

Fixed: Wiping a disk is part of installing a machine, not something to remember

Servers reused from a previous life carried old volume-manager metadata that made the installer abort halfway. Erasing every disk is now a step in the standard install sequence, which fixes that — and means a machine that enters service carries nothing from whatever ran on it before.

— Cloud platform

New: Machines are now installed automatically, from power-on to running operating system

New hardware is discovered, inventoried, wiped and installed without anyone touching it, driven over the management controller. It shortens the time between a server arriving and capacity being available, and it makes every machine come up identically instead of however the last person did it by hand.

— Cloud platform

New: Second zone network is live, with redundant 100G uplinks

A second availability zone is being built. Its network foundation is now in service: two independent 100 Gbit/s uplinks to the border router, each carrying traffic at the same time through BGP multipath. If one link, optic or cable fails, the other keeps forwarding and no session is interrupted. The zone also has its own public address block, so customers who take resources in both zones get addresses that route between them.

— Cloud platform

New: Now peering at SwissIX, our seventh internet exchange

Direct peering in Zurich over the exchange's route servers, which shortens the path to Swiss networks that peer there instead of sending the traffic through a transit carrier abroad. Brings us to seven exchanges and 260 Gbit/s of exchange port capacity.

— Network — AS43141

New: Live network data for AS43141, as one feed and a one-line embeddable panel

Any site can now show current, accurate facts about the network — carriers, internet exchanges, installed capacity, address space — without anyone keeping them up to date by hand. Two lines of HTML for the ready-made panel, or read the JSON and lay it out yourself. It refreshes hourly and states when it was built, so a number on someone else\s page is never older than the hour.

— Network — AS43141

Fixed: Our published transit list showed two carriers out of four

It listed only the carriers that had an active measurement probe attached, which quietly understated the network to anyone reading it. All four now appear, ordered by installed capacity, and the list is derived from the same table as the capacity total so the two cannot drift apart.

— Network — AS43141

Fixed: Removed the per-carrier latency figures we published: they did not mean what they appeared to mean

The numbers came from a single router probing a single anycast address. "1 ms" therefore said that destination was close from that one point, not that the carrier was fast, and placed side by side the figures invited a comparison the measurement cannot support. They also said nothing about the sites where most of our capacity sits. We would rather publish less than publish a number that does not survive being checked. The measurement stays where it belongs, choosing routes.

— Network — AS43141

Fixed: Public network data can now actually be cached by CDNs and shared caches

The endpoints ran through session handling, so every response carried a cookie — and many shared caches and CDNs refuse to store a response with a cookie no matter how cacheable it claims to be. They now set none. Sites embedding our data serve it from their own cache, and keep serving it for up to 24 hours if our host is unreachable.

— Network — AS43141

Improved: Latency figures withdrawn, for a good reason

They were measured from a single router towards an anycast address, so they described that path and not the carrier. Removed rather than dressed up. If you want to know the path to you, ask us for a trace from the site you care about.

— Network — AS43141

Improved: The network publishes its own numbers

The institute built a live feed of the AS43141 data — capacity, exchanges, carriers, address space — so the numbers on this site are read rather than typed. Three versions in one day, each one more precise than the last.

— Network — AS43141

Fixed: The API could refuse 60 requests a minute for the whole site

The limit was counted against the shared token the site uses, not against each visitor, so a busy minute would have turned into refusals for everyone — sign-ups included. Found during a check after launch, before anyone hit it.

New: security@gigakube.ch

A mailbox for vulnerability reports, first in our security.txt ahead of the contact form. For a CSIRT at three in the morning, a form is not a channel.

New: What we do about backups, in writing

The Security page now states the schedule, the retention, the second copy — and the limit: both copies are in the same datacentre, so it does not yet cover losing the site. An off-site copy is on the list.

New: A page about the network, with numbers that check themselves

Capacity, exchanges, carriers and peers, read every hour from the public looking glass of the network we are part of, with the time of the last reading next to them. If they drop, you will watch them drop.

New: Two-step verification

Turn it on under Account: your password, then a six-digit code from any authenticator app. Eight single-use recovery codes for the day you lose the phone. Not SMS — a phone number can be moved to someone else with a phone call, which is the attack this is meant to stop.

Improved: Faults are detected in minutes, by thresholds taken from real failures

The platform keeps thirteen months of history and raises alarms on thresholds derived from failures that actually happened, rather than on round numbers — wired to a destination that reaches a person on call.

— Kubernetes platform

Improved: A cluster's own components no longer take the long way round

Components inside a cluster were being told to reach their control plane at its public address, leaving and re-entering the network to talk to something a few metres away. They are now given the internal address, and the public one is used only by people connecting from outside.

— Kubernetes platform

Improved: Faster and steadier storage under every control plane

The database that holds the state of each managed control plane now sits on the synchronous-write NVMe tier. The effect is a control plane that stays responsive when several clusters are busy at the same time.

— Kubernetes platform

Improved: Shorter addresses for clusters

Clusters are reachable at a shorter name: about fifteen characters have gone from every cluster address, and from the kubeconfig handed to the customer.

— Kubernetes platform

Fixed: Deleting a cluster always finishes

Cluster deletion could stop part-way and stay there, waiting on a resource that had been left marked as in use. That resource is now released, and a deletion completes on its own.

— Kubernetes platform

Fixed: A cluster's name is published and withdrawn with the cluster itself

The public DNS record for a cluster with a dedicated address is now written when the cluster is created and removed when it is deleted, instead of being maintained by hand. A deleted cluster can no longer leave a name behind pointing at an address that has since been reassigned.

— Kubernetes platform

Fixed: Clusters no longer stall when disk creation fails

A cluster whose worker disks could not be created would sit with a healthy control plane and no worker nodes, indefinitely, until somebody looked at it. Worker nodes that fail to start are now replaced automatically within ten minutes. If every node in a group fails at once the platform deliberately does not replace them: that means the cause is upstream, and a person should look at it instead.

— Kubernetes platform

Improved: The nameservers no longer announce their version or hostname

— Cloud platform

Fixed: Volume creation failed for a couple of minutes during storage maintenance, without saying why

It now waits instead of refusing.

— Cloud platform

Fixed: Some volumes were left out of the backups

Inclusion depended on a flag someone had to set by hand, which is a design that eventually forgets something. It no longer does.

— Cloud platform

Fixed: A DNS name still pointed at a public address that had been reassigned

— Cloud platform

Fixed: One compute node could not attach replicated volumes while still accepting new machines

The node was advertising itself as healthy for work it could not do. It is now taken out of rotation when that check fails.

— Cloud platform

Improved: Tenant clusters moved to their own domain

Cluster API endpoints now live under a name that belongs to this service, instead of an internal platform domain.

— Kubernetes platform

Fixed: Clusters with a shared endpoint could not be reached from inside

The platform team traced it to a missing internal endpoint record and to the management tunnel pointing at an address that was not reachable. Both fixed within a day of being reported.

— Kubernetes platform

Fixed: The terminal now tells you why it will not open

It used to answer "Unexpected server response: 500", which is a way of saying nothing. Now it says which part is not ready.

New: Anyone can report a problem to us

A single entry point for alerts, from us or from anyone who monitors something that touches us, fanned out to the people on call. If you see something wrong before we do, we want it to be easy to tell us.

New: An email when your cluster is ready, with a way straight in

It arrives when the cluster answers, not when the order is accepted, and the button takes you to the cluster rather than to a login page.

New: Per-hour billing for clusters and public addresses

Clusters and public addresses are charged for what you have allocated, by the hour, on the same wallet as everything else. What you see on the card is what is being counted.

New: Node pools: different worker sizes in one cluster

Groups of workers with their own size, labels and taints, so a memory-hungry job and a small web front end can live in the same cluster without paying for the larger machine twice.

Fixed: A cluster name that is too long is now refused when you order it

A name that exceeded the length a DNS label allows produced an unusable address and certificate, and the problem only appeared when someone first tried to connect. Names like that are now refused at the moment of ordering, with an explanation.

— Kubernetes platform

New: Node pools, each with its own size, labels and taints

A cluster can now be ordered as several groups of nodes instead of one, each group with its own machine size, its own Kubernetes labels and its own taints. Workloads that need particular machines can be kept on them.

— Kubernetes platform

New: Two projects can be joined on a private network

Useful when the same customer keeps environments in separate projects and still wants them to talk without going out to the internet.

— Cloud platform

Improved: The whole internal network renumbered, with nothing to do on your side

Every machine moved onto a private range with room to grow. Planned, rehearsed and done with the old addresses kept alive until the new ones carried everything.

New: A terminal and k9s in the browser

kubectl against your own cluster from the portal, with a session that lives as long as the tab. Useful precisely on the day something is broken and you do not have your laptop.

New: Look inside your cluster without installing anything

Nodes, pods that are not happy, deployments short of replicas, recent warnings. Read-only, and the paths are chosen by us: your secrets are not reachable through this, because there is no call that asks for them.

Fixed: The join token had never worked

Three separate reasons stacked on top of each other, which is why it looked like one stubborn bug. It now returns a token, the CA hash and the endpoint, ready to paste.

New: Resize a cluster, and watch it being built

Add or remove workers after creation. While a cluster is being built the card shows the actual steps the platform is going through, not a spinner: three minutes of "Creating…" is three minutes of not knowing whether it is working.

New: Private networks, routers and public addresses

Your own private network for your instances, a router when you need to reach out, and a public address when you need to be reached.

Fixed: Shared links showed the wrong page

Social crawlers do not run JavaScript, so every shared link showed the home page title whatever page you had shared. Each public route is now also a static page with its own title and description.

Fixed: The terms checkbox stopped responding

Confirming the second password field cleared the acceptance you had just given. Reported by the first person to try the full sign-up.

Fixed: Uploads over 1 MB failed without saying why

The proxy in front of the application was cutting them. It now allows what the application allows.

Fixed: A deploy could remove images you had uploaded

It happened, and it is the reason the second copy of customer files exists. Fixed, and those files are now pulled to another machine where nothing on the application server can delete them.

New: Daily backups, and each one read back

A logical dump every night, verified as a readable archive immediately after it is written and deleted if it cannot be read. A second copy on another machine, with an alarm if the chain stops.

New: Security, Imprint and Cookies pages — and no fonts from Google

The fonts are served from our own machines. A page that explains privacy while asking a third party for a typeface is telling on itself.

New: Create a Kubernetes cluster from the panel

Version, control plane replicas, worker size and count up to 32, shared or dedicated API endpoint. Three minutes to a cluster, most of it spent copying the golden image.

Improved: Line monitoring tells a line that is down from a session that was never closed

We probe only the sessions that are genuinely online, and a session still held open on an address that has not answered for months is now reported as its own state rather than as a fault. Real faults stand out instead of sitting in a list of things that were never going to answer.

— Network — AS43141

Improved: Mail deliverability is checked against live DNS instead of assumed

For every domain our relay sends for, we now verify on a schedule that the records deliverability rests on — SPF, DKIM, DMARC and MTA-STS — are really published and still current. A record dropped from a zone gets caught here, rather than by someone's message bouncing.

— Network — AS43141

New: All of our address space is registered with a working abuse contact

Every block we hold now has a registry entry with correct contact details and a monitored abuse address. Abuse reports reach somebody instead of bouncing, and nothing we announce is missing from the registry.

— Network — AS43141

New: Published a geofeed so geolocation databases can correct themselves

A published file in the standard format (RFC 8805) gives the real location of each of our address blocks, and our registry entries point at it. Geolocation providers collect it on their own, so a wrong city or country now corrects itself instead of needing a separate support request to every provider that got it wrong.

— Network — AS43141

Fixed: Reverse DNS moved to our own nameservers with every existing record preserved

Receiving mail servers judge a sender on its reverse DNS, so moving a block's delegation is an efficient way to have mail rejected. We enumerated the existing records first, carried all of them over, and verified each one against the new nameservers before and after the switch. No interruption to mail.

— Network — AS43141

Fixed: DNS changes could silently stop reaching the secondary nameservers

Re-provisioning a zone could write a version number lower than the one the secondaries already held. They correctly ignored it, so every later edit to that zone stopped propagating — with no error raised anywhere. Fixed, and held in place by a test so it cannot come back quietly.

— Network — AS43141

Fixed: IP addresses in Ticino no longer geolocate to central Switzerland

Our address blocks were registered with a country but no location, so geolocation databases fell back to the middle of Switzerland. Every block now carries its real coordinates, and we publish a machine-readable geofeed those databases can read. Services that pick a region from an IP address — streaming, speed tests, regional content — now place these addresses where they actually are.

— Network — AS43141

New: A card before resources, and 5 CHF to start

Nothing is created until a card is verified — it is what keeps the platform from being free compute for whoever finds it. The welcome credit lands when you add the card, not when you register, for the same reason.

New: Pricing published, per hour and without lock-in

Worker prices set against a survey of what comparable Swiss and European providers charge, not against a margin target. Shared control plane free, dedicated endpoint 25 CHF a month. You pay for what you allocated, by the hour.

Improved: An order that cannot be served now waits in a queue and says so

When the platform is at capacity, a new cluster is held in an explicit waiting state with the reason, instead of being accepted and then failing quietly. Existing clusters are no longer squeezed to make room for a new one.

— Kubernetes platform

Improved: You get what the order asks for, and nothing else

Parts of a cluster are provisioned only when the order asks for them. Order a cluster without workers, or without a dedicated address, and nothing extra is built alongside it.

— Kubernetes platform

New: Join your own machines to your cluster

A cluster now hands out a join command for worker nodes you run yourself, together with the conditions under which those nodes can reach the control plane. Useful when some of the work has to stay on your own hardware.

— Kubernetes platform

New: A cluster can have an address of its own

Alongside the shared entry point, a cluster can now be given a dedicated address. It is announced as a name rather than a number, so the address behind it can change without invalidating kubeconfig files already in use.

— Kubernetes platform

Improved: Customer DNS zones moved onto a Swiss domain we own

— Cloud platform

Improved: Every cluster gets its own network security group

Each cluster is created with a firewall group generated for it alone, and that group is removed when the cluster is deleted. A cluster's traffic rules are its own: they are not shared with any other cluster, and nothing outside it can change them.

— Kubernetes platform

Fixed: A refused Kubernetes version change no longer blocks the rest of a cluster

Asking for a Kubernetes version that cannot be served used to leave the whole cluster stuck, including changes that had nothing to do with the version. The version request is now refused on its own, with the reason, and everything else about the cluster keeps working.

— Kubernetes platform

Fixed: Larger worker sizes can be ordered again

Clusters asking for the bigger machine types failed to build, and the error suggested the cloud was out of room when it was not. Worker disks are now created as volumes rather than on the compute node's local disk, so the size of a worker no longer depends on how much room happens to be left on the machine hosting it.

— Kubernetes platform

New: A 100 Gbit/s speed-test server on the Swiss backbone

So you can measure the bandwidth to our network yourself instead of taking a number off a page.

— Network — AS43141

Fixed: Live-migrating a machine booted from a volume did not work

It does now, which is what makes maintenance on a compute node invisible to whoever is running on it.

— Cloud platform

Fixed: Every load balancer after the tenth failed to build

— Cloud platform

Improved: New machines are born on replicated storage, not on the host's local disk

Losing a compute node no longer means losing what was on it.

— Cloud platform

Fixed: Link drops shorter than the monitoring interval are no longer invisible

Polling once a minute missed interruptions of six and thirty-four seconds: real drops that left no trace anywhere. We now read each interface's own last-change counter, so a flap is recorded with the second it happened however brief it was. A fault on a link can be shown rather than argued about.

— Network — AS43141

Improved: A second independent transit in Milan

A second tier-1 carrier at the Milan site: if one upstream has a bad day, traffic continues on the other instead of waiting for it to come back.

— Network — AS43141

New: Two more exchanges in Milan: MIX and MINAP

On the route servers at both, so paths to the Italian and international networks present in Milan are local and direct rather than long trips through transit.

— Network — AS43141

New: Direct peering at AMS-IX Amsterdam

— Network — AS43141

Improved: Automatic volume backups, with rotation and a weekly test restore

The backups rotate, and one is restored every week to prove it can be. A backup nobody has restored is a hope, not a backup.

— Cloud platform

New: Per-project metering, with rates applied

What each project consumed, measured and priced, which is what makes per-hour billing possible upstream of us.

— Cloud platform

New: Memory encrypted by the hardware, and per-network bandwidth limits

Virtual machine memory can be encrypted by the CPU itself, so the contents are not readable from the host. Bandwidth limits can be set per network.

— Cloud platform

New: Shared filesystems, and reserving capacity ahead of time

A filesystem several machines can mount at once, and the ability to book resources for a date instead of hoping they are free.

— Cloud platform

Fixed: Creating a disk without naming its type failed

The default volume type was not visible to projects, so the most obvious way to ask for a disk was the one that did not work.

— Cloud platform

Fixed: Machines booting from a volume created out of an image would not start

— Cloud platform

Improved: Jumbo frames on the internal networks

Nine thousand bytes per packet between virtual machines, which means more of each packet is your data and less is header.

— Cloud platform

New: DNS as a service: your own zones, managed from the API

Each project can hold its own zones and edit them through the API, instead of asking someone to add a record.

— Cloud platform

New: Email that works

Verification, welcome, password changes and security notices, sent through a European relay. Two silent failures found and fixed while testing them.

Fixed: Large packets were disappearing on the way out

The internal network offers 9000-byte frames, the path to the internet carries 1500. Anything in between vanished without an error — the failure mode that looks like "the internet is slow today". Now the route carries the right size, set by a service at boot instead of a config file that gets overwritten.

New: Gigakube goes up

One machine per job — site, application, database, cache — each tier allowed to talk only to the next one it actually needs. HTTPS from the first day, with the web server configuration owned by Ansible rather than edited by hand.

Improved: Images and backups moved off the install node, which was a single point of failure

The image catalogue and the backup service no longer depend on one service machine being alive.

— Cloud platform

Fixed: Public addresses restored after a gateway misconfiguration took all of them down

Two incompatible settings on the network nodes dropped the traffic of every public address, with no error anywhere — the kind of failure you find out about from customers. Fixed, and a check now runs after every gateway change, because a silent fault deserves a loud test.

— Cloud platform

New: Managed load balancers, and long operations that survive a restart

Create a load balancer from the API or the console and it gets a public address. Operations that take a while now resume if the service running them restarts, so a balancer does not end up half-built.

— Cloud platform

New: GPU flavours, with the card handed whole to the machine

The GPU is passed through to the virtual machine with nothing in between, so the performance is the card's. Available on a subset of the compute nodes.

— Cloud platform

New: Volumes replicated across two nodes, with a third as arbiter

Virtual machine disks are written to two synchronous copies plus an arbiter that holds no data. If one node goes down the volume stays writable, and nobody has to intervene.

— Cloud platform

New: The cloud API, Horizon and Skyline reachable over HTTPS

The OpenStack API and both web consoles answer on public addresses over TLS, with certificates that renew themselves, across two network nodes rather than one.

— Cloud platform

New: Peering at the Piacenza exchange

— Network — AS43141

Fixed: Some networks were unreachable depending on which site you entered from

Traffic arriving at one site could fail to reach destinations served by another. Routing corrected so every prefix is reachable from every entry point.

— Network — AS43141

New: A direct path to a large Swiss network

Direct peering with Init7 at CH-IX: traffic towards their users takes a direct route rather than going through our upstreams.

— Network — AS43141

Improved: The backbone between sites is now a fully redundant 100/400G ring

More capacity, and no single link whose failure takes a site off the map.

— Network — AS43141

New: DDoS mitigation, dropped at the transits rather than at our edge

Volumetric attack traffic aimed at one address is discarded upstream, before it can saturate the links. The point is that everything else stays reachable while an attack is in progress.

— Network — AS43141

Improved: RPKI origin validation on every upstream and every peer

Routes whose origin fails RPKI validation are discarded instead of accepted. It is the difference between noticing a route hijack and carrying it: traffic to your services cannot be pulled towards a network with no right to announce the address.

— Network — AS43141

Fixed: Our IPv6 block could stop being announced during a failover

In some failover scenarios the IPv6 prefix stopped being originated, which makes every IPv6 service unreachable while everything still looks healthy from inside. It is now always originated by the network.

— Network — AS43141

New: Direct peering at DE-CIX Frankfurt

Shorter paths to the hundreds of networks present there, instead of going through transit.

— Network — AS43141