Data Architect, Ph.D, Information Technologist, Gamer
7088 stories
·
30 followers

Schlock Mercenary: December 7, 2019

1 Share

Read the whole story
denubis
2 hours ago
reply
Sydney, Australia
Share this story
Delete

That packet looks familiar, and that one, and that one...

2 Shares

Sometimes I tell stories about service outages from the point of view of it having already happened, and provide this kind of "omniscient narrator" perspective as it goes along. I can tell you to look out for things, since they'll come in handy later, and so on.

Other times, I try to present them in much the same way that they happened, complete with the whole "fog of war" thing that goes with being in the moment. This is where you can't see obvious things that are right in front of you because they seem too ridiculous to contemplate.

This is one of the latter, based on multiple events I've lived through. Yes, multiple: it's happened at several distinct companies that I've either worked for, contracted for, heard in stories from friends, or read somewhere over the years.

So, let's dive in.

You're at work at the office. It's a normal business day. Lots of other people are also at the office, doing their regular jobs. Much of this work involves poking at vaguely Unix-flavored boxes which are not physically on the premises. They're somewhere relatively far away, and so you have to telnet, ssh, or whatever to get to them. The point is, you're having to cross at least one "WAN" link to reach these things.

Things are going okay. But then it seems like everything is getting slower and slower. Interactive connections are lagging: keystrokes don't echo back quickly any more. Web browsing and other "batch" network stuff is dragging: the "throbber" in the browser is busy for far too long for ordinary activities. The chat system gets slower and slower.

You manage to hear from other people via the chat system ever so briefly. Some of them are seeing it as well. Others aren't sure, but at least you know it's not just you.

Over the next minute or so, the whole thing grinds to a halt. EVERYTHING is now dead. Nothing's working.

Depending on what decade it is, you whip out a modem and dial back in to the corporate network remote access pool, jump on the guest wireless network, or start tethering through your cell phone and get back in touch with whoever managed to stay online. You find others who have done the same thing, and it's clear there's a very big problem, but only where you are. The rest of the organization's offices are fine, but everything where you are is toast.

Someone immediately thinks it's the link to the Internet and picks up the phone to the ISP to yell at them, because it has to be them. It's the network. It's always the network. Particularly when there's someone else to yell at, right? That happens in the background.

Maybe an hour into this, someone unrelated to the problem but who knows their way around a network makes a curious observation: they have a hard-wired connection, and they are seeing a LOT of crazy traffic going past their host. It's way more multicast and broadcast traffic than they are used to. (Apparently, they have done this before, and have their own internal metric for what "normal" is, which is particularly valuable now. Otherwise, how would they know?) Hooray for tcpdump!

Some of the broadcast traffic identifies its origin. More than just a MAC address or a source IP address, some of it (depending on the decade again) is NetBIOS over TCP/IP broadcasts, or MDNS broadcasts, or whatever, and they helpfully have the names of the hosts embedded. These are workstations, and they include some aspect of the human user's unixname, so you can go "oh, that's so and so".

Did that person just unleash a hell torrent on the network? Better go check. They're right around the corner on the same floor, so why not? You get there, and they're very friendly, and no, they aren't doing anything funny. Their host DOES have the IP address and MAC address corresponding to some of the packets flying past the other box over and over, but they swear up and down they didn't do anything.

You ask nicely if they'll disable their network interface(s) temporarily just for a while to attempt isolating things, and they agree. They even do you one better and power the whole machine off and will leave it off until you give the all-clear. They even unplug the network cable(s) to thwart any Wake-on-LAN magic. There's no way it's going to cause you any trouble now.

You walk back around the corner to the first person's machine where they spotted all of of the bcast/mcast traffic, expecting it to have subsided. It hasn't. Indeed, it's still running full tilt, and then you notice in the spew that the host you just disabled is still in the packet traces. This box has its NICs disabled, is unplugged from the hard-wired network AND is powered off... and yet there are Ethernet frames going by which claim to be from it.

At this point, people who have lived through this before probably know what happened. They might not know how, or why it was able to happen, but somewhere deep down inside, they have this sinking feeling that someone did something terrible to the network and this is how it's manifesting.

The story diverges from here based on which version of it I've either lived through or heard about, but it goes along similar lines in any case.

Someone introduced a loop to the network. The most likely case is that a person showed up in the morning and went crawling under their desk to plug something in, like a power cord. Maybe it "came unplugged" overnight or over the weekend, and they're fixing it.

Then, while they're down there, they see this random Ethernet patch cable just kind of hanging out on the floor. It looks like it should be plugged in to one of the ports down there, but it's not. They plug it in to "do that person a solid" so they don't have to waste time also dealing with unplugged stuff.

Of course, as it turned out, the loose cable end was one that was ON TOP of a desk, and had fallen down. It was the end that went into a computer, and did not need to be plugged "back in"... for the other end of that same cable was already plugged into a port!

Taking that cable and jamming it into another port introduced a loop.

Again, here, a bunch of people are jumping up and down, looking for the comment form, or popping back to the Hacker News tab to say something. Patience. We'll get there.

Radia Perlman figured this out a long time ago. It's called spanning-tree protocol, and it's how your network can detect and defend against such clownery as someone looping it back onto itself.

But, again, the story branches. One time, it didn't exist yet in the networking equipment at the company. Another time, it did, but the people running the network had no idea what it was, or why they'd use it. A third time, they knew about it and decided to "not use it" since "nobody here is that stupid". Yet another time, they had just turned it off days earlier because "it was breaking stuff, and things were fine without it".

For the benefit of those who haven't lived through it yet, here's what happens. Let's take a simple loop where you've managed to plug two ports on the same switch into each other. We'll say that ports 15 and 16 are looped back.

Someone, somewhere, sends out a broadcast, multicast, or possibly even a unicast packet (for a destination not yet mapped to a port). The switch (or hub, if you're "back in the day"...) takes that packet and floods it out to all ports except the one where it came from. Maybe it came from its uplink (port 24), and so it sends the packet down ports 1-23, skipping 24.

The packet goes out port 15, makes that neat little hairpin turn, and arrives back at the switch on port 16 at nearly the same time. Assuming it's a broadcast, multicast, or a still-unknown unicast address, we're right back where we started: the switch has to take THIS packet, then, and flood it out to all of its ports less the source port. It gets sent to 1-15 and 17-24, skipping 16. Hold that thought right there for now.

Back up in time a few nanoseconds. That packet ALSO went out port 16, made the hairpin turn, passing itself going the other direction (possibly on the same wires, if in in full-duplex mode), and showed up back at the switch on port 15. At that point, the switch proceeded to flood it out to 1-14 and 16-24.

So you see, every time the packet leaves 15, it arrives at 16, and every time it leaves 16, it arrives at 15. If this was a one-way situation, you'd "merely" have a train of packets flying around forever, never dying, because there's no such thing as a TTL at this layer of your network. But, since it's happening in both directions, you actually get double the fun.

None of these packets stop being forwarded, and new ones show up, so before long, your switch is doing nothing but spraying those frames everywhere just as fast as it can. Eventually you starve out all other traffic, and everything probably grinds to a halt.

Eventually, the organization learns about things like switches, STP, 802.1x, and not having massive broadcast domains, and that crazy chapter of their history ends. However, just down the street, yet another company is waiting to add it to their own history.

So, here's a challenge for anyone who's managed to get this far: pick a nice quiet time outside of business hours, declare a maintenance window, then go deliberately loop back some Ethernet ports. Look in offices, conference rooms, and on the backs of things like VOIP phones. Be creative. See if your network actually survives it.

Or, you know, don't. If you don't test it, someone else will do it for you... eventually. They won't wait for a quiet time and won't declare a maintenance window, and they sure won't know to unplug it when things go sideways, but that's life, right?

Oh, finally, don't forget to go back to that nice person and tell them that they are okay to plug their machine back in, power it up, and re-enable the network. Then also tell them they didn't cause the whole business to tank for multiple hours, because they might be sweating bullets that somehow the whole thing will land on their head and poison their next performance review.

People deserve to know when they didn't cause a problem, particularly when they think they did! Don't forget that.

Read the whole story
denubis
2 hours ago
reply
Sydney, Australia
smarkwell
17 hours ago
reply
Share this story
Delete

Virtue signalling and vice signalling

1 Share

One of the stranger terms of political abuse to enter the lexicon in recent years is “virtue signalling”. It’s used almost exclusively by the political right and covers many different kinds of statements, actions and policies, mostly associated with the culture wars.

A particularly striking feature of this is that, until recently, “virtue” was a term primarily associated with the right. Bill Bennett (Education Secretary under GW Bush) had a big hit with The Book of Virtues back in the 1990s. He’s now an apologist for Trumpism.

It’s too complicated to cover all aspects of this in one post, but it may be useful to compare two symbolic actions

  • displaying a rainbow flag; and
  • wearing a MAGA hat.

Clearly the term “virtue signalling” would be applied only to the first of these. And this is just a not a matter of semantics, as it would be if the left had a corresponding term.

People who display the rainbow flag are virtue signalling in the obvious sense of the word: the flag says something like “equal marriage is a good cause. I support it, and so should you”.

Normally, the opposing response would be to say “No, it’s not a good cause, and those who support it are wrong’

The problem for the right is that they don’t have any moral standing for a claim like this, and they know it. While many rightwingers undoubtedly believe homosexuality to be sinful, they know that this belief violates norms of equal treatment and personal freedom they claim to accept, and they therefore can’t put it forward without inviting condemnation, or at least rejection, including from their own side. So, they have to resort to terms like “virtue signalling”, in this case implying an ostentatious moral superiority, combined with hypocrisy.

And the same is true across the whole range of issues summed up in the cognate term “Social Justice Warrior”.

The MAGA hat is the mirror image of this. The MAGA hat
(unlike, say, an American flag lapel pin) is not a claim, legitimate or otherwise, to be a patriotic American. Rather, it’s a deliberately offensive statement of support for Trump’s racism, misogyny and corruption.

The whole point is to “trigger the libs” as Trump Jr’s recent book puts it. No claim to virtue is being put forward. It’s a pure piece of identity politics, making the assertion that the wearers should be treated as superior without having any actual justification for this claim, moral or otherwise. Again, this can’t be spelt out; being an explicit white nationalist remains beyond the pale, and the conduct of the Trumpists defies any credible defense.

So, the intellectual apologists of the right can only resort to tu quoque, making the claim, in various forms, that the left is just as bad as their own side. This started with the Republican War on Science, but is now virtually universal.

The point of “virtue signalling” is to make this claim, without having to say what is wrong with the virtue being signalled.

Virtue signalling and hypocrisy

Most of the time, the accusation of “virtue signalling” includes an implicit connotation of “hypocrisy”. But then, why introduce a new and obscure term for something we have known about for millennia?

The answer is that hypocrisy is a specific accusation that can be backed up, or refuted, by evidence. For example, if a church leader who claims to be a Christian advocates locking up innocent children, the case is pretty clear-cut.

By contrast, “virtue signalling” is an insinuation rather than a factual claim. It doesn’t need to be backed up, and usually isn’t. If the person accused of virtue signalling on the basis of a symbolic action shows that they are in fact making costly efforts in support of their cause, these actions are just added to the charge sheet.

The charge of virtue signalling doesn’t rely on the actual inconsistencies of individuals. Rather it relies on in-group shared negative perceptions of out-groups (inner city latte sipping lefties and so on).

To restate the central point, accusations of virtue signalling aren’t meant to promote actual virtue over fraudulent signals: rather to argue against virtue and in favour of vice. Those who use the accusation want to score points in favor of behavior they aren’t willing to defend openly.

In all of this, it’s worth remembering the observation of La Rouchefoucald that “hypocrisy is the tribute vice pays to virtue”. The accusation of virtue signalling represents the refusal of vice to pay this tribute.

Read the whole story
denubis
20 hours ago
reply
Sydney, Australia
kazriko
10 hours ago
The thing is, the opposite of virtue signalling isn't "vice signalling" but rather being genuine about your point of view, and following through with actions that support that point of view. Virtue signalling is just putting a sign up to indicate what side you nominally support without taking further action.
Share this story
Delete

'Exhaustion point': Worst to come for NSW bushfires

1 Share
The Rural Fire Service founder said NSW faced "a very dire situation" as fire authorities were stretched to the limit, with the worst of the season yet to come.
Read the whole story
denubis
21 hours ago
reply
Sydney, Australia
Share this story
Delete

Claims about real rates of return

1 Share

With recourse to archival, printed primary, and secondary sources, this paper reconstructs global real interest rates on an annual basis going back to the 14th century, covering 78% of advanced economy GDP over time. I show that across successive monetary and fiscal regimes, and a variety of asset classes, real interest rates have not been “stable”, and that since the major monetary upheavals of the late middle ages, a trend decline between 0.6-1.8bps p.a. has prevailed. A consistent increase in real negative-yielding rates in advanced economies over the same horizon is identified, despite important temporary reversals such as the 17th Century Crisis. Against their long-term context, currently depressed sovereign real rates are in fact converging “back to historical trend” – a trend that makes narratives about a “secular stagnation” environment entirely misleading, and suggests that – irrespective of particular monetary and fiscal responses – real rates could soon enter permanently negative territory. I also posit that the return data here reflects a substantial share of “nonhuman wealth” over time: the resulting R-G series derived from this data show a downward trend over the same timeframe: suggestions about the “virtual stability” of capital returns, and the policy implications advanced by Piketty (2014) are in consequence equally unsubstantiated by the historical record.

That is from a new paper by Paul Schelzing, via the excellent Kevin Lewis.

The post Claims about real rates of return appeared first on Marginal REVOLUTION.

Read the whole story
denubis
23 hours ago
reply
Sydney, Australia
Share this story
Delete

Schlock Mercenary: December 5, 2019

1 Share

Read the whole story
denubis
2 days ago
reply
Sydney, Australia
Share this story
Delete
Next Page of Stories