Unroutable

About password security and credential stuffing

Thu, 04 Jan 2024 00:00:00 +0000

Common Falsehoods About Password Security and Credential Stuffing

TechCrunch published an article that gives class-action lawyers suing 23andMe a mouthpiece to editorialize about password security practices, masquerading as a news article. The upshot of the ~article~ editorial is this:

A large number of 23andMe’s customers were subject to a credential stuffing attack.
Design choices in 23andMe’s site allowed other customer data to be exposed via accounts compromised in the credential stuffing attack.
23andMe’s mitigations for credential stuffing attacks were inadequate.
23andMe should be held liable for this.

I want to focus here on the third point: credential stuffing attack mitigations. I’ve worked quite a bit on analyzing large credential stuffing attacks and recommending mitigations for them. I also served as a technical escalation point for customers who had a wide variety of strongly held false beliefs about password security credential stuffing mitigation. In reading various social media responses the 23andMe case, I see all these false beliefs turning up again. Let’s have a look at some.

The first question we need to answer here is: what do you mean by rate limiting? Usually there are two main rate limits that people consider: per-user and per-IP address. For example, you could allow a maximum of N failed logins per username during time window T1, after which point you allow no logins for that username until time window T2 expires. The intention here is to prevent an attacker who has a valid username from trying a large list of passwords for that username. The advantage to this is that for a given single target user, it makes large scale credential stuffing slow. The problems are substantial, however:

For an attacker with a large number of valid usernames, they can spread their attempts across the username list at a rate just below the limit for any given single user. For example, if the lockout trigger is 10 failures in 1 minute with a lockout window of 10 minutes and the attacker has 10,000 valid usernames, they can try 9 passwords per user per minute without ever triggering a lockout, or 90,000 tries per minute across their target list. If they do trigger a lockout, they can just come back in 10 minutes.

The more serious problem with per-user rate limiting is its utility as a denial-of-service attack against the legitimate account owner: a harrassment-oriented attacker need only trigger the required number of intentional failures in order to temporarily lock out the legitimate owner. This can continue indefinitely.

The next response to these problems is usually to propose per-IP-address rate limiting: a given address is only allowed a certain number of failures or a certain number of logins on different accounts in a given time window before being blocked. Ten years ago, this was pretty effective. Today, it’s much less effective due to the widespread availability of high-quality proxy services to attackers. It’s now very common for an attacker to be able to cycle through tens of thousands of different IP addresses in a short period of time, with some of those addresses also carrying legitimate traffic. The widespread use of carrier-grade NAT complicates this even further: with widespread NAT, you often end up blocking legitmate user traffic that just happens to share an IP address with the attacker at a point in time.

You can also combine the two: allow only N failures per address A before imposing restrictions. This might work, but requires that the application track a lot of state, and also requires a predictable distribution of normal behavior. As a service grows, the compliexity of managing this increases quickly. Blocking legitimate traffic leads to angry customers and increased support tickets.

The truth is that you can probably come up with a rate-limiting policy that will help to mitigate unsophisticated credential stuffing attacks, which helps a bit. But is it a killer app? No.

False: Most credential stuffing attacks use “brute force” techniques

Again, it depends on what you mean by “brute force”. It’s true that sites see combinatoric attacks against the most common usernames, such as:

Administrative usernames (“admin”, “root”, etc)
Very popular names in different languages (“tom”, “antonio”, etc)
“OG” usernames (“s3x”, “l33t”, etc)

A few hundred usernames like these will see constant low volume dictionary-style attacks. But it’s uncommon to see a large dictionary-style attack against a username chosen at random. Instead, the vast majority of credential-stuffing attacks use so-called “combolists”, which are lists of username/email and password pairs that are taken from password database leaks or malware dumps, and traded in the criminal underground. This technique relies on users’ proclivity to reuse passwords across sites, or to change their password back to a previously compromised one after a forced reset. This is likely the technique used in the 23andMe attack.

False: A stolen unique password means a compromised site

One of the most common accusations I saw form customers who had compromised accounts went something like this:

I use unique, random passwords for EVERY site. Yet an attacker logged into my account interactively. This means with CERTAINTY that your site is compromised and my password was stolen from YOU, and that you OBVIOUSLY store user passwords in clear text.

Sigh. No, what this means is that almost certainly, a system you use was compromised by malware and your cleartext password stolen from your browser or another application cache. This kind of malware is incredibly common on Windows systems and is becoming more common on other operating systems over time. It’s almost certainly the largest source of net-new credential data as of this writing. While malware-sourced credential feeds usually require criminals to pay for them in cryptocurrency, some of these credentials will become incorporated into more widely distributed combolists over time as well.

Worse, stealer malware also captures browser session cookies, which enable an attacker to hijack an authenticated web session, bypassing multifactor authentication.

What is to be done?

The attorney in the TechCrunch editorial seems to be arguing that multifactor authentication (MFA) should be mandatory. This is not an insane argument, but it comes with a large set of additional questions that need answers:

Should all sites, no matter how small, be required to use MFA? If not, what is the bar?
What types of MFA should be mandated? It’s beyond the scope of this blog post, but each MFA method has advantages and disadvantages.
If a company mandates MFA, should they be allowed to outsource it? Should they be required to outsource it? If so, how do we ensure the security of the outsourced provider, since we’ve now concentrated risks there?
What kind of account recovery should be mandated?
What penalties and reporting requirements should be enforced?

I’m not opposed to mandatory MFA in certain contexts, but I also don’t have perfect – or even good – answers to these questions.

Another option is to try to make password-only security better. The best version of this is email verification, sometimes called “MFA Lite”:

After signup or a successful login, the site sets a long-lived “device cookie” in the user’s browser. When the user logs in from a new device, they are required to click a link in an email in order to acknowledge the new device, at which point a new cookie is saved on that device. This doesn’t help prevent targeted account takeovers leveraging stealer malware (because the attacker will typically just steal the device cookie and the user’s email credentials anyway), but it helps a lot with standard combolist credential stuffing; it’s common for account takeovers to drop by 90% or more when this technique is implemented.
Another option is to subscribe to a service that prevents users from setting passwords that show up in combolists. There are any number of companies that collect combolists from the criminal underground and sell this sort of service. No company will ever have complete coverage, but this can work fairly well. The drawbacks are a) it’s expensive, b) it requires a fair amount of additional authentication code, and c) it leads to more support tickets when users get confused by it.

Books 2020

Thu, 31 Dec 2020 00:00:00 +0000

Books 2020

I read at least 41 books in 2020. Here are some highlights.

Intellectual History (?)

This is the category I’m using for my two overall favorites for the year, although it doesn’t quite fit. I love books like this that tell the stories of historic figures in unusual ways.

A Madman Dreams of Turing Machines - Janna Levin

This is generally described as a “philosophical novel”, a description which is both accurate and suitably vague, as the novel is difficult to describe. Janna Levin is famous for a lot of things: she’s possibly the most influential living black hole physicist, an author of a number of well-known books, and a well-known science communicator. This was her first book (I think), and doesn’t seem to be particularly well-known despite being an amazing work.

It’s a fictionalized account of the lives of Kurt Gödel and Alan Turing. The writing is alernately strange and beautiful. I can’t recommend it highly enough.

The Weil Conjectures - Karen Olsson

I read this back to back with the previous book, and they will always be connected in my mind although they don’t intersect except for a few biographical years.

The book explores the lives of the French writer Simone Weil (who died young and was little known during her lifetime) and her brother Andrew Weil, an influential mathemetician who survived to old age. The biographical stories are interwoven with meditations from the author’s own life as a mathemetician turned writer. Also beautifully written, it tells the stories of historic figures who deserve to be better known.

Classic Fiction

I only read two books in this category this year, and I’m noting one:

Pride and Prejudice - Jane Austen

At some point this year I realized that even though I had read this 30-ish years ago, I couldn’t remember a thing about it other than “Lizzie” and “Darcy”. I’m always curious about whether I feel years later that classics or old favorites stand the test of time. This one certianly does. After the re-read, I went on to binge watch all the screen adaptations I could find, including the Keira Knightly film, the two BBC adaptations, and the remarkable “Lizzie Bennett Diaries” modern-dress series on YouTube. They were all great fun, but the Keira Knightly film is my favorite.

Science Fiction

I didn’t read a lot of SF this year. My single pick is:

Binti - Nnedi Okorofor

I read the whole trilogy, but the first was by far my favorite. It’s just so different from any other SF novel I’ve read in recent years that I enjoyed every page. Somehow it manages to fit five or six classic archetypal stories into a very short novella.

Espionage History

This was by far my largest reading genre this year. I read so many great books in this category that it’s hard to narrow them down, but I’ll try. Maybe I’ll get crazy and do a separate blog post on this topic; you never know.

Active Measures - Thomas Rid

This history of disfornmation campaigns (mostly but not exclusively Soviet-bloc and Russian) is justifiably a huge 2020 bestseller. Anyone working in social media, politics, intelligence, or any related field should read it carefully. There’s a lot of misinformation about disinformation out there, but Rid cuts through it brilliantly.

The Main Enemy - Milt Rearden and James Risen

This is a long, detailed history of the KGB/CIA “spy wars” from roughly the mid-1970s to the fall of the Soviet Union. I’ve read a lot of books about this period and have more on my list, but this is the best survey that I’ve read thus far. In order to reduce mandatory CIA pre-publication review problems, the authors wrote their respective conributions separately and Rearden (a CIA veteran) submitted his half for review, while Risen (a journalist) did not. In the final version, though, they don’t call out the authors of each chapter, so it’s hard to keep track of what’s first-hand reporting and what’s not. I highly recommend the book for its thoroughness and the quality of the writing, but if you intend to study the period in depth it’s important to realize that there’s a lot of opinion in the reporting here. It’s a good idea to read other books about particular cases or periods to better understand the controversies surrounding some of the stories.

Circle of Treason - Sandra Grimes and Jeanne Vertefeuille

I’m calling this a “bonus pick” because the writing is a bit rough and repetitive in places, but it’s written from an analyst’s perspective rather than a journalist’s or ops officer’s perspective. I read a number of good books about the Aldrich Ames case, and I think it’s worth getting the take from the analysts who worked the case most closely. This also provides a useful counterpoint to some of Rearden’s opinions.

2019 Books

Mon, 30 Dec 2019 00:00:00 +0000

2019 Books

I read at least 32 books in 2019. The high count is due primarily to burning through a bunch of mediocre thriller novels on road trips, but I also read a number of really good books in diverse categories. Here are some highlights:

(Intellectual History) At the Existentialist Cafe - Sarah Bakewell

I love wide-ranging intellectual histories, and this fits that description completely. While I am not a huge fan of existentialism as a philosophical movement, its history and personalities are fascinating, and this book does justice to all of it. I particularly enjoyed the chapters about Simone de Beauvoir, Iris Murdoch, and Maurice Merleau-Ponty.

(Mystery) The Infinite Blacktop - Sara Gran

I don’t read a lot of mysteries, but Sara Gran’s Claire DeWitt series is certainly one of my all-time favorites. Detective noir with a touch of magical realism. This is the latest; I recommend reading them in order. I hope there are more to come.

(Micro History) Lady on the Beach - Norah Berg

This memoir is out of print, but worth finding if you have any interest in Pacific Northwest history. Battling alcoholism during the Great Depression, the author leaves Seattle for a subsistence life of gathering and beachcombing in a tiny, remote community on the Washington Coast. My dad was born in the same area and grew up during the years Berg describes. It was fascinating to see some of the wild stories he told confirmed by Norah Berg in this book.

(Philosophy and Mind) Conscious - Annaka Harris

I’ve read a great many of the popular books on “consciousness studies” over the years, this is one of the best. It’s short, concise, and clear. This is a difficult topic and Harris explains all the major problems and proposed solutions with impressive clarity.

(History of Espionage) The Spy and the Traitor - Ben Macintyre

Fascinating Cold War history. This was recommended to me by someone who doesn’t read espionage stuff who described it as one of the best books of her year, on a subject she didn’t think would interest her. That made it an easy sell for me, since I do like espionage books. I’ve since given it to a number of people who all enjoyed it.

(Cybersecurity) Sandworm - Andy Greenberg

This is my cybersecurity pick of the year. Enough said. If you’re interested at all in the subject, read it.

(Literary History/Autobiography) The Heart to Artemis - Bryher

Bryher may be best remembered as the longtime partner of the great Imagist poet H.D. (Hilda Doolittle), but she was a fascinating person in her own right: novelist, traveler, heiress, film critic, art patron, and more. This memoir covers her life from childhood to 1940 and her intersection with virutally all the artistic and intellectual figures of the early 20th century.

Book Report: Machete Season

Mon, 16 Jan 2017 00:00:00 +0000

Overview

Jean Hatzfeld’s Machete Season: the Killers in Rwanda Speak is a much different book than the Pol Pot history that I covered a couple of weeks ago. It’s harder to write about, because it’s just what the title describes: the killers in their own words, interspersed with short contextual explanations of the events surrounding the Rwandan genocide.

Hatzfeld – who has also written two books about the horrific Baltic wars of the 1990s – argues that many of what the mainstream media call genocides should be described as war crimes instead: brutal, unacceptable mass killings of defenseless humans that nonetheless take place in the the context of reducing a population’s ability to wage war. Genocide, he argues, is a term that should be reserved to describe an effort to completely exterminate a population and leave it incapable of ever recovering. In the Rwandan genocide, for example, the Hutu killers often preferred to murder women and children first, because it would leave the Tutsi population less capable of carrying on to the next generation.

History

Modern Rwanda has three main ethnic groups: the majority Hutu, the minority Tutsi, and a small population of Twa jungle-dwelling hunter-gatherers. At the time of the 1994 genocide the Hutus made up 80-90% of the population, the Tutsis 10-20%, and the Twa around 1%. The Twa were not involved in the genocide or the civil war that led up to it, although they experienced considerable hardship at the hands of the Hutu government in the prior two decades.

Before the colonial era, the area that makes up modern-day Rwanda was organized around Hutu and Tutsi tribal groups, with a relatively small Tutsi aristocracy wielding the most historical influence. The European colonial period began in the 1880s, when Germany took political control of the area as part of German East Africa. Colonial influence was relatively mild until Belgium took control from Germany during World War I. Both Europoean countries encouraged and assisted the Tutsi-dominated power structure, partially due to a racialist belief that the Tutsis were more “Aryan” than the Hutus. The Belgians also formalized the division between Hutu and Tutsi, requiring that Rwandans carry identity cards that listed their ethnic group.

With the decline of colonial power following World War II, however, a Rwandan independence movement took shape that unsurprisingly started to concentrate power in the vastly more numerous Hutu population. Slowly, Belgium switched its support from the Tutsis to the Hutus, a move backed in the country by the powerful and popular Roman Catholic church. In 1959 the “Rwandan Revolution” took place: following the murder of a popular Hutu chief by members of a Tutsi political group, Hutu activists started mass killings of Tutsis. Several hundred thousand Tutsis fled to neighboring countries, and Belgium organized elections that immediately resulted in Hutu dominance.

Frequent anti-Tutsi violence continued for the next thirteen years, when a military coup installed the Hutu Juvenal Habyarimana as president in 1973. Under military rule, anti-Tutsi violence diminished, but an undercurrent always remained. Through the 1980s, Tutsi refugees in Uganda fought as part of guerilla campaigns in the Ugandan Bush War and eventually started planning a campaign against Rwanda as part of an organization called the Rwandan Patriotic Front, or RPF. The RPF attacked Rwanda from Uganda on several occasions, leading to military interventions by both France and Zaire. Anti-government, pro-Hutu extremist parties gained strength as a result of the attacks, and small scale reprisal killings of Tutsis began.

Genocide

In 1993, the hardline Hutu Power movement gained strength on an anti-Tutsi platform and began compiling formal lists of “traitors” for extermination. A popular new radio station founded by Hutu extremists began broadcasting hardline anti-Tutsi programming and violently racist comedy. The new Hutu president of neighboring Burundi was assassinated by Tutsi army officers, which led to detailed planning for a Hutu-led massacre in Rwanda.

The final trigger event occurred in April of 1994, when President Habariyama’s plane was shot down and the president killed. It reamains unknown who was responsible for the the attack on the plane; blame has been placed on both pro-Tutsi RPF forces and on pro-Hutu extremists. Regardless of who was responsible, the Hutu response was immediate: a combined group of Rwandan soldiers and civilians murdered the regime’s successor and Prime Minister, along with a small group of Belgian soldiers assigned to protect her. Within hours, Hutu extremists began systematically murdering moderate politicians and journalists. As the genocide continued, thousands of moderate Hutus were killed along with Tutsis.

Within days, army and civilian militia leaders ordered Hutus to kill their Tutsi neighbors, and the Hutus obeyed without hesitation. Orders were broadcast by radio and verbally at public gatherings in many locations. The explicit goal was to kill every Tutsi, without exception.

Machete Season

The killers interviewed in the book were from the Ntarama area of central Rwanda. Their explanation of the massacre begins starkly:

During that killing season we rose earlier than usual, to eat lots of meat, and we went up to the soccer field at around nine or ten o’clock. The leaders would grumble about latecomers, and we would go off on the attack. Rule number one was to kill. There was no rule number two. It was an organization without complications.

Ntarama was the location of one of the largest massacres in the early days of the genocide: over 5000 people were killed in the Ntarama Church over the course of a day or two, hacked by machete, shot, killed by grenade, or eventually, burned alive.

Over the next three months, an estimated 5 out of 6 Tutsis in Rwanda were murdered and the country lost almost 20% of its population. The Hutus and Tutsis shared region, language and religion: for the most part, Hutus knew who to kill simply because they were killing their neighbors. The stories of the killers in the book reflect at length on the practical aspects of the genocide: they remember the camaraderie with their Hutu cohort, the relief from agricultural work, the short-term material gains of goods taken from their victims, and the physical hardship of killing so many people by hand, by machete.

In response, the RPF launched an invasion from Uganda at the start of the genocide that was ultimately successful. The Hutu military (along with the civilian population) was largely occupied with the genodicde and was too disorganized to mount much of a defense. The genocide moved too fast for the RPF invasion to help much, though: the killings stopped simply because there were few Tutsis left to kill.

Aftermath

The machete season left Rwanda in a shambles. Following the RPF invasion and fearing reprisals, hundreds of thousands of Hutus fled to Zaire, leading to a severe refugee crisis in the neighboring country. Many villages were left uninhabitable. At least a million Hutus, constituting over 20% of the remaining population, are thought to have participated in the genocide. The RPF-led government began mass imprisonments and mass trials, overwhelming the country’s tiny judicial infrastructure. The overcrowded prisons are where Hatzfeld interviewed the killers, over a period of several months. The judicial authorities ultimately decided to categorize participants in the genocide into four classes, according to the severity of their crimes. Penalties ranged from death to a few years’ imprisonment.

The twenty years since the genocide have been difficult and complex, with difficulties in conducting fair trials, accusations of large-scale reprisal killings by the RPF, and continued warfare on multiple fronts.

Summary of Pol Pot: Anatomy of a Nightmare

Mon, 02 Jan 2017 00:00:00 +0000

Overview

I’ve decided to start a reading project on genocides and violent totalitarian dictators. Most education about these topics in the US is focused around Nazi Germany, or occasionally the Soviet Union under Stalin. While I’d like to come back to those events if I can endure the topic that long, I’m starting with non-Western events.

First up is Pol Pot and the Khmer Rouge, partly because I grew up in the 1980s around a lot of first or second-generation Cambodian and Vietnamese immigrants, but never knew much about the politics behind their flight from Southeast Asia. It’s a particularly strange case of different cultural, political, and historic influences converging in a disastrous way. The term “genocide” has been controversial with reference to the Khmer Rouge regime: while they systematically murdered or starved somewhere between 1.7 and 2.3 million people, for the most part the killings didn’t target a specific racial, ethnic, or religious group. While there were certainly elements of this – as I’ll discuss – Pol Pot’s regime was more about brutal slavery and vicious punishment of any deviance, regardless of the person.

This post is based on the book Pol Pot: Anatomy of a Nightmare, by Philip Short. The book is exhaustively detailed and very well-written; I highly recommend it if you want to learn more.

Background

Cambodia entered the post-WW2 era chafing under a century of French colonialism. The hereditary monarch, King Sihanouk, ruled in name only. Cambodia held a cultural memory of greatness that hearkened back to the 12th century, when the lost kingdom of Angkor Wat was the jewel of southeast Asian civilization. The simultaneous stress of a communist insurgency in Vietnam and the rise of nationalist political movements in Cambodia was pushing an already strained French colonial rule close to the breaking point.

Into this mix came Pol Pot. As the child of an middle-class (relative to the deep poverty that defined the agrarian under-classes) family, he won a prestigious scholarship to study radio technology in Paris. There, he and other Cambodian students quickly become involved with the French Communist Party. The Cambodians weren’t ingesting European Marxism wholesale, though: being relatively new to the French language, they were impatient with French translations of abstract, technical Russian and German texts on the nuances of Marxist theory. Furthermore, Marxism’s requirement for a culture to pass through a phase of industrial captialism before communist revolution didn’t sit well with Cambodia’s pre-industrial, agrarian cultural background. For Cambodia, Mao’s interpretation was a better fit: the industrial phase was not required.

Pol ended up returning to Cambodia with a mixture of Marxism, Maoism, and Kropotkin’s anarchism (which admired the French revolution rather than Bolshevism) under his belt. To this he added a cultural history of Theravada Buddhism.

Independence

In the early 1950s, France agreed to Cambodian independence with the stipulation that hereditary monarchy give way to a constitutional monarchy with open elections. Faced with a popular nationalist opposition party, Sihanouk abdicated his throne to his father to participate in politics. The elections were heavily rigged in his party’s favor. After winning, the new government entered into a balancing act between captialist and communist influences: likely due to a secret agreement with Vietnam, Vietnamese communist activity was limited to border areas in exchange for Cambodia’s refusal to accept anti-communist aid from the United States or Western Europe.

This pattern continued into the mid-1960s. The Khmer communists in Cambodia, led in secret by Pol Pot, slowly and steadily gained strength, quietly forming alliances with China. China, no ally of the Soviets, sought to mitigate Soviet influence in Vietnam by aiding the Khmer. Sihanouk continued the violent repression of opposition parties while keeping the United States at a distance.

Revolution

In the late 1960s, disputes over rice trade and further oppression by Sihanouk led to widespread unrest that set the stage for an open insurrection by Pol Pot and the Khmer Rouge communists. An ostensibly anti-communist but largely pointless heavy bombing campaign by the United States near the end of the Vietnam War further destabilized the rural areas of the country. Guerrilla warfare between various factions continued until 1975, when the Khmer Rouge took power under Pol.

The Killing Fields

Pol Pot’s vision of an agrarian revolution quickly led to slavery. The country’s larger cities were forcibly evacuated and their inhabitants resettled as peasants in rural areas, forming an underclass called “base people”. Virtually all educated people, including teachers, engineers, and technicians of every type were forced into re-education through hard labor. Part of Pol’s vision included forced equality in every way: everyone needed to dress the same, eat the same amount, and so on. Some leaders were reportedly vexed that people had differing heights. Families were alloted the same amount of rice regardless of size. Those who couldn’t tolerate the required labor were killed or starved.

Ethnic Chinese, ethnic Thai, and Christian and Muslim minority populations were often singled out for starvation or execution.

Rampant anti-intellectualism rapidly led to practical problems: Pol’s goal of increased rice production required irrigation systems to be built, but the lack of engineering or construction expertise led to a series of dam failures that only made food shortages worse. Village cadres who failed to meet their rice production quotas were likely to be imprisoned or executed, so they systematically inflated their production numbers in a predictable case of “post-fact” self-interest.

Conspiracy theories of every sort were put forth to explain the failures of the revolutionary cause. With neighboring Vietnam allied with the Soviet Union, the Khmer Rouge entered into a relucant alliance with China to counter Vietnamese aggression. A series of increasingly savage purges resulted in the murder of thousands of suspected traitors, many of whom were killed after torture-induced confessions in a secret prison called S-21. Victims were often killed by draining their blood to supply military hospitals. Of the 15,000 - 25,000 people thought to have been sent to S-21, only twelve are known to have survived.

Both the Vietnamese and the US were blamed in these conspiracy theories: forced confessions under torture typically produced detailed false tales of complex CIA or Vietnamese involvement in counter-revolutionary activities. Constant border clashes between Khmer and Vietnamese forces stoked these theories, with each side making multiple invasion attempts.

China eventually attacked Vietnam in defense of the Khmer Rouge, but their respite was brief: a final Vietnamese invasion in 1979 toppled what remained of the Pol’s government, but not until at least 1.7 million Cambodians and ethnic or religious minorities had been killed or starved to death by the Khmer Rouge. The Khmer Rouge retreated into the mountains of Thailand and continued a Chinese-backed guerrilla war against Vietnam for more than a decade, until a 1992 treaty was signed that began a coalition government made up of Vietnamese communists and non-communist remnants of the Khmer Rouge.

Pol Pot died peacefully in his sleep in 1996.

CCIE Recert Should Be Like Wilderness Medicine Recert

Thu, 17 Sep 2015 00:00:00 +0000

Last winter I had to recertify CCIE. This time it felt like a negative, adversarial ordeal: reviewing and relearning a lot of stuff that I don’t use in order to justify the sunk costs of obtaining the certification. It’s also a zero-sum game: time spent on recertification is time not spent learning newer, more relevant things. I’ve seen a couple of blog posts (here and here) lately related to this issue. How could recertification be done better?

Outside my professional life, I’ve long been a search and rescue volunteer here in rural Colorado. As part of that, I maintain a Wilderness First Responder (WFR) certification. WFR is a certification for remote emergency medical care that starts as an 80-hour class. It’s required for most types of guiding and outdoor education careers.

Unlike with the CCIE, I always look forward to WFR recertification, even though it’s expensive and I have to take vacation time in order to do it. Why? It’s fun, cooperative, progressive, educational, and encouraging. It’s done as a 16-24 hour class that mixes classroom review, hands-on lab practice, and new material that’s been introduced or updated in the preceding years. This allows recertification candidates to interact with their peers, learn new skills, refine old ones, and demonstrate their competency in front of live instructors. By the end of the class, the test is largely a formality: it’s not that hard, and you really have to screw up badly to fail. The instructors might not even let you take it if you’ve gone that far astray.

Although I have little hope of it ever happening, I think this should be the model for CCIE recertification: interaction with peers, review of basics, updates on new technology, and lab exercises.

There are a couple of objections to ideas like this:

It would be expensive, and thus unfair to applicants from different countries or different income levels. I understand this; maybe Cisco could offer this idea in addition to the traditional path.
It “devalues” the certification, presumably by making recertification easier. I don’t understand this objection; it’s not that far from wishing for an artificial cap on the CCIE population. I took the CCIE exam because it was challenging to me, not because of what it does or doesn’t do for other people. I’m interested in being challenged in cooperative and constructive ways, not in excluding other people from the club. Rather than worrying that other people might not be good enough, we could use opportunities like this recertification model to help them get better.

Installing netmiko on Windows

Wed, 15 Jul 2015 19:12:00 +0000

Netmiko is a Python module by Kirk Byers that provides a wrapper around the Paramiko SSH module for doing screen scraping and CLI automation on network devices.

Paramiko has some dependencies that make installation on Windows a tad tricky. Here's a quick way to get it done:

Install Anaconda.
From the Anaconda shell, run "conda install paramiko".
From the Anaconda shell, run "pip install scp".
Install git for Windows.
Clone netmiko with "git clone https://github.com/ktbyers/netmiko"
cd into the netmiko directory and run "python setup.py install".

Done! Screen scrape away, and don't forget to hound your vendors for real APIs... :-)

Extracting Traffic from Rolling Capture Files

Tue, 14 Jul 2015 17:35:00 +0000

Every so often I need to extract a subset of traffic from a set of rolling timestamped pcap files. One common place I do this is with Security Onion; one of the great features of SO is its full-packet-capture feature: you can easily pivot from Snort, Suricata, or Bro logs to a full packet capture view, or download the associated pcap file.

But what if you don't have an associated alert or Bro log entry? Or if you're doing pcap on some system that's not as user-friendly as Security Onion, but nonetheless supports rolling captures?

The way I usually do this is with find and xargs. Here's an example of my most common workflow, using timestamps as the filtering criteria for find:

> find . -newerct "16:07" ! -newerct "16:10" | xargs -I {} tcpdump -r {} -w /tmp/{} host 8.8.8.8
> cd /tmp
> mergecap -w merged.pcap *.pcap

Translated:

Find all files in the current directory created after 16:07 but not created after 16:10. This requires GNU find 4.3.3 or later. It supports many different time and date formats.
Using xargs, filter each file with the "host 8.8.8.8" BPF expression and write it to /tmp with the same filename.
Merge all the .pcap files in /tmp into merged.pcap.

You can easily modify this workflow to fit other use cases.

More ADN (Awk Defined Networking)

Fri, 15 May 2015 20:52:00 +0000

Want to know how many IPv4 nodes are in each of your VLANs? Use ADN:

ssh myswitch 'sh arp | i Vlan' | awk '{print $NF}' | sort | uniq -c | sort -rn

     79 Vlan38
     65 Vlan42
     58 Vlan34
     22 Vlan36
     21 Vlan32
     20 Vlan40
      9 Vlan3
      7 Vlan8
      5 Vlan6
      5 Vlan204
      5 Vlan203
      5 Vlan2
      4 Vlan74
      3 Vlan82
      3 Vlan4

ADN - Awk Defined Networking

Fri, 24 Apr 2015 16:39:00 +0000

Because I have yet to transition to a completely software-defined network in which everything configures itself (wink wink), I still have to do tasks like bulk VLAN changes.

Thanks to a recent innovation called ADN, or "AWK Defined Networking", I can do this in a shorter time window that the average bathroom break. For example, I just had a request to change all ports on a large access switch stack that are currently in VLAN 76 to VLAN 64:

# ssh switch_name.foo.com 'show int status | i _76_' | grep Gi | awk '{print "int ",$1,"\n","description PC/Phone","\n","switchport access vlan 64"}'
Password: ***

int Gi1/0/25
description PC/Phone
switchport access vlan 64
int Gi1/0/26
description PC/Phone
switchport access vlan 64
[many more deleted]

Then I copied and pasted the results into config mode. Back to lounging on the beach.

Not even any Python skills required!

Quick Example: Elasticsearch Bulk Index API with Python

Fri, 27 Mar 2015 02:17:00 +0000

A quick example that shows how to use Elasticsearch bulk indexing from the Python client. This is dramatically faster than indexing documents one at a time in a loop with the index() method.

Filtering .raw fields with Python Elasticsearch DSL High-Level Client

Sun, 01 Mar 2015 03:27:00 +0000

It took me a while to figure out how to search the not_analyzed ".raw" fields created by Logstash in Elasticsearch indices, using the high-level Python Elasticsearch client. Because keyword arguments can't have attributes, Python throws an error if you try it the intuitive way (this assumes you've already set up a client as es and an index as i, as shown in the docs):

Instead, you create a dictionary with your parameters and unpack it using the ** operator:

This produces the Elasticsearch query we want:

Experimenting with the CSR1000v REST API

Thu, 12 Feb 2015 18:18:00 +0000

This all started because we occasionally want to block traffic from an IP address or two for a short time. Our firewall is a pain to configure for this sort of thing: adding a drop for a single IP address literally takes 10 minutes. You have to open a fat client, create an object, add the object to a group, save the config, verify the config, push the config, etc.

I thought that SRTBH (Source-based Real-Time Black Hole) implemented by BGP would be the ticket: fast, easy, and theoretically easy to automate with the REST API in the Cisco Cloud Services Router 1000v. SRTBH is a simple and elegant way of dropping selected traffic on BGP speaking routers. In a nutshell:

You configure a "trigger router" that speaks iBGP with the rest of your BGP-speaking routers (usually your Internet edge or transit routers), but doesn't participate in traffic forwarding.
On each edge/transit router you configure a static route to null0 for an unused /32, usually 192.0.2.1: ip route 192.0.2.1 255.255.255.255 null0
On each edge/transit router you configure loose-mode unicast RPF filtering on your outside interfaces: ip verify source reachable-via any. This has a special property: packets from any source for which the uRPF next-hop resolves to Null0 will be dropped.
When you want to drop traffic from a particular host (e.g. 1.1.1.2), you configure a static host route on the trigger router and redistribute it into BGP by matching its tag: ip route 1.1.1.2 255.255.255.255 null0 tag 666.
At redistribution, you set its next-hop to 192.0.2.1:

route-map SRTBH permit 10
description BGP Black Hole
match tag 666
set origin igp
set community no-export
set ip next-hop 192.0.2.1

The effect of this is that the edge routers see the iBGP route to the "bad" address as a /32 with a next-hop of 192.0.2.1. Because the next-hop for 192.0.2.1 recursively resolves to null0, the loose uRPF feature drops traffic from the bad host at the edge.

In my case, I wanted to use the CSR1000v as the trigger router. (At this point, somebody's thinking "Why not use a Linux BGP daemon?") That's a valid approach; I just want to use the CSR1000v.

I followed the REST setup instructions, then wrote a simple CLI tool to make testing easier. It's easy enough to test the API with cURL:

$ curl -X POST https://10.1.1.1:55443/api/v1/auth/token-services -H "Accept:application/json" -u "testuser" -d "" --insecure -3
Enter host password for user 'testuser':
{"kind": "object#auth-token", "expiry-time": "Thu Feb 12 17:24:30 2015", "token-id": "xNOvrJh[...]rW3x0=", "link": "https://10.1.1.1:55443/api/v1/auth/token-services/2285580579"}

but because the API requires that you obtain a temporary authorization token (demonstrated above) before doing anything interesting, writing the Python tool made sense.

Then the problems started: the REST API on the CSR1000v doesn't support tags on static routes. You can't add them, or even retrieve them. Here's what happens normally if you retrieve static routes:

>python rest.py --device 10.1.1.1 --user testuser --resource /routing-svc/static-routes
Password:
{u'items': [{u'admin-distance': 1,
             u'destination-network': u'10.0.0.0/8',
             u'kind': u'object#static-route',
             u'next-hop-router': u'10.30.254.152',
             u'outgoing-interface': u''},
u'kind': u'collection#static-route'}

If one of the routes has a tag, it throws an error:

{u'detail': u' ',
u'error-code': -1,
u'error-message': u"invalid literal for int() with base 10: 'tag'"}

You can't set tags either. Furthermore, the same thing holds true for static routes with the "name" argument. (Incidentally, the error message seems to indicate that the IOS XE API is written in Python.)

The problem with this is that without tags, there's no convenient way to filter the RTBH black hole trigger routes at the time of redistribution. You can't set route-maps on BGP network statements via the API either, so no joy there.

The best option to me seems to be to redistribute all static routes, but filter on prefix-length rather than tag:

route-map SRTBH permit 10
description BGP Black Hole
match ip address prefix-list HOST_ROUTES_ONLY
set origin igp
set community no-export
set ip next-hop 192.0.2.1

But then you have to make sure that you never have any /32s that aren't trigger routes... a strategy which is prone to configuration mistakes.

Because SRTBH seems like a natural use-case for API automation, I hope Cisco fixes this soon.

[added]
Other interesting things I've found: the static route configuration API doesn't allow you to add equal-cost static routes. For example, if I already have a 1.1.1.5/32 route and I add another one, I get a 404:

python rest.py --device 10.1.1.1 --user testuser --password foo --resource /routing-svc/static-routes --method post --json '{"destination-network":"1.1.1.5/32","outgoing-interface":"null0"}' --verbose
[snip]
DEBUG:root:got status code: 404
DEBUG:root:reponse content: Static route already exists
got status code: 404

The same thing goes for adding other items: I tried adding a name-server twice, and got a similar error. I would think that the API would be idempotent where possible (such as when adding a duplicate nameserver), and require clarifying attributes otherwise. With equal-cost static routes for example, it could throw a 40x unless an extra attribute like {"allow-equal-cost":true} is present in the POST request.

Pleasing terminal colors on Security Onion

Thu, 15 Jan 2015 22:17:00 +0000

To get the lovely Solarized theme working in Security Onion:

sudo apt-get install gnome-terminal

I'm sure there's a way to get in working in the default xfce4 terminal, but I couldn't figure it out.

Follow instructions here: http://stackoverflow.com/questions/23118916/configuring-solarized-colorscheme-in-gnome-terminal-tmux-and-vim

Problems with kvm-ok in VIRL with VMWare Player

Thu, 08 Jan 2015 22:21:00 +0000

I'm installing Cisco VIRL, and despite following the instructions regarding nested virtualization settings, the kvm-ok command was still complaining. I needed to edit the .vmx file for the VIRL VM and add/edit the following:

monitor.virtual_mmu = "hardware"
monitor.virtual_exec = "hardware"
vhv.enable = "TRUE"
monitor_control.restrict_backdoor = "true"

My Network Toolkit

Fri, 02 Jan 2015 17:45:00 +0000

A while back, Chris Marget of Fragmentation Needed posted a run-down of his comprehensive and extremely clever network toolkit. Because I'm something of a weight weenie, mine is a lot more slimmed down. I thought I'd post it here:

The contents:

Two random USB drives (in case I need to leave one with somebody).
Single-mode and multi-mode LC fiber loopback plugs.
Rack PDU plug adapter.
Awesome PicQuic compact screwdriver (thanks to Chris's post).
T1 loopback plug (red) (because we still have T1s out here in the boonies).
Cat-6 pass-through plug (white).
Crossover adapter (orange).
Sharpie.
Console setup:

USB-to-DB9 adapter.
DB9-to-RJ45 adapter.
Flat Cat-6 cable.
Rollover adapter.
Velcro tie
Flat Cat-6 cable with velcro tie.

The console setup could probably be improved by adding a DB9 null-modem adapter. The coolest thing (IMO) that I'm missing from Chris's setup is the Bluetooth console adapter -- maybe one day.

The Fenix AA light and Leatherman Skeletool CX almost always live in a pocket rather than the kit and go with me everywhere. The kit all fits into a small zippered case that used to hold a Dell laptop power supply.

My main goal here was to have all the hard-to-find professional stuff in one small package. I have a separate "personal" kit that contains stuff like headphones, USB cables, and chargers for personal electronics.

Imposing Artificial Limitations to Develop Skills

Fri, 05 Dec 2014 15:49:00 +0000

I'm a big fan of imposing artificial limitations on yourself in order to aid skill development. Here are some quick ideas:

When troubleshooting network devices from the CLI, try not to look at the configuration. Use only "show" or "debug" commands instead. I found this enormously beneficial when practicing for CCIE.
When troubleshooting larger operational issues or learning a new environment, try not to log into individual devices at all. Force yourself to use only your network management system, NetFlow, packet captures, or host-based tools like ping, traceroute, or nmap.
When learning automation or orchestration skills, force yourself to write scripts, run API calls, or use your favorite orchestration tool to do simple things, even if it doesn't seem like they merit the extra effort.

Simple Python Syslog Counter

Wed, 02 Jul 2014 01:24:00 +0000

Recently I did a Packet Pushers episode about log management. In it, I mentioned some of the custom Python scripts that I run to do basic syslog analysis, and someone asked about them in the comments.

The script I'm presenting here isn't one of the actual ones that I run in production, but it's close. The real one sends emails, does DNS lookups, keeps a "rare messages" database using sqlite3, and a few other things, but I wanted to keep this simple.

One of the problems I see with getting started with log analysis is that people tend to approach it like a typical vendor RFP project: list some requirements, survey the market, evaluate and buy a product to fit your requirements. Sounds good, right? The problem with log analysis is that often you don't know what your requirements really are until you start looking at data.

A simple message counting script like this lets you look at your data, and provides a simple platform on which you can start to iterate to find your specific needs. It also lets us look at some cool Python features.

I don't recommend pushing this too far: once you have a decent idea of what your data looks like and what you want to do with it, set up Logstash, Graylog2, or a similar commercial product like Splunk (if you can afford it).

That said, here's the Python:

I tried to make this as self-documenting as possible. You run it from the CLI with a syslog file as the argument, and you get this:

$ python simple_syslog_count.py sample.txt
214   SEC-6-IPACCESSLOGP
15    SEC-6-IPACCESSLOGRL
10    LINEPROTO-5-UPDOWN
10    LINK-3-UPDOWN
7     USER-3-SYSTEM_MSG
4     STACKMGR-4-STACK_LINK_CHANGE
4     DUAL-5-NBRCHANGE
3     IPPHONE-6-UNREGISTER_NORMAL
3     CRYPTO-4-PKT_REPLAY_ERR
3     SEC-6-IPACCESSLOGRP
3     SEC-6-IPACCESSLOGSP
2     SSH-5-SSH2_USERAUTH
2     SSH-5-SSH2_SESSION
2     SSH-5-SSH2_CLOSE

10.1.16.12

     6     SEC-6-IPACCESSLOGP

10.1.24.3

     2     LINEPROTO-5-UPDOWN
     2     LINK-3-UPDOWN

[Stuff deleted for brevity]

For Pythonistas, the script makes use of a few cool language features:

Named, Compiled rRgexes

We can name a regex match with the (?PPATTERN) syntax, which makes it easy to understand it when it's referenced later with the .group('') method on the match object.
This is demonstrated in lines 36-39 and 58-59 of the gist shown above.
It would be more efficient to capture these fields by splitting the line with the .split() string method, but I wanted the script to work for unknown field positions -- hence the regex.

Multiplication of Strings

We control indentation by multiplying the ' ' string (that a single space enclosed in quotes) by an integer value in the print_counter function (line 50).

The reason this works is that the Python str class defines a special __mul__ method that controls how the * operator works for objects of that class:
>>> 'foo'.__mul__(3)
'foofoofoo'
>>> 'foo' * 3
'foofoofoo'

collections.Counter Objects

Counter objects are a subclass of dictionaries that know how to count things. Jeremy Schulman talked about these in a comment on the previous post. Here, we use Counters to build both the overall message counts and the per-device message counts:

>>> my_msg = 'timestamp ip_address stuff %MY-4-MESSAGE:other stuff'
>>> CISCO_MSG = re.compile('%(?P.*?):')
>>> from collections import Counter
>>> test_counter = Counter()
>>> this_msg = re.search(CISCO_MSG,my_msg).group('msg')
>>> this_msg
'MY-4-MESSAGE'
>>> test_counter[this_msg] += 1
>>> test_counter
Counter({'MY-4-MESSAGE': 1})

collections.defaultdict Dictionaries

It could get annoying when you're assigning dictionary values inside a loop, because you get errors when the key doesn't exist yet. This is a contrived example, but it illustrates the point:

>>> reporters = {}
>>> for reporter in ['1.1.1.1','2.2.2.2']:
... reporters[reporter].append['foo']
...
Traceback (most recent call last):
File "", line 2, in
KeyError: '1.1.1.1'
To fix this, you can catch the exception:

>>> reporters = {}
>>> for reporter in ['1.1.1.1','2.2.2.2']:
...     try:
...         reporters[reporter].append['foo']
...         reporters[reporter].append['bar']
...     except KeyError:
...         reporters[reporter] = ['foo']
...         reporters[reporter].append('bar')

As usual, though, Python has a more elegant way in the collections module: defaultdict

>>> from collections import defaultdict
>>> reporters = defaultdict(list)
>>> for reporter in ['1.1.1.1','2.2.2.2']:
... reporters[reporter].append('foo')
... reporters[reporter].append('bar')
>>> reporters
defaultdict(, {'1.1.1.1': ['foo', 'bar'], '2.2.2.2': ['foo', 'bar']})

In the syslog counter script, we use a collections.Counter object as the type for our defaultdict. This allows us to build a per-syslog-reporter dictionary that shows how many times each message appears for each reporter, while only looping through the input once (line 66):

per_reporter_counts[reporter][msg] += 1

Here, the dictionary per_reporter_counts has the IPv4 addresses of the syslog reporters as keys, with a Counter object as the value holding the counts for each message type:

>>> from collections import Counter,defaultdict
>>> per_reporter_counts = defaultdict(Counter)
>>> per_reporter_counts['1.1.1.1']['SOME-5-MESSAGE'] += 1
>>> per_reporter_counts
defaultdict(, {'1.1.1.1': Counter({'SOME-5-MESSAGE': 1})})
>>> per_reporter_counts['1.1.1.1']['SOME-5-MESSAGE'] += 5
>>> per_reporter_counts
defaultdict(, {'1.1.1.1': Counter({'SOME-5-MESSAGE': 6})})

If you got this far, you can go implement it for IPv6 addresses. :-)

Python Sets: Handy for Network Data

Fri, 20 Jun 2014 16:55:00 +0000

My Python-related posts seem to get the most reads, so here's another one!

A problem that comes up fairly often in networking is finding the number of occurrences of unique items in a large collection of data: let's say you want to find all of the unique IP addresses that accessed a website, traversed a firewall, got denied by an ACL, or whatever. Maybe you've extracted the following list from a log file:

1.1.1.1
2.2.2.2
3.3.3.3
1.1.1.1
5.5.5.5
5.5.5.5
1.1.1.1
2.2.2.2
...

and you need to reduce this to:

1.1.1.1
2.2.2.2
3.3.3.3
5.5.5.5

In other words, we're removing the duplicates. In low-level programming languages, removing duplicates is a bit of a pain: generally you need to implement an efficient way to sort an array of items, then traverse the sorted array to check for adjacent duplicates and remove them. In a language that has dictionaries (also known as hash tables or associative arrays), you can do it by adding each item as a key in your dictionary with an empty value, then extract the keys. In Python:

>>> items = ['1.1.1.1','2.2.2.2','3.3.3.3','1.1.1.1','5.5.5.5','5.5.5.5','1.1.1.1','2.2.2.2']
>>> d = {}
>>> for item in items:
... d[item] = None
...
>>> d
{'5.5.5.5': None, '3.3.3.3': None, '1.1.1.1': None, '2.2.2.2': None}
>>> unique = d.keys()
>>> unique
['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2']

or, more concisely using a dictionary comprehension:

>>> {item:None for item in items}.keys()
['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2']

Python has an even better way, however: the "set" type, which emulates the mathematical idea of a set as a collection of distinct items. If you create an empty set and add items to it, duplicates will automatically be thrown away:

>>> s = set()
>>> s.add('1.1.1.1')
>>> s
set(['1.1.1.1'])
>>> s.add('2.2.2.2')
>>> s.add('1.1.1.1')
>>> s
set(['1.1.1.1', '2.2.2.2'])
>>> for item in items:
... s.add(item)
...
>>> s
set(['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2'])

Predictably, you can use set comprehensions just like list comprehensions to do the same thing as a one liner:

>>> {item for item in items}
set(['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2'])

Or, if you have a list built already you can just convert it to a set:

>>> set(items)
set(['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2'])

Python also provides methods for the most common types of set operations: union, intersection, difference and symmetric difference. Because these methods accept lists or other iterables, you can quickly find similarities between collections of items:

>>> items
['1.1.1.1', '2.2.2.2', '3.3.3.3', '1.1.1.1', '5.5.5.5', '5.5.5.5', '1.1.1.1', '2.2.2.2']
>>> more_items = ['1.1.1.1','8.8.8.8','1.1.1.1','7.7.7.7','2.2.2.2']
>>> set(items).intersection(more_items)
set(['1.1.1.1', '2.2.2.2'])
>>> set(items).difference(more_items)
set(['5.5.5.5', '3.3.3.3'])

Have fun!

Fun with Router IP Traffic Export and NSM

Wed, 02 Apr 2014 15:54:00 +0000

The Basics
I finally got around to setting up Security Onion (the best network security monitoring package available) to monitor my home network, only to discover that my Cisco 891 router doesn't support support the right form of SPAN. Here's how I worked around it. The topology looks like this:

The 891 router has an integrated 8-port switch module, so the simple case would have been a traditional SPAN setup; something like this:

! vlan 10 is the user VLAN
monitor session 1 source interface vlan 10
monitor session 1 destination interface FastEthernet0

with the server's monitoring NIC connected to FastEthernet0.

The problem is that the 891 doesn't support using a VLAN as a source interface, and because of the way the embedded WAP works, a physical source interface won't work either. Hence, I turned to an obscure feature that's helped me occasionally in the past: Router IP Traffic Export. This is a feature for IOS software platforms that enables you to enable SPAN-like functions for almost any source interface.

The configuration looks like this:

ip traffic-export profile RITE_MIRROR
interface FastEthernet0
bidirectional
mac-address 6805.ca21.2ddd

interface Vlan10
ip traffic-export apply RITE_MIRROR

This takes all traffic routed across the Vlan10 SVI and sends it out the FastEthernet0 interface, rewriting the destination MAC address to the specified value. I used the MAC address of my monitoring NIC, but it shouldn't matter in this case because the monitoring NIC is directly attached. If I wanted to copy the traffic across a switched interface, it would matter.

My ESXi host (a low-cost machine from Zareason with 32GB of RAM) has two physical NICs; one for all of the regular VM traffic (using 802.1q to separate VLANs if needed) and one for monitoring. The monitoring pNIC is attached to a promiscuous mode vSwitch in ESXi, which in turn is connected to the monitoring vNIC on the Security Onion VM. The effect of this is identical to SPAN-ing all the traffic from VLAN 10 to my Security Onion monitoring system; I get Snort, Bro, Argus, and full packet capture with just the built-in software tools in IOS and ESXi.

Oddity: RITE Capture, Tunnels?
Interestingly, you can also use RITE to capture traffic to a RAM buffer and export it to a pcap file. I don't understand why you would use this instead of the much more flexible Embedded Packet Capture Feature, though.

Another thing I've wondered is whether you could use a L2 tunnel to send the mirrored traffic elsewhere in the network. The destination interface must be a physical Ethernet interface, but it would be interesting to try using a L2TPv3 tunnel from an Ethernet interface to another router--I have no idea if this would work.

Production Use?
Cisco makes the UCS E-series blades for ISR G2 routers that let you run a hypervisor on a blade inside your router chassis. These things have an external Ethernet port on them, so you should be able to connect a RITE export interface to the external port on an E-series blade, and run Security Onion inside your router. I've always wanted to try this, but I haven't been able to get funding yet to test it.

Unroutable

About password security and credential stuffing

Common Falsehoods About Password Security and Credential Stuffing

False: Rate limiting login attempts is a great mitigation

False: Most credential stuffing attacks use “brute force” techniques

False: A stolen unique password means a compromised site

What is to be done?

Books 2020

Books 2020

Intellectual History (?)

A Madman Dreams of Turing Machines - Janna Levin

The Weil Conjectures - Karen Olsson

Classic Fiction

Pride and Prejudice - Jane Austen

Science Fiction

Binti - Nnedi Okorofor

Espionage History

Active Measures - Thomas Rid

The Main Enemy - Milt Rearden and James Risen

Circle of Treason - Sandra Grimes and Jeanne Vertefeuille

2019 Books

2019 Books

(Intellectual History) At the Existentialist Cafe - Sarah Bakewell

(Mystery) The Infinite Blacktop - Sara Gran

(Micro History) Lady on the Beach - Norah Berg

(Philosophy and Mind) Conscious - Annaka Harris

(History of Espionage) The Spy and the Traitor - Ben Macintyre

(Cybersecurity) Sandworm - Andy Greenberg

(Literary History/Autobiography) The Heart to Artemis - Bryher

Book Report: Machete Season

Overview

History

Genocide

Machete Season

Aftermath

Summary of Pol Pot: Anatomy of a Nightmare

Overview

Background

Independence

Revolution

The Killing Fields

CCIE Recert Should Be Like Wilderness Medicine Recert

Installing netmiko on Windows

Extracting Traffic from Rolling Capture Files

More ADN (Awk Defined Networking)

ADN - Awk Defined Networking

Quick Example: Elasticsearch Bulk Index API with Python

Filtering .raw fields with Python Elasticsearch DSL High-Level Client

Experimenting with the CSR1000v REST API

Pleasing terminal colors on Security Onion

Problems with kvm-ok in VIRL with VMWare Player

My Network Toolkit

Imposing Artificial Limitations to Develop Skills

Simple Python Syslog Counter

Named, Compiled rRgexes

Multiplication of Strings

collections.Counter Objects

collections.defaultdict Dictionaries

Python Sets: Handy for Network Data

Fun with Router IP Traffic Export and NSM