programmatic generation of sequential Junos set commands

While building out a new enterprise network for a client using Juniper hardware (SRX240s and EX3300s), I ran into a decision I’ve encountered a number of times: how to generate repetitive configuration. For example:

set interface ge-2/0/0 unit 0 family ethernet-switching port-mode access vlan members corporate-data
set interface ge-2/0/1 unit 0 family ethernet-switching port-mode access vlan members corporate-data
, etc.

Obviously interface-range exists to solve this issue, at least for switchport config, but other members of my team take issue with the transparency of configuration in that form, and I agree somewhat – to verify an interface’s configuration, you’re either following a chain of config items or using the less-than-ideal show vlans. Additionally, interface-range only helps with interface configuration, not for security zones or policies.

In the past, I’ve used a quick bash script that I modify as necessary for outputting the required set commands. This is fine for me, but I wanted to make something a bit easier and more portable for my (Windows-using) team, so I converted it to a quick Python script.

Python is still not the most accessible interpreter for Windows users, so I figured I’d go all the way and create a javascript web version. It’s barebones and the interactivity is a bit weird – I intended to go for the full terminal experience, but my JS and CSS skills are weak at best. Anyway, hope this is useful to someone other than me and my coworkers.

manually set AWS EC2 RHEL7 hostname

I use a few t2.micro EC2 instances running RHEL7 for various general-purpose applications best suited to a small VPS. A personal preference of mine is to have short, descriptive hostnames on each system I regularly interact with, and by default, RHEL7 on AWS uses dynamic hostnames – the system’s hostname is defined by the private IP address of the instance:

Screen Shot 2016-04-10 at 00.01.22

I’m sure that’s useful in some applications (“at scale”), but it’s not what I prefer. Amazon has an article on how to set a static hostname, but it doesn’t seem correct – I have a hard time believing that every step is required. I put this to a test: rather than follow the steps to the letter, I tested each step independently, then in combination with multiple iterations, to see what was actually required. Here’s all that’s needed:

  1. Replace the contents of /etc/hostname with your desired hostname. No idea why Amazon tells you to use HOSTNAME=newhostname, just echo "newhostname" | sudo tee /etc/hostname
  2. Append preserve_hostname: true to /etc/cloud/cloud.cfg
  3. Reboot.

Caveat: I could be wrong, there could be good reasons to go through all of the other steps that Amazon’s doc explains. Can’t imagine why though!

converting powers of ten to powers of two

In this post on cracking md5crypt hashes, I listed the sizes of a number of password search spaces as exponents of 10, e.g. 1.9 * 1017, but I realize now that there’s a much better way to discuss large search spaces. I’ve been taking a Stanford online course on cryptography over the last couple months which has made me realize, among many other things, that it’s much more useful to think in powers of two in matters of cryptography, computing, and information theory in general.

However, powers of ten are easy to deal with in back-of-the-envelope calculations, and many articles/papers/posts deal in powers of ten. I had no idea how to convert from 10x to 2y in any straightforward fashion, so I talked to a couple friends with much stronger math backgrounds than mine and did a bit of reading. Here’s what I found:

For x = 2n, we want to solve for n.
The solution:
n = log(x) / log(2)
For example:
log(1017) / log(2) = 56.472777613085164
thus
1017 ≈ 256

N.B. that’s a very rough approximation at that scale. Definitely good enough for the purposes of estimating the nearest power of two, but worth keeping in mind there’s still a large difference (well, exactly double) between any two integer powers of two.

I was confused about the log operations – is this log base-10? if not, what actual operation is taking place? Turns out it’s the natural logarithm, aka loge x. However, it’s not actually important that this is base-e, just that the base of both logarithm operations is the same – it cancels out. We can show this in Python, where a base can be provided as an optional second argument to math.log:

>>> log(10000) / log(2)
13.28771237954945
>>> log(10000, 10) / log(2, 10)
13.287712379549452

How to actually use this in practice? Lots of ways:

Wolfram Alpha:

Screen Shot 2015-12-10 at 14.44.55

Python:

Screen Shot 2015-12-10 at 14.37.38

Javascript:

Screen Shot 2015-12-10 at 14.43.50

OS X spotlight (different result…?):

Screen Shot 2015-12-10 at 14.46.12

UPDATE 2015-12-29:

I wrote a command-line tool to do this conversion quickly, on GitHub here. It’s in my PATH so I can do this:

[tkerr@pro ~]$ tentotwo 39
10 ^ 39   =   2 ^ 129.55519570060713   ≈  2 ^ 130
[tkerr@pro ~]$ tentotwo -r 128
2 ^ 128   =   10 ^ 38.531839444989586   ≈  10 ^ 39

split DNS/dns-proxy in Junos

I spent some time last night learning about and implementing split DNS on SRXes via
Junos’ proxy-dns feature, and ran into some interesting complications.

Here’s a diagram of the environment I was working on:

proxydnstopology

This environment consists of five branches, each with an SRX210. There’s a full mesh of IPsec VPN tunnels between them (10 links total, not pictured). Last night’s task was building VPN tunnels from each site to an AWS VPC so that the MS guys on my team could build an AD domain in AWS. Amazon makes this extremely easy, even providing example Junos configs, but even better you can download automatically generated Junos configs with your tunnel’s info prepopulated. The entire process took about 30 minutes for all five sites.

It got interesting when setting up split DNS. The goal here is to have the corporate networks at all five locations able to work with a domain controller in AWS, but without hairpinning all DHCP and DNS traffic through the VPN tunnels to AWS. We can leave DHCP on the SRXes, but AD requires DNS to function, so we need it to at least resolve requests for the AD domain. The natural solution is split DNS: requests for any names in company.com. are forwarded to the domain controller(s), and all other requests are handled by public DNS. Users on the internal network should be using the local firewall as their resolving nameserver (passed via DHCP), and company.com should resolve to its AWS internal IP address of 10.200.1.10 rather than the public A record of 74.122.237.76, company.com’s webserver.

Juniper has some docs here and here, and there are a couple quick articles I also took a look at.

Here’s the config:

tylerkerr@site-a-srx210> show configuration system services dns dns-proxy
interface {
    vlan.10;
}
default-domain * {
    forwarders {
        8.8.8.8;
        8.8.4.4;
    }
}
default-domain company.com {
    forwarders {
        10.200.1.10;
    }
}

I threw the config in, committed, and tested. dig company.com @10.50.1.1 returned 74.122.237.76, the external address – not working. On first attempt there was a delay of about three seconds – this ended up being very important, although I didn’t realize it until later. I cleared the dns-proxy cache (clear system services dns-proxy cache), which didn’t resolve the issue but did reintroduce the three-second delay on first resolution.

At this point I reread the config as well as the blog posts and KB articles to make sure I wasn’t missing something. A guess was that the order of default-domain statements mattered, a la ACLs, but everything I saw indicated otherwise, that this is more of a longest-prefix style ruleset. I tried breaking the config up into a ‘view’ statement, matching clients only in specific CIDR ranges, but the results were identical.

I set up a trace for destination-prefix 10.200.1.10/32 and reproduced the issue. I immediately saw that the source IP for the DNS queries was the IP address for st0.8, the address that AWS’ automatically generated VPN config files gave me: 169.254.45.122, a link-local address. Since I have a static route for the VPC, the SRX was correctly sourcing the queries from this link-local address, but there was no way for the responses to come back in for a number of reasons. This meant that the SRX tried three queries to 10.200.1.10, each timing out after one second, before falling back to Google DNS. This explains the initial symptoms above – a three-second delay followed by a response with the public IP address.

My next plan was to attempt to change the source IP for outbound dns-proxy queries. Some reading pointed me towards the default-address-selection selection, which evidently tells system services to use the firewall’s loopback address rather than sourcing based on outbound interface. I set the loopback address to an unused IP in the corporate subnet, which immediately let DNS flow between the corporate network and AWS, but broke other system services (like all other DNS to Google) since they were now being sourced from an RFC1918 address.

Some more googling found me this j-nsp thread, which was extremely helpful. The key suggestion further down was to “number the st0 interface with a /32 from the corporate range and source my queries from there.” I wasn’t actually sure that this would work – did Amazon give me specific link-local addresses for a reason? would changing these break the tunnels? – but I gave it a try. After a brief dropout while the IPsec tunnel re-established itself, communication to AWS came right back up, meaning that AWS doesn’t care how my side of the tunnel is addressed.

However, after committing this change, I was no longer able to query the SRX for ANY DNS records, getting this from dig:

;; WARNING: recursion requested but not available

I was completely at a loss for this, checking the differences between all of my commits trying to find when DNS stopped working and what could have broken it. I fixed it by committing a “deactivate system services dns dns-proxy” then a reactivation – just a bug.

After this, everything was working properly – company.com resolved to its internal address from the corporate network, other DNS traffic was properly forwarded to Google DNS, and no huge workarounds were necessary.

SHA-1 hashes in Junos

I previously mentioned I’d write about using set system login password format sha1 in Junos. I haven’t had a ton of time, and I blew $80 by launching a g2.8xlarge AWS instance and forgetting about it (do billing alerts only proc weekly?) so I’m postponing the real research, but suffice it to say that SHA-1 password hashes in Junos are appropriately more difficult than $1. I’m still confused about the lack of bcrypt, but I suppose in serious environments nobody else has access to your configs anyway. Someday…

vindication

Is there anything better than installing Wireshark on the VoIP PBX in response to “the firewall is blocking phone registration” to show not only prolific arrivals of multicast SIP registration requests, but the PBX silently responding with HTTP 500s?

Junos’ $1 hashes

Previously I wrote a bit about Junos’ $9 ‘hashes’ (they’re actually reversibly encrypted passwords). While considering the ramifications of nearly-cleartext credentials stored in config files, I thought about the $1 hashes used to authenticate users via telnet, HTTP, console, HTTPS, and SSH (don’t use the first two). I’ve been told multiple times that the “system login user authentication encrypted-password” statements, usually at the beginning of a config, can be copied around as desired to grant that user access. This is similar in function to an SSH public key, but that line of thinking is dangerous – while properly-sized RSA public keys can’t realistically factored to their original private keys, some password hashes can be.

$1 hashes are from md5crypt(). md5crypt was written in 1995 and added to FreeBSD version 2 shortly after. It’s a few extra operations on top of what is essentially 1000 rounds of md5 (stretching) operating on the password and an 8-byte salt. It outputs a base64-encoded version of a 32-byte md5 hash. Apple’s well-commented code from OS X is a good way to see exactly how it works. The output format is $s separating the format, salt, and hash:

$1$SSSSSSSS$hhhhhhhhhhhhhhhhhhhhhh

(id)      (salt)                    (hash)

MD5 is extremely fast, which means it’s extremely broken as a password hash algorithm. My desktop’s fairly new mid-end GPU (a GTX 960) can do 7.1 billion MD5 hashes/second. md5crypt’s stretch factor of 1000 (plus a bit more with the overhead of the extra operations) is still very cooperative to cracking – my GTX 960 does 2.6 million hashes/sec, roughly 2700x slower than MD5 but all things considered very, very fast for password hashing. I won’t write too much about hashing algorithms and speed, as it’s been covered extremely thoroughly by many people. Bottom line: use as slow of an algorithm as is reasonable, with a variable work factor. This leaves bcrypt, scrypt, and PBKDF2.  This article and its 2007 predecessor are excellent primers on password hashing, salts, and rainbow tables.

Back to md5crypt – 2.6 million hashes/sec on commodity hardware is very fast. Poul-Henning Kamp, the original author of md5crypt(), wrote in 2012 that md5crypt is now “as prone to brute-force attacks as the DES based UNIX crypt was back in 1995: Any 8 character password can be found in a couple of days.” That’s damning, but how about some numbers?

I’m using oclHashcat, specifically cudahashcat64.exe, and their mask syntax is useful to describe keyspaces:

?l = a-z

?u = A-Z

?d = 0-9

?s«space»!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~

?a = ?l?u?d?s

Here are a few password search spaces along with the time it’d take my GTX960 to run through the entire space using md5crypt:

?a x 7 : 308915776 (308 million) – 3 x 10^8 : 2 minutes

?a x 8 : 69833729609375 (69 billion) – 6.9 x 10^13 : 310 days

?l?u?d x 8 : 218340105584896 (218 trillion) – 2.1 x 10^14 : 972 days

?a x 8 : 6634204312890625 (6 quadrillion) – 6.6 x 10^15 : 80.86 years

?l x 10 : 141167095653376 (141 trillion) – 1.4 x 10^14 : 628 days

?l?u x 10 : 2779905883635712 (2 quadrillion) – 2.7 x 10^15 : 33.88 years

?l?u?d x 10 : 13537086546263552 (13 quadrillion) – 1.3 x 10^16 : 165 years

?a x 10 : 47720649502202481 (47 quadrillion) – 4.7 x 10^16 : 581 years

?a x 12 : 190107953018704786 (190 quadrillion) – 1.9 x 10^17 : 2317 years

I assume there’s some mismatch in the symbol sets, because the search space numbers from Hashcat don’t quite match up with the GRC Haystack. I’ve duplicated a bit of effort from the Haystack, but its three estimated cracking speeds don’t quite cover md5crypt on commodity hardware, and it’s fun to get numbers that are relevant to the subject at hand.

You can see that the times escalate extremely quickly, and that as you get past 10 characters it becomes unfeasible. Remember, though, that these times are for my single $300 GPU – a well-resourced entity (be it corporate, state, or criminal) that could afford hundreds or thousands of GTX 960s changes the playing field greatly. Using napkin math, a farm of 3000 GTX 960s retailing for $900k could now break a 10-length ?l?u password in 4 days.

Also remember that the times listed above are to exhaust the entire search space. Hashcat outputs a somewhat sequential ordering of the search space, so if my password is ‘Andrew15’ it’ll be found much sooner than ‘Rachel99’ (assuming a pure bruteforce with no strategic masking). Humans aren’t random and neither are their passwords, but if they were, the average discovery would occur 50% of the way through the search space.

What does this look like in practice? First, we get a password hash straight out of Junos:

tylerkerr@srx210# set system login user test authentication plain-text-password
New password:
Retype new password:

[edit]
tylerkerr@srx210# show | compare
[edit system login]
+ user test {
+ authentication {
+ encrypted-password "$1$6Ub0uM5t$08QKpPT1ZO0GjwcVe6mTP1"; ## SECRET-DATA

OK, our hash is $1$6Ub0uM5t$08QKpPT1ZO0GjwcVe6mTP1. Let’s take it to Hashcat.

To keep the demo short, let’s say that I’m going to try the correct search space – ?l?u x 6. The Hashcat syntax is (on Windows):
cudahashcat64.exe -m500 -a3 $1$6Ub0uM5t$08QKpPT1ZO0GjwcVe6mTP1 -1 ?l?d ?1?1?1?1?1?1
-m500 is hash type 500 – see Hashcat’s impressive list of supported hashes here
-a3 is attack type 3, mask attack, which is just a tunable bruteforce
-1 defines a custom mask variable, which in this case is ?l?d, and six ?1s sets our search space at 6 chars. It’s possible to automatically increment if you’ve chosen or know the character set but not the password length.

This is what it looks like:

gtx-960-working

And on completion:

gtx-960-done

The password was “easy0n” (“easy0ne” would’ve taken longer than I wanted). 4:05 of real time, from an estimated 12 minutes.

Out of curiosity, I asked a colleague to try the same operation on his Titan X, a $1000 single-GPU card:

titanx-done

Done in 1:37, around 2.5x faster.

 

So what does this mean regarding Junos’ $1 hashes? Even though it’s an old algorithm, good passwords are hard enough to crack that coworkers with access to your config files probably aren’t going to be able to crack them. Even a determined attacker that somehow obtains your hash would still need to devote significant resources to cracking it (again, as long as you used a reasonable password).

All of that said, it’s still concerning that these are even realistically breakable at all. A state or multinational entity would be able to break even very good passwords hashes with md5crypt – it’s an old algorithm. Cisco has already started moving to both scrypt and PBKDF2. $1 hashes should probably be treated as secret data, which, to their credit, Juniper appends as a comment to all $1 hashes and $9 passwords. Now, Next time: what happens when you use set system login password format sha1?

Junos configuration archiving

Junos supports automatic archiving of the current configuration via a few different methods: file (local), ftp, and scp. You’re able to force passive ftp via the pasvftp:// URI scheme if necessary. Both forms of ftp use cleartext creds though, so those should be immediately forgotten about. scp is great, however, so here’s some info:

You have the option of transferring at specific intervals (between 15 minutes and two days) or upon commit. These are mutually exclusive.

Here are the top-level set commands:

set system archival configuration transfer-on-commit
set system archival configuration transfer-interval [15 - 2880]
set system archival configuration archive-sites "scp://configbackups@hostname:/home/configbackups/sitename/" password "sshpassword"

A couple things I’ve learned:

  • The syntax is pretty strict. It looks like some special characters are out, including ~, thus the /home/ in the example above. Any @s other than the one in user@host are also out.
  • Transfers are not instant – sometimes it’s near-instant, and sometimes it’s more like 30 seconds. I assume the /var/transfer/config/ dir is polled on an interval.
  • Transferred configs are gzipped, so if you want to glance at them, use zless/zgrep/zcat.
  • Using a keypair instead of passwords is evidently possible, but I haven’t tried it.

Note the archive-sites format – you’re also able to use “scp://configbackups:sshpassword@hostname” etc., but this means the password is stored plaintext in your config. You can instead use the above syntax – with separate “password” term – to store an encrypted version.

Caveat: This is stored in a simple reversible format. Juniper’s $9 “hashes” are actually a proprietary reversible encryption that’s been reverse-engineered for quite a while. Same idea as Cisco’s type 7 passwords. Both are instantly reversible on a number of sites like this, although it’s an unbelievably bad idea to send what is presumably a sensitive password to a public website. m00nie’s site uses TLS at least (and he seems like a straightforward guy), but the logic is still in serverside perl rather than js so you can’t be 100% sure where it goes.

Point of the caveat: make sure that any configs that contain sensitive info only end up on secure systems. $9 passwords are only very slightly better than plaintext.

Here’s Juniper’s kbase article.

Next time: how secure are Junos’ $1 hashes?