logo rabbitude
rabbit  hinted  that  our  emails  were  fake:  here's  why  it's  not 
evil-rabbitude

yesterday, rabbitude announced that rabbit had left hardcoded api keys to their email provider, sendgrid, in code that we had access to.

this was proved with sample emails sent from rabbit’s sendgrid account to a number of individuals. these include:

rabbit’s response

although rabbit have issued a short statement on our previous disclosure, they are yet to formally acknowledge the sendgrid compromise.

instead, their ceo, jesse lyu, suggested in their community discord that the emails could have been spoofed / faked:

discord user:

[…] why was rabbitude successfully able to send me an DKIM verified email from [email protected] and pass DMARC

jesse:

i did asked [sic] about this. turns out to be many ways. use common sense to tell if it’s legit email [sic] or not.

discord user:

jesse were they able to gain access to your domain to create the subdomain r1. ? or was it a spoof

jesse:

there are many ways, spoof included. either way this is not ok.

it is categorically false that these emails were spoofed. let me explain why.

background

i suspect the notion these emails could have been spoofed stems from a misunderstanding of how modern email verification differs from traditional email.

it is certainly true that, not that long ago, it was laughably easy for anyone to spoof email “from” addresses…

under the hood, an email is not much more than a text file. here’s a simple example:

From: [email protected]
To: [email protected]
Subject: This is an example email

hi recipient,

this is the main body of the email

yours,
sender

notice that the metadata - the From and To addresses as well as the Subject - are simply lines at the top of the email. these are called headers, and if you change them, you could make it look like the email came from pretty much anyone.

modern email

clearly, this should not be possible in the modern world. nowadays, a whole host of email verification systems and protocols exist.

ryan fae, one of the journalists sent an email, has kindly shared the full raw version of the email they recieved here. i’m going to comment on a cut down version here with the relavent bits:

Authentication-Results: aspmx1.migadu.com;
	dkim=pass header.d=r1.rabbit.tech header.s=s1 header.b=JO6tvaFz;
	spf=pass (aspmx1.migadu.com: domain of "[email protected]" designates 159.183.84.99 as permitted sender) smtp.mailfrom="[email protected]";
	dmarc=pass (policy=quarantine) header.from=r1.rabbit.tech
Received: from wfbtrqsp.outbound-mail.sendgrid.net (wfbtrqsp.outbound-mail.sendgrid.net [159.183.84.99])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
	(No client certificate requested)
	by aspmx1.migadu.com (Postfix) with ESMTPS id 1E8C2C7AD
	for <[email protected]>; Wed, 26 Jun 2024 17:25:38 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=r1.rabbit.tech;
	h=content-transfer-encoding:content-type:from:mime-version:subject:to:
	cc:content-type:from:subject:to;
	s=s1; bh=gS8mted2QORXmZD6JV1EQqM7imUQ+hJDWoIEHnot+PQ=;
	b=JO6tvaFzlgeiYtS/QTKgbxlQXz44SX4jCFYmqyAXOJLkANUpHSteIRBybW14/qXj2QHJ
	+9r/qwr9tjqtBVj0yoF454whAIHmUweZmrsDY/lBjpyLT8empBcqz6cFjMeGSTOXo6ol/g
	MN1nPqacrcAxgCtb8d/2yeMZ4Un186vv5/q5Ap9NFMjcUjGqSKtaYNX770XKEMkQFc/m8Q
	SaNYWr2vjeQazrlGDixT0vhEjzcyKInOHnz/fyPuP1gv4g3CGr8tWjmt1BTlGOSvh/JG9C
	QdP2ZV14np1DgH46WYhbdwpAS5yI0yK4uJsg4xWrDv++yGDzof2jZA346JROP9MA==
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=us-ascii
From: [email protected]
Mime-Version: 1.0
Subject: Hi, we got hacked.
To: [email protected]

We got breached. And didn't admit it like cowards

this is similar to the previous example, but with more stuff. we still have the From, To, and Subject headers, and the body of the email at the bottom, but we also have some extra headers too, most interestingly Received, DKIM-Signature, and Authentication-Results.

received header

let’s break the Received header down to its different pieces:

from wfbtrqsp.outbound-mail.sendgrid.net

this tells us that the email was received from a mailserver claiming to be called wfbtrqsp.outbound-mail.sendgrid.net.

(wfbtrqsp.outbound-mail.sendgrid.net [159.183.84.99])

the contents of the brackets gives the information the receiving server discovered when looking up the sending server via reverse dns, namely:

  • that it is indeed called wfbtrqsp.outbound-mail.sendgrid.net
  • that its ip address is 159.183.84.99, which a simple whois lookup shows is owned by twilio sendgrid

by aspmx1.migadu.com

this tells us the name of the receiving server which performed the verification. in this case it was the server for ryan’s mail provider, migadu.

dkim-signature header

the Recieved header is fairly useful for confirming the email’s legitimacy, but relying on it requires us to trust that migadu verified the sending server correctly. the DKIM-Signature header, however, is much stronger proof, and can be independantly verified.

dkim - domain key identified mail - is a system used to digitally sign parts of the email itself. let’s look at some components of the header:

h=content-transfer-encoding:content-type:from:mime-version:subject:to:cc:content-type:from:subject:to;

this tells us which headers have been signed:

d=r1.rabbit.tech

this is the domain which owns the keys used to sign the email and is, by extension, claiming ownership of the email itself.

s=s1

this is the name of the key used to sign the email. this, in combination with the domain, can be used to look up the public key using something like linux’s dig:

> dig TXT s1._domainkey.r1.rabbit.tech

;; ANSWER SECTION:
s1._domainkey.r1.rabbit.tech. 300 IN	CNAME	s1.domainkey.u43543998.wl249.sendgrid.net.
s1.domainkey.u43543998.wl249.sendgrid.net. 904 IN TXT "k=rsa; t=s; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApnk/avjznUTwdYpcVVqCLJVN2+OYwz7eLPePahndd4xGCgYaAA3/mEGjSNnYKIzfaSuhuUVi9sIAaDtM57pzPNmRb2tCTFYZ9jiFIV3uudyf14B2HApq8vjcpDDnGcx1+OiM0+NkPCkhBZOrEjZUSZ9WwZv675rYkq6hBcSJ4M6ssTPB6p7dt5BQYI+mky2tEcPf1" "PqlzsiQw1EZCCWipYhuNFEyoYZpukCLEmp9QO1to1uRMNbHjWbXRxsLWzkwg1ESLi0KvJzcYUwcTVs+bjMtjrEMOOAbDAJwP99K11+gJuYL4IZTA42BFq3kVVrg15oEARyUR7bYnvP5x1Lh2wIDAQAB"

we can see r1.rabbit.tech has delegated control of its s1 key to… you guessed it… sendgrid. we also get the public component of the signing key, so we can verify the signature later.

bh=gS8mted2QORXmZD6JV1EQqM7imUQ+hJDWoIEHnot+PQ=;

this is a hash of the body content of the email itself.

b=JO6…

and finally, we get to the digital signature of the body and headers listed above. if i were to change the body, or change a header, the signature would cease to be valid.

ryan’s mail provider checks this siguature - and we can see that it passes via the Authentication-Results header:

Authentication-Results: aspmx1.migadu.com;
	dkim=pass header.d=r1.rabbit.tech header.s=s1 header.b=JO6tvaFz;

but you don’t have to (and shouldn’t!) take migadu’s word for it! dkim can be verified independently: download the raw email yourself and check using one of the many tools that can verify DKIM signatures.

i use dkimpy:

> cat ~/Downloads/Hi, we got hacked.-9efb2684ff65631ac6b8f5c17e479ea5.eml | dkimverify
signature ok

and, just to check, i went in and changed the From address and tried again:

> cat ~/Downloads/Hi, we got hacked.-9efb2684ff65631ac6b8f5c17e479ea5.eml | dkimverify
signature verification failed

i’d encourage you to verify this yourself!

the presence of this digital signature confirms the content of the email body and signed headers were:

a) sent by a system with access to the private component of r1.rabbit.tech’s s1 key. b) unchanged since sending

conclusion

so, while emails back in the day were spoofable, we aren’t back in the day anymore - modern email systems, including the ones rabbit uses through sendgrid, are much, much harder effectively impossible to spoof when configured correctly.

you can see that the emails were sent from sendgrid’s servers.

we have shown that they were digitally signed by a key, looked up on the r1.rabbit.tech domain, and delegated to sendgrid’s control. there’s no “faking” this sort of digital signature.

sadly, this technical proof shouldn’t actually be needed to convince rabbit. they already know the emails are legit - all they have to do is check their sendgrid logs, the emails will be there. those logs aren’t available to everyone else, but the technical proof in this article proves that anyway.

so, rabbit, it’s time to be honest. the jig is well and truly up. there is no hiding behind denials this time. this happened, and you know it - the emails came from your sendgrid account, which was available via the exposed key.

please do the right thing and inform your customers.