Snowden and Compilers

Matej Kovacic matej.kovacic at
Thu Feb 13 00:19:13 PST 2014


> It was demonstrated well before then, Arrigo Triulzi had demonstrated running 
> an SSH server inside a NIC several years earlier.

Below is a text of interview with Triulzi, from 2009. Website where this
was published has some technical problems (only slovenian version is
published, but english will be recovered in a day or two).

Menawhile, web.archive version is available:

Slovenian version ( has a
screenschoot of NIC SSH in action.

“All your firmware are belong to us”

by Matej Kovačič :: 26. september 2009

Interview with independent security and networking consultant Arrigo Triulzi

Arrigo Triulzi is an independent security and networking consultant,
working out of Geneva in Switzerland. He is a mathematician by training
but has been a consultant for over 20 years. His hobby is firmware
research. In November 2008 he had a presentation about project Maux –
his research about firmware rootkits.
Since firmware rootkits and exploits are interesting area of research,
we decided to make a short interview with him to present this topic to
our readers.

Matej Kovačič: First question – could you describe rootkits for our
readers in a few words?

Arrigo Triulzi: A rootkit is normally defined as a collection of tools
which allow the remote control of a system. They generally consist of at
least a sniffer, a password cracker and a backdoor.

Matej Kovačič: What is the difference between software and hardware

Arrigo Triulzi: A software rootkit is tailored to an operating system,
often a specific version of an operating system, a hardware rootkit is
tailored to the hardware of a specific system and is therefore operating
system independent. A reinstall of software on a system infected with a
hardware rootkit has no effect on the rootkit’s operation.

Matej Kovačič: Last year you had the presentation about malicious code,
stored on a network card. Could you describe what is the general idea of
hardware rootkits and what exactly have you done?

Arrigo Triulzi: The idea behind hardware rootkits (see John Heasman's
ACPI work, Jason Larson’s work on in-memory rootkits, earlier work by
the Austrian TESO group and Rutkowska's recent work on AMT) is simply to
have rootkits which are operating system independent and, in some cases,
hardware independent.
Hardware independence is given by the fact that a rootkit running in a
network card works independently of where that network card is used: if
it is in an IBM Power workstation, a Dell PC, a high-end switch or a
MIPS-based SOHO router. As long as the NIC chip is the same then it will
What I have done is to modify the NIC firmware in such a way that it can
react to external commands and, in association with other equipment on
the motherboard (the video card, specifically), provide a remote shell.
This remote shell is capable of viewing the system memory from which you
can extract whatever is of interest (passwords stored in clear, keyboard
buffers, graphics buffers, etc.).
An extension of this work is to provide a direct communications channel
between two NICs thereby providing a bypass in a firewall which is
totally invisible to the firewall's operating system.

Matej Kovačič: How were you able to load your firmware into the card?
You need some special hardware device or you can do it simply from the
computer? Is there any protection, like digital signatures of new firmware?

Arrigo Triulzi: Well, as all research projects it evolved from using the
flash update program which came with the NIC and simply feeding it my
modified firmware to using more sophisticated techniques like
discovering a remote update capability. The current version simply sends
the appropriate packets to vulnerable cards and updates them.
Different manufacturers use different layers of protection and it is a
general observation that the cheaper the NIC the worse the protection
is. Quite a few high-end cards make use of digital signatures to verify
that the new firmware is indeed from the manufacturer.

Matej Kovačič: How hard was to develop such a code, what kind of a
knowledge and equipment do you need?

Arrigo Triulzi: It all started out of curiosity and took about two years
of working when I had time to spare. The knowledge I built as I went
along and, for the hardware, by asking my father who used to design

Matej Kovačič: Approximate, about how many man-hours? By your opinion,
how much would need some well funded criminal organization?

Arrigo Triulzi: Well, I would say that it took me approximately 500
man-hours to get to the stage of having a working “shell” (it cannot
really be called a real shell as the commands are very limited). I doubt
that a well-funded organization would need much more time than I needed
but it should be noted that the ROI does not make the endeavour
worthwhile: while the current mechanisms work (phishing, viruses, etc.)
there is no point in spending time and money to go down the hardware route.

Matej Kovačič: There are several producers of network cards. How many
chipsets are in use? Rootkits for every branch should probably differ.
How much? Would it be possible to develop generic ethernet card rootkit?

Arrigo Triulzi: As I describe in my paper the marketing department is
your worst enemy: there are literally thousands of variants, many of
which are only ever documented in the OEM datasheets, and often little
changes (deltas) in the NIC production line cause my modified firmware
to fail.
A perfect example: I bought two 10-pack of allegedly identical NICs
(identical model numbers) and the production batches differed enough
that I needed to have two separate versions. The changes were minor but
it was only when I looked carefully at the main chip on the NIC that I
realized that they were different revisions of the same chip.
Now, can a generic NIC rootkit be developed? I don’t know, it might be
possible to write a NIC rootkit which works across families in the same
way that a kernel can run (at times suboptimally) on CPUs from Pentium
through Core i7 but I have not really looked at it.

Matej Kovačič: If you can run a custom code on a network card firmware,
you have direct memory access to RAM...

Arrigo Triulzi: Yes, you do, via DMA.

Matej Kovačič: Which means you can easily read and write memory without
CPU knowing anything about it... What are the other security impacts of
this and of your research?

Arrigo Triulzi: Well, the first problem is that it is invisible except
for the traffic on the network side - if you want to take data out then
you must pass it over the network and this means that it could be
detected. On the other hand the operating system has no hope unless it
has some way of comparing the current firmware with the original one.
The security impacts are a simple extrapolation of the above: you can
read (and write) data into any system in a way which is currently
totally undetectable.

Matej Kovačič: How to detect such attack? It seems software techniques
are long time not enough to keep us secure...

Arrigo Triulzi: The only way would be trusted computing if implemented
properly and without the DRM halo which it normally carries: it would
mean that the firmware on the NIC is trusted by the system and therefore
any modification would be detected and prevented.
There is probably no easy way to detect it in software - a simply
firmware comparison could be tricked by keeping a copy of the parts
which were modified on the firmware and returning those to anyone trying
to download the firmware for verification.

Matej Kovačič: How exactly would be possible to detect if firmare is
correct? TPM chip on every device?

Arrigo Triulzi: Well, if we were to implement TPM in such a way that
every component of a system is verified for both security and integrity
(it would be nice to know if your NIC is about to fail, independently of
the security of the device). This makes the boot process complicated
because you start having issues with chicken & eggs: what part of the
hardware needs to be powered up and in what order? How do you execute
correctly and, more importantly, securely, a WOL (wake-on-LAN) packet?

If you start thinking about it you realize that it is not a trivial
issue at all: if you need to process a WOL packet it means that you have
to have the NIC wake up first, but then how does the TPM verify that the
WOL packet has not tampered the NIC? Do you do it later in the process?
But then how do you know that the NIC has not tampered with other parts
of the system or, worse, is replying pretending to be other devices on
the PCI bus to the TPM queries? The paranoid can head for the hills now.

Matej Kovačič: What if someone installs malicious code in the factory?

Arrigo Triulzi: This is obviously a nightmare – you get a pre-installed
botnet with your Christmas PC purchase. It is one of the examples I
always use: someone installing modified NIC firmware at Dell just before
the Christmas rush, Dell ships loads of PCs and mid-January we have a
gigantic botnet distributed around the planet by DHL. No chance of
seeing the infection vector because it never touches the Internet until
the botnet is unleashed. See it as an out-of-band infection mechanism.

Matej Kovačič: How much memory does have network card? How big can be
malicious firmware?

Arrigo Triulzi: Very very little. This is why I had to branch out to the
GPU (video card) to find memory to put my shell in. I only really use
the smallest amount of RAM and firmware space on the NIC device,
everything else is done via DMA on the video card. In my first paper on
NIC firmware modification (presented at a closed conference) I gave the
figure of approximately 5s of sniffer data being held on the device
before the RAM filled up.

Matej Kovačič: Have you heard of Phenoelit research of security of IOS
operating system? Last year they demonstrated an interesting attack on
Cisco routers – they sent one special packet through router and were
able - for instance - to change firewall rules.
It seems it would be possible to bypass the network filters by smuggling
network card into the company, penetrate network routers from inside,
get internet access to compromised ethernet card and... the sky is the

Arrigo Triulzi: Indeed I have read Phenoelit’s research and rather than
smuggling a NIC into the company a better trick would be to take over
the firewall directly: if the firewall has NICs which are vulnerable you
can use something I call the “Jedi Packet Trick” to take over the
external NIC, then via the PCI bus infect the internal NIC and pass
packets directly between the two NICs. The name obviously comes from the
Jedi Mind Trick used to bypass the Imperial Stormtroopers…

Matej Kovačič: Last month we saw a malicious code could be also runned
on a Mac keyboard. Joanna Rutkowska and her team had shown how
vulnerable are hypervisors. Which hardware could also be exploited?

Arrigo Triulzi: Well, anything with a CPU: from the service processors
(IPMI, AMT, HP iLO) to SATA disks, to SCSI controllers, to video cards
(these I've already done some work on), network cards, etc.

Matej Kovačič: Yes, in your presentation you mentioned, you are able to
connect to video cards through network card (via PCI-to-PCI transport)
and run a malicious code there. This is much more powerful, because
video cards have much more memory and GPU has much more computing
power... Could you explain this in more detail?

Arrigo Triulzi: Quite simply the NIC is not powerful enough to do much
more than take packets off the wire and route them to a different
location over the PCI bus. You therefore need somewhere to run more
complex code and the obvious answer is (avoiding software, of course)
the GPU: the current crop from both ATI and nVidia are extremely
powerful and nVidia comes with a good open-source development kit.
So the packets come into the NIC, they are checked to see if they are to
be forwarded to the video card or are legitimate packets, they get to
the video card which processes the command and sends the reply back
through the same route.

Matej Kovačič: We haven't seen much security research about hardware
based attacks (comparative to software based attacks). Is there any
security testing of a new hadrware?

Arrigo Triulzi: Not really, no. At the moment hardware security rests
with manufacturers. I suspect that governments will test hardware
security for their more sensitive applications.

Matej Kovačič: Maybe some advice what to do if you are really paranoid
about security?

Arrigo Triulzi: As things stand I would recommend using pen and paper...
More seriously now: hardware rootkits are very sexy but hardly what is
needed these days. What is the point of a sophisticated hardware rootkit
when you can gain control of a user's machine by simply sending a
malicious image as part of a website or a malicious attachment with an
e-mail? At the moment the return on investment for hardware rootkits do
not make them viable so they are simply not going to be used except
where they are really needed which, in my opinion, is at the level of
serious industrial espionage. So the answer is that you had better
concern yourself with securing what you already use and make sure that
your online behaviour is not reckless, mail programs should come with
three to five layers of big red dialog boxes before they let you open an
attachment of any kind...

Matej Kovačič: I agree. But if you are a big company or government
organization, this things can become important. And then it is only a
question of trust. I mean, at the end of the day, you have to trust
someone – your software, your OS, your hardware. Who do you trust?

Arrigo Triulzi: As I mentioned earlier the more expensive cards do come
with signed firmware. If you decide to purchase the very cheapest
equipment then chances are that one of the areas the company didn’t
invest in is security. If you are a government organization you often
already have a source license for Windows, for example, and it is just
routine for approved vendors to provide hardware specifications
including the firmware update mechanism. You simply need to check that
it uses signed firmware and then verify the claim.
In many ways it is the usual story: “you get what you pay for”.

Matej Kovačič: Thank you very much.




More information about the cypherpunks mailing list