Micah Lee's guide to analysing leaked, hacked and otherwise acquired data

📚 Finished reading Hacks, Leaks, and Revelations by Micah Lee.

This is a very hands-on book, here to teach you the art and science of analysing big datasets, structured or not, primarily from the point of view of an investigative reporter who’s digging into potentially rather dubiously acquired big ol' datasets.

That said, data is data, so the techniques herein may be of use to anyone who is trying to dredge interesting or important insights out of a big mass of non-descript data. It’s just that all the examples come from the titular hacks and leaks and are all the more fascinating for having done so. Many relate to stories the author published at The Intercept.

There’s some useful meta-info from a citizen-or-professional journalistic POV about how to protect yourself and your sources, and ideas around validating your findings. Anyone in this field with an eye on their own privacy should take heed. We’re talking about using tools like disk encryption, Signal, Tor, SecureDrop and so on, including the author’s own “Dangerzone”.

And then it’s onto the data itself: how to download massive datasets (including via BitTorrent), how to explore through their inner depths, how to turn a gigabyte of ascii ramblings into searchable evidence, and so on.

Firstly the author teaches us the use of some handy command-line tools present on or gettable for your average home computer system (it’s probably slightly easier if you’re a Linux or Mac user, although there are instructions for Windows too, if you’re happy to use WSL).

That’s followed up by some information about using some GUI software packages useful for specific types of data; e.g. the Thunderbird email client for interpreting email dumps, and a tool called Aleph for general searching and drawing connections that I’ll definitely be giving a go (even if it looks like the Open Source edition isn’t any more supported by the organisation concerned :( ) .

Then we learn the more in-depth arts of Python programming and SQL data querying, with the most advanced projects being around how to build your own mini-app to help efficiently explore your dataset of choice. The author has created or contributed to several such tools, including BlueLeaks Explorer or DiscordLeaks.

The book is full of practical exercises, learning by doing, using real-life freely available datasets of the type the author has used in his own published investigations. Most of them come from the Distributed Denial of Secrets website, somewhere you should absolutely explore if you’re interested in this sort of practice.

That includes a bunch of leaked police data known as Blueleaks, the archives of some chat groups that weird right-wing lunatics hang out in, exports from the Parler social network around the date of the January 6th US insurrection, dumps of some medical company facilitating Covid anti-vaxxers and so on.

There’s the odd diversion into stories about the actual stories the author reported on, and the real-world impact he produced using these sorts of techniques. Very inspiring.

From the choice of datasets, you can perhaps tell the political salience of this book. I’m very much on its side, but even if, somehow, you’re not, obviously the tools and techniques work just as well on any equivalent dataset.

I read the paper version of the book. For people that don’t like re-typing, the code the author shares in the book is freely available in this git repo. And if you prefer screen-reading, well, the entire book’s text is freely available to anyone who wants it under the Creative Commons licence. Superb. Although if you like it and can afford to, it would be nice to provide some kind of support, whether in $ by buying a DRM-free copy of the eBook, or via citation whenever you too publish your next world-changing data-driven investigation.

I would say that the scope of the book is so large that the instructions are necessarily rather succinct. If you really want to get into programming your own interactive data exploring tools with Python, and are not already a Python programmer, you will need more than this book provides to get started. Links are provided. But this remains a great source for opening one’s mind as to what is reasonably possible given a bit of nerd-effort.