I Saw You Coming

Much of our world revolves around data—a lot of data. I’m talking about petabytes, yottabytes and the like. Put simply, you could fit all the academic research libraries in the US into two petabytes.

What kind of data are we talking about? Everything, including consumer products, news, crime, and weather.

That begs two questions. Where does the data come from, and who pays for it?

The data origin—not its provision—varies: records stored by humans provide a good deal of it. You are for instance able to tap into two hundred million records of crime data for the US. The data you access costs you money, and is the result of millions of security-related filings, including arrests, sentences, and paroles.

A second major source of data are sensors. These can be weather station sensors for wind speed or air temperature, buoys at sea measuring wave height, or satellites sitting high above you as you read these words. You are yourself part of the sensor network—as you read, your cellphone informs the cloud about your latitude and longitude. From that sensed data, we know whether you are sitting (and we know exactly where), and what time it is—if you sit there long enough, we will know where you live, or where you work—in a matter of days we’ll know both.

It’s a simple matter to find out who you live with, based on your coincidence in time and space, and build a relationship tree. If you’re moving slowly as you read these words, we’ll know you’re walking or strolling. If you’re moving fast, we know you’re in a vehicle—discovering whether it’s a car, bus, or train is a trivial matter. We can cross your trajectory with a highway map—if your vehicle makes frequent stops, you’re on a bus—or maybe you’re a UPS driver (but do me a favor, don’t read while you drive). Sensors provide huge amounts of data because they’re measuring stuff all the time.

The final source of data are models—these models don’t sashay on the catwalk, they run on computers, often using those very same sensor outputs to make forecasts—here’s that weather thing again.

The second question is even easier to answer. Who pays for it?

You do. You really should have seen that coming.

As a taxpayer, you fund the justice system, the weather office, the health system, the education system… delete as applicable, depending on where you live—I dearly hope no deletion is required.

When I was in the US, I picked up the latest book by Michael Lewis, called The Fifth Risk. I picked it up in the usual fashion, by seeing it an an airport store and promptly buying it on Kindle.

Now, Michael Lewis has been a favorite for years, so (unusually) I’ll plug an(other) author in these pages. Lewis has had a go at US investment bankers, the sub-prime mortgage scandal, the whole austerity deal in Europe, and HFT—High Frequency Trading is another scandalously well-kept secret—and yes, Big Data is at the heart of it. Oh, and he’s had a few goes at the orang-u-bang.

In summary, it’s a good job Lewis is not a Saudi national, otherwise he’d be part of an erector set by now. As an aside, the only simple question I want answered: if Khashoggi died in a fist fight, as claimed today, where’s the body? I suppose I’m also curious about why he got into a fist fight with fifteen guys.

The Fifth Risk took me two days to read, and I was fascinated by a chapter called ‘All the President’s Data.’ I want you to read the book, particularly in the lead-up to the mid-terms, so I won’t be a spoiler.

I will, however, tell you that US federal agencies such as the department of agriculture, NOAA, USGS, and NASA, have to provide data to the public as part of their mission statement. They are obviously not in the business of making the most sophisticated viewing interfaces for the consumer market—these are often done by third parties, but the key point is that those parties would be unable to source data were it not for the fact that you have paid for it to be made public, and more importantly, accessible in a simple way.

As an example, if you’re a US taxpayer, you support Wind Guru. The business model for this wind and wave forecasting website is fascinating—not least because the site is based in the Czech Republic, a land-locked nation. Every surfer knows Wind Guru—what most don’t know is that it isn’t a guru at all, the gurus are the US National Weather Service (NWS), the US Geological Survey, and others. Wind Guru accesses models run by NWS (a part of NOAA) using a special toolset known as web services.

Many government agencies worldwide provide such services, and this has allowed the private sector to develop some really nice tools for public use. The problem is when the private sector lobbies the government to try and stop the agencies that run the models being able to do anything but supply data.

One of the current discussions revolves around AccuWeather, which charges for its services, and its alleged efforts to limit how NOAA presents its own data, acquired through sensor networks paid for by (you saw it coming) the US taxpayer.

At the forefront of all this excitement is an American lawyer called Barry Myers, who is at present the CEO of AccuWeather. The exciting bit is that in October 2017, Myers was picked by Trump to lead NOAA.

The confirmation hearing is holding this one up. If Myers is confirmed, it means that an operator in the private sector of the multi-million weather forecast business will be in charge of a government agency that collects twenty terabytes of data every day, much of it weather-related.

The line between public and private becomes thinner and grayer than an old man’s hair.

If the orangutan buffoonery get away with this one, the fox will be firmly placed in charge of the henhouse.

The India Road, Atmos Fear, Clear Eyes, and Folk Tales For Future Dreamers. QR links for smartphones and tablets.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.


%d bloggers like this: