Big Data in practice

(This is a translation of an earlier post in my Norwegian blog. This translation was done by Ragnvald Sannes using Google Translate with a few amendments. This technology malarky is getting better and better, isn’t it?).
ml_mapI have just finished teaching four days of data analytics – proper programming and data collection. We (Chandler, Alessandra and the undersigned) have managed to trick over 30 executives and middle managers in Norway to attend a programming and statistics course (more or less, this is actually what analytics basically is), while sort of wondering how we did that. The students are motivated and hard-working and have many and smart questions – in a course taught in English. It is almost enough to make me stop complaining about the state of the world and education and other things.
Anyway – what are these students going to do with this course? We are working on real projects, in the sense that we require people to come up with a problem they will find out in their own job – preferably something that is actually important and where deep data analysis can make a difference. This has worked for almost all the groups: They work on real issues in real organizations – and that is incredibly fun for the teacher. Here is a list of the projects, so judge by yourself. (I do not identify any students here, but believe me – these people face these issues every day.) Well worth spending time on:
  • What is the correct price for newly built homes? A group is working to figure out how to price homes that are not built yet, for a large residential building company.
  • What is the tax effect of the sharing economy? This group (where one student works for the Tax Administration) tries to figure out how to identify people who cheat on the tax as Uber drivers – while making suggestions on how tax rules can be adapted to make it easy to follow the law.
  • What characterizes successful consulting proposals? A major consulting firm wants to use data from their CRM system (which documents the bidding process) to understand what kind of projects they will win or lose.
  • How to recognize money laundering transactions? A bank wants to find out if any of their customers are doing money laundering through online gaming companies.
  • How to offer benefits to customers with automated analysis? A company that supplies stock trading terminals wants to use data analysis to create a competitive edge.
  • How to segment Norwegian shareholders? A company that offers online trading of shares wants to identify segments of its customers to pinpoint and improve its marketing strategy.
  • How to lower costs and reduce the risk of production stoppages in a process business? A hydropower company wants to better understand when and why your power stations need repairs or maintenance.
  • How to identify customers who are in the process of terminating? A TV company wants to understand what characterizes “churn” – how can they identify customers who are about to leave them?
  • Why are some wines more popular than others? A group will work with search data from a wine site to find out what makes some wines more sought after than others.
  • Which customers will buy a new product? A group is working on data from a large bank that wants to offer its existing customers more services.
  • How to increase the recycling rate for waste in Oslo? REN – Oslo’s municipal trash service – wants to find out if you can organize routes and routines differently to better utilize trash trucks and recycling plants.
  • How to avoid being sold out for promotional items? One of Norway’s largest grocery chains wishes to improve their ordering routines so that customers do not get to the store and find out that there is no more left of the offer they wanted.
  • How to model fraud risk in maritime insurance? An insurance company wants to build a model to understand how to find customers attempting to fraud companies or authorities.
  • Which customers are about to leave us? A large transport company wants to find out which customers are about to go to a competitor so that they can take action before it happens.
  • What characterize students who drop out? BI enters 3500 new students each year, but some of them end after the first year. How can we find evidence that a student is about to drop out?
Common to all the projects – and so it’s with all the student projects I have advised since I started in this industry – is that you start with a big question and reduce it to something that can actually be answered. Then you look for data and find that you need to reduce it even more. Then you get problems that the data is either not found, unreliable or inadequate – and one has to figure out what to do with it. Finally, after about 90% of the time and money budget is gone, one can begin to think about analysis. And then there is a risk that you find nothing…
And that is an important lesson of this course: The goal is that the student should be able to know about actual data analysis to ask the right questions and have a realistic expectation of what kind of answer you actually can get.
There is a great demand for this course – so we have set up an additional course this fall. See you there!

The reassembler

James May – Captain Slow, the butt of many Top Gear jokes about nerds and pedants – has a fantastic little show called The Reassembler, where he takes some product that has been taken apart into little pieces, and puts it together again. It works surprisingly well, especially when he goes off on tangents about corporate history, kids waiting for their birthdays to come, and whether something is a bolt or a screw.

Slow television, nerd style.

Here is one example, you can find others on Youtube:

Smarthelp: Locating and messaging passengers

 

If you are a public transportation company: How do you tell your prospective passengers that their travel plans may have to change?

Public transportation companies know a lot about their passengers’ travel patterns, but not as much as you would think – and, surprisingly, they know less now when ticket sales have been automated than they used to know before.

screen696x696

RuterBillett – a ticketing app

Let’s take a concrete company as an example: Ruter AS, the public transportation authority of Greater Oslo. Ruter is a publicly owned company that coordinates various suppliers of transportation services (bus, tram, train, some ferries) in the Oslo area. The company has been quite innovative in their use of apps, selling most of their tickets on the RuterBillett app, and having many of their customers plan their journey on the RuterReise app. The apps are very popular because they make it very easy both to figure out which bus or train to take, and to buy a ticket.

The company has a problem, though: While they know that someone bought a ticket on the ticketing app, they don’t know which particular bus, tram or other service the passenger took (a ticket typically gives you one hour of open travel on their services, no matter how many of them you use).

screen696x6961

RuterReise – a journey planning app

They could get some information from what people have been searching for, but the two apps are not linked, and they don’t know whether a passenger who searched for a particular route actually bought a ticket and did the journey – or not. There are many reasons for this lack of knowledge, but privacy issues – Norway has very strict laws on privacy – are important. Ruter does not want to track where its customers are travelling, at least not if it in any way involves identifying who a passenger actually is.

Not knowing where passengers are is a problem in many situations: It creates difficulties for dimensioning capacity, and it makes it difficult to communicate with passengers when something happens – such as a bus delay or cancellation.

Identifying travel patterns and communicating with passengers

The problem for Ruter is that they want to know where people are travelling (so they can figure out how many buses or trams they need to schedule), they ned to know who regularly takes certain journeys (so they know whom to send a message to if that route is not working) and they need to know who is in a certain area at a certain time (so they don’t send you a message about your bus being delayed if you are out of town, for instance). All of this is easy, except for one thing: Norway has very strict privacy laws – already quite similar to EU’s General Data Protection Regulation, which goes into effect in 2018 – and Ruter cares deeply about not being seen as a company that monitors where people travel.

In short, they need to know where you travel, but do not want to know who you are.

This is a seemingly impossible challenge, but Smarthelp Secure Infrastructure, in combination with Smart Decision Support, makes it possible. The communications platform creates an end-to-end encrypted communication channel between a central system and the smartphone. Using technology developed because we had to solve the problem of medical-level encrypted communication between emergency centers and individual users, Smarthelp has technology that allows someone to track specific information you allow access to – say, the fact that you are in a certain area, or that you regularly travel certain paths – without sharing other information, such as your name.

This would allow Ruter, when something happens, to send a message to people who a) regularly takes, say, bus route 85, and who b) is in an area where it is conceivable that they could take the bus, given their prior patterns, the time of day, and so on. For the individual passenger, this would mean that you only get pertinent messages – you don’t get messages about bus routes you don’t normally take (unless you actually get on the bus), and you don’t get messages when you are far enough from the bus that it is clear you are not going to take it anyway. In a world of information overload, this is extremely important – flood the user with many messages, and they do not read them.

The future of public transportation

A selective message and geolocation service, such as Smarthelp provides, is an evolutionary step, an optimization of the current way transportation is coordinated. In the long term (especially if we start to talk about seld-driving vehicles), the whole way we coordinate public transportation will change. As one Ruter employee told me: A public transportation company is “someone who takes you from a place you are not to a place you don’t want to go.”

The next step in public transportation is that the users tells the company not just that they want to get on the bus, but also where they want to go. I have been told that in an experiment, Telenor found that, one sunny summer afternoon, fully half of their employees (located at Fornebu outside Oslo) planned to go to Huk, a public beach on Bygdøy. The distance from Telenor’s headquarters at Fornebu is 10 minutes by car, but takes more than 30 minutes by public transportation, involving two bus routes. If Ruter had known about these travel plans, though, it could have just rolled up some buses and driven the employees directly, vastly improving the service – and avoiding clogging up the regular buses to Bygdøy.

And that is the future of public transportation: Instead of planning where you will go in terms of geography, you will tell the public transportation company where you want to go, and they will get you there. With self-driving cars, they will be able to tell you when you will be at your destination – but, perhaps, not willing to tell you the actual route. As a passenger, you probably will not care – after all, what matters to you is when you arrive, not by which route.

That would, in effect, mean that we have transitioned public transportation from line switching to packet switching, effectively turning the bus into the Internet. But that is for the future.

In the meantime, there is Smarthelp.


(I am on the board of Råd AS, a company that has developed the platform SmartHelp for Norwegian emergency services, allowing shared situational awareness, communication and privacy. The company is now seeking customers and collaborators outside this market.)

Smarthelp is a platform technology consisting of, at present, three elements: Smarthelp Rescue, an app for iPhone and Android that allows users to transmit their position to an emergency service; Smarthelp Decision Support, a decision support system which allows an operator to locate and communicate with users (both with the app and without), and Smarthelp Secure Infrastructure, a granularly encrypted communications platform for secure, private communication. If you want more information, please contact me or Fredrik Øvergård, CEO of SmartHelp.

After Moore: Landauer

Very interesting blog by the very readable Ted: Is computing in reverse the next big thing?

As Moore’s law continues, it will reach certain physical limitations, such as electrons behaving less dependently the thinner the conduits become (think individual electrons instead of a more predictable stream. Another (they are linked, I suspect) is Landauer’s principle, which dictates that there is a certain lower limit on how much power that is necessary to flip a bit, and that forms a hard stop in terms of how much you can lower power consumption (and with it, heat dissipation.) (See Denning, P. J. and T. G. Lewis (2016). “Exponential laws of computing growth.” Communications of the ACM 60(1): 54-65, for an excellent discussion of Moore’s law and its remaining life.)

Turns out computing capability as a function of electric power consumption might be the next big obstacle (or at least measurement.) The BitCoin miners certainly know that.

Reverse, computing, which Ted writes about, is essentially computing where the power can be reversed, recreating the initial state. While difficult technically, it certainly would reduce power consumption to almost nothing.

To learn how, read the article. Recommended!

SmartHelp: Locating employees in a crisis

If there is a crisis – do you know where your people are?

Imagine the situation: An event (terrorist attack, industrial accident, public transportation accident) of some proportion happens. Many people are hurt, lots of rumors abound, emergency services are responding. Almost immediately, the question arises: Are any of my employees affected by this – and do they need help?

At present, most organizations locate their employees by calling them or sending emails. This is slow and ineffective – when Norway was hit by a terrorist bomb in the Oslo city centre in 2011 during the summer holiday, it took one of the large newspapers more than two days of frantic telephoning to find all their employees. Most of the employees were, of course, just fine, but the company still had to locate them all. In such a situation, knowing who is not in danger quickly is very important, because it lets you concentrate resources on those who need help.

Smarthelp Decision Support, the emergency service communication platform, allows an organization to quickly – within minutes – determine where its employees are and whether they need help. Smarthelp does this while maintaining privacy of the individual employee.

Most large organizations have a system where employees register where they travel on business. For this service to work, the employee has to remember to update it, though for some companies, this happens automatically if they purchase their tickets through a specific travel agency. While this may help, people travel for pleasure, deviate from their itineraries, forget to register their travels, and purchase their tickets from the cheapest, rather than the official source. Consequently, nobody knows where they really are.

SmartHelp Decision Support (see picture) allows the company to set up a geographical area surrounding the event, and contact all their employees (based on lists of telephone numbers) to determine whether they are inside this area or not.

terroreksempel

Here is another example: You are responsible for security in a large company facility – say, an office building. The company receives a bomb threat which necessitates evacuating the building with thousands of employees. If the employees have SmartHelp on their phones, you can communicate with them all, and determine whether they (or at least their smartphones have left the building (limited by GPS accuracy). You can define a rallying point or area and get an automatic message as soon as someone enters the area, allowing you to quickly determine who is not accounted for. (At this point, GPS location – which we use – does not allow precise location inside a building, but that could change as WiFi locationing services get better.)

rumorsparisAnother advantage is information: In the November 2015 terrorist attack in Paris happened, there where (as is usual) lots of rumors circulating in the hundreds of thousands of Twitter messages and other social channels. With SmartHelp, the authorities would have been able to send targeted messages to specific areas, conveying a precise and autorative message across a cacophony of noise and misinformation.

SmartHelp works anywhere in the world where there is mobile reception (I have used it to signal my position to my host in Shanghai, for instance.) Privacy is handled through an ingenious cryptographic architecture that is secure and fast – the platform is certified for the medical information under the Norwegian data privacy laws, among the strictest in the world.

If you want more information, please contact me or Fredrik Øvergård, CEO of SmartHelp.


(I am on the board of Råd AS, a company that has developed the platform SmartHelp for Norwegian emergency services, allowing shared situational awareness, communication and privacy. The company is now seeking customers and collaborators outside this market.)

Smarthelp is a platform technology consisting of, at present, three elements: Smarthelp Rescue, an app for iPhone and Android that allows users to transmit their position to an emergency service; Smarthelp Decision Support, a decision support system which allows an operator to locate and communicate with users (both with the app and without), and Smarthelp Secure Infrastructure, a granularly encrypted communications platform for secure, private communication. If you want to see how the system works in a 911 central situation, see this video:

Made my day!

digøkskjermI just got the message that the new bachelor program Informatikk: Digital Økonomi og Ledelse (Informatics: Digital Economics and Management) is now the most sought-after study program in Norway, with 19 applicants per available place (514 first-priority applicants for 27 available places).

Since I have taken the initiative to this program and developed it with colleagues at the University of Oslo (where I have an adjunct position, this definitely made my day. Week, actually.

Just sayin’…

What to do in Oslo (and Norway) in summer

Yesterday I got an email from an old friend and former colleague who is visiting Norway and wondering what to see and do. His is not the first email of that kind – and it dawned on me that the rational and productive thing to do (well, it is early Sunday morning and I am still undercaffeinated) is to write this up as a blog post. (Did the same thing for my Norwegian friends asking me about what to do in Boston.) So – what would I recommend if you should find yourself in Oslo during the summer, with some time to look around?

First of all, you need to understand a few basic things about Norway. Norway has lots to recommend it, but we are mainly about nature – while Oslo, Bergen, Trondheim and some of the other cities are nice, they are a bit like a one-star Michelin restaurant – worthy of a stop, but not destinations in themselves. To appreciate Norway, you need to appreciate nature and natural beauty – and be willing to put in some effort to see it. On that note:

So: If you are going to Norway, pack good shoes, rain gear, basically the light hiking setup. You may not need it, but you certainly won’t regret packing it. As the saying goes here, there is no such thing as bad weather, only bad clothing. And we mean it.

Oslo

Caveat: I like to list things that are (arguably, somewhat) unique to Oslo. We have nice cafes and tolerably good museums (such as the National Gallery), but those you will find in any capital in Europe. Most people who come here do it for a short visit – and you want to stay away from that which is mostly for the tourists and not different from anywhere else in Europe. So, here is where I take my (mostly) American friends:

  • norway-vigeland-sculpture-park-8The Vigeland park (or Frogner park). This is the world’s largest sculpture park dedicated to one artist, Gustav Vigeland. And, as every American says who has been there – they are all naked! Vigeland was a pupil of Rodin and the sheer size and rather playful humor of the place is enjoyable and a sure hit with the kids. (My favorite: The little girl who, unseen by her parents, is picking up a snake. See if you can find it.) One of Oslo’s city symbols is “Sinnataggen”, a little boy having a tantrum. This is also where people in Oslo go to relax and hang out if the weather is good, so you might see some locals.
  • 10414The Holmenkollen ski jump, especially if we can throw in a little walk in the woods. The ski jump is impressive in itself (though ski jumping is no longer the enormous draw it used to be for Norwegians) and you get an excellent view of Oslo. There is a zip line if you feel adventurous. But, to really have a Norwegian experience in the sense of doing something the locals like, walk from Frognerseteren to Tryvannsstua (pictured, about 3 km) or Skjennungstua (5 km, view, delightful pastries) to get a sense of Oslomarka. Oslo is 63% protected forest, with gravel roads, paths and (in winter) hundreds of kilometers of prepared ski tracks. Heavily used by the citizens of Oslo and surrounding municipalities.
  • The Munch museum, a bit run down, will probably move to a new building in 2018/19, but the paintings are, of course, worth seeing.
  • Exhibition_in_Viking_Ship_Museum,_Oslo_01The Viking Ship museum. The museum itself is crowded and due for an update, but it is the only place in the world, as far as I know, where you can see a complete, real Viking ship. Worth the trip. If you are feeling energetic, there are lots of other museums (at the Bygdøy peninsula, take a boat there from the City Hall). I have taken people to the Folk Museum a lot – mainly because two of my daughters have had summer jobs there and can provide an inside perspective – but it requires a deeper interest in folklore and rural history than most people have (but if you go there, make sure to sample the Hardangerlefse, cooked over open fire. Carbs galore.) There is the Fram and Kon-Tiki museum, as well, though I haven’t been for many years. Plus Huk, a nice beach, if you feel like a dip.
  • The Ekebergparken sculpture park, a relatively new sculpture park donated by property investor and man-about-town Christian Ringnes. The park itself is more of a lightly edited forest than an art installation, but it works well, and the collection (themed as an homage to women) is very good. And the view of the city is excellent – this is, quite literally, where The Scream was painted. (There is a metal frame at one point where you can take photos recreating the painting yourself. Including screaming.)
  • 1920px-full_opera_by_nightExplore the fjord – Oslo is built around the fjord, and if you can get on a boat – even just taking a summer ferry out to some of the islands – by all means do it. (I live on an island, perhaps I am a bit partial here.) In later years, the city has tried to make the shoreline available for walking (a project called Fjordbyen) and it works rather well. The high point, of course, is walking on the roof of the Oslo Opera House, but the Akershus Fort and just hanging out slowly sipping a cold (and expensive) drink while ogling the historical boats you find here and there is nice, too.
  • Møllerfossene i AkerselvaExplore Oslo by bike or foot. The urban bike project is pretty good, you download an app to your smartphone and check out and in bikes from many places. Best path: Along Akerselva. Grünerløkka (partially along the river) is the “bohemian” (well, gentrified) district, Frogner is more upscale, Grønland for immigrant food and culture, and Gamlebyen for history.
  • Eat at a restaurant. Food is somewhat expensive, drinks horribly expensive, but the quality is very high. My personal favorite is Kampen Bistro, a semi-hidden neighborhood restaurant/community house with a semi-fixed menu and the world’s best cheese platter. But you will find lots of good food and a constantly changing set of restaurants – Google for a list. (Alternatively: My friend Bill Schiano once spent most of a day walking from bakery to bakery all over town, a very happy man.)
  • If you want to say you have seen something not many Norwegians have seen: The Emanuel Vigeland mausoleum, created by Gustav Vigeland’s, well, rather eccentric brother. Opening hours are complicated, but if you have been there, you have definitely seen more of Oslo than most people.

Norway outside Oslo

Well, now it gets difficult. There are, literally, hundreds of web pages devoted to Norwegian tourist spots, from Prekestolen to Trolltunga to Hurtigruten to Flåmsbanen to whatever. They are all fantastic, but since you can read about them in lots of detail there and also plan your route via visitnorway.com and other sites, I will stay away from the obvious choices, and, again, go for something a bit more hidden and for the local (though not very hidden.)

bamsemums_smallExtremely important warning: In addition to the weather and clothing issue mentioned above, Norwegian culture is fundamentally built on the principle that you, yourself, and nobody else, is responsible for being knowledgeable and physically capable of doing whatever you set out doing. Norwegians do not believe in protecting people against their own stupidity. (And forget about suing anyone, you’ll be laughed out of court.)

Norwegians are quite cynical about this. Do not expect there to be signposts and security fences in the mountains or along the fjords – but if there are, obey them religiously, because they denote real dangers, not risks of lawsuits. Every year, some tourist will fall off Trolltunga or Vøringsfossen, step too close to a calving glacier or drown in a fjord because they did not have the wits, skills or equipment necessary. That is considered to be the tourists’ fault. At Svalbard, tourists are referred to as “Bamsemums” (a popular foam-bear-covered-in-chocolate candy) because of their propensity to end up as polar bear lunch, on account of going where they are not supposed to without a gun. Something of the same spirit prevails in most of Norway, though in some places (such as Trolltunga), the local authorities are thinking about fencing things in a bit, mostly because it is so expensive to haul the bodies out.

Yes, I am exaggerating a bit. But not much.

With that out of the way: The best way to see Norway is to amble around (preferably in an electric car), stop whenever you feel like it, and (thanks to the Viking-age freedom to roam) go for a walk in your hiking shoes. Norway is all about nature, and there is so much of it that practically anything is worth seeing. That being said: The south coast is mellow and sunny and idyllic, the west coast is spectacular with deep fjords and high mountains, the North (north from Trondheim, that is) goes from New Zealand fjords to Alaskan tundra. The Eastern forests offer opportunities for canoeing and fishing, if you are into that. The mountains in the middle have peaks and glaciers and mountain huts, managed by the truly wonderful Norwegian Tourist Association. Explore at your leisure, spend time, and above all, do not feel forced to see the famous sites, when you can find something almost as good quite close.

I have never been to Geiranger, for instance. (But I haven’t seen Star Wars either.)

As for recommendations, here is a fairly random list of things I like:

  • knutshøMountaineering is great, but where to go if you don’t have much time? Some suggestions from Jotunheimen (but there are thousands of others):
    • Knutshø (pictured, steeper than it may look) is dramatic, close to the road, has fantastic views, and there are not many people there. Perfect one-day walk: Park the car and do it, walk along the ridge, return along the lake on south side. Great view of Besseggen.
    • Bitihorn is another, less dramatic alternative.
    • If you want to get above 2000 meters, try Rasletind.
  • The Rosendal Barony is breathtakingly beautiful, as is the view. Not far away (south of Odda there is a path up to Buarbreen, a part of Folgefonna, nice walk to a glacier.)
  • If you don’t fancy walking, taking the boat from Gjendesheim to Gjendebu and back is worth it. While reading Three in Norway by two of them.
  • Solvorn is a sleepy little hamlet with a fantastic little family hotel called Walaker. From there, you can drive to Veitastrond and Tungastølen and walk in to Austerdalsisen (fantastic glacier views, while they are still there.)
  • While I am at it – the Glacier Museum in Fjærland is worth a visit. As well as the nearby glacier (mind your step.)
  • Ålesund is a beautiful little coastal city which burned in 1904 and was rebuilt in Art Noveau style. Many Norwegian cities were architecturally brutalized in the sixties and later, but Ålesund has kept its style.
  • bakklandetTrondheim is nice – and Baklandet Skydstation has been named the best cafe in the world several times. I love it. And it is close to the only bike lift I know of. Trondheim is a university town – primarily the technical university, Norway’s MIT.
  • Bergen is nice – the whole city – perhaps with an exception for the fish market, which has transmogrified from local and lively market to a canned tourist experience the last few years. Will probably be fixed. Take a tram or cable car to the surrounding hilltops*.
  • The south coast has lots of beautiful little towns with white-painted “skipper” houses. I like them all – Lillesand, Grimstad, Arendal, Tvedestrand, Risør, Kragerø – but if you somehow can get to Lyngør, it is absolute perfection in summer.
  • Tromsø has nightlife and nature and the most active local patriots in Norway. And that is saying a lot.
  • I quite like the Henrik Sørensen museum near Holmsbu.
  • Svolvær and Lofoten and the other Hurtigruten things are fantastic, of course, don’t misunderstand me, but they tend to fill up.
  • Fishing is, apparently, fantastic. (I don’t fish myself…)

A few notes:

  • You cannot see the northern lights in the summer. Sorry.
  • Nor the midnight sun in the winter.
  • Norway is expensive and service is friendly but overstretched (we have near wage equality and employees are very expensive…). Do not expect fawning service, even at fancy hotels.
  • pictures_funny2_grandeNorwegians are rather reserved and have a fairly expansive sense of personal space. When the other people at the bus stop are keeping their distance to you, it has nothing to do with you. It’s them. (See The Social Guidebook to Norway for further information).

And that is that. Comments welcome. More to follow as I remember things.

PS: Found this video with Morten Rustad’s 10 favorite places in Norway. I spent much of my summers as a boy in Valdres and Jotunheimen and agree with him. But note also: Solitude and challenge is a plus, crowds a minus, for the quintessential Norwegian tourist/explorer:

*No, not mountains. I Norway, any mountain with trees on top is a hill, no matter how steep or tall.