Category Archives: Teaching

Analytics projects

asm_topTogether with Chandler Johnson and Alessandra Luzzi, I currently teach a course called Analytics for Strategic Management. In this course (now in its second iteration), executive students work on real projects for real companies, applying various forms of machine learning (big data, analytics, whatever you want to call it) to business problems. We have just finished the second of five modules, and the projects are now defined.

Here is a (mostly anonymised) list:

  • The Agency for Public Management and eGovernment (Difi) wants to understand and predict which citizens are likely to reserve themselves against electronic communications from the government. The presumption is that these people may be mostly old, not on electronic media, or in other ways digitally unsophisticated – but that may not be true, so they want to find out.
  • An electric power distribution company wants to investigate power imbalances in the electric grid: In the electric grid, production has to match consumption at all times, or you will get (sometimes rather large) price fluctuations. Can they predict when imbalances (more consumption that production, for instance) will occur, so that they can adjust accordingly?
  • A company in the food and beverage industry want to offer recommendations to their (business) customers: When you order products from them, how can they suggest other products that may either sell well or differentiate the customer from the competition?
  • A petroleum producing company wants to predict unintended shutdowns and slowdowns in their production infrastructure. Such problems are costly and risky, but predictions are difficult because they are rather rare – and that creates difficulties with unbalanced data sets.
  • A major bank wants to look into the security profiles of their online customers and investigate whether some customers are less likely to be exposed to security risks (and therefore may be able to use less cumbersome security procedures than others).
  • An insurance company wants to investigate which of their new customers are likely to leave them (churn analysis) – and why. They want to find them early, while there is still time to do something to make them stay.
  • A ship management company wants to investigate the use of certain types of oil and optimise the delivery and use of it. (Though the oil is rather specialised, the ships are large and the expense significant.)
  • Norsk Tipping runs a service helping people who are in danger of becoming addicted to gaming, an important part of their societal responsibility which they take very seriously. They want to identify which of their customers are most likely to benefit from intervention. This is a rather tricky and interesting problem – you need to identify not only those who are likely to become addicted, but also make a judgement as to whether the intervention (of which there is limited capacity) is likely to help.
  • A major health club chain wants to identify customers who are not happy with their services, and they want to find them early, so they can make offers to activate them and make them stay.
  • A regional bank wants to identify customers who are about to leave them, particularly those who want to move their mortgage somewhere else. (This is also a problem of unbalanced data sets, since most customers stay.)
  • A major electronic goods retailer wants to do market basket analysis to be able to recommend and stock products that customers are likely to buy together with others.

All in all, a fairly typical set of examples of the use of machine learning and analytics in business – and I certainly like to work with practical examples with very clearly defined benefits. Now – a small matter of implementation!

Advertisements

Cool ad. Literally speaking.

This is our (NI Norwegian Business School) new ad for attracting international students. Which, if they do master’s courses in strategy, might mean that they will meet me. Be forewarned, I am just like this guy, with somewhat shorter hair and beard…

Looking forward to seeing you!

Big Data in practice

(This is a translation of an earlier post in my Norwegian blog. This translation was done by Ragnvald Sannes using Google Translate with a few amendments. This technology malarky is getting better and better, isn’t it?).
ml_mapI have just finished teaching four days of data analytics – proper programming and data collection. We (Chandler, Alessandra and the undersigned) have managed to trick over 30 executives and middle managers in Norway to attend a programming and statistics course (more or less, this is actually what analytics basically is), while sort of wondering how we did that. The students are motivated and hard-working and have many and smart questions – in a course taught in English. It is almost enough to make me stop complaining about the state of the world and education and other things.
Anyway – what are these students going to do with this course? We are working on real projects, in the sense that we require people to come up with a problem they will find out in their own job – preferably something that is actually important and where deep data analysis can make a difference. This has worked for almost all the groups: They work on real issues in real organizations – and that is incredibly fun for the teacher. Here is a list of the projects, so judge by yourself. (I do not identify any students here, but believe me – these people face these issues every day.) Well worth spending time on:
  • What is the correct price for newly built homes? A group is working to figure out how to price homes that are not built yet, for a large residential building company.
  • What is the tax effect of the sharing economy? This group (where one student works for the Tax Administration) tries to figure out how to identify people who cheat on the tax as Uber drivers – while making suggestions on how tax rules can be adapted to make it easy to follow the law.
  • What characterizes successful consulting proposals? A major consulting firm wants to use data from their CRM system (which documents the bidding process) to understand what kind of projects they will win or lose.
  • How to recognize money laundering transactions? A bank wants to find out if any of their customers are doing money laundering through online gaming companies.
  • How to offer benefits to customers with automated analysis? A company that supplies stock trading terminals wants to use data analysis to create a competitive edge.
  • How to segment Norwegian shareholders? A company that offers online trading of shares wants to identify segments of its customers to pinpoint and improve its marketing strategy.
  • How to lower costs and reduce the risk of production stoppages in a process business? A hydropower company wants to better understand when and why your power stations need repairs or maintenance.
  • How to identify customers who are in the process of terminating? A TV company wants to understand what characterizes “churn” – how can they identify customers who are about to leave them?
  • Why are some wines more popular than others? A group will work with search data from a wine site to find out what makes some wines more sought after than others.
  • Which customers will buy a new product? A group is working on data from a large bank that wants to offer its existing customers more services.
  • How to increase the recycling rate for waste in Oslo? REN – Oslo’s municipal trash service – wants to find out if you can organize routes and routines differently to better utilize trash trucks and recycling plants.
  • How to avoid being sold out for promotional items? One of Norway’s largest grocery chains wishes to improve their ordering routines so that customers do not get to the store and find out that there is no more left of the offer they wanted.
  • How to model fraud risk in maritime insurance? An insurance company wants to build a model to understand how to find customers attempting to fraud companies or authorities.
  • Which customers are about to leave us? A large transport company wants to find out which customers are about to go to a competitor so that they can take action before it happens.
  • What characterize students who drop out? BI enters 3500 new students each year, but some of them end after the first year. How can we find evidence that a student is about to drop out?
Common to all the projects – and so it’s with all the student projects I have advised since I started in this industry – is that you start with a big question and reduce it to something that can actually be answered. Then you look for data and find that you need to reduce it even more. Then you get problems that the data is either not found, unreliable or inadequate – and one has to figure out what to do with it. Finally, after about 90% of the time and money budget is gone, one can begin to think about analysis. And then there is a risk that you find nothing…
And that is an important lesson of this course: The goal is that the student should be able to know about actual data analysis to ask the right questions and have a realistic expectation of what kind of answer you actually can get.
There is a great demand for this course – so we have set up an additional course this fall. See you there!

Made my day!

digøkskjermI just got the message that the new bachelor program Informatikk: Digital Økonomi og Ledelse (Informatics: Digital Economics and Management) is now the most sought-after study program in Norway, with 19 applicants per available place (514 first-priority applicants for 27 available places).

Since I have taken the initiative to this program and developed it with colleagues at the University of Oslo (where I have an adjunct position, this definitely made my day. Week, actually.

Just sayin’…

Effective student feedback

In our book Teaching with cases: A practical guide, Bill Schiano and I talk at a fairly high level about how to give effective student feedback by using a spreadsheet and personalized emails. Our argument is that by giving every student individual feedback in addition to the grade, you reduce the number of grade justification requests and complaints. This blog post is a detailed guide on how to do it – too detailed for the technically inclined, probably, but we all have to start somewhere.

You probably already have all the tools you need on your computer – a spreadsheet and an email client that works with your spreadsheet – such as Excel and Outlook, for example. (For myself, I use Excel and SerialMailer, a cheap serial mail client for Mac.) You can probably use online software as well, for instance a Google spreadsheet (which is nice because editing by more than one teacher is easy) and Gmail, though I have never tried it.

The idea is to use the spreadsheet to organize all your feedback, and to set it up so you use as little time as possible to give as much feedback as possible. I will demonstrate this, with an example for a fictitious course with three in-class sessions (with participation grading, 40% of grade), an individual written examination (30% of grade), and a group assignment (30%). I will show the various details of building the spreadsheet below – if you want to skip ahead and inspect the thing your self, I have made it available for downloading.

I start with a spreadsheet of students and email addresses, provided to me by the administration or downloaded from our LMS. Let’s say it looks something like this:

1eval

(If I have the time or can get the administration to create it from their databases, I ask to have first and last names in separate columns. For this demonstration, I won’t bother.)

Then I add columns for each of the assignments that I am going to grade:

2eval

(In this example, the individual examination has six questions, of which the students should answer four.)

For the group project, I create a separate sheet (in the same workbook, called “Groups”

3eval

The group sheet is exceptionally simple, just group number, points and comment. If you have several group assignments, this is where you will put them:

4eval

Note that I also create a group numbered 0. This is what I use for students who drop the course or don’t do the group assignment.

With that done, I assign students to groups in the Students sheet…

5eval.jpg

…and then I am ready to start teaching my course.

As the course rolls along, I enter points and comments for each student. As mentioned in the book, it is extremely important that you do the participation evaluation immediately after each class. I tend to give the students a score of 1-3, sometimes 1-6, with some definition. As I will show later, what scale you use does not really matter, as you can normalize them to whatever you want later in the process.

Anyway, assume the course is finished, and you have entered comments and points for everything – for the individual student, in the Student sheet….

7eval.jpg

…for the groups, in the Groups sheet:

6eval

To finish the evaluation part (we’ll get to communication later), you need to a) match the group points and comments to each individual student, b) calculate a final score for each student, and c) determine the letter grade for each student.

First: Group grades and comments. For a small class like this, this is probably a bit of overkill – you could just copy comments and points over to each individual student. But doing it the way I show here has the advantage (aside from being correct from a database administrator’s point of view) of error-correction (any error you make will be systematic and therefore easily spotted) and repeatable (the first time you do this, it is a chore, the second time, you just copy your previous spreadsheet and fiddle with it). Moreover, if you have a class with more than 40 students, a bit of “programming” saves time and effort. (I have done this for classes of 350 students, a situation where participation grading is not really possible – except as a small reward for exceptional students – but where the group feedback mechanism becomes extremely valuable.)

So, first – link individual students with their group’s scores and comments, using the LOOKUP function:

8eval

The formula is

           =LOOKUP($E2;Groups!$A$2:$A$8;Groups!B$2:B$8)

and what it does is take the student’s group number (E2), look it up the first column of the “Groups” sheet (Groups!$A$2:$A$8), using fixed references to be able to copy the formula to the rest of the sheet, and displaying the group score (which is in column Groups!$B2:$B8). This nicely picks up all the group scores and comments:

9eval

We now have all the information for each student, then we have to calculate the various scores. We have the exam score already (the average of the exam points) and the participation score (a sum of the score for each session.) We now need to calculate the total points, which isn’t too hard: The max for participation is 9, for the others it is 10, so the formula for the total will be:

01eval

         =(W2*0,3)+(O2*0,3)+(J2*0,4*10/9)

(I use comma as a decimal delimiter, parentheses for readability). We can now add a student ranking in the G column (students are always interested in this, so why not tell them?)

02eval

Now we are ready to set the grades. The simple way to do this is to sort the students by their scores (or rank, if you will):

03eval

How you set the letter grades is up to you, of course, but it helps to have the students sorted. I set grades by starting at the top, trying to get a reasonable distribution, and make sure that I don’t use absolutes so that some unlucky student narrowly misses a better grade. Let’s say we end up here:

04eval

Now we are ready to communicate the results to the students. We will do that by writing a letter to them, composed largely of common text (i.e., feedback that is the same to all students), and them use the mail/merge function of word to merge in the individual details from the spreadsheet.

1feed

Example letter using SerialMailer

As said above, I use a product called SerialMailer on my Mac. The concept is simple: You write the letter, link to the spreadsheet, and insert field names into the text. When the letter is sent out (or printed), the field names are substituted for the values for each individual student.

Here is how to do it in Word (if you want to send it out via email, you need to have Outlook as well.):

First, open Word and write the letter:

2feed

(As you can see, I recycle much of my texts…)

Let’s start by replacing “student” in the opening salutation with the student’s name. Then you open the Mailings tab in Word and hit Select recipients from an existing list…

3feed

…and select your spreadsheet:

000ff

I get this message, click OK:

5feed

And open the “Students” sheet (i.e. the individual sheet):

6feed

If you click “Insert Merge Field” now, you should get a list of the column headings in the spreadsheet:

7feed

Delete the word “student”, choose “Insert Merge Field” and choose “Name”, and the field code will be in the document:

8feed

If you hit “Preview Results”, the code will be replaced by the content for each student:

9feed

Now write field labels and insert the fields you want to share with the students. I like to add the listing at the end of the letter, but you can do whatever you want:

1ff

Hit Preview, and this is what each student will see:

2ff

And there you go. Now hit “Finish and Mail Merge”, and select whether you want to print the documents on paper (or PDF) or send them out via email (shown).

3ff

You must tell Outlook where the email addresses are in the spreadsheet:

5ff

Then specify a subject and choose HTML Message (if you want formatted text):

6ff

…and, well, this is where I will have to stop, since I do not use Outlook. But trust me, it works well, the students love having individualized feedback, and it really isn’t that much more work than just providing the grade. As an added bonus: If students want a grade justification, you can just tell them that they already have it…

(Corrections and feedback welcome, of course.)

Teaching Hacks: Using Google Docs under It’s Learning

(This is a new category I just dreamed up – will post little snippets of useful stuff for teaching. My view is that technology should make your life easier and the experience of the student better – otherwise, don’t use the technology.)

At BI Norwegian School of Business we use a learning management system called It’s Learning. As these systems go, it is (I think) no better or worse than any other system, but the interface is a bit clunky. However, it has a very useful feature (which I learned from Ragnvald Sannes), namely the ability to display Google Docs within the page:

screendocs

This is very useful because

  • you can create all your course documents (syllabi etc.) in Google Docs, which is much better for editing and everything else. You can even edit the docs right in the It’s Learning window.
  • you can give the students read, comment or write capability as you please. Giving the students write access to a shared document is useful for many purposes – I use it as a shared arena for proposing term papers, for instance. Linda Rademaker uses a shared spreadsheet for student group formation – the students write themselves into groups, and she has a tab with “Lost sheep” who have not found groups to work in.
  • you can also share a Google Folder with the students and link that right from It’s Learning.

To set up a page like this, first create the document in Google Drive, copy the link to the document (“Share” in Google Doc, set the access rights to whatever you want), go to It’s Learning, click “Add” in the left column, choose “File or link”. Here you can choose various options, but what has worked for me is choosing “link” and pasting in the link. Make sure the “Embed page within itslearning” is checked, write the Title, and there you go.

Certainly has made my life easier, and hopefully made the students’ experience better.

(By the way, this does not work in China, of course (no Google Doc access), in case you teach there.)

Cases: How to prepare for and learn from them

My versatile and creative colleague Hanno Roberts and I have made a series of five videos on case learning and preparation, originally for students at the BI/Fudan MBA program. This teaching method is difficult both for teacher and student, but highly rewarding provided you give it proper attention – which means effective preparation. Hanno and I talk about the goal of case teaching, how students can prepare individually, how to prepare as a group, how to go through the case discussion in the classroom, and then we sum up with some strategies for how to retain what you have learned.

Hanno and I did these videos against a green-screen, with little preparation – we basically met, outlined a structure with some keywords, decided broadly on who should say what, and dove right into it. Most of the videos were shot once, and then the very capable Milosz Tuszko edited them, added background, logos and keywords.

The updated videos are a less wooden than the previous version, methinks, and available in high resolution and with better sound. We clarified the differences between my version of case teaching and Hanno’s (both work, by the way). Over the years the original videos have been much watched – hopefully, our students (and others) will watch them carefully, and the result will be better case teaching, more learning, and an even more enjoyable experience teaching.

Details about each video below the fold…

Continue reading