Monthly Archives: February 2009

FAST Forward 2009: Notes from the third day

Bjørn Olstad: Microsoft’s vision for enterprise search

Search as a transparent and ubiquitous layer providing information and context seamlessly – from a search box (tell me what you want in 1.4 words and I will answer) to a conversational interface (giving pointers to more information and suggestions for continued searches, to a natural interface.

Demo of Microsoft Surface: Camera interface, can recognize things. Multiuser (as opposed to Apple. Showed an application built on search with touch – whenever you touch an information object a query goes towards an ESP implementation and brings up all the information available on that object.

Very impressive demo of Excel Gemini: How do you fit enterprise data into Excel. (Picture of a VW bug with a jet engine.) Pulls 100 million rows into Excel, sort them (instantly), slices and dices. Built on top of ESP, does extreme compression, takes advantage of high memory, allows publishing of live spreadsheets to Sharepoint. Extremely impressive, worth the whole conference.

Bjørn continues talking about search as a platform: Demoing Globrix.com, where you can ask questions about apartments and houses and get a rich search experience where you can change attributes and the data changes dynamically. Globrix does not hold content themselves, but crawls available content on the web and shows it (much like Kayak.com for airline tickets).

Another demo: Search for entertainment based on location, friends and content. Moving from there to a focused movie site. This is federated search that understands some of the semantics (understands that “David Bowie” refers to a person and therefore only search certain databases.) Also incorporates community (letting users edit the results and feed them back).

FAST AdMomentum – advertising network – has had tremendous growth.

Content analytics: How can you lay a foundation for a good search experience by focusing on data quality? Demo: Content Integration Studio, sucking out semantics from unstructured text and writing it back both to the search engine and to databases (such as an HR database).

Panel session on enterprise search

Hitachi consulting (Ellen): Very big focus on the economy now, almost all conversations are about that topic. eDiscovery is important: Looking at many sources with a view towards risk discovery and risk mitigation.

EMC consulting (Mark Stone): Natural interfaces will be important, frees up the mind to focus on the information rather than the interface. Shows a video of a small girls using the Surface table and how she very quickly starts to focus on the pictures she is manipulating rather than the interface – she completely forgets that she is working with a computer.

Sue Feldman, IDC: We have to get beyond the document paradigm. I want to see interfaces that will immerse me in the sea of information and explore it, without having to think about what application it is in.

Sue Feldman: Core issue with search: Data quality and making it a rich experience for the user. Anthropological, linguistic and cultural issues, getting people to understand both what they are seeing and what they are looking for.  We are just beginning on this journey. From keyword matching and relevance ranking to pulling the user in, having a dialogue with the information. What we are seeing is hybrid systems that combine collaboration, search, analysis etc.

AMR Research: There is a religious war going on, between collaborative systems, portals, content management systems, and search. They all claim to be the answer to the problem of connecting users with their data. There is also consolidation in the market, partially driven by the economy, but there is also a consolidation of functionality and an explosion in new ideas, many small companies coming up with new ideas.  No one technology is going to solve all of these problems. Lots of opportunity because Microsoft is gobbling up all these technologies, trying to provide one product that covers most (Sharepoint).

Q: Examples of interaction management?

Hitachi consulting: Best examples currently found in collaboration and community software.

EMC: There is a tool out there that searches not only blogs, but specifically the comment sections of blogs, looking for mentions of products. Do sentiment analysis, find out what the customers are saying about you.

Sue Feldman: Searching through corporate communications in lawsuit situations. Ad targeting. And what is the relationship between search and innovation?

Hitachi: Innovation comes from finding what you did not expect to find.

Q: This question always comes up: Search is a commodity – or is it? What is the current market doing for search adoption?

AMR: I am not sure who says that, there is so much room for innovation, so I can’t understand why anyone would say it is commoditized. Go out there and find the opportunities.

Sue F: Well, search is a tool, like a screwdriver. But I really need a screwdriver. The toolbox has expanded so much. I see the search market continuing to explode even though the technology is tanking. Possible that we will see a disruption with a new platform based on information management, access and collaboration.

EMC: We are seeing growth, the business will mature because companies have to focus on what the business really needs.

Sue Feldman & others: Search use awards

Customer awards:

  • Best productivity advancement: Verizon Business.
  • Best digital market application (I): McGraw-Hill Platts (doing industry-specific searches, 50% increase in trial subscriptions, 40% increase in revenue.)
  • Best digital market application (II): SPH Search (reader interaction and content integrated with newspaper sources, federated search.)
  • Social computing: Accenture (internal search on people profiles and content)
  • User engagement: Kakaku.com, Japan (700m pageviews, 18m unique users)
  • User engagement: AutoTrader (peak query level of 1500 qps)

Partner awards:

  • Digital market solution: Comperio (use of search for user interaction)
  • Social computing solution: NewsGator (enterprise social computing on top of Sharepoint)
  • User experience solutions: EMC Consulting
  • Partner of the year: Hitachi consulting.

FAST – technology futures and optimization

Notes from various presentations at FAST Forward

Bjørn Olstad and Svein Arne Gylterud: Technology briefing

Two main directions: Fast for Sharepoint and FAST for Internet Business. Various other licensing options. Richer search experience, taking into account time, user profile data, and tagging.

Some new features, available directly: In-picture thumbnails view of docs, can collect powerpoint slides without starting ppt.

FAST Search for Internet Business: Content Integration Studio, (new version), scalability increased (better performance on less server).

FAST AdMomentum: Competitor to Google Adwords and Adsense. AdMomentum is an integrated platform for managing ads, including display ads. Can track user behavior across devices and platforms.

Data increasingly residing in a hybrid infrastructure, need to move to a model with intent in, content out. Moving from text-centric approach to richer media. Will continue multiplatform, but some new components are based on .net and will therefore only run on Windows.

Main innovations: Configuration tool: Same tool for doing indexing of content as for evaluating queries, based on the graphical user interface from CIS. Also innovations in the search core: More context awareness, more scalability.

Mark Stone and Richard Griffin, EMC2: Beneath the Surface: Search Without the (Text) Box – An Insight into the Next-Generation of User Experiences

Going from a command line interface – not really different from Archie – to conversational interfaces (such as FAST) to natural user interfaces (example: Look at an apple through a screen and get information about it superimposed on the image.) Another example: An umbrella, made by Ambient Devices, which has a handle that glows blue when it is going to rain. All this is powered by search.

Key areas for NUI (Natural User Interfaces): Relevancy, simplicity, speed (reducing the time consequence of errors), and unification (get everything in one interface).

Demo: The Look Finder. Dynamic changing of pictures of clothes based on attributes, role models (Kate Moss), color etc.

Richard Griffin: How to design NUIs.: Start with the user, finding out what attributes that are important for them. Do a lot of sketching, Flash-based technology etc. to create an information design. Need three roles: Designer, developer and integrator.

Designers use Photoshop and Illustrator, Integrator uses Blend, developer uses Visual Studio.

Demo: Surface table with an application

Dan Benson and Paul Summers, Microsoft: Making FAST ESP Shine: Best Practices from the Solution Architects

Performance tuning: Indexing latency: Document processing: Entity extraction, clustering, lemmatization, doc conversion, etc. Doc processing is CPU intensive.

Index performance tuning: Use 4-5 partitions, keep partition 0 and 1 small, so that they can be indexed quickly and used for late insertion documents. Consider doing lemmatization in the query expansion rather than in document expansion.

Put small partitions on RAMdisk or SSD.

Query performance tuning: QPS is a critical issue. Queries enter the system through the QR server. Processed and passed down to top level dispatcher, which distributes the query to the low level dispatchers, which sits on individual nodes. In high QPS scenarious, you want fewer partitions, typically 3. Typically, you turn off spellchecking and query-side lemmatization as well as synonym-expansion on the query side (which means you have to do that on the document processing side, which is always the tradeoff. Not much tuning done on the dispatchers. Much tuning can be done on the low level search engines, for each partition.

Navigators and document summaries allows for a lot of performance gain. Can get a lot of performance from not sending fields back that you are not going to display, even though they are searchable. Navigators are costly both for memory and query performance because of the CPU computations necessary. Be careful with the number of navigators requested – on send those that the interfaces need. Wildcards are expensive because it needs to index all varieties of terms. Turn them off on long fields. Hit highlighting is also costly and can be turned off if you don’t need it. Reduce nesting, use filters before you do dynamic ranking, so you have fewer results to rank. Higher up: Minimize the number of hits.

Document capacity: Try to save hardware costs when storing documents. If you want to optimize for this, you need to create more partitions. Keep in mind that a 32bit system has a max of 4G per processor. Archive Indexing Feature allows for adding new nodes on the fly, because it sends data to columns that have capacity. This is useful when sizing the installation.

FASTForward 2009 – impressions from the second day

The second day has less of the “big picture” and more of product announcements and more technical detail. Here are some notes as the day progresses:

Kirk Koenigsbauer, Microsoft: Our enterprise search vision & roadmap

Kirk is responsible for the business side of FAST after the acquisition. He is speaking on Microsoft’s commitment to search, the roadmap and future business directions, including pricing.

About 15% of the research done in MS Research is search-oriented.  10 years support on current FAST products, even non-MS platform.

Search server express now has more than 100,000 downloads. 1/3 of MS enterprise customers have deployed a MS search solution. Partner #s have doubled.

MS vision: Create experiences that combine the magic of software with the power of Internet services across a world of devices. Search is integral to vision.

Demo: Use of search in a business setting, showing documents in a viewer format, extracting keywords and concepts.

Announcing two new products:

  • FAST for Sharepoint, which is FAST ESP integrated into Sharepoint, available at a substantially lower price than FAST ESP, typically 50% lower price. Simpler pricing model: Per-user charge for FAST ESP standalone, included in Sharepoint. Still need to buy a server at 25K a pop, but this is substantially lower price. Will be available from next rollout of Office (wave 14). Will also provide a licensing bridge for those who purchase Sharepoint now.
  • FAST Search for Internet business. New functionality for interaction management (promotions, campaigns etc.), Content Integration Studio (graphical interface for managing content restructuring and content integration), and simplified licensing: Language pack and connectors will be part of the standard package.

Valentin Richter, Raytion: User engagement

Low satisfaction with many search solutions, and 70% of search managers do not study search logs with an eye to improve the experience. Went through a list of common myths about search (such as “people know what they are looking for”.) People want simplicity – they cannot handle expressions and need more of a drill down approach navigating through related information. Installing search platforms immediately needs to a focus on information quality: You find duplicates, you find confidential documents everywhere, and so on – be ready for it both in a technical and organizational sense.

Walton Smith, Booz Allen Hamilton: Case study of use of FAST and Sharepoint

BAH based in Virginia, traditionally centralized, but expanding. 300 partners, all wanting to go in different directions. De facto collaboration tool was Outlook. Created a social computing platform called hello.bah.com. Among the results: Have given access to more esoteric material, which caused issues with indexing. Were able to pull new people from other parts of the organization on a project. Other application: staffing.bah.com, finding people with the right credentials and experience, pulling information from many sources. search.bah.com crawls hell and iShare. About 1/3 of the firm is now using the platform, lots of information on individuals.

Charlene Li: Transformation  based on social technologies

It is all about engaging users in dialogue: H&R Block has a page on Facebook where they discuss tax issues – not trying to pull people in, at least not explicitly. Comcast is on Twitter with their customer service people. Starbucks testing ideas, such as automated purchasing based on a customer card. Beth Israel’s CEO blogs about what it is like to run a hospital. Necessary to change search to include social software: Technorati searches blogs, del.icio.us allows social bookmarking. You can use Twitter mapping to see what people are discussing – showing that what is rated high somewhere may not be what is most discussed. Amazon now lets you filter reviews by friends.

Conclusion: Social networks will be like air, and will transform companies from the outside in. Social media is impacting search at multiple levels, refining results based on personalization details derived from their social circles.

Jørn Ellefsen, Comperio: In search of profits

Comperio has more than 100 customers and have created a front application, Comperio Front, that sits between the customer’s web pages and their search engine. Introduced Drew Brunell who works with SEO for, among others, News International. Paid search is the growing part of the advertising market, everything else is either flat (display ads) or sinking (traditional ads). Doing a lot of experimentation linking into customer behavior – for instance, matching content with areas that see a lot of conmments, “invisible newspapers”. Another notion is the “curated content model”, setting up pages with a blend of original content with stuff from the outside web. Topic pages based on “zero-term search”, offering editorial content put together automatically around. Stefan Sveen, CTO Comperio, demonstrated topic pages from Times Online: User and journalists can create their own topic pages, based on search results and mark entries coming in after the page is created.

Venkat Krishnamoorthy, Thomson Reuters: Delivering Contextual and Intelligent Information to Premium Customers

Reuters delivers context-sensitive information for pre-investment analysis to premiere customers. They have done this for a long time, but want to change from being a data-delivery company, but to integrate into  the user’s workflow. Challenges here included having too many applications the customers needed to stitch together, finding information was difficult, especially across different kinds of assets – more than 40 content databases.  Solution: Put in a search and navigation layer between their desktop products (they have two, a web-based one and a premium, client-based one).

FASTForward 2009 – impressions from the first day

This year, FASTForward is in Las Vegas, which is quite a shock to the system. I haven’t been here since 1987, when I visited COMDEX and stayed at the Bally, then the largest hotel here. Now you can hardly see it for all the new and much bigger ones. (Las Vegas has changed in other ways as well, the business model is no longer “cheap booze and food, make it up in the casino”. Since most people no longer come here primarily to gamble, so a glass of red wine was $15 in the bar..)

The theme for the conference this year is “Engage your user” and the program builds on the one from last year, but with, it seems to me, fewer circuit speakers and more company examples, which is fine with me.

Don Tapscott: The Net Generation

First speaker out: Don Tapscott, actually a colleague of mine from nGenera, talking about his new book Grown up digital. Don is very optimistic about the net generation, in contrast to the many articles that have cropped up saying that the current net-addicted generation is the dumbest ever. The population in USA has gone from boom to bust to echo: The echo group is called the Net Generation, the digital natives. For the first time in history, kids are authorities (in relation to their parents) on something that is important. In contrast to their parents, when asked whether they want to be smarter or better-looking, two thirds of them answer “smarter” (except in England.) Are they coddled? 40% of them move home after university. but is this a bad thing?

HTML beaten by XML: Tapscott’s son created a Facebook community around Wikinomics, got 126 users in 7 countries in one day. Then the community started pointing out errors in the book and requesting participation from Tapscott himself. Company ad spending online is growing faster than time people spend online. Prescription: Don’t create products, create Consumer experiences. Don’ create place, create Anyplace. Don’t focus on price, focus on Discovery as a mechanism for price setting. Don’t focus on promotion, focus on Engagement. Marketer are losing control of the brand, which now needs to have integrity, or the influence network will kick in and trounce you on the social networks.

There are many things young people don’t know – so perhaps it is time to tear down these generational firewalls and get some mentoring going.

Clay Shirky: Here comes everybody

Five word summary of the book: Group interaction just got easier.

Example: Someone posts an HDR picture on Flickr and asks what the best software for this is. It balloons into a long and very technical discussion, with more photos. Happens every day, but is a very powerful technology that has profoundly changed how we interact. Since you can attach conversations to everything, every URL is a possible platform for interaction.

Another example: HSBC sold many students on penalty-free checking, but reneged on it during summer vacation, wanting to charge their customers 140 pounds. Didn’t count on Facebook. The information advantage of HBSC disappeared – one person found out how to simply move his money to another bank, posted it, and that was it. Then the thousand of Facebook users started to protest – and eventually HSBC caved in to this group of coordinated participants.

Information moves faster: The earthquake in Sichuan last year got reported immediately via qq.ch and Twitter. The last time something similar happened, in the 1970s, it took three months before the Chinese authorities would even admit it would happen.

Another example: Chris Avenir, student at Ryerson University, taking Chemistry 101. Started a study group on Facebook, 146 participants. And then he gets a note from the university saying he may be expelled because he was putting course stuff online, whereas he saw it as simply taking what he was doing in the real world online. This is a clash of metaphors: The university sees it as a media outlet, the students as regular social life. That is the wrong question to ask: Facebook is Facebook, not a manifestation of something else.

Crowsourcing: Howardsforums now has engineers from handset manufacturers sending their customers there. Companies look at this as a source of free labor, but fail to see the passion necessary – and the fact that they cannot, inside their companies, model the complexity and the uses of their products that takes place out there in the real world.

Example: Dogear, an internal service in IBM similar to del.icio.us, in that it let users tag and save URLs. It led to two research groups (one in England and one in the US) finding out they were working on the same problem and initiating collaboration.

How to get started with this? Amazon has lots of social applications, but have never had a strategy for it, nor a button marked “community”. Their social interaction tools have evolved through trial and error, which takes a tolerance of failure and a willingness to learn.

Dan Rasmus: Navigating the new world of business

Dan is Director of Insight in Microsoft (now, there is a title I have given to myself many times). His role is to look at changes in business and the workplace on a rolling basis. Quotes Schopenhauer: “The task is not so much to see what no one yet has seen, but to think what nobody has yet thought about that which everybody has seen.”

Microsoft uses scenario planning to try to make sure that investments will work in several scenarios as opposed to depend on one and only one future change. Key dimensions: Globalization vs. bordering, centralization vs. decentralization. Four scenarios: Proud tower, Border Friction, Freelance Planet and one more. [This was presented in a rather busy video.]  Credit crisis has a tendency to overpower everything, but we still have to worry about environment, mobile tech, and other things. Globalization is there in spite of “buy American”: A standard breakfast cereal has components from all over the world. How do you manage a workforce of outsourced people, freelancers and consultants?

Other examples: Data everywhere, placeless work (punctuated workday), need to be open to attract younger workers. Going towards a blended future, with blended jobs – such as “authenticity engineer”

Acer Aspire One experience

Before going to the FastForward 2009 i bought a netbook, an Acer Aspire One, with 1Gb RAM and 160Gb harddisk, running XP. The idea was to use this for notetaking during the conference, since the machine is small and has good battery capability (I got the 6 cell version.)

A funny thing: When I went to buy it, the salespeople tried very hard to get me to buy a subnotebook instead – which would cost upwards of $1000 instead of $329. Margin call? Anyway, a true sign of a disruptive technology is when the salespeople sneer at it, so I predict a great future for these machines. (That being said, I would have liked to get the Sony Vaio P, but it was not yet available and the interesting version was more than $1100. In other words, I can get tree Acers for one Vaio..)

So far, this thing is working very well. Its wireless network access is not as good as my Lenovo X61 – the speed is lower and it seems to not be as good in keeping the connection. The processing power is lower, but as long as I don’t have more than 4 big applications open concurrently, and make sure I quit Firefox about once per 12 hours (it is something of a memory hog) it is just fine.

The keyboard is surprisingly good for such a small size, and I do pack a real keyboard and a mouse in my large travel bag. I do make a few mistypes occasionally, with somewhat unpredictable results, but mostly touch typing goes well, though I would not want to write a dissertation on it.

The touch pad is a tad bit sensitive, I will need to fiddle with some of the settings – it is quite easy to hit it and inadvertently increase the size of the font in Firefox or to accidentally delete large parts of text in Live Writer or Thunderbird. Ctrl-Z is very useful. I would prefer a navigation knob such as is available on the Vaio or the Lenovo computers – more control, fewer accidents.

The thing comes with a camera which I haven’t tried yet and a various other ports and functions which, presumably, work (and were one reason I got this one rather than the HP Mini, which lacked a few.)

So far, so good. Excellent note-taker and internet device, not strong enough to be your main machine unless you are either doing very simple stuff or mainly working online, in which case it is surprisingly good. And it does fit on the tray in economy class, which is a bonus.

Update Feb 10: I find that, with the small screen, I tend to use all applications in full screen mode and switch between them using alt-tab. This is a little bit of a throwback to the mid-90s, when I used DesqView on a text-based DOS machine in the same way. Works fine, though. And the battery worked the whole day yesterday, from 9 to 17 with a small 20 minute charge at the hotel room – which I must say is rather good for a machine at a third of the price of a regular notebook.

Wheels…

Since I am staying in the USA for the spring semester, I have to buy a car (though renting a car is cheap here, it is much less expensive to buy one.) At first, I thought I would just buy a car and sell it when I leave – either a very cheap one that I could just get rid of, or something nicer that I could sell back to a dealer.

Then it occurred to me – why not buy a car which I could take back home to Norway when I am done here? After all, cars are expensive in Norway, and shipping one across the Atlantic is $2000-3000, depending on the size and type.

Now, this is more complicated than it sounds. Import duties on a new car average 200% in Norway, they are prorated for age, but the larger the engine and heavier the car, the higher the duties. So even though nice used cars are ridiculously cheap in the States, the taxes would make the car more expensive than just getting it in Norway.

But there is one loophole: Cars older than 30 years are called "vintage" and subject to a mere 25% sales tax. So I have been trying to find a nice 30 year old car, which I could drive for a few months in the States (so I find out whether there was anything seriously wrong with it) and then ship back to Norway. The requirement was that a) it should not be too expensive (so a Ferrari or something more exotic is out of the question,) b) be reliable (that rules out Jaguars and American cars,) and c) have more than two seats, since there are children and dogs to be transported (and that rules out all those nice Mercedes SLs which are quite easy to come by.)

image So what did I end up with? Well, here it is, a 1977 Mercedes 450 SEL 6.9 (known to car enthusiasts just as a "6.9",) silver paint with a nice blue leather interior and a truly fearsome engine. It has 140K miles on it and a few spots here and there that will need to be fixed, but it as been well maintained, image runs really well and will allow me to tool around Boston and environs with some style. After all, this particular model was more costly than a Rolls Royce when it was launched.

The car handles like a sports car although it is a rather large sedan. This is the first time I have had a car capable of producing instant tire squeal and effortless (and silent) acceleration past 100 mph. As for gas consumption, let’s think about that some other time. I still have to take it to a body shop to deal with some dings and a few rust spots that need welding, but if things work out, I should eventually have a rather stylish vehicle for summer driving back in Norway.

Mercedes 6.9 009 Update February 23: Have now driven the monster for three days (including going through a hail storm in Connecticut.) It does take a bit of getting used to – especially the large steering wheel, which feels like a bus wheel though it has power assisted steering. The climate control is excellent: Set it and forget it. And I find that, despite the zip, it induces the same kind of driving that I used to do with my old Chevrolet Caprice Classic model 1980 back in the mid-90s: Low-stress ambling down the highway, listening to WBUR (one of Boston’s PBS radio stations) and eventually getting there. But this time without the seasickness in curves and with plenty of power to pass other cars, should I feel the urge. Aahhh, relaxation…. I have ordered a chauffeur hat, so I can sneak into the limo line when picking up people at airports….

Incidentally, the picture on the left is of Tom Rossiter, who runs The Stable Ltd, where I bought the 6.9. He has a wonderful collection of what he terms "interesting" cars, so a visit to Gladstone, NJ, is highly recommended.