Interesting times? (or, the political correctness of scientist superstition)

I am about 3/4 through Eric Hobsbawm’s autobiography Interesting times (Amazon, Google search.) I picked this up at Heathrow airport on a lark (buying almost random books leads to unexpected discoveries.) Hobsbawm is a highly respected and famous historian – respected for his books, famous for remaining a Marxist to the bitter end.
The book makes me a bit uneasy. One one hand, it makes me want to buy some of his books – notably The Age of Extremes (Amazon), a history of the years 1914 to 1991, which I haven’t read yet and look forward to. On the other hand, something about Robsbawm’s recollections disturbs me.

Robsbawm is highly intelligent and learned – his analyses of the evolution of politics in the UK fascinates, partly from his excellent writing, partly from the people he has met and known (almost like Peter Drucker’s Adventures of a bystander.) Moreover, he is aware of his surroundings, almost in real time. But a nagging question remains: How can such a smart person remain wedded to a political ideology that has proved its moral and practical impossibility over and over?
I have known a number of smart people myself, and some of them have, despite an awe-inspiring analytical brilliance in their work, espoused bizarre views and beliefs – political, religious, or just plain old superstitious – in their private lives. I ask myself how someone so smart can succumb to something so patently irrational? One thing is that entrepreneurs and businesspeople sometimes have non-rational views, but that may come with the territory – after all, one of the most important attributes of an entrepreneur is the willingness to suspend disbelief in a business idea. However, how can that be the case for academics, where scientific positivism (or, at least, scientific methods), openness to criticism, and continual revision of one’s assumptions and prejudices is supposed to be not only the norm, but the whole justification for the activity? If you are a smart, learned and critical researcher – how come you do not apply those same criteria to the belief you have attached yourself to, and start to wonder why you pledge allegiance to such claptrap?
One thing is whether this belief is harmless – quite another if it’s something that will cause other people harm – and quite another if the believer is in a position to use his or her “rational” background to influence others to adopt the same “irrational” beliefs. Most of the smart people with weird beliefs I know solve this by not mentioning their weird ideas at work – or to refer to them jokingly, in a way that cuts off discussion or at least signals that discussion should only proceed in a harmless fashion, not challenging the belief.
Moreover, what happens when the professional and private lives meet? Hobsbawm’s loyalty to the Communist Party precluded him from writing modern history until the Party went out of existence. He offers this as an explanation, but does not apologize for it. In other words, he let his faith – or, if you will, irrational belief – get in the way of doing the scientific work he obviously wanted to do (instead writing about the 19th century). Scientists are usually quick to denounce this when it is forced on them, but do not seem to want to discuss it when the limitation in what is researchable is imposed by themselves.
J. M. Roberts, in his History of the world (Oxford, 2003, Amazon) observed that despite all our scientific progress, there does not seem to be any reduction in superstition and magical beliefs in the world. Religious sects, new-ageism, and most spam messages shows this. I wonder to what extent if would help if also scientists would be willing to acknowledge their superstitions and apply the same rigor to them?
Addendum two days later, when finished
The last chapter is about the US, with a Coda on the world post 9/11/01. Excellent observations, though I still feel uncomfortable…..