Posts

Showing posts from 2010

ParaView, VTK files and endianness

Image
Recently, I've been doing some work with ParaView, a rather nice open-source visualization application that's aimed at the display and analysis of large data sets. It has a wide range of visualization functionality, an interface that allows for interaction via its graphical user interface, or through scripting, and a distributed architecture. Executable versions are available for a variety of platforms, or you can download the source and compile it yourself. I'm currently running the Windows 64bit distribution on my laptop, and have also been building the source on HECToR, with a view to trying the application out on that machine, and using it for the analysis of some of the large data sets generated there.ParaView uses the open-source Visualization Toolkit (VTK) for data processing and rendering; this consists of a core C++ class library and interpreted interface layers for Tcl/Tk, Java and Python. Much of VTK's functionality - specifically, its visualization techn…

Half-way through HECToR

HECToR, the UK national supercomputer service, had its third birthday in October. NAG, as many readers will know, provides the Computational Science and Engineering (CSE) support for the service, helping users with application problems, and with porting and tuning their codes to make them as efficient as possible. We also provide an extensive programme of training courses throughout the UK covering both basic and advanced topics. To date we have had over 900 attendees on these courses and delivered them in 16 locations. We are currently putting together our programme for next year, upgrading our course materials in response to the latest hardware upgrade (from a Cray XT6 to an XE6), and developing material for some new topics that we haven't addressed before. A novel aspect of the HECToR service is the Distributed CSE Service (DCSE) which funds dedicated resources to work on specific codes. Those resources can come from within the research community itself, or from specialis…

Christmas Comes Early

Christmas came early for one little girl at NAG Ltd. Being in charge of office services has its ups and downs (hands down a blocked loo or physically being up a ladder changing light bulbs) are part of my normal day. But a call from the local fire brigade certainly cheered me up. Since the fire service has stopped its annual building inspections we are now reliant on paying somebody to come in every year to carry out a risk assessment and for some reason these people are always rather bland and lacking in making their work (although serious) fun. Anyone who was here two years ago and was lucky enough to have training on the use of fire extinguishers would know this has to have been the best training course ever; it was a laugh from start to finish. The request was to familiarise themselves with the layout of the building so if there was ever a real emergency and they would have some background knowledge. They had promised to arrive in a full rig and I had expected one or two officers,…

Now you see it...

Image
A few weeks ago, I presented the HECToR scientific visualisation training course to a group of researchers at the University of Reading. This course looks at the use of visualisation as a tool for the improved understanding of numerical data (as produced, for example, by calculations or simulations run on HECToR). We started by exploring the different types of data - characterizing each according to its structure, dependencies and dimensionality - before reviewing the different visualisation techniques (such as contouring, particle tracing, volume rendering) that are applicable to each type. Some examples of the techniques, used to display a variety of data types from several application areas, are given in the figures below.Figure 1. Fluid flow around a cylinder Figure 2. Visualisation of MRI data, showing an aneurism Figure 3. Potential energy in the space between three atoms Next, we briefly reviewed a few applications for performing visualisation, of which there are a plethora …

Fortran Fame

I was standing on the NAG booth here at SC10 yesterday, muttering to myself about the difficulty of maintaining an internet connection in the exhibition hall. The organisers claim that there's more bandwidth available at SC10 than is used by any but the five largest countries in the world - I think they're exaggerating because I can never get any of it. Anyway, along came a dapper fellow wearing a baseball cap who started a converstaion about the way standards were being implemented in the NAG Fortran compiler. I asked whether he was interested in using Fortran. "Well, I used to be", he said. Belatedly I looked down at his name badge. It turned out to be Tom Lahey, founding father of the Lahey Fortran compiler. What could I do but shake his hand and bow.

It's a hard life.

Every year in November NAG staff attend SC, formerly known as SuperComputing, an annual showcase where computer hardware and software vendors can display their goods and services, along with a huge technical agenda covering all aspects of HPC and scientific computing. The show moves around different cities in the United States. This year SC10 is in New Orleans, Louisiana, and running from 14th to 21st November, with around 10,000 registered attendees. It's the 23rd outing since the first one which took place in Orlando, Florida in 1988. NAG has attended every single one of these events, and is one of just a handful of outfits to do so. New Orleans was founded in 1718 by French speculators interested in making money from trade and finance, and named after the then French Regent, Philippe, Duke of Orléans. In the mid-eighteenth century it was ceded to the Spanish. They're the ones who actually created most of the characteristic architecture in the heart of the city now known as…

Looking forward to meeting people with new ideas at Supercomputing.

I’m someone who enjoys talking with people at exhibition booths and the International High Performance Computing event, which is being staged next week (http://sc10.supercomputing.org/ ), will provide good opportunities. Here is a chance to spend time with many current and future users of NAG products and services as well as with NAG collaborators. Attendees at this conference and exhibition work in such a variety of different areas that the conversations are always fascinating and informative. Meeting with people face-to-face inevitably triggers more new ideas, and results in innovative solutions being created faster, than any electronic communication method. I look forward to meeting some of you there (NAG have Booth #3131 – right at the heart of the main hall).

Creative Programming

Image
As I was being asked to write this blog post my brain awoke from it's number crunching stupor and I realised that I couldn't think of anything to write. The past few hours of sitting, programming in front of a computer seemed to have wiped out the artistic half of my mind. I know at least for myself that staring at a monitor seems to obliterate all of my thoughts until I've finished the click clacking and click clicking of my current task and I can finally put blessed ink to physical paper to plan my next step. So in order to re-enliven my thinking and my creativity, I sought a nice picture of a stunning scene. There's nothing like the beauty of creation to awaken something deep in your heart. As I was quickly sketching the above picture to get the other half of my brain warmed up to it's full creative potential, I started wondering about creativity in programming. Firstly there is the beauty of the written code itself, the ideas encapsulated within it and t…

Comparing HPC across China, USA and Europe

In my earlier blog post today on China announcing the world's faster supercomputer, I said I'd be back with more later on the comparisons with the USA, Europe and others. In this morning's blog, I made the point that the world's fastest supercomputer, in itself, is not world changing. But leading supercomputers, critically matched with appropriate expertise in programming and using them, togther with the vision to ensure use across basic research, industry and defence applications can indeed be strategically beneficial to a nation - including real economic impact.

There are plenty of reports and studies describing the strategic impact of HPC within a given organisation or at national levels (some are catalogued by IDC here), so let's take it as a premise for the following thoughts.

With this in mind, there are some comparisons to be made between the approaches to supercomputing across the USA, Europe and China.

The USA has long enjoyed near total dominance of the h…

Why does the China supercomputer matter to western governments?

There is a lot of fuss in the mainstream media (BBC, FT, CNET, even the Daily Mail!) the last few days about the world's fastest supercomputer being in China for the first time. And much ado on Twitter (me too - @hpcnotes).

But much of the mainstream reporting, twitter-fest, and blogging is missing the point I think. China deploying the world's fastest supercomputer is news (the fastest supercomputer has almost always been American for decades, with the occasional Japanese crown). But the machine alone is not the big news.

Imagine that China announced a new prototype passenger aircraft, half the cost of the latest Boeing or Airbus. It has 50% greater fuel efficiency too. And an order of magnitude greater predicted reliability statistics. That would be major news. Sure it uses a lot of US designed components too.

But what if China announced this new aircraft wasn't just a prototype. It was a commercially available product now. And they have the capacity to make lots of them…

Carbon Footprint

In these austere times I am always on the lookout for ways to reduce our expenditure without reducing our service to customers. NAG have just been awarded a matched funding grant from http://www.sustainableroutes.co.uk/ for £1000 to help us reduce our carbon footprint and we intend to use the grant (and some) to improve our video conferencing and cut down on some travelling costs and CO2 for meetings and training. Currently we are looking at some state of the art video conference equipment which we hope to start trialling with our Manchester office. There are some other useful tips and grants available on the Sustainable Routes website so I would recommend go taking a look.

Source-level debugging of Python in Emacs

Image
To help me track problems in Python code (yes, even sometimes in my own...) I usually rely on good-old print/trace debugging. Owing to Python's speed of interpretation and execution, this is a pretty convenient approach. Python does have its own interactive debugger though— pdb—for those odd occasions when it's desirable to poke about in a program while it's running. The debugging mode in Emacs even supports pdb by default, but there's a little snag: you probably don't have a pdb in your path, so M-x pdb will just fail. Solution? Add a pdb script to your path and make it executable
#!/bin/shpdb_path=`python -c "import pdb, sys; sys.stdout.write(pdb.__file__ + '\n')"`exec python ${pdb_path} "$@"

2010: A Retail Store Odyssey

Stanley Kubrick’s film “2001: A Space Odyssey” was the epic 1968 science fiction film that explored human evolution, technology and artificial intelligence with both realism (and surrealism) and remains one of the top films of all time. In it, two astronauts battle the computer HAL for control of their spaceship and for their lives while investigating a series of strange monoliths left from an earlier civilization. For many years and for many people, the film has been symbolic of our struggle to master, and not be mastered by, computers. Kubrick and his co-author Sir Arthur C. Clark were both brilliant and far ahead of their time. In many respects, they still are.


In 2010 we are certainly wrestling with computers that occasionally seem to get the better of us. In our world, computers are ubiquitous and the software behind them has a pervasive impact on our daily lives but hardly in the way Kubrick and Clark envisioned. Consider this absolutely mundane sequence of events, at least in t…

The Shoulders of Giants

Last week NAG held its 34th AGM and prior to the business part of the meeting we had the pleasure of hearing a lecture by Professor Nick Higham of the University of Manchester on the subject How and How Not to Compute the Exponential of a Matrix. The lecture was filmed and can be viewed here. Matrix functions are useful in a number of areas, in particular for describing the solutions of certain types of differential equations. NAG is working with Nick to include his algorithms in future marks of the Library.

One of the things I particularly enjoyed about Nick's talk was the way that he set the subject in its proper historical context, starting with the work of 19th Century mathematicians such as Sylvester and Cayley. Working at the cutting edge of computing technology its easy to forget how much of what we do depends on work that goes back hundreds of years. It wasn't always this way: Fermat, having claimed to have found a proof of what became known as his last theorem, added…

HTML5: Possible implications for technical documentation

A New Version of HTMLAs you may have heard, HTML5, a new version of HTML, the main markup language for the web is under development. This is the first version for quite some time, HTML4.01, the current version, was released in December, 1999. There was an XML version, XHTML 1.0, XHTML (Released in January 2000 and revised August 2002), but it did not contain any new features.Much of the buzz around HTML5 concentrates on new features such as video (as an alternative to using flash plugins etc) and canvas (a JavaScript API for 2 dimensional graphics), but in this article I want to look at the features that impact on the kind of technical, mathematical documents used by NAG.SVG and MathMLSVG is an XML format for describing scalar graphics, using a similar graphics model to PostScript or PDF, but using XML syntax and CSS for styling, just as for HTML. It has been around for some years, but HTML5, for the first time, specifies how it works in the context of HTML (rather than XML). In a rel…

Do you want ice with your supercomputer?

“Would you like ice with your drink?” It’s a common question of course. One that divides people – few will think “I don’t mind” – most have a firm preference one way or the other. There are people who hate ice with their drink and those who freak if there is none. National stereotypes have a role to play – in the USA the question is not always asked – it’s assumed you want ice with everything. In the UK, you often have to ask specifically to get ice.

Yet the role of ice in making our drinks chilled is misleading. I once had a discussion with a leading American member of the international HPC community about this. “No ice”, he was complaining as we headed out of a European country, “they had no ice for the drink”.

“I don’t get this obsession with ice”, I chipped in. “What?!” He looked at me as if I were mad. “Why do you like your coke warm?”

“Ah, but that’s just it”, I replied. “I hate warm drinks – I really like my coke chilled. But surely, in this modern world over a century after th…

Working on the ADVISE project

Image
For the past three and a half years, my colleagues and I have been working on ADVISE, a TSB-funded collaborative research project which has been developing a new toolkit for visualization and analysis. Besides NAG, the partners in the project were VSNi and the University of Leeds. VSNi have expertise in statistics, as implemented in their GenStat product, while Leeds have an international reputation for their work in visualization research. As for NAG, we've had some success with IRIS Explorer, a popular visualization toolkit which allows users to construct applications by connecting modules together via a visual programming interface.Figure 1. The ADVISE pipeline builder We retained that interface in ADVISE (see Figure 1) because it has proved to be a rather intuitive way to create, modify and interact with applications. Thus, in this figure, the user selects modules from the repository on the right and connects them together in the area on the left. The widgets for controll…

NAG at Quant Congress, New York

Image
Last month I was in New York with my sales colleagues Mike Modica and Rick Guido from NAG's US office, attending Quant Congress USA. This is a meeting which is devoted to the latest developments in financial derivatives, risk management and the associated use of numerical techniques; NAG has been associated with it for the past five years because our software has found extensive use in the quantitative analysis community. Each day of the conference program opened with a plenary session of two presentations, followed by around twenty talks divided across two streams. The meeting had nearly a hundred attendees, not all of whom attended both days (perhaps it's worth noting that, whilst the technical program undoubtedly had its own attractions and merits, the attendance might have been enhanced still further by the fact that the two days of the meeting saw heavy rain fall on the city - in the middle of an otherwise-sweltering week). NAG was one of five exhibitors at the meeting, a…

Why quality has always been essential for NAG’s own internal process (and a holiday).

Quality is one of the things that matters a lot to our users.

That is certainly true, but another reality is that the NAG Library could not exist at all without the checks and tests that result in the best quality software. The NAG Library is particularly complex and involves many different people doing very different types of task and activities. The Library is created from intricate actions including:

- Investigating lots of numerical problems and potential approaches and solutions
- Designing hundreds of methods for the various chapters
- Producing example code, data and plots
- Creating installer and user notes
- Writing the detailed documentation and descriptions of routines
- Building the multiple different processor and compiler implementations of the Library
- etc…

The best approach to quality is essential, in making the NAG Library, precisely because it is this multifaceted combination of numerical mathematics, software engineering and evolved process. If it were not for the…

What's in a Name?

Newcomers to NAG often ask about the names of NAG routines. Certainly the names appear strange and unfamiliar at first, but to many NAG users the name encapsulates the heritage of quality and precision offered by NAG.

Originally the NAG Libraries were created by contributors from U.K. academia. Some of these wrote in Algol 60 and others in Fortran. It was agreed to have two libraries of identical content, so that every contribution in one language had a counterpart in the other. Thus both communities were appeased. Early printed documentation included both language versions.

It was clearly desirable to be able to distinguish one language version of an algorithm from the same algorithm implemented in the alternate language and in general this is still desirable if a single program needs to call both instances of the algorithm.

Equally the early developers saw that, as new algorithms were developed, replacement routines would be required with perhaps different calling sequences, so t…

A day in the life of...

Image
... a Numerical Analyst: Wow, this could be fairly boring, but here goes... I arrive at my desk, slightly dishevelled from my daily cycle to work, whip my laptop out of the draw and unhibernate my old faithful friend. He and I have had some ups and downs over the past year but I am happy to say that we are still on good terms. Having gone through the usual email checking routine, I ssh (remotely logon) onto one of the local linux work servers, to pick up from where I finished the day before. And then it's on to slogging at the current routine, improving it's accuracy or speed, checking it works properly, documenting the code, screaming (under my breath) in frustration at the latest problem, or quickly writing a blog post to give my mind a break from the overflows and NaN's (think they are grandmas? Think again) of the numererical analyst's life. But it's not all slog. One of the highlights for me, is in the cracking of problems. The simple answers, and the no…

Hey, you sass that hoopy Fortran?

Image
We have to work with lots of different Fortran compilers at NAG. So far at Mark 22 of our Fortran Library we have built using products from six distinct sources (GNU, IBM, Intel®, NAG, PGI and Sun). The list of what we build with naturally changes from Mark to Mark as different vendors fall by the wayside or as commercial considerations preclude us from creating a particular Library implementation.

With the six vendors listed above we are blessed that the base Fortran coverage is Fortran 95. At previous Marks we still supported compilers covering only Fortran 77 (g77 and pgf77 for example). This forced us to jump through all manner of hoops so that we could do some of the really useful stuff from Fortran 90—like, woah, dynamically allocating memory—in a Fortran 77 way.

Some hoops still exist though, even with Fortran 90 code.


For example, we recently ran into the following:
PROGRAMstr_read! .. Implicit None Statement .. IMPLICITNONE! .. Parameters .. INTEGER, PARAMETER …

A Life Well-Lived: Erwin Ruppenthal (1958 – 2010)

There’s probably some unwritten rule that blogs are off-limits to memorials. If so, I’m happy to break it for our colleague Erwin (or Erv as he was known at home). He died early this week after a long struggle with brain cancer, at home, surrounded by his wife Heidi and sons Alex and Kurt. If we take a closer look at his life we can learn a lot about the value of work and about having a purpose in life greater than earning a paycheck.Erwin taught me a number of lessons about work and I’ll tell you about some of them so that things he valued might live on in each of us, but first let me tell you a little about him. He came to NAG in early August 1990 to work in IT at our US office. What I’m told was that he was an earnest, hardworking German citizen in his early 30’s who most would describe as shy and self-effacing. It was nine years later that I encountered him on my first day as his boss. He was a man of two countries and two cultures who embraced both. Following sports was a passion…

Time Machines and Supercomputers

I found a Linpack App for the iPhone last week. Nothing special, just a bit of five minute fun. It seems a 3G model achieves about 20 MFLOPS. [Note 1]

What's that got to do with time machines? Well it got me thinking "I wonder when 20 MFLOPS was the performance of a leading edge supercomputer?" Actually, it was before the start of the Top500 list (1993), so finding out was beyond the research I was prepared to do for this blog.

So I thought instead about the first supercomputer I used in anger. As soon as I name it, if anyone is still reading this waffle, you will immediately fall into two camps - those who think I'm too young to be nostalgic about old supercomputers yet - and those who think I'm too old to be talking about modern supercomoputers :-).

It was a Cray T3D.

You're still waiting for the time machine bit ... hang on in there.

My application on that T3D sustained about 25 GFLOPS. Which is about the same as a high end PC of recent years. What this me…

Fantasy Football – a classic Portfolio Optimisation problem

Image
England out the World Cup, German colleagues, customers, collaborators (actually not just Germans, Americans, Scots,…) and so called friends all sending me mocking e-mails and texts.
“I hear OXO are making a new product. The packaging is white with a red cross and they're calling it the laughing stock.”
How could I channel my frustration? NAG Blog to the rescue.
In my last blog I promised to reflect on my early career at NAG which isn’t that long ago compared with some of colleagues.
My commute to the NAG office is longer than I’d like, but there is a positive side…
·Valuable thinking time ·Audio books - I recently enjoyed “No One Would Listen” by Harry Markopolos. The exclusive story of the Harry Markopolos–lead investigation into Bernie Madoff and his $65 billion Ponzi scheme.
Back to “thinking time,” on one commute into Oxford I’d been pondering the previous day’s internal training course. I was recalling my senior technical colleague’s wise words. That wise colleague was David …