Introducing clusterfun

In computer vision research, ML model predictions are related to images. You often want to plot these predictions to get a sense of how well your model is doing on your data. For example, in fraud detection, you might want to plot a histogram of your model scores for both your genuine and fraudulent data, to see if and how these two distributions overlap. In such a context, it can be useful to be able to quickly inspect the images related to the scores. For example, you might want to have a look at the images that were not predicted correctly, i.e. fraudulent images that overlap with the genuine distribution, and genuine images that overlap with the fraudulent distribution. However, most common plotting libraries don't allow you to do that. You'd end up first creating a histogram like this:

What do the genuine images in the fraudulent distribution look like? What do the genuine images close to zero look like? How are they different compared to the images in the middle of the distribution?

After creating this plot, you'd have to manually single out the images that you want to inspect in a separate process. In the most extreme, laborious cases this is done by saving images in a directory and inspecting them that way. Some libraries, such as bokeh make this a little bit easier, but the UX is not mainly made for inspecting images - you can hover over images, but that's about it. However, understanding data is not only about looking at an image, one at a time, at a glance. Understanding data means being able to zoom in on an image, view multiple images next to each other and see aggregate statistics of such a selection. Clusterfun allows you to do that. Let's dive into each of these components:

• 1. Hover to inspect

  In a plot, you can quickly hover over images. Hovering does not only show you the image, but also the metadata associated with the image. For example, you can see a model score, the image location, and any other metadata you added to the dataframe used to create the plot.

• 2. Select to inspect

  Hovering over images is fine for a quick glance at the data, but looking at multiple items at the same time can give you a much better sense of your data distribution. This allows you to e.g. quickly spot similarities between images that are wrongly predicted.

• 3. Statistics of a selection

  Once you have a selection of data points, it can be helpful to see aggregate statistics of a selection. For example, in a face recognition context, you might want to see the country of origin of the faces in your selection. This can help you to quickly spot biases in your data. In the example below, you can see how we can dive into particular subsets of a dataset. Here, we see a group of paintings that look very similar, but are still from different painters. Picasso and Braque made a few very similar paintings!

Q&A

• How is this different from model analysis platforms such as Scale, Encord, etc.?

  • Scale and Encord offer big, feature-rich solutions for model analysis. Clusterfun is more lightweight and should allow you to review data within a minute given a pandas dataframe with, per image, a location (S3 or local) and a score.
  • Moreover, these platforms are not mainly for plotting. They do allow you to inspect subsets of your data (e.g. wrong predictions), but don't offer the flexibility of a plotting library where you can determine what the x and y-axis represent. The additional benefit of a plot is that it allows you to easily see how close a data point is to another data point.
  • All of clusterfun is open source.

Try clusterfun