CSL - Contextual Snapshot Library

CSL is a library implementing contextual snapshots. This page contains links to download the CSL, as well as a tutorial on how to use it. The tutorial describes how to transform this:

into this:

using the CSL.

The authors of this work are Peter Mindek, Meister Eduard Gröller, and Stefan Bruckner.

Tutorial

To demonstrate how to integrate the CSL with an existing visualization system, we use a simple OpenGL application and explain, step-by-step, what needs to be done in order to integrate the CSL into it. The application, as well as the version with the integrated CSL, are available in the Download section.

Sample application

The sample application used in this tutorial renders a graphics effect using OpenGL. Namely, it ray-marches a signed distance function of several morphing objects. This is implemented as a GLSL shader which is used for rendering a single quad covering the whole screen. The application is shown in Figure 1.

Click to enlarge

Figure 1. A simple OpenGL application.

It is possible to select the rendering style from the palette on the left bottom (Figure 2). The user can either click on the icon of the desired style, or they can use the slider to smoothly change between the rendering styles.

Click to enlarge

Figure 2. The rendering style palette and the rendering style slider.

Figure 3 shows the object rendered with different rendering styles.

Click to enlarge

Figure 3. Different rendering styles.

Integrating the CSL

We are going to extend the application with a possibility to select regions on the screen. Multiple regions can be selected for each rendering style, and the user will be able to restore the selections made with each style. The selected regions will be rendered with a different style than the rest of the image.

To implement this functionality, we use the CSL. This will demonstrate how to integrate the CSL with an existing application.

Linking and initializing the library

First, we need to include the sources of the CSL with our project. The CSL uses Qt and OpenGL. As our application already uses both of these libraries, we do not need to link any additional dependencies. Important: the CSL needs EXT_shader_image_load_store extension for its function.

The CSL contains two classes which implement its API: Presentation and PRSShaderEnhancer. We start with including them into the renderer class, and creating instances of them:

renderer.h

#include "csl/presentation.h"
#include "csl/prsshaderenhancer.h"

...

protected:
  Presentation *prs;
  PRSShaderEnhancer *se;

renderer.cpp

void Renderer::initializeGL()
{
  prs = new Presentation(width(), height(), this);
  se = new PRSShaderEnhancer();
  
  prs->setAnchors2D(true);
  prs->setSelectionsVisible(true);
  prs->setAutoActivateSelections(true);
  prs->setAutoActivateSnapshots(true);
  prs->setEasingCurve(QEasingCurve::InOutQuad);
  ...
}

We create the instances in the initializeGL(), since we need the OpenGL context for creating an instance of the Presentation class. Immediately after creating the instances, we can set the properties of the CSL. The important ones are set by these functions:

Afterwards, we create QGLShaderProgram instances representing the shaders used by the CSL. The CSL does not load them automatically so that the user can choose their own shaders. We load four different selection shaders to show how easily they can be interchanged. Also, we connect parametersChanged signal to updateParameters slot. This will be explained later. Finally, we enable OpenGL texturing, since we are going to need it for the rendering of the CSL's output.

renderer.h

protected:
  QGLShaderProgram *currentSelectionShader;
  QGLShaderProgram *selectionShader0;
  QGLShaderProgram *selectionShader1;
  QGLShaderProgram *selectionShader2;
  QGLShaderProgram *selectionDisplayShader;

renderer.cpp

selectionDisplayShader = Util::createShader("csl/shaders/vs.txt",
                                            "csl/shaders/selection_fs1.txt");

selectionShader0 = Util::createShader("csl/shaders/vs.txt",
                                      "csl/shaders/stroke_fs0.txt");
selectionShader1 = Util::createShader("csl/shaders/vs.txt",
                                      "csl/shaders/stroke_fs1.txt");
selectionShader2 = Util::createShader("csl/shaders/vs.txt",
                                      "csl/shaders/stroke_fs2.txt");
selectionShader3 = Util::createShader("csl/shaders/vs.txt",
                                      "csl/shaders/stroke_fs3.txt");

currentSelectionShader = this->selectionShader0;

prs->addSelectionDisplayShader(selectionDisplayShader);

connect(prs, SIGNAL(parametersChanged(QList<QVariant>)),
        this, SLOT(updateParameters(QList<QVariant>)));

glEnable(GL_TEXTURE_2D);

Rendering the scene

After the CSL is set up, we need to modify the rendering pipeline of the application. Instead of rendering to the framebuffer, we will render the scene to a texture using a framebuffer object. We will pass this texture to the CSL and then render a quad covering the whole screen textured with an output texture from the CSL.

renderer.h

protected:
  QGLFramebufferObject *fbo;

renderer.cpp

void Renderer::initializeGL()
{
  fbo = 0;
}

...

void Renderer::resizeGL(int w, int h)
{
  prs->resizeFbo(w, h);

  if (fbo != 0)
  {
    delete fbo;
  }
  if (w > 0 && h > 0)
  {
    fbo = new QGLFramebufferObject(w, h, QGLFramebufferObject::CombinedDepthStencil);
    prs->setInputFbo(fbo);
  }
  else
  {
    fbo = 0;
  }
  ...
}

...

void Renderer::paintGL()
{
  ...
  fbo->bind();

//original rendering
  shader->bind();
  glRectf(-1,-1,1,1);
  shader->release();
//end of original rendering

  fbo->release();

  prs->setCurrentParameter(shading);
  prs->render();

  glLoadIdentity();
  glEnable(GL_TEXTURE_2D);
  glBindTexture(GL_TEXTURE_2D, fbo->texture());
  glBegin(GL_QUADS);
    glTexCoord2f(0, 0);
    glVertex2f(-1, -1);
    glTexCoord2f(1, 0);
    glVertex2f(1, -1);
    glTexCoord2f(1, 1);
    glVertex2f(1, 1);
    glTexCoord2f(0, 1);
    glVertex2f(-1, 1);
  glEnd();
}

We create a new FBO whenever the windows resizes, and we set it as an input FBO for the CSL. We also call resizeFbo() function of the Presentation class, which resizes the output FBO used by the CSL.

In the rendering method, we bind the FBO before the actual rendering, and release it afterwards. We process it by the CSL, and render the output as a textured quad covering the whole screen. What is displayed now is the enhanced rendering containing the CSL annotations.

Before the CSL rendering, we set current parameters values to the CSL using the setCurrentParameter(QVariant) method. This informs the CSL about the current context. The method takes QVariant parameter, so we can use a parameter of any type supported by QVariant (qreal including).

In our case, the only parameter defining the current context is the shading parameter specifying the current rendering style. At this point, we could also set a list of multiple parameter values to the CSL using the setCurrentParameters(QList<QVariant>) method and supplying it with a list of the parameter values.

Interaction

We are going to implement two types of interaction: creating a selection, and activating anchors representing individual contextual snapshots.

renderer.cpp

void Renderer::mousePressEvent(QMouseEvent *e)
{
  ...
  if (!guiInteraction)
  {
    if (e->button() == Qt::LeftButton)
    {
      QList<QString> parameterNames;
      parameterNames << "shading";
      QList<QVariant> parameterValues;
      parameterValues << shading;

      QVector3D position = QVector3D(75.0f, shading * 50.0f + 30.0f, 0.0f);
      prs->createSelection(position, parameterValues, parameterNames);

      drag = DRAG_SELECTION;
    }
    else if (e->button() == Qt::RightButton)
    {
      int selectedAnchor = prs->anchorOnPosition(e->pos().x(),
                                                 prs->invertY(e->pos()).y());
      if (selectedAnchor >= 0)
      {
        prs->animateToAnchorParameters(selectedAnchor);
      }
    }
  }
}

void Renderer::mouseMoveEvent(QMouseEvent *e)
{
  ...
  if (drag == DRAG_SELECTION)
  {
    prs->getSketcher()->addPoint(SelectionPoint(prs->invertY(e->posF())));
    prs->getSketcher()->renderLastSelection(currentSelectionShader);
  }
  else
    prs->showAnchorThumbnail(prs->anchorOnPosition(e->pos().x(),
                                                   prs->invertY(e->pos()).y()));
}

void Renderer::mouseReleaseEvent(QMouseEvent *e)
{
  if (drag == DRAG_SELECTION)
  {
    prs->beginSelection();
    prs->selectAllSelections();
    prs->endSelection();
  }

  drag = DRAG_NOTHING;
}

The interaction is implemented in the mouse event handlers. The guiInteraction tells us, whether the given event has been handled as an interaction with the rendering style controls. If it was not, we create a new selection on left mouse button, and activate an anchor on right mouse button. For creating the selection, we pass parameter names and their values (in our case only the shading parameter) to the CSL, and position an anchor on respective position next to the rendering style slider. The position of the anchor will tell us what was the context for creating selections stored within the respective contextual snapshot.

The createSelection method returns Selection * pointer to the created selection. We could use it to add an embedded visualization to this selection using the addIntegratedView(Selection *, QString). The second parameter is the ID of the desired embedded visualization, which has to be previously registered with the registerIntegratedView method. An embedded visualization can be any Qt widget (QWidget) and it is possible to add multiple embedded visualizations to each selection.

On mouse move event, if we are currently creating a selection, a new point is added to the last selection, which is then transformed into a selection mask. The transformation is performed by the shader currentSelectionShader. We can simply change this shader for any of the four we have previously loaded. If we are not currently creating a selection, we check whether the mouse hovers over any anchor, and if so, we display its thumbnail.

On mouse release event, if we were creating a selection, we activate all selections of the current contextual snapshots, so that the newly created selection will be available in our ray-marching shader.

The method animateToAnchorParameters interpolates between current parameter values and the parameter values of the respective contextual snapshot. During the interpolation, it continously emits the parametersChanged signal, which we have connected to the updateParameters slot:

renderer.h

public slots:
  void updateParameters(QList<QVariant> parameters);

renderer.cpp

void Renderer::updateParameters(QList<QVariant> parameters)
{
  if (parameters.size() > 0)
  {
    setShading(parameters[0].toReal(), true);
    update();
  }
}

...

void Renderer::setShading(GLfloat shading, bool interpolating)
{
  this->shading = shading;
  shader->bind();
  shader->setUniformValue("shading", this->shading);
  shader->release();

  if (!interpolating)
    prs->viewChanged();
{

When the parameter values are changed by the CSL, they are updated in the application using the setShading function. In this function, we call method viewChanged();. This tells the CSL, that the parameters have changed, and if there is an active contextual snapshot, it should be deactivated because it is no longer valid since the context have changed. However, we do not call this function when an anchor is activated and we interpolate the parameters, since this would deactivate the anchor to which we are interpolating.

Custom parameter types

The CSL uses QVariant to store parameter values in the contextual snapshots. This allows arbitrary parameter types, to be used. The CSL provides functionality to interpolate parameter values, as well as to match them in order to activate contextual snapshots with parameter settings matching the current parameter settings of the system. Therefore, the interpolation and the matching functions need to be provided for individual types. The CSL provides these function for the following types:

It is possible to use different parameter types. In this case, the interpolation and the matching for this type have to be implemented. It is also possible to change the default implementation of the interpolation and the matching functions.

The interpolation function is implemented by the static method PRSUtil::mix(QVariant a0, QVariant a1, qreal x). The method takes the two parameter values to be interpolated (a0, a1) and the interpolation parameter x. The method retruns the interpolated value of the two parameters. The default implementation contains a switch statement, where a respective interpolation function is evaluated based on the type of a0.

The matching function is implemented by the static method bool PRSUtil::equal(QVariant a0, QVariant a1, QString parameterName). The parameters of the method are the two parameter values to be compared, and their name. By checking the provided parameter name, it is possible to implement different matching function for each individual parameter, even if they are of the same type. The function returns true, if the parameter values are matching. Default implementation evaluates two parameter values to be matching, if they are exactly equal:

prsutil.cpp

bool PRSUtil::equal(QVariant a0, QVariant a1, QString parameterName)
{
  if (a0.type() != a1.type())
    return false;

  bool result = false;

  switch (a0.type())
  {
  default:
  case QVariant::Double:
  case QVariant::Int:
  case QVariant::Bool:
  case QVariant::Vector2D:
  case QVariant::Vector3D:
  case QVariant::Vector4D:
  case QVariant::Matrix4x4:
  case QVariant::Rect:
    result = a0 == a1;
    break;
  }

  return result;
}

It is possible to implement parameter matching, where a certain different is tollerated to still evaluate two values to be match. This is useful when we want a contextual snapshot to be activated even when its stored parameter settings are a little bit different from the current parameter settrings of the visualization system.

In the sample application, we allow the shading parameter to be different from the value stored in the contextual snapshot by 0.1 for the contextual snapshot to still be automatically activated. This is achieved by modifying the respective case statement in the PRSUtil::equal() method. All other parameters of the type Double would have to be equal in order for the contextual snapshot to be automatically activated:

prsutil.cpp

case QVariant::Double:
  if (parameterName == "shading")
  {
    result = qAbs(a0.toDouble() - a1.toDouble()) < 0.1;
  }
  else
  {
    result = a0 == a1;
  }
  break;

Shader enhancement

So far, we can create image-space selections in different contexts (rendering styles), and switch between contexts to see respective selections. Now we would like to use the selections in our ray-marching shader. For this we use the PRSShaderEnhancer.

Instead of just compilig the source code our ray-marching shader, we first process it with the PRSShaderEnhancer:

renderer.cpp

QString enhancedFSource = se->enhance(QString(fsSourceArray), "330");
shader = Util::createShader(":/shaders/vs.txt", enhancedFSource, true, false);
se->init(shader);

fsSourceArray is the source code of the shader. The enhance() method inserts additional functionality into it. It will allow us to easily access all selection masks in the shader, and use them to modify the rendering. After the shader is compiled, we initialize the PRSShaderEnhancer.

In the rendering loop, we need to pass the selection masks to our shader. To do this, we call following method after the shader is bound:

renderer.cpp

se->prepareShader(1, prs->getSelectionMasksArray(), prs->getActiveSelectionsCount());

The first parameter specified the texture unit used for passing the selection masks to the shader. They are stored within a texture array, so only one unit is needed for all selections.

Using the PRSShaderEnhancer allows as to easily query the DOI (degree of interest) of each fragment in a GLSL fragment shader. The DOI is a float number in the interval [0,1] which specifies to what degree is the given fragment selected. This is to enable so called smooth brushing, or partial selections. To query the DOI of a fragment, we can call the prsFragmentDOI() function:

fs.txt

col = mix(col, selectionCol, prsFragmentDOI());

col is the shaded color of a fragment of the rendered object, selectionCol is the color of the fragment in a distinct rendering style (using diagonal red and yellow stripes). We use the fragment's DOI to mix these two colors. The result is a different rendering style inside of the selected regions, as illustrated in Figure 4.

Click to enlarge

Figure 4. Various image-space selections.

Figure 4 shows three different types of selections, from which we can choose: a circular, a rectangular, and a free-hand lasso selection. All of these are binary selections (each point in the image is either fully selected or not selected at all). The fourth selection type is a fuzzy lasso, where the selection mask is blurred between the begining and the end of the user-made stroke. This allows users to create gradients between selected and not selected areas. The fuzzy lasso selection is shown in Figure 5. The user can choose which type of selection to use by pressing numbers 1 - 4 on the keyboard.

Click to enlarge

Figure 5. Fuzzy lasso selection.

Conclusion

This tutorial describes how to integrate the CSL into an existing application. It does not serve as a full documentation of the CSL, as it does not describe all of its functions. Please feel free to send any questions and comments to mindek@cg.tuwien.ac.at. The source code of the CSL, the example application, as well as the version enhanced with the CSL are available in the Download section. There is also another use case of the CSL - a manuscript analysis tool created by extending a simple manuscript reader application with the CSL. All downloads contain source code as well as Windows binaries.

Download

CSL v0.9.3

sources of the Contextual Snapshot Library (GitHub repository)

Demo Application

demo application without CSL

Demo Application + CSL

demo application with integrated CSL

Manuscript reader

a use case of the CSL

Requirements

The CSL as well as the demo applications are written in C++ and GLSL. The source code requires Qt 4.x.

The software uses OpenGL. The shaders, except for the demo application without CSL, require EXT_shader_image_load_store extension, which means OpenGL 3.0 and GLSL 1.30 are required.

License

The software available on this page is released under the LGPL license.