I'm at the Vis on the web workshop talking about how I use this blog, how I get my students to use it, and such (like @dr_tj on twitter). Go here!

by R.F. Erbacher 
 
This paper talks about analyzing dynamically network infrastructure in conjunction with information on system load, remote access and network topology. 
 
Glyphs have been made use of to represent nodes in a network. These glyphs represent routers and servers. Individual users are not a focus of this paper. Visualizing loads on individual routers and servers and network infrastructure is the focus. The glyph borders are used to represent the capacity of the router. Thicker borders indicate 100MBPS and thinner border indicates 10MBPS capacity. The glyphs are interconnected using directed lines, where the directions indicate the flow of traffic. Hashes are used on the top of the directed lines to indicate the type of information (How the hashes encode the type of information is not provided). The lines extending from the perimeter of a glyph indicate the number of users on the system. Each line indicates 10 users. The glyph interiors are shaded as rings. The thickness of the interior ring indicates the load on that particular node. 

Year: 2005 
Summary written by Chris Lewis 
 
This paper describes 3 methods of displaying network intrusion data in 3D space. The researchers used 2 data sources. One was NIDS Snort data and the other source of data was raw network activity. The first method is using an "island". The island is a big circle that represents ports in a range of 1 to 65535 and there are 3 big invisible circles inside the island that represent 1 to 1024, the most popular ports. Any hits are drawn that look like trees on the island. The trees are drawn at different heights and have leaves that shoot off at different angles and have different colored leaves depending on the state of differing variables. The second method is an alert plot. The best that I can understand about this view is that the middle cube represents alerts and the left is a cube that represents internal IP addresses and the right cube represents external IP addresses. There are lines that connect two cubes to each other. The third method is a 3D bar graph that has x, y, and z planes that represent time and port numbers. The planes show alerts and the network data inside those alerts at the appropriate time. Each of these views shows a unique way of showing intrusion alerts. 

Year: 2006 
Summary written by Chris Lewis 
 
The author does a good job in documenting the history of techniques used to display network data and visualization methods that show various kinds of attacks. The important element that they stress is time. Sometimes it is difficult to see something like a worm on a network. If the display only shows data for one specific point in time, then the activity might go unnoticed. However, if the data is displayed that shows the traffic from a starting point in time to an ending point, a pattern should be visible. The developers of this tool tried to make a system that did not need an expert to understand. They got security experts to develop an ontology that identified what types of patterns to look for in network data that would signify a possible attack. They took this information and used it to display possible warnings to users that could be nieve on the subject. I think what is important here is that the visualizations should be clear in what they represent and it is important to keep track of the network data over time to be able to look for patterns. 

Year: 2004 
Summary written by Chris Lewis 
 
This article gives a good idea of how to show information from SNORT files. The data is represented and organized by destination and source. Visual lines can be drawn in between the two to make it easier to see. When the shapes are drawn in the appropriate time placement, an admin can easily see patterns that might have gone unnoticed by viewing text. Each type of protocol is represented by a different shape. The service identified as being used also has its own shape. Green is used for preprocessor events, red, yellow, and blue are used for priority alert 1, 2, and 3.  

Year: 2002 
Summary written by Chris Lewis 
This paper has many good ideas on how to show a server as a glyph and the attempted and successful connections around it with lines and other glyphs. This system has ways of identifying alerts. The author also makes note of how it is important to design your glyph so that it represents the data in the database. Each line and each shape needs to represent something. You also need an idea of when the data is normal and when the display would need to be changed to represent a change in the value.  

by K.C. Cox, S.G. Eick, T. He 
 
An explination about Seenet 3D visualization system, which is an improvement over Seenet 2D (as explained by me before in R.A. Becker paper). As opposed to the 2D geometric maps used in Seenet 2D this method uses 3D globe to visualize the network information. Unlike Seenet 2D, which uses straight lines, this paper utilizes 3D arcs to represent the links between nodes. 
 
The paper explains about 3 views. 
 
1. Global network view: In this view the nodes are countries represented at their capital using 3D box glyphs extending particular to the globe. 3D arcs are used to link countries using the traffic as statistic. The glyph is colored and sized to represent the total packet count emanating from the country. The arcs are color coded to indicate the traffic. Redder arc is used color larger traffics (must be temperature color scale). The time frame is indicated by light position, where the light indicates the light from the sun and the position of sun is taken as metaphor to indicate the time. The globe is translucent to see through the other side of the globe. Various interactions like scaling and rotation are possible, However the interactions are restricted to keep the globe always in focus. The routing of arcs can also be altered by the user, so that they can pass through the center of the globe. Filtering of arcs is permitted to focus on a certain part of the globe. The authors claim that the arcs usage reduces the clutter that is seen in traditional link maps. 
 
2nd view: Arc Map: This is similar to the Global network view, except that instead of using a 3D globe for positioning the nodes, the nodes are placed on a flat 2D world map. The links remain arcs and the display 3D. The advantage here is it can be used on any map and for any scale. One can focus on within country traffic only leaving other countries from the view. The large traffic flow arcs can be rendered using highest height. Arcs can be rendered translucent which solves the occlusion problem partially. 
 
3rd view: Drill down view: To view the traffic patterns for a particular node this particular view is suitable. This view shows details on demand for a particular node. Information like which code has the highest interest in a particular node and which node has the least amount of interest is seen from this view. The example given in paper shows a spoke like 2D view to see interesting traffic patterns between US and other countries. All countries are arranged in alphabetical order around a circle and the nodes are sized to represent the link traffic. This could be altered and nodes can be presented in geographical order. Another refinement given in paper includes presenting the nodes on a helix or a sphere and links indicating traffic. 
 
The whole system was implemented using C++ and Vz library. The code was 5000 lines long and appears to be published long back, since the frame rates claimed by the paper were less than 2 per second and were run on Windows 95 and on Windows NT systems. 

by S.G. Eick and G.J. Wills 
 
This paper talks about drawbacks in other papers in terms of placement of nodes, and propose some techniques for the placement of nodes for graphs with hierarchies. 
 
First they propose techniques for an Email network. They first provide the overview of the network in a graph layout. Here the graph node sizes are first calculated. The area of each node is proportional to the messages sent or received. They are colored based on their roles (example in the paper colors red for clerical duties, blue for technical etc.). The links show email connection between individuals. The links are colored using common heat scale. The nodes are placed by minimizing SIGMA(w_ij - 1/d_ij)^2 function, where d_ij is link length and w_ij is the size of the link. Though it is not fully clear for me about the minimization function.  
 
Interaction is provided to change different visualization parameters and technique similar to scatter plot brushing is used to link different displays used for visualization.  
 
The different displays used for visualization are the main network view (explained in previous paragraph), a time slider view showing the period under scrutiny, and a histogram view of link strength (horizontal axis) vs number of links (verical axis). By appropriately placing nodes and properly navigating the author explains how communication patterns can be found by looking at the node displacements or group displacements. 
 
Second they explain the visualization of software engineering, where a hierarchy of files and interdependency between files is investigated. Any file change has to happen after a modification request (MR). A link between two files A and B is thus defined as the sum of MRs which change both files. Using this definition three types of links are generated. 
1. A file-file link between files in a module, 2. file-module link which is the sum of links between a file and all files in other module and 3. Module-Module link which is the sum of links between files in one module and files in other module.  
(The three types of links and their construction is a little confusing) 
 
In order to obtain a overview only 1% of strongest module-module links are visualized. This reduces the visual clutter. The node placement is not described however I am assuming the same strategy as used for the email network. The visualization used three visual cues for node size and color. The width represents the number of files in a module and height the number of changes made to a module. Color emphasizes the aspect ratio. 

by R.A. Becker, S.G. Eick and A.R. Wilks 
 
In this paper three methods to display network data and a tool seenet using these techniques is described 
 
The three methods include 
 
Link Map: This is used to mainly depict the link statistics between different nodes. The nodes are represented on a ggeographical map. The links are represented using partial lines between nodes. Different cues like line thickness and color can be used to depict the link statistics. In order to avoid visual clutter only the exception data is shown. Even then there might be visual clutter, so instead of showing full line segments only a partial line starting from the originating node is shown. The main disadvantage of this method is the visual clutter that happens in the graphical display. Seenet can be used to dynamically select the current and anchor node and displaying the traffic between these nodes. It can also be used to dynamically adjust various parameters and also to animate between different time frames. 
 
Traffic Matrix: Like the link map the traffic matrix displays the link statistics between nodes, but unlike the Linkmap the traffic matrix displays the nodes as a matrix. The nodes are represented along horizontal rows and vertical columns. The traffic is represented using small squares (or glyphs). The different time frames are depicted by animation. There is no visual clutter in traffic matrix, However it cannot be applied to large networks (from my perspective). The traffic matrix can also be confusing if the nodes are not depicted on the rows and columns properly. The paper has shown an example where the nodes are presented in east-west order. Seenet can be used to adjust dynamic sliders for the presentation of data. 
 
Nodemap: Unlike the previous two techniques this technique displays the node statistics. The statistics provided in the paper involves the incoming and outgoing traffic. The nodal data is represented using glyphs. The example given in the paper depicts incoming traffic size as the width of the glyph and the outgoing traffic size using height of the glyph. The Seenet can be used to control statistics like the glyph size and color etc. 

For Tree-Maps the internal nodes information is not present. However following the paper, the level information can be misleading if some internal node has only one child. Note that this information may be wrong but can only be verified once I implement it. 
 
This happens because the drawing direction is changed alternatively from vertical to horizontal, and when a node has a single child, since the internal node information is presented via its children, this information is lost. 

Reference: http://ieeexplore.ieee.org.proxy.library.msstate.edu/search/srchabstract.jsp?arnumber=1249009&isnumber=27965&punumber=8837&k2dockey=1249009@ieeecnfs&query=((jankun-kelly)%3Cin%3Emetadata+)&pos=7&access=no 
 
Paper: MoireGraphs: radial focus+context visualization and interaction for graphs with visual nodes 
 
Moire graphs are visual node graphs which display spanning tree using radial focus+context technique. The nodes are area nodes presented in a polar 2D space. The nodes are interconnected or linked using lines. The layout of the graph is radial. Node information is augmented via texturing the node area with information regarding the node (Like screenshot of the document or an image).  
 
Advantages include presentation of visual information along with the node, aesthetically pleasing and interactive graphs, animated interactions to avoid abrupt transition between states. 
 
Disadvantages include limited size of the graphs, visual occlusion which can happen when the angular spread for two adjacent siblings is not enough to accomodate their node size. In order to limit the size of the graphs, the nodes can be intentionally removed from the graph. 

Reference: http://portal.acm.org.proxy.library.msstate.edu/citation.cfm?id=949607.949654&coll=portal&dl=ACM&CFID=28483634&CFTOKEN=38616739 
 
Vis 91 proceedings, Tree-Maps: a space-filling approach to the visualization of hierarchical information structures. 
 
An interactive visualization method for presenting hierarchical information. Based on the paper I am highlighting some of the important information that treemaps present. 
 
Tree-maps are suitable for Hierarchical data. The presentation is done on a 2 Dimensional space, with no links in between the nodes. How ever the nodes were presented as rectangles. Different augmentations to these nodes such as texturing and coloring can be done on top of these nodes to sub-categorize these nodes. The primary categorization is on the hierarchical level, which is intuitive in tree-maps. 
 
Advantages of Tree-maps include effective utilization of screen space (100%), interactive control and highlighting more important nodes (by assigning high weights).  
 
Disadvantages I could gather from the paper (though I did not see any specific mentioning in the paper) is that some internal nodes may be occluded from the view. This could happen in 2 cases. One where the size of a node is 0. The other case is when the node has a single child. 

SCons is a "next generation" build tool written in Python. It is a make replacement, and much more. It will create PDFs from latex, compile C, C++, JAVA, MS Visual Studio Projects, and anything else you can think of. This would be a wonderful tool for cross-platform development, or development of a very heterogeneous system. 

I did a little more 'thorough' testing on the differences between dict.get(key) and dict[key]. I tested for both keys that do and do not exist, and I seem to have different results than what I did last time =( 
 
It seems dict[key] is about twice as fast as dict.get(key) when the key does exist in the dict. On the other hand, dict[key] + exception handling is twice as slow as dict.get(key) when key is not in dict. Here's the extra twist: checking if the key is in dict and then getting the appropriate value is only slightly slower than simply table[key]. This is something like: 

if key in dict: 
    value = dict[key] 
else: 
    value = None

or, more elegantly: 

value = None if key not in dict else dict[key]

 
This method is fractionally slower than dict[key] when key is in the table, but it's considerably faster when key is not in the table. Also, this method is consistantly faster than dict.get(key) in both cases. 
 
The script can be found [here] 
The output for 100,000 iterations of each case can be found [here] 
 
The output shows the name of each test case followed by the commands executed in the test. Total runtime for each follow each. 
 
I may have done the cases a little naively, but I still think it shows some powerful differences. 

Installing GLEW 
Download the GLEW source from [here] 
Run the following command in the glew folder: 

> GLEW_DEST=/usr/local sudo make install

 
Installing GLEWpy 
Download the GLEWpy source from [here] 
Replace the setup.py in the glewpy folder with the modified setup.py from [here] 
Run the following command in the glewpy folder: 

> sudo python setup.py install

 

This is useful for 'faster' logins, or programs that use SSH and don't support a login (ex: SCPlugin SVN GUI for OSX) 
 
On local machine, create and upload the key: 

> ssh-keygen -t dsa 
** overwrite if you need to ** 
** enter empty passphrase (is this safe?) ** 
> scp ~/.ssh/id_dsa.pub username@server

 
Login to remote server and do: 

> cat id_dsa.pub >> .ssh/authorized_keys 
> rm id_dsa.pub

 
I did this a while back, but never took notes. *sigh* 

Very impressive demo of a large scale multi-touch display wall. 

PyGPU 

Here's an image resizing technique presented at Siggraph '07. 
 
http://www.youtube.com/watch?v=6NcIJXTlugc 

On my laptop I have a fork of the keyboard/mouse input code that can be used to take an arbitrary number of computers and control them with one keyboard & mouse. This post is just a reminder to myself to add it to svn. I know there are 3rd party applications for this, but who knows, it could still be useful. 

Here's a collection of Python 3D software. I'm tempted to experiment with some of these, but there just aren't enough hours in a day. 

Ketan and Swati have just had a new addition to their family. Congratulations and Best Wishes!! 

This list enumerates the various (3rd party) software used on the cluster. It will be amended as I think of more. 
 
 
Python 
PyOpenGL 
Python Imaging Library 
Glew 
Pyrex 
GlewPy 
Python Cryptography Toolkit 
PyGame 
 
I also have some installation/configuration notes in the works. I will add these after I get them organized. 

A lucid explaination of the purpose and logic behind indentation in Python.

def GetWXBitmap(myImage): 
    image = apply(wx.EmptyImage, myImage.size) 
    image.SetData(myImage.convert("RGB").tostring()) 
    # if the image has an alpha channel, 
    # you can set it with this line: 
    myImage = myImage.convert("RGBA") 
    image.SetAlphaData(myImage.tostring()[3::4] ) 
    return image.ConvertToBitmap()

 
FROM: 
http://wiki.wxpython.org/WorkingWithImages#head-7aa43a4a1e066fd28640ce86066ba0617afe2a8b 
 
Using wx.StaticBitmap (widget on the window to show the bitmap): 
http://wiki.wxpython.org/wxStaticBitmap?highlight=%28Bitmap%29