All this pattern matching is producing lots of numbers and I don't want to examine long lists of matching scores anymore. I came up with the idea to build a viewer that would show me the similarity of a large number of sources in an attractive way.
Let me show you the result first and afterwards tell you how it was done. I took the same set of graphical icons as before and matched them with each other. I'm afraid the resolution of the uploaded video is worse than I expected, but hopefully your eyes can manage.
The icons are firstly placed in a random way and then start to mingle and cluster with similar icons. Notice for example how the red hearts start clustering almost immediately. Then the two red alarm clocks on the left edge decide they want to meet desperately. Finally a blue ID pass starts moving up from the bottom to get intimate with its best friend. See how they move in groups when they have found their 'best friends'.
I can tell you, for me it is so much more pleasing to see those icons crawl around instead of analysing all the matching scores in Excel!
Nerd-mode on:
The underlying technique is based on a force-directed graph of interconnected nodes. Each node is connected to every other node by a spring with a natural length based on the matching score of the two nodes. Similar nodes therefore have a tendency to stay close.
Apart from this spring attraction force we need a repulsion force to prevent the nodes from overlapping too much. Each node is considered an electrical charged particle with equal charge and therefore every pair of nodes undergoes Coulomb repulsion. There are some parameters to tweak to influence the behaviour such as the stiffness of the spring and the strength of repulsion. The code was actually far more simpler than I expected. The visualisation was done with DirectX 9.
Talk to you later!
Thursday, December 13, 2007
Saturday, December 8, 2007
Dancing Emptiness
Yesterday I was looking for an image that really expressed the essence of this research. No, actually this goes further than 'research' because this sounds too much like a laboratory set-up where the observer is not part of the experiment. Wrong, the observer is right in the middle of it, participating and influencing the outcome. So let's call this an inquiry instead.
The theme of this inquiry is the similarity of patterns arising in our world. There is no real need to understand why and how these patterns arise. Their manifestation is miraculous to us and can be admired without knowing the reason of their existence. Instead this inquiry focuses on similarity between these patterns. Actually the similarity of patterns is a (meta-)pattern itself. Observation is an action. An observer pattern interacts with an object pattern, like interfering waves, creating a new pattern. The pattern is not built from subatomic particles, it is more like condensed nothingness. It is not a dead, inactive nothingness, but an emptiness full of potential. A lot of people call it Consciousness. Consciousness is the stage for things to happen.
This next picture expresses this relation between formless Consciousness and its patterns very well for me. It is a scene from Hindu mythology where Shiva, representing Consciousness, and Shakti personified as the goddess Parvati, representing the patterns (energy) manifesting in our universe, dance.
This is the dance of Consciousness observing itself through the manifestations of patterns. This is the dance that takes place every second, everywhere in every universe.
The theme of this inquiry is the similarity of patterns arising in our world. There is no real need to understand why and how these patterns arise. Their manifestation is miraculous to us and can be admired without knowing the reason of their existence. Instead this inquiry focuses on similarity between these patterns. Actually the similarity of patterns is a (meta-)pattern itself. Observation is an action. An observer pattern interacts with an object pattern, like interfering waves, creating a new pattern. The pattern is not built from subatomic particles, it is more like condensed nothingness. It is not a dead, inactive nothingness, but an emptiness full of potential. A lot of people call it Consciousness. Consciousness is the stage for things to happen.
This next picture expresses this relation between formless Consciousness and its patterns very well for me. It is a scene from Hindu mythology where Shiva, representing Consciousness, and Shakti personified as the goddess Parvati, representing the patterns (energy) manifesting in our universe, dance.
This is the dance of Consciousness observing itself through the manifestations of patterns. This is the dance that takes place every second, everywhere in every universe.
Thursday, December 6, 2007
Making Money
Last week I told you I wanted to link some other data sources to my Crazy Diamonds system besides 2D images. I found a nice dutch stock market site (www.belegger.nl) that provided me with price history for all AEX (Amsterdam Exchange index) stocks. I hooked this source of financial data up to my system and was ready to look for similarity patterns.
I decided it would be nice to take the AEX index itself into account, to really see interesting movements of a stock price relative to the main direction of the financial market, represented by this AEX index. For you nerds out there: I added the capability for amplitude modulation and demodulation using the AEX index as the carrier and the actual stock price as the output value. From the stock value and AEX carrier value at a given time, I derive the signal which is the movement of the stock compared to the current AEX index. This signal is used for the pattern matching.
My first goal is to find similarities between AEX stocks in the last year (2007). Not all AEX stocks were included because I selected the stocks with at least a 5 year history. The Crazy system started matching all stocks and came up with some amazing similarities. Take a look at the next picture for yourself:
You can click it to enlarge. You see pairs of stocks with the right one being the best match for the left stock. It really struck me that some are really similar, check out source "Fortis" with best match "Randstad" for example. I'm not a financial analyst and have no idea if there is a reason for this match. I'd better check data from the previous years to find out if this is a reoccuring event. We also see a clear similarity between "Fortis" and "ING" which seems very possible, them being both banks.
If you would like to see more stock similarity pictures, give me a note and I'll send you some.
I will start thinking about ways to visualise similarity in large sets of data sources. For example it would be nice to have 2D or even 3D overviews showing similar data sources in clusters, enabling us to categorise sources automatically. Oh well, that's enough for now!
I decided it would be nice to take the AEX index itself into account, to really see interesting movements of a stock price relative to the main direction of the financial market, represented by this AEX index. For you nerds out there: I added the capability for amplitude modulation and demodulation using the AEX index as the carrier and the actual stock price as the output value. From the stock value and AEX carrier value at a given time, I derive the signal which is the movement of the stock compared to the current AEX index. This signal is used for the pattern matching.
My first goal is to find similarities between AEX stocks in the last year (2007). Not all AEX stocks were included because I selected the stocks with at least a 5 year history. The Crazy system started matching all stocks and came up with some amazing similarities. Take a look at the next picture for yourself:
You can click it to enlarge. You see pairs of stocks with the right one being the best match for the left stock. It really struck me that some are really similar, check out source "Fortis" with best match "Randstad" for example. I'm not a financial analyst and have no idea if there is a reason for this match. I'd better check data from the previous years to find out if this is a reoccuring event. We also see a clear similarity between "Fortis" and "ING" which seems very possible, them being both banks.
If you would like to see more stock similarity pictures, give me a note and I'll send you some.
I will start thinking about ways to visualise similarity in large sets of data sources. For example it would be nice to have 2D or even 3D overviews showing similar data sources in clusters, enabling us to categorise sources automatically. Oh well, that's enough for now!
Subscribe to:
Posts (Atom)
