Dna-microarray

howdy!

i m trying to visualize dna-microarray data in 3d space. as you maybe know such microarray datafiles are really huge. right now i am workin on a 20mb excell file holding 43000 rows and 90 columns. each row is represented by a sphere.

my patch now works perfectly for one sector on the microarray (== 883 rows)… as i expected performance drops when loading more and more sectors. - right now i have 10 sectors running but really very very very slow. i thought this performance draw back is because of the huge polygon amount. - the strange thing is that it doesn`t get better even when a reduce the sphere resolution to 2x2.

any idea how i can trick around to get a smooth navigation? do i have the draw back because of the parsing of the large csv file or because of limitations on the graphic side (bad graphic card - macbook pro 2y old).

i also have to admit that i am still a vvvv newbie as i really hardly use it.

best
wolfgang

If you could attach your patch and an example dataset…

ah - yes…

sorry

here is the patch…

csvbig08.v4p (132.7 kB)

and a reduced data set (still pretty large)

please excuse my dirty patching…

ok… this one was too large to upload (4mb?)
so only one sector… (this is the one without performance problems)

sector01.csv (426.9 kB)

the data set is missing!
edit: ah ok

seems that most power is consumed by button and the second separate node. (CTRL-F9 for debug mode)

maybe i have time to dig into why these nodes are so lame.

Just had a look at the separate node and made a little performance optimization. Before your Example-Patch ran with 16 FPS on my machine now its 24. Will be available next beta.

You could use RegExpr(String) instead of Separate. I think it will be faster.

thanks for having a look at it!

happy to hear that its not actually really stupid patching ;)… thought of using RegExpr before but was a little afraid of the syntax (had some sleepless nights with regular expressions long time ago ;) )

2 little questions en plus:

is there any workaround for the reader node as it takes up to five minutes to load the 4mb file? would it make sense to use more than one reader and split up the data file?

is there any recommendable workaround on using spheres (i think of something like “points” with a shader applied to generate volume? or using particles?)

so far so good

thanks again

best wolfgang

death by parsing x’n’y out of a csv table…

is this necessary?
couldn’t be those generated automatical?
then you could parse only that row of your table that you really need.
much faster this way.
see attached…

furthermore it seems that you don’t know Select (Value) and Count (Value) because you patched some workarounds…

urgently i want to recommend reading this
also have a look at diki’s performer approach

csvbig08.v4p (131.7 kB)

and i would S+H on
leftclick AND no keypressed.

so the renderer (html) doesn’t need your performance when moving the camera.

csvbig08.v4p (136.4 kB)

…"death by parsing x’n’y out of a csv table…

is this necessary?
couldn’t be those generated automatical?
then you could parse only that row of your table that you really need.
much faster this way."…

well yes - the actual data i will have to work on later won`t have any regular x/y distrubtion. x/y values will for example also represent known relationships and dependencies between various genes. - after all i will only get csv files with the results and i just wanted to test where the performance barriers are when working this way…

big thanks for the S+H, select and count hints. that was what i meant by stupid patching ;)…

best regards
wolfgang

but see diki’s performer approach.
might save your performance.
i assume you only once read the file…

ok - so far so good

those tipps help me pretty much…

the next thing i want to implement is something like a 2d context menu popping up on sphere rollover in the ex9 renderer… - just like the gene name (which works pretty nice).

is there a way to get two renderers in one window (except from ie texturing a gdi to a ex9 geometry)…

should i start a new thread about this? - i don`t want to annoy too much with those newbie questions…

best
wolfgang

again posted too early… (i guess)

i found remote (vvvv)… will that do the job? (good/bad idea?)… - doesn`t seem to be tooo easy to handle…

best
wolfgang

why don’t you use just texture a quad (positioned with the sphere) with the appropriate information and only make it visible on rollover? seems to be the most simple method…which is also a way to get 2 renderers in one window…
renderer1->dxtexture->quad->renderer2

yes shure… the thing is that the menu should be interactive (choosing menu items - ie. weblinks)…

it would be also ok if there would be a way to keep a relatively small gdi or flash render on top while the ex9 render is in fullscreen… - so i could have the menu in the top right corner…

best
wolfgang

hi everybody…

some months passed by since my last patching approaches on this thing… two much work and so on…

anyway - i found something called “point sprites” while looking for something replacing my spheres by something more “performing”:

http://www.two-kings.de/tutorials/dxgraphics/dxgraphics17.html

sounds perfect for my approaches.

has anybody ever worked with that (is there maybe a module i don`t know)

and… should i start a new thread on that?

best
wolfgang

have a look at the User Shaders page, there is a point sprite shader…