Feature / design suggestions

[danielparton]

(copying this issue thread over from https://github.com/choderalab/choderalab/issues/46)

dannyparton:

Hi, this is to start collecting ideas for what to show on the status display monitor in Z1145, once it has been installed. We could have some things displayed constantly, with others cycling periodically.

Suggestions welcome!

A few ideas so far:

• Dropcam feed
• Some kind of status from Thermo Momentum
• Cluster status / node availability
• Recent GitHub activity
• Travis CI status
• Progress of FAH projects
• Kinase leaderboard (top 10 from the prioritization scheme)

[danielparton]

kyleabeauchamp:

http://cbc.arizona.edu/njardarson/group/sites/default/files/Top200%20Pharmacetical%20Products%20by%20US%20Retail%20Sales%20in%202012_0.pdf

[danielparton]

jchodera:

Graphical monitoring of cluster jobs:
http://sourceforge.net/projects/myjam/

[danielparton]

jhprinz:

If we use slack, we could add this or other interesting feeds from twitter, etc... Dannys post was already pretty complete.

If we have the MacMini running it would be easy to setup a simple web app that can integrate all of these. If Danny is interested we could patch something together using some "state-of-the-art web 2.0 technology".

Most of the stuff Danny suggested should be available though some API, Twitter or RSS feed I guess. Also Dropcam can easily be added to that, too.

Here are some example of templates that could be cheaply purchased and easily adapted.

http://kopyov.com/demo/londinium/info_blocks.html
http://flatfull.com/themes/scale/index.html
http://pixelcave.com/demo/proui/page_ready_timeline.php

all of these can be found at http://themeforest.net for $15-25

Just an idea. Of course we could go with little less design for free :)
Most of this stuff runs from predefined Javascript libraries or might need a little php or python script to process the data. We can probably set up something nice in 1-2 days and add fun stuff, when we find it.

Also, I have seen software that can turn a MacMini into a AppleTV like thing with multiple client support.

http://www.airserver.com/

Just tried it and it works fine. It's $15 but it works really well. It activates once someone log in automatically and supports multiple mirroring at the same time which I could not confirm yet.

Then, I still have this leapmotion thingy I wanted to show you. We could use one of these to make the board interactive for the website. This would allow with single gestures to controle the screen. I can show you. A browser interface is already available and I have tried it before...

Any other stuff?

[danielparton]

jchodera:

Monitoring the cluster via ganglia:

ssh -L 8081:localhost:80 chodera@sd.cbio.mskcc.org -f sleep 1000; open http://localhost:8081/ganglia/

[danielparton]

jhprinz:

Danny and me have discussed the status monitor some more. Here is a first list with things we came up with:

Web frontend: using JavaScript, HTML5, Bootstrap
Design: Maybe buy something nice from theme forest for a few bucks. I assume doing ourselves is too complicated
Backend: Python Server with Flask (preferred), node.js (Javascript Runtime Server) (alternatively)

Open conceptual questions:

• Do we want interactive elements, like different view to be accessible, interaction with messages or automatic dismissing.
• Do we want this accessible from outside?
• Do we need a responsive design, e.g. different view for mobile, tablets, laptop in the back, etc

Once we have agreed on the general setting I can setup a github repository with a project structure and we go from there. Still all the stuff should run locally on the MacMini.

Dropcam

Include streams by hijacking the id from the Dropcam site. This should be done one manually and then run as long as the application is not stopped. Might be automated but could be cumbersome to implement. The element looks like this

<object type="application/x-shockwave-flash" data="//cdn1.dropcam.com/flash/video.swf?2037cd0985" width="640px" height="360px">
    <param name="wmode" value="opaque">
    <param name="bgcolor" value="#e3eaed">
    <param name="allowFullScreen" value="false">
    <param name="allowScriptAccess" value="always">
    <param name="flashvars" value="uuid={UUID}&cameraScale=1.0&cameraCenterX=0.5&cameraCenterY=0.5&sessionToken={TOKEN}&...">
</object>    

Width and Height needs to be in the correct aspect ratio otherwise bars in the background color appear. Multiple streams can run without a problem but consume many resources.

Robot

When Momentum is running it constantly writes into an XML file named AuditLog.xml which is located inside the Audit . In addition, when a worklist is running a worklist specific folder is created inside the Worklist folder. This contains also an AuditLog.xml file that is constantly filled with all commands being done during the worklist run. Once it is finishd the results are also stored in an Archive that has special Gantt formatted XML files in it. Once a XML file exceeds a size of about 3MB it is split. Also a new audit file is started when Momentum is started.

1. Status

For this the general Auditfile should be consulted. Here also errors when a device experienced an error, etc. are stored. The newest file is always called AuditLog.xml while previous versions are renamed to AuditLog1.xml etc... It also contains general things like robot run in simulation mode, a worklist has successfully finished, etc.

2. Error messages

I have not checked this, but according to Munis error messages should appear also in the Worklist AuditLog.xml file. We should check both files for specific types of error messages and display them well visible on the screen. Still, we would need some way to dismiss such messages

3. Worklist progress

Although the worklist file contains the progress there is no way to figure out how far a process has progressed. Only when e.g. an Iteration is finished it is reported. We could use the results from a simulation and attach these to a running simulation. That way we could show a complete gantt chart which is updated online to see the progress. But this would require to manuall assign simulations to real runs. Might be too cumbersome and we would not do it. Still the current progress could be shown in an realtime gantt chart.

4. Analysis

Sonya has some python scripts running that allow to analyze infinite XML files and produce figures. This could be incorporated into the monitor. The file locations for the infinite are specified in the audit files and could easily accessed. We might report a nice timeline that has a report on the read plate in it.

Cluster Status

Retrieve XML file from

ssh username@sd.cbio.mskcc.org ganglia --xml

parse it and update the display accordingly. Charts and gauges should be rendered in the browser and not using images

Temperatures

We have not looked into APIs to access the temperature sensors on the robot, but there is a website, that we could parse and use. This would mainly be to get the temperature at the cytomat and we should get another monitor for the fridge. Problems here can also be displayed as large error messages.

Github / travis CI

Both provide APIs to access e.g. last commits or build status of projects after a simple authentication.

FAH Status

This might be integrated once siegetank is working (or is it?)

Twitter, etc...

Might also be accessed by an API and added to our stream of events.

Timeline

I think we should have a general timeline of events where everything that happened gets logged. Especially on a mobile device this makes sense so we could check what the robot did, etc...
Alternatively we only have specialized dashboards for each source separately.

Open Deliveries

This requires to access the sciquest or quertzy. Or some manual solution -> ask John

Calendar

Make a google calendar that is displayed and shows upcoming events. Might be combined with the Open Deliveries

[danielparton]

jhprinz:

I have added a first running example in choderalab/clsm to be changed. It has so far a php-based REST Server and a nodejs-based WebSocket Server. Would be cool, if Danny could copy a simple python-based REST Server too in there. I would continue to add a simple server that runs on the momentum machine and can be accesses using a RESTful interface and provides all sorts of data about the status of the robot.
I have also used the "TODO" layout which I have been using so far. We should also think about a layout.
Right now we have:

• Both Dropcams working.
• A rudimentary momentum AuditLog.xml reader and
• A showing of the cluster status using ganglia (right now it updates every few seconds. From Germany it took about 20s + 5s wait) and the loads change very little so it looks, as if nothing is happening. But the clusters on the left and on the top change! I tested it.
  Would be good to gather things to do next.

[danielparton]

jchodera:

Awesome, @jhprinz!

Why don't we create a separate repository for this, maybe something like status-display or command-center or similar? That way, we can keep issues grouped with the code?

[danielparton]

jchodera:

I also don't think anybody has requested a static IP for the Mac Mini or put TeamViewer on it yet, but we should do that so that we can log in remotely to maintain this stuff.

[danielparton]

jhprinz:

I like command-center! Chodela-Lab Command Center (CCC) or C3. A static IP would be very useful. Is there actually any news on the bridge to access the EVO machine? Herea little pic from the actual "Command Center"

[danielparton]

jchodera:

That looks awesome!

I will work on getting a static IP address and DNS alias for the command center Mac Mini, though it will be difficult to allow remote login or HTTP access from outside (though I can try).

@jhprinz : Do you have a VPN account already? I realize that we should really, really get you one, since you won't easily be able to get into the network otherwise. Had you filled out a VPN request before you left MSKCC?

I don't actually know what the status of the bridge machine is---did anybody actually talk to Ronan while he did this?

[danielparton]

jhprinz:

@jchodera No I have not yet a VPN account. Can I sign something here and send it to you?

I have just updated the code a little and found a way to access a server on a different machine from a webapp like our status monitor. It uses JSONP and so you can just set some server and port and it asks for data which the server can deliver. Apparently there are lots of restrictions on what files a webpage can access and which not. This is also true for things on the local machine. Seems to make sense though.

@sonyahanson I have updated the README.md file in the monitor folder. It contains more details about the files you need to look at and what there role it. You do not need to install a server, it runs just from the plain file. Let me know, if I can help.

@dannyparton Could you have a look at the server.py which uses Flask? It works now, but I guess it can be made much simpler. Let me know what you think.

[jchodera]

SiegeTank stats for our projects:
https://www.proteneer.com/stats
(There are JSON versions of this available for parsing too.)