How do I manage our microscopy data?

So today I solved one major lab issue (I think).

I have always used “personal” file servers to archive our data: essentially simple hard drive arrays in RAID5 configuration. These have been accessible to all in the lab from undergraduate project students to postdocs. These are great but they do fail .

Recently two of ours failed.

Had these two drives been in the same array, we would have lost a lot of our imaging data. So I felt the need to do something about this. My University has fortunately just implemented 5Tb (yes, Tb not Gb) of network storage for each “data steward” (essentially each academic). This is automatically backed up and can (in future) be archived to tape. This seems to solve the problem for 5Tb at least but also for more (if we pay). Local IT have also provided a possible solution for local file storage (i.e. within the building) which could be a very useful solution for our core research facilities such as Bioimaging and Proteomics that generate large data volumes.

The large network data store required me to write a data management plan which I did using this online Data Management Planning Tool by the UK Digital Curation Centre through a JISC funded project. This was surprisingly useful in the way it prompted me to think about the way that I organize lab data and consider its future use, access etc.

So, now to the question – HOW do I organize these data sets.

We take a lot of microscopy data from a lot of different imaging systems. Thus formats are complex and not always future-proofed. one plan here is to export as OME-TIFF using the Open Microscopy format that includes the META data. This is simple and fine. In fact we acquire most of our data using Volocity from Improvision/Perkin Elmer – this includes good metadata and (at least for now) the core software is free from Perkin-Elmer on registration.The OME-TIFF option would allow us to take everything including the meta data into ImageJ or equivalent very easily.

Its more the folder structure that requires better organization. Currently I list everything in date order (yy-mm-dd) followed by a brief experiment name. This date format means I can sort by date acquired (which to a large extent is how my brain remembers experiments). I am wondering whether to then include a Word/Text doc alongside detailing methods, labelling etc is enough. We can usually get this annotated directly on the microscope. The other thing I plan to include is a short description of the experimental goal and outcomes where possible. This might in fact be within a higher level directory where we organize by project/sub-project. I need to get the lab to properly commit to this. Progressive creep away from such rigid systems is also a concern but that is another issue…

So….does anyone have a better way? I wish to have something that is simple but highly effective. I want to look through our data archives and know exactly what everything is without referring to the person who acquired the data. I should already be there but I am not.

Before someone suggests OME, we don’t have the wherewithal to move to OMERO – the server setup is beyond me and not something that we can implement easily through our IT support. This is one for the future…

The good news is that I am taking data management more seriously as required by Research Councils (in the UK at least but I guess also internationally by all funders). The bad news is that I clearly have a long way to go with this.


8 thoughts on “How do I manage our microscopy data?

  1. Would a web-based front-end running off a database be more suitable than a Word/text document, but considerably less complicated than installing OMERO? You could then store all the experiment metadata in one place, where it can then be browsed (e.g. by project) and also queried by text search (e.g. show all experiments where a specific label was used).

    With some additional work, it would probably also be possible to employ a workflow that will take the metadata held within the existing data files and put this into the database (to aid in searching the data).

    I appreciate this is more complex than a basic text file, but may save a lot of time in the long run.

    • Thanks Bruce.
      This is a really good suggestion; in all honesty I wouldn’t be entirely sure where to start with this though. My IT skills are very much “personal” computing not server/database stuff at all. Any suggestions where to look for a template for this? David

      • I believe there are some “off the shelf” lab management systems available, but I have no experience of installing these or modifying them to meet specific needs.

        The programmer inside me would recommend making your own system. Someone with a bit of PHP/Java and MySQL knowledge could probably knock something together over a weekend. Any old Linux web server (of which I’m sure the university has many) would be able to host the service. Do you have a bioinformatician on speed dial?

        As a similar suggestion, but perhaps considerably simpler to implement, could be to have an Excel spreadsheet containing the same data. This would offer more tools than a Word document (such as being able to search specific columns and sort data) but doesn’t need any programming skills!

  2. The format of using date-based archiving should work, however i agree that the difficult part is having an accurate log of exactly what was performed to take that image. This should all be down to lab-book management. I’ve had both a postdoc and a PhD student with meticulous lab-books, and the process of finding out what they did is straightforward and links the date in the lab-book to the experiment and the images (with associated metadata) on the server. For the sake of argument/discussion, surely properly maintained lab-books are a stable record (discounting flood, fire and brimstone), have no compatibility issues (handwriting aside) and have so many more benefits (easing paper/thesis write up, IP issues), meaning that instead of looking for a new system for image storage, you should be looking for a new system to record and archive all things done in the lab, with imaging just being a part of this.

    • You are right on two points here I think Pete.
      1) lab books have a lot going for them if managed properly.
      2) this needs to be a system for all of our data, microscopy is just the core of that. In fact nearly all of our data is now in the form of digital images in one way or another (Odyssey scanner for blots, digital camera for gels, plates etc).

      Regarding the first point – I simply find that lab books become impenetrable over time. Even for current lab members I find it hard to go to their notes and find out what they do – a better culture of annotation is clearly required because this is something usually found within the software anyway (e.g. Volocity).

      I have wondered whether Labguru or similar would help as it would enable integration f lab collections of antibodies, plasmids, siRNAs etc. However, the cost in time of inputting all of our collection data is daunting because its currently all handwritten! The cost of integrating the lab is also putting me off – free for individual use but our small lab would give rise to an annual fee of $720 for Labguru! Perhaps a simple database solution as suggested above by Bruce would be more appropriate.

      • I have looked at LabGuru too – I used to use LabLife – but the fees are annoying, and the lack of a local application is also a problem, since our internet is not as reliable as I would like. Ideally an Evernote-style local app that syncs when required would be better than a pure web interface. We have been looking at a local, backed-up, semi-custom solution based upon OpenFreezer, but have yet to take the plunge.
        Being a new lab gives us the opportunity to start from nearly-fresh, but nothing jumps out as an ideal “ready-to-roll” solution. I was nearly burned once before when VectorNTI went from free to pay-only, and had to get my data out using a free trial, so I am wary of anything I don’t have a copy of locally in an open format.

  3. Alan makes a very good point here – online tools that don’t last because of reasons beyond one’s control. The Labguru issue of no local files is understandable from the collaborative point of view. The PDF export looks very limiting in terms of what you could do offline. It could be great but I get the sense you would have to buy in to this long term to get real benefit. Realistically I can’t see us using valuable research income for this.

  4. Pingback: Update on microscopy data management | Stephens Lab Blog

Comments are closed.