Cloud-based storage in MCS 2.0.6: setting up MScan and MView to access the cloud

In MScan and MView, each user (chosen or created from the User window shown when the programs start) has his or her own cloud settings.

Click on the Settings/Cloud Storage Settings menu item on uppermost window of MScan or MView.  

For Amazon S3:

Choose Amazon S3 for the service and  enter the Access key ID and the Secret key ID.

For Microsoft Azure

Choose Microsoft Azure for the service and enter the Storage Account Name and the Primary Access Key.

Cloud-based storage in MCS 2.0.6: introduction to S3 and Azure

Amazon S3 (Simple Storage Service) and Microsoft Azure services are very similar in the way they operate. Their services can be tried for free, either for a limited time or for a certain amount of data transfer. In addition, data files located on the cloud are organized in a similar way (see table below). The the top level is called an account, which would be similar to a disk drive. The account holds either buckets or containers, which are very similar to disk folders or directories. Buckets or containers contain objects or blobs. For all practical intent, objects or blobs are MCS data files. A major difference between a file system on a disk and the S3 or Azure cloud storages is the fact that buckets or containers cannot contain other buckets or containers, only objects or blobs.

Your Amazon S3 Microsoft Azure

computer

Drive Account Storage Account

Folder Bucket     Container

File         Object Blob 

An Azure storage can store up to 200 TB of data. Blobs can be as large as 200 GB.

MScan allows you

• to create new buckets and containers
• and directly upload data files to one of these "folders" at the end of an imaging session.

MView is capable of

• dowloading data files on the cloud to a local folder on your computer
• create new buckets or containers.
• upload files to the cloud
• delete files in the cloud

Cloud-based storage in MCS: setting up your S3 or Azure account

For Amazon S3:

1. You need first to set up an account. Go to:

Amazon S3: http://aws.amazon.com/s3/

2. Once you are activated, you will receive a confirmation e-mail. Follow the instructions in the e-mail to set up Access Identifiers. Use the Web-based AWS Management Console to try creating a bucket, manually upload a file etc… Click on the Security Credentials menu item (under the menu item bearing your user name).

The two most important information to retrieve from your account are:

• Your Access Key ID: it is a 20 character-long string
• Your Secret Key: it is a 40 character-long string

Copy these two values somewhere (on paper, in a text file or in an e-mail) to register later in MScan and MView.

For Microsoft Azure:

1. You need first to set up an account. Go to:

Microsoft Azure: http://www.windowsazure.com/en-us/

This requires an account name and a password.

Log into the Windows Azure Management Portal.

Click on the NEW button at the bottom of the navigation pane 

Click on DATA SERVICES menu item on the left of the window, then click on STORAGE, QUICK CREATE.

Enter a name for the storage account in the URL input box (e.g., jimtwophotonfiles), select a location for the file repository (e.g., West US) and enable/disable geo-replication.

2. Once the storage account is created

Click on the Manage Access Key button at the bottom of the screen 

The two most important information to retrieve from your account are:

• Your Storage account name
• Your Primary Accesst Key

Copy these two values somewhere (on paper, in a text file or in an e-mail) to register later in MScan and MView.

Backing up and archiving your multiphoton data: the MCS 2.0.6 cloud-based solution

Optical imaging instruments such as multiphoton microscopes generate large amount of data. Because MOMs are often used to collect two-photon images many times per second over periods ranging from minutes to hours, typical files can routinely exceed hundreds of megabytes. The trend towards producing tens or hundred of gigabytes of data per day is accelerating with the use of video-rate resonant scanning imaging in behavioral experiments. As a resut, the problem of backing up and archiving terabytes of two-photon datasets is becoming particularly acute. One could use a local file server but this requires funds to buy the hardware, expertise to set up the server and some kind of periodic administration. And of course, when the file server is full, someone will have to take care of the problem.

The computer industry has already a solution, called "cloud storage" to safely store large amount of data without having to invest in a costly infrastructure. In exchange for a small fee, Web behemoths such as Amazon, Google and Microsoft offer disk space on their servers to end-users, provide a Web-based interface to administrate accounts and programming tools to upload or download files. The advantages of cloud storage include:

• virtually unlimited or extremely large storage capacity.
• reliability: the data is usually stored in triplicate and, optionally, at sites that are geographically distinct to prevent losing data in case of a catastrophic hardware failure.
• universal availability through an Internet connection.
• flexible pricing that scales up with the amount of data you store.

MCS 2.0.6 now leverages cloud storage capabilities by allowing:

• in MScan, to systematically upload your newly created data files to a user-specific location on the cloud upon finishing an imaging session. You won't have to back up (or forget to archive...) your data files manually anymore.
• in MView, to download data files from the cloud to your local disk storage to perform analysis. Your data is available anywhere and at all times as long as you have an Internet connection to access your cloud account.

MCS 2.0.6 at present supports two services, Amazon S3 (Simple Storage Service) and Microsoft Azure. If you opt for their fee-based plan, Amazon has no restriction on the amount of data you can store while Microsoft now caps the amount of data per storage to 200 TB. However, you can create as many storage accounts as you want. Maximal file size is 200 GB and 5 TB for Azure and S3, respectively.

There are many other cloud storage companies. Google has a service similar to S3 and Azure called Drive, but files cannot be larger than 10 GB, which could be a problem with data collected in resonant scanning mode. For those who are accustomed to Dropbox, it appears that this company uses Amazon S3 as its underlying infrastructure.

MCS 2.0.5 released soon

MCS 2.0.5 is currently being tested for imminent release.


MScan 2.0.5

New features:
(1) NON-RESONANT SCANNING.  The 3DMap window demagnification can be changed to be less than one (i.e., one can zoom in on views).
(2)  NON-RESONANT SCANNING. Photostimulation targets can now be designated in the 3DMap window with a demagnification < 1.
(3) Time-lapse imaging can now be done with intervals less than 1 s (the interval between frames is now set in milliseconds). You can take frames every 500 ms when your frame duration is less than that (for instance ~ 330 ms at the default frame rate of 3.05 fps).
(4) There is now a zPiezo Automation object (of the Piezo type) that allows outputting voltages to the z-piezo controller during scripts.
Features (3) and (4) allow scripts to move the objective with the piezo in time-lapse mode after taking a frame. The idea is to take a frame and while waiting for the next frame, to move the piezo. The interval between each frame will allow the objective lens to settle when performing large z- changes.
Bug fixes:(6) RESONANT SCANNING. There was a bug in generating the waveform controlling the Pockels cell.(5) NON-RESONANT SCANNING. When imaging a subframe (less than 512 x 512) consecutive images are not flipped anymore (subframes are selected by dragging the mouse over a region inside the display area of the Viewer).


MView 2.0.5

Improvements
- When converting to TIFF files, information is now written as an ASCII string (not as UNICODE).

Bug fixes
- Files taken during photostimaging in modes other than region scans can now be open in MView.

The on-line documentation has been updated.

On avoiding TIFF files

TIFF files.

TIFF (Tagged Image File Format) files are used by several notable two-photon imaging software packages to store frame data. TIFF files appeared in the mid-1980s and the latest update, TIFF 6.0 was issued in 1992. Needless to say, TIFF files are woefully outdated - one of the biggest limitations of TIFF file is their size, which cannot be larger than 4 GB.

With conventional scanners, when imaging 512 x 512 frames at 2 fps on two channels, a TIFF file would be full in about 34 min. As I mentioned earlier, this would happen in a little more than 2 minutes in resonant scanning. Writing files in the BigTIFF format could bypass the 4 GB size limitation, but this new file format is not widespread.

Another disadvantage of TIFF (and BigTIFF) files is their internal organization (or lack of). Frame data can be placed anywhere in the file, thus defeating most streaming strategies. In particular, TIFF readers cannot expect the n-th frame to be located at a predictable location related to the frame index - the entire file has to be read until that frame is found, which is arguably a performance problem with large files.

Lastly, the collection of image tags is not extensible so important two-photon scanning information has to be placed arbitrarily in the image description section of the file.

For those who are accustomed to analyzing data in TIFF files, MView provides MCS to TIFF conversion, either for individual files or for batches of files in the same folder.

Structured Storage files.

MCS uses an advanced file format that completely overcome the shortcomings of TIFF files. Firstly, MCS files are based on the Structured Storage technology available under Windows. Structured storage files are like miniature file systems that contain arbitrary numbers of folders called storages which in turn can hold either other storages or internal files called streams.

An immediate advantage of such files is the ability to store analog data (e.g., electrophysiology) in a separate stream alongside imaging data. Individual streams can be as large as 16 TB, so there is little concern that Structured Storage files will run out of space in the foreseeable future. Writing or reading data to or from  the data file is straightforward, thus efficient, during streaming operations. MView has no difficulty opening and reviewing data files larger than 100 GB. In principle you can think of adding all kinds of parallel streams of data, for instance visual stimuli patterns without disturbing the structure of the file.

Similarly, file properties are stored in an independent stream. MScan writes standard file property / value pairs (e.g., microscope lens resolution, stimulation parameters, etc…) but you can add also your own property names and values to the file (e.g., "Animal Line" = "Wfs1-Tg2-CreERT2 : Ai38").

During analysis by either MView or MCSX, file properties can be queried programmatically for property values. You can also visualize the property name / value list of any file without opening it by right-clicking on the file name in any Explorer window.

Structured Storage is a powerful technology used by many professional Windows programs including Microsoft Office, MSSQL, Dassault Systemes SolidWorks and other CAD programs. By using Structured Storage, MCS provides two-photon experimenters with a whole new level of capabilities during imaging and data analysis.

MCS 2.0: 32-bit and 64-bit versions

The programs in MCS are now either 32-bit or 64-bit.


MScan 2.0

The new MCS 2.0 workstation runs MScan 2.0 64-bit under Windows 7 64-bit Professional Edition. This system, which allows either conventional or resonant scanning, is now the standard computer for new MOM and MCS orders.

We provide a 32-bit version of MScan 2.0 for customers who are running MCS on a 32-bit computer. To allow resonant scanning, we recommend switching to the 64-bit version of MCS.


MView 2.0

MView, the analysis program of MCS and the software component MCSX (which allows, for instance, to read MCS files from Matlab) are free of charge and can be downloaded from the web site of Sutter Instrument (www. sutter.com). They can be installed on any version of Windows starting from XP.

The MCS analysis package is now available in either 32-bit or 64-bit versions. Prior to installation, choose the correct version that matches the "bitness" (32-bit or 64-bit) of your operating system.

MOM and MCS 2.0: now with resonant scanning

Resonant scanning allows video rate two-photon imaging at 31 frames per second, a near 10-fold increase in frame rate over conventional scanning. Sutter Instrument is now proposing its Moveable Objective Microscope with either conventional or resonant scanners.  Current owners of the MOM can also benefit from the advantages of resonant scanning without having to buy a new microscope. The unique design of the resonant scanning unit allows swapping it with conventional scanners in a matter of minutes. The resonant scanning module provides also much needed sound muffling. 

MCS, the computer system of the MOM, has been upgraded in version 2.0 to control either type of scanning from the same Windows 64-bit MCS workstation. There is, by design, much commonality between the hardware and wiring of the two systems. Switching between conventional and scanning mode in MScan 2.0, the acquisition program of MCS, requires only restarting the application.

Notable features of MScan 2.0. operating in resonant scanning mode include the possibility to:

• acquire and display 31 frames per second on two channels with 512 x 512 frames, or even at a higher rate by selecting a subframe (e.g., 124 fps with 128 x 512 frames), while simultaneously
• perform the "Sanderson correction" to linearize the field of view
• acquire up to 8 analog channels
• stimulate the preparation with 2 analog channels and 8 digital channels
• plot intensities of regions of interest in real-time (very useful when doing calcium imaging)
• run JavaScript or VBScript automation programs in the background
• and stream data to disk.

Because resonant scanning generates a large number of frames very quickly, it is a challenge to store the incoming data on file. Many programs other than MScan use TIFF as their main data file format, which is inadequate for resonant scanning due to the 4 GB limitation in size of these files. With two imaging channels, it would take only 2 minutes and 16 seconds to fill a TIFF file to capacity. To allow continuous recordings on a much longer scale, MScan features a high-speed, large capacity disk streaming system. This allows MScan to record more than 100 GB of uninterrupted imaging data in the same file. In addition, a new triggering mode allows starting and stopping imaging and data streaming upon sensing the level of a TTL line. The possibility to perform gap-free resonant imaging for several hours is a major advantage of using MCS in experiments involving behavior.

Welcome to the Sutter MOM blog

The Sutter MOM blog.

You will find news, updates and tutorials related to the hardware and software of the Moveable Objective Microscope from Sutter Instrument.