Collaborative Test: Methodology
Goals
This tests are focused on the collaboration technology that handles interactions between the users and the document they’re collaborating upon. While it may imply to evaluate some aspects that are more depending on the UI than transaction management occasionally, it leaves a lot of collaborative features such as cloud sharing of chat out of our scope.
Specifically, this tests will:
- Assess the main design choices at hand
- Evaluate operational stability and scalability
- Evaluate the extensiveness of document related collaborative features
The essentials qualities expected from a real time collaborative tool to get significant user adoption are, based upon the Google Doc’s model:
- Document creation and edition should require no or little specific learning from the users compared to mono user applications
- Workflow should be as fluent, or more, than single user application
- The risk to loose user created content shall be equal or smaller than mono user applications
- User shall be able to witness collaboration happening in real time and understand who does what.
Setup description
We use two Surface Pro (a 3 and 4) connected via Wifi to Internet. Each one has a wireless mouse and keyboard connected. We’re filming at 25 ips. Latency is measured by frame by frame analysis. By convention “User A” is on the left (Surface Pro 3) and “User B” on the right. User A will always be the first to act in each sequence.
Tests list
Overview
Check view: user A changes her view on the document. If it doesn’t impact B each user has her own view on the document.
Note: while the dominant option some applications have all users sharing the same view. Forced share view hampers workflow significantly.
Check synchronisation/locks: user A edit the document. Do we see the edition happening on B’s screen or do we need to the user(s) to perform a manual synchronisation?
Note: Manual synchronisation and/or mandatory locks requires specific learning to the user. Moreover, it considerably increases the risk to loose significant amount of content because of conflicts or stability issues. While the occasional ability to edit the document without synchronisation or lock is a good feature (such as online mode, or private/locked part within the document) we consider it essential that the engine supports automatic synchronisation to get user’s adoption and reflect positively on product image.
Check Undo behaviour: user A moves an object, User B Moves it, User A undoes, User B undoes
Note: When User A undoes it’s essential that her actions, not the latest performed on the document (that would be B) is the one undone.
Quality
Basic Latency: user A select a single object and moves it. Frame of the first move displayed on her screen is marked in video editing. Frame of the first move displayed on B is marked. Difference is used to determine basic latency.
Note: We consider 1 second to be the threshold above which collaboration with vocal chat becomes uncomfortable. Also the more the latency, the bigger the risk of data loss in conflicting actions. Only relevant for collaborative tools offering continuous sync but for the much more problematic manual synch we’ll still evaluate of fast a sync happen with two users knowingly collaborating simultaneously.
Load test: user A select a large amount of objects (that also has to be a plausible use case) and moves it. Latency on B screen is measured.
Note: same rule applies. Some collaboration solution will work at the expense of their efficiency at handling transactions (ie the ability to hand one cluster of edit as one single action for one user) in which case it will typically shows here. In the same way any solution synchronizing the whole document rather than what has been edited will take an excessive amount of time in such test.
Advanced Undo: user A edit an object with one intention (eg moves it horizontaly). User B then moves it with a distinct intention (eg moves it vertically). User A undoes. Either the undo performed only undoes her intention (in above example, performing a reversed horizontal move while keeping the vertical position affected by user B) in which case we’ll call it relative undo. Or it will bring back the object in the initial position, cancelling both actions. In such case, we’ll call it absolute undo. If B undoes then, object will come back at a different position than the initial one.
Note: while relative undo may look more desirable it’s actually not always the case. Relative and absolute undo both have their use depending on the context. Still we find it interesting to take note of various strategies ends up being used.
Collaborative features
Communication: what way do the users have to communicate? Aside for verbal messages (text or voice), can they point to an element of the document, for instance to make know they’re about to work on it?
Note: project’s topography is typically largely optional in a single user environment yet essential in a multi user ones where none of them will have automatically a full mental map of the document. Without the help of collaboration oriented locators in the UI basic collaborative workflow can become painful on large documents with many iteration of several elements, for instance.
Project history: does the app allows for any ability to revert a project to a previous state ? does it allow for projects duplication, then any form of merging ?
Note: an essential in our experience, as user’s made document corruption (generally unwillingly) can happen easily as soon as there is a manual synchronisation. This was one of the very first issues raised by Ohmstudio’s beta testers.
Traceability: do we have a way to track who did what in the document ?
Note: another big request from Ohmstudio’s users, especially as musician has been frightened a lot by IP theft issues over the years. But it’s easy to see how a company collaborating with 3rd party independent workers will need such info just for project management or knowing who to ask about the thinking behind some edits.
