The synchronization « tracker » consists in a database to store the files' states along with an HTTP web service answering to a specific set of operations. You can either access:
- your dedicated tracker, usually accessible behind a domain like
https://<trackername>.syncplanet.io/api, - the main server at https://syncplanet.io/api,
- a demo server explaining basics at https://demo.damas.io/api
The tracker is based on the free and open source project damas-core (data management system core) to store the objects needed by the synchronization system and to provide basic CRUD operations to interact with them.
You can visit the Environment Setup Guide to setup a Python, Javascript or Shell scripting environment.
¶ Objects Types
The database stores objects using the JSON format and are indexed using the _id key being unique identifiers Strings.
¶ Files
The files are indexed using their path, as a unique identifier, under the _id key. A few other keys are used to control the synchronization system (see the list below). Any number of custom keys can be added without limitation, this might be useful for an integration in a custom asset management system.
¶ File Publish
When some files are created and need to be propagated on several sites, at publish time, a JSON object is provided, so the synchronization service can process the file specified by the _id key:
{
"_id": "/file/path", /* UNIX path format (String) */
"comment": "<text>", /* Text comment of user when published (String) */
"origin": "<sitename>", /* Volume name where the file is coming from (String)*/
"sync_priority": 7, /* Positive or negative, 0 is default normal priority (Number) */
"syncEverywhere": true /* overide default behavior in globalParameters (Boolean) */
}
_id:
File's identifier, using UNIX path format (String)
A path is relative from the project's root directory
A path always begins with a slash (
/) character
Slash (
/) characters are used as sub level delimiters
A folder can be indexed using a trailing slash, see Packages
comment:
User's text comment, specified at publish. (String)
If the file was discovered by an scan agent, the comment will be
scanned at <sitename>
or
modified at <sitename>
origin:
Site's name, on which the file was created, and has to be retrieved from (String)
sync_priority:
Order preference to process this file (files with highersync_priorityare processed first, default is0, can be negative) (Number)
syncEverywhere:
Does this file has to be propagated everywhere? (on every other site that is was originated from?)
(Boolean)
To select specific targets : put
syncEverywheretoFalseand use specific keys for each site:
sync_<studio1>: True (Boolean)
so, as an example:
{ "syncEverywhere": False, "sync_studio1": True, "sync_studio3": True }
propagates to studio1 and studio3 and not studio2
your_key:
Use the JSON file object to store any key-value pair useful for your asset management system
¶ New Keys at Creation
The author and time keys are automatically inserted by the tracker when a new file is created:
{
"_id": "/file/path",
[...]
"author": "<@username>", /* String, authenticated user who did the publish */
"time": 1567529676000 /* Number, time of the element, used at creation and element modification */
}
author:
The authenticated username who did the publish.
The
supportusername is used if the file was discovered by the automatic file system scan
time:
The publish time used to display and sort the elements in the interfaces.
The
timekey is also used to compare thesynced_*keys so the system knows if a synchronized copy is outdated (yellow color is then used instead of green)
¶ New Keys at Process Time
Some keys are created and updated as the synchronization process runs. These keys are useful for the monitoring interfaces, to get the file's history log, or to debug the file's synchronization:
{
"_id": "/file/path",
[...]
"file_mtime": 1567529676000, /* Number, modification time on the origin file system (epoch time in milliseconds) */
"file_size": 1286230019, /* Number, number of bytes on the origin file system */
"modified_site5": 15676534821582, /* Number, a modification was scanned on site5 */
"syncError_site2": 23, /* Number, error encountered on site2 (rsync exit status code) */
"syncLastAttempt_site2": 1567516933000, /* Number, time of last failed attempt on site2 */
"synced_site1": 15675564912347, /* Number, successful delivery time on site1 */
"synced_site3": 15675564591231, /* Number, successful delivery time on site3 */
"synced_site4": 15675565175645, /* Number, successful delivery time on site4 */
"syncing_site6": 1754897397000 /* the service is propagating this file on site6 since epoch (Number) */
}
file_mtimethe value of the mtime attribute of the file in the file system (unix timestamp - epoch - in milliseconds)file_sizethe value of the size attribute of the file in the file system (size in bytes)modified_sitenameunix timestamp in millisecondssyncError_sitenameexit status code if an error occured on that site. 23 is: file not foundsyncLastAttempt_sitename: time of the last unsuccessful sync attempt for sitename. unix timestamp in millisecondssynced_<sitename>: time of the successful sync for sitename. unix timestamp in millisecondssyncing_<sitename>: the service is processing this file for this site since specified epoch
¶ Optional Keys
If you need a multi-site locking system please contat us.
{
"lock": "<@username>" /* String, username who is editing the file */
}
¶ Users
The users are described as elements wearing some reserved keys: username, password, class plus optional keys.
{
"_id": "@username", /* String, unique user name identifying the user in the database */
"class": "editor", /* String, a user class for permissions. Could be guest, user, editor, admin */
"email": "usermail@address.com", /* String */
"fullname": "Firstname Lastname", /* String */
"password": "13d3a2a16c0cd2f7bf115d471999377e" /* String, hashed password */
}
The Authentication page gives more details about the authentication mechanism.
¶ Generic function lists and objects
¶ CRUD Function List
A set of generic CRUD functions is provided to create, read, modify and delete elements:
create- insert element(s) in the databaseread- retrieve element(s) using identifier(s)update- modify keys on existing element(s)upsert- create or modify element(s)delete- delete element(s)
¶ User Management
signIn- request an authentication token from the serversignOut- revoke a tokenverify- ask the server for the authentication status and user
¶ Search queries
graph- retrieve connected edges and nodes (recursive)search- find elements matching a query stringsearch_one- find first element matching a query stringsearch_mongo- find elements using a MongoDB query object (if MongoDB is the back-end)
¶ Function Details
¶ lock
Lock file(s) for edition. Sets a lock key on elements with current authenticated username as value)
lock( identifiers [, callback] )
¶ parameters
identifiers{string | array} : identifier(s) string to lockcallback{function} (js only, optional) : if specified, the request is asynchronous
¶ return values
- returns true on success, false otherwise
- returns false if the element is already locked
# Python
# lock one asset
project.lock('/project/path/to/file')
# True
# lock multiple assets in 1 request
project.lock(['/project/path/to/file1', '/project/another_file_path'])
# True
¶ publish
Add files to the index
publish( elements [, callback] )
Many options are provided at publish time, allowing Folder Packages and Destination Filtering.
¶ parameters
elements{object | array} : object(s) to insert in the databasecallback{functon} (js only, optional) : if specified, the request is asynchronous
¶ return values
- returns an array of nodes (containing parent nodes and child nodes) on success
- returns
null(Javascript) orNone(Python) on failure
Same specifications as /api/create, except that it is expecting specific keys and a child element is created to keep track of the states upon publish
When we publish a file that is already tracked, the function returns an error
¶ examples
{
"_id": "/project/path/new_file",
"comment": "text",
"origin": "sitename"
}
Child node :
{
"_id": "55ae0b1ed81e88357d77d0e9",
"#parent" : "/project/path/new_file",
"comment": "text",
"origin": "sitename"
}
_idpath or an array of paths- key
originalphanumerical name without space, for ease of use. For multi-site.
optional keys (these keys are not mandatory but could ease multi sites configurations and version control):
file_mtimeNumber (milliseconds since 1 Jan 1970 00:00)file_sizeNumber (number of bytes)versionNumber
In a multi-site environment, the
originand_idpath are used to retrieve the file from the source server.
¶ unlock
Unlock a locked asset
unlock ( identifiers [, callback] )
¶ parameters
identifiers{string | array} : identifier(s) string(s) to lockcallback{function} (js only, optional) : if specified, the request is asynchronous
¶ return values
- returns a boolean or an array of booleans (depending on the input)
If the asset is not locked or locked for someone else (
lockkey value != authenticated user name) it returns false. If it was successfully unlocked, returns true.
¶ comment
Add a comment to one or several element(s)
comment ( elements [, callback] )
¶ parameters
elements{object | array} : object(s) specifying the elements identifiers and comment stringcallback{function} (js only, optional) : if specified, the request is asynchronous
¶ return values
- returns an element or an array of elements on success
null(Javascript) orNone(Python) on failure
Sets
authorkey on elements with the authenticated username as value, as well astimekey with the current time at creation.
# Python
# single parent id
>>> project.comment({"#parent" : "asset_id", "comment" : "text"})
{u'author' : u'username', u'time' : 1480588505449, u'#parent' : u'asset_id', u'comment' : u'text'}
# multiple parent ids
>>> project.comment({"#parent" : ["asset_id1", "asset_id2"], "comment" : "text"})
[{u'author' : u'username', u'time' : 1480588505449, u'#parent' : u'asset_id1', u'comment' : u'text'}, {u'author' : u'username', u'time' : 1480588505449, u'#parent' : u'asset_id2', u'comment' : u'text'}]
// Javascript
damas.comment({'#parent' : "asset_id", comment : "text"});
>> Object { author: "damas", time: 1480588505449, '#parent': "asset_id", comment: "text" }
// comment using an asynchronous call
damas.comment({'#parent' : "asset_id", comment : "text"}, function (node) {
// asynchronous mode
console.log(node.time);
});
¶ SyncPlanet-specific older documentation
¶ Global Parameters
The global configuration parameters are accessible through a JSON element that is used by the synchronization agents.
{
"_id": "rsyncGlobalParameters", /* generic parameter */
"enableScans": true, /* parameter for scans */
"excludePatternRules": "", /* parameter for scans */
"excludedFiles": "", /* parameter for scans */
"ignoreExistingFiles": false, /* parameter for scans */
"includePatternRules": "", /* parameter for scans */
"rsaPublicKey": "", /* key to install under volumes' .ssh/authorized_keys */
"sleepBetweenScans": 60, /* parameter for scans */
"syncEverywhere": true, /* [Destination Filtering](Destination-Filtering) */
"time": 1555711910004 /* generic parameter */
}
¶ Volumes
Each volume is described as an element which holds its configuration and options to control its behavior.
{
"_id": "sit/inthebox/mice/amopix", /* the volume identifier */
"agentStartTime": 1599731439000, /* agent time when launched */
"agentStatus": "on", /* status of the agent, on / off / undefined */
"agentVersion": "1.0.5", /* version of the running agent */
"author": "support", /* generic parameter */
"enableDiscovery": true, /* Boolean, filesystem scan switch */
"enableDiscoveryPackages": true, /* Boolean, filesystem scan switch */
"enableDiscoveryUpdatedFiles": true, /* Boolean, filesystem scan switch */
"enableDiscoveryDeletedFiles": true, /* Boolean, filesystem scan switch */
"enableEmission": true, /* Boolean, file emission switch */
"enableReception": true, /* Boolean, file reception switch */
"error": 0, /* Number, error encountered */
"exit_emission": 0, /* Number, error encountered */
"exit_reception": 0, /* Number, error encountered */
"exit_scan": 0, /* Number, error encountered */
"last_emission": 1571918493000, /* Number, last emission */
"last_reception": 1571910212000, /* Number, last reception */
"last_scan": 1571918421000, /* Number, last filesystem scan */
"lastConnectionAttempt": 1600942773000, /* Number, time of last connection attempt */
"lastConnectionStatusChange": 1600512139000, /* Number, time of previous status */
"lastConnectionSuccess": 1600942773000, /* Number, last successful connection to the volume */
"last_emission": 1600942573000, /* Number, last file emission time */
"last_reception": 1598770303000, /* Number, last file reception time */
"last_scan": 1600942965000, /* Number, last volume scan time */
"name": "amopix", /* String, volume's name */
"notifiedEmails": "" /* String, comma separated email addresses */
"rsync_args": "", /* String, arguments for rsync */
"rsync_env": "", /* String, environment for rsync */
"scan_duration": 14, /* Number, last filesystem scan duration */
"stderr_emission": "", /* String, stderr if emission error */
"stderr_scan": "", /* String, stderr if scan error */
"syncAmountEmission": 10, /* Number, Number of elements uploaded each time */
"syncAmountReception": 10, /* Number, Number of elements downloaded each time */
"time": 1563285980221, /* generic parameter */
"url": "ssh://user@adress:port/project/" /* String, URL of the volume */
}