Like all database files, Quicken files should be closed before backing them up. If Quicken has an "Export" or "Save as..." option, it would be best to use these to save the data in a separate file, then have SpiderOak back up the separate file. Otherwise, close the Quicken application before you back up the Quicken file.
Again like all database files, Quicken files should not be synced. They can be backed up if the above precautions are followed by databases should not by synced.
Quicken uses different file formats on Mac and Windows. On Mac, Quicken uses the package file format. A package "file" is in fact a folder that the Mac Finder and certain other software shows as if it were a file. Our job however is to faithfully back up and display files and folders without manipulation, so SpiderOak will show you what the "file" really is. When restoring Quicken files for Mac, or indeed any package file, you should restore the entire folder as presented by SpiderOak as a single unit.
A Quicken file for Mac will not open in Quicken for Windows, and a Quicken file for Windows will not open in Quicken for Mac. More details on the Mac-to-Windows issue are available from Intuit's website at Converting Quicken for Mac files to Quicken for Windows.