Redesign for autosaving campaigns (WIP)

Developer discussion regarding MapTool 1.4

Moderators: dorpond, trevor, Azhrei

Forum rules
Posting now open to all registered forum users.
User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Redesign for autosaving campaigns (WIP)

Post by Azhrei »

This thread is to discuss features/options needed for autosaving campaigns in MapTool 1.4+.

From this posting:
Full Bleed wrote:I'd like to see granular control of the "Backup" feature in MT. That is, allow users to 1) set how often and 2) how many backups of their campaigns are made. Right now, the "Backup" feature in MT is voodoo magic (and not in a good way.) No one has any control over it, and when tragedy strikes, they just hope something useful might be in that directory when they go desperately looking to see if MT has been backing up their work.

As for "Auto Save", I think it needs to be renamed to what it is: "Crash Recovery"

Once you make "Backups" function like people expect backups to function, then they won't expect "Auto Save" (aka "Crash Recovery") to do something it was not designed to do.

Then link the two by only having two settings in MT:

A) How often do you want to make a Campaign Backup?
B) How many Campaign Backups do you want to keep?

Require at least 1 Backup file (default to or suggest a higher number).
When a backup is written, a tiny "Crash Recovery File" would be created with the backup's name in it.
When MT is closed properly, the Crash Recovery File flag would be deleted.
If MT loads and sees a crash recovery flag, it will ask if the user wants to load the backup file associated with it (basically, the way it looks for Autosave files now.) However, if it doesn't see the flag, that means MT closed properly and there is no need for "Crash Recovery".
So here's what I'm thinking of doing. Tell me how this sounds to you. (I have my own opinions about some of the discussion points, but I'm not going to voice them until there's been some discussion.)

1. The autosave feature will have a setting that controls how frequently the autosaves occur. If the user is manually performing a save when it's time for an autosave to occur, the autosave will wait for the manual save to finish and, as soon as it's completed, the autosave will occur.

Discussion point #1: When an autosave is delayed like this, should it just be skipped entirely? If autosaving is for 'crash recovery", then there's no need to recover anything right after a manual save.

Discussion point #2: If the decision from DP#1 is to skip the autosave because a crash recovery file isn't needed, what about previous autosaves? Should they be removed? Suppose there are four autosaves and the user manually does a save (and there's no autosave being delayed), are crash recovery files really needed at that point?

2. There should be a limit (or limits) on how many autosave files MapTool should be managing at any one given time. For example, the filename pattern could be autosave-number.cmpgn, but if these files are for crash recovery only, do we need more than one?

3. Autosaves are currently put under the ~/.maptool/autosave/ directory. Should there be an option to change this? The main concern is that autosave files should (ideally) be on a different partition from where the .maptool directory is kept.

Discussion point #3: Consider also that multiple campaigns may need autosave files. If autosave files always use the same filename pattern (as described in #2, above), how will MapTool be able to tell an autosave for one campaign from an autosave for another campaign? What happens if MapTool is running multiple times simultaneously, each instance working on a different campaign?

4. Should network shares be disallowed for autosave files?

Discussion point #4: Consider that network shares generally take longer to write to and, on Windows, hold exclusive write locks to open files. Corollary: what should MapTool do if the interval between autosaves elapses while a previous autosave is still running?

Discussion point #5: It's probably expected that the "interval" field for autosaves is the amount of time between the start of one autosave and the start of the next one. But perhaps it should be the interval from when the first one completes until the start of the next one? What happens then if the first autosave takes an extremely long time (a network timeout, for example, if writing to network shares is going to be allowed)?

User avatar
Full Bleed
Demigod
Posts: 4736
Joined: Sun Feb 25, 2007 11:53 am
Location: FL

Re: Redesign for autosaving campaigns (WIP)

Post by Full Bleed »

Azhrei wrote:1. The autosave feature will have a setting that controls how frequently the autosaves occur.
Check!
If the user is manually performing a save when it's time for an autosave to occur, the autosave will wait for the manual save to finish and, as soon as it's completed, the autosave will occur.

Discussion point #1: When an autosave is delayed like this, should it just be skipped entirely? If autosaving is for 'crash recovery", then there's no need to recover anything right after a manual save.
I do not think skipping the save would be a deal breaker... but I do suspect that you'll get reports of the saves not saving consistently if there are "holes" in the save record that were created when it collides with the manual saves.

That said, unlike a Manual Save, I've never had an issue with Autosave grabbing 100% of the computer's focus. It has always seemed to just do its thing without interrupting my games... so if it autosaved immediately after a Manual Save, and I didn't "feel it", I probably would not care.
Discussion point #2: If the decision from DP#1 is to skip the autosave because a crash recovery file isn't needed, what about previous autosaves? Should they be removed? Suppose there are four autosaves and the user manually does a save (and there's no autosave being delayed), are crash recovery files really needed at that point?
I would not remove them.

Having a number of autosaves will offer a secondary "time machine" feature where we could step back into the state of the campaign at various stages if necessary.
2. There should be a limit (or limits) on how many autosave files MapTool should be managing at any one given time. For example, the filename pattern could be autosave-number.cmpgn, but if these files are for crash recovery only, do we need more than one?
As I outlined, I don't think it should be just for "crash recovery". That's where we start running into problems.

"Crash Recovery" and "Auto Backup" are separate things. But most users *think* "Autosave" is "Auto Backup."

I would rename the whole process to "Auto Backup" and have Crash Recovery simply draft off of the Auto Backup feature as I suggested above.

Otherwise, you're really developing two separate features (neither of which currently works well right now.)
3. Autosaves are currently put under the ~/.maptool/autosave/ directory. Should there be an option to change this?
Being able to manually set the directory would be nice. Being able to set it to a faster, safer, drive would be a big plus.
The main concern is that autosave files should (ideally) be on a different partition from where the .maptool directory is kept.
Ideally, yes. But requiring people to choose this on install is going too far. It will stump many users. It has to have a single partition default.
Discussion point #3: Consider also that multiple campaigns may need autosave files. If autosave files always use the same filename pattern (as described in #2, above), how will MapTool be able to tell an autosave for one campaign from an autosave for another campaign? What happens if MapTool is running multiple times simultaneously, each instance working on a different campaign?
I think the Auto Backup names should be based on the campaign names so this wouldn't be a problem.

And, yes, this will inevitably lead to a request for an "Auto Clean-Up" feature. ;) With something like an option to "delete backups more than X months old" and maybe a size warning if the backup directory was exceeding a certain size.
4. Should network shares be disallowed for autosave files?
Discussion point #4: Consider that network shares generally take longer to write to and, on Windows, hold exclusive write locks to open files.
I wouldn't disallow it. Most probably won't have a problem. It'll be the user's responsibility to manage their permissions and performance with regard to using an optional network share location.
Corollary: what should MapTool do if the interval between autosaves elapses while a previous autosave is still running?
Warn (of the collision), Skip (the new save), Inform (that the save didn't happen and that changes should be made to address the issue.)
Discussion point #5: It's probably expected that the "interval" field for autosaves is the amount of time between the start of one autosave and the start of the next one. But perhaps it should be the interval from when the first one completes until the start of the next one? What happens then if the first autosave takes an extremely long time (a network timeout, for example, if writing to network shares is going to be allowed)?
I'd say the interval should start after the save finishes.

As for saves taking so long that they collide with other saves... there should probably be a timeout that is shorter than the interval (and, as above: warn, skip, inform.)


Thanks for opening this up for discussion.
Last edited by Full Bleed on Tue Sep 05, 2017 6:57 am, edited 2 times in total.
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."

User avatar
wolph42
Winter Wolph
Posts: 9999
Joined: Fri Mar 20, 2009 5:40 am
Location: Netherlands
Contact:

Re: Redesign for autosaving campaigns (WIP)

Post by wolph42 »

1. check.
FB wrote:That said, unlike a Manual Save, I've never had an issue with Autosave grabbing 100% of the computer's focus. It has always seemed to just do its thing without interrupting my games... so if it autosaved immediately after a Manual Save, and I didn't "feel it", I probably would not care.
I *did* notice the autosaves.Basically entire MT grinds to a halt, quite expectantly and remains unresponsive for 30s or so and then everything works again. It took me a while to figure out that it was the autosave... But as FB says IF you don't notice the autosave THEN there is not reason not to save even after the file has just been saved.

2. what FB said. (so join crash and backup). probably have a setting where you can set max amount of backups in total and per campaign. E.g. 10 total, 3 per campaign, would store campaignA.cmpgn (cA) as cA_1, cA_2 and cA_3, cB and cC in a similar way and simply constantly rename the files when a new one is save, pushing the old ones back (so _3 is always the last save). When campaign cD is introduced it pushes out the oldest stored campaign e.g. cA_1, then cA_2 and then it again cycles through cD_1,2 and 3.
I hope this makes sense.

I guess the biggest hurdle is to keep track of these in some matter, keeping in mind that one could be running two instances. But basically in the directory you have everything there, the campaign file, the last numbered name and the 'oldest' save. So this could work. Of course you could theoretically have interesting instances with two MT running, both saving to the same dir but one has max 5/2 and the other 10/3 which means that the former encounters a 'last save' _3 while in its 'world' the only possible last can be _2. Also one instance allows a max saves of 5 while the other 10...so if there are 6 files, what happens when 3/5 instance saves?
Other things could be that people start to manually rename files in that dir...etc.

3. obviously there should be that option, if not only because you would want it on a separate drive.

discussion#3: it does not, it keep the pattern described above. This means that if you did NOT change the store directory for the backups AND you have 2 instances running AND with 2 exactly the same campaigns AND you have set the max backup saves to less than 2 THEN yes you're screwed for one of the campaigns when MT crashes. (that and you're an idiot).

4. nope. Its bloody annoying when thats disallowed. At you're own risk!

User avatar
Jagged
Great Wyrm
Posts: 1306
Joined: Mon Sep 15, 2008 9:27 am
Location: Bristol, UK

Re: Redesign for autosaving campaigns (WIP)

Post by Jagged »

In my opinion the best auto backup (or crash recovery if you prefer) services work on a count down. The countdown starts after a modification is made and is reset by a manual save.

User avatar
Full Bleed
Demigod
Posts: 4736
Joined: Sun Feb 25, 2007 11:53 am
Location: FL

Re: Redesign for autosaving campaigns (WIP)

Post by Full Bleed »

wolph42 wrote:I *did* notice the autosaves.Basically entire MT grinds to a halt, quite expectantly and remains unresponsive for 30s or so and then everything works again. It took me a while to figure out that it was the autosave... But as FB says IF you don't notice the autosave THEN there is not reason not to save even after the file has just been saved.
Hmmm... I guess I've always had MT's data directories on fast drives (Raptors when I used mechanical and now SSDs)... otherwise an autosave every 5 minutes would have been prohibitive if it impacted games like you experienced.
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."

Elkad
Kobold
Posts: 5
Joined: Tue Apr 29, 2014 2:13 pm

Re: Redesign for autosaving campaigns (WIP)

Post by Elkad »

1. Autosaves should never be skipped. If you have to delay one because of a manual save, that's fine, but run it immediately after.
Previous autosaves should never be deleted just because of a manual save. They should always follow the retention rules.

2. Filename should be campaignname_autosave_00X.cmpgn
Number of autosaves should be Per Campaign File, default to something low-ish (4?), but be fully user configurable. If I want 99 autosaves, I want 99 autosaves. Maybe I want to use it for version control.

3. Save locations. Give me a checkbox for (maptool/autosave/) or (campaignfilelocation/autosave/). And an option to put in yet another path if I want. By including the filename in the autosave, multiple instances are handled (mostly) correctly. Letting me choose the location covers the instances where I'm running 2 copies from 2 directories with the same filename.

4. Network shares. No restrictions. My "network share" might be in the same box, because I'm running MT in a virtual machine. Or maybe I have 10gig networking and a big cache on the network drive, and it'll actually be faster. Besides, it's annoying.

User avatar
RPTroll
TheBard
Posts: 3159
Joined: Tue Mar 21, 2006 7:26 pm
Location: Austin, Tx
Contact:

Re: Redesign for autosaving campaigns (WIP)

Post by RPTroll »

Some sort of notification that an autosave is in progress as well as how long it is taking. (bottom of screen?)
If Autosaves are taking longer than X seconds, somehow notify the GM to let them decide what autosave features to enable. (could be a tooltip on the indicator above)
While new autosave functionality is being worked out - a timer to the next autosave would be handy (could be combined with the first line above).

Delay/disable a manual save if an autosave is in progress.
ImageImage ImageImageImageImage
Support RPTools by shopping
Image
Image

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Redesign for autosaving campaigns (WIP)

Post by Azhrei »

Great stuff, folks. Thanks. :)

There are some issues in terms of how things should be done given the answers above, but I'll wait another day or two to see if anyone else wants to chime in, then I'll respond.

PS to FB: I found a couple of bugs in how autosaving currently works and have fixed those, so if a build comes out before this rewrite is finished, at least the existing autosave process has been cleaned up.

User avatar
JamzTheMan
Great Wyrm
Posts: 1872
Joined: Mon May 10, 2010 12:59 pm
Location: Chicagoland
Contact:

Re: Redesign for autosaving campaigns (WIP)

Post by JamzTheMan »

For #2, assuming we keep x saves, I'd like to propose not saving as .cmpgn file/format.

Or rather, how about storing them all in a single file? saving 10 copies of the same 200mb file can eat up space. But 10 copies should compress well considering the considerable overlap. I would then like to keep backups of each campaign. Again, assuming we keep for x days vs per instance of mt.

Which also should be considered, 2+ copies of mt running...

As a side note, how about option to set compression level. It may be nice to have faster saves using store vs high level compression? Maybe this is a setting for the campaign and normal saving instead of backup/recovery?

Lastly, macro save? Auto save during gm prep is one thing to prevent loss of work. During play, I like saving before/after combat. But stopping play for 30 seconds can kill a mood. If my initiative macro started an autosave in the background, that would be a peace of mind...
-Jamz
____________________
Custom MapTool 1.4.x.x Fork: maptool.nerps.net
Custom TokenTool 2.0 Fork: tokentool.nerps.net
More information here: MapTool Nerps! Fork

taustinoc
Dragon
Posts: 516
Joined: Mon Aug 03, 2015 6:30 pm

Re: Redesign for autosaving campaigns (WIP)

Post by taustinoc »

JamzTheMan wrote:Or rather, how about storing them all in a single file?
I see the appeal, but it's an additional failure mode. If that one file gets corrupted, all your autosaves are gone. If they're separate files, there is at least a possibility that some are still good.
JamzTheMan wrote:Lastly, macro save? Auto save during gm prep is one thing to prevent loss of work. During play, I like saving before/after combat. But stopping play for 30 seconds can kill a mood. If my initiative macro started an autosave in the background, that would be a peace of mind...
Perhaps a keyboard shortcut or on screen button to turn autosave on and off quickly? Or (or, rather, and), as you suggest, the ability to do so with a macro.

paulstrait
Dragon
Posts: 304
Joined: Mon Mar 23, 2009 4:48 pm

Re: Redesign for autosaving campaigns (WIP)

Post by paulstrait »

I don't think that we necessarily need more than one, but I strongly believe that it shouldn't be automatically deleted until it is ready to make another one (rather than as soon as it is used, because what do you do if it crashes right away?)

User avatar
Full Bleed
Demigod
Posts: 4736
Joined: Sun Feb 25, 2007 11:53 am
Location: FL

Re: Redesign for autosaving campaigns (WIP)

Post by Full Bleed »

Azhrei wrote:PS to FB: I found a couple of bugs in how autosaving currently works and have fixed those, so if a build comes out before this rewrite is finished, at least the existing autosave process has been cleaned up.
Excellent. That's an important bug fix.

Btw, since you've been poking around in there, how is the current backup feature supposed to be working?
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Redesign for autosaving campaigns (WIP)

Post by Azhrei »

Full Bleed wrote:Btw, since you've been poking around in there, how is the current backup feature supposed to be working?
There's one save file, and the timer is reset when the autosave completes (so a 10 minute interval is actually 10 minutes plus whatever time is needed to perform the autosave). If a manual save is underway when an autosave is supposed to start, the autosave is canceled. If an autosave has started when the user tries to manually save, a message appears in the chat panel that there's an autosave in progress and to try again.

(There were two bugs, one regarding the interval timer not always being reset when the autosave completed, and the other dealt with a race condition in which two saves could end up running simultaneously -- if you thought it was slow waiting for a large campaign to be saved, try doing two of them at the same time! :roll: )

User avatar
Full Bleed
Demigod
Posts: 4736
Joined: Sun Feb 25, 2007 11:53 am
Location: FL

Re: Redesign for autosaving campaigns (WIP)

Post by Full Bleed »

I was referring to the "backup" save file... which we don't seem to be able to set...

I don't see how we know the default conditions for it... when it saves... when it deletes... how it handles multiple campaign files or instances... etc. Not sure how (or if) it interacts with the autosave.
if you thought it was slow waiting for a large campaign to be saved, try doing two of them at the same time!
Yipes.
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Redesign for autosaving campaigns (WIP)

Post by Azhrei »

Full Bleed wrote:I was referring to the "backup" save file... which we don't seem to be able to set...
You're correct, those things are completely opaque right now. There's only one file -- no rotating files, for example -- and no options for setting the autosave name. If two instances of MT try to autosave at the same time, I have no idea what happens, but I'm sure it's spectacular. :roll:

(Hm, makes me think we should have some kind of "file rotation" service within MT.)

In any case, right now we're focused on requirements gathering in regards to autosave functionality. :)

Post Reply

Return to “MapTool 1.4”