Redesign for autosaving campaigns (WIP)

Developer discussion regarding MapTool 1.4

Moderators: dorpond, trevor, Azhrei

Forum rules
Posting now open to all registered forum users.
User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Redesign for autosaving campaigns (WIP)

Post by Azhrei »

Alright, so here are some issues...

Full Bleed: I would rename the whole process to "Auto Backup" and have Crash Recovery simply draft off of the Auto Backup feature as I suggested above.
This isn't 100% clear to me, but how about this:

The first/most recent autosave file will be called recovery.cmpgn, then older versions of that file will be named autosave-number.cmpgn.

If the interval specified in Preferences is non-zero, the former file will be created at the given interval. A second field in Preferences will control how many should be created and if the user selects 1, they only get that file; if they select >1, the previous file is renamed as described above.

what do you think? How would you change that?

Being able to manually set the directory would be nice. Being able to set it to a faster, safer, drive would be a big plus.
What's the best way to do this? If we store the directory name in the Preferences, it would apply to all campaigns. If we store it in the campaign, it would be specific to each campaign. I see pros and cons each way, but I don't think it's a good idea to provide options for both! (For example, specify the default directory in Preferences and then let a specific campaign override that default.) I'm with Craig on something like this -- too many options is just confusing and besides, how many people would actually use this feature? I know I wouldn't.

Elkad: Number of autosaves should be Per Campaign File, default to something low-ish (4?), but be fully user configurable. If I want 99 autosaves, I want 99 autosaves. Maybe I want to use it for version control.
Well, you might want to use it for version control, but then you're using it the wrong way. ;)

There are plenty of other ways to implement version control and that's not what we're attempting to do with this. Besides, MT automatically taking snapshots periodically doesn't give you any clue where you were at when the autosave occurred.

I think the Auto Backup names should be based on the campaign names so this wouldn't be a problem.
Then you won't get any autosaves until you've manually saved the campaign at least once, because until then there is no campaign name. You would get your recovery file, but that would be it (and it wouldn't work if you have two instances of MT open). Is that what you want?

Give me a checkbox for (maptool/autosave/) or (campaignfilelocation/autosave/). And an option to put in yet another path if I want.
This is what I'm thinking, too. If the autosaves were under .maptool/autosaves/ and each campaign had it's own subdirectory under that, then you'd be able to tell at a glance which campaigns had autosaves and which didn't. It would be easy to clean up as well ("We finished TombOfHorrors, so I can delete everything under that directory.") and make it easy to store them somewhere else (using a symbolic link for the entire .maptool/autosaves/ directory, for example).

The problem is that we don't have a campaign name until the user does their first save. I'm thinking we should use some generic string in that case (like "unknown" + task ID), so that multiple instances of MT don't step on each other's toes. (I doubt anyone is going to have two unsaved campaigns working simultaneously, but if we don't account for it in the code then users will simply report problems that occur as bugs. Much better to consider these things and include the necessary if-statements or whatever to prevent them in the first place. :))

And, yes, this will inevitably lead to a request for an "Auto Clean-Up" feature. ;) With something like an option to "delete backups more than X months old" and maybe a size warning if the backup directory was exceeding a certain size.
Those types of options will likely never be implemented as part of MT! I have no issue adding a count limit (no more than 10, for example), but adding time or size expirations means people complaining if MT doesn't delete something even though they haven't run MT for days or weeks. The counter limit is easy to implement and is reliable and predictable. :)

It'll be the user's responsibility to manage their permissions and performance with regard to using an optional network share location.
Yeah, that would be my inclination, but Windows is pretty brain-dead when it comes to opening files.

Warn (of the collision), Skip (the new save), Inform (that the save didn't happen and that changes should be made to address the issue.)
That sounds like a plan.

I'd say the interval should start after the save finishes.
Okay.

As for saves taking so long that they collide with other saves... there should probably be a timeout that is shorter than the interval (and, as above: warn, skip, inform.)
Hm, good plan. There's no reason that an overlapping autosave couldn't produce the same warn/skip/inform sequence. (And I'm sure Jamz will want to create an "overlapping save" event that will invoke a macro! :wink:)

wolph42: I *did* notice the autosaves. Basically entire MT grinds to a halt, quite expectantly and remains unresponsive for 30s or so and then everything works again.
Wow. I can tell you're going to be one of the prime testers, then! I don't see why it would be so bad, but there are a few things that I can do to help ameliorate it -- dropping thread priority, intermittently adding yield(), and maybe other stuff. (See this page if you're interested in reading about some of the complexities involved.)

Jagged: In my opinion the best auto backup (or crash recovery if you prefer) services work on a count down. The countdown starts after a modification is made and is reset by a manual save.
I agree!! But unfortunately, MT doesn't have a single point at which to determine that "something has changed". It will need something like that in the future, though, since we want token data to eventually be split off from the image that appears on the screen. If we implement that properly, we can send only the information that changes from client<->server and save a lot of bandwidth/effort. It can also eliminate some of our current "one token updated on two systems" concurrency issues.

RPTroll: Some sort of notification that an autosave is in progress as well as how long it is taking. (bottom of screen?)
If Autosaves are taking longer than X seconds, somehow notify the GM to let them decide what autosave features to enable. (could be a tooltip on the indicator above)
While new autosave functionality is being worked out - a timer to the next autosave would be handy (could be combined with the first line above).

Wow, I like all of these ideas!

I can easily arrange a notification to appear in the chat window, but I think a progress bar in the status bar at the bottom of the window would be much better. I don't think I can accurately estimate a percentage while the save is occurring, though. At the time the save happens, I don't know what the file size will be, nor do I know how many components will need to be saved, since they are saved on the fly as they are referenced by the maps. I suppose a bar that swipes left and right will be enough, since you'll know an autosave is running. (Maybe the actual file writing can be changed in the future so that a true percentage could be calculated.)

The progress bar area could include some status information when the autosave is not running, like "how long to the next save, how long did the last save take, how much space did the last save consume."

Wow, good stuff everyone. Thanks for the input. :)

(I'm hunkered down for Hurricane Irma. Don't worry, though; I have my off-site Git repositories of MapTool code fully synced. ;))

User avatar
RPTroll
TheBard
Posts: 3159
Joined: Tue Mar 21, 2006 7:26 pm
Location: Austin, Tx
Contact:

Re: Redesign for autosaving campaigns (WIP)

Post by RPTroll »

Stay safe, my friend.
ImageImage ImageImageImageImage
Support RPTools by shopping
Image
Image

taustinoc
Dragon
Posts: 516
Joined: Mon Aug 03, 2015 6:30 pm

Re: Redesign for autosaving campaigns (WIP)

Post by taustinoc »

A couple of thoughts:

If you put the save path in the campaign file, you'll get bug reports from shared campaign files, when the path points to a location that doesn't exist on the computer it's been opened on. Perhaps there's a clever way to check for that, and maybe open a "Save As" window when it happens.

And on the autosave file names, if you include the campaign name in the file name, you'll end up with a growing number of autosaves as new campaigns are created, megabytes each, that never get deleted unless it's done manually. Maybe just some kind of startup notice to the effect of "You have autosave files older than date (or more than XXX megabytes), would you like to whatever seems like the right thing to offer (like show a list to delete, or open the save folder, or something). Seems like there should be some provision for informing the user that they're using up more and more disk space.

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Redesign for autosaving campaigns (WIP)

Post by Azhrei »

taustinoc wrote:If you put the save path in the campaign file, you'll get bug reports from shared campaign files, [...]
Good point, and another reason to put the path in Preferences. I suppose the name could be forced to be relative to .maptool/autosaves/ but any such wizardry can still be broken.
And on the autosave file names, if you include the campaign name in the file name, you'll end up with a growing number of autosaves as new campaigns are created, [...]
Another good point. Consider what happens if I have a large campaign, then use "Save As..." to give it a new name. A lot more autosaves are created and the whole pruning problem comes back.

I'm beginning to think that autosave files should be deleted when MT is closed. They would be strictly usable as "crash recovery" and nothing else. If the user wants multiple save files, they should save them on their own. (Just like users will be expected to handle saving to network shares themselves. :))

Otherwise, multiple instances of MT create a growing number of autosave files and no good way to remove them without building some kind of expiration system. That feels like a bad idea to me.

So... MT would use its task ID in all autosave files that it creates. The first would be recover-taskID.cmpgn and at each interval, they would be renamed to autosave-taskID-number.cmpgn. When MT exits cleanly, all such files are deleted. If MT crashes, those files would still be there and when you start MT again, it locates all recovery files, presents a list to the user, and asks if any should be recovered. Options are Yes (recover and delete), No (don't recover but still delete), and Cancel (don't do anything and terminate). When deleting, all files with the same task ID would be deleted, cleaning up everything from the crash.

Now there's never an issue with naming of the files (no conflict with multiple instances of MT), no problem with files that grow unbounded in quantity or size (since they're all removed), and no requirement that the user save their campaign once in order for crash recovery/autosave to work. It would not be called autosave anywhere -- it would be called "crash recover". A user that knows what they're doing can copy their autosave files somewhere else, but then such a user would just save the campaign using the menu.

taustinoc
Dragon
Posts: 516
Joined: Mon Aug 03, 2015 6:30 pm

Re: Redesign for autosaving campaigns (WIP)

Post by taustinoc »

Azhrei wrote:I'm beginning to think that autosave files should be deleted when MT is closed. They would be strictly usable as "crash recovery" and nothing else.
There is an appeal to the idea of auso-saves, but it would require more thought than "just keep x number of auto-saves of each campaign file." It could as simple as adding a "do you want to delete auso-saves" prompt on exit, or as complicated as a "manage auto-saves" function, but you are absolutely right that crash recovery and auto-saves are different functions, and have different requirements that can easily conflict. Maybe it should be a completely separate feature.

User avatar
Full Bleed
Demigod
Posts: 4736
Joined: Sun Feb 25, 2007 11:53 am
Location: FL

Re: Redesign for autosaving campaigns (WIP)

Post by Full Bleed »

Azhrei wrote:I'm beginning to think that autosave files should be deleted when MT is closed. They would be strictly usable as "crash recovery" and nothing else. If the user wants multiple save files, they should save them on their own. (Just like users will be expected to handle saving to network shares themselves. :))
I think you're losing ground here... Basically, you're pretty much just saying, "Nevermind about this redesign thing, I'll just fix what we have." ;)

Which is fine. But then you need to start a new thread for: "Redesign for Auto-Backup."

I really think that there is a simpler way, noted in my original post, to roll both features into one system... but if you want to keep them separate, fix the crash recovery (aka "auto save"), and then revisit the functionality of the backup system so that it's actually useful... that still sounds like progress.
Otherwise, multiple instances of MT create a growing number of autosave files and no good way to remove them without building some kind of expiration system. That feels like a bad idea to me.
How is an imperfect expiration system somehow *worse* than no backup redundancy at all?

Some of us care more about redundancy, not losing campaign data, and not losing game progress more than some drive "space". Heck, I have a campaign file (actually two campaigns that I eventually merged into the same campaign file) dating back to 2010... 500+ copies... 24GB. I have a few rules I've followed over the years... if the date changes while I'm developing/prepping... I rename the file... I rename and save the file into a "begin session" version... and an "end session" version... etc. I know I'm willing to spend some GB's adding some additional protection to my games. Space is cheap. My time isn't.
A user that knows what they're doing can copy their autosave files somewhere else, but then such a user would just save the campaign using the menu.
The truth is that the people that will benefit the most from useful auto-save/crash recovery and auto-backup features are the users that "don't really know what they are doing." The ones who aren't particularly good at being so anal about creating backups, renaming campaign files, etc. MT doesn't owe those people any protections against their ignorance and bad habits... but they sure will appreciate it. :)
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."

User avatar
wolph42
Winter Wolph
Posts: 9999
Joined: Fri Mar 20, 2009 5:40 am
Location: Netherlands
Contact:

Re: Redesign for autosaving campaigns (WIP)

Post by wolph42 »

I agree with FB, MT should create backups and I do see your point: auto-save and backups should be separate functions.
Azhrei wrote:So... MT would use its task ID in all autosave files that it creates. The first would be recover-taskID.cmpgn and at each interval, they would be renamed to autosave-taskID-number.cmpgn. When MT exits cleanly, all such files are deleted. If MT crashes, those files would still be there and when you start MT again, it locates all recovery files, presents a list to the user, and asks if any should be recovered. Options are Yes (recover and delete), No (don't recover but still delete), and Cancel (don't do anything and terminate). When deleting, all files with the same task ID would be deleted, cleaning up everything from the crash.
you're forgetting the part when someone runs two instances...

A simple an VERY RL example: beamer-laptop setup. This often means a dual screen setup where the gm runs two instances: master and slave (server/gm and client/player instance) where the gm runs server and player is logged in.
As both MT versions are on the same pc, they will auto-save whatever they have on a regular basis. Now since its likely that the master was started first and the slave a bit later, this means that most of the time the auto-save will be that of the slave (when master save is created then a tiny bit later the slave save will overwrite it). Obviously the slave save is useless on a crash.

Note that it can also be server/player and client/gm. So 'master' really means: server!

Perhaps this particular example could be fixed by letting slaves not save...but I'm not sure of the consequences.
And of course there are other examples, e.g. working on my framework and opening up bot framework cause I 'need to look something up'. One instance crashes and I end up with the wrong auto-save... you get the picture.

User avatar
Full Bleed
Demigod
Posts: 4736
Joined: Sun Feb 25, 2007 11:53 am
Location: FL

Re: Redesign for autosaving campaigns (WIP)

Post by Full Bleed »

Azhrei wrote:The first/most recent autosave file will be called recovery.cmpgn, then older versions of that file will be named autosave-number.cmpgn.

If the interval specified in Preferences is non-zero, the former file will be created at the given interval. A second field in Preferences will control how many should be created and if the user selects 1, they only get that file; if they select >1, the previous file is renamed as described above.

what do you think? How would you change that?
I'll take this.

But I'd rather have it have the campaign name in there. Yes, I've already acknowledged that it will blow up the backup/autosave/recovery (whatever this it) directory over time... but that's a toll I'm more than willing to pay.
Being able to manually set the directory would be nice. Being able to set it to a faster, safer, drive would be a big plus.
What's the best way to do this? If we store the directory name in the Preferences, it would apply to all campaigns. If we store it in the campaign, it would be specific to each campaign. I see pros and cons each way, but I don't think it's a good idea to provide options for both! (For example, specify the default directory in Preferences and then let a specific campaign override that default.) I'm with Craig on something like this -- too many options is just confusing and besides, how many people would actually use this feature? I know I wouldn't.
Well, that's addressing multiple use cases. You brought up being able to set network shares... and said:
"The main concern is that autosave files should (ideally) be on a different partition from where the .maptool directory is kept."
I just offered support. ;)

When Wolph talked about his long auto-save issue I thought, "Maybe if he could easily set it's location to a faster option that would fix his problem."

Right now I basically get this functionality by creating a custom bat with a unique maptool data directory for *each* of my campaigns. I started doing this, initially, to save the different screen layouts. Having control where the respective backups and autosaves were was just a side benefit. Most users, however, simply don't have that functionality because they don't know how to make a custom bat file with a custom data directory. But that doesn't mean that it isn't something many users wouldn't like to take advantage of if it was easy.
Elkad: Number of autosaves should be Per Campaign File, default to something low-ish (4?), but be fully user configurable. If I want 99 autosaves, I want 99 autosaves. Maybe I want to use it for version control.
Well, you might want to use it for version control, but then you're using it the wrong way. ;)
Well, sure. But that doesn't mean you can't develop a system that kills more than one bird.
Then you won't get any autosaves until you've manually saved the campaign at least once, because until then there is no campaign name. You would get your recovery file, but that would be it (and it wouldn't work if you have two instances of MT open). Is that what you want?
Better than nothing... and something of a corner case. Besides, isn't there already a default: "Grasslands"? Why not use the Grasslands name for unnamed backups until it was renamed to something more accurate.

Edit: As for multiple instances of unamed campaign files (or even same named campaign files)... see my post after this one.
RPTroll: Some sort of notification that an autosave is in progress as well as how long it is taking. (bottom of screen?)
If Autosaves are taking longer than X seconds, somehow notify the GM to let them decide what autosave features to enable. (could be a tooltip on the indicator above)
While new autosave functionality is being worked out - a timer to the next autosave would be handy (could be combined with the first line above).

Wow, I like all of these ideas!

I can easily arrange a notification to appear in the chat window,
It currently indicates when it's happening in the status bar... any reason to change that?
but I think a progress bar in the status bar at the bottom of the window would be much better. I don't think I can accurately estimate a percentage while the save is occurring, though. At the time the save happens, I don't know what the file size will be, nor do I know how many components will need to be saved, since they are saved on the fly as they are referenced by the maps. I suppose a bar that swipes left and right will be enough, since you'll know an autosave is running. (Maybe the actual file writing can be changed in the future so that a true percentage could be calculated.)
Yeah... that is kind a fluff feature. Might not be worth the engineering needed to have it be accurate.

The progress bar area could include some status information when the autosave is not running, like "how long to the next save, how long did the last save take, how much space did the last save consume."
Neat idea.
(I'm hunkered down for Hurricane Irma. Don't worry, though; I have my off-site Git repositories of MapTool code fully synced. ;))
Oh yeah... forgot you lived in the Tampa area. My mother lives in the Sarasota/Bradenton area. It sounds like it has been quite a ride...
Last edited by Full Bleed on Mon Sep 11, 2017 5:00 am, edited 1 time in total.
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."

User avatar
Full Bleed
Demigod
Posts: 4736
Joined: Sun Feb 25, 2007 11:53 am
Location: FL

Re: Redesign for autosaving campaigns (WIP)

Post by Full Bleed »

I just thought of something on the issue of using the campaign name in the backup/autosave file name and have file renames generating a bunch of new backups...

Why not have a field to set a static prefix or suffix that will append to any backup file name? Then have MT only look for that id when doing the backups.

Something like:

Winter's last gasp-09-11-2017-(Seasonal Sins-1).autosave
Summer's first breath-09-11-2017-(Seasonal Sins-2).autosave

Someone would have to go out of their way to change that static field to generate more backups than anticipated.

As for concurrency issues between two MT instances, with the same file name open... I guess you could create a "file open flag" in the autosave directory... and if the second instance sees it, have it disable autosave (and make note of it on the status bar.)
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."

taustinoc
Dragon
Posts: 516
Joined: Mon Aug 03, 2015 6:30 pm

Re: Redesign for autosaving campaigns (WIP)

Post by taustinoc »

Full Bleed wrote:How is an imperfect expiration system somehow *worse* than no backup redundancy at all?
That depends on how much hard drive space you have (granted, most people have a lot, but not everybody), and whether or not you have the expertise to figure out why you've run out (and most people do not have that).

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Redesign for autosaving campaigns (WIP)

Post by Azhrei »

taustinoc wrote:[...] and whether or not you have the expertise to figure out why you've run out (and most people do not have that).
Bingo.

So I've got those two bugs mentioned earlier fixed and the "crash recovery" scheme only needs one more thing for it to work properly when there are multiple MT campaigns open simultaneously. (MT needs to delete the crash recovery file when it closes cleanly, and it needs to scan for recovery files and give the user a chance to recover them. It already does this, but if there were multiple MT campaigns open, there could be multiple recovery files, so the pop ups need to be adjusted for that.)

I will make that additional change and then I'll close issue#144 and consider crash recovery complete. And I'll write up some documentation for how it works. I've been thinking we need a "glossary" on the Help menu, so maybe I'll put it there or maybe it'll be added to the wiki. We'll see what makes more sense.

I'll open a separate issue to cover having multiple automatic backups. I'll come back to that when I get the crash recovery stuff done.

Note to wolph: there is no problem for multiple MT instances because only servers will generate recovery files. Even if you don't manually "Start a Server", MT still has one running -- it's called a "personal server" in the source code -- which means there will always be a recovery file. But when a client connects to a remote server, the personal server is turned off, thus clients won't generate recovery files. I apologize if my references to "multiple instances of MT" confused that issue!

PS: We fared very well in the face of the hurricane. We got some gusts of up to 90 mph, but we haven't been hurt, our power never went out, and we didn't have any damage to our home. Others in Florida weren't so lucky; I've seen reporting that said 3.4 million "customers" were without power and the census says there are 6.6 million "households". There's not a 1:1 correspondence between those terms, but it certainly sounds like a LOT of people are without power. We dodged a bullet this time.

taustinoc
Dragon
Posts: 516
Joined: Mon Aug 03, 2015 6:30 pm

Re: Redesign for autosaving campaigns (WIP)

Post by taustinoc »

Azhrei wrote:I'll open a separate issue to cover having multiple automatic backups. I'll come back to that when I get the crash recovery stuff done.
I think that's a wise choice. One thing at a time. To quote Charles Emerson Winchester, " I do one thing at a time, I do it very well, and then I move on."
Azhrei wrote:PS: We fared very well in the face of the hurricane.
That's good to hear.

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Redesign for autosaving campaigns (WIP)

Post by Azhrei »

taustinoc wrote:I think that's a wise choice. One thing at a time. To quote Charles Emerson Winchester, " I do one thing at a time, I do it very well, and then I move on."
Hah-hah, another M*A*S*H fan!

:D

User avatar
Full Bleed
Demigod
Posts: 4736
Joined: Sun Feb 25, 2007 11:53 am
Location: FL

Re: Redesign for autosaving campaigns (WIP)

Post by Full Bleed »

taustinoc wrote:
Full Bleed wrote:How is an imperfect expiration system somehow *worse* than no backup redundancy at all?
That depends on how much hard drive space you have (granted, most people have a lot, but not everybody), and whether or not you have the expertise to figure out why you've run out (and most people do not have that).
Kind of a soft concern. There are so many ways to address this... and most wouldn't even be that hard.

But no reason to spend any more time on it... it looks like the "redesign" is going to just be a fix. Which, as mentioned, is fine by me. I think it's an important fix and am glad someone (Az!) has taken the time to do it.
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."

taustinoc
Dragon
Posts: 516
Joined: Mon Aug 03, 2015 6:30 pm

Re: Redesign for autosaving campaigns (WIP)

Post by taustinoc »

Azhrei wrote:
taustinoc wrote:I think that's a wise choice. One thing at a time. To quote Charles Emerson Winchester, " I do one thing at a time, I do it very well, and then I move on."
Hah-hah, another M*A*S*H fan!

:D
Is there anyone who isn't?

Post Reply

Return to “MapTool 1.4”