FEATURE: MapTool Database

[RPTroll]

I agree with Craig that an external data source adds a level of dependency that would cause headaches. I do like the idea for large framework data, though.

Cleanest codewise is a single database within the MapTool server. It will also decrease development time on the front end. As Jamz says, KISS it in the beginning.

Completely agree with the caching statement. Let the database do it. Most of the frequently requested info is going to state in memory. However, things will slow down if a macro has to make a call to a centralized database vs. pulling a token property and some of those properties are pretty dynamic. In some ways this may be unavoidable but a local data cache would test to see if there is a new timestamp and update as needed. I guess when I start thinking in circles it's time to quit.

[JamzTheMan]

Well, true, I'd hate my game to have to stop because I lost the connection to the central database. We could make it download/update from the central database, but at that point we could do that with flat files or anything else.

I do want to see a (eventually) the components broken apart like Craig said earlier so a "campaign" can be more independent of a "framework". It really is a PITA to upgrade, say, LM's pathfinder framework when you have mods on top of it...

[RPTroll]

How's this for a concept - It seems that token properties may still need to be truth for the character specific data like stats. The server database (I use the term loosely) should be used for the campaign lists, effects, and so on. Restated, the DB holds framework data while the token holds character data.

[Azhrei]

How's this for a concept - It seems that token properties may still need to be truth for the character specific data like stats. The server database (I use the term loosely) should be used for the campaign lists, effects, and so on. Restated, the DB holds framework data while the token holds character data.

Well, if the db is really just a replacement for MT's tables in the beginning, this is what we'll have, right?

If the plan is still to keep "macros" -- meaning JavaScript code blocks -- on the server, then perhaps only the server needs the db. As Craig said, MT has traditionally copied everything across the network on first connection, but that's clearly not a very scalable approach (as many of you already know). Any client that needs db information would cache it locally and just ask the server "has it changed?" This is what web browsers do now with images and other downloaded information. Since we don't want to carry a bunch of timestamps around, the client generates a hash and asks the server, "is this what you have?" This keeps network traffic down.

Is it possible that the hash could be the same even though the data is different? Sure, but it's pretty unlikely with a good/long hash, especially if you include a "generation number" in the hash (a monotonically increasing value; think "simple timestamp").

Starting with a read-only db that mimics how tables currently work is probably the best starting point. And there are no "has it changed" issues, since the data is read-only.

[RPTroll]

I agree.

Is there a way to split the DB portions of this into a new topic under the 1.4 forum or do we want to just crank up a new topic there?

[JamzTheMan]

I say leave the assignment listed with links back to here for now. Once someone picks that up as an owner they can start a new topic and drive the discussion?

[Azhrei]

I agree.

Is there a way to split the DB portions of this into a new topic under the 1.4 forum or do we want to just crank up a new topic there?

Yes, as a moderator you should be able to split the topic. When you split it, you can specify the location of the new piece.

Remember that the 1.4 forum is currently read-only for most users -- should we change that? I was originally thinking that it would be for developers and contributors to discuss technical ideas but maybe with feedback from users in the form of polls, but I'm happy to make it full access for everyone as well. I guess it kind of depends on what you see the new forum being used for. Maybe the top-level forum should be read-write for users and there should be a developer-only subform that is read-only?

I'm easy either way.

[JamzTheMan]

I agree.

Is there a way to split the DB portions of this into a new topic under the 1.4 forum or do we want to just crank up a new topic there?

Yes, as a moderator you should be able to split the topic. When you split it, you can specify the location of the new piece.

Remember that the 1.4 forum is currently read-only for most users -- should we change that? I was originally thinking that it would be for developers and contributors to discuss technical ideas but maybe with feedback from users in the form of polls, but I'm happy to make it full access for everyone as well. I guess it kind of depends on what you see the new forum being used for. Maybe the top-level forum should be read-write for users and there should be a developer-only subform that is read-only?

I'm easy either way.

I think we need a read-only forum, at least in part. We definitely need/want feed back in part, but we also need just a small area so we can collect our thoughts and assignments and report/update on tasks. Rest of community probably doesn't need to see it (I have no problem with them seeing it, just may be boring to read the 20th post on some technical aspect).

So maybe a Public area (so they can view respond to new updates and see features added that we want some feed back on the UI).
And a subforum that is read only and subscriber only (any one can subscribe), that we can use to talk on technical going on's and project related tasks/issues/etc. And if users don't want to clutter up there "Unread" forum stuff, they don't have to.

On a side note, I kinda wish the Looking for Group was subscriber only for same reason, I don't really need to see all the campaign requests/game updates for online games. But that's just me. Maybe I'm anti-social

[aliasmask]

I can't say what is best, but I'm thinking the db to only exist on the server machine in an encrypted file. I think that's what the mote team did with H2. But then again, having a copy on the clients or actually copies of the tables created on the clients would be good.

[RPTroll]

Just a few trollish thoughts about an implementation path for MapTool DB. My goal is to keep the time between code releases small while still giving the users new, needed functionality.

0th step: pick a technology and code implementation strategy
1st step: tables use database
2nd step: framework callable tables
3rd step: server uses database to populate token properties. Changes to tokens are pushed to db from server. (orm?)
4th step: all data lives in database and is pulled on demand

[Craig]

Just a few trollish thoughts about an implementation path for MapTool DB. My goal is to keep the time between code releases small while still giving the users new, needed functionality.

0th step: pick a technology and code implementation strategy
1st step: tables use database
2nd step: framework callable tables
3rd step: server uses database to populate token properties. Changes to tokens are pushed to db from server. (orm?)
4th step: all data lives in database and is pulled on demand

This mostly sounds good -- I will explain mostly just a little below.
For 0th step I know No SQL/Document databases are all the rage these days but I think we should stick to a relational DB as as that seems to match the features we want the best. As for which one, obviously one that allows us to locate the DB anywhere we want on the file system and not only in a specific place...

1st step, yeah this makes sense.
2nd step, fairly early on in the build process (still single digits) I want start working on JS macros, and add a way for MTScript to call JS macros/exposed functions/what have you, and at that point we should focus more on the JS macros and a nicer cleaner start. (Yeah I know boo hiss from the crowd, but I did say MTScript should be able to call JS Macros), eventually I could see some of the more advanced call back stuff being only JS macros though.
3rd Step I am good with this up to the parentheses.
4th all good.

Now for the bit that doesn't sound so good to me its those three letters ORM. I think this stuff is so simple ORM is just making things more complex and slower and less flexible and more of a pain to set up and harder to debug and harder to recover from errors. The SQL required for this is so simple that if we can't manage that we cant manage any ORM.

[wolph42]

as long as mt supports mt script you have my blessing. mixing in js in the total and with that allowing for speed and versatility sounds like a great plan.

[RPTroll]

Got it on the ORM but thought I'd check. OK. So now for technology. Do we use the built-in java (Derby) DB as a starting point. If we use JDBC does it matter what the initial technology is as long as it is (mostly) ANSI SQL compliant?

[Craig]

Got it on the ORM but thought I'd check. OK. So now for technology. Do we use the built-in java (Derby) DB as a starting point. If we use JDBC does it matter what the initial technology is as long as it is (mostly) ANSI SQL compliant?

Yeah if you use JDBC it doesn't really matter. Sure you can still end up depending on something one DB has but another doesn't but really if you do the fix is mostly quick and painless when you move over (unless someone puts most of the code in stored procedures...)

[RPTroll]

oh boy. I hadn't thought of stored procedures. There will be a use/need for them when called from MapTool proper (good way to insulate from backend database changes). I can see builtins calling them.

User defined stored procedures? hmmmm. How many worms are in that can?