MapTool build notes for Eclipse

Progress reports and musings from the developers on the current gaming tools.

Moderators: dorpond, trevor, Azhrei

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

MapTool build notes for Eclipse

Post by Azhrei »

I just put out the MapTool 1.3.b64 build and I learned a lot during that process of how to build that beast! This post/thread will be my ramblings of how it works and what setup may be required.

I will be editing this post over time (paragraphs with blue dates in front of them) so be aware that replies posted below may reference an older version of this post. Caveat emptor.



First, the MapTool SVN repository is just one project. It requires at least one other project checked out into the workspace in order to be built: common.build. Be sure to checkout that project as well.

(Aside: I've done the same thing with a few of the other projects as well, such as the parser, rplib, rplib.resource, and maptool.resource. More about that later.)

You should be able to right-click on MapTool's build.xml and choose Run As -> Ant Build... That will open a dialog where you can fill in various attributes of how to perform the build. I'm going to describe what I did to make it work and why I think there has to be a better approach. In the Run As dialog click on the Classpath tab and you'll notice that ANT has a couple of places where it can pull plugins from. You'll need to add another so click on Add JARs... on the right and navigate to the common.build project's lib directory. Inside there you'll find three JAR files. Add each one to the classpath. One is the Pack200 task, one is the Java implementation of JavaScript (an old one), and I don't know what the third one is (haven't looked). Now you can click the Run button and execute the build. The next time you execute the build.xml those values will already populate the classpath so no changes will be needed.

2010-06-20: I also found that in order to recreate the parser JAR files I needed to have the ANTLR plugin installed in Eclipse. After doing so, it's necessary to right-click on the parser project and select Toggle ANTLR project nature. You'll see a pair of brown antlers appear on the icon next to the project name. :) This plugin reads the grammar files (such as expression.g) and creates the corresponding LR(1) grammar parser as Java source code. Took me forever to figure this one out. :roll:

A build of MapTool puts the resulting files into the ../dist/ directory so that directory should already exist. (It's part of the workspace but you don't want to check it out of SVN or you'll get a HUGE directory full of all revisions of all older builds of all tools! Instead, just create the directory using your OS tools.) You'll have a file called maptool-development.txt (a config file needed by the web site for the WebStart feature), a directory called zip/, and a directory called lib/ (used to hold individual JARs; also for WebStart). I've recently discovered that I can do a simple rsync --ignore-existing -SHavz ./ rptools:www/download/ and any new files will be sent to the proper directory on the web server. (Although I currently need to copy the text file myself. In the future, the text file will be named based on the particular MapTool build, such as maptool-1.3.b64.txt, and as such the --ignore-existing won't affect it. This new naming scheme will allow the web site to serve older versions of MapTool via WebStart, something it can't do right now.)



Another thing I've noticed is how clunky it is to replace a MapTool library with another project. For example, let's say I wanted to fix something in a macro function. I find the code and make some changes, but changes are required in the parser project as well. The parser library is already part of MapTool's build path (in my case, parser-1.1.b20.jar) so I first need to remove that library, then make the parser project a dependency for MapTool. I can now test MapTool and not need the parser JAR at all as long as testing stays within Eclipse (because the parser project will be added to MapTool's classpath).

When I'm ready for a final build, I go into the parser project and right-click on the build.xml and choose Run As -> Ant Build... I verify the parameters (no classpath changes are needed here, since this build file doesn't use Pack200 or other plugins) and click Run. Presto! I have a new JAR file; this one would be parser-1.1.b21.jar. I then copy it back to the MapTool project and turn off the dependency on parser.

I think there must be a better way. Perhaps MapTool should depend on the parser project (so that it gets built as needed, when source code changes) but the project appears higher in the classpath than the (older) JAR file so the JAR file simply isn't used. But even that isn't perfect, since an update to the JAR file won't affect MapTool at all if the project dependency is still there. :roll:

There must be a better mouse trap...

kkragenbrink
Cave Troll
Posts: 33
Joined: Mon May 03, 2010 3:03 am
Location: Los Angeles, CA

Re: MapTool build notes for Eclipse

Post by kkragenbrink »

This post was a huge help. Thank you for it.

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: MapTool build notes for Eclipse

Post by Azhrei »

Heh, you're welcome. :)

I still don't know how Trevor organizes his Eclipse workspace.

In doing the b65 build last night I also determined there has to be a simpler way to handle that as well. What I did was to do ANT builds on rplib and parser, copy the JARs out of their respective dist/ directories into maptool/lib, then right-click on those libraries under maptool/Referenced Libraries and choose Build Path > Remove from build path. Then go to the lib/ directory and right-click on each of the new libraries and choose Build Path > Add to build path.

As I mentioned above, there must be a better way because after the build I'll need to undo all of that because I want the project dependency to take precedence over the library JARs... :roll:

kkragenbrink
Cave Troll
Posts: 33
Joined: Mon May 03, 2010 3:03 am
Location: Los Angeles, CA

Re: MapTool build notes for Eclipse

Post by kkragenbrink »

I actually set up my environment to depend entirely on the other projects, and not the jars. Takes a little longer to process, but rplibs and parser get compiled at the same time as MapTools, now, which means if I make a change to them the jar gets repackaged and reupdated. Makes me happy.

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: MapTool build notes for Eclipse

Post by Azhrei »

kkragenbrink wrote:I actually set up my environment to depend entirely on the other projects, and not the jars. Takes a little longer to process, but rplibs and parser get compiled at the same time as MapTools, now, which means if I make a change to them the jar gets repackaged and reupdated. Makes me happy.
And I'm happy that you have something you're happy with. ;)

But that doesn't help the process of testing/building/deploying to the web site. Otherwise, I agree: your technique is the one I typically use.

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: MapTool build notes for Eclipse

Post by Azhrei »

In a private message, dorpond wrote:Not being a developer, I am not all that familiar with Eclipse but I can see that there are some 'projects' missing. If I delete them, eclipse is happy and I can launch Maptool but every time I do an update, these dependancies come back.

Can one of you give me step by step details on how to fix this so I can get up to date to all the changes that have taken place? I would like to get testing soon than later.
This will hopefully all change this week when we make the switchover to Git. In fact, I wanted to confirm with Craig that he's ready to make that switch.

Right now the configuration for MapTool that I have requires that the other projects be open or you get the build error you have. You can eliminate the requirements yourself by right-clicking on the maptool project and choosing Build Path > Configure Build Path.... Once inside there, select the Projects tab and remove those two projects. Now it should build properly.

Here's a short synopsis of why it's configured this way.
Spoiler
When I'm working in multiple projects at once (as I have been recently with the file I/O work that is part of rplib) I can't use the JAR libraries that are part of the maptool project because I'd have to do builds of rplib (thus bumping the version number) so I add the project to the build path above the JAR file -- now Eclipse uses the project in preference to the JAR. Once everything works, I build rplib and bump the version number, copy that JAR into maptool/lib, modify the build path to use the new JAR, and leave the reference to rplib in there for future development. But when I commit the changes to SVN the build path is committed as well. :(
One solution is to check out all of the rptools repository as a single project. This would allow Eclipse to treat the whole thing as a single project and eliminate this particular problem.
Spoiler
But it produces others as it makes it more difficult for me to track changes and then commit only certain ones. The correct way to do that in SVN would be to create branches, but I've not been impressed with the SVN/Eclipse integration in this regard. The one time I tried switching the maptool project to a branch, it was a PITA to switch it back. That implies that I should've created an entirely separate project to represent the branch. But that seems like overkill. And if I created a new project for, say, rplib, then other projects that reference that project wouldn't be referencing the branch but would be referencing the original and I'd have to change the build path again anyway. But apparently that's the correct way to do things in SVN based on my Google-fu... :(
Part of the problem is that SVN has trunk/branches/tags directories in each project so when I check out maptool/trunk I can't see the other two directories within Eclipse. If I were to check out the entire repo then those directories would appear as just that: directories. Doesn't fix the dependency problem though.

Anyway, I think Git's mechanism for branches should allow me to check out the entire rptools repo at once and still create and manage branches in a reasonable way. If so, I would modify the build path for maptool so that it always relies on the JAR files from a certain directory (I'd use ~/.m2/repository for reasons given below) and the builds of other projects would always put their JARs in that location. I believe that's the right way to handle it. (I could do this now using SVN, but it's a big change to the build process and I'd like to keep SVN pristine during the switch to Git, if possible. Meaning I'd switch to Git and make the build changes there so if something is broken other contributors can continue to use SVN in the mean time.) User whited has Maven set up to properly build all of the MapTool-related projects and has checked his changes into his own project at github. I can run a single mvn install now and build all related projects and put the JARs into ~/.m2/repository with that command. I have the Maven plugin in Eclipse, but haven't done the build from there yet.

If anyone knows that this is NOT the right way to do things, please speak up! If it's too involved to type, I'll jump on Skype or Vent or something and we can chat. :) I want to switch to Git by the end of this week if possible. I have Git configured at SourceForge and when the switchover occurs I will use git-svn make sure the Git repo is in sync with the SVN repo and make an announcement here. For the first few builds I'll probably keep the SVN repo up to date with Git, but all new development will be Git-based.

Phergus
Deity
Posts: 7132
Joined: Fri May 12, 2006 8:56 pm
Location: Middle of Nowhere, NM
Contact:

Re: MapTool build notes for Eclipse

Post by Phergus »

Azhrei wrote:Anyway, I think Git's mechanism for branches should allow me to check out the entire rptools repo at once and still create and manage branches in a reasonable way.
From my readings of the Git website and docs a Git repository should only have a single project in it. This sounds like you are going to throw the whole lot into one git repo which would force someone wanting to look at TokenTool to download the entire RPTools codebase.

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: MapTool build notes for Eclipse

Post by Azhrei »

Hm. Back to do some more reading then... :)

With Maven a lot of that goes away anyway. Maven will build separate projects and dump all of the JARs into one place. Then other projects can depend on the JARs. But this means building a JAR every time I make a change to rplib (for example) and I'm back to the version number issue.

Hm, maybe not. If Maven builds JARs without version numbers, then my push to the RPTools web site could assign the version number. So the JAR for rplib would be just rplib.jar until I push it to RPTools where it becomes rplib-1.0.b125 (or whatever). That allows the dependencies to be static.

Doesn't help with Eclipse though -- Eclipse still has the issue of the build path referring to either a project or a JAR. I guess using projects in all possible cases would be the best thing to do since the build/push process can automate generating the JARs.

I'm going to have to go back and re-read the multi-project examples I've seen see how they recommend it should be done...

Thanks for the heads-up, Phergus. :)

Phergus
Deity
Posts: 7132
Joined: Fri May 12, 2006 8:56 pm
Location: Middle of Nowhere, NM
Contact:

Re: MapTool build notes for Eclipse

Post by Phergus »

I've been trying to find a good description on using Git with multiple projects and so far everything I find has been either for very trivial cases or just advice along the "try this I think it will work" line.

What I do keep seeing over and over again is the one repo - one project advice.

User avatar
whited
Cave Troll
Posts: 45
Joined: Sun Aug 29, 2010 8:58 pm
Location: Seattle, WA

Re: MapTool build notes for Eclipse

Post by whited »

One git repo per sub-project would probably be better. I haven't researched it much, but I'm betting that anything you want to version independently should have its own source repository.

You should read up on the standard Maven versioning, particularly SNAPSHOT builds. If we use SNAPSHOT builds, it should greatly simplify things.

The idea being that when you are developing before a release, you can build 1.3.75-SNAPSHOT, and deploy that. "SNAPSHOT" is a special keyword that ends up in the repository as something like "1.3.75-<DATETIME>". When you want a release candidate, you change the build version to a *-RC1, *-RC2 etc. A release usually has a "-RELEASE" suffix. I probably haven't described it very well, but that's the idea.

I'm thinking it would require converting the "1.3.b75" into a "1.3.75-SNAPSHOT", "1.3.75-RC1", or "1.3.75-RELEASE" depending on the maven deployment being built.

We probably need to figure out a place to deploy snapshots and releases to. This way, with separated git repositories, anyone could for example fetch the "maptool" project, and build the source using SNAPSHOT dependencies from the maven package repository. Whenever someone does a check-in, they would build a SNAPSHOT build.

It's late here and I need to crash - hopefully this was coherent...

Alhazred
Cave Troll
Posts: 77
Joined: Tue Dec 30, 2008 8:46 am

Re: MapTool build notes for Eclipse

Post by Alhazred »

whited wrote:One git repo per sub-project would probably be better. I haven't researched it much, but I'm betting that anything you want to version independently should have its own source repository.

You should read up on the standard Maven versioning, particularly SNAPSHOT builds. If we use SNAPSHOT builds, it should greatly simplify things.

The idea being that when you are developing before a release, you can build 1.3.75-SNAPSHOT, and deploy that. "SNAPSHOT" is a special keyword that ends up in the repository as something like "1.3.75-<DATETIME>". When you want a release candidate, you change the build version to a *-RC1, *-RC2 etc. A release usually has a "-RELEASE" suffix. I probably haven't described it very well, but that's the idea.

I'm thinking it would require converting the "1.3.b75" into a "1.3.75-SNAPSHOT", "1.3.75-RC1", or "1.3.75-RELEASE" depending on the maven deployment being built.

We probably need to figure out a place to deploy snapshots and releases to. This way, with separated git repositories, anyone could for example fetch the "maptool" project, and build the source using SNAPSHOT dependencies from the maven package repository. Whenever someone does a check-in, they would build a SNAPSHOT build.

It's late here and I need to crash - hopefully this was coherent...
Just thought I would comment on this since I have spent the last 4 years or so setting up and running this kind of environment.

Maven really relies on the concept of layered binary repos. You have what is in your Eclipse projects and then you have the artifacts maven produces, which go in a local repo and can be the classpath (maven plugin helps with this). The dev can simply build the one thing he's working on and test it. He's got his own local maven repo where the jars go, from there they can be posted on up to a group level maven repo when they are ready. The other devs will now see them instantly as maven hits the repos in order, closest to farthest. If a dev happens to work on the same module as another dev, then his locally fresher copy will take precendence, but likely that shouldn't happen often. When it does that is when SNAPSHOT is really helpful.

So basically maven keeps a local binary repo on each devs desktop, then there is a project level one for testing within the dev group, then maybe a top level if you need it where only released binaries need to go. One nice implication of all this is you never have to use SVN or GIT to hold anything binary at all.

Frankly if you are going to use maven I'd really suggest getting a good book on it. It is a non-trivial tool and it can be a major driver of the whole build process. It can also be a real bear if the development cycle isn't built to fit with how it works.

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: MapTool build notes for Eclipse

Post by Azhrei »

Alhazred wrote:Frankly if you are going to use maven I'd really suggest getting a good book on it. It is a non-trivial tool and it can be a major driver of the whole build process. It can also be a real bear if the development cycle isn't built to fit with how it works.
Thanks for the advice. :) I have the Maven book in my wishlist at Amazon, but I just don't have time right now to do any reading! I'll order it and maybe get some time before Thanksgiving.

And we might want to chat with you some more about setup and build process. ;)

Alhazred
Cave Troll
Posts: 77
Joined: Tue Dec 30, 2008 8:46 am

Re: MapTool build notes for Eclipse

Post by Alhazred »

Azhrei wrote:
Alhazred wrote:Frankly if you are going to use maven I'd really suggest getting a good book on it. It is a non-trivial tool and it can be a major driver of the whole build process. It can also be a real bear if the development cycle isn't built to fit with how it works.
Thanks for the advice. :) I have the Maven book in my wishlist at Amazon, but I just don't have time right now to do any reading! I'll order it and maybe get some time before Thanksgiving.

And we might want to chat with you some more about setup and build process. ;)
Sure, there are a lot of clever people in the maven community too. Actually I finally ended up writing another framework in perl that does a lot of the same stuff. The one thing that maven can be frustrating about is it has an odd sort of rigidity. It can be hard to say repurpose a plugin that does a specific task to do the same task in a different context (it is possible but it takes a rather significant amount of understanding of maven internals and some repackaging). Still, it is the best choice as something that can do the job and which everyone using Eclipse can fairly easily set up in a consistent way.

Anyhow, I can certainly help you out here and there. We here still haven't advanced beyond SVN, but build tooling is something I can get into.

I have to wonder, if you guys are going to OSGi, then isn't the next step RCP? lol.

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: MapTool build notes for Eclipse

Post by Azhrei »

Yeah, Trevor had suggested using Eclipse as the platform for the next MT!

But the switch from Swing to SWT would be a huge one. And it didn't seem to provide any benefits significant enough. Especially considering that the announced roadmap for Java 7 includes HTML5 support in Swing for 2011. (Probably one of the things that MT most sorely lacks is a good HTML implementation.)

Of course, if MT 2.0 does make that switch, we'll already have OSGi support under our belts and the transition will be much smoother! 8)

Craig
Great Wyrm
Posts: 2107
Joined: Sun Jun 22, 2008 7:53 pm
Location: Melbourne, Australia

Re: MapTool build notes for Eclipse

Post by Craig »

Azhrei wrote:Yeah, Trevor had suggested using Eclipse as the platform for the next MT! But the switch from Swing to SWT would be a huge one. And it didn't seem to provide any benefits significant enough.
From what I understand he looked at it but very quickly dismissed the idea because the SWT canvas component is just too feature limited. I know he was keen to move to java 6 because of added features to Java2D, so I see it unlikely he would go SWT where there is pretty much no Java2D.

You would then have to use the SWT OpenGL component, but if you wanted OpenGL you are better off going with JLWGL or JOGL which are more mature.

There are certainly a lot more downsides than upsides to moving to SWT
Azhrei wrote: Especially considering that the announced roadmap for Java 7 includes HTML5 support in Swing for 2011. (Probably one of the things that MT most sorely lacks is a good HTML implementation.)

Of course, if MT 2.0 does make that switch, we'll already have OSGi support under our belts and the transition will be much smoother! 8)
One thing we should keep an eye on is the Netbeans RCP (think of it as Eclipse's swing RCP competitor). Given that Oracle stated Netbeans will be their Java development editor of choice -- they also have another IDE JDeveloper and contribute quite a bit to Eclipse -- hopefully they will update Netbeans GUI with the new JavaFX 2.0 goodies (HTML 5 and all that) and that could become a viable candidate for a platform -- we could finally get rid of the current docking framework, yay!!! :)

Post Reply

Return to “Developer Notes”