Well, I Am Not a Lawyer, but any printed reproduction of copyrighted material could reasonably be argued to violate US copyright laws. (There are exceptions for what is called "fair use" of the material in scholarly works and reviews.) However, when you describe a game system to friends, you do not reproduce the copyrighted material, but rather describe it in your own words. When you share books across a table, you do not reproduce the text, but rather allow others to read it themselves. I doubt that anyone would argue that you can't tell friends about a game, or that you can't loan them your rulebooks. Steve Jackson Games is usually quite friendly to gamers and, as far as I know, doesn't have a problem with loaning out or sharing rulebooks. Frankly, any other position would be both absurd and unenforceable.
As for the bigger question of making MapTools "work as you want," I don't know whether it's technically illegal or not to code in a game system that you don't distribute. It might be an open question in US copyright terms. But as long as you don't distribute your code, no one will know about it, and no one will take you to court about it.
Of course, many people produce MapTool conversions of leading game systems, including D&D, and do distribute them. Without the express agreement of the copyright holder, this is likely a violation of copyright law. As long as the copyright holder doesn't complain, however, they can get away with it. The copyright holder might even be willing to tolerate such copyright infringement if it increases the number of people who play the game and does not compete with their own digital products. (Remember, IANAL, and This Does Not Constitute Legal Advice.)
You should check out Steve Jackson Games' Online Policy
, if you haven't already. It's written in plain English and is remarkably generous, as such policies go.