Technical Question: defer macro and java heap space

[wolph42]

Im curious. I've build a macro that maps the coordinates that an image covers and saves these coordinates to a bunch of lib:tokens. The details are irrelevant.

What I'm curious about is this: I've created a defer loop (using execlink) which I feed 900 tokens to 'scan' or 'map'. This defer macro scan one token, removes it from the list and then calls yet another defer macro feeding this list and then ends. This goes on until the list is empty. The good part of this is that you can port stuff to the chat, update a status bar etc. (and run heavy duty process that would otherwise fail).

Now I would expect with a deferred macro that they actually finish (that is no longer linger in the heap space) when done however when I do this I notice 2 things:
1. the process starts to slow down considerably after say 100 scans
2. if I feed too many token (value differs per pc) but say >300 I get a java heapspace out of memory error.

If however I do the scans in tranches of say 100 scans per execution, meaning that the user actually needs to run the macro 9 times, everything works fine.

This has lead me to believe that a defer macro does NOT end, when it ends. Is this correct? and if so why?

[Craig]

If however I do the scans in tranches of say 100 scans per execution, meaning that the user actually needs to run the macro 9 times, everything works fine.

This has lead me to believe that a defer macro does NOT end, when it ends. Is this correct? and if so why?

Garbage Collection is not instantaneous, what you are doing will create a LOT of objects very quickly (not just the threads, but parsing/running macros creates a lot of objects which is usually fine but when you are doing it so often without a pause it is not), if you keep creating objects faster than the garbage collector can free them then you will run out of memory. Your objects/threads must be garbage collected before you can get the memory back, you don't get it back as soon as the thread has completed. By requiring the user to run the macro again after each 100 scans you are giving the garbage collector time to catch up which is why you are not experiencing the issue in that case.

[wolph42]

So... Would it work if I insert some delay or input() request say every 100 iterations?

Not sure how I can create a delay though, any clues on that?

Note though that 'very quickly' is a relative word as it takes about 3 minutes to make 100 iterations or scans.

[Craig]

So... Would it work if I insert some delay or input() request say every 100 iterations?

Not sure how I can create a delay though, any clues on that?

Try opening an input box and waiting, if it is just objects are being created faster than they can be garbage collected then this will also work.

Note though that 'very quickly' is a relative word as it takes about 3 minutes to make 100 iterations or scans.

Yes but during those 3 minutes your macro is busy running and doing things, given the amount of execLinks going around its likely that there are a lot of objects being created quickly for the whole of those 3 minutes.

[aliasmask]

Depending on your method of calling these execLinks, you can chain them where the next one isn't created until the previous one is finished. I have a feeling you're just creating a bunch per macro run creating a huge stack waiting. This is where recursion is useful. I did this in the last incarnation of macroio because of loop constraints. Once if finished processing 500 lines and got to the next macro break point, it would call itself and finish the job until all lines were processed.

I assume you're doing something like this:

Code: Select all

[H, c(900), code: {
   [H: link = macroLinkText("macroName@lib:token",...)]
   [H: execLink(link,1)]
}]

But if you let is end and call its self with the necessary counter data, you'll never have more than one execLink in memory. It really depends on your data as well. If you have large complex data structure, you won't want to pass that data or even save it as is in to a single prop. You'll want to break it up in to say, 900 props. You can then just pass the prop name and counter to process the data.

Code: Select all

[H: count = macro.args]
[H: link = macroLinkText("macroName@lib:token",...)]
[H: execLink(link,1)]

[wolph42]

nope. If I do them at once I could not have progress bar. the last line of the deferred macro calls itself with the updated list. Else I could no t have a progress bar.

So there is always only one macro active!

[aliasmask]

nope. If I do them at once I could not have progress bar. the last line of the deferred macro calls itself with the updated list. Else I could no t have a progress bar.

So there is always only one macro active!

Sure, you can have a progress bar. Just call a frame and refresh. You'll need to pass the max count in your args. Also, doing it this way doesn't lock up your system while it runs because it's using small slices of time. This is what I do for my animation as well.

[wolph42]

You're right. However that's not the point of this post. The question is why the heap space gets cluttered even though I run a serialized chain of macros.
I understand that if I fire them all at once that it goes very bad very soon. But this is not the case. It goes very bad a bit later and I'm trying to understand why and how I can prevent it.

I created this preferred chain event actually to prevent this from happening and was surprised that it did anyway. Granted it happened later. If I run this from within one macro in a for loop the system crashses after roughly half (150) of the scans.

So it does help but does not solve the issue.

The other question is : how can I create a delay (that is IF it's the garbage collection is the issue here). I like to minimize the user interaction. The input() trick is an option but it requires the user to check every 3to 5 minuttes to click 'next'

[aliasmask]

However that's not the point of this post.

Granted. I was just addressing the other issue brought up about using input as a solution. Making the function recursive, IMO would be a better hands off solution.

[wolph42]

However that's not the point of this post.

Granted. I was just addressing the other issue brought up about using input as a solution. Making the function recursive, IMO would be a better hands off solution.

Would recursive not increase the heap space a lot?

[aliasmask]

However that's not the point of this post.

Granted. I was just addressing the other issue brought up about using input as a solution. Making the function recursive, IMO would be a better hands off solution.

Would recursive not increase the heap space a lot?

Not with a deferral, but normally, yes. The method I suggest only creates the link ahead of itself before creating the next link. Start 1, create 2, finish 1, start 2, create 3,...

[wolph42]

*looking back* ah yes that's exactly how I did it. I wouldn't call this recursive but yes.

So that brings us back to the original question: why the memory error?

Might this be a memory leak bug?

[aliasmask]

If you're only storing one deferred link at a time, I can't say what the problem is. Perhaps some more testing is needed.

edit: Here's what I'm talking about:

progress bar defer test.rptok

(55.75 KiB) Downloaded 61 times

This is only a loop of 200 with minimal code and IMO is pretty slow. My guess the slowdown is from the trash collection. I wouldn't think printing the frame would be the slowdown.. I suppose I could just turn on debugging and find out.

edit: well, 47ms isn't all that long I suppose, but yeah, 1/3rd of the time is from the call.

[wolph42]

yup, its exactly the same method I use... safe the fact that your progress bar look cooler... as usual...

this still does not explain the heap space error...

[Azhrei]

this still does not explain the heap space error...

But Craig already covered that. Is there something else that isn't clear?

Maybe the garbage collector operation itself isn't clear? Think of the GC as a very low priority thread that is part of the Java runtime. MapTool doesn't specifically run the GC (at least, not for this discussion). Instead, the GC thread waits in a "sleeping" state for some event to trigger it and wake it up. That event could be related to heap space being low (the most common) or a general JVM-wide timeout (such as the JVM implementers deciding that the GC should run every 60 seconds or something; the Sun desktop JVM doesn't use a timeout, IIRC).

The problem in your situation is that even if the GC is triggered due to heap space filling up with unused objects, the thread will never run because it's a lower priority than the application threads that chewing up all available cpu (all available cpu within the JVM, although perhaps not on the entire machine).

Java programmers are taught (or at least, should be taught!) how to avoid this issue, but it's something that MTscript can't effectively protect you from.