[Pogamut-list] Proper Server Termination

[jacob.schrum]

Re: Proper Server Termination

Author: jacob.schrum

>> If I run my evolution code, but make it launch only one server with an unlimited time limit (so basically, one server
>> gets launched with some bots), there is no memory/thread leak.
>
>This means using UCCWrapper?

Yes. This scenario does not involve any bots or servers shutting down, which
I think is the source of the problem. I should do a test where I use one server,
but I load and kill several bots ...

>> If I evolve using just one server at a time (no multithreading, but still using the same code that can launch multiple server threads),
>> with a population size of 1 for several generations, the code leaks both memory and threads.
>
>You mean code as a whole or yours? Or this could not have been verified?

Can't be verified, but I think the answer might be a little of both. See answers below:

>The key point here is to discover which threads were not killed. This could be found out from the debugger itself
>(no need to use profiler) either in Eclipse or NetBeans... whenever such FatalErrorEvent occurs and your JVM won't terminate
>(meaning there are still some threads running), simply pause JVM and examine how many threads it has and what their names
>are. You will notice that all threads (that Pogamut Library is spawning) are nicely named, so it will be easy to distinguish between them.

I actually did use the profiler, but to look at which Threads were running the whole time. In particular,
with each new server, several threads are created, but some of these are still around when the server and
its bots are killed. Some of these were mine, and I've dealt with them. The threads on your end that are not
going away are threads that start with:

RMI TCP Connection
JMX server connection timeout
ClientNotifForwarder

The fact that these don't go away could be a consequence either of how I'm starting or stopping the servers.

>> HashMaps, ArrayLists, Locations, Strings, char arrays and Object arrays. The char arrays are inside the
>> Strings, and the Object Arrays are inside the ArrayLists. Strings are probably the keys for a lot of the HashMaps,
>> and Locations are probably what's being looked up. From all of this, I have the feeling that navigation and
>> path planning information is not being cleared out when the bots die, but maybe I'm jumping to conclusions.
>> Without the ability to follow the stack traces, I can't know for certain.
>
>No HashMap is using String as a key in Pogamut, this is wrong - we always use Token which is "named" String
>that has much faster equals() method. 

Actually, Tokens seem to be leaking too, as are UnrealIds and something called WeakReference. I know that
UnrealId's contain Strings, and I'm guessing Tokens do too, so that explains why Strings are leaking.

>When you're using profiler, there is a problem that all objects in the core
>consist of many lists / maps / Strings, thus it seems that these objects leak the most, but they are roots of the
>mountain of objects that leaked just because some "high-level object" were not gc()ed. Thus you should look for objects
>name "XYZModule" or "XYZAgent" or "XYZBot" which will be probably at the end of the list of objects (when sorted
>according to number of instances).

The most long lived class with "Module" at the end is AgentModule$1, which corresponds to the ComponentControlHelper
within AgentModule. PlayerMaps within Players also looks like it might be staying around too long.

>> On the thread side, RMI connections, sockets, and TCP connections seem to be responsible. These threads
>> tend to stay alive long after they should be closed.
>
>There is JMX problem - to be able to publish agent's JMX interface one need to have demon threads running RMI registry.
>If those registry are not shut down properly (via PogamutPlatform.getPlatform().close();) it will run forever.

As I mentioned above, there are some RMI/JMX threads that are created when a new server is created, and stay around
even after the server is killed. I think these are in addition to the demon threads, which I see at the top of the
profiler timeline: RMI TCP Accept, RMI Scheduler, RMI Reaper.

Are you saying that I should run PogamutPlatform.getPlatform().close() when my code terminates after the final generation,
or should I be running it after the initial settings are loaded and the first server is launched? Should I run it every
time I kill a server?

>> I can provide more specific profiler data if needed. I would really like to figure this out.
>
>Try JProfiler for your code, it is paid, but you can obtain 14 days trial license that is very powerful, much more useful
>than NetBeans profiler.

I will do that.

However, the main reason I'm doing all of this is for the Humanlike Bots/BorPtize competitions, which use a slightly
modified version of Gamebots. Unfortunately, the mod being used for the upcoming competition in June doesn't seem
to work with Pogamut 3.2: There are message parsing errors and, most alarmingly, my bot can't choose to use the
Link Gun/Judging gun. I put the following code in the logic() method:

{CODE()}
weaponry.changeWeapon(ItemType.LINK_GUN);
body.getShooting().shoot();
{CODE}

But when I went to the bot, it was firing the Assault Rifle. But all players START with the Link Gun in Botprize, since it
is the judging gun. My 3.1 bot works though, so I'm going to have to go back to that for the competition. Still, I would like
to figure this problem out, but since I'm using 3.1 for a while, I may not be as focused on it.

-- 
Reply Link: <http://diana.ms.mff.cuni.cz/main/tiki-view_forum_thread.php?forumId=4&comments_reply_threadId=4&comments_parentId=682&post_reply=1#form>