aerogear-users

Default heap size and OOM

classic Classic list List threaded Threaded
♦
♦
 6 messages Options
mo
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦

Default heap size and OOM

mo
Hi,

We use the OpenShift cartridge to deploy AeroGear UnifiedPush server version 1.0.3.  We deploy it on a medium gear.

Am I expected to increase the maximum heap size for JVM, set by default to be 512m?

We are getting OutOfMemoryError frequently.  And it's often very easily reproducible by sending 10 or so consecutive messages from AeroGear's admin console.

I can provide more details, but I wanted to see if I was supposed to increase the heap size.

Thank you.
Matthias Wessendorf
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦

Re: [Aerogear-users] Default heap size and OOM

Matthias Wessendorf
Hi,

Farah and I are looking to remove the JGroups subsystem. This has been a boost on other cartridges (that do not need clustering) as well. 

Will keep you updated


On Wed, Jun 17, 2015 at 7:14 PM, mo <[hidden email]> wrote:
Hi,

We use the  OpenShift cartridge
<https://github.com/aerogear/openshift-origin-cartridge-aerogear-push>   to
deploy AeroGear UnifiedPush server version 1.0.3.  We deploy it on a medium
gear.

Am I expected to increase the maximum heap size for JVM, set by default to
be 512m?

We are getting OutOfMemoryError frequently.  And it's often very easily
reproducible by sending 10 or so consecutive messages from AeroGear's admin
console.

I can provide more details, but I wanted to see if I was supposed to
increase the heap size.

Thank you.



--
View this message in context: http://aerogear-users.1116366.n5.nabble.com/Default-heap-size-and-OOM-tp56.html
Sent from the aerogear-users mailing list archive at Nabble.com.
_______________________________________________
Aerogear-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-users



--

_______________________________________________
Aerogear-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-users
mo
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦

Re: [Aerogear-users] Default heap size and OOM

mo
Thanks, Matthias.  I take your reply to mean "No, one shouldn't have to increase JVM heap size."  

Let me list some more parameters, in the hope that you might spot something obvious that we can correct on our side.  It's been difficult to troubleshoot this, because when the problem happens we can't even SSH into the server.  I believe whatever the problem is, it's exhausting resources on the server.

Parameters:

1. Deployed on OpenShift using the cartridge found here: https://github.com/aerogear/openshift-origin-cartridge-aerogear-push
2. AeroGear UnifiedPush Server 1.0.3 and WildFly, MySQL 5.5, Medium gear.
3. One application is defined with one variant (simple-push), 131 installations registered.
4. We are using all default settings.  We haven't customized the JVM max heap size, for example.  The server log tells me "-Xmx512m".

Here are the reproduction steps.  This happened reliably two days ago.  We couldn't reproduce this yesterday after a fresh restart of AeroGear.

1. Logged into AeroGear as admin
2. Select "Send Push"
3. Select an application
4. Hit "Send Push Notification", multiple times one after another.  About 10 consecutive messages would cause OutOfMemoryError on our server.

Here's a relevant snippet of the server log:

17:03:29,511 INFO  [PushNotificationSenderEndpoint] (default task-28) Message submitted to PushNetworks for further processing
17:03:29,513 INFO  [JPAPushMessageInformationDao] (EJB default - 6) Processing send request with '[alert=, criteria=[aliases=null, deviceTypes=null, categories=null, variants=null], time-to-live=-1]' payload
17:03:29,837 ERROR [org.jboss.as.ejb3.invocation] (EJB default - 6) JBAS014134: EJB Invocation failed on component SenderServiceImpl for method public abstract void org.jboss.aerogear.unifiedpush.message.SenderService.send(org.jboss.aerogear.unifiedpush.api.PushApplication,org.jboss.aerogear.unifiedpush.message.UnifiedPushMessage): javax.ejb.EJBException: JBAS014580: Unexpected Error
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187) [wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275) [wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]

.
.
.

[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_79]
        at java.lang.Thread.start(Thread.java:714) [rt.jar:1.7.0_79]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1056) [jsse.jar:1.7.0_79]
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332) [jsse.jar:1.7.0_79]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359) [jsse.jar:1.7.0_79]
.
.
.

org.jboss.aerogear.unifiedpush.message.sender.SimplePushNotificationSender.put(SimplePushNotificationSender.java:111) [unifiedpush-push-1.0.3.jar:1.0.3]
        at org.jboss.aerogear.unifiedpush.message.sender.SimplePushNotificationSender.sendPushMessage(SimplePushNotificationSender.java:65) [unifiedpush-push-1.0.3.jar:1.0.3]
        at org.jboss.aerogear.unifiedpush.message.SenderServiceImpl.send(SenderServiceImpl.java:110) [unifiedpush-push-1.0.3.jar:1.0.3]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)

.
.
.

        at org.jboss.as.ejb3.component.interceptors.NonPooledEJBComponentInstanceAssociatingInterceptor.processInvocation(NonPooledEJBComponentInstanceAssociatingInterceptor.java:59) [wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:273) [wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
        ... 38 more
Matthias Wessendorf
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦

Re: [Aerogear-users] Default heap size and OOM

Matthias Wessendorf
Hi ,




On Thu, Jun 18, 2015 at 4:57 PM, mo <[hidden email]> wrote:
Thanks, Matthias.  I take your reply to mean "No, one shouldn't have to
increase JVM heap size."

yeah, if you really want to, you can try tweaking the jvm heap size (as long as you still use a function of the max memory available on the gear)


But also please review the changes done in 

You could port them over to your instance. Makes sense ?

More coming soon.

 

Let me list some more parameters, in the hope that you might spot something
obvious that we can correct on our side.  It's been difficult to
troubleshoot this, because when the problem happens we can't even SSH into
the server.  I believe whatever the problem is, it's exhausting resources on
the server.

Parameters:

1. Deployed on OpenShift using the cartridge found here:
https://github.com/aerogear/openshift-origin-cartridge-aerogear-push
<https://github.com/aerogear/openshift-origin-cartridge-aerogear-push>
2. AeroGear UnifiedPush Server 1.0.3 and WildFly, MySQL 5.5, Medium gear.
3. One application is defined with one variant (simple-push), 131
installations registered.
4. We are using all default settings.  We haven't customized the JVM max
heap size, for example.  The server log tells me "-Xmx512m".

Here are the reproduction steps.  This happened reliably two days ago.  We
couldn't reproduce this yesterday after a fresh restart of AeroGear.

1. Logged into AeroGear as admin
2. Select "Send Push"
3. Select an application
4. Hit "Send Push Notification", multiple times one after another.  About 10
consecutive messages would cause OutOfMemoryError on our server.

Here's a relevant snippet of the server log:

 [0m [0m17:03:29,511 INFO  [PushNotificationSenderEndpoint] (default
task-28) Message submitted to PushNetworks for further processing
 [0m [0m17:03:29,513 INFO  [JPAPushMessageInformationDao] (EJB default - 6)
Processing send request with &apos;[alert=, criteria=[aliases=null,
deviceTypes=null, categories=null, variants=null], time-to-live=-1]&apos;
payload
 [0m [31m17:03:29,837 ERROR [org.jboss.as.ejb3.invocation] (EJB default - 6)
JBAS014134: EJB Invocation failed on component SenderServiceImpl for method
public abstract void
org.jboss.aerogear.unifiedpush.message.SenderService.send(org.jboss.aerogear.unifiedpush.api.PushApplication,org.jboss.aerogear.unifiedpush.message.UnifiedPushMessage):
javax.ejb.EJBException: JBAS014580: Unexpected Error
        at
org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
        at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]

.
.
.

[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[rt.jar:1.7.0_79]
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.7.0_79]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_79]
        at java.lang.Thread.start(Thread.java:714) [rt.jar:1.7.0_79]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1056)
[jsse.jar:1.7.0_79]
        at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
[jsse.jar:1.7.0_79]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)
[jsse.jar:1.7.0_79]
.
.
.

org.jboss.aerogear.unifiedpush.message.sender.SimplePushNotificationSender.put(SimplePushNotificationSender.java:111)
[unifiedpush-push-1.0.3.jar:1.0.3]
        at
org.jboss.aerogear.unifiedpush.message.sender.SimplePushNotificationSender.sendPushMessage(SimplePushNotificationSender.java:65)
[unifiedpush-push-1.0.3.jar:1.0.3]
        at
org.jboss.aerogear.unifiedpush.message.SenderServiceImpl.send(SenderServiceImpl.java:110)
[unifiedpush-push-1.0.3.jar:1.0.3]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.7.0_79]
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[rt.jar:1.7.0_79]
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_79]
        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
        at
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)

.
.
.

        at
org.jboss.as.ejb3.component.interceptors.NonPooledEJBComponentInstanceAssociatingInterceptor.processInvocation(NonPooledEJBComponentInstanceAssociatingInterceptor.java:59)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
        at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
        at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:273)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
        ... 38 more




--
View this message in context: http://aerogear-users.1116366.n5.nabble.com/Default-heap-size-and-OOM-tp56p58.html
Sent from the aerogear-users mailing list archive at Nabble.com.

_______________________________________________
Aerogear-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-users



--
Matthias Wessendorf

blog: http://matthiaswessendorf.wordpress.com/
sessions: http://www.slideshare.net/mwessendorf
twitter: http://twitter.com/mwessendorf

_______________________________________________
Aerogear-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-users
Matthias Wessendorf
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦

Re: [Aerogear-users] Default heap size and OOM

Matthias Wessendorf
Hi Mo,

the PR has been merged, and it should improve the situation a bit 

PS: next will be removing SimplePush from the cartridge

On Thu, Jun 18, 2015 at 5:17 PM, Matthias Wessendorf <[hidden email]> wrote:
Hi ,




On Thu, Jun 18, 2015 at 4:57 PM, mo <[hidden email]> wrote:
Thanks, Matthias.  I take your reply to mean "No, one shouldn't have to
increase JVM heap size."

yeah, if you really want to, you can try tweaking the jvm heap size (as long as you still use a function of the max memory available on the gear)


But also please review the changes done in 

You could port them over to your instance. Makes sense ?

More coming soon.

 

Let me list some more parameters, in the hope that you might spot something
obvious that we can correct on our side.  It's been difficult to
troubleshoot this, because when the problem happens we can't even SSH into
the server.  I believe whatever the problem is, it's exhausting resources on
the server.

Parameters:

1. Deployed on OpenShift using the cartridge found here:
https://github.com/aerogear/openshift-origin-cartridge-aerogear-push
<https://github.com/aerogear/openshift-origin-cartridge-aerogear-push>
2. AeroGear UnifiedPush Server 1.0.3 and WildFly, MySQL 5.5, Medium gear.
3. One application is defined with one variant (simple-push), 131
installations registered.
4. We are using all default settings.  We haven't customized the JVM max
heap size, for example.  The server log tells me "-Xmx512m".

Here are the reproduction steps.  This happened reliably two days ago.  We
couldn't reproduce this yesterday after a fresh restart of AeroGear.

1. Logged into AeroGear as admin
2. Select "Send Push"
3. Select an application
4. Hit "Send Push Notification", multiple times one after another.  About 10
consecutive messages would cause OutOfMemoryError on our server.

Here's a relevant snippet of the server log:

 [0m [0m17:03:29,511 INFO  [PushNotificationSenderEndpoint] (default
task-28) Message submitted to PushNetworks for further processing
 [0m [0m17:03:29,513 INFO  [JPAPushMessageInformationDao] (EJB default - 6)
Processing send request with &apos;[alert=, criteria=[aliases=null,
deviceTypes=null, categories=null, variants=null], time-to-live=-1]&apos;
payload
 [0m [31m17:03:29,837 ERROR [org.jboss.as.ejb3.invocation] (EJB default - 6)
JBAS014134: EJB Invocation failed on component SenderServiceImpl for method
public abstract void
org.jboss.aerogear.unifiedpush.message.SenderService.send(org.jboss.aerogear.unifiedpush.api.PushApplication,org.jboss.aerogear.unifiedpush.message.UnifiedPushMessage):
javax.ejb.EJBException: JBAS014580: Unexpected Error
        at
org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
        at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]

.
.
.

[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[rt.jar:1.7.0_79]
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.7.0_79]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_79]
        at java.lang.Thread.start(Thread.java:714) [rt.jar:1.7.0_79]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1056)
[jsse.jar:1.7.0_79]
        at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
[jsse.jar:1.7.0_79]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)
[jsse.jar:1.7.0_79]
.
.
.

org.jboss.aerogear.unifiedpush.message.sender.SimplePushNotificationSender.put(SimplePushNotificationSender.java:111)
[unifiedpush-push-1.0.3.jar:1.0.3]
        at
org.jboss.aerogear.unifiedpush.message.sender.SimplePushNotificationSender.sendPushMessage(SimplePushNotificationSender.java:65)
[unifiedpush-push-1.0.3.jar:1.0.3]
        at
org.jboss.aerogear.unifiedpush.message.SenderServiceImpl.send(SenderServiceImpl.java:110)
[unifiedpush-push-1.0.3.jar:1.0.3]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.7.0_79]
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[rt.jar:1.7.0_79]
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_79]
        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
        at
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)

.
.
.

        at
org.jboss.as.ejb3.component.interceptors.NonPooledEJBComponentInstanceAssociatingInterceptor.processInvocation(NonPooledEJBComponentInstanceAssociatingInterceptor.java:59)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
        at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
        at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:273)
[wildfly-ejb3-8.1.0.Final.jar:8.1.0.Final]
        ... 38 more




--
View this message in context: http://aerogear-users.1116366.n5.nabble.com/Default-heap-size-and-OOM-tp56p58.html
Sent from the aerogear-users mailing list archive at Nabble.com.

_______________________________________________
Aerogear-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-users



--



--
Matthias Wessendorf

blog: http://matthiaswessendorf.wordpress.com/
sessions: http://www.slideshare.net/mwessendorf
twitter: http://twitter.com/mwessendorf

_______________________________________________
Aerogear-users mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/aerogear-users
mo
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦

Re: [Aerogear-users] Default heap size and OOM

mo
Awesome.  Thank you, Matthias.

I'm currently revising our client side code, as it is likely contributing to wasted resources on the server side.  I will post a report after the revision and testing.
Loading...
Free forum by Nabble Edit this page