Athena Freezing Desktop
Bruce, Sorry for the accusatory subject line - I can't prove Athena is the culprit but I figured that others with the same problem would be searching for similar terms. Running Athena from the current debian package for Lenny (2:1.2.11d-1) we have been seeing desktop freezes after some hard to characterize period of use. The time period seems to usually be a few hours, and there is a belief that it is connected to having more than a few raw data files open in Athena. Users have previously seen Athena crash/freeze when they have "too many" files open, or Athena open for "too long" but I haven't been able to get any firm numbers. The concern is that while previously when Athena crashed or froze, only Athena was effected. Now it seems that the whole desktop becomes non-responsive. The reason I'm suggesting there is a link between Athena and the Desktop (XFCE4) problem is that if you connect to the system via SSH and kill Athena, the Desktop becomes unfrozen and the user can continue with what they are doing. We have seen this occur on both beamlines, with different users and different data sets so I doubt it is data specific, but can send your some sample files if that will be useful. In one case I was able to collect an error message from the .xsession-errors file of a user who was experiencing the problem: ** (xfwm4:21854): WARNING **: Unmanaged net_wm_state (window 0x3800008) X Error: BadWindow (invalid Window parameter) 3 Major opcode: 7 Minor opcode: 0 Resource id: 0x329 kbuildsycoca running... Reusing existing ksycoca kbuildsycoca running... Reusing existing ksycoca kbuildsycoca running... Reusing existing ksycoca Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. I'm afraid I don't know if the lines before the lines that explicitly mention athena are related, or if these messages have anything to do with the desktop freezing problem - I think I have the seen the freeze without the errors. Please advise if we can provide you with other information to make figuring out easier. When this problem occurs at 3AM it really upsets our users (which tends to upset the staff when they get called about it...) so we have started running a second system just for Athena. Can you make any suggestions on the minimum vs optimal hardware for Athena? For tracking many (30+) raw data files to see how a run is progressing, can you suggest minimum RAM, CPU power, # of Cores? Thanks!! Please CC ehle@iit.edu with any responses as I'm not an active subscriber to this list. David Ehle MRCAT -IIT
Hi Dave, I suspect that the issue is related to the fact that, under the hood, Athena is using a library written in Fortran (Ifeffit is written in Fortran). Fortran does not do dynamic memory allocation. As Athena projects get bigger and bigger, you will eventually use up all of the statically allocated memory. Unless you have done something special in the compilation or run-time environment, there is probably nothing stopping the process from then touching memory outside of its static block. Then weird things will happen. This is explained in some detail here: http://millenia.cars.aps.anl.gov/pipermail/ifeffit/2007-May/007577.html (Yes, this does indeed represent a way to attack a computer, but its a hammer, not a scalpel. It is hard to imagine how to make something worse than what you have already observed happen.) This issue is well known and long discussed on the mailing list. It is also known (or should be...) to all the staff scientists at your beamline, who should be warning their users to limit the size of their project files by saving and restarting every 40 or 50 or so scans. Alternately, you could compile up a version of Ifeffit with a much larger statically allocated block of memory for use at the beamlines. (Carlo would certainly know how to do so, Ken also.) I am open to other suggestions, but your statement that it only happens after long use suggests it is the memory thing. It seems odd but certainly not impossible that it would freeze X. B On Monday 15 February 2010, 02:28:35 pm, David Ehle wrote:
Bruce,
Sorry for the accusatory subject line - I can't prove Athena is the culprit but I figured that others with the same problem would be searching for similar terms.
Running Athena from the current debian package for Lenny (2:1.2.11d-1) we have been seeing desktop freezes after some hard to characterize period of use. The time period seems to usually be a few hours, and there is a belief that it is connected to having more than a few raw data files open in Athena.
Users have previously seen Athena crash/freeze when they have "too many" files open, or Athena open for "too long" but I haven't been able to get any firm numbers. The concern is that while previously when Athena crashed or froze, only Athena was effected. Now it seems that the whole desktop becomes non-responsive.
The reason I'm suggesting there is a link between Athena and the Desktop (XFCE4) problem is that if you connect to the system via SSH and kill Athena, the Desktop becomes unfrozen and the user can continue with what they are doing.
We have seen this occur on both beamlines, with different users and different data sets so I doubt it is data specific, but can send your some sample files if that will be useful.
In one case I was able to collect an error message from the .xsession-errors file of a user who was experiencing the problem: ** (xfwm4:21854): WARNING **: Unmanaged net_wm_state (window 0x3800008) X Error: BadWindow (invalid Window parameter) 3 Major opcode: 7 Minor opcode: 0 Resource id: 0x329 kbuildsycoca running... Reusing existing ksycoca kbuildsycoca running... Reusing existing ksycoca kbuildsycoca running... Reusing existing ksycoca Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700.
I'm afraid I don't know if the lines before the lines that explicitly mention athena are related, or if these messages have anything to do with the desktop freezing problem - I think I have the seen the freeze without the errors.
Please advise if we can provide you with other information to make figuring out easier.
When this problem occurs at 3AM it really upsets our users (which tends to upset the staff when they get called about it...) so we have started running a second system just for Athena. Can you make any suggestions on the minimum vs optimal hardware for Athena? For tracking many (30+) raw data files to see how a run is progressing, can you suggest minimum RAM, CPU power, # of Cores?
Thanks!! Please CC ehle@iit.edu with any responses as I'm not an active subscriber to this list.
David Ehle MRCAT -IIT _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
-- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973 My homepage: http://xafs.org/BruceRavel EXAFS software: http://cars9.uchicago.edu/~ravel/software/exafs/
Bruce, Thanks for the explanation I will read thread you suggested. I really appreciate the quick response. Can you think of anything that may have changed in recent versions of Athena that may have aggravated the problem? It seems like it's happening more often now, and suddenly taking X with it every time. Freezing X makes this a much bigger problem than if we just crash Athena. In the short term, any advice on what is important hardware-wise for running Athena would be greatly appreciated. If we can't find a fast solution for Athena related X crashes we will need to set up a dedicated system to run it so it doesn't take down our control stations. Thanks again! David. On Mon, 15 Feb 2010, Bruce Ravel wrote:
Hi Dave,
I suspect that the issue is related to the fact that, under the hood, Athena is using a library written in Fortran (Ifeffit is written in Fortran). Fortran does not do dynamic memory allocation. As Athena projects get bigger and bigger, you will eventually use up all of the statically allocated memory. Unless you have done something special in the compilation or run-time environment, there is probably nothing stopping the process from then touching memory outside of its static block. Then weird things will happen.
This is explained in some detail here:
http://millenia.cars.aps.anl.gov/pipermail/ifeffit/2007-May/007577.html
(Yes, this does indeed represent a way to attack a computer, but its a hammer, not a scalpel. It is hard to imagine how to make something worse than what you have already observed happen.)
This issue is well known and long discussed on the mailing list. It is also known (or should be...) to all the staff scientists at your beamline, who should be warning their users to limit the size of their project files by saving and restarting every 40 or 50 or so scans.
Alternately, you could compile up a version of Ifeffit with a much larger statically allocated block of memory for use at the beamlines. (Carlo would certainly know how to do so, Ken also.)
I am open to other suggestions, but your statement that it only happens after long use suggests it is the memory thing. It seems odd but certainly not impossible that it would freeze X.
B
On Monday 15 February 2010, 02:28:35 pm, David Ehle wrote:
Bruce,
Sorry for the accusatory subject line - I can't prove Athena is the culprit but I figured that others with the same problem would be searching for similar terms.
Running Athena from the current debian package for Lenny (2:1.2.11d-1) we have been seeing desktop freezes after some hard to characterize period of use. The time period seems to usually be a few hours, and there is a belief that it is connected to having more than a few raw data files open in Athena.
Users have previously seen Athena crash/freeze when they have "too many" files open, or Athena open for "too long" but I haven't been able to get any firm numbers. The concern is that while previously when Athena crashed or froze, only Athena was effected. Now it seems that the whole desktop becomes non-responsive.
The reason I'm suggesting there is a link between Athena and the Desktop (XFCE4) problem is that if you connect to the system via SSH and kill Athena, the Desktop becomes unfrozen and the user can continue with what they are doing.
We have seen this occur on both beamlines, with different users and different data sets so I doubt it is data specific, but can send your some sample files if that will be useful.
In one case I was able to collect an error message from the .xsession-errors file of a user who was experiencing the problem: ** (xfwm4:21854): WARNING **: Unmanaged net_wm_state (window 0x3800008) X Error: BadWindow (invalid Window parameter) 3 Major opcode: 7 Minor opcode: 0 Resource id: 0x329 kbuildsycoca running... Reusing existing ksycoca kbuildsycoca running... Reusing existing ksycoca kbuildsycoca running... Reusing existing ksycoca Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700.
I'm afraid I don't know if the lines before the lines that explicitly mention athena are related, or if these messages have anything to do with the desktop freezing problem - I think I have the seen the freeze without the errors.
Please advise if we can provide you with other information to make figuring out easier.
When this problem occurs at 3AM it really upsets our users (which tends to upset the staff when they get called about it...) so we have started running a second system just for Athena. Can you make any suggestions on the minimum vs optimal hardware for Athena? For tracking many (30+) raw data files to see how a run is progressing, can you suggest minimum RAM, CPU power, # of Cores?
Thanks!! Please CC ehle@iit.edu with any responses as I'm not an active subscriber to this list.
David Ehle MRCAT -IIT _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
-- Bruce Ravel ------------------------------------ bravel@bnl.gov
National Institute of Standards and Technology Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973
My homepage: http://xafs.org/BruceRavel EXAFS software: http://cars9.uchicago.edu/~ravel/software/exafs/
On Monday 15 February 2010, 03:30:51 pm, David Ehle wrote:
Can you think of anything that may have changed in recent versions of Athena that may have aggravated the problem? It seems like it's happening more often now, and suddenly taking X with it every time. Freezing X makes this a much bigger problem than if we just crash Athena.
I cannot imagine what's causing this. I wouldn't say that anything substantial has changed in Ifeffit+friends.
In the short term, any advice on what is important hardware-wise for running Athena would be greatly appreciated. If we can't find a fast solution for Athena related X crashes we will need to set up a dedicated system to run it so it doesn't take down our control stations.
That's not a horrible idea in any case. You could compile up a ginormous version of Ifeffit, link Athena to it, and run into the memory problem much less often. B -- Bruce Ravel ------------------------------------ bravel@bnl.gov National Institute of Standards and Technology Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973 My homepage: http://xafs.org/BruceRavel EXAFS software: http://cars9.uchicago.edu/~ravel/software/exafs/
Matt, Stefan, I'll be responding to both posts sent to list in the order recieved. Matt, Thank your for your response. I'm going to run athena from a terminal window today (our users use a launcher) and see if I can provoke the problem and collect some actual worthwhile info on the problem. I'm afraid I've been handicapped by only getting 2nd or 3rd hand reports so far - normally I wouldn't even try to file a bug based on that, but it was considered urgent enough by users and staff that I wanted to at least make sure this wasn't a known problem or something with a trivial fix. To clear up a little confusion - Athena is being run from the desktop, not via an SSH connection. SSH was only used as a method to kill the Athena process when the desktop was non-responsive. I suspect your right and the problem is some sort of combination issue. Stefan, Thank you for your suggestion. The systems in question has not shown this problem in the past and there is a very strong correlation between Athena running and the "freeze" occuring, and the Athena process being killed and the freeze going away (without restarting X) so the video card is not one of my primary suspects - but I will look and see if there has been a recent change in the xorg version or drivers that might match the timeline of when we started seeing the problem. Thanks for the suggestion! David. All, Hopefully by the end of the day I'll be able to provide some actual information. Thanks again for your help! David. Combining messages sent only to list below- Stefen Mangold on Feb 16th 2:38am: Dear David Ehle, I had already some trouble at the beamline with a new computer with sudden crashes of X during "doing nothing". We could track this problem down to a bug in the graphic card driver. We could solve this problem by exchanging the consumer graphic card with a 50 Euro more expensive one. Regards Stefan Matt Newville on Feb 15th 3:14pm: David, I suspect that you *should* get sensible error messages from Athena about running out of Ifeffit resources well before seeing a crash of X. Not in the sense of "Bruce ought to change Athena to report this", but in the sense of "Bruce already does report this". Generally, Ifeffit/Athena should run out of memory well before a Linux system runs out of memory and swap space. Crashing X over ssh seems really peculiar to me -- that's not an out-of-memory issue. I don't often leave Athena open for hours at a time on Linux, but have never seen such crashes on Ubuntu. My suspicion is that this might be caused by Perl/Tk, X, or the lengthy ssh session. --Matt On Mon, 15 Feb 2010, Bruce Ravel wrote:
On Monday 15 February 2010, 03:30:51 pm, David Ehle wrote:
Can you think of anything that may have changed in recent versions of Athena that may have aggravated the problem? It seems like it's happening more often now, and suddenly taking X with it every time. Freezing X makes this a much bigger problem than if we just crash Athena.
I cannot imagine what's causing this. I wouldn't say that anything substantial has changed in Ifeffit+friends.
In the short term, any advice on what is important hardware-wise for running Athena would be greatly appreciated. If we can't find a fast solution for Athena related X crashes we will need to set up a dedicated system to run it so it doesn't take down our control stations.
That's not a horrible idea in any case. You could compile up a ginormous version of Ifeffit, link Athena to it, and run into the memory problem much less often.
B
-- Bruce Ravel ------------------------------------ bravel@bnl.gov
National Institute of Standards and Technology Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973
My homepage: http://xafs.org/BruceRavel EXAFS software: http://cars9.uchicago.edu/~ravel/software/exafs/
Bruce, list, It appears that the freezes were probably not caused by running out of ifeffit resources. When I overloaded Athena with 50+ files I got a very well behaved warning, and then it finally declared it simply couldn't do what was asked of it and refused to do anything until I removed some groups. At no point did it become unresponsive or take the desktop with it. On a different computer I did manage to provoke a segfault when I was just fiddling around without even attempting to imitate actual use, but it still did not take the desktop with it. No significant error messages were produced. I will look at possible driver issues as Stefan suggested. Can anyone else suggest lines of inquiry? David. On Tue, 16 Feb 2010, David Ehle wrote:
Matt, Stefan,
I'll be responding to both posts sent to list in the order recieved.
Matt,
Thank your for your response. I'm going to run athena from a terminal window today (our users use a launcher) and see if I can provoke the problem and collect some actual worthwhile info on the problem. I'm afraid I've been handicapped by only getting 2nd or 3rd hand reports so far - normally I wouldn't even try to file a bug based on that, but it was considered urgent enough by users and staff that I wanted to at least make sure this wasn't a known problem or something with a trivial fix.
To clear up a little confusion - Athena is being run from the desktop, not via an SSH connection. SSH was only used as a method to kill the Athena process when the desktop was non-responsive.
I suspect your right and the problem is some sort of combination issue.
Stefan,
Thank you for your suggestion. The systems in question has not shown this problem in the past and there is a very strong correlation between Athena running and the "freeze" occuring, and the Athena process being killed and the freeze going away (without restarting X) so the video card is not one of my primary suspects - but I will look and see if there has been a recent change in the xorg version or drivers that might match the timeline of when we started seeing the problem. Thanks for the suggestion!
David.
All,
Hopefully by the end of the day I'll be able to provide some actual information.
Thanks again for your help!
David.
Combining messages sent only to list below-
Stefen Mangold on Feb 16th 2:38am: Dear David Ehle,
I had already some trouble at the beamline with a new computer with sudden crashes of X during "doing nothing". We could track this problem down to a bug in the graphic card driver. We could solve this problem by exchanging the consumer graphic card with a 50 Euro more expensive one.
Regards
Stefan
Matt Newville on Feb 15th 3:14pm: David,
I suspect that you *should* get sensible error messages from Athena about running out of Ifeffit resources well before seeing a crash of X. Not in the sense of "Bruce ought to change Athena to report this", but in the sense of "Bruce already does report this". Generally, Ifeffit/Athena should run out of memory well before a Linux system runs out of memory and swap space. Crashing X over ssh seems really peculiar to me -- that's not an out-of-memory issue. I don't often leave Athena open for hours at a time on Linux, but have never seen such crashes on Ubuntu. My suspicion is that this might be caused by Perl/Tk, X, or the lengthy ssh session.
--Matt
On Mon, 15 Feb 2010, Bruce Ravel wrote:
On Monday 15 February 2010, 03:30:51 pm, David Ehle wrote:
Can you think of anything that may have changed in recent versions of Athena that may have aggravated the problem? It seems like it's happening more often now, and suddenly taking X with it every time. Freezing X makes this a much bigger problem than if we just crash Athena.
I cannot imagine what's causing this. I wouldn't say that anything substantial has changed in Ifeffit+friends.
In the short term, any advice on what is important hardware-wise for running Athena would be greatly appreciated. If we can't find a fast solution for Athena related X crashes we will need to set up a dedicated system to run it so it doesn't take down our control stations.
That's not a horrible idea in any case. You could compile up a ginormous version of Ifeffit, link Athena to it, and run into the memory problem much less often.
B
-- Bruce Ravel ------------------------------------ bravel@bnl.gov
National Institute of Standards and Technology Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973
My homepage: http://xafs.org/BruceRavel EXAFS software: http://cars9.uchicago.edu/~ravel/software/exafs/
Hi David, Do you (or can you) see such issues with other applications? I would probably first try /usr/bin/widget, the Perl/Tk demo code (I think that should be installed with the debian package), to see if that ever has the same problem. I might try re-installing (or even rebuilding from source) Perl/Tk. Cheers, --Matt PS: I think Perl/Tk does not at all depend on Tk, but you might verify that.
Bruce, I'm afraid I'm not actually a Athena user, so could you clarify something for me in regards the discussion of data groups in the "Care and feeding of Athena" thread? It looks like users should be wary starting around 45-50 data groups. Does that translate to 45-50 raw datafile that are "open" for active processing/graphing (i.e. are checked in the top right hand pane), or 45-50 raw datafiles that athena is aware of (are available to BE checked in the upper right hand pane)? Was there a resolution to the issue that Mark was saying that 150+ data groups still only uses up 5% of IEFFIT's heap? Thanks!! David. On Mon, 15 Feb 2010, Bruce Ravel wrote:
Hi Dave,
I suspect that the issue is related to the fact that, under the hood, Athena is using a library written in Fortran (Ifeffit is written in Fortran). Fortran does not do dynamic memory allocation. As Athena projects get bigger and bigger, you will eventually use up all of the statically allocated memory. Unless you have done something special in the compilation or run-time environment, there is probably nothing stopping the process from then touching memory outside of its static block. Then weird things will happen.
This is explained in some detail here:
http://millenia.cars.aps.anl.gov/pipermail/ifeffit/2007-May/007577.html
(Yes, this does indeed represent a way to attack a computer, but its a hammer, not a scalpel. It is hard to imagine how to make something worse than what you have already observed happen.)
This issue is well known and long discussed on the mailing list. It is also known (or should be...) to all the staff scientists at your beamline, who should be warning their users to limit the size of their project files by saving and restarting every 40 or 50 or so scans.
Alternately, you could compile up a version of Ifeffit with a much larger statically allocated block of memory for use at the beamlines. (Carlo would certainly know how to do so, Ken also.)
I am open to other suggestions, but your statement that it only happens after long use suggests it is the memory thing. It seems odd but certainly not impossible that it would freeze X.
B
On Monday 15 February 2010, 02:28:35 pm, David Ehle wrote:
Bruce,
Sorry for the accusatory subject line - I can't prove Athena is the culprit but I figured that others with the same problem would be searching for similar terms.
Running Athena from the current debian package for Lenny (2:1.2.11d-1) we have been seeing desktop freezes after some hard to characterize period of use. The time period seems to usually be a few hours, and there is a belief that it is connected to having more than a few raw data files open in Athena.
Users have previously seen Athena crash/freeze when they have "too many" files open, or Athena open for "too long" but I haven't been able to get any firm numbers. The concern is that while previously when Athena crashed or froze, only Athena was effected. Now it seems that the whole desktop becomes non-responsive.
The reason I'm suggesting there is a link between Athena and the Desktop (XFCE4) problem is that if you connect to the system via SSH and kill Athena, the Desktop becomes unfrozen and the user can continue with what they are doing.
We have seen this occur on both beamlines, with different users and different data sets so I doubt it is data specific, but can send your some sample files if that will be useful.
In one case I was able to collect an error message from the .xsession-errors file of a user who was experiencing the problem: ** (xfwm4:21854): WARNING **: Unmanaged net_wm_state (window 0x3800008) X Error: BadWindow (invalid Window parameter) 3 Major opcode: 7 Minor opcode: 0 Resource id: 0x329 kbuildsycoca running... Reusing existing ksycoca kbuildsycoca running... Reusing existing ksycoca kbuildsycoca running... Reusing existing ksycoca Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700.
I'm afraid I don't know if the lines before the lines that explicitly mention athena are related, or if these messages have anything to do with the desktop freezing problem - I think I have the seen the freeze without the errors.
Please advise if we can provide you with other information to make figuring out easier.
When this problem occurs at 3AM it really upsets our users (which tends to upset the staff when they get called about it...) so we have started running a second system just for Athena. Can you make any suggestions on the minimum vs optimal hardware for Athena? For tracking many (30+) raw data files to see how a run is progressing, can you suggest minimum RAM, CPU power, # of Cores?
Thanks!! Please CC ehle@iit.edu with any responses as I'm not an active subscriber to this list.
David Ehle MRCAT -IIT _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
-- Bruce Ravel ------------------------------------ bravel@bnl.gov
National Institute of Standards and Technology Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2 Building 535A Upton NY, 11973
My homepage: http://xafs.org/BruceRavel EXAFS software: http://cars9.uchicago.edu/~ravel/software/exafs/
David,
I suspect that you *should* get sensible error messages from Athena
about running out of Ifeffit resources well before seeing a crash of
X. Not in the sense of "Bruce ought to change Athena to report
this", but in the sense of "Bruce already does report this".
Generally, Ifeffit/Athena should run out of memory well before a Linux
system runs out of memory and swap space. Crashing X over ssh seems
really peculiar to me -- that's not an out-of-memory issue. I don't
often leave Athena open for hours at a time on Linux, but have never
seen such crashes on Ubuntu. My suspicion is that this might be
caused by Perl/Tk, X, or the lengthy ssh session.
--Matt
On Mon, Feb 15, 2010 at 1:28 PM, David Ehle
Bruce,
Sorry for the accusatory subject line - I can't prove Athena is the culprit but I figured that others with the same problem would be searching for similar terms.
Running Athena from the current debian package for Lenny (2:1.2.11d-1) we have been seeing desktop freezes after some hard to characterize period of use. The time period seems to usually be a few hours, and there is a belief that it is connected to having more than a few raw data files open in Athena.
Users have previously seen Athena crash/freeze when they have "too many" files open, or Athena open for "too long" but I haven't been able to get any firm numbers. The concern is that while previously when Athena crashed or froze, only Athena was effected. Now it seems that the whole desktop becomes non-responsive.
The reason I'm suggesting there is a link between Athena and the Desktop (XFCE4) problem is that if you connect to the system via SSH and kill Athena, the Desktop becomes unfrozen and the user can continue with what they are doing.
We have seen this occur on both beamlines, with different users and different data sets so I doubt it is data specific, but can send your some sample files if that will be useful.
In one case I was able to collect an error message from the .xsession-errors file of a user who was experiencing the problem: ** (xfwm4:21854): WARNING **: Unmanaged net_wm_state (window 0x3800008) X Error: BadWindow (invalid Window parameter) 3 Major opcode: 7 Minor opcode: 0 Resource id: 0x329 kbuildsycoca running... Reusing existing ksycoca kbuildsycoca running... Reusing existing ksycoca kbuildsycoca running... Reusing existing ksycoca Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700.
I'm afraid I don't know if the lines before the lines that explicitly mention athena are related, or if these messages have anything to do with the desktop freezing problem - I think I have the seen the freeze without the errors.
Please advise if we can provide you with other information to make figuring out easier.
When this problem occurs at 3AM it really upsets our users (which tends to upset the staff when they get called about it...) so we have started running a second system just for Athena. Can you make any suggestions on the minimum vs optimal hardware for Athena? For tracking many (30+) raw data files to see how a run is progressing, can you suggest minimum RAM, CPU power, # of Cores?
Thanks!! Please CC ehle@iit.edu with any responses as I'm not an active subscriber to this list.
David Ehle MRCAT -IIT _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Dear David Ehle, I had already some trouble at the beamline with a new computer with sudden crashes of X during "doing nothing". We could track this problem down to a bug in the graphic card driver. We could solve this problem by exchanging the consumer graphic card with a 50 Euro more expensive one. Regards Stefan Am 15.02.2010 um 20:28 schrieb David Ehle:
Bruce,
Sorry for the accusatory subject line - I can't prove Athena is the culprit but I figured that others with the same problem would be searching for similar terms.
Running Athena from the current debian package for Lenny (2:1.2.11d-1) we have been seeing desktop freezes after some hard to characterize period of use. The time period seems to usually be a few hours, and there is a belief that it is connected to having more than a few raw data files open in Athena.
Users have previously seen Athena crash/freeze when they have "too many" files open, or Athena open for "too long" but I haven't been able to get any firm numbers. The concern is that while previously when Athena crashed or froze, only Athena was effected. Now it seems that the whole desktop becomes non-responsive.
The reason I'm suggesting there is a link between Athena and the Desktop (XFCE4) problem is that if you connect to the system via SSH and kill Athena, the Desktop becomes unfrozen and the user can continue with what they are doing.
We have seen this occur on both beamlines, with different users and different data sets so I doubt it is data specific, but can send your some sample files if that will be useful.
In one case I was able to collect an error message from the .xsession-errors file of a user who was experiencing the problem: ** (xfwm4:21854): WARNING **: Unmanaged net_wm_state (window 0x3800008) X Error: BadWindow (invalid Window parameter) 3 Major opcode: 7 Minor opcode: 0 Resource id: 0x329 kbuildsycoca running... Reusing existing ksycoca kbuildsycoca running... Reusing existing ksycoca kbuildsycoca running... Reusing existing ksycoca Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 3891. Use of uninitialized value in concatenation (.) or string at /usr/bin/athena line 4700.
I'm afraid I don't know if the lines before the lines that explicitly mention athena are related, or if these messages have anything to do with the desktop freezing problem - I think I have the seen the freeze without the errors.
Please advise if we can provide you with other information to make figuring out easier.
When this problem occurs at 3AM it really upsets our users (which tends to upset the staff when they get called about it...) so we have started running a second system just for Athena. Can you make any suggestions on the minimum vs optimal hardware for Athena? For tracking many (30+) raw data files to see how a run is progressing, can you suggest minimum RAM, CPU power, # of Cores?
Thanks!! Please CC ehle@iit.edu with any responses as I'm not an active subscriber to this list.
David Ehle MRCAT -IIT _______________________________________________ Ifeffit mailing list Ifeffit@millenia.cars.aps.anl.gov http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
-- Dr. Stefan Mangold Institut für Synchrotronstrahlung Forschungszentrum Karlsruhe Tel.: +49-7247-826073
participants (4)
-
Bruce Ravel -
David Ehle -
Matt Newville -
Stefan Mangold