Short run times for some workunits

Questions and Answers : Bugs : Short run times for some workunits
Message board moderation

To post messages, you must log in.

AuthorMessage
AnandBhat

Send message
Joined: 14 May 22
Posts: 10
Credit: 197,581
RAC: 1,779
Message 652 - Posted: 1 Dec 2022, 23:18:13 UTC

A few of my workunits have had abnormally small run times. I thought the run times for all work units was set to 2 hours and thought it was best to report this here if this warrants a closer look.

https://boinc.loda-lang.org/loda/workunit.php?wuid=2599141 - 297s
https://boinc.loda-lang.org/loda/workunit.php?wuid=2599156 - 270s
https://boinc.loda-lang.org/loda/workunit.php?wuid=2599261 - 153s
https://boinc.loda-lang.org/loda/workunit.php?wuid=2599315 - 53s
etc.

I'm not sure if these were marked completed due to some out of memory condition or some other system/ network error as these appear to have completed at the same time. However, they've all passed validation and I've received credits for them. The task outputs do not show any errors and the LODA logs page is blank.
ID: 652 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Christian Krause
Project administrator

Send message
Joined: 9 May 22
Posts: 250
Credit: 449,267
RAC: 198
Message 653 - Posted: 3 Dec 2022, 21:58:23 UTC - in response to Message 652.  

Thanks for reporting it. We'll look into. I checked some other results and they look normal. Not sure why these were so quick.
ID: 653 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
AnandBhat

Send message
Joined: 14 May 22
Posts: 10
Credit: 197,581
RAC: 1,779
Message 655 - Posted: 6 Dec 2022, 1:56:21 UTC - in response to Message 653.  

I saw this happen again today. I have LODA running with resource share 0 (i.e., only download work when my other projects do not have any work). I had 16 LODA tasks running when I received some LHC tasks with an earlier deadline. My system automatically paused the LODA tasks (status: "Waiting to run") and proceeded to run and complete the LHC tasks. After a couple of hours, when the LODA tasks that were at varying %s of progress got their chance to run, the tasks immediately "completed" and were reported as valid.

E.g.,
https://boinc.loda-lang.org/loda/workunit.php?wuid=2685909
https://boinc.loda-lang.org/loda/workunit.php?wuid=2685639

I have the option for setting "Leave non-GPU tasks in memory when suspended" checked in BOINC.

I managed to freeze network communications and captured this output file for wu_1670076947_34367_0. It appears the task abruptly "completed" after resuming:
2022-12-06 10:39:13|INFO |Starting LODA v22.12.2. See https://loda-lang.org/
2022-12-06 10:39:13|INFO |Found environment variable: PROJECT_DIR=.\
2022-12-06 10:39:13|INFO |Loading init data from file: .\init_data.xml
2022-12-06 10:39:13|INFO |Platform: windows, system memory: 15734 MiB
2022-12-06 10:39:13|INFO |User name: AnandBhat, host ID: 139
2022-12-06 10:39:13|INFO |Using LODA home directory "C:\ProgramData\BOINC/projects/boinc.loda-lang.org_loda\"
2022-12-06 10:39:13|INFO |Checking environment
2022-12-06 10:39:13|WARN |Setting environment variable: COMSPEC=C:\WINDOWS\system32\cmd.exe
2022-12-06 10:39:13|WARN |Setting environment variable: SYSTEMROOT=C:\WINDOWS
2022-12-06 10:39:13|WARN |Setting environment variable: PATH=C:\WINDOWS\system32;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\Program Files\Git\cmd;C:\ProgramData\BOINC/projects/boinc.loda-lang.org_loda\git\cmd;C:\Program Files\Git\usr\bin;C:\ProgramData\BOINC/projects/boinc.loda-lang.org_loda\git\usr\bin
2022-12-06 10:39:13|WARN |Setting environment variable: TMP=C:\ProgramData\BOINC/projects/boinc.loda-lang.org_loda\
2022-12-06 10:39:13|WARN |Setting environment variable: TEMP=C:\ProgramData\BOINC/projects/boinc.loda-lang.org_loda\
2022-12-06 10:39:13|INFO |Loading sequences from the OEIS index
2022-12-06 10:39:19|INFO |Loaded 337318/358544 sequences in 5.74s
2022-12-06 10:39:27|INFO |Initialized 5 matchers (ignoring 123210 sequences)
2022-12-06 10:39:27|INFO |Initialized 1 generators (profile: mutate3, overwrite: none)
2022-12-06 10:39:27|INFO |Mining programs in client mode (extended validation mode)
2022-12-06 10:39:49|INFO |Processed 49 programs, 0.5%
2022-12-06 10:40:48|INFO |Processed 2104 programs, 1.3%
2022-12-06 10:41:30|INFO |Processed 53 programs, 1.9%
2022-12-06 10:44:04|INFO |Processed 9 programs, 4.0%
2022-12-06 10:44:53|INFO |Processed 159 programs, 4.7%
2022-12-06 10:45:46|INFO |Processed 469 programs, 5.4%
2022-12-06 10:46:22|INFO |Processed 229 programs, 5.9%
2022-12-06 10:46:58|INFO |Processed 2035 programs, 6.4%
2022-12-06 10:47:11|INFO |Fetched http://api.loda-lang.org/miner/v1/oeis/b245433.txt.gz
2022-12-06 10:47:34|INFO |Processed 3391 programs, 6.9%
2022-12-06 10:48:10|INFO |Processed 229783 programs, 7.4%
2022-12-06 10:49:04|INFO |Processed 308607 programs, 8.2%
2022-12-06 10:50:30|INFO |Processed 893 programs, 9.4%
2022-12-06 10:51:06|INFO |Processed 439748 programs, 9.9%
2022-12-06 10:52:49|INFO |Processed 335372 programs, 11.3%
2022-12-06 12:41:00|INFO |Finished mining after 121 minutes
ID: 655 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
AnandBhat

Send message
Joined: 14 May 22
Posts: 10
Credit: 197,581
RAC: 1,779
Message 656 - Posted: 6 Dec 2022, 3:28:39 UTC - in response to Message 655.  

I noticed something that may be related.

1. When LODA tasks are in a Waiting to run or Suspended status in BOINC, the process in Windows Task Manager shows it is Suspended. Tasks for other projects do not appear to do this.

2. The memory for the LODA tasks in the Suspended state as seen in Task Manager drops over time.

3. When resumed, the LODA tasks "jump" in % completed, either directly to 100% or to a higher percent complete. It's almost as if the system thinks the task has been processing something while it was suspended.

Here's a snippet from when this LODA task fluctuated between Running and Waiting to run, with my indicators for when the switch happened. The progress completed %s appeared to increase proportionally to the time the task was not running (related to the standard 2 hour runtime?):
2022-12-06 12:40:55|INFO |Starting LODA v22.12.2. See https://loda-lang.org/
2022-12-06 12:40:55|INFO |Found environment variable: PROJECT_DIR=.\
2022-12-06 12:40:55|INFO |Loading init data from file: .\init_data.xml
2022-12-06 12:40:55|INFO |Platform: windows, system memory: 15734 MiB
2022-12-06 12:40:55|INFO |User name: AnandBhat, host ID: 139
2022-12-06 12:40:55|INFO |Using LODA home directory "C:\ProgramData\BOINC/projects/boinc.loda-lang.org_loda\"
2022-12-06 12:40:55|INFO |Checking environment
2022-12-06 12:40:55|WARN |Setting environment variable: COMSPEC=C:\WINDOWS\system32\cmd.exe
2022-12-06 12:40:55|WARN |Setting environment variable: SYSTEMROOT=C:\WINDOWS
2022-12-06 12:40:55|WARN |Setting environment variable: PATH=C:\WINDOWS\system32;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\Program Files\Git\cmd;C:\ProgramData\BOINC/projects/boinc.loda-lang.org_loda\git\cmd;C:\Program Files\Git\usr\bin;C:\ProgramData\BOINC/projects/boinc.loda-lang.org_loda\git\usr\bin
2022-12-06 12:40:55|WARN |Setting environment variable: TMP=C:\ProgramData\BOINC/projects/boinc.loda-lang.org_loda\
2022-12-06 12:40:55|WARN |Setting environment variable: TEMP=C:\ProgramData\BOINC/projects/boinc.loda-lang.org_loda\
2022-12-06 12:40:55|INFO |Loading sequences from the OEIS index
2022-12-06 12:40:59|INFO |Loaded 337318/358544 sequences in 4.45s
2022-12-06 12:41:06|INFO |Initialized 5 matchers (ignoring 123223 sequences)
2022-12-06 12:41:07|INFO |Loaded 500 patterns
2022-12-06 12:41:07|INFO |Initialized 1 generators (profile: pattern, overwrite: none)
2022-12-06 12:41:07|INFO |Mining programs in client mode (extended validation mode)
2022-12-06 12:42:18|INFO |Processed 46 programs, 1.2%  <=====Entered Waiting to Run, suspended for ~20 minutes
2022-12-06 13:01:14|INFO |Processed 147 programs, 16.9% <=====Entered Running, progress jumped by 15%
2022-12-06 13:03:21|INFO |Processed 414 programs, 18.7% <=====Entered Waiting to Run, suspended for ~40 minutes
2022-12-06 13:42:05|INFO |Processed 57 programs, 51.0% <=====Entered Running, progress jumped by 32%
2022-12-06 13:42:41|INFO |Processed 536 programs, 51.5%
2022-12-06 13:43:24|INFO |Processed 149 programs, 52.1%
2022-12-06 13:44:01|INFO |Processed 261 programs, 52.6%
2022-12-06 13:46:41|INFO |Processed 62 programs, 54.8%
2022-12-06 13:47:22|INFO |Processed 15 programs, 55.4%
2022-12-06 13:48:26|INFO |Processed 682 programs, 56.3%
2022-12-06 13:49:02|INFO |Processed 257 programs, 56.8% <=====Entered Waiting to Run, suspended for ~30 minutes
2022-12-06 14:19:14|INFO |Processed 133 programs, 81.9% <=====Entered Running, progress jumped by 25%
ID: 656 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Christian Krause
Project administrator

Send message
Joined: 9 May 22
Posts: 250
Credit: 449,267
RAC: 198
Message 659 - Posted: 6 Dec 2022, 20:43:29 UTC - in response to Message 656.  

Can you maybe try to reduce the number of parallel / sleeping tasks?
You can set the Max Number of Jobs here (maybe also supported in other projects):
https://boinc.loda-lang.org/loda/prefs.php?subset=project
ID: 659 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
AnandBhat

Send message
Joined: 14 May 22
Posts: 10
Credit: 197,581
RAC: 1,779
Message 660 - Posted: 6 Dec 2022, 23:57:10 UTC - in response to Message 659.  

As I run LODA with resource share 0, I only get the tasks needed to keep my CPUs busy (16 at a time for the 16 threads) and have no queued LODA tasks at any given time. However, if I get work from another project with an earlier deadline, BOINC suspends the LODA tasks while it works on the ones that expire earlier.

I will stop requesting work for other projects if I'm running LODA going forward until this is addressed.
ID: 660 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Questions and Answers : Bugs : Short run times for some workunits

©2024 LODA Language