User: dajiang@af.uchicago.edu Condor Job Id: 1447666.0
Walltime (from condor_q): 35:49:17 CPU time (from condor_q): 35:23:24 ServerTime: 1773567302 JobStartDate: 1773438345 RemoteUserCpu: 127300.0 RemoteSysCpu: 104.0 ResidentSetSize: 2250000 x509UserProxyVOName: RemoteHost: slot1_2@c076.af.uchicago.edu
ActivationSetupDuration: 1 Args: 1000_687.5 mHpdM_fit_withSystematics_medium /data/dajiang/2hdm/v0.5_histograms_rebinned/ /data/dajiang/2hdm/trexfitter_outputs//step_trexfitter_batchType_condor_13-03-2026-164307 /home/dajiang/2hdm/test/trexfitter/dagman_run_step_trexfitter_batchType_condor_13-03-2026-164307 AutoClusterAttrs: draining,for,yum,updates,ERROR,cannot,access,/cvmfs/atlas,FirstUpdateUptimeGPUsSeconds,LastUpdateUptimeGPUsSeconds,MachineLastMatchTime,No,cache,information,for,host,workers-ads-68f974c8b9-24qwf,No,cache,information,for,host,workers-ads-68f974c8b9-46h6t,No,cache,information,for,host,workers-ads-68f974c8b9-5865f,No,cache,information,for,host,workers-ads-68f974c8b9-6bpwb,No,cache,information,for,host,workers-ads-68f974c8b9-7hth5,No,cache,information,for,host,workers-ads-68f974c8b9-7kjz6,No,cache,information,for,host,workers-ads-68f974c8b9-7v5lc,No,cache,information,for,host,workers-ads-68f974c8b9-87h96,No,cache,information,for,host,workers-ads-68f974c8b9-8j7c7,No,cache,information,for,host,workers-ads-68f974c8b9-92xgk,No,cache,information,for,host,workers-ads-68f974c8b9-94mn5,No,cache,information,for,host,workers-ads-68f974c8b9-9hpv4,No,cache,information,for,host,workers-ads-68f974c8b9-9jvbf,No,cache,information,for,host,workers-ads-68f974c8b9-9k82n,No,cache,information,for,host,workers-ads-68f974c8b9-c2c54,No,cache,information,for,host,workers-ads-68f974c8b9-cx7dq,No,cache,information,for,host,workers-ads-68f974c8b9-dc5d8,No,cache,information,for,host,workers-ads-68f974c8b9-dg4kc,No,cache,information,for,host,workers-ads-68f974c8b9-dhm5m,No,cache,information,for,host,workers-ads-68f974c8b9-dnssg,No,cache,information,for,host,workers-ads-68f974c8b9-dpc22,No,cache,information,for,host,workers-ads-68f974c8b9-flxpn,No,cache,information,for,host,workers-ads-68f974c8b9-gbbbd,No,cache,information,for,host,workers-ads-68f974c8b9-gsrh2,No,cache,information,for,host,workers-ads-68f974c8b9-hs47h,No,cache,information,for,host,workers-ads-68f974c8b9-jjt5k,No,cache,information,for,host,workers-ads-68f974c8b9-jlzcn,No,cache,information,for,host,workers-ads-68f974c8b9-jx4k4,No,cache,information,for,host,workers-ads-68f974c8b9-kbqvx,No,cache,information,for,host,workers-ads-68f974c8b9-ksgk4,No,cache,information,for,host,workers-ads-68f974c8b9-l6vs8,No,cache,information,for,host,workers-ads-68f974c8b9-lpmcf,No,cache,information,for,host,workers-ads-68f974c8b9-m9kkd,No,cache,information,for,host,workers-ads-68f974c8b9-mncd9,No,cache,information,for,host,workers-ads-68f974c8b9-mnvgn,No,cache,information,for,host,workers-ads-68f974c8b9-nxrhw,No,cache,information,for,host,workers-ads-68f974c8b9-p8ccq,No,cache,information,for,host,workers-ads-68f974c8b9-pjvtf,No,cache,information,for,host,workers-ads-68f974c8b9-pmv2f,No,cache,information,for,host,workers-ads-68f974c8b9-pshrz,No,cache,information,for,host,workers-ads-68f974c8b9-q47kj,No,cache,information,for,host,workers-ads-68f974c8b9-qkfv5,No,cache,information,for,host,workers-ads-68f974c8b9-t7tfr,No,cache,information,for,host,workers-ads-68f974c8b9-v9x2g,No,cache,information,for,host,workers-ads-68f974c8b9-vckkz,No,cache,information,for,host,workers-ads-68f974c8b9-xcd4n,No,cache,information,for,host,workers-ads-68f974c8b9-xrmj6,No,cache,information,for,host,workers-ads-68f974c8b9-xsf2d,No,cache,information,for,host,workers-ads-68f974c8b9-xxmx9,No,cache,information,for,host,workers-ads-68f974c8b9-zpt2r,No,cache,information,for,host,workers-gpu-59bc5ffc7c-v46w8,No,cache,information,for,host,workers-login-864c89bcdc-6khjm,No,cache,information,for,host,workers-login-864c89bcdc-6qbhs,No,cache,information,for,host,workers-login-864c89bcdc-fnz5v,No,cache,information,for,host,workers-login-864c89bcdc-h725p,No,cache,information,for,host,workers-login-864c89bcdc-nmjph,No,cache,information,for,host,workers-login-864c89bcdc-s6865,No,cache,information,for,host,workers-login-864c89bcdc-tlbfm,No,cache,information,for,host,workers-login-864c89bcdc-v7h2m,queue,RemoteOwner,RequestCpus,RequestDisk,RequestGPUs,RequestMemory,StartOfJobUptimeGPUsSeconds,TotalJobRuntime,UptimeGPUsSeconds,ConcurrencyLimits,FlockTo,Rank,Requirements,ALLOW_MWT2,/cvmfs/atlas-condb,workers-login-864c89bcdc-5pcrd,workers-login-864c89bcdc-cp5qd,workers-login-864c89bcdc-b674h,NodeOnline,workers-login-864c89bcdc-hpllh,workers-login-864c89bcdc-mhghh,workers-login-864c89bcdc-shgcq,workers-login-864c89bcdc-8cshl,workers-login-864c89bcdc-vn7mr,workers-login-864c89bcdc-sckck,workers-login-864c89bcdc-8vl24,workers-login-864c89bcdc-zd8gj,c056,c089,c106,c049,/data:,Permission,denied,c001,workers-gpu-59bc5ffc7c-rccbg,c078,c045,c092,c058,c061,c004,c070,c096,c087,c100,workers-ads-68f974c8b9-gsc7x,workers-ads-68f974c8b9-2tz86,workers-ads-68f974c8b9-blsph,workers-ads-68f974c8b9-chqm4,workers-ads-68f974c8b9-dxpsm,workers-ads-68f974c8b9-wvrj7,workers-ads-68f974c8b9-btrr9,workers-ads-68f974c8b9-c6mpf,workers-ads-68f974c8b9-fkl6d,workers-ads-68f974c8b9-gzrrm,workers-ads-68f974c8b9-k5982,workers-ads-68f974c8b9-twqjg,workers-ads-68f974c8b9-l5f76,workers-ads-68f974c8b9-9mvsg,workers-ads-68f974c8b9-jldbp,workers-ads-68f974c8b9-pl8bn,workers-ads-68f974c8b9-wr2j5,workers-ads-68f974c8b9-5l6vc,workers-ads-68f974c8b9-pdfgv,workers-ads-68f974c8b9-466m5,workers-ads-68f974c8b9-6cbl6,workers-ads-68f974c8b9-jjfft,workers-ads-68f974c8b9-6r5b2,workers-ads-68f974c8b9-ckrrj,workers-ads-68f974c8b9-cw2jz,workers-ads-68f974c8b9-lv7dp,workers-ads-68f974c8b9-zrcqz,workers-ads-68f974c8b9-5mwcn,workers-ads-68f974c8b9-wvhh9,workers-ads-68f974c8b9-lh6xd,workers-login-864c89bcdc-llsfr,workers-login-864c89bcdc-7s457,c019,c039,c098,c018,c101,c047,c010,c002,c003,c021,c036,c048,c057,c059,c065,c072,c073,c074,c075,c079,c090,c093,c107,c006,c016,c017,c023,c025,c038,c044,c063,c077,c081,c085,c091,c097,c104,c108,c007,c009,c024,c040,c041,c043,c053,c054,c060,c062,c069,c076,c080,c082,c088,c095,c099,c103,c105,c022,c046,c051,c052,c055,c064,c066,c068,c094,c013,c037,c042,c050,c067,c071,c084,c086,c109,c110,workers-gpu-59bc5ffc7c-rrk7s,workers-gpu-59bc5ffc7c-lwzj9,c030,c012,workers-ads-68f974c8b9-6d66m,workers-ads-68f974c8b9-7skfd,workers-ads-68f974c8b9-zkf8v,workers-ads-68f974c8b9-9gkdv,workers-ads-68f974c8b9-2qqtv,workers-ads-68f974c8b9-w7xdw,workers-ads-68f974c8b9-79zjc,workers-ads-68f974c8b9-85fjn,workers-ads-68f974c8b9-dj58n,workers-ads-68f974c8b9-dkw2k,workers-ads-68f974c8b9-m4b5v,workers-ads-68f974c8b9-npqmp,workers-ads-68f974c8b9-v7grk,workers-ads-68f974c8b9-kd87j,workers-ads-68f974c8b9-f9lm6,workers-ads-68f974c8b9-p6bx5,workers-ads-68f974c8b9-sj62w,workers-ads-68f974c8b9-5bm75,workers-ads-68f974c8b9-gghmq,workers-ads-68f974c8b9-f6lzx,workers-ads-68f974c8b9-l5nh8,workers-ads-68f974c8b9-bl962,workers-ads-68f974c8b9-bmtqd,workers-ads-68f974c8b9-2s9dt,workers-ads-68f974c8b9-6mzgw,workers-ads-68f974c8b9-fr6lp,workers-ads-68f974c8b9-5qhjx,workers-ads-68f974c8b9-c5s5z,workers-ads-68f974c8b9-r56rp,workers-ads-68f974c8b9-sxt9s,workers-login-864c89bcdc-7vdkx,workers-login-864c89bcdc-l8j9n,workers-login-864c89bcdc-pw72c,workers-login-864c89bcdc-pdb67,workers-login-864c89bcdc-w8v4p,PDU,replacement,workers-login-864c89bcdc-wkrjv,workers-login-864c89bcdc-8zt7j,workers-login-864c89bcdc-5mtth,DiskUsage,FileSystemDomain AutoClusterId: 71 BlockReadKbytes: 0 BlockReads: 0 BlockWriteKbytes: 0 BlockWrites: 0 BytesRecvd: 2906.0 BytesSent: 0.0 ClusterId: 1447666 Cmd: /home/dajiang/2hdm/test/trexfitter/dagman_run_step_trexfitter_batchType_condor_13-03-2026-164307/runTRExFitter.sh CommittedSlotTime: 0 CommittedSuspensionTime: 0 CommittedTime: 0 CondorPlatform: $CondorPlatform: x86_64_AlmaLinux9 $ CondorVersion: $CondorVersion: 25.0.3 2025-10-31 BuildID: 847298 PackageID: 25.0.3-1 GitSHA: dc94bfbb $ CpusProvisioned: 1 CpusUsage: 0.9975567398800385 CumulativeRemoteSysCpu: 104.0 CumulativeRemoteUserCpu: 127300.0 CumulativeSlotTime: 0 CumulativeSuspensionTime: 0 CurrentHosts: 1 DAGManJobId: 1447662 DAGManNodesLog: /home/dajiang/2hdm/test/trexfitter/dagman_run_step_trexfitter_batchType_condor_13-03-2026-164307/./trexfitter_subdag.dag.nodes.log DAGManNodesMask: 0,1,2,4,5,7,8,9,10,11,12,13,16,17,24,27,35,36 DAGNodeName: job_1000_687.5_medium DAGParentNodeNames: DiskProvisioned: 2360 DiskUsage: 125000 DiskUsage_RAW: 102721 EnteredCurrentStatus: 1773438345 Environment: Err: submit_trexfitter_logs/submit_trexfitter_1000_687.5_medium.err ExecutableSize: 3 ExecutableSize_RAW: 3 ExecuteDirWasEncrypted: False ExitBySignal: False ExitStatus: 0 FileSystemDomain: nfs.af.uchicago.edu FirstJobMatchDate: 1773438345 GPUsProvisioned: 0 GlobalJobId: head01.af.uchicago.edu#1447666.0#1773438324 ImageSize: 2250000 ImageSize_RAW: 2046912 In: /dev/null InitialWaitDuration: 21 Iwd: /home/dajiang/2hdm/test/trexfitter/dagman_run_step_trexfitter_batchType_condor_13-03-2026-164307 JobBatchId: 1447660.0 JobBatchName: trexfitter.dag+1447660 JobCurrentReconnectAttempt: Value.Undefined JobCurrentStartDate: 1773438345 JobCurrentStartExecutingDate: 1773438346 JobFlavour: testmatch JobLeaseDuration: 2400 JobNotification: 0 JobPrio: 0 JobRunCount: 1 JobStartDate: 1773438345 JobStatus: 2 JobSubmitFile: /home/dajiang/2hdm/test/trexfitter/dagman_run_step_trexfitter_batchType_condor_13-03-2026-164307/trexfitter_submits/submitTRExFitter_1000_687.5_medium.condor JobSubmitMethod: 1 JobUniverse: 5 LastJobLeaseRenewal: 1773566968 LastJobStatus: 1 LastMatchTime: 1773438345 LastSuspensionTime: 0 LeaveJobInQueue: False MachineAttrCpus0: 1 MachineAttrSlotWeight0: 1 MaxHosts: 1 MemoryProvisioned: 65536 MemoryUsage: ((ResidentSetSize + 1023) / 1024) MinHosts: 1 MyType: Job NumCkpts: 0 NumCkpts_RAW: 0 NumInputTransferStarts: 1 NumJobCompletions: 0 NumJobMatches: 1 NumJobStarts: 1 NumRestarts: 0 NumShadowStarts: 1 NumSystemHolds: 0 OrigMaxHosts: 1 Out: submit_trexfitter_logs/submit_trexfitter_1000_687.5_medium.out Owner: dajiang ProcId: 0 PublicClaimId: <192.170.240.241:9618?CCBID=192.170.240.14:9618%3faddrs%3d192.170.240.14-9618+[2605-9a00-10-200a-e63d-1aff-fe01-80e0]-9618%26alias%3dhead01.af.uchicago.edu%26noUDP%26sock%3dcollector#192041&PrivNet=af.uchicago.edu&addrs=192.170.240.241-9618+[2605-9a00-10-200a-266e-96ff-fe19-be38]-9618&alias=c076.af.uchicago.edu&noUDP&sock=startd_46997_9c86>#1773413395#164#... QDate: 1773438324 Rank: 0.0 RecentBlockReadKbytes: 0 RecentBlockReads: 0 RecentBlockWriteKbytes: 0 RecentBlockWrites: 0 RecentStatsLifetimeStarter: 1200 RemoteHost: slot1_2@c076.af.uchicago.edu RemoteSlotID: 1 RemoteSysCpu: 104.0 RemoteUserCpu: 127300.0 RemoteWallClockTime: 0.0 RequestCpus: 1 RequestDisk: DiskUsage RequestMemory: 65536 Requirements: (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory) && ((TARGET.FileSystemDomain == MY.FileSystemDomain) || (TARGET.HasFileTransfer)) ResidentSetSize: 2250000 ResidentSetSize_RAW: 2046692 ScratchDirFileCount: 17 ServerTime: 1773567302 ShadowBday: 1773438345 ShouldTransferFiles: IF_NEEDED SlotAccount: SlotOwner: dajiang StartdIpAddr: <192.170.240.241:9618?CCBID=192.170.240.14:9618%3faddrs%3d192.170.240.14-9618+[2605-9a00-10-200a-e63d-1aff-fe01-80e0]-9618%26alias%3dhead01.af.uchicago.edu%26noUDP%26sock%3dcollector#192041&PrivNet=af.uchicago.edu&addrs=192.170.240.241-9618+[2605-9a00-10-200a-266e-96ff-fe19-be38]-9618&alias=c076.af.uchicago.edu&noUDP&sock=startd_46997_9c86> StartdPrincipal: execute-side@matchsession/192.170.240.241 StatsLifetimeStarter: 128622 StderrMtime: 1773468620 StdoutMtime: 1773566960 StreamErr: False StreamOut: False SubmitEventNotes: DAG Node: job_1000_687.5_medium TargetType: Machine TotalSubmitProcs: 1 TotalSuspensions: 0 TransferIn: False TransferInFinished: 1773438345 TransferInStarted: 1773438345 TransferInputFileCounts: [ CEDAR = 1 ] TransferInputSizeMB: 0 TransferInputStats: [ CedarSizeBytesTotal = 2906; CedarFilesCountTotal = 1; CedarSizeBytesLastRun = 2906; CedarFilesCountLastRun = 1 ] TransferOutputStats: [ ] TransferQueued: False TransferringInput: False User: dajiang@af.uchicago.edu UserLog: /home/dajiang/2hdm/test/trexfitter/dagman_run_step_trexfitter_batchType_condor_13-03-2026-164307/submit_trexfitter_logs/submit_trexfitter_1000_687.5_medium.log WallClockCheckpoint: 128282 WhenToTransferOutput: ON_EXIT cputime: 127404 csecs: 127404.0 effcy: 98.79572260520949 job_id: 1447666.0 rss: 2.1G submitter: head01.af.uchicago.edu vm: 2.0G walltime: 128957 wsecs: 128957.0