User: fengping@af.uchicago.edu Condor Job Id: 2738235.0
Walltime (from condor_q): 02:34:35 CPU time (from condor_q): 00:03:24 ServerTime: 1785166202 JobStartDate: 1785156927 RemoteUserCpu: 361.0 RemoteSysCpu: 47.0 ResidentSetSize: 175000 x509UserProxyVOName: RemoteHost: slot1_15@c075.af.uchicago.edu
ActivationSetupDuration: 1 Arguments: -c .' './env.sh;' 'python3' '-m' 'distributed.cli.dask_worker' 'tls://dask-2f02dd704b654f8bb63ccf5a5db69fd5.dask-gateway:8786' '--name' 'dask-2f02dd704b654f8bb63ccf5a5db69fd5-000146' '--nthreads' '2' '--memory-limit' '4.00GiB' '--nanny' '--death-timeout' '60' '--dashboard-address' ':8787' '--listen-address' 'tls://0.0.0.0:$CONTAINER_PORT' '--contact-address' 'tls://$HOST:$PORT AuthTokenId: d51a66fbe9c03c3190eaf007f698afd2 AuthTokenIssuer: head01.af.uchicago.edu AuthTokenSubject: fengping@af.uchicago.edu AutoClusterAttrs: FirstUpdateUptimeGPUsSeconds,LastUpdateUptimeGPUsSeconds,MachineLastMatchTime,No,cache,information,for,host,workers-ads-8b88d7889-4bvng,No,cache,information,for,host,workers-ads-8b88d7889-4jxgt,No,cache,information,for,host,workers-ads-8b88d7889-4tsf4,No,cache,information,for,host,workers-ads-8b88d7889-4x72r,No,cache,information,for,host,workers-ads-8b88d7889-5f9kh,No,cache,information,for,host,workers-ads-8b88d7889-5kcll,No,cache,information,for,host,workers-ads-8b88d7889-6sqkx,No,cache,information,for,host,workers-ads-8b88d7889-76gth,No,cache,information,for,host,workers-ads-8b88d7889-87gj6,No,cache,information,for,host,workers-ads-8b88d7889-b4jwh,No,cache,information,for,host,workers-ads-8b88d7889-f4l48,No,cache,information,for,host,workers-ads-8b88d7889-hjfx6,No,cache,information,for,host,workers-ads-8b88d7889-j9vzx,No,cache,information,for,host,workers-ads-8b88d7889-jqgt7,No,cache,information,for,host,workers-ads-8b88d7889-k6mj5,No,cache,information,for,host,workers-ads-8b88d7889-l28b5,No,cache,information,for,host,workers-ads-8b88d7889-l7jbl,No,cache,information,for,host,workers-ads-8b88d7889-lrm6n,No,cache,information,for,host,workers-ads-8b88d7889-mbzkq,No,cache,information,for,host,workers-ads-8b88d7889-mmszb,No,cache,information,for,host,workers-ads-8b88d7889-nqlmv,No,cache,information,for,host,workers-ads-8b88d7889-nzw4f,No,cache,information,for,host,workers-ads-8b88d7889-p5qvf,No,cache,information,for,host,workers-ads-8b88d7889-pw8gz,No,cache,information,for,host,workers-ads-8b88d7889-qt7mj,No,cache,information,for,host,workers-ads-8b88d7889-r2rmw,No,cache,information,for,host,workers-ads-8b88d7889-r9rll,No,cache,information,for,host,workers-ads-8b88d7889-rlkbv,No,cache,information,for,host,workers-ads-8b88d7889-vh95k,No,cache,information,for,host,workers-ads-8b88d7889-vqkgz,No,cache,information,for,host,workers-ads-8b88d7889-wbqpk,No,cache,information,for,host,workers-ads-8b88d7889-wzpn8,No,cache,information,for,host,workers-ads-8b88d7889-zkldx,No,cache,information,for,host,workers-gpu-5d8bdf5f6f-nrh79,No,cache,information,for,host,workers-gpu-5d8bdf5f6f-r27sq,No,cache,information,for,host,workers-gpu-5d8bdf5f6f-vk89q,No,cache,information,for,host,workers-login-7686fc76f7-jhtzl,No,cache,information,for,host,workers-login-7686fc76f7-nrx98,No,cache,information,for,host,workers-login-7686fc76f7-xlfjx,queue,RemoteOwner,RequestCpus,RequestDisk,RequestGPUs,RequestMemory,StartOfJobUptimeGPUsSeconds,TotalJobRuntime,UptimeGPUsSeconds,ConcurrencyLimits,FlockTo,Rank,Requirements,workers-ads-8b88d7889-cfvrk,workers-ads-8b88d7889-5qnpd,workers-login-7686fc76f7-pmxkp,workers-ads-8b88d7889-4w7sv,workers-ads-8b88d7889-9rm8l,workers-ads-8b88d7889-ddl2x,workers-ads-8b88d7889-j9498,workers-ads-8b88d7889-z7xbg,workers-ads-8b88d7889-5llv2,workers-ads-8b88d7889-2xgw6,workers-ads-8b88d7889-9hzxw,workers-ads-8b88d7889-dg244,workers-ads-8b88d7889-j8hnn,workers-ads-8b88d7889-l4hrj,workers-ads-8b88d7889-lsxq6,workers-ads-8b88d7889-mfsnr,workers-ads-8b88d7889-wr68p,workers-ads-8b88d7889-r6mjx,workers-ads-8b88d7889-msdw8,workers-ads-8b88d7889-q2c7t,workers-ads-8b88d7889-r5wgh,workers-ads-8b88d7889-x4j5d,workers-ads-8b88d7889-lwtcd,workers-ads-8b88d7889-j5jxk,workers-ads-8b88d7889-dzwfk,workers-ads-8b88d7889-qr9rd,workers-ads-8b88d7889-wpm77,workers-ads-8b88d7889-f8tzx,workers-ads-8b88d7889-krg2r,workers-ads-8b88d7889-pcjrl,workers-ads-8b88d7889-rtlld,workers-ads-8b88d7889-drx5j,workers-ads-8b88d7889-8fn9n,workers-ads-8b88d7889-9bmvd,workers-ads-8b88d7889-c65f7,workers-ads-8b88d7889-ckkr5,workers-ads-8b88d7889-4cwrd,workers-ads-8b88d7889-6cc67,workers-ads-8b88d7889-9psrf,workers-ads-8b88d7889-7ssd7,workers-ads-8b88d7889-9m7tv,workers-ads-8b88d7889-pnxbj,workers-ads-8b88d7889-rbq9g,workers-gpu-5d8bdf5f6f-9klrm,workers-ads-8b88d7889-bjf7l,workers-ads-8b88d7889-cbdhz,workers-ads-8b88d7889-4cjr6,workers-ads-8b88d7889-qrmst,workers-ads-8b88d7889-vr9j7,workers-ads-8b88d7889-56tlc,workers-ads-8b88d7889-9kt79,workers-ads-8b88d7889-nm4mp,workers-ads-8b88d7889-p4xm6,workers-ads-8b88d7889-m26c9,workers-ads-8b88d7889-n8xbm,workers-ads-8b88d7889-qnrhj,workers-ads-8b88d7889-4d4dd,workers-ads-8b88d7889-7z88d,workers-ads-8b88d7889-4xn27,workers-ads-8b88d7889-76pw7,workers-ads-8b88d7889-lrvnt,workers-ads-8b88d7889-p4spm,workers-ads-8b88d7889-xn7vx,workers-ads-8b88d7889-ztlpv,workers-gpu-5d8bdf5f6f-b8s9m,workers-ads-8b88d7889-cxxf2,workers-ads-8b88d7889-jsrmb,workers-ads-8b88d7889-wbjmr,workers-ads-8b88d7889-bplmd,workers-ads-8b88d7889-xmdhz,workers-ads-8b88d7889-44dgp,workers-ads-8b88d7889-jf7rh,workers-ads-8b88d7889-q795f,workers-login-7686fc76f7-5zwdb,workers-ads-8b88d7889-fqsvk,workers-ads-8b88d7889-2mx8s,workers-ads-8b88d7889-dktsz,workers-ads-8b88d7889-h5l2s,workers-ads-8b88d7889-6hw2c,workers-ads-8b88d7889-b9pm9,workers-ads-8b88d7889-cw6p9,workers-ads-8b88d7889-dvbs6,workers-ads-8b88d7889-h8tpx,workers-ads-8b88d7889-t6vn4,workers-gpu-5d8bdf5f6f-9djs9,workers-login-7686fc76f7-bctnl,workers-ads-8b88d7889-2rrsc,workers-ads-8b88d7889-8c525,workers-login-7686fc76f7-xt578,workers-ads-8b88d7889-cbz7k,workers-ads-8b88d7889-mkw7v,workers-login-7686fc76f7-wzkwf,workers-ads-8b88d7889-hfz2c,workers-ads-8b88d7889-x5ctl,workers-ads-8b88d7889-hgvkk,workers-ads-8b88d7889-lt46h,workers-ads-8b88d7889-2tpql,workers-ads-8b88d7889-7mbf2,workers-ads-8b88d7889-wlxrx,workers-ads-8b88d7889-8wp8j,workers-ads-8b88d7889-9c9lf,ALLOW_MWT2,ERROR,only,0,GB,free,in,/,NodeOnline,benchmarking,cannot,access,/data:,Permission,denied,DaskWorkerCores,DaskWorkerDisk,DaskWorkerMemory,Machine AutoClusterId: 9371 BytesRecvd: 133375.0 BytesSent: 0.0 ClusterId: 2738235 Cmd: sh CommittedSlotTime: 0 CommittedSuspensionTime: 0 CommittedTime: 0 CondorPlatform: $CondorPlatform: X86_64-Debian_11 $ CondorVersion: $CondorVersion: 24.12.19 2026-04-15 BuildID: 896241 PackageID: 24.12.19-1+deb11 GitSHA: c68f3e0e $ ContainerServiceNames: dask,nanny CpusProvisioned: 2 CpusUsage: 0.0006680513153007725 CumulativeRemoteSysCpu: 47.0 CumulativeRemoteUserCpu: 361.0 CumulativeSlotTime: 0 CumulativeSuspensionTime: 0 CurrentHosts: 1 DaskWorkerCores: 2 DaskWorkerDisk: 5000000000 DaskWorkerMemory: 4294967296 DaskWorkerName: htcondor--2738235.0-- DiskProvisioned: 4884803 DiskUsage: 175 DiskUsage_RAW: 159 DockerImage: hub.opensciencegrid.org/usatlas/analysis-dask-base:dev EnteredCurrentStatus: 1785156927 Environment: DASK_GATEWAY_API_URL=http://api-dask-gateway.dask-gateway:8000/api DASK_GATEWAY_CLUSTER_NAME=dask-gateway.2f02dd704b654f8bb63ccf5a5db69fd5 Err: /dev/null ExecutableSize: 125 ExecutableSize_RAW: 123 ExecuteDirWasEncrypted: False ExitBySignal: False ExitStatus: 0 FirstJobMatchDate: 1785156927 GPUsProvisioned: 0 GlobalJobId: head01.af.uchicago.edu#2738235.0#1785156919 HoldReason: Value.Undefined HoldReasonCode: Value.Undefined ImageSize: 175 ImageSize_RAW: 163 In: /dev/null InitialWaitDuration: 8 Iwd: /var/lib/condor/spool/8235/0/cluster2738235.proc0.subproc0 JobBatchName: dask-2f02dd704b654f8bb63ccf5a5db69fd5-000146 JobCurrentReconnectAttempt: Value.Undefined JobCurrentStartDate: 1785156927 JobCurrentStartExecutingDate: 1785156928 JobId: 2738235.0 JobLeaseDuration: 2400 JobNotification: 0 JobPrio: 0 JobRunCount: 1 JobStartDate: 1785156927 JobStatus: 2 JobSubmitFile: /tmp/tmp5vq44fq9.sh JobSubmitMethod: 0 JobUniverse: 5 LastHoldReason: Spooling input data files LastHoldReasonCode: 16 LastJobLeaseRenewal: 1785165653 LastJobStatus: 1 LastMatchTime: 1785156927 LastSuspensionTime: 0 LeaveJobInQueue: JobStatus == 4 && (CompletionDate is undefined || CompletionDate == 0 || ((time() - CompletionDate) < 864000)) MachineAttrCpus0: 2 MachineAttrSlotWeight0: 2 MaxHosts: 1 MemoryProvisioned: 4096 MemoryUsage: 163 MinHosts: 1 MyType: Job NetworkInputMb: 3.088394 NetworkOutputMb: 13.579862 NumCkpts: 0 NumCkpts_RAW: 0 NumInputTransferStarts: 1 NumJobCompletions: 0 NumJobMatches: 1 NumJobStarts: 1 NumRestarts: 0 NumShadowStarts: 1 NumSystemHolds: 0 OrigMaxHosts: 1 Out: /dev/null Owner: fengping ProcId: 0 PublicClaimId: <192.170.240.240:9618?CCBID=192.170.240.14:9618%3faddrs%3d192.170.240.14-9618+[2605-9a00-10-200a-e63d-1aff-fe01-80e0]-9618%26alias%3dhead01.af.uchicago.edu%26noUDP%26sock%3dcollector#302221&PrivNet=af.uchicago.edu&addrs=192.170.240.240-9618+[2605-9a00-10-200a-266e-96ff-fe19-cd60]-9618&alias=c075.af.uchicago.edu&noUDP&sock=startd_56691_e39d>#1784528345#88864#... QDate: 1785156919 Rank: 0.0 ReleaseReason: Data files spooled RemoteHost: slot1_15@c075.af.uchicago.edu RemoteSlotID: 1 RemoteSysCpu: 47.0 RemoteUserCpu: 361.0 RemoteWallClockTime: 0.0 RequestCpus: MY.DaskWorkerCores RequestDisk: floor(MY.DaskWorkerDisk / 1024) RequestMemory: floor(MY.DaskWorkerMemory / 1048576) Requirements: ( !(Machine is "c111.af.uchicago.edu" || Machine is "c112.af.uchicago.edu" || Machine is "c113.af.uchicago.edu" || Machine is "c114.af.uchicago.edu" || Machine is "c115.af.uchicago.edu")) && (TARGET.Arch == "X86_64") && TARGET.HasDocker && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory) && (TARGET.Cpus >= RequestCpus) && (TARGET.HasFileTransfer) ResidentSetSize: 175000 ResidentSetSize_RAW: 167284 SUBMIT_Cmd: /bin/sh SUBMIT_Iwd: /home/fengping SUBMIT_TransferInput: /etc/condor/dask/env.sh,/tmp/tmp9l1dbphi/api-token,/tmp/tmp9l1dbphi/dask.crt,/tmp/tmp9l1dbphi/dask.pem ScratchDirFileCount: 20 ServerTime: 1785166202 ShadowBday: 1785156927 ShouldTransferFiles: YES SlotAccount: SlotOwner: fengping StageInFinish: 1785156919 StageInStart: 1785156919 StartdIpAddr: <192.170.240.240:9618?CCBID=192.170.240.14:9618%3faddrs%3d192.170.240.14-9618+[2605-9a00-10-200a-e63d-1aff-fe01-80e0]-9618%26alias%3dhead01.af.uchicago.edu%26noUDP%26sock%3dcollector#302221&PrivNet=af.uchicago.edu&addrs=192.170.240.240-9618+[2605-9a00-10-200a-266e-96ff-fe19-cd60]-9618&alias=c075.af.uchicago.edu&noUDP&sock=startd_56691_e39d> StartdPrincipal: execute-side@matchsession/192.170.240.240 TargetType: Machine TotalSubmitProcs: 1 TotalSuspensions: 0 TransferErr: False TransferIn: False TransferInFinished: 1785156927 TransferInStarted: 1785156927 TransferInput: env.sh,api-token,dask.crt,dask.pem TransferInputFileCounts: [ CEDAR = 5 ] TransferInputSizeMB: 0 TransferInputStats: [ CedarSizeBytesTotal = 133375; CedarFilesCountTotal = 5; CedarSizeBytesLastRun = 133375; CedarFilesCountLastRun = 5 ] TransferOut: False TransferOutputRemaps: Value.Undefined TransferOutputStats: [ ] TransferQueued: False TransferringInput: False User: fengping@af.uchicago.edu WallClockCheckpoint: 7277 WantDocker: True WhenToTransferOutput: ON_EXIT cputime: 204 csecs: 204.0 dask_ContainerPort: 8786 dask_HostPort: 38997 effcy: 2.1994609164420487 job_id: 2738235.0 nanny_ContainerPort: 8001 nanny_HostPort: 34783 queue: short rss: 170.9M submitter: head01.af.uchicago.edu vm: 163.0K walltime: 9275 wsecs: 9275.0