Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 207.11806ms FAIL (2025-08-07 23:59:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 207.143327ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.XXXXX..X...................................................] duration: 223.222388ms FAIL (2025-08-07 23:59:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 223.241745ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.XXXXX..X............................................XXXXXXX] duration: 1m0.003043095s FAIL (2025-08-07 23:59:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.003057852s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....X.X..........................................X.........] duration: 544.6098ms FAIL (2025-08-07 23:59:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 544.627984ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.394191895s FAIL (2025-08-07 23:59:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== 1345s) [upload=7262]: Got next segment. Inline: false Aug 7 23:31:37.583906637 (7h4m51.170962878s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:31:37.583953414 (7h4m51.171009656s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:31:37.613963041 (7h4m51.201019262s) [seg_pos={0 0}, upload=7262]: Begin upload segment... Aug 7 23:31:37.614008966 (7h4m51.201065176s) [seg_pos={0 0}, upload=7262]: Done begin upload segment. Aug 7 23:31:37.614029103 (7h4m51.201085324s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614108311 (7h4m51.201164542s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614344612 (7h4m51.201400833s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614634754 (7h4m51.201690975s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614919926 (7h4m51.201976137s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614943150 (7h4m51.201999391s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615146719 (7h4m51.20220294s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615454163 (7h4m51.202510384s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615514677 (7h4m51.202570918s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615688160 (7h4m51.202744381s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615932236 (7h4m51.202988457s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616184166 (7h4m51.203240378s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616521716 (7h4m51.203577937s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616643614 (7h4m51.203699855s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621231117 (7h4m51.208287338s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621561163 (7h4m51.208617374s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621861323 (7h4m51.208917534s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.622156223 (7h4m51.209212444s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.726040884 (7h4m51.313097094s) [upload=7262]: Next returned nil segment Aug 7 23:31:37.726066772 (7h4m51.313122983s) [upload=7262]: Waiting for error group managing segments... Aug 7 23:31:37.736232517 (7h4m51.323288738s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.737656014 (7h4m51.324712245s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.739754932 (7h4m51.326811153s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.740628113 (7h4m51.327684344s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741712657 (7h4m51.328768867s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741742653 (7h4m51.328798883s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.748242274 (7h4m51.335298505s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.750434577 (7h4m51.337490798s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.758962364 (7h4m51.346018586s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.769693706 (7h4m51.356749917s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.800540734 (7h4m51.387596975s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.924012655 (7h4m51.511068866s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.059695219 (7h4m51.64675144s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102515917 (7h4m51.689572138s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102541606 (7h4m51.689597827s) [seg_pos={0 0}, upload=7262]: Segment reached optimal threshold of 14 pieces. Aug 7 23:31:38.102658574 (7h4m51.689714795s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF Aug 7 23:31:38.102748111 (7h4m51.689804332s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE): protocol: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection Aug 7 23:31:38.102832158 (7h4m51.689888399s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GcyWwKkfavsCsmtq55U72n6XeZKqQiGGYHY9D6UwST8sC14Ac): protocol: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection Aug 7 23:31:38.102967200 (7h4m51.690023421s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12syasUPsyYf4UstQ6cFAr6qamze3vCyAm9dnobk5e83w4bTz69): protocol: context canceled; context canceled Aug 7 23:31:38.102995132 (7h4m51.690051374s) [seg_pos={0 0}, upload=7262]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:31:38.103097553 (7h4m51.690153775s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:31:38.103102412 (7h4m51.690158633s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:31:38.203633212 (7h4m51.790689433s) [upload=7262]: Done uploading segments... Aug 7 23:31:38.203649102 (7h4m51.790705323s) [upload=7262]: Upload finished. err: <nil> Duration: 1.39421606s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXX.X.X...........................................X.X..XXX..] duration: 1.838960153s FAIL (2025-08-07 23:59:06 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.838986975s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [X.XX............................................XXXXXXXXX...] duration: 184.883459ms FAIL (2025-08-07 23:58:59 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 (7h23m15.650779768s) [upload=7574]: Upload finished. err: <nil> Aug 7 23:50:41.160166556 (7h23m54.747222777s) [upload=7588]: Starting upload Aug 7 23:50:41.160220555 (7h23m54.747276777s) [upload=7588]: Done starting upload Aug 7 23:50:41.160264659 (7h23m54.74732088s) [upload=7588]: Uploading segments... Aug 7 23:50:41.160295956 (7h23m54.747352177s) [upload=7588]: Got next segment. Inline: true Aug 7 23:50:41.160299753 (7h23m54.747355964s) [upload=7588]: Waiting for error group managing segments... Aug 7 23:50:41.160331633 (7h23m54.747387844s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:41.160335861 (7h23m54.747392072s) [upload=7588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:41.160338396 (7h23m54.747394617s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:41.222859735 (7h23m54.809915976s) [upload=7588]: Done uploading segments... Aug 7 23:50:41.222879913 (7h23m54.809936154s) [upload=7588]: Upload finished. err: <nil> Aug 7 23:51:17.131690986 (7h24m30.718747197s) [upload=7599]: Starting upload Aug 7 23:51:17.131738926 (7h24m30.718795147s) [upload=7599]: Done starting upload Aug 7 23:51:17.131792947 (7h24m30.718849178s) [upload=7599]: Uploading segments... Aug 7 23:51:17.131835896 (7h24m30.718892117s) [upload=7599]: Got next segment. Inline: true Aug 7 23:51:17.131842027 (7h24m30.718898239s) [upload=7599]: Waiting for error group managing segments... Aug 7 23:51:17.131883385 (7h24m30.718939627s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:17.131889146 (7h24m30.718945388s) [upload=7599]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:17.132006205 (7h24m30.719062435s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:38.787433326 (7h24m52.374489567s) [upload=7599]: Done uploading segments... Aug 7 23:51:38.787452212 (7h24m52.374508453s) [upload=7599]: Upload finished. err: <nil> Aug 7 23:52:00.256204647 (7h25m13.843260858s) [upload=7610]: Starting upload Aug 7 23:52:00.256263547 (7h25m13.843319758s) [upload=7610]: Done starting upload Aug 7 23:52:00.256316195 (7h25m13.843372406s) [upload=7610]: Uploading segments... Aug 7 23:52:00.256345671 (7h25m13.843401881s) [upload=7610]: Got next segment. Inline: true Aug 7 23:52:00.256349758 (7h25m13.843405969s) [upload=7610]: Waiting for error group managing segments... Aug 7 23:52:00.256399992 (7h25m13.843456203s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:00.256404780 (7h25m13.843460991s) [upload=7610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:00.256407806 (7h25m13.843464017s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:30.237600164 (7h25m43.824656416s) [upload=7610]: Done uploading segments... Aug 7 23:52:30.237630862 (7h25m43.824687113s) [upload=7610]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:53.146251246 (7h26m6.733307456s) [upload=7626]: Starting upload Aug 7 23:52:53.146307339 (7h26m6.733363561s) [upload=7626]: Done starting upload Aug 7 23:52:53.146350962 (7h26m6.733407173s) [upload=7626]: Uploading segments... Aug 7 23:52:53.146411344 (7h26m6.733467565s) [upload=7626]: Got next segment. Inline: true Aug 7 23:52:53.146415271 (7h26m6.733471493s) [upload=7626]: Waiting for error group managing segments... Aug 7 23:52:53.146449966 (7h26m6.733506177s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:53.146454174 (7h26m6.733510395s) [upload=7626]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:53.146456709 (7h26m6.73351293s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.132536521 (7h26m36.719592742s) [upload=7626]: Done uploading segments... Aug 7 23:53:23.132560907 (7h26m36.719617128s) [upload=7626]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:28.217132518 (7h26m41.804188759s) [upload=7638]: Starting upload Aug 7 23:53:28.217216185 (7h26m41.804272426s) [upload=7638]: Done starting upload Aug 7 23:53:28.217288009 (7h26m41.80434426s) [upload=7638]: Uploading segments... Aug 7 23:53:28.217336129 (7h26m41.80439237s) [upload=7638]: Got next segment. Inline: true Aug 7 23:53:28.217341890 (7h26m41.804398131s) [upload=7638]: Waiting for error group managing segments... Aug 7 23:53:28.217421438 (7h26m41.804477679s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:28.217427650 (7h26m41.804483901s) [upload=7638]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:28.217431617 (7h26m41.804487859s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:58.204830989 (7h27m11.79188721s) [upload=7638]: Done uploading segments... Aug 7 23:53:58.204849294 (7h27m11.791905505s) [upload=7638]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:01.364416813 (7h27m14.951473054s) [upload=7648]: Starting upload Aug 7 23:54:01.364544481 (7h27m14.951600732s) [upload=7648]: Done starting upload Aug 7 23:54:01.364639518 (7h27m14.95169577s) [upload=7648]: Uploading segments... Aug 7 23:54:01.364703858 (7h27m14.951760109s) [upload=7648]: Got next segment. Inline: true Aug 7 23:54:01.364713517 (7h27m14.951769757s) [upload=7648]: Waiting for error group managing segments... Aug 7 23:54:01.364781374 (7h27m14.951837625s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:01.364791231 (7h27m14.951847472s) [upload=7648]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:01.364797502 (7h27m14.951853754s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:31.351186626 (7h27m44.938242847s) [upload=7648]: Done uploading segments... Aug 7 23:54:31.351241118 (7h27m44.938297339s) [upload=7648]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:31.416140839 (7h27m45.003197049s) [upload=7656]: Starting upload Aug 7 23:54:31.416189309 (7h27m45.00324553s) [upload=7656]: Done starting upload Aug 7 23:54:31.416235946 (7h27m45.003292157s) [upload=7656]: Uploading segments... Aug 7 23:54:31.416277384 (7h27m45.003333595s) [upload=7656]: Got next segment. Inline: true Aug 7 23:54:31.416297601 (7h27m45.003353813s) [upload=7656]: Waiting for error group managing segments... Aug 7 23:54:31.416336454 (7h27m45.003392665s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:31.416340652 (7h27m45.003396873s) [upload=7656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:31.416342965 (7h27m45.003399177s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:01.405081824 (7h28m14.992138035s) [upload=7656]: Done uploading segments... Aug 7 23:55:01.405101451 (7h28m14.992157672s) [upload=7656]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:45.764200449 (7h28m59.35125669s) [upload=7673]: Starting upload Aug 7 23:55:45.764300708 (7h28m59.351356948s) [upload=7673]: Done starting upload Aug 7 23:55:45.764398370 (7h28m59.351454611s) [upload=7673]: Uploading segments... Aug 7 23:55:45.764515538 (7h28m59.35157178s) [upload=7673]: Got next segment. Inline: true Aug 7 23:55:45.764521980 (7h28m59.351578222s) [upload=7673]: Waiting for error group managing segments... Aug 7 23:55:45.764570060 (7h28m59.3516263s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:45.764575871 (7h28m59.351632111s) [upload=7673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:45.764579898 (7h28m59.351636149s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:15.747212127 (7h29m29.334268369s) [upload=7673]: Done uploading segments... Aug 7 23:56:15.747307315 (7h29m29.334363566s) [upload=7673]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:15.823083783 (7h29m29.410140035s) [upload=7682]: Starting upload Aug 7 23:56:15.823169003 (7h29m29.410225243s) [upload=7682]: Done starting upload Aug 7 23:56:15.823245125 (7h29m29.410301367s) [upload=7682]: Uploading segments... Aug 7 23:56:15.823289758 (7h29m29.410346009s) [upload=7682]: Got next segment. Inline: true Aug 7 23:56:15.823295359 (7h29m29.410351599s) [upload=7682]: Waiting for error group managing segments... Aug 7 23:56:15.823343018 (7h29m29.410399269s) [upload=7682]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:15.823348678 (7h29m29.41040492s) [upload=7682]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:15.823352376 (7h29m29.410408616s) [upload=7682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:45.810149218 (7h29m59.397205439s) [upload=7682]: Done uploading segments... Aug 7 23:56:45.810167081 (7h29m59.397223322s) [upload=7682]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:58.169907263 (7h30m11.756963514s) [upload=7692]: Starting upload Aug 7 23:56:58.169972405 (7h30m11.757028647s) [upload=7692]: Done starting upload Aug 7 23:56:58.170060319 (7h30m11.75711657s) [upload=7692]: Uploading segments... Aug 7 23:56:58.170104551 (7h30m11.757160792s) [upload=7692]: Got next segment. Inline: true Aug 7 23:56:58.170110071 (7h30m11.757166313s) [upload=7692]: Waiting for error group managing segments... Aug 7 23:56:58.170155567 (7h30m11.757211808s) [upload=7692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.170161878 (7h30m11.75721812s) [upload=7692]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.170165555 (7h30m11.757221807s) [upload=7692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:28.152393221 (7h30m41.739449472s) [upload=7692]: Done uploading segments... Aug 7 23:57:28.152443946 (7h30m41.739500187s) [upload=7692]: Upload finished. err: metaclient: context deadline exceeded Duration: 184.903867ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 138.831835ms FAIL (2025-08-07 23:58:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 138.860098ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X....XXX.X......................................XXXXX.] duration: 1.087498435s FAIL (2025-08-07 23:58:48 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522017724 (7h27m3.109073975s) [noise=true , address=45.91.116.12:4226 , piece_id=YELH2JJ2TFX6A2FUHKFGZFH3RKFQCDUBTA4OBW6B6BSWPJ4FTQSQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522360804 (7h27m3.109417046s) [noise=true , address=45.91.116.14:4190 , piece_id=PPYDTYVAD7ESR52JJNH64JB22XD5B3WTXTZ7X72X6KCG2FVK5MXA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522463095 (7h27m3.109519336s) [noise=true , address=104.36.82.11:4113 , piece_id=N5ABMUNW7Q2MPYAXORNWQMJYIGMHHLYKHZSM7FD265TVN7QPBILQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522783393 (7h27m3.109839644s) [noise=true , address=45.91.116.5:4297 , piece_id=WYHPPXNXBWTSD4LO2RWXPUN37P7JULRPJSIA3G5KNBYBXXBUDEXQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522884061 (7h27m3.109940302s) [noise=true , address=104.36.82.6:4874 , piece_id=FZBS75TSUST27CMCEC5UZAAC2ZUNZ5F3PX5SIFHLN72GALS4XNTA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523171748 (7h27m3.110228s) [noise=true , address=104.36.82.9:4006 , piece_id=6RYH7RTUR45VDJQKTMW6QXWWPPC2FRU2PGECQWGIKUAQOG4DCVEA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523292534 (7h27m3.110348775s) [noise=true , address=45.91.116.8:4048 , piece_id=JS3BGIHYWKZYEAXNUCCIYHEDRA2YQMBM2OBPJOUWIAWNEBO6BYFQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523596000 (7h27m3.110652251s) [noise=true , address=77.90.31.17:4770 , piece_id=QQOG5XJMG2CHWVCGAYWMQMRAHGEKQ5AITZOOOOOVUEFWDFHGBSEA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523713870 (7h27m3.110770111s) [noise=true , address=45.91.116.8:4047 , piece_id=SJVTTSCB7BDZRLEJX4WXD4PLXBALDVO2GWIQ5RUI6GAOVMNCERLQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.524251423 (7h27m3.111307674s) [noise=true , address=45.91.116.6:4974 , piece_id=RHPHBCUSDUVI23GGMG2WTVNLMQAAK436N3FXYHPBJ6FV7WOMSFMA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.524718816 (7h27m3.111775057s) [noise=true , address=45.91.116.11:4117 , piece_id=H4RP6BXPN4GRE2TMY5GZEOZ5UKKVDOEAGAS72ZRBIOURV7OBB5MQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.647138359 (7h27m3.2341946s) [upload=7640]: Next returned nil segment Aug 7 23:53:49.647155511 (7h27m3.234211752s) [upload=7640]: Waiting for error group managing segments... Aug 7 23:53:49.655170752 (7h27m3.242226993s) [noise=true , address=45.91.116.14:4190 , piece_id=PPYDTYVAD7ESR52JJNH64JB22XD5B3WTXTZ7X72X6KCG2FVK5MXA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.655792202 (7h27m3.242848433s) [noise=true , address=45.91.116.5:4297 , piece_id=WYHPPXNXBWTSD4LO2RWXPUN37P7JULRPJSIA3G5KNBYBXXBUDEXQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.656163956 (7h27m3.243220187s) [noise=true , address=45.91.116.8:4048 , piece_id=JS3BGIHYWKZYEAXNUCCIYHEDRA2YQMBM2OBPJOUWIAWNEBO6BYFQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.656798870 (7h27m3.243855091s) [noise=true , address=45.91.116.12:4226 , piece_id=YELH2JJ2TFX6A2FUHKFGZFH3RKFQCDUBTA4OBW6B6BSWPJ4FTQSQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.657996216 (7h27m3.245052458s) [noise=true , address=45.91.116.4:4939 , piece_id=LWLSQN6BSOJY4PA7I7HZL5PSRI4DNR6E4GMYWPUB6DF6BLMPU26A, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.658644787 (7h27m3.245701017s) [noise=true , address=45.91.116.11:4117 , piece_id=H4RP6BXPN4GRE2TMY5GZEOZ5UKKVDOEAGAS72ZRBIOURV7OBB5MQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.659153586 (7h27m3.246209817s) [noise=true , address=77.90.31.17:4770 , piece_id=QQOG5XJMG2CHWVCGAYWMQMRAHGEKQ5AITZOOOOOVUEFWDFHGBSEA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.660604835 (7h27m3.247661076s) [noise=true , address=45.91.116.8:4047 , piece_id=SJVTTSCB7BDZRLEJX4WXD4PLXBALDVO2GWIQ5RUI6GAOVMNCERLQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.660941473 (7h27m3.247997694s) [noise=true , address=45.91.116.6:4974 , piece_id=RHPHBCUSDUVI23GGMG2WTVNLMQAAK436N3FXYHPBJ6FV7WOMSFMA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.661457717 (7h27m3.248513959s) [noise=true , address=77.90.31.18:4807 , piece_id=5CCUKS55ROKTHMZA7NKUNYXOTAKYZIDQGDJBSMFM5TSBEFVDJMPA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.662043810 (7h27m3.249100041s) [noise=true , address=104.36.82.6:4874 , piece_id=FZBS75TSUST27CMCEC5UZAAC2ZUNZ5F3PX5SIFHLN72GALS4XNTA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665007553 (7h27m3.252063794s) [noise=true , address=104.36.82.11:4113 , piece_id=N5ABMUNW7Q2MPYAXORNWQMJYIGMHHLYKHZSM7FD265TVN7QPBILQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665952497 (7h27m3.253008727s) [noise=true , address=77.90.31.16:4734 , piece_id=AIAJVZAYQTA4OR6WDJ6V2XH3TXBO6FA4VRUCYJGIFXXUO3ZGIT4A, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665980449 (7h27m3.25303667s) [seg_pos={0 0}, upload=7640]: Segment reached optimal threshold of 13 pieces. Aug 7 23:53:49.666316626 (7h27m3.253372857s) [noise=true , address=104.36.82.14:4221 , piece_id=JKDWKECRRZCOLROOZ5QXNOGIRN3LLLAUGDQRFBPDYBHBV6LTPH7Q, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: rpc: dial tcp 104.36.82.14:4221: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666330082 (7h27m3.253386322s) [noise=true , address=104.36.82.9:4006 , piece_id=6RYH7RTUR45VDJQKTMW6QXWWPPC2FRU2PGECQWGIKUAQOG4DCVEA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: read tcp 5.161.89.252:14750->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666492835 (7h27m3.253549076s) [noise=true , address=104.36.82.16:4294 , piece_id=PL7A2YIDU7ZU2F5J4IE37IFPYADS2VNPLY443J46RIBG7J6ZNZJA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 5.161.89.252:34080->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666519565 (7h27m3.253575806s) [seg_pos={0 0}, upload=7640]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:53:49.666614813 (7h27m3.253671053s) [upload=7640]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:53:49.666620242 (7h27m3.253676484s) [upload=7640]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:17.275326827 (7h30m30.862383048s) [upload=7640]: Done uploading segments... Aug 7 23:57:17.275353585 (7h30m30.862409807s) [upload=7640]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:17.349138524 (7h30m30.936194735s) [upload=7698]: Starting upload Aug 7 23:57:17.349187294 (7h30m30.936243515s) [upload=7698]: Done starting upload Aug 7 23:57:17.349221879 (7h30m30.93627809s) [upload=7698]: Uploading segments... Aug 7 23:57:17.349404010 (7h30m30.93646022s) [upload=7698]: Got next segment. Inline: false Aug 7 23:57:17.349444765 (7h30m30.936500976s) [upload=7698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:17.349448802 (7h30m30.936505023s) [upload=7698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:47.334809460 (7h31m0.921865702s) [upload=7698]: Next segment err: context deadline exceeded Aug 7 23:57:47.334985600 (7h31m0.922041831s) [upload=7698]: Done uploading segments... Aug 7 23:57:47.335006539 (7h31m0.922062781s) [upload=7698]: Upload finished. err: context deadline exceeded Aug 7 23:57:47.412546828 (7h31m0.99960307s) [upload=7705]: Starting upload Aug 7 23:57:47.412626116 (7h31m0.999682357s) [upload=7705]: Done starting upload Aug 7 23:57:47.412754536 (7h31m0.999810777s) [upload=7705]: Uploading segments... Aug 7 23:57:47.413018508 (7h31m1.000074749s) [upload=7705]: Got next segment. Inline: false Aug 7 23:57:47.413067800 (7h31m1.000124042s) [upload=7705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:47.413073351 (7h31m1.000129592s) [upload=7705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:17.397856277 (7h31m30.984912519s) [upload=7705]: Next segment err: context deadline exceeded Aug 7 23:58:17.397964409 (7h31m30.98502065s) [upload=7705]: Done uploading segments... Aug 7 23:58:17.397983265 (7h31m30.985039505s) [upload=7705]: Upload finished. err: context deadline exceeded Aug 7 23:58:17.497021948 (7h31m31.084078188s) [upload=7712]: Starting upload Aug 7 23:58:17.497166287 (7h31m31.084222528s) [upload=7712]: Done starting upload Aug 7 23:58:17.497376509 (7h31m31.084432751s) [upload=7712]: Uploading segments... Aug 7 23:58:17.497668814 (7h31m31.084725066s) [upload=7712]: Got next segment. Inline: false Aug 7 23:58:17.497771346 (7h31m31.084827597s) [upload=7712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:17.497783308 (7h31m31.08483955s) [upload=7712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:47.474157820 (7h32m1.061214041s) [upload=7712]: Next segment err: context deadline exceeded Aug 7 23:58:47.474221107 (7h32m1.061277319s) [upload=7712]: Done uploading segments... Aug 7 23:58:47.474236366 (7h32m1.061292587s) [upload=7712]: Upload finished. err: context deadline exceeded Duration: 1.08752203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X....XXX.X......................................XXXXX] duration: 30.000539083s FAIL (2025-08-07 23:58:47 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522017724 (7h27m3.109073975s) [noise=true , address=45.91.116.12:4226 , piece_id=YELH2JJ2TFX6A2FUHKFGZFH3RKFQCDUBTA4OBW6B6BSWPJ4FTQSQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522360804 (7h27m3.109417046s) [noise=true , address=45.91.116.14:4190 , piece_id=PPYDTYVAD7ESR52JJNH64JB22XD5B3WTXTZ7X72X6KCG2FVK5MXA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522463095 (7h27m3.109519336s) [noise=true , address=104.36.82.11:4113 , piece_id=N5ABMUNW7Q2MPYAXORNWQMJYIGMHHLYKHZSM7FD265TVN7QPBILQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522783393 (7h27m3.109839644s) [noise=true , address=45.91.116.5:4297 , piece_id=WYHPPXNXBWTSD4LO2RWXPUN37P7JULRPJSIA3G5KNBYBXXBUDEXQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522884061 (7h27m3.109940302s) [noise=true , address=104.36.82.6:4874 , piece_id=FZBS75TSUST27CMCEC5UZAAC2ZUNZ5F3PX5SIFHLN72GALS4XNTA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523171748 (7h27m3.110228s) [noise=true , address=104.36.82.9:4006 , piece_id=6RYH7RTUR45VDJQKTMW6QXWWPPC2FRU2PGECQWGIKUAQOG4DCVEA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523292534 (7h27m3.110348775s) [noise=true , address=45.91.116.8:4048 , piece_id=JS3BGIHYWKZYEAXNUCCIYHEDRA2YQMBM2OBPJOUWIAWNEBO6BYFQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523596000 (7h27m3.110652251s) [noise=true , address=77.90.31.17:4770 , piece_id=QQOG5XJMG2CHWVCGAYWMQMRAHGEKQ5AITZOOOOOVUEFWDFHGBSEA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523713870 (7h27m3.110770111s) [noise=true , address=45.91.116.8:4047 , piece_id=SJVTTSCB7BDZRLEJX4WXD4PLXBALDVO2GWIQ5RUI6GAOVMNCERLQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.524251423 (7h27m3.111307674s) [noise=true , address=45.91.116.6:4974 , piece_id=RHPHBCUSDUVI23GGMG2WTVNLMQAAK436N3FXYHPBJ6FV7WOMSFMA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.524718816 (7h27m3.111775057s) [noise=true , address=45.91.116.11:4117 , piece_id=H4RP6BXPN4GRE2TMY5GZEOZ5UKKVDOEAGAS72ZRBIOURV7OBB5MQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.647138359 (7h27m3.2341946s) [upload=7640]: Next returned nil segment Aug 7 23:53:49.647155511 (7h27m3.234211752s) [upload=7640]: Waiting for error group managing segments... Aug 7 23:53:49.655170752 (7h27m3.242226993s) [noise=true , address=45.91.116.14:4190 , piece_id=PPYDTYVAD7ESR52JJNH64JB22XD5B3WTXTZ7X72X6KCG2FVK5MXA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.655792202 (7h27m3.242848433s) [noise=true , address=45.91.116.5:4297 , piece_id=WYHPPXNXBWTSD4LO2RWXPUN37P7JULRPJSIA3G5KNBYBXXBUDEXQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.656163956 (7h27m3.243220187s) [noise=true , address=45.91.116.8:4048 , piece_id=JS3BGIHYWKZYEAXNUCCIYHEDRA2YQMBM2OBPJOUWIAWNEBO6BYFQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.656798870 (7h27m3.243855091s) [noise=true , address=45.91.116.12:4226 , piece_id=YELH2JJ2TFX6A2FUHKFGZFH3RKFQCDUBTA4OBW6B6BSWPJ4FTQSQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.657996216 (7h27m3.245052458s) [noise=true , address=45.91.116.4:4939 , piece_id=LWLSQN6BSOJY4PA7I7HZL5PSRI4DNR6E4GMYWPUB6DF6BLMPU26A, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.658644787 (7h27m3.245701017s) [noise=true , address=45.91.116.11:4117 , piece_id=H4RP6BXPN4GRE2TMY5GZEOZ5UKKVDOEAGAS72ZRBIOURV7OBB5MQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.659153586 (7h27m3.246209817s) [noise=true , address=77.90.31.17:4770 , piece_id=QQOG5XJMG2CHWVCGAYWMQMRAHGEKQ5AITZOOOOOVUEFWDFHGBSEA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.660604835 (7h27m3.247661076s) [noise=true , address=45.91.116.8:4047 , piece_id=SJVTTSCB7BDZRLEJX4WXD4PLXBALDVO2GWIQ5RUI6GAOVMNCERLQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.660941473 (7h27m3.247997694s) [noise=true , address=45.91.116.6:4974 , piece_id=RHPHBCUSDUVI23GGMG2WTVNLMQAAK436N3FXYHPBJ6FV7WOMSFMA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.661457717 (7h27m3.248513959s) [noise=true , address=77.90.31.18:4807 , piece_id=5CCUKS55ROKTHMZA7NKUNYXOTAKYZIDQGDJBSMFM5TSBEFVDJMPA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.662043810 (7h27m3.249100041s) [noise=true , address=104.36.82.6:4874 , piece_id=FZBS75TSUST27CMCEC5UZAAC2ZUNZ5F3PX5SIFHLN72GALS4XNTA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665007553 (7h27m3.252063794s) [noise=true , address=104.36.82.11:4113 , piece_id=N5ABMUNW7Q2MPYAXORNWQMJYIGMHHLYKHZSM7FD265TVN7QPBILQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665952497 (7h27m3.253008727s) [noise=true , address=77.90.31.16:4734 , piece_id=AIAJVZAYQTA4OR6WDJ6V2XH3TXBO6FA4VRUCYJGIFXXUO3ZGIT4A, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665980449 (7h27m3.25303667s) [seg_pos={0 0}, upload=7640]: Segment reached optimal threshold of 13 pieces. Aug 7 23:53:49.666316626 (7h27m3.253372857s) [noise=true , address=104.36.82.14:4221 , piece_id=JKDWKECRRZCOLROOZ5QXNOGIRN3LLLAUGDQRFBPDYBHBV6LTPH7Q, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: rpc: dial tcp 104.36.82.14:4221: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666330082 (7h27m3.253386322s) [noise=true , address=104.36.82.9:4006 , piece_id=6RYH7RTUR45VDJQKTMW6QXWWPPC2FRU2PGECQWGIKUAQOG4DCVEA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: read tcp 5.161.89.252:14750->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666492835 (7h27m3.253549076s) [noise=true , address=104.36.82.16:4294 , piece_id=PL7A2YIDU7ZU2F5J4IE37IFPYADS2VNPLY443J46RIBG7J6ZNZJA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 5.161.89.252:34080->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666519565 (7h27m3.253575806s) [seg_pos={0 0}, upload=7640]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:53:49.666614813 (7h27m3.253671053s) [upload=7640]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:53:49.666620242 (7h27m3.253676484s) [upload=7640]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:17.275326827 (7h30m30.862383048s) [upload=7640]: Done uploading segments... Aug 7 23:57:17.275353585 (7h30m30.862409807s) [upload=7640]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:17.349138524 (7h30m30.936194735s) [upload=7698]: Starting upload Aug 7 23:57:17.349187294 (7h30m30.936243515s) [upload=7698]: Done starting upload Aug 7 23:57:17.349221879 (7h30m30.93627809s) [upload=7698]: Uploading segments... Aug 7 23:57:17.349404010 (7h30m30.93646022s) [upload=7698]: Got next segment. Inline: false Aug 7 23:57:17.349444765 (7h30m30.936500976s) [upload=7698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:17.349448802 (7h30m30.936505023s) [upload=7698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:47.334809460 (7h31m0.921865702s) [upload=7698]: Next segment err: context deadline exceeded Aug 7 23:57:47.334985600 (7h31m0.922041831s) [upload=7698]: Done uploading segments... Aug 7 23:57:47.335006539 (7h31m0.922062781s) [upload=7698]: Upload finished. err: context deadline exceeded Aug 7 23:57:47.412546828 (7h31m0.99960307s) [upload=7705]: Starting upload Aug 7 23:57:47.412626116 (7h31m0.999682357s) [upload=7705]: Done starting upload Aug 7 23:57:47.412754536 (7h31m0.999810777s) [upload=7705]: Uploading segments... Aug 7 23:57:47.413018508 (7h31m1.000074749s) [upload=7705]: Got next segment. Inline: false Aug 7 23:57:47.413067800 (7h31m1.000124042s) [upload=7705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:47.413073351 (7h31m1.000129592s) [upload=7705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:17.397856277 (7h31m30.984912519s) [upload=7705]: Next segment err: context deadline exceeded Aug 7 23:58:17.397964409 (7h31m30.98502065s) [upload=7705]: Done uploading segments... Aug 7 23:58:17.397983265 (7h31m30.985039505s) [upload=7705]: Upload finished. err: context deadline exceeded Aug 7 23:58:17.497021948 (7h31m31.084078188s) [upload=7712]: Starting upload Aug 7 23:58:17.497166287 (7h31m31.084222528s) [upload=7712]: Done starting upload Aug 7 23:58:17.497376509 (7h31m31.084432751s) [upload=7712]: Uploading segments... Aug 7 23:58:17.497668814 (7h31m31.084725066s) [upload=7712]: Got next segment. Inline: false Aug 7 23:58:17.497771346 (7h31m31.084827597s) [upload=7712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:17.497783308 (7h31m31.08483955s) [upload=7712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:47.474157820 (7h32m1.061214041s) [upload=7712]: Next segment err: context deadline exceeded Aug 7 23:58:47.474221107 (7h32m1.061277319s) [upload=7712]: Done uploading segments... Aug 7 23:58:47.474236366 (7h32m1.061292587s) [upload=7712]: Upload finished. err: context deadline exceeded Duration: 30.000553791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [XXXX.X...............................................XX.X...] duration: 344.850454ms FAIL (2025-08-07 23:58:45 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:39.988158721 (7h20m53.575214932s) [upload=7536]: Done uploading segments... Aug 7 23:47:39.988176785 (7h20m53.575232995s) [upload=7536]: Upload finished. err: <nil> Aug 7 23:48:17.585904864 (7h21m31.172961076s) [upload=7548]: Starting upload Aug 7 23:48:17.585990855 (7h21m31.173047066s) [upload=7548]: Done starting upload Aug 7 23:48:17.586065835 (7h21m31.173122035s) [upload=7548]: Uploading segments... Aug 7 23:48:17.586102964 (7h21m31.173159205s) [upload=7548]: Got next segment. Inline: true Aug 7 23:48:17.586107262 (7h21m31.173163473s) [upload=7548]: Waiting for error group managing segments... Aug 7 23:48:17.586144451 (7h21m31.173200662s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:17.586148969 (7h21m31.17320518s) [upload=7548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:17.586151854 (7h21m31.173208066s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:17.642355873 (7h21m31.229412094s) [upload=7548]: Done uploading segments... Aug 7 23:48:17.642371582 (7h21m31.229427794s) [upload=7548]: Upload finished. err: <nil> Aug 7 23:49:19.929340015 (7h22m33.516396255s) [upload=7565]: Starting upload Aug 7 23:49:19.929497669 (7h22m33.51655391s) [upload=7565]: Done starting upload Aug 7 23:49:19.929593468 (7h22m33.516649709s) [upload=7565]: Uploading segments... Aug 7 23:49:19.929640947 (7h22m33.516697188s) [upload=7565]: Got next segment. Inline: true Aug 7 23:49:19.929647679 (7h22m33.516703931s) [upload=7565]: Waiting for error group managing segments... Aug 7 23:49:19.929695078 (7h22m33.516751329s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:19.929701059 (7h22m33.5167573s) [upload=7565]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:19.929704956 (7h22m33.516761197s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:19.992636681 (7h22m33.579692922s) [upload=7565]: Done uploading segments... Aug 7 23:49:19.992654385 (7h22m33.579710625s) [upload=7565]: Upload finished. err: <nil> Aug 7 23:50:07.176468149 (7h23m20.76352438s) [upload=7576]: Starting upload Aug 7 23:50:07.176537798 (7h23m20.763594019s) [upload=7576]: Done starting upload Aug 7 23:50:07.176592070 (7h23m20.763648311s) [upload=7576]: Uploading segments... Aug 7 23:50:07.176626473 (7h23m20.763682705s) [upload=7576]: Got next segment. Inline: true Aug 7 23:50:07.176631002 (7h23m20.763687213s) [upload=7576]: Waiting for error group managing segments... Aug 7 23:50:07.176664084 (7h23m20.763720295s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:07.176668132 (7h23m20.763724343s) [upload=7576]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:07.176670586 (7h23m20.763726807s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:07.240851653 (7h23m20.827907864s) [upload=7576]: Done uploading segments... Aug 7 23:50:07.240866771 (7h23m20.827922992s) [upload=7576]: Upload finished. err: <nil> Aug 7 23:50:45.893754262 (7h23m59.480810504s) [upload=7591]: Starting upload Aug 7 23:50:45.893825847 (7h23m59.480882088s) [upload=7591]: Done starting upload Aug 7 23:50:45.893912749 (7h23m59.48096899s) [upload=7591]: Uploading segments... Aug 7 23:50:45.893955949 (7h23m59.4810122s) [upload=7591]: Got next segment. Inline: true Aug 7 23:50:45.893961970 (7h23m59.481018211s) [upload=7591]: Waiting for error group managing segments... Aug 7 23:50:45.894005482 (7h23m59.481061723s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:45.894011062 (7h23m59.481067304s) [upload=7591]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:45.894014729 (7h23m59.48107097s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:05.079621048 (7h24m18.666677259s) [upload=7591]: Done uploading segments... Aug 7 23:51:05.079636297 (7h24m18.666692508s) [upload=7591]: Upload finished. err: <nil> Aug 7 23:51:21.725622238 (7h24m35.312678469s) [upload=7603]: Starting upload Aug 7 23:51:21.725696627 (7h24m35.312752838s) [upload=7603]: Done starting upload Aug 7 23:51:21.725758231 (7h24m35.312814443s) [upload=7603]: Uploading segments... Aug 7 23:51:21.725796774 (7h24m35.312852985s) [upload=7603]: Got next segment. Inline: true Aug 7 23:51:21.725802845 (7h24m35.312859056s) [upload=7603]: Waiting for error group managing segments... Aug 7 23:51:21.725848871 (7h24m35.312905082s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:21.725855313 (7h24m35.312911523s) [upload=7603]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:21.725859100 (7h24m35.312915351s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:21.784505948 (7h24m35.371562158s) [upload=7603]: Done uploading segments... Aug 7 23:51:21.784521767 (7h24m35.371577978s) [upload=7603]: Upload finished. err: <nil> Aug 7 23:52:10.285208176 (7h25m23.872264397s) [upload=7615]: Starting upload Aug 7 23:52:10.285263228 (7h25m23.872319449s) [upload=7615]: Done starting upload Aug 7 23:52:10.285310727 (7h25m23.872366937s) [upload=7615]: Uploading segments... Aug 7 23:52:10.285339711 (7h25m23.872395932s) [upload=7615]: Got next segment. Inline: true Aug 7 23:52:10.285343438 (7h25m23.872399649s) [upload=7615]: Waiting for error group managing segments... Aug 7 23:52:10.285391158 (7h25m23.872447379s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:10.285395065 (7h25m23.872451276s) [upload=7615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:10.285397770 (7h25m23.872453991s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:10.400888570 (7h25m23.987944781s) [upload=7615]: Done uploading segments... Aug 7 23:52:10.400904530 (7h25m23.987960741s) [upload=7615]: Upload finished. err: <nil> Aug 7 23:53:11.795016020 (7h26m25.382072262s) [upload=7635]: Starting upload Aug 7 23:53:11.795108102 (7h26m25.382164354s) [upload=7635]: Done starting upload Aug 7 23:53:11.795175588 (7h26m25.382231829s) [upload=7635]: Uploading segments... Aug 7 23:53:11.795254645 (7h26m25.382310886s) [upload=7635]: Got next segment. Inline: true Aug 7 23:53:11.795262130 (7h26m25.38231837s) [upload=7635]: Waiting for error group managing segments... Aug 7 23:53:11.795319296 (7h26m25.382375538s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:11.795326170 (7h26m25.38238242s) [upload=7635]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:11.795329857 (7h26m25.382386097s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:11.782974743 (7h27m25.370030954s) [upload=7635]: Done uploading segments... Aug 7 23:54:11.782991474 (7h27m25.370047695s) [upload=7635]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:11.855031501 (7h27m25.442087722s) [upload=7650]: Starting upload Aug 7 23:54:11.855087627 (7h27m25.442143838s) [upload=7650]: Done starting upload Aug 7 23:54:11.855131608 (7h27m25.442187829s) [upload=7650]: Uploading segments... Aug 7 23:54:11.855163227 (7h27m25.442219448s) [upload=7650]: Got next segment. Inline: true Aug 7 23:54:11.855167005 (7h27m25.442223226s) [upload=7650]: Waiting for error group managing segments... Aug 7 23:54:11.855198905 (7h27m25.442255126s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:11.855202661 (7h27m25.442258882s) [upload=7650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:11.855205036 (7h27m25.442261247s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:11.844108363 (7h28m25.431164634s) [upload=7650]: Done uploading segments... Aug 7 23:55:11.844129542 (7h28m25.431185763s) [upload=7650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:11.924363315 (7h28m25.511419527s) [upload=7668]: Starting upload Aug 7 23:55:11.924429198 (7h28m25.511485409s) [upload=7668]: Done starting upload Aug 7 23:55:11.924484862 (7h28m25.511541083s) [upload=7668]: Uploading segments... Aug 7 23:55:11.924519887 (7h28m25.511576098s) [upload=7668]: Got next segment. Inline: true Aug 7 23:55:11.924523564 (7h28m25.511579785s) [upload=7668]: Waiting for error group managing segments... Aug 7 23:55:11.924554131 (7h28m25.511610342s) [upload=7668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:11.924558279 (7h28m25.51161449s) [upload=7668]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:11.924560643 (7h28m25.511616855s) [upload=7668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:12.020746910 (7h28m25.607803121s) [upload=7668]: Done uploading segments... Aug 7 23:55:12.020765404 (7h28m25.607821625s) [upload=7668]: Upload finished. err: <nil> Aug 7 23:55:57.071847012 (7h29m10.658903254s) [upload=7678]: Starting upload Aug 7 23:55:57.071985611 (7h29m10.659041851s) [upload=7678]: Done starting upload Aug 7 23:55:57.072082963 (7h29m10.659139204s) [upload=7678]: Uploading segments... Aug 7 23:55:57.072231269 (7h29m10.65928751s) [upload=7678]: Got next segment. Inline: true Aug 7 23:55:57.072240506 (7h29m10.659296748s) [upload=7678]: Waiting for error group managing segments... Aug 7 23:55:57.072359458 (7h29m10.6594157s) [upload=7678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:57.072367143 (7h29m10.659423384s) [upload=7678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:57.072371320 (7h29m10.659427562s) [upload=7678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:57.049680168 (7h30m10.636736379s) [upload=7678]: Done uploading segments... Aug 7 23:56:57.049696499 (7h30m10.636752709s) [upload=7678]: Upload finished. err: metaclient: context deadline exceeded Duration:* 344.873787ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 13 out of 60 is failed (threshold=3) [XX.X..X..X......................................XXX.X.XXX.X.] duration: 186.953985ms FAIL (2025-08-07 23:58:43 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 186.982438ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 13 out of 60 is failed (threshold=3) [XX.XX............................................XXXXXXXXX..] duration: 175.096172ms FAIL (2025-08-07 23:58:34 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 (7h23m15.650779768s) [upload=7574]: Upload finished. err: <nil> Aug 7 23:50:41.160166556 (7h23m54.747222777s) [upload=7588]: Starting upload Aug 7 23:50:41.160220555 (7h23m54.747276777s) [upload=7588]: Done starting upload Aug 7 23:50:41.160264659 (7h23m54.74732088s) [upload=7588]: Uploading segments... Aug 7 23:50:41.160295956 (7h23m54.747352177s) [upload=7588]: Got next segment. Inline: true Aug 7 23:50:41.160299753 (7h23m54.747355964s) [upload=7588]: Waiting for error group managing segments... Aug 7 23:50:41.160331633 (7h23m54.747387844s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:41.160335861 (7h23m54.747392072s) [upload=7588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:41.160338396 (7h23m54.747394617s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:41.222859735 (7h23m54.809915976s) [upload=7588]: Done uploading segments... Aug 7 23:50:41.222879913 (7h23m54.809936154s) [upload=7588]: Upload finished. err: <nil> Aug 7 23:51:17.131690986 (7h24m30.718747197s) [upload=7599]: Starting upload Aug 7 23:51:17.131738926 (7h24m30.718795147s) [upload=7599]: Done starting upload Aug 7 23:51:17.131792947 (7h24m30.718849178s) [upload=7599]: Uploading segments... Aug 7 23:51:17.131835896 (7h24m30.718892117s) [upload=7599]: Got next segment. Inline: true Aug 7 23:51:17.131842027 (7h24m30.718898239s) [upload=7599]: Waiting for error group managing segments... Aug 7 23:51:17.131883385 (7h24m30.718939627s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:17.131889146 (7h24m30.718945388s) [upload=7599]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:17.132006205 (7h24m30.719062435s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:38.787433326 (7h24m52.374489567s) [upload=7599]: Done uploading segments... Aug 7 23:51:38.787452212 (7h24m52.374508453s) [upload=7599]: Upload finished. err: <nil> Aug 7 23:52:00.256204647 (7h25m13.843260858s) [upload=7610]: Starting upload Aug 7 23:52:00.256263547 (7h25m13.843319758s) [upload=7610]: Done starting upload Aug 7 23:52:00.256316195 (7h25m13.843372406s) [upload=7610]: Uploading segments... Aug 7 23:52:00.256345671 (7h25m13.843401881s) [upload=7610]: Got next segment. Inline: true Aug 7 23:52:00.256349758 (7h25m13.843405969s) [upload=7610]: Waiting for error group managing segments... Aug 7 23:52:00.256399992 (7h25m13.843456203s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:00.256404780 (7h25m13.843460991s) [upload=7610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:00.256407806 (7h25m13.843464017s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:30.237600164 (7h25m43.824656416s) [upload=7610]: Done uploading segments... Aug 7 23:52:30.237630862 (7h25m43.824687113s) [upload=7610]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:53.146251246 (7h26m6.733307456s) [upload=7626]: Starting upload Aug 7 23:52:53.146307339 (7h26m6.733363561s) [upload=7626]: Done starting upload Aug 7 23:52:53.146350962 (7h26m6.733407173s) [upload=7626]: Uploading segments... Aug 7 23:52:53.146411344 (7h26m6.733467565s) [upload=7626]: Got next segment. Inline: true Aug 7 23:52:53.146415271 (7h26m6.733471493s) [upload=7626]: Waiting for error group managing segments... Aug 7 23:52:53.146449966 (7h26m6.733506177s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:53.146454174 (7h26m6.733510395s) [upload=7626]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:53.146456709 (7h26m6.73351293s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.132536521 (7h26m36.719592742s) [upload=7626]: Done uploading segments... Aug 7 23:53:23.132560907 (7h26m36.719617128s) [upload=7626]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:28.217132518 (7h26m41.804188759s) [upload=7638]: Starting upload Aug 7 23:53:28.217216185 (7h26m41.804272426s) [upload=7638]: Done starting upload Aug 7 23:53:28.217288009 (7h26m41.80434426s) [upload=7638]: Uploading segments... Aug 7 23:53:28.217336129 (7h26m41.80439237s) [upload=7638]: Got next segment. Inline: true Aug 7 23:53:28.217341890 (7h26m41.804398131s) [upload=7638]: Waiting for error group managing segments... Aug 7 23:53:28.217421438 (7h26m41.804477679s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:28.217427650 (7h26m41.804483901s) [upload=7638]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:28.217431617 (7h26m41.804487859s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:58.204830989 (7h27m11.79188721s) [upload=7638]: Done uploading segments... Aug 7 23:53:58.204849294 (7h27m11.791905505s) [upload=7638]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:01.364416813 (7h27m14.951473054s) [upload=7648]: Starting upload Aug 7 23:54:01.364544481 (7h27m14.951600732s) [upload=7648]: Done starting upload Aug 7 23:54:01.364639518 (7h27m14.95169577s) [upload=7648]: Uploading segments... Aug 7 23:54:01.364703858 (7h27m14.951760109s) [upload=7648]: Got next segment. Inline: true Aug 7 23:54:01.364713517 (7h27m14.951769757s) [upload=7648]: Waiting for error group managing segments... Aug 7 23:54:01.364781374 (7h27m14.951837625s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:01.364791231 (7h27m14.951847472s) [upload=7648]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:01.364797502 (7h27m14.951853754s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:31.351186626 (7h27m44.938242847s) [upload=7648]: Done uploading segments... Aug 7 23:54:31.351241118 (7h27m44.938297339s) [upload=7648]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:31.416140839 (7h27m45.003197049s) [upload=7656]: Starting upload Aug 7 23:54:31.416189309 (7h27m45.00324553s) [upload=7656]: Done starting upload Aug 7 23:54:31.416235946 (7h27m45.003292157s) [upload=7656]: Uploading segments... Aug 7 23:54:31.416277384 (7h27m45.003333595s) [upload=7656]: Got next segment. Inline: true Aug 7 23:54:31.416297601 (7h27m45.003353813s) [upload=7656]: Waiting for error group managing segments... Aug 7 23:54:31.416336454 (7h27m45.003392665s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:31.416340652 (7h27m45.003396873s) [upload=7656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:31.416342965 (7h27m45.003399177s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:01.405081824 (7h28m14.992138035s) [upload=7656]: Done uploading segments... Aug 7 23:55:01.405101451 (7h28m14.992157672s) [upload=7656]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:45.764200449 (7h28m59.35125669s) [upload=7673]: Starting upload Aug 7 23:55:45.764300708 (7h28m59.351356948s) [upload=7673]: Done starting upload Aug 7 23:55:45.764398370 (7h28m59.351454611s) [upload=7673]: Uploading segments... Aug 7 23:55:45.764515538 (7h28m59.35157178s) [upload=7673]: Got next segment. Inline: true Aug 7 23:55:45.764521980 (7h28m59.351578222s) [upload=7673]: Waiting for error group managing segments... Aug 7 23:55:45.764570060 (7h28m59.3516263s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:45.764575871 (7h28m59.351632111s) [upload=7673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:45.764579898 (7h28m59.351636149s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:15.747212127 (7h29m29.334268369s) [upload=7673]: Done uploading segments... Aug 7 23:56:15.747307315 (7h29m29.334363566s) [upload=7673]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:15.823083783 (7h29m29.410140035s) [upload=7682]: Starting upload Aug 7 23:56:15.823169003 (7h29m29.410225243s) [upload=7682]: Done starting upload Aug 7 23:56:15.823245125 (7h29m29.410301367s) [upload=7682]: Uploading segments... Aug 7 23:56:15.823289758 (7h29m29.410346009s) [upload=7682]: Got next segment. Inline: true Aug 7 23:56:15.823295359 (7h29m29.410351599s) [upload=7682]: Waiting for error group managing segments... Aug 7 23:56:15.823343018 (7h29m29.410399269s) [upload=7682]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:15.823348678 (7h29m29.41040492s) [upload=7682]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:15.823352376 (7h29m29.410408616s) [upload=7682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:45.810149218 (7h29m59.397205439s) [upload=7682]: Done uploading segments... Aug 7 23:56:45.810167081 (7h29m59.397223322s) [upload=7682]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:58.169907263 (7h30m11.756963514s) [upload=7692]: Starting upload Aug 7 23:56:58.169972405 (7h30m11.757028647s) [upload=7692]: Done starting upload Aug 7 23:56:58.170060319 (7h30m11.75711657s) [upload=7692]: Uploading segments... Aug 7 23:56:58.170104551 (7h30m11.757160792s) [upload=7692]: Got next segment. Inline: true Aug 7 23:56:58.170110071 (7h30m11.757166313s) [upload=7692]: Waiting for error group managing segments... Aug 7 23:56:58.170155567 (7h30m11.757211808s) [upload=7692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.170161878 (7h30m11.75721812s) [upload=7692]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.170165555 (7h30m11.757221807s) [upload=7692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:28.152393221 (7h30m41.739449472s) [upload=7692]: Done uploading segments... Aug 7 23:57:28.152443946 (7h30m41.739500187s) [upload=7692]: Upload finished. err: metaclient: context deadline exceeded Duration: 175.117161ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 518.635931ms FAIL (2025-08-07 23:58:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== 1345s) [upload=7262]: Got next segment. Inline: false Aug 7 23:31:37.583906637 (7h4m51.170962878s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:31:37.583953414 (7h4m51.171009656s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:31:37.613963041 (7h4m51.201019262s) [seg_pos={0 0}, upload=7262]: Begin upload segment... Aug 7 23:31:37.614008966 (7h4m51.201065176s) [seg_pos={0 0}, upload=7262]: Done begin upload segment. Aug 7 23:31:37.614029103 (7h4m51.201085324s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614108311 (7h4m51.201164542s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614344612 (7h4m51.201400833s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614634754 (7h4m51.201690975s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614919926 (7h4m51.201976137s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614943150 (7h4m51.201999391s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615146719 (7h4m51.20220294s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615454163 (7h4m51.202510384s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615514677 (7h4m51.202570918s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615688160 (7h4m51.202744381s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615932236 (7h4m51.202988457s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616184166 (7h4m51.203240378s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616521716 (7h4m51.203577937s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616643614 (7h4m51.203699855s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621231117 (7h4m51.208287338s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621561163 (7h4m51.208617374s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621861323 (7h4m51.208917534s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.622156223 (7h4m51.209212444s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.726040884 (7h4m51.313097094s) [upload=7262]: Next returned nil segment Aug 7 23:31:37.726066772 (7h4m51.313122983s) [upload=7262]: Waiting for error group managing segments... Aug 7 23:31:37.736232517 (7h4m51.323288738s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.737656014 (7h4m51.324712245s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.739754932 (7h4m51.326811153s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.740628113 (7h4m51.327684344s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741712657 (7h4m51.328768867s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741742653 (7h4m51.328798883s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.748242274 (7h4m51.335298505s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.750434577 (7h4m51.337490798s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.758962364 (7h4m51.346018586s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.769693706 (7h4m51.356749917s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.800540734 (7h4m51.387596975s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.924012655 (7h4m51.511068866s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.059695219 (7h4m51.64675144s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102515917 (7h4m51.689572138s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102541606 (7h4m51.689597827s) [seg_pos={0 0}, upload=7262]: Segment reached optimal threshold of 14 pieces. Aug 7 23:31:38.102658574 (7h4m51.689714795s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF Aug 7 23:31:38.102748111 (7h4m51.689804332s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE): protocol: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection Aug 7 23:31:38.102832158 (7h4m51.689888399s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GcyWwKkfavsCsmtq55U72n6XeZKqQiGGYHY9D6UwST8sC14Ac): protocol: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection Aug 7 23:31:38.102967200 (7h4m51.690023421s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12syasUPsyYf4UstQ6cFAr6qamze3vCyAm9dnobk5e83w4bTz69): protocol: context canceled; context canceled Aug 7 23:31:38.102995132 (7h4m51.690051374s) [seg_pos={0 0}, upload=7262]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:31:38.103097553 (7h4m51.690153775s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:31:38.103102412 (7h4m51.690158633s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:31:38.203633212 (7h4m51.790689433s) [upload=7262]: Done uploading segments... Aug 7 23:31:38.203649102 (7h4m51.790705323s) [upload=7262]: Upload finished. err: <nil> Duration: 518.664114ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 14 out of 60 is failed (threshold=3) [XXX.X..X..X......................................XXX.X.XXX.X] duration: 30.001297801s FAIL (2025-08-07 23:58:27 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001317249s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [X.XXXXX..X..................................................] duration: 187.474195ms FAIL (2025-08-07 23:58:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 187.491958ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X....X.X..........................................X........] duration: 396.521888ms FAIL (2025-08-07 23:58:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 396.548879ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXX.X.X...........................................X.X..XXX.] duration: 1.813139509s FAIL (2025-08-07 23:58:18 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.813160659s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........X....XXX.X......................................XXXX] duration: 30.000914646s FAIL (2025-08-07 23:58:17 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== iece_id=LWLSQN6BSOJY4PA7I7HZL5PSRI4DNR6E4GMYWPUB6DF6BLMPU26A, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.520814889 (7h27m3.10787119s) [noise=true , address=77.90.31.18:4807 , piece_id=5CCUKS55ROKTHMZA7NKUNYXOTAKYZIDQGDJBSMFM5TSBEFVDJMPA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.521449253 (7h27m3.108505494s) [noise=true , address=77.90.31.16:4734 , piece_id=AIAJVZAYQTA4OR6WDJ6V2XH3TXBO6FA4VRUCYJGIFXXUO3ZGIT4A, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.521498244 (7h27m3.108554486s) [noise=true , address=104.36.82.14:4221 , piece_id=JKDWKECRRZCOLROOZ5QXNOGIRN3LLLAUGDQRFBPDYBHBV6LTPH7Q, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.521948114 (7h27m3.109004366s) [noise=true , address=104.36.82.16:4294 , piece_id=PL7A2YIDU7ZU2F5J4IE37IFPYADS2VNPLY443J46RIBG7J6ZNZJA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522017724 (7h27m3.109073975s) [noise=true , address=45.91.116.12:4226 , piece_id=YELH2JJ2TFX6A2FUHKFGZFH3RKFQCDUBTA4OBW6B6BSWPJ4FTQSQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522360804 (7h27m3.109417046s) [noise=true , address=45.91.116.14:4190 , piece_id=PPYDTYVAD7ESR52JJNH64JB22XD5B3WTXTZ7X72X6KCG2FVK5MXA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522463095 (7h27m3.109519336s) [noise=true , address=104.36.82.11:4113 , piece_id=N5ABMUNW7Q2MPYAXORNWQMJYIGMHHLYKHZSM7FD265TVN7QPBILQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522783393 (7h27m3.109839644s) [noise=true , address=45.91.116.5:4297 , piece_id=WYHPPXNXBWTSD4LO2RWXPUN37P7JULRPJSIA3G5KNBYBXXBUDEXQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522884061 (7h27m3.109940302s) [noise=true , address=104.36.82.6:4874 , piece_id=FZBS75TSUST27CMCEC5UZAAC2ZUNZ5F3PX5SIFHLN72GALS4XNTA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523171748 (7h27m3.110228s) [noise=true , address=104.36.82.9:4006 , piece_id=6RYH7RTUR45VDJQKTMW6QXWWPPC2FRU2PGECQWGIKUAQOG4DCVEA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523292534 (7h27m3.110348775s) [noise=true , address=45.91.116.8:4048 , piece_id=JS3BGIHYWKZYEAXNUCCIYHEDRA2YQMBM2OBPJOUWIAWNEBO6BYFQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523596000 (7h27m3.110652251s) [noise=true , address=77.90.31.17:4770 , piece_id=QQOG5XJMG2CHWVCGAYWMQMRAHGEKQ5AITZOOOOOVUEFWDFHGBSEA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523713870 (7h27m3.110770111s) [noise=true , address=45.91.116.8:4047 , piece_id=SJVTTSCB7BDZRLEJX4WXD4PLXBALDVO2GWIQ5RUI6GAOVMNCERLQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.524251423 (7h27m3.111307674s) [noise=true , address=45.91.116.6:4974 , piece_id=RHPHBCUSDUVI23GGMG2WTVNLMQAAK436N3FXYHPBJ6FV7WOMSFMA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.524718816 (7h27m3.111775057s) [noise=true , address=45.91.116.11:4117 , piece_id=H4RP6BXPN4GRE2TMY5GZEOZ5UKKVDOEAGAS72ZRBIOURV7OBB5MQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.647138359 (7h27m3.2341946s) [upload=7640]: Next returned nil segment Aug 7 23:53:49.647155511 (7h27m3.234211752s) [upload=7640]: Waiting for error group managing segments... Aug 7 23:53:49.655170752 (7h27m3.242226993s) [noise=true , address=45.91.116.14:4190 , piece_id=PPYDTYVAD7ESR52JJNH64JB22XD5B3WTXTZ7X72X6KCG2FVK5MXA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.655792202 (7h27m3.242848433s) [noise=true , address=45.91.116.5:4297 , piece_id=WYHPPXNXBWTSD4LO2RWXPUN37P7JULRPJSIA3G5KNBYBXXBUDEXQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.656163956 (7h27m3.243220187s) [noise=true , address=45.91.116.8:4048 , piece_id=JS3BGIHYWKZYEAXNUCCIYHEDRA2YQMBM2OBPJOUWIAWNEBO6BYFQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.656798870 (7h27m3.243855091s) [noise=true , address=45.91.116.12:4226 , piece_id=YELH2JJ2TFX6A2FUHKFGZFH3RKFQCDUBTA4OBW6B6BSWPJ4FTQSQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.657996216 (7h27m3.245052458s) [noise=true , address=45.91.116.4:4939 , piece_id=LWLSQN6BSOJY4PA7I7HZL5PSRI4DNR6E4GMYWPUB6DF6BLMPU26A, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.658644787 (7h27m3.245701017s) [noise=true , address=45.91.116.11:4117 , piece_id=H4RP6BXPN4GRE2TMY5GZEOZ5UKKVDOEAGAS72ZRBIOURV7OBB5MQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.659153586 (7h27m3.246209817s) [noise=true , address=77.90.31.17:4770 , piece_id=QQOG5XJMG2CHWVCGAYWMQMRAHGEKQ5AITZOOOOOVUEFWDFHGBSEA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.660604835 (7h27m3.247661076s) [noise=true , address=45.91.116.8:4047 , piece_id=SJVTTSCB7BDZRLEJX4WXD4PLXBALDVO2GWIQ5RUI6GAOVMNCERLQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.660941473 (7h27m3.247997694s) [noise=true , address=45.91.116.6:4974 , piece_id=RHPHBCUSDUVI23GGMG2WTVNLMQAAK436N3FXYHPBJ6FV7WOMSFMA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.661457717 (7h27m3.248513959s) [noise=true , address=77.90.31.18:4807 , piece_id=5CCUKS55ROKTHMZA7NKUNYXOTAKYZIDQGDJBSMFM5TSBEFVDJMPA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.662043810 (7h27m3.249100041s) [noise=true , address=104.36.82.6:4874 , piece_id=FZBS75TSUST27CMCEC5UZAAC2ZUNZ5F3PX5SIFHLN72GALS4XNTA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665007553 (7h27m3.252063794s) [noise=true , address=104.36.82.11:4113 , piece_id=N5ABMUNW7Q2MPYAXORNWQMJYIGMHHLYKHZSM7FD265TVN7QPBILQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665952497 (7h27m3.253008727s) [noise=true , address=77.90.31.16:4734 , piece_id=AIAJVZAYQTA4OR6WDJ6V2XH3TXBO6FA4VRUCYJGIFXXUO3ZGIT4A, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665980449 (7h27m3.25303667s) [seg_pos={0 0}, upload=7640]: Segment reached optimal threshold of 13 pieces. Aug 7 23:53:49.666316626 (7h27m3.253372857s) [noise=true , address=104.36.82.14:4221 , piece_id=JKDWKECRRZCOLROOZ5QXNOGIRN3LLLAUGDQRFBPDYBHBV6LTPH7Q, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: rpc: dial tcp 104.36.82.14:4221: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666330082 (7h27m3.253386322s) [noise=true , address=104.36.82.9:4006 , piece_id=6RYH7RTUR45VDJQKTMW6QXWWPPC2FRU2PGECQWGIKUAQOG4DCVEA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: read tcp 5.161.89.252:14750->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666492835 (7h27m3.253549076s) [noise=true , address=104.36.82.16:4294 , piece_id=PL7A2YIDU7ZU2F5J4IE37IFPYADS2VNPLY443J46RIBG7J6ZNZJA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 5.161.89.252:34080->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666519565 (7h27m3.253575806s) [seg_pos={0 0}, upload=7640]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:53:49.666614813 (7h27m3.253671053s) [upload=7640]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:53:49.666620242 (7h27m3.253676484s) [upload=7640]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:17.275326827 (7h30m30.862383048s) [upload=7640]: Done uploading segments... Aug 7 23:57:17.275353585 (7h30m30.862409807s) [upload=7640]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:17.349138524 (7h30m30.936194735s) [upload=7698]: Starting upload Aug 7 23:57:17.349187294 (7h30m30.936243515s) [upload=7698]: Done starting upload Aug 7 23:57:17.349221879 (7h30m30.93627809s) [upload=7698]: Uploading segments... Aug 7 23:57:17.349404010 (7h30m30.93646022s) [upload=7698]: Got next segment. Inline: false Aug 7 23:57:17.349444765 (7h30m30.936500976s) [upload=7698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:17.349448802 (7h30m30.936505023s) [upload=7698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:47.334809460 (7h31m0.921865702s) [upload=7698]: Next segment err: context deadline exceeded Aug 7 23:57:47.334985600 (7h31m0.922041831s) [upload=7698]: Done uploading segments... Aug 7 23:57:47.335006539 (7h31m0.922062781s) [upload=7698]: Upload finished. err: context deadline exceeded Aug 7 23:57:47.412546828 (7h31m0.99960307s) [upload=7705]: Starting upload Aug 7 23:57:47.412626116 (7h31m0.999682357s) [upload=7705]: Done starting upload Aug 7 23:57:47.412754536 (7h31m0.999810777s) [upload=7705]: Uploading segments... Aug 7 23:57:47.413018508 (7h31m1.000074749s) [upload=7705]: Got next segment. Inline: false Aug 7 23:57:47.413067800 (7h31m1.000124042s) [upload=7705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:47.413073351 (7h31m1.000129592s) [upload=7705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:58:17.397856277 (7h31m30.984912519s) [upload=7705]: Next segment err: context deadline exceeded Aug 7 23:58:17.397964409 (7h31m30.98502065s) [upload=7705]: Done uploading segments... Aug 7 23:58:17.397983265 (7h31m30.985039505s) [upload=7705]: Upload finished. err: context deadline exceeded Duration: 30.000931206s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [..XXXXX..X............................................XXXXXX] duration: 1m0.000512902s FAIL (2025-08-07 23:58:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000525517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 14 out of 60 is failed (threshold=3) [XXX.XX............................................XXXXXXXXX.] duration: 194.876643ms FAIL (2025-08-07 23:58:13 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 (7h23m15.650779768s) [upload=7574]: Upload finished. err: <nil> Aug 7 23:50:41.160166556 (7h23m54.747222777s) [upload=7588]: Starting upload Aug 7 23:50:41.160220555 (7h23m54.747276777s) [upload=7588]: Done starting upload Aug 7 23:50:41.160264659 (7h23m54.74732088s) [upload=7588]: Uploading segments... Aug 7 23:50:41.160295956 (7h23m54.747352177s) [upload=7588]: Got next segment. Inline: true Aug 7 23:50:41.160299753 (7h23m54.747355964s) [upload=7588]: Waiting for error group managing segments... Aug 7 23:50:41.160331633 (7h23m54.747387844s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:41.160335861 (7h23m54.747392072s) [upload=7588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:41.160338396 (7h23m54.747394617s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:41.222859735 (7h23m54.809915976s) [upload=7588]: Done uploading segments... Aug 7 23:50:41.222879913 (7h23m54.809936154s) [upload=7588]: Upload finished. err: <nil> Aug 7 23:51:17.131690986 (7h24m30.718747197s) [upload=7599]: Starting upload Aug 7 23:51:17.131738926 (7h24m30.718795147s) [upload=7599]: Done starting upload Aug 7 23:51:17.131792947 (7h24m30.718849178s) [upload=7599]: Uploading segments... Aug 7 23:51:17.131835896 (7h24m30.718892117s) [upload=7599]: Got next segment. Inline: true Aug 7 23:51:17.131842027 (7h24m30.718898239s) [upload=7599]: Waiting for error group managing segments... Aug 7 23:51:17.131883385 (7h24m30.718939627s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:17.131889146 (7h24m30.718945388s) [upload=7599]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:17.132006205 (7h24m30.719062435s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:38.787433326 (7h24m52.374489567s) [upload=7599]: Done uploading segments... Aug 7 23:51:38.787452212 (7h24m52.374508453s) [upload=7599]: Upload finished. err: <nil> Aug 7 23:52:00.256204647 (7h25m13.843260858s) [upload=7610]: Starting upload Aug 7 23:52:00.256263547 (7h25m13.843319758s) [upload=7610]: Done starting upload Aug 7 23:52:00.256316195 (7h25m13.843372406s) [upload=7610]: Uploading segments... Aug 7 23:52:00.256345671 (7h25m13.843401881s) [upload=7610]: Got next segment. Inline: true Aug 7 23:52:00.256349758 (7h25m13.843405969s) [upload=7610]: Waiting for error group managing segments... Aug 7 23:52:00.256399992 (7h25m13.843456203s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:00.256404780 (7h25m13.843460991s) [upload=7610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:00.256407806 (7h25m13.843464017s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:30.237600164 (7h25m43.824656416s) [upload=7610]: Done uploading segments... Aug 7 23:52:30.237630862 (7h25m43.824687113s) [upload=7610]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:53.146251246 (7h26m6.733307456s) [upload=7626]: Starting upload Aug 7 23:52:53.146307339 (7h26m6.733363561s) [upload=7626]: Done starting upload Aug 7 23:52:53.146350962 (7h26m6.733407173s) [upload=7626]: Uploading segments... Aug 7 23:52:53.146411344 (7h26m6.733467565s) [upload=7626]: Got next segment. Inline: true Aug 7 23:52:53.146415271 (7h26m6.733471493s) [upload=7626]: Waiting for error group managing segments... Aug 7 23:52:53.146449966 (7h26m6.733506177s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:53.146454174 (7h26m6.733510395s) [upload=7626]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:53.146456709 (7h26m6.73351293s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.132536521 (7h26m36.719592742s) [upload=7626]: Done uploading segments... Aug 7 23:53:23.132560907 (7h26m36.719617128s) [upload=7626]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:28.217132518 (7h26m41.804188759s) [upload=7638]: Starting upload Aug 7 23:53:28.217216185 (7h26m41.804272426s) [upload=7638]: Done starting upload Aug 7 23:53:28.217288009 (7h26m41.80434426s) [upload=7638]: Uploading segments... Aug 7 23:53:28.217336129 (7h26m41.80439237s) [upload=7638]: Got next segment. Inline: true Aug 7 23:53:28.217341890 (7h26m41.804398131s) [upload=7638]: Waiting for error group managing segments... Aug 7 23:53:28.217421438 (7h26m41.804477679s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:28.217427650 (7h26m41.804483901s) [upload=7638]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:28.217431617 (7h26m41.804487859s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:58.204830989 (7h27m11.79188721s) [upload=7638]: Done uploading segments... Aug 7 23:53:58.204849294 (7h27m11.791905505s) [upload=7638]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:01.364416813 (7h27m14.951473054s) [upload=7648]: Starting upload Aug 7 23:54:01.364544481 (7h27m14.951600732s) [upload=7648]: Done starting upload Aug 7 23:54:01.364639518 (7h27m14.95169577s) [upload=7648]: Uploading segments... Aug 7 23:54:01.364703858 (7h27m14.951760109s) [upload=7648]: Got next segment. Inline: true Aug 7 23:54:01.364713517 (7h27m14.951769757s) [upload=7648]: Waiting for error group managing segments... Aug 7 23:54:01.364781374 (7h27m14.951837625s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:01.364791231 (7h27m14.951847472s) [upload=7648]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:01.364797502 (7h27m14.951853754s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:31.351186626 (7h27m44.938242847s) [upload=7648]: Done uploading segments... Aug 7 23:54:31.351241118 (7h27m44.938297339s) [upload=7648]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:31.416140839 (7h27m45.003197049s) [upload=7656]: Starting upload Aug 7 23:54:31.416189309 (7h27m45.00324553s) [upload=7656]: Done starting upload Aug 7 23:54:31.416235946 (7h27m45.003292157s) [upload=7656]: Uploading segments... Aug 7 23:54:31.416277384 (7h27m45.003333595s) [upload=7656]: Got next segment. Inline: true Aug 7 23:54:31.416297601 (7h27m45.003353813s) [upload=7656]: Waiting for error group managing segments... Aug 7 23:54:31.416336454 (7h27m45.003392665s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:31.416340652 (7h27m45.003396873s) [upload=7656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:31.416342965 (7h27m45.003399177s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:01.405081824 (7h28m14.992138035s) [upload=7656]: Done uploading segments... Aug 7 23:55:01.405101451 (7h28m14.992157672s) [upload=7656]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:45.764200449 (7h28m59.35125669s) [upload=7673]: Starting upload Aug 7 23:55:45.764300708 (7h28m59.351356948s) [upload=7673]: Done starting upload Aug 7 23:55:45.764398370 (7h28m59.351454611s) [upload=7673]: Uploading segments... Aug 7 23:55:45.764515538 (7h28m59.35157178s) [upload=7673]: Got next segment. Inline: true Aug 7 23:55:45.764521980 (7h28m59.351578222s) [upload=7673]: Waiting for error group managing segments... Aug 7 23:55:45.764570060 (7h28m59.3516263s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:45.764575871 (7h28m59.351632111s) [upload=7673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:45.764579898 (7h28m59.351636149s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:15.747212127 (7h29m29.334268369s) [upload=7673]: Done uploading segments... Aug 7 23:56:15.747307315 (7h29m29.334363566s) [upload=7673]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:15.823083783 (7h29m29.410140035s) [upload=7682]: Starting upload Aug 7 23:56:15.823169003 (7h29m29.410225243s) [upload=7682]: Done starting upload Aug 7 23:56:15.823245125 (7h29m29.410301367s) [upload=7682]: Uploading segments... Aug 7 23:56:15.823289758 (7h29m29.410346009s) [upload=7682]: Got next segment. Inline: true Aug 7 23:56:15.823295359 (7h29m29.410351599s) [upload=7682]: Waiting for error group managing segments... Aug 7 23:56:15.823343018 (7h29m29.410399269s) [upload=7682]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:15.823348678 (7h29m29.41040492s) [upload=7682]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:15.823352376 (7h29m29.410408616s) [upload=7682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:45.810149218 (7h29m59.397205439s) [upload=7682]: Done uploading segments... Aug 7 23:56:45.810167081 (7h29m59.397223322s) [upload=7682]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:58.169907263 (7h30m11.756963514s) [upload=7692]: Starting upload Aug 7 23:56:58.169972405 (7h30m11.757028647s) [upload=7692]: Done starting upload Aug 7 23:56:58.170060319 (7h30m11.75711657s) [upload=7692]: Uploading segments... Aug 7 23:56:58.170104551 (7h30m11.757160792s) [upload=7692]: Got next segment. Inline: true Aug 7 23:56:58.170110071 (7h30m11.757166313s) [upload=7692]: Waiting for error group managing segments... Aug 7 23:56:58.170155567 (7h30m11.757211808s) [upload=7692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.170161878 (7h30m11.75721812s) [upload=7692]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.170165555 (7h30m11.757221807s) [upload=7692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:28.152393221 (7h30m41.739449472s) [upload=7692]: Done uploading segments... Aug 7 23:57:28.152443946 (7h30m41.739500187s) [upload=7692]: Upload finished. err: metaclient: context deadline exceeded Duration: 194.89696ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXXX.X.X...........................................X.X..XXX] duration: 30.000674441s FAIL (2025-08-07 23:58:05 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000693636s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.137487832s FAIL (2025-08-07 23:58:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.137518039s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 9 out of 60 is failed (threshold=3) [XXXXX.X...............................................XX.X..] duration: 169.05193ms FAIL (2025-08-07 23:58:00 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:39.988158721 (7h20m53.575214932s) [upload=7536]: Done uploading segments... Aug 7 23:47:39.988176785 (7h20m53.575232995s) [upload=7536]: Upload finished. err: <nil> Aug 7 23:48:17.585904864 (7h21m31.172961076s) [upload=7548]: Starting upload Aug 7 23:48:17.585990855 (7h21m31.173047066s) [upload=7548]: Done starting upload Aug 7 23:48:17.586065835 (7h21m31.173122035s) [upload=7548]: Uploading segments... Aug 7 23:48:17.586102964 (7h21m31.173159205s) [upload=7548]: Got next segment. Inline: true Aug 7 23:48:17.586107262 (7h21m31.173163473s) [upload=7548]: Waiting for error group managing segments... Aug 7 23:48:17.586144451 (7h21m31.173200662s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:17.586148969 (7h21m31.17320518s) [upload=7548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:17.586151854 (7h21m31.173208066s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:17.642355873 (7h21m31.229412094s) [upload=7548]: Done uploading segments... Aug 7 23:48:17.642371582 (7h21m31.229427794s) [upload=7548]: Upload finished. err: <nil> Aug 7 23:49:19.929340015 (7h22m33.516396255s) [upload=7565]: Starting upload Aug 7 23:49:19.929497669 (7h22m33.51655391s) [upload=7565]: Done starting upload Aug 7 23:49:19.929593468 (7h22m33.516649709s) [upload=7565]: Uploading segments... Aug 7 23:49:19.929640947 (7h22m33.516697188s) [upload=7565]: Got next segment. Inline: true Aug 7 23:49:19.929647679 (7h22m33.516703931s) [upload=7565]: Waiting for error group managing segments... Aug 7 23:49:19.929695078 (7h22m33.516751329s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:19.929701059 (7h22m33.5167573s) [upload=7565]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:19.929704956 (7h22m33.516761197s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:19.992636681 (7h22m33.579692922s) [upload=7565]: Done uploading segments... Aug 7 23:49:19.992654385 (7h22m33.579710625s) [upload=7565]: Upload finished. err: <nil> Aug 7 23:50:07.176468149 (7h23m20.76352438s) [upload=7576]: Starting upload Aug 7 23:50:07.176537798 (7h23m20.763594019s) [upload=7576]: Done starting upload Aug 7 23:50:07.176592070 (7h23m20.763648311s) [upload=7576]: Uploading segments... Aug 7 23:50:07.176626473 (7h23m20.763682705s) [upload=7576]: Got next segment. Inline: true Aug 7 23:50:07.176631002 (7h23m20.763687213s) [upload=7576]: Waiting for error group managing segments... Aug 7 23:50:07.176664084 (7h23m20.763720295s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:07.176668132 (7h23m20.763724343s) [upload=7576]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:07.176670586 (7h23m20.763726807s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:07.240851653 (7h23m20.827907864s) [upload=7576]: Done uploading segments... Aug 7 23:50:07.240866771 (7h23m20.827922992s) [upload=7576]: Upload finished. err: <nil> Aug 7 23:50:45.893754262 (7h23m59.480810504s) [upload=7591]: Starting upload Aug 7 23:50:45.893825847 (7h23m59.480882088s) [upload=7591]: Done starting upload Aug 7 23:50:45.893912749 (7h23m59.48096899s) [upload=7591]: Uploading segments... Aug 7 23:50:45.893955949 (7h23m59.4810122s) [upload=7591]: Got next segment. Inline: true Aug 7 23:50:45.893961970 (7h23m59.481018211s) [upload=7591]: Waiting for error group managing segments... Aug 7 23:50:45.894005482 (7h23m59.481061723s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:45.894011062 (7h23m59.481067304s) [upload=7591]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:45.894014729 (7h23m59.48107097s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:05.079621048 (7h24m18.666677259s) [upload=7591]: Done uploading segments... Aug 7 23:51:05.079636297 (7h24m18.666692508s) [upload=7591]: Upload finished. err: <nil> Aug 7 23:51:21.725622238 (7h24m35.312678469s) [upload=7603]: Starting upload Aug 7 23:51:21.725696627 (7h24m35.312752838s) [upload=7603]: Done starting upload Aug 7 23:51:21.725758231 (7h24m35.312814443s) [upload=7603]: Uploading segments... Aug 7 23:51:21.725796774 (7h24m35.312852985s) [upload=7603]: Got next segment. Inline: true Aug 7 23:51:21.725802845 (7h24m35.312859056s) [upload=7603]: Waiting for error group managing segments... Aug 7 23:51:21.725848871 (7h24m35.312905082s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:21.725855313 (7h24m35.312911523s) [upload=7603]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:21.725859100 (7h24m35.312915351s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:21.784505948 (7h24m35.371562158s) [upload=7603]: Done uploading segments... Aug 7 23:51:21.784521767 (7h24m35.371577978s) [upload=7603]: Upload finished. err: <nil> Aug 7 23:52:10.285208176 (7h25m23.872264397s) [upload=7615]: Starting upload Aug 7 23:52:10.285263228 (7h25m23.872319449s) [upload=7615]: Done starting upload Aug 7 23:52:10.285310727 (7h25m23.872366937s) [upload=7615]: Uploading segments... Aug 7 23:52:10.285339711 (7h25m23.872395932s) [upload=7615]: Got next segment. Inline: true Aug 7 23:52:10.285343438 (7h25m23.872399649s) [upload=7615]: Waiting for error group managing segments... Aug 7 23:52:10.285391158 (7h25m23.872447379s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:10.285395065 (7h25m23.872451276s) [upload=7615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:10.285397770 (7h25m23.872453991s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:10.400888570 (7h25m23.987944781s) [upload=7615]: Done uploading segments... Aug 7 23:52:10.400904530 (7h25m23.987960741s) [upload=7615]: Upload finished. err: <nil> Aug 7 23:53:11.795016020 (7h26m25.382072262s) [upload=7635]: Starting upload Aug 7 23:53:11.795108102 (7h26m25.382164354s) [upload=7635]: Done starting upload Aug 7 23:53:11.795175588 (7h26m25.382231829s) [upload=7635]: Uploading segments... Aug 7 23:53:11.795254645 (7h26m25.382310886s) [upload=7635]: Got next segment. Inline: true Aug 7 23:53:11.795262130 (7h26m25.38231837s) [upload=7635]: Waiting for error group managing segments... Aug 7 23:53:11.795319296 (7h26m25.382375538s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:11.795326170 (7h26m25.38238242s) [upload=7635]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:11.795329857 (7h26m25.382386097s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:11.782974743 (7h27m25.370030954s) [upload=7635]: Done uploading segments... Aug 7 23:54:11.782991474 (7h27m25.370047695s) [upload=7635]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:11.855031501 (7h27m25.442087722s) [upload=7650]: Starting upload Aug 7 23:54:11.855087627 (7h27m25.442143838s) [upload=7650]: Done starting upload Aug 7 23:54:11.855131608 (7h27m25.442187829s) [upload=7650]: Uploading segments... Aug 7 23:54:11.855163227 (7h27m25.442219448s) [upload=7650]: Got next segment. Inline: true Aug 7 23:54:11.855167005 (7h27m25.442223226s) [upload=7650]: Waiting for error group managing segments... Aug 7 23:54:11.855198905 (7h27m25.442255126s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:11.855202661 (7h27m25.442258882s) [upload=7650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:11.855205036 (7h27m25.442261247s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:11.844108363 (7h28m25.431164634s) [upload=7650]: Done uploading segments... Aug 7 23:55:11.844129542 (7h28m25.431185763s) [upload=7650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:11.924363315 (7h28m25.511419527s) [upload=7668]: Starting upload Aug 7 23:55:11.924429198 (7h28m25.511485409s) [upload=7668]: Done starting upload Aug 7 23:55:11.924484862 (7h28m25.511541083s) [upload=7668]: Uploading segments... Aug 7 23:55:11.924519887 (7h28m25.511576098s) [upload=7668]: Got next segment. Inline: true Aug 7 23:55:11.924523564 (7h28m25.511579785s) [upload=7668]: Waiting for error group managing segments... Aug 7 23:55:11.924554131 (7h28m25.511610342s) [upload=7668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:11.924558279 (7h28m25.51161449s) [upload=7668]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:11.924560643 (7h28m25.511616855s) [upload=7668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:12.020746910 (7h28m25.607803121s) [upload=7668]: Done uploading segments... Aug 7 23:55:12.020765404 (7h28m25.607821625s) [upload=7668]: Upload finished. err: <nil> Aug 7 23:55:57.071847012 (7h29m10.658903254s) [upload=7678]: Starting upload Aug 7 23:55:57.071985611 (7h29m10.659041851s) [upload=7678]: Done starting upload Aug 7 23:55:57.072082963 (7h29m10.659139204s) [upload=7678]: Uploading segments... Aug 7 23:55:57.072231269 (7h29m10.65928751s) [upload=7678]: Got next segment. Inline: true Aug 7 23:55:57.072240506 (7h29m10.659296748s) [upload=7678]: Waiting for error group managing segments... Aug 7 23:55:57.072359458 (7h29m10.6594157s) [upload=7678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:57.072367143 (7h29m10.659423384s) [upload=7678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:57.072371320 (7h29m10.659427562s) [upload=7678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:57.049680168 (7h30m10.636736379s) [upload=7678]: Done uploading segments... Aug 7 23:56:57.049696499 (7h30m10.636752709s) [upload=7678]: Upload finished. err: metaclient: context deadline exceeded Duration:* 169.079301ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.X.XXXXX..X.................................................] duration: 197.111014ms FAIL (2025-08-07 23:57:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 197.127685ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X....XXX.X......................................XXX] duration: 30.000654956s FAIL (2025-08-07 23:57:47 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ed Aug 7 23:53:32.466167823 (7h26m46.053224095s) [upload=7640]: Starting upload Aug 7 23:53:32.466240499 (7h26m46.05329672s) [upload=7640]: Done starting upload Aug 7 23:53:32.466316471 (7h26m46.053372692s) [upload=7640]: Uploading segments... Aug 7 23:53:32.466494353 (7h26m46.053550574s) [upload=7640]: Got next segment. Inline: false Aug 7 23:53:32.466534318 (7h26m46.053590529s) [upload=7640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:32.466538625 (7h26m46.053594846s) [upload=7640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:49.520645082 (7h27m3.107701324s) [seg_pos={0 0}, upload=7640]: Begin upload segment... Aug 7 23:53:49.520694595 (7h27m3.107750835s) [seg_pos={0 0}, upload=7640]: Done begin upload segment. Aug 7 23:53:49.520735481 (7h27m3.107791722s) [noise=true , address=45.91.116.4:4939 , piece_id=LWLSQN6BSOJY4PA7I7HZL5PSRI4DNR6E4GMYWPUB6DF6BLMPU26A, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.520814889 (7h27m3.10787119s) [noise=true , address=77.90.31.18:4807 , piece_id=5CCUKS55ROKTHMZA7NKUNYXOTAKYZIDQGDJBSMFM5TSBEFVDJMPA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.521449253 (7h27m3.108505494s) [noise=true , address=77.90.31.16:4734 , piece_id=AIAJVZAYQTA4OR6WDJ6V2XH3TXBO6FA4VRUCYJGIFXXUO3ZGIT4A, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.521498244 (7h27m3.108554486s) [noise=true , address=104.36.82.14:4221 , piece_id=JKDWKECRRZCOLROOZ5QXNOGIRN3LLLAUGDQRFBPDYBHBV6LTPH7Q, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.521948114 (7h27m3.109004366s) [noise=true , address=104.36.82.16:4294 , piece_id=PL7A2YIDU7ZU2F5J4IE37IFPYADS2VNPLY443J46RIBG7J6ZNZJA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522017724 (7h27m3.109073975s) [noise=true , address=45.91.116.12:4226 , piece_id=YELH2JJ2TFX6A2FUHKFGZFH3RKFQCDUBTA4OBW6B6BSWPJ4FTQSQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522360804 (7h27m3.109417046s) [noise=true , address=45.91.116.14:4190 , piece_id=PPYDTYVAD7ESR52JJNH64JB22XD5B3WTXTZ7X72X6KCG2FVK5MXA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522463095 (7h27m3.109519336s) [noise=true , address=104.36.82.11:4113 , piece_id=N5ABMUNW7Q2MPYAXORNWQMJYIGMHHLYKHZSM7FD265TVN7QPBILQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522783393 (7h27m3.109839644s) [noise=true , address=45.91.116.5:4297 , piece_id=WYHPPXNXBWTSD4LO2RWXPUN37P7JULRPJSIA3G5KNBYBXXBUDEXQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522884061 (7h27m3.109940302s) [noise=true , address=104.36.82.6:4874 , piece_id=FZBS75TSUST27CMCEC5UZAAC2ZUNZ5F3PX5SIFHLN72GALS4XNTA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523171748 (7h27m3.110228s) [noise=true , address=104.36.82.9:4006 , piece_id=6RYH7RTUR45VDJQKTMW6QXWWPPC2FRU2PGECQWGIKUAQOG4DCVEA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523292534 (7h27m3.110348775s) [noise=true , address=45.91.116.8:4048 , piece_id=JS3BGIHYWKZYEAXNUCCIYHEDRA2YQMBM2OBPJOUWIAWNEBO6BYFQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523596000 (7h27m3.110652251s) [noise=true , address=77.90.31.17:4770 , piece_id=QQOG5XJMG2CHWVCGAYWMQMRAHGEKQ5AITZOOOOOVUEFWDFHGBSEA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523713870 (7h27m3.110770111s) [noise=true , address=45.91.116.8:4047 , piece_id=SJVTTSCB7BDZRLEJX4WXD4PLXBALDVO2GWIQ5RUI6GAOVMNCERLQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.524251423 (7h27m3.111307674s) [noise=true , address=45.91.116.6:4974 , piece_id=RHPHBCUSDUVI23GGMG2WTVNLMQAAK436N3FXYHPBJ6FV7WOMSFMA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.524718816 (7h27m3.111775057s) [noise=true , address=45.91.116.11:4117 , piece_id=H4RP6BXPN4GRE2TMY5GZEOZ5UKKVDOEAGAS72ZRBIOURV7OBB5MQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.647138359 (7h27m3.2341946s) [upload=7640]: Next returned nil segment Aug 7 23:53:49.647155511 (7h27m3.234211752s) [upload=7640]: Waiting for error group managing segments... Aug 7 23:53:49.655170752 (7h27m3.242226993s) [noise=true , address=45.91.116.14:4190 , piece_id=PPYDTYVAD7ESR52JJNH64JB22XD5B3WTXTZ7X72X6KCG2FVK5MXA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.655792202 (7h27m3.242848433s) [noise=true , address=45.91.116.5:4297 , piece_id=WYHPPXNXBWTSD4LO2RWXPUN37P7JULRPJSIA3G5KNBYBXXBUDEXQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.656163956 (7h27m3.243220187s) [noise=true , address=45.91.116.8:4048 , piece_id=JS3BGIHYWKZYEAXNUCCIYHEDRA2YQMBM2OBPJOUWIAWNEBO6BYFQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.656798870 (7h27m3.243855091s) [noise=true , address=45.91.116.12:4226 , piece_id=YELH2JJ2TFX6A2FUHKFGZFH3RKFQCDUBTA4OBW6B6BSWPJ4FTQSQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.657996216 (7h27m3.245052458s) [noise=true , address=45.91.116.4:4939 , piece_id=LWLSQN6BSOJY4PA7I7HZL5PSRI4DNR6E4GMYWPUB6DF6BLMPU26A, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.658644787 (7h27m3.245701017s) [noise=true , address=45.91.116.11:4117 , piece_id=H4RP6BXPN4GRE2TMY5GZEOZ5UKKVDOEAGAS72ZRBIOURV7OBB5MQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.659153586 (7h27m3.246209817s) [noise=true , address=77.90.31.17:4770 , piece_id=QQOG5XJMG2CHWVCGAYWMQMRAHGEKQ5AITZOOOOOVUEFWDFHGBSEA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.660604835 (7h27m3.247661076s) [noise=true , address=45.91.116.8:4047 , piece_id=SJVTTSCB7BDZRLEJX4WXD4PLXBALDVO2GWIQ5RUI6GAOVMNCERLQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.660941473 (7h27m3.247997694s) [noise=true , address=45.91.116.6:4974 , piece_id=RHPHBCUSDUVI23GGMG2WTVNLMQAAK436N3FXYHPBJ6FV7WOMSFMA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.661457717 (7h27m3.248513959s) [noise=true , address=77.90.31.18:4807 , piece_id=5CCUKS55ROKTHMZA7NKUNYXOTAKYZIDQGDJBSMFM5TSBEFVDJMPA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.662043810 (7h27m3.249100041s) [noise=true , address=104.36.82.6:4874 , piece_id=FZBS75TSUST27CMCEC5UZAAC2ZUNZ5F3PX5SIFHLN72GALS4XNTA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665007553 (7h27m3.252063794s) [noise=true , address=104.36.82.11:4113 , piece_id=N5ABMUNW7Q2MPYAXORNWQMJYIGMHHLYKHZSM7FD265TVN7QPBILQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665952497 (7h27m3.253008727s) [noise=true , address=77.90.31.16:4734 , piece_id=AIAJVZAYQTA4OR6WDJ6V2XH3TXBO6FA4VRUCYJGIFXXUO3ZGIT4A, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665980449 (7h27m3.25303667s) [seg_pos={0 0}, upload=7640]: Segment reached optimal threshold of 13 pieces. Aug 7 23:53:49.666316626 (7h27m3.253372857s) [noise=true , address=104.36.82.14:4221 , piece_id=JKDWKECRRZCOLROOZ5QXNOGIRN3LLLAUGDQRFBPDYBHBV6LTPH7Q, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: rpc: dial tcp 104.36.82.14:4221: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666330082 (7h27m3.253386322s) [noise=true , address=104.36.82.9:4006 , piece_id=6RYH7RTUR45VDJQKTMW6QXWWPPC2FRU2PGECQWGIKUAQOG4DCVEA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: read tcp 5.161.89.252:14750->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666492835 (7h27m3.253549076s) [noise=true , address=104.36.82.16:4294 , piece_id=PL7A2YIDU7ZU2F5J4IE37IFPYADS2VNPLY443J46RIBG7J6ZNZJA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 5.161.89.252:34080->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666519565 (7h27m3.253575806s) [seg_pos={0 0}, upload=7640]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:53:49.666614813 (7h27m3.253671053s) [upload=7640]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:53:49.666620242 (7h27m3.253676484s) [upload=7640]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:17.275326827 (7h30m30.862383048s) [upload=7640]: Done uploading segments... Aug 7 23:57:17.275353585 (7h30m30.862409807s) [upload=7640]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:17.349138524 (7h30m30.936194735s) [upload=7698]: Starting upload Aug 7 23:57:17.349187294 (7h30m30.936243515s) [upload=7698]: Done starting upload Aug 7 23:57:17.349221879 (7h30m30.93627809s) [upload=7698]: Uploading segments... Aug 7 23:57:17.349404010 (7h30m30.93646022s) [upload=7698]: Got next segment. Inline: false Aug 7 23:57:17.349444765 (7h30m30.936500976s) [upload=7698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:17.349448802 (7h30m30.936505023s) [upload=7698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:57:47.334809460 (7h31m0.921865702s) [upload=7698]: Next segment err: context deadline exceeded Aug 7 23:57:47.334985600 (7h31m0.922041831s) [upload=7698]: Done uploading segments... Aug 7 23:57:47.335006539 (7h31m0.922062781s) [upload=7698]: Upload finished. err: context deadline exceeded Duration: 30.000670305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 946.596478ms FAIL (2025-08-07 23:57:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 946.618389ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.040739835s FAIL (2025-08-07 23:57:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== 1345s) [upload=7262]: Got next segment. Inline: false Aug 7 23:31:37.583906637 (7h4m51.170962878s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:31:37.583953414 (7h4m51.171009656s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:31:37.613963041 (7h4m51.201019262s) [seg_pos={0 0}, upload=7262]: Begin upload segment... Aug 7 23:31:37.614008966 (7h4m51.201065176s) [seg_pos={0 0}, upload=7262]: Done begin upload segment. Aug 7 23:31:37.614029103 (7h4m51.201085324s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614108311 (7h4m51.201164542s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614344612 (7h4m51.201400833s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614634754 (7h4m51.201690975s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614919926 (7h4m51.201976137s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614943150 (7h4m51.201999391s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615146719 (7h4m51.20220294s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615454163 (7h4m51.202510384s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615514677 (7h4m51.202570918s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615688160 (7h4m51.202744381s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615932236 (7h4m51.202988457s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616184166 (7h4m51.203240378s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616521716 (7h4m51.203577937s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616643614 (7h4m51.203699855s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621231117 (7h4m51.208287338s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621561163 (7h4m51.208617374s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621861323 (7h4m51.208917534s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.622156223 (7h4m51.209212444s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.726040884 (7h4m51.313097094s) [upload=7262]: Next returned nil segment Aug 7 23:31:37.726066772 (7h4m51.313122983s) [upload=7262]: Waiting for error group managing segments... Aug 7 23:31:37.736232517 (7h4m51.323288738s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.737656014 (7h4m51.324712245s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.739754932 (7h4m51.326811153s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.740628113 (7h4m51.327684344s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741712657 (7h4m51.328768867s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741742653 (7h4m51.328798883s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.748242274 (7h4m51.335298505s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.750434577 (7h4m51.337490798s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.758962364 (7h4m51.346018586s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.769693706 (7h4m51.356749917s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.800540734 (7h4m51.387596975s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.924012655 (7h4m51.511068866s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.059695219 (7h4m51.64675144s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102515917 (7h4m51.689572138s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102541606 (7h4m51.689597827s) [seg_pos={0 0}, upload=7262]: Segment reached optimal threshold of 14 pieces. Aug 7 23:31:38.102658574 (7h4m51.689714795s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF Aug 7 23:31:38.102748111 (7h4m51.689804332s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE): protocol: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection Aug 7 23:31:38.102832158 (7h4m51.689888399s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GcyWwKkfavsCsmtq55U72n6XeZKqQiGGYHY9D6UwST8sC14Ac): protocol: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection Aug 7 23:31:38.102967200 (7h4m51.690023421s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12syasUPsyYf4UstQ6cFAr6qamze3vCyAm9dnobk5e83w4bTz69): protocol: context canceled; context canceled Aug 7 23:31:38.102995132 (7h4m51.690051374s) [seg_pos={0 0}, upload=7262]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:31:38.103097553 (7h4m51.690153775s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:31:38.103102412 (7h4m51.690158633s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:31:38.203633212 (7h4m51.790689433s) [upload=7262]: Done uploading segments... Aug 7 23:31:38.203649102 (7h4m51.790705323s) [upload=7262]: Upload finished. err: <nil> Duration: 1.04076381s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X....X.X..........................................X.......] duration: 452.339833ms FAIL (2025-08-07 23:57:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 452.361293ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 15 out of 60 is failed (threshold=3) [XXXX.XX............................................XXXXXXXXX] duration: 30.001601937s FAIL (2025-08-07 23:57:28 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 (7h23m15.650779768s) [upload=7574]: Upload finished. err: <nil> Aug 7 23:50:41.160166556 (7h23m54.747222777s) [upload=7588]: Starting upload Aug 7 23:50:41.160220555 (7h23m54.747276777s) [upload=7588]: Done starting upload Aug 7 23:50:41.160264659 (7h23m54.74732088s) [upload=7588]: Uploading segments... Aug 7 23:50:41.160295956 (7h23m54.747352177s) [upload=7588]: Got next segment. Inline: true Aug 7 23:50:41.160299753 (7h23m54.747355964s) [upload=7588]: Waiting for error group managing segments... Aug 7 23:50:41.160331633 (7h23m54.747387844s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:41.160335861 (7h23m54.747392072s) [upload=7588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:41.160338396 (7h23m54.747394617s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:41.222859735 (7h23m54.809915976s) [upload=7588]: Done uploading segments... Aug 7 23:50:41.222879913 (7h23m54.809936154s) [upload=7588]: Upload finished. err: <nil> Aug 7 23:51:17.131690986 (7h24m30.718747197s) [upload=7599]: Starting upload Aug 7 23:51:17.131738926 (7h24m30.718795147s) [upload=7599]: Done starting upload Aug 7 23:51:17.131792947 (7h24m30.718849178s) [upload=7599]: Uploading segments... Aug 7 23:51:17.131835896 (7h24m30.718892117s) [upload=7599]: Got next segment. Inline: true Aug 7 23:51:17.131842027 (7h24m30.718898239s) [upload=7599]: Waiting for error group managing segments... Aug 7 23:51:17.131883385 (7h24m30.718939627s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:17.131889146 (7h24m30.718945388s) [upload=7599]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:17.132006205 (7h24m30.719062435s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:38.787433326 (7h24m52.374489567s) [upload=7599]: Done uploading segments... Aug 7 23:51:38.787452212 (7h24m52.374508453s) [upload=7599]: Upload finished. err: <nil> Aug 7 23:52:00.256204647 (7h25m13.843260858s) [upload=7610]: Starting upload Aug 7 23:52:00.256263547 (7h25m13.843319758s) [upload=7610]: Done starting upload Aug 7 23:52:00.256316195 (7h25m13.843372406s) [upload=7610]: Uploading segments... Aug 7 23:52:00.256345671 (7h25m13.843401881s) [upload=7610]: Got next segment. Inline: true Aug 7 23:52:00.256349758 (7h25m13.843405969s) [upload=7610]: Waiting for error group managing segments... Aug 7 23:52:00.256399992 (7h25m13.843456203s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:00.256404780 (7h25m13.843460991s) [upload=7610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:00.256407806 (7h25m13.843464017s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:30.237600164 (7h25m43.824656416s) [upload=7610]: Done uploading segments... Aug 7 23:52:30.237630862 (7h25m43.824687113s) [upload=7610]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:53.146251246 (7h26m6.733307456s) [upload=7626]: Starting upload Aug 7 23:52:53.146307339 (7h26m6.733363561s) [upload=7626]: Done starting upload Aug 7 23:52:53.146350962 (7h26m6.733407173s) [upload=7626]: Uploading segments... Aug 7 23:52:53.146411344 (7h26m6.733467565s) [upload=7626]: Got next segment. Inline: true Aug 7 23:52:53.146415271 (7h26m6.733471493s) [upload=7626]: Waiting for error group managing segments... Aug 7 23:52:53.146449966 (7h26m6.733506177s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:53.146454174 (7h26m6.733510395s) [upload=7626]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:53.146456709 (7h26m6.73351293s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.132536521 (7h26m36.719592742s) [upload=7626]: Done uploading segments... Aug 7 23:53:23.132560907 (7h26m36.719617128s) [upload=7626]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:28.217132518 (7h26m41.804188759s) [upload=7638]: Starting upload Aug 7 23:53:28.217216185 (7h26m41.804272426s) [upload=7638]: Done starting upload Aug 7 23:53:28.217288009 (7h26m41.80434426s) [upload=7638]: Uploading segments... Aug 7 23:53:28.217336129 (7h26m41.80439237s) [upload=7638]: Got next segment. Inline: true Aug 7 23:53:28.217341890 (7h26m41.804398131s) [upload=7638]: Waiting for error group managing segments... Aug 7 23:53:28.217421438 (7h26m41.804477679s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:28.217427650 (7h26m41.804483901s) [upload=7638]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:28.217431617 (7h26m41.804487859s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:58.204830989 (7h27m11.79188721s) [upload=7638]: Done uploading segments... Aug 7 23:53:58.204849294 (7h27m11.791905505s) [upload=7638]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:01.364416813 (7h27m14.951473054s) [upload=7648]: Starting upload Aug 7 23:54:01.364544481 (7h27m14.951600732s) [upload=7648]: Done starting upload Aug 7 23:54:01.364639518 (7h27m14.95169577s) [upload=7648]: Uploading segments... Aug 7 23:54:01.364703858 (7h27m14.951760109s) [upload=7648]: Got next segment. Inline: true Aug 7 23:54:01.364713517 (7h27m14.951769757s) [upload=7648]: Waiting for error group managing segments... Aug 7 23:54:01.364781374 (7h27m14.951837625s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:01.364791231 (7h27m14.951847472s) [upload=7648]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:01.364797502 (7h27m14.951853754s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:31.351186626 (7h27m44.938242847s) [upload=7648]: Done uploading segments... Aug 7 23:54:31.351241118 (7h27m44.938297339s) [upload=7648]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:31.416140839 (7h27m45.003197049s) [upload=7656]: Starting upload Aug 7 23:54:31.416189309 (7h27m45.00324553s) [upload=7656]: Done starting upload Aug 7 23:54:31.416235946 (7h27m45.003292157s) [upload=7656]: Uploading segments... Aug 7 23:54:31.416277384 (7h27m45.003333595s) [upload=7656]: Got next segment. Inline: true Aug 7 23:54:31.416297601 (7h27m45.003353813s) [upload=7656]: Waiting for error group managing segments... Aug 7 23:54:31.416336454 (7h27m45.003392665s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:31.416340652 (7h27m45.003396873s) [upload=7656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:31.416342965 (7h27m45.003399177s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:01.405081824 (7h28m14.992138035s) [upload=7656]: Done uploading segments... Aug 7 23:55:01.405101451 (7h28m14.992157672s) [upload=7656]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:45.764200449 (7h28m59.35125669s) [upload=7673]: Starting upload Aug 7 23:55:45.764300708 (7h28m59.351356948s) [upload=7673]: Done starting upload Aug 7 23:55:45.764398370 (7h28m59.351454611s) [upload=7673]: Uploading segments... Aug 7 23:55:45.764515538 (7h28m59.35157178s) [upload=7673]: Got next segment. Inline: true Aug 7 23:55:45.764521980 (7h28m59.351578222s) [upload=7673]: Waiting for error group managing segments... Aug 7 23:55:45.764570060 (7h28m59.3516263s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:45.764575871 (7h28m59.351632111s) [upload=7673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:45.764579898 (7h28m59.351636149s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:15.747212127 (7h29m29.334268369s) [upload=7673]: Done uploading segments... Aug 7 23:56:15.747307315 (7h29m29.334363566s) [upload=7673]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:15.823083783 (7h29m29.410140035s) [upload=7682]: Starting upload Aug 7 23:56:15.823169003 (7h29m29.410225243s) [upload=7682]: Done starting upload Aug 7 23:56:15.823245125 (7h29m29.410301367s) [upload=7682]: Uploading segments... Aug 7 23:56:15.823289758 (7h29m29.410346009s) [upload=7682]: Got next segment. Inline: true Aug 7 23:56:15.823295359 (7h29m29.410351599s) [upload=7682]: Waiting for error group managing segments... Aug 7 23:56:15.823343018 (7h29m29.410399269s) [upload=7682]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:15.823348678 (7h29m29.41040492s) [upload=7682]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:15.823352376 (7h29m29.410408616s) [upload=7682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:45.810149218 (7h29m59.397205439s) [upload=7682]: Done uploading segments... Aug 7 23:56:45.810167081 (7h29m59.397223322s) [upload=7682]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:58.169907263 (7h30m11.756963514s) [upload=7692]: Starting upload Aug 7 23:56:58.169972405 (7h30m11.757028647s) [upload=7692]: Done starting upload Aug 7 23:56:58.170060319 (7h30m11.75711657s) [upload=7692]: Uploading segments... Aug 7 23:56:58.170104551 (7h30m11.757160792s) [upload=7692]: Got next segment. Inline: true Aug 7 23:56:58.170110071 (7h30m11.757166313s) [upload=7692]: Waiting for error group managing segments... Aug 7 23:56:58.170155567 (7h30m11.757211808s) [upload=7692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.170161878 (7h30m11.75721812s) [upload=7692]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.170165555 (7h30m11.757221807s) [upload=7692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:28.152393221 (7h30m41.739449472s) [upload=7692]: Done uploading segments... Aug 7 23:57:28.152443946 (7h30m41.739500187s) [upload=7692]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001623948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXXXX.X.X...........................................X.X..XX] duration: 30.000634442s FAIL (2025-08-07 23:57:23 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000657797s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X....XXX.X......................................XX] duration: 3m44.82124374s FAIL (2025-08-07 23:57:17 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ug 7 23:53:02.405645052 (7h26m15.992701273s) [upload=7627]: Starting upload Aug 7 23:53:02.405704283 (7h26m15.992760504s) [upload=7627]: Done starting upload Aug 7 23:53:02.405806533 (7h26m15.992862744s) [upload=7627]: Uploading segments... Aug 7 23:53:02.406021685 (7h26m15.993077906s) [upload=7627]: Got next segment. Inline: false Aug 7 23:53:02.406061369 (7h26m15.99311764s) [upload=7627]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:02.406068102 (7h26m15.993124323s) [upload=7627]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:32.387466061 (7h26m45.974522302s) [upload=7627]: Next segment err: context deadline exceeded Aug 7 23:53:32.387608347 (7h26m45.974664588s) [upload=7627]: Done uploading segments... Aug 7 23:53:32.387634907 (7h26m45.974691158s) [upload=7627]: Upload finished. err: context deadline exceeded Aug 7 23:53:32.466167823 (7h26m46.053224095s) [upload=7640]: Starting upload Aug 7 23:53:32.466240499 (7h26m46.05329672s) [upload=7640]: Done starting upload Aug 7 23:53:32.466316471 (7h26m46.053372692s) [upload=7640]: Uploading segments... Aug 7 23:53:32.466494353 (7h26m46.053550574s) [upload=7640]: Got next segment. Inline: false Aug 7 23:53:32.466534318 (7h26m46.053590529s) [upload=7640]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:32.466538625 (7h26m46.053594846s) [upload=7640]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:49.520645082 (7h27m3.107701324s) [seg_pos={0 0}, upload=7640]: Begin upload segment... Aug 7 23:53:49.520694595 (7h27m3.107750835s) [seg_pos={0 0}, upload=7640]: Done begin upload segment. Aug 7 23:53:49.520735481 (7h27m3.107791722s) [noise=true , address=45.91.116.4:4939 , piece_id=LWLSQN6BSOJY4PA7I7HZL5PSRI4DNR6E4GMYWPUB6DF6BLMPU26A, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.520814889 (7h27m3.10787119s) [noise=true , address=77.90.31.18:4807 , piece_id=5CCUKS55ROKTHMZA7NKUNYXOTAKYZIDQGDJBSMFM5TSBEFVDJMPA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.521449253 (7h27m3.108505494s) [noise=true , address=77.90.31.16:4734 , piece_id=AIAJVZAYQTA4OR6WDJ6V2XH3TXBO6FA4VRUCYJGIFXXUO3ZGIT4A, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.521498244 (7h27m3.108554486s) [noise=true , address=104.36.82.14:4221 , piece_id=JKDWKECRRZCOLROOZ5QXNOGIRN3LLLAUGDQRFBPDYBHBV6LTPH7Q, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.521948114 (7h27m3.109004366s) [noise=true , address=104.36.82.16:4294 , piece_id=PL7A2YIDU7ZU2F5J4IE37IFPYADS2VNPLY443J46RIBG7J6ZNZJA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522017724 (7h27m3.109073975s) [noise=true , address=45.91.116.12:4226 , piece_id=YELH2JJ2TFX6A2FUHKFGZFH3RKFQCDUBTA4OBW6B6BSWPJ4FTQSQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522360804 (7h27m3.109417046s) [noise=true , address=45.91.116.14:4190 , piece_id=PPYDTYVAD7ESR52JJNH64JB22XD5B3WTXTZ7X72X6KCG2FVK5MXA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522463095 (7h27m3.109519336s) [noise=true , address=104.36.82.11:4113 , piece_id=N5ABMUNW7Q2MPYAXORNWQMJYIGMHHLYKHZSM7FD265TVN7QPBILQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522783393 (7h27m3.109839644s) [noise=true , address=45.91.116.5:4297 , piece_id=WYHPPXNXBWTSD4LO2RWXPUN37P7JULRPJSIA3G5KNBYBXXBUDEXQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.522884061 (7h27m3.109940302s) [noise=true , address=104.36.82.6:4874 , piece_id=FZBS75TSUST27CMCEC5UZAAC2ZUNZ5F3PX5SIFHLN72GALS4XNTA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523171748 (7h27m3.110228s) [noise=true , address=104.36.82.9:4006 , piece_id=6RYH7RTUR45VDJQKTMW6QXWWPPC2FRU2PGECQWGIKUAQOG4DCVEA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523292534 (7h27m3.110348775s) [noise=true , address=45.91.116.8:4048 , piece_id=JS3BGIHYWKZYEAXNUCCIYHEDRA2YQMBM2OBPJOUWIAWNEBO6BYFQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523596000 (7h27m3.110652251s) [noise=true , address=77.90.31.17:4770 , piece_id=QQOG5XJMG2CHWVCGAYWMQMRAHGEKQ5AITZOOOOOVUEFWDFHGBSEA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.523713870 (7h27m3.110770111s) [noise=true , address=45.91.116.8:4047 , piece_id=SJVTTSCB7BDZRLEJX4WXD4PLXBALDVO2GWIQ5RUI6GAOVMNCERLQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.524251423 (7h27m3.111307674s) [noise=true , address=45.91.116.6:4974 , piece_id=RHPHBCUSDUVI23GGMG2WTVNLMQAAK436N3FXYHPBJ6FV7WOMSFMA, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.524718816 (7h27m3.111775057s) [noise=true , address=45.91.116.11:4117 , piece_id=H4RP6BXPN4GRE2TMY5GZEOZ5UKKVDOEAGAS72ZRBIOURV7OBB5MQ, seg_pos={0 0}, upload=7640]: Uploading piece... Aug 7 23:53:49.647138359 (7h27m3.2341946s) [upload=7640]: Next returned nil segment Aug 7 23:53:49.647155511 (7h27m3.234211752s) [upload=7640]: Waiting for error group managing segments... Aug 7 23:53:49.655170752 (7h27m3.242226993s) [noise=true , address=45.91.116.14:4190 , piece_id=PPYDTYVAD7ESR52JJNH64JB22XD5B3WTXTZ7X72X6KCG2FVK5MXA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.655792202 (7h27m3.242848433s) [noise=true , address=45.91.116.5:4297 , piece_id=WYHPPXNXBWTSD4LO2RWXPUN37P7JULRPJSIA3G5KNBYBXXBUDEXQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.656163956 (7h27m3.243220187s) [noise=true , address=45.91.116.8:4048 , piece_id=JS3BGIHYWKZYEAXNUCCIYHEDRA2YQMBM2OBPJOUWIAWNEBO6BYFQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.656798870 (7h27m3.243855091s) [noise=true , address=45.91.116.12:4226 , piece_id=YELH2JJ2TFX6A2FUHKFGZFH3RKFQCDUBTA4OBW6B6BSWPJ4FTQSQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.657996216 (7h27m3.245052458s) [noise=true , address=45.91.116.4:4939 , piece_id=LWLSQN6BSOJY4PA7I7HZL5PSRI4DNR6E4GMYWPUB6DF6BLMPU26A, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.658644787 (7h27m3.245701017s) [noise=true , address=45.91.116.11:4117 , piece_id=H4RP6BXPN4GRE2TMY5GZEOZ5UKKVDOEAGAS72ZRBIOURV7OBB5MQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.659153586 (7h27m3.246209817s) [noise=true , address=77.90.31.17:4770 , piece_id=QQOG5XJMG2CHWVCGAYWMQMRAHGEKQ5AITZOOOOOVUEFWDFHGBSEA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.660604835 (7h27m3.247661076s) [noise=true , address=45.91.116.8:4047 , piece_id=SJVTTSCB7BDZRLEJX4WXD4PLXBALDVO2GWIQ5RUI6GAOVMNCERLQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.660941473 (7h27m3.247997694s) [noise=true , address=45.91.116.6:4974 , piece_id=RHPHBCUSDUVI23GGMG2WTVNLMQAAK436N3FXYHPBJ6FV7WOMSFMA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.661457717 (7h27m3.248513959s) [noise=true , address=77.90.31.18:4807 , piece_id=5CCUKS55ROKTHMZA7NKUNYXOTAKYZIDQGDJBSMFM5TSBEFVDJMPA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.662043810 (7h27m3.249100041s) [noise=true , address=104.36.82.6:4874 , piece_id=FZBS75TSUST27CMCEC5UZAAC2ZUNZ5F3PX5SIFHLN72GALS4XNTA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665007553 (7h27m3.252063794s) [noise=true , address=104.36.82.11:4113 , piece_id=N5ABMUNW7Q2MPYAXORNWQMJYIGMHHLYKHZSM7FD265TVN7QPBILQ, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665952497 (7h27m3.253008727s) [noise=true , address=77.90.31.16:4734 , piece_id=AIAJVZAYQTA4OR6WDJ6V2XH3TXBO6FA4VRUCYJGIFXXUO3ZGIT4A, seg_pos={0 0}, upload=7640]: Done uploading piece. err: <nil> Aug 7 23:53:49.665980449 (7h27m3.25303667s) [seg_pos={0 0}, upload=7640]: Segment reached optimal threshold of 13 pieces. Aug 7 23:53:49.666316626 (7h27m3.253372857s) [noise=true , address=104.36.82.14:4221 , piece_id=JKDWKECRRZCOLROOZ5QXNOGIRN3LLLAUGDQRFBPDYBHBV6LTPH7Q, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: rpc: dial tcp 104.36.82.14:4221: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666330082 (7h27m3.253386322s) [noise=true , address=104.36.82.9:4006 , piece_id=6RYH7RTUR45VDJQKTMW6QXWWPPC2FRU2PGECQWGIKUAQOG4DCVEA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: read tcp 5.161.89.252:14750->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666492835 (7h27m3.253549076s) [noise=true , address=104.36.82.16:4294 , piece_id=PL7A2YIDU7ZU2F5J4IE37IFPYADS2VNPLY443J46RIBG7J6ZNZJA, seg_pos={0 0}, upload=7640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 5.161.89.252:34080->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:53:49.666519565 (7h27m3.253575806s) [seg_pos={0 0}, upload=7640]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:53:49.666614813 (7h27m3.253671053s) [upload=7640]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:53:49.666620242 (7h27m3.253676484s) [upload=7640]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:17.275326827 (7h30m30.862383048s) [upload=7640]: Done uploading segments... Aug 7 23:57:17.275353585 (7h30m30.862409807s) [upload=7640]: Upload finished. err: metaclient: context deadline exceeded Duration: 3m44.821258437s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...XXXXX..X............................................XXXXX] duration: 1m0.001259588s FAIL (2025-08-07 23:57:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001279494s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX.X....X.X..........................................X......] duration: 443.710312ms FAIL (2025-08-07 23:57:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 443.758944ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 10 out of 60 is failed (threshold=3) [XXXXXX.X...............................................XX.X.] duration: 214.979935ms FAIL (2025-08-07 23:57:01 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:39.988158721 (7h20m53.575214932s) [upload=7536]: Done uploading segments... Aug 7 23:47:39.988176785 (7h20m53.575232995s) [upload=7536]: Upload finished. err: <nil> Aug 7 23:48:17.585904864 (7h21m31.172961076s) [upload=7548]: Starting upload Aug 7 23:48:17.585990855 (7h21m31.173047066s) [upload=7548]: Done starting upload Aug 7 23:48:17.586065835 (7h21m31.173122035s) [upload=7548]: Uploading segments... Aug 7 23:48:17.586102964 (7h21m31.173159205s) [upload=7548]: Got next segment. Inline: true Aug 7 23:48:17.586107262 (7h21m31.173163473s) [upload=7548]: Waiting for error group managing segments... Aug 7 23:48:17.586144451 (7h21m31.173200662s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:17.586148969 (7h21m31.17320518s) [upload=7548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:17.586151854 (7h21m31.173208066s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:17.642355873 (7h21m31.229412094s) [upload=7548]: Done uploading segments... Aug 7 23:48:17.642371582 (7h21m31.229427794s) [upload=7548]: Upload finished. err: <nil> Aug 7 23:49:19.929340015 (7h22m33.516396255s) [upload=7565]: Starting upload Aug 7 23:49:19.929497669 (7h22m33.51655391s) [upload=7565]: Done starting upload Aug 7 23:49:19.929593468 (7h22m33.516649709s) [upload=7565]: Uploading segments... Aug 7 23:49:19.929640947 (7h22m33.516697188s) [upload=7565]: Got next segment. Inline: true Aug 7 23:49:19.929647679 (7h22m33.516703931s) [upload=7565]: Waiting for error group managing segments... Aug 7 23:49:19.929695078 (7h22m33.516751329s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:19.929701059 (7h22m33.5167573s) [upload=7565]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:19.929704956 (7h22m33.516761197s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:19.992636681 (7h22m33.579692922s) [upload=7565]: Done uploading segments... Aug 7 23:49:19.992654385 (7h22m33.579710625s) [upload=7565]: Upload finished. err: <nil> Aug 7 23:50:07.176468149 (7h23m20.76352438s) [upload=7576]: Starting upload Aug 7 23:50:07.176537798 (7h23m20.763594019s) [upload=7576]: Done starting upload Aug 7 23:50:07.176592070 (7h23m20.763648311s) [upload=7576]: Uploading segments... Aug 7 23:50:07.176626473 (7h23m20.763682705s) [upload=7576]: Got next segment. Inline: true Aug 7 23:50:07.176631002 (7h23m20.763687213s) [upload=7576]: Waiting for error group managing segments... Aug 7 23:50:07.176664084 (7h23m20.763720295s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:07.176668132 (7h23m20.763724343s) [upload=7576]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:07.176670586 (7h23m20.763726807s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:07.240851653 (7h23m20.827907864s) [upload=7576]: Done uploading segments... Aug 7 23:50:07.240866771 (7h23m20.827922992s) [upload=7576]: Upload finished. err: <nil> Aug 7 23:50:45.893754262 (7h23m59.480810504s) [upload=7591]: Starting upload Aug 7 23:50:45.893825847 (7h23m59.480882088s) [upload=7591]: Done starting upload Aug 7 23:50:45.893912749 (7h23m59.48096899s) [upload=7591]: Uploading segments... Aug 7 23:50:45.893955949 (7h23m59.4810122s) [upload=7591]: Got next segment. Inline: true Aug 7 23:50:45.893961970 (7h23m59.481018211s) [upload=7591]: Waiting for error group managing segments... Aug 7 23:50:45.894005482 (7h23m59.481061723s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:45.894011062 (7h23m59.481067304s) [upload=7591]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:45.894014729 (7h23m59.48107097s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:05.079621048 (7h24m18.666677259s) [upload=7591]: Done uploading segments... Aug 7 23:51:05.079636297 (7h24m18.666692508s) [upload=7591]: Upload finished. err: <nil> Aug 7 23:51:21.725622238 (7h24m35.312678469s) [upload=7603]: Starting upload Aug 7 23:51:21.725696627 (7h24m35.312752838s) [upload=7603]: Done starting upload Aug 7 23:51:21.725758231 (7h24m35.312814443s) [upload=7603]: Uploading segments... Aug 7 23:51:21.725796774 (7h24m35.312852985s) [upload=7603]: Got next segment. Inline: true Aug 7 23:51:21.725802845 (7h24m35.312859056s) [upload=7603]: Waiting for error group managing segments... Aug 7 23:51:21.725848871 (7h24m35.312905082s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:21.725855313 (7h24m35.312911523s) [upload=7603]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:21.725859100 (7h24m35.312915351s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:21.784505948 (7h24m35.371562158s) [upload=7603]: Done uploading segments... Aug 7 23:51:21.784521767 (7h24m35.371577978s) [upload=7603]: Upload finished. err: <nil> Aug 7 23:52:10.285208176 (7h25m23.872264397s) [upload=7615]: Starting upload Aug 7 23:52:10.285263228 (7h25m23.872319449s) [upload=7615]: Done starting upload Aug 7 23:52:10.285310727 (7h25m23.872366937s) [upload=7615]: Uploading segments... Aug 7 23:52:10.285339711 (7h25m23.872395932s) [upload=7615]: Got next segment. Inline: true Aug 7 23:52:10.285343438 (7h25m23.872399649s) [upload=7615]: Waiting for error group managing segments... Aug 7 23:52:10.285391158 (7h25m23.872447379s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:10.285395065 (7h25m23.872451276s) [upload=7615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:10.285397770 (7h25m23.872453991s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:10.400888570 (7h25m23.987944781s) [upload=7615]: Done uploading segments... Aug 7 23:52:10.400904530 (7h25m23.987960741s) [upload=7615]: Upload finished. err: <nil> Aug 7 23:53:11.795016020 (7h26m25.382072262s) [upload=7635]: Starting upload Aug 7 23:53:11.795108102 (7h26m25.382164354s) [upload=7635]: Done starting upload Aug 7 23:53:11.795175588 (7h26m25.382231829s) [upload=7635]: Uploading segments... Aug 7 23:53:11.795254645 (7h26m25.382310886s) [upload=7635]: Got next segment. Inline: true Aug 7 23:53:11.795262130 (7h26m25.38231837s) [upload=7635]: Waiting for error group managing segments... Aug 7 23:53:11.795319296 (7h26m25.382375538s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:11.795326170 (7h26m25.38238242s) [upload=7635]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:11.795329857 (7h26m25.382386097s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:11.782974743 (7h27m25.370030954s) [upload=7635]: Done uploading segments... Aug 7 23:54:11.782991474 (7h27m25.370047695s) [upload=7635]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:11.855031501 (7h27m25.442087722s) [upload=7650]: Starting upload Aug 7 23:54:11.855087627 (7h27m25.442143838s) [upload=7650]: Done starting upload Aug 7 23:54:11.855131608 (7h27m25.442187829s) [upload=7650]: Uploading segments... Aug 7 23:54:11.855163227 (7h27m25.442219448s) [upload=7650]: Got next segment. Inline: true Aug 7 23:54:11.855167005 (7h27m25.442223226s) [upload=7650]: Waiting for error group managing segments... Aug 7 23:54:11.855198905 (7h27m25.442255126s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:11.855202661 (7h27m25.442258882s) [upload=7650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:11.855205036 (7h27m25.442261247s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:11.844108363 (7h28m25.431164634s) [upload=7650]: Done uploading segments... Aug 7 23:55:11.844129542 (7h28m25.431185763s) [upload=7650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:11.924363315 (7h28m25.511419527s) [upload=7668]: Starting upload Aug 7 23:55:11.924429198 (7h28m25.511485409s) [upload=7668]: Done starting upload Aug 7 23:55:11.924484862 (7h28m25.511541083s) [upload=7668]: Uploading segments... Aug 7 23:55:11.924519887 (7h28m25.511576098s) [upload=7668]: Got next segment. Inline: true Aug 7 23:55:11.924523564 (7h28m25.511579785s) [upload=7668]: Waiting for error group managing segments... Aug 7 23:55:11.924554131 (7h28m25.511610342s) [upload=7668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:11.924558279 (7h28m25.51161449s) [upload=7668]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:11.924560643 (7h28m25.511616855s) [upload=7668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:12.020746910 (7h28m25.607803121s) [upload=7668]: Done uploading segments... Aug 7 23:55:12.020765404 (7h28m25.607821625s) [upload=7668]: Upload finished. err: <nil> Aug 7 23:55:57.071847012 (7h29m10.658903254s) [upload=7678]: Starting upload Aug 7 23:55:57.071985611 (7h29m10.659041851s) [upload=7678]: Done starting upload Aug 7 23:55:57.072082963 (7h29m10.659139204s) [upload=7678]: Uploading segments... Aug 7 23:55:57.072231269 (7h29m10.65928751s) [upload=7678]: Got next segment. Inline: true Aug 7 23:55:57.072240506 (7h29m10.659296748s) [upload=7678]: Waiting for error group managing segments... Aug 7 23:55:57.072359458 (7h29m10.6594157s) [upload=7678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:57.072367143 (7h29m10.659423384s) [upload=7678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:57.072371320 (7h29m10.659427562s) [upload=7678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:57.049680168 (7h30m10.636736379s) [upload=7678]: Done uploading segments... Aug 7 23:56:57.049696499 (7h30m10.636752709s) [upload=7678]: Upload finished. err: metaclient: context deadline exceeded Duration:* 214.99377ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 163.584777ms FAIL (2025-08-07 23:57:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 163.605787ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 10 out of 60 is failed (threshold=3) [.XXXXXX.X...............................................XX.X] duration: 1m0.000752295s FAIL (2025-08-07 23:56:57 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:39.988158721 (7h20m53.575214932s) [upload=7536]: Done uploading segments... Aug 7 23:47:39.988176785 (7h20m53.575232995s) [upload=7536]: Upload finished. err: <nil> Aug 7 23:48:17.585904864 (7h21m31.172961076s) [upload=7548]: Starting upload Aug 7 23:48:17.585990855 (7h21m31.173047066s) [upload=7548]: Done starting upload Aug 7 23:48:17.586065835 (7h21m31.173122035s) [upload=7548]: Uploading segments... Aug 7 23:48:17.586102964 (7h21m31.173159205s) [upload=7548]: Got next segment. Inline: true Aug 7 23:48:17.586107262 (7h21m31.173163473s) [upload=7548]: Waiting for error group managing segments... Aug 7 23:48:17.586144451 (7h21m31.173200662s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:17.586148969 (7h21m31.17320518s) [upload=7548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:17.586151854 (7h21m31.173208066s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:17.642355873 (7h21m31.229412094s) [upload=7548]: Done uploading segments... Aug 7 23:48:17.642371582 (7h21m31.229427794s) [upload=7548]: Upload finished. err: <nil> Aug 7 23:49:19.929340015 (7h22m33.516396255s) [upload=7565]: Starting upload Aug 7 23:49:19.929497669 (7h22m33.51655391s) [upload=7565]: Done starting upload Aug 7 23:49:19.929593468 (7h22m33.516649709s) [upload=7565]: Uploading segments... Aug 7 23:49:19.929640947 (7h22m33.516697188s) [upload=7565]: Got next segment. Inline: true Aug 7 23:49:19.929647679 (7h22m33.516703931s) [upload=7565]: Waiting for error group managing segments... Aug 7 23:49:19.929695078 (7h22m33.516751329s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:19.929701059 (7h22m33.5167573s) [upload=7565]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:19.929704956 (7h22m33.516761197s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:19.992636681 (7h22m33.579692922s) [upload=7565]: Done uploading segments... Aug 7 23:49:19.992654385 (7h22m33.579710625s) [upload=7565]: Upload finished. err: <nil> Aug 7 23:50:07.176468149 (7h23m20.76352438s) [upload=7576]: Starting upload Aug 7 23:50:07.176537798 (7h23m20.763594019s) [upload=7576]: Done starting upload Aug 7 23:50:07.176592070 (7h23m20.763648311s) [upload=7576]: Uploading segments... Aug 7 23:50:07.176626473 (7h23m20.763682705s) [upload=7576]: Got next segment. Inline: true Aug 7 23:50:07.176631002 (7h23m20.763687213s) [upload=7576]: Waiting for error group managing segments... Aug 7 23:50:07.176664084 (7h23m20.763720295s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:07.176668132 (7h23m20.763724343s) [upload=7576]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:07.176670586 (7h23m20.763726807s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:07.240851653 (7h23m20.827907864s) [upload=7576]: Done uploading segments... Aug 7 23:50:07.240866771 (7h23m20.827922992s) [upload=7576]: Upload finished. err: <nil> Aug 7 23:50:45.893754262 (7h23m59.480810504s) [upload=7591]: Starting upload Aug 7 23:50:45.893825847 (7h23m59.480882088s) [upload=7591]: Done starting upload Aug 7 23:50:45.893912749 (7h23m59.48096899s) [upload=7591]: Uploading segments... Aug 7 23:50:45.893955949 (7h23m59.4810122s) [upload=7591]: Got next segment. Inline: true Aug 7 23:50:45.893961970 (7h23m59.481018211s) [upload=7591]: Waiting for error group managing segments... Aug 7 23:50:45.894005482 (7h23m59.481061723s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:45.894011062 (7h23m59.481067304s) [upload=7591]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:45.894014729 (7h23m59.48107097s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:05.079621048 (7h24m18.666677259s) [upload=7591]: Done uploading segments... Aug 7 23:51:05.079636297 (7h24m18.666692508s) [upload=7591]: Upload finished. err: <nil> Aug 7 23:51:21.725622238 (7h24m35.312678469s) [upload=7603]: Starting upload Aug 7 23:51:21.725696627 (7h24m35.312752838s) [upload=7603]: Done starting upload Aug 7 23:51:21.725758231 (7h24m35.312814443s) [upload=7603]: Uploading segments... Aug 7 23:51:21.725796774 (7h24m35.312852985s) [upload=7603]: Got next segment. Inline: true Aug 7 23:51:21.725802845 (7h24m35.312859056s) [upload=7603]: Waiting for error group managing segments... Aug 7 23:51:21.725848871 (7h24m35.312905082s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:21.725855313 (7h24m35.312911523s) [upload=7603]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:21.725859100 (7h24m35.312915351s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:21.784505948 (7h24m35.371562158s) [upload=7603]: Done uploading segments... Aug 7 23:51:21.784521767 (7h24m35.371577978s) [upload=7603]: Upload finished. err: <nil> Aug 7 23:52:10.285208176 (7h25m23.872264397s) [upload=7615]: Starting upload Aug 7 23:52:10.285263228 (7h25m23.872319449s) [upload=7615]: Done starting upload Aug 7 23:52:10.285310727 (7h25m23.872366937s) [upload=7615]: Uploading segments... Aug 7 23:52:10.285339711 (7h25m23.872395932s) [upload=7615]: Got next segment. Inline: true Aug 7 23:52:10.285343438 (7h25m23.872399649s) [upload=7615]: Waiting for error group managing segments... Aug 7 23:52:10.285391158 (7h25m23.872447379s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:10.285395065 (7h25m23.872451276s) [upload=7615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:10.285397770 (7h25m23.872453991s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:10.400888570 (7h25m23.987944781s) [upload=7615]: Done uploading segments... Aug 7 23:52:10.400904530 (7h25m23.987960741s) [upload=7615]: Upload finished. err: <nil> Aug 7 23:53:11.795016020 (7h26m25.382072262s) [upload=7635]: Starting upload Aug 7 23:53:11.795108102 (7h26m25.382164354s) [upload=7635]: Done starting upload Aug 7 23:53:11.795175588 (7h26m25.382231829s) [upload=7635]: Uploading segments... Aug 7 23:53:11.795254645 (7h26m25.382310886s) [upload=7635]: Got next segment. Inline: true Aug 7 23:53:11.795262130 (7h26m25.38231837s) [upload=7635]: Waiting for error group managing segments... Aug 7 23:53:11.795319296 (7h26m25.382375538s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:11.795326170 (7h26m25.38238242s) [upload=7635]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:11.795329857 (7h26m25.382386097s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:11.782974743 (7h27m25.370030954s) [upload=7635]: Done uploading segments... Aug 7 23:54:11.782991474 (7h27m25.370047695s) [upload=7635]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:11.855031501 (7h27m25.442087722s) [upload=7650]: Starting upload Aug 7 23:54:11.855087627 (7h27m25.442143838s) [upload=7650]: Done starting upload Aug 7 23:54:11.855131608 (7h27m25.442187829s) [upload=7650]: Uploading segments... Aug 7 23:54:11.855163227 (7h27m25.442219448s) [upload=7650]: Got next segment. Inline: true Aug 7 23:54:11.855167005 (7h27m25.442223226s) [upload=7650]: Waiting for error group managing segments... Aug 7 23:54:11.855198905 (7h27m25.442255126s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:11.855202661 (7h27m25.442258882s) [upload=7650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:11.855205036 (7h27m25.442261247s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:11.844108363 (7h28m25.431164634s) [upload=7650]: Done uploading segments... Aug 7 23:55:11.844129542 (7h28m25.431185763s) [upload=7650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:11.924363315 (7h28m25.511419527s) [upload=7668]: Starting upload Aug 7 23:55:11.924429198 (7h28m25.511485409s) [upload=7668]: Done starting upload Aug 7 23:55:11.924484862 (7h28m25.511541083s) [upload=7668]: Uploading segments... Aug 7 23:55:11.924519887 (7h28m25.511576098s) [upload=7668]: Got next segment. Inline: true Aug 7 23:55:11.924523564 (7h28m25.511579785s) [upload=7668]: Waiting for error group managing segments... Aug 7 23:55:11.924554131 (7h28m25.511610342s) [upload=7668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:11.924558279 (7h28m25.51161449s) [upload=7668]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:11.924560643 (7h28m25.511616855s) [upload=7668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:12.020746910 (7h28m25.607803121s) [upload=7668]: Done uploading segments... Aug 7 23:55:12.020765404 (7h28m25.607821625s) [upload=7668]: Upload finished. err: <nil> Aug 7 23:55:57.071847012 (7h29m10.658903254s) [upload=7678]: Starting upload Aug 7 23:55:57.071985611 (7h29m10.659041851s) [upload=7678]: Done starting upload Aug 7 23:55:57.072082963 (7h29m10.659139204s) [upload=7678]: Uploading segments... Aug 7 23:55:57.072231269 (7h29m10.65928751s) [upload=7678]: Got next segment. Inline: true Aug 7 23:55:57.072240506 (7h29m10.659296748s) [upload=7678]: Waiting for error group managing segments... Aug 7 23:55:57.072359458 (7h29m10.6594157s) [upload=7678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:57.072367143 (7h29m10.659423384s) [upload=7678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:57.072371320 (7h29m10.659427562s) [upload=7678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:57.049680168 (7h30m10.636736379s) [upload=7678]: Done uploading segments... Aug 7 23:56:57.049696499 (7h30m10.636752709s) [upload=7678]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.000765641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 667.511227ms FAIL (2025-08-07 23:56:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== 1345s) [upload=7262]: Got next segment. Inline: false Aug 7 23:31:37.583906637 (7h4m51.170962878s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:31:37.583953414 (7h4m51.171009656s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:31:37.613963041 (7h4m51.201019262s) [seg_pos={0 0}, upload=7262]: Begin upload segment... Aug 7 23:31:37.614008966 (7h4m51.201065176s) [seg_pos={0 0}, upload=7262]: Done begin upload segment. Aug 7 23:31:37.614029103 (7h4m51.201085324s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614108311 (7h4m51.201164542s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614344612 (7h4m51.201400833s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614634754 (7h4m51.201690975s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614919926 (7h4m51.201976137s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614943150 (7h4m51.201999391s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615146719 (7h4m51.20220294s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615454163 (7h4m51.202510384s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615514677 (7h4m51.202570918s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615688160 (7h4m51.202744381s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615932236 (7h4m51.202988457s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616184166 (7h4m51.203240378s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616521716 (7h4m51.203577937s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616643614 (7h4m51.203699855s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621231117 (7h4m51.208287338s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621561163 (7h4m51.208617374s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621861323 (7h4m51.208917534s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.622156223 (7h4m51.209212444s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.726040884 (7h4m51.313097094s) [upload=7262]: Next returned nil segment Aug 7 23:31:37.726066772 (7h4m51.313122983s) [upload=7262]: Waiting for error group managing segments... Aug 7 23:31:37.736232517 (7h4m51.323288738s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.737656014 (7h4m51.324712245s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.739754932 (7h4m51.326811153s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.740628113 (7h4m51.327684344s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741712657 (7h4m51.328768867s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741742653 (7h4m51.328798883s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.748242274 (7h4m51.335298505s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.750434577 (7h4m51.337490798s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.758962364 (7h4m51.346018586s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.769693706 (7h4m51.356749917s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.800540734 (7h4m51.387596975s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.924012655 (7h4m51.511068866s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.059695219 (7h4m51.64675144s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102515917 (7h4m51.689572138s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102541606 (7h4m51.689597827s) [seg_pos={0 0}, upload=7262]: Segment reached optimal threshold of 14 pieces. Aug 7 23:31:38.102658574 (7h4m51.689714795s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF Aug 7 23:31:38.102748111 (7h4m51.689804332s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE): protocol: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection Aug 7 23:31:38.102832158 (7h4m51.689888399s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GcyWwKkfavsCsmtq55U72n6XeZKqQiGGYHY9D6UwST8sC14Ac): protocol: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection Aug 7 23:31:38.102967200 (7h4m51.690023421s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12syasUPsyYf4UstQ6cFAr6qamze3vCyAm9dnobk5e83w4bTz69): protocol: context canceled; context canceled Aug 7 23:31:38.102995132 (7h4m51.690051374s) [seg_pos={0 0}, upload=7262]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:31:38.103097553 (7h4m51.690153775s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:31:38.103102412 (7h4m51.690158633s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:31:38.203633212 (7h4m51.790689433s) [upload=7262]: Done uploading segments... Aug 7 23:31:38.203649102 (7h4m51.790705323s) [upload=7262]: Upload finished. err: <nil> Duration: 667.540172ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 14 out of 60 is failed (threshold=3) [XXXX.X..X..X......................................XXX.X.XXX.] duration: 204.572178ms FAIL (2025-08-07 23:56:54 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 204.588859ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 14 out of 60 is failed (threshold=3) [.XXXX.X..X..X......................................XXX.X.XXX] duration: 30.000568053s FAIL (2025-08-07 23:56:53 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000590214s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [..X.XXXXX..X................................................] duration: 185.921716ms FAIL (2025-08-07 23:56:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 185.94485ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 15 out of 60 is failed (threshold=3) [XXXXX.XX............................................XXXXXXXX] duration: 30.001830394s FAIL (2025-08-07 23:56:45 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Aug 7 23:48:58.840462661 (7h22m12.427518891s) [upload=7557]: Upload finished. err: <nil> Aug 7 23:50:01.971851413 (7h23m15.558907624s) [upload=7574]: Starting upload Aug 7 23:50:01.971890696 (7h23m15.558946907s) [upload=7574]: Done starting upload Aug 7 23:50:01.971977749 (7h23m15.559033961s) [upload=7574]: Uploading segments... Aug 7 23:50:01.972022943 (7h23m15.559079154s) [upload=7574]: Got next segment. Inline: true Aug 7 23:50:01.972026850 (7h23m15.559083062s) [upload=7574]: Waiting for error group managing segments... Aug 7 23:50:01.972060975 (7h23m15.559117185s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:01.972064511 (7h23m15.559120712s) [upload=7574]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:01.972066685 (7h23m15.559122896s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:02.063709030 (7h23m15.650765241s) [upload=7574]: Done uploading segments... Aug 7 23:50:02.063723547 (7h23m15.650779768s) [upload=7574]: Upload finished. err: <nil> Aug 7 23:50:41.160166556 (7h23m54.747222777s) [upload=7588]: Starting upload Aug 7 23:50:41.160220555 (7h23m54.747276777s) [upload=7588]: Done starting upload Aug 7 23:50:41.160264659 (7h23m54.74732088s) [upload=7588]: Uploading segments... Aug 7 23:50:41.160295956 (7h23m54.747352177s) [upload=7588]: Got next segment. Inline: true Aug 7 23:50:41.160299753 (7h23m54.747355964s) [upload=7588]: Waiting for error group managing segments... Aug 7 23:50:41.160331633 (7h23m54.747387844s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:41.160335861 (7h23m54.747392072s) [upload=7588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:41.160338396 (7h23m54.747394617s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:41.222859735 (7h23m54.809915976s) [upload=7588]: Done uploading segments... Aug 7 23:50:41.222879913 (7h23m54.809936154s) [upload=7588]: Upload finished. err: <nil> Aug 7 23:51:17.131690986 (7h24m30.718747197s) [upload=7599]: Starting upload Aug 7 23:51:17.131738926 (7h24m30.718795147s) [upload=7599]: Done starting upload Aug 7 23:51:17.131792947 (7h24m30.718849178s) [upload=7599]: Uploading segments... Aug 7 23:51:17.131835896 (7h24m30.718892117s) [upload=7599]: Got next segment. Inline: true Aug 7 23:51:17.131842027 (7h24m30.718898239s) [upload=7599]: Waiting for error group managing segments... Aug 7 23:51:17.131883385 (7h24m30.718939627s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:17.131889146 (7h24m30.718945388s) [upload=7599]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:17.132006205 (7h24m30.719062435s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:38.787433326 (7h24m52.374489567s) [upload=7599]: Done uploading segments... Aug 7 23:51:38.787452212 (7h24m52.374508453s) [upload=7599]: Upload finished. err: <nil> Aug 7 23:52:00.256204647 (7h25m13.843260858s) [upload=7610]: Starting upload Aug 7 23:52:00.256263547 (7h25m13.843319758s) [upload=7610]: Done starting upload Aug 7 23:52:00.256316195 (7h25m13.843372406s) [upload=7610]: Uploading segments... Aug 7 23:52:00.256345671 (7h25m13.843401881s) [upload=7610]: Got next segment. Inline: true Aug 7 23:52:00.256349758 (7h25m13.843405969s) [upload=7610]: Waiting for error group managing segments... Aug 7 23:52:00.256399992 (7h25m13.843456203s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:00.256404780 (7h25m13.843460991s) [upload=7610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:00.256407806 (7h25m13.843464017s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:30.237600164 (7h25m43.824656416s) [upload=7610]: Done uploading segments... Aug 7 23:52:30.237630862 (7h25m43.824687113s) [upload=7610]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:53.146251246 (7h26m6.733307456s) [upload=7626]: Starting upload Aug 7 23:52:53.146307339 (7h26m6.733363561s) [upload=7626]: Done starting upload Aug 7 23:52:53.146350962 (7h26m6.733407173s) [upload=7626]: Uploading segments... Aug 7 23:52:53.146411344 (7h26m6.733467565s) [upload=7626]: Got next segment. Inline: true Aug 7 23:52:53.146415271 (7h26m6.733471493s) [upload=7626]: Waiting for error group managing segments... Aug 7 23:52:53.146449966 (7h26m6.733506177s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:53.146454174 (7h26m6.733510395s) [upload=7626]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:53.146456709 (7h26m6.73351293s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.132536521 (7h26m36.719592742s) [upload=7626]: Done uploading segments... Aug 7 23:53:23.132560907 (7h26m36.719617128s) [upload=7626]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:28.217132518 (7h26m41.804188759s) [upload=7638]: Starting upload Aug 7 23:53:28.217216185 (7h26m41.804272426s) [upload=7638]: Done starting upload Aug 7 23:53:28.217288009 (7h26m41.80434426s) [upload=7638]: Uploading segments... Aug 7 23:53:28.217336129 (7h26m41.80439237s) [upload=7638]: Got next segment. Inline: true Aug 7 23:53:28.217341890 (7h26m41.804398131s) [upload=7638]: Waiting for error group managing segments... Aug 7 23:53:28.217421438 (7h26m41.804477679s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:28.217427650 (7h26m41.804483901s) [upload=7638]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:28.217431617 (7h26m41.804487859s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:58.204830989 (7h27m11.79188721s) [upload=7638]: Done uploading segments... Aug 7 23:53:58.204849294 (7h27m11.791905505s) [upload=7638]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:01.364416813 (7h27m14.951473054s) [upload=7648]: Starting upload Aug 7 23:54:01.364544481 (7h27m14.951600732s) [upload=7648]: Done starting upload Aug 7 23:54:01.364639518 (7h27m14.95169577s) [upload=7648]: Uploading segments... Aug 7 23:54:01.364703858 (7h27m14.951760109s) [upload=7648]: Got next segment. Inline: true Aug 7 23:54:01.364713517 (7h27m14.951769757s) [upload=7648]: Waiting for error group managing segments... Aug 7 23:54:01.364781374 (7h27m14.951837625s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:01.364791231 (7h27m14.951847472s) [upload=7648]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:01.364797502 (7h27m14.951853754s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:31.351186626 (7h27m44.938242847s) [upload=7648]: Done uploading segments... Aug 7 23:54:31.351241118 (7h27m44.938297339s) [upload=7648]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:31.416140839 (7h27m45.003197049s) [upload=7656]: Starting upload Aug 7 23:54:31.416189309 (7h27m45.00324553s) [upload=7656]: Done starting upload Aug 7 23:54:31.416235946 (7h27m45.003292157s) [upload=7656]: Uploading segments... Aug 7 23:54:31.416277384 (7h27m45.003333595s) [upload=7656]: Got next segment. Inline: true Aug 7 23:54:31.416297601 (7h27m45.003353813s) [upload=7656]: Waiting for error group managing segments... Aug 7 23:54:31.416336454 (7h27m45.003392665s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:31.416340652 (7h27m45.003396873s) [upload=7656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:31.416342965 (7h27m45.003399177s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:01.405081824 (7h28m14.992138035s) [upload=7656]: Done uploading segments... Aug 7 23:55:01.405101451 (7h28m14.992157672s) [upload=7656]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:45.764200449 (7h28m59.35125669s) [upload=7673]: Starting upload Aug 7 23:55:45.764300708 (7h28m59.351356948s) [upload=7673]: Done starting upload Aug 7 23:55:45.764398370 (7h28m59.351454611s) [upload=7673]: Uploading segments... Aug 7 23:55:45.764515538 (7h28m59.35157178s) [upload=7673]: Got next segment. Inline: true Aug 7 23:55:45.764521980 (7h28m59.351578222s) [upload=7673]: Waiting for error group managing segments... Aug 7 23:55:45.764570060 (7h28m59.3516263s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:45.764575871 (7h28m59.351632111s) [upload=7673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:45.764579898 (7h28m59.351636149s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:15.747212127 (7h29m29.334268369s) [upload=7673]: Done uploading segments... Aug 7 23:56:15.747307315 (7h29m29.334363566s) [upload=7673]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:15.823083783 (7h29m29.410140035s) [upload=7682]: Starting upload Aug 7 23:56:15.823169003 (7h29m29.410225243s) [upload=7682]: Done starting upload Aug 7 23:56:15.823245125 (7h29m29.410301367s) [upload=7682]: Uploading segments... Aug 7 23:56:15.823289758 (7h29m29.410346009s) [upload=7682]: Got next segment. Inline: true Aug 7 23:56:15.823295359 (7h29m29.410351599s) [upload=7682]: Waiting for error group managing segments... Aug 7 23:56:15.823343018 (7h29m29.410399269s) [upload=7682]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:15.823348678 (7h29m29.41040492s) [upload=7682]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:15.823352376 (7h29m29.410408616s) [upload=7682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:45.810149218 (7h29m59.397205439s) [upload=7682]: Done uploading segments... Aug 7 23:56:45.810167081 (7h29m59.397223322s) [upload=7682]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001844761s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX.X....X.X..........................................X.....] duration: 562.269088ms FAIL (2025-08-07 23:56:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 562.28632ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XXXXXX.X.X...........................................X.X..X] duration: 30.000364608s FAIL (2025-08-07 23:56:21 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000385067s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 13 out of 60 is failed (threshold=3) [..XXXX.X..X..X......................................XXX.X.XX] duration: 30.000410383s FAIL (2025-08-07 23:56:16 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000432884s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 14 out of 60 is failed (threshold=3) [.XXXXX.XX............................................XXXXXXX] duration: 30.00150159s FAIL (2025-08-07 23:56:15 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad=7545]: Done uploading segments... Aug 7 23:48:09.951286926 (7h21m23.538343148s) [upload=7545]: Upload finished. err: <nil> Aug 7 23:48:58.772502770 (7h22m12.359559011s) [upload=7557]: Starting upload Aug 7 23:48:58.772574805 (7h22m12.359631046s) [upload=7557]: Done starting upload Aug 7 23:48:58.772645527 (7h22m12.359701767s) [upload=7557]: Uploading segments... Aug 7 23:48:58.772695300 (7h22m12.359751541s) [upload=7557]: Got next segment. Inline: true Aug 7 23:48:58.772701031 (7h22m12.359757271s) [upload=7557]: Waiting for error group managing segments... Aug 7 23:48:58.772770249 (7h22m12.359826501s) [upload=7557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:58.772780228 (7h22m12.359836469s) [upload=7557]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:58.772787010 (7h22m12.359843252s) [upload=7557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:58.840436652 (7h22m12.427492893s) [upload=7557]: Done uploading segments... Aug 7 23:48:58.840462661 (7h22m12.427518891s) [upload=7557]: Upload finished. err: <nil> Aug 7 23:50:01.971851413 (7h23m15.558907624s) [upload=7574]: Starting upload Aug 7 23:50:01.971890696 (7h23m15.558946907s) [upload=7574]: Done starting upload Aug 7 23:50:01.971977749 (7h23m15.559033961s) [upload=7574]: Uploading segments... Aug 7 23:50:01.972022943 (7h23m15.559079154s) [upload=7574]: Got next segment. Inline: true Aug 7 23:50:01.972026850 (7h23m15.559083062s) [upload=7574]: Waiting for error group managing segments... Aug 7 23:50:01.972060975 (7h23m15.559117185s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:01.972064511 (7h23m15.559120712s) [upload=7574]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:01.972066685 (7h23m15.559122896s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:02.063709030 (7h23m15.650765241s) [upload=7574]: Done uploading segments... Aug 7 23:50:02.063723547 (7h23m15.650779768s) [upload=7574]: Upload finished. err: <nil> Aug 7 23:50:41.160166556 (7h23m54.747222777s) [upload=7588]: Starting upload Aug 7 23:50:41.160220555 (7h23m54.747276777s) [upload=7588]: Done starting upload Aug 7 23:50:41.160264659 (7h23m54.74732088s) [upload=7588]: Uploading segments... Aug 7 23:50:41.160295956 (7h23m54.747352177s) [upload=7588]: Got next segment. Inline: true Aug 7 23:50:41.160299753 (7h23m54.747355964s) [upload=7588]: Waiting for error group managing segments... Aug 7 23:50:41.160331633 (7h23m54.747387844s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:41.160335861 (7h23m54.747392072s) [upload=7588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:41.160338396 (7h23m54.747394617s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:41.222859735 (7h23m54.809915976s) [upload=7588]: Done uploading segments... Aug 7 23:50:41.222879913 (7h23m54.809936154s) [upload=7588]: Upload finished. err: <nil> Aug 7 23:51:17.131690986 (7h24m30.718747197s) [upload=7599]: Starting upload Aug 7 23:51:17.131738926 (7h24m30.718795147s) [upload=7599]: Done starting upload Aug 7 23:51:17.131792947 (7h24m30.718849178s) [upload=7599]: Uploading segments... Aug 7 23:51:17.131835896 (7h24m30.718892117s) [upload=7599]: Got next segment. Inline: true Aug 7 23:51:17.131842027 (7h24m30.718898239s) [upload=7599]: Waiting for error group managing segments... Aug 7 23:51:17.131883385 (7h24m30.718939627s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:17.131889146 (7h24m30.718945388s) [upload=7599]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:17.132006205 (7h24m30.719062435s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:38.787433326 (7h24m52.374489567s) [upload=7599]: Done uploading segments... Aug 7 23:51:38.787452212 (7h24m52.374508453s) [upload=7599]: Upload finished. err: <nil> Aug 7 23:52:00.256204647 (7h25m13.843260858s) [upload=7610]: Starting upload Aug 7 23:52:00.256263547 (7h25m13.843319758s) [upload=7610]: Done starting upload Aug 7 23:52:00.256316195 (7h25m13.843372406s) [upload=7610]: Uploading segments... Aug 7 23:52:00.256345671 (7h25m13.843401881s) [upload=7610]: Got next segment. Inline: true Aug 7 23:52:00.256349758 (7h25m13.843405969s) [upload=7610]: Waiting for error group managing segments... Aug 7 23:52:00.256399992 (7h25m13.843456203s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:00.256404780 (7h25m13.843460991s) [upload=7610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:00.256407806 (7h25m13.843464017s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:30.237600164 (7h25m43.824656416s) [upload=7610]: Done uploading segments... Aug 7 23:52:30.237630862 (7h25m43.824687113s) [upload=7610]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:53.146251246 (7h26m6.733307456s) [upload=7626]: Starting upload Aug 7 23:52:53.146307339 (7h26m6.733363561s) [upload=7626]: Done starting upload Aug 7 23:52:53.146350962 (7h26m6.733407173s) [upload=7626]: Uploading segments... Aug 7 23:52:53.146411344 (7h26m6.733467565s) [upload=7626]: Got next segment. Inline: true Aug 7 23:52:53.146415271 (7h26m6.733471493s) [upload=7626]: Waiting for error group managing segments... Aug 7 23:52:53.146449966 (7h26m6.733506177s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:53.146454174 (7h26m6.733510395s) [upload=7626]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:53.146456709 (7h26m6.73351293s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.132536521 (7h26m36.719592742s) [upload=7626]: Done uploading segments... Aug 7 23:53:23.132560907 (7h26m36.719617128s) [upload=7626]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:28.217132518 (7h26m41.804188759s) [upload=7638]: Starting upload Aug 7 23:53:28.217216185 (7h26m41.804272426s) [upload=7638]: Done starting upload Aug 7 23:53:28.217288009 (7h26m41.80434426s) [upload=7638]: Uploading segments... Aug 7 23:53:28.217336129 (7h26m41.80439237s) [upload=7638]: Got next segment. Inline: true Aug 7 23:53:28.217341890 (7h26m41.804398131s) [upload=7638]: Waiting for error group managing segments... Aug 7 23:53:28.217421438 (7h26m41.804477679s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:28.217427650 (7h26m41.804483901s) [upload=7638]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:28.217431617 (7h26m41.804487859s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:58.204830989 (7h27m11.79188721s) [upload=7638]: Done uploading segments... Aug 7 23:53:58.204849294 (7h27m11.791905505s) [upload=7638]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:01.364416813 (7h27m14.951473054s) [upload=7648]: Starting upload Aug 7 23:54:01.364544481 (7h27m14.951600732s) [upload=7648]: Done starting upload Aug 7 23:54:01.364639518 (7h27m14.95169577s) [upload=7648]: Uploading segments... Aug 7 23:54:01.364703858 (7h27m14.951760109s) [upload=7648]: Got next segment. Inline: true Aug 7 23:54:01.364713517 (7h27m14.951769757s) [upload=7648]: Waiting for error group managing segments... Aug 7 23:54:01.364781374 (7h27m14.951837625s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:01.364791231 (7h27m14.951847472s) [upload=7648]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:01.364797502 (7h27m14.951853754s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:31.351186626 (7h27m44.938242847s) [upload=7648]: Done uploading segments... Aug 7 23:54:31.351241118 (7h27m44.938297339s) [upload=7648]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:31.416140839 (7h27m45.003197049s) [upload=7656]: Starting upload Aug 7 23:54:31.416189309 (7h27m45.00324553s) [upload=7656]: Done starting upload Aug 7 23:54:31.416235946 (7h27m45.003292157s) [upload=7656]: Uploading segments... Aug 7 23:54:31.416277384 (7h27m45.003333595s) [upload=7656]: Got next segment. Inline: true Aug 7 23:54:31.416297601 (7h27m45.003353813s) [upload=7656]: Waiting for error group managing segments... Aug 7 23:54:31.416336454 (7h27m45.003392665s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:31.416340652 (7h27m45.003396873s) [upload=7656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:31.416342965 (7h27m45.003399177s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:01.405081824 (7h28m14.992138035s) [upload=7656]: Done uploading segments... Aug 7 23:55:01.405101451 (7h28m14.992157672s) [upload=7656]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:45.764200449 (7h28m59.35125669s) [upload=7673]: Starting upload Aug 7 23:55:45.764300708 (7h28m59.351356948s) [upload=7673]: Done starting upload Aug 7 23:55:45.764398370 (7h28m59.351454611s) [upload=7673]: Uploading segments... Aug 7 23:55:45.764515538 (7h28m59.35157178s) [upload=7673]: Got next segment. Inline: true Aug 7 23:55:45.764521980 (7h28m59.351578222s) [upload=7673]: Waiting for error group managing segments... Aug 7 23:55:45.764570060 (7h28m59.3516263s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:45.764575871 (7h28m59.351632111s) [upload=7673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:45.764579898 (7h28m59.351636149s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:15.747212127 (7h29m29.334268369s) [upload=7673]: Done uploading segments... Aug 7 23:56:15.747307315 (7h29m29.334363566s) [upload=7673]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001527678s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [....XXXXX..X............................................XXXX] duration: 1m0.000632978s FAIL (2025-08-07 23:56:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000660942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 159.510726ms FAIL (2025-08-07 23:56:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 159.543086ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [...X.XXXXX..X...............................................] duration: 172.546725ms FAIL (2025-08-07 23:55:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 172.568827ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.XX.X....X.X..........................................X....] duration: 416.867638ms FAIL (2025-08-07 23:55:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 416.886053ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)