Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................X....XXX.X.......] duration: 19.853076453s FAIL (2025-08-07 23:27:04 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ite tcp 5.161.89.252:59232->104.36.82.13:4187: use of closed network connection Aug 7 23:17:27.468792451 (6h50m41.055848663s) [noise=true , address=104.36.82.15:4257 , piece_id=YSUUNUNXXR4XZZRQBZVTYSP7WZNQVDHHS7KVVRKIAY5AQI3EMGCQ, seg_pos={0 0}, upload=7021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: read tcp 5.161.89.252:16228->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:27.468805356 (6h50m41.055861567s) [seg_pos={0 0}, upload=7021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:17:27.468898489 (6h50m41.055954701s) [upload=7021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:27.468902838 (6h50m41.055959049s) [upload=7021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:27.512149772 (6h50m41.099205993s) [upload=7021]: Done uploading segments... Aug 7 23:17:27.512163628 (6h50m41.099219849s) [upload=7021]: Upload finished. err: <nil> Aug 7 23:17:51.618188257 (6h51m5.205244499s) [upload=7028]: Starting upload Aug 7 23:17:51.618315174 (6h51m5.205371416s) [upload=7028]: Done starting upload Aug 7 23:17:51.618432584 (6h51m5.205488824s) [upload=7028]: Uploading segments... Aug 7 23:17:51.618706434 (6h51m5.205762675s) [upload=7028]: Got next segment. Inline: false Aug 7 23:17:51.618753392 (6h51m5.205809634s) [upload=7028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:51.618759304 (6h51m5.205815554s) [upload=7028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:02.217185080 (6h51m15.804241291s) [seg_pos={0 0}, upload=7028]: Begin upload segment... Aug 7 23:18:02.217213693 (6h51m15.804269904s) [seg_pos={0 0}, upload=7028]: Done begin upload segment. Aug 7 23:18:02.217258727 (6h51m15.804314948s) [noise=true , address=45.91.116.7:4009 , piece_id=VMY27LD3MBT4O65CXJJEYBIP7QVB543GSJOTQLYBELK4G7DOBQRA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217278985 (6h51m15.804335206s) [noise=true , address=77.90.31.12:4592 , piece_id=X4RIDUL5RUENCF4ZVRNIBWIUAHGQYW6XLBZWDMOUYGEIWQL5ZH7A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217653263 (6h51m15.804709484s) [noise=true , address=104.36.82.11:4115 , piece_id=MIW4TCCVX7O3KTEBU2RQRHD4GKPBYQOBTDKXGLH3B3NYCKTBDQ3A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217754012 (6h51m15.804810233s) [noise=true , address=77.90.31.17:4772 , piece_id=A7XUBHQKTJRVAMBQAO24KOU4GFGENHMFY6DBETNJDZ7J5YR7VO6Q, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217977880 (6h51m15.805034101s) [noise=true , address=104.36.82.14:4222 , piece_id=BJCIBUJTQDIS64F2XTL6XYNMICZJUXBP2NUA3UXOCORROZWONL4Q, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218105467 (6h51m15.805161678s) [noise=true , address=77.90.31.16:4735 , piece_id=QBT3GKFTCJBCONV4W4E3V74IQBXZO2R4M3SQBJBGRPEXMN4L4V2A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218314438 (6h51m15.805370659s) [noise=true , address=45.91.116.10:4082 , piece_id=GCFQIVXLPEIWQXT4A5Y6Z4N2E54HKSLGB2QDXR62SYQ4AISIXIMA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218477351 (6h51m15.805533563s) [noise=true , address=45.91.116.6:4974 , piece_id=MQWUSYBCOBJPCBLKUWG5R2YGX2UXHJAYTFXLU5G45I7ADEEJOSIA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218633864 (6h51m15.805690076s) [noise=true , address=77.90.31.21:4877 , piece_id=C7HY6GFRIWXX43OOGKBM3EXJVIMPUQP36KHHSVSGOVFJYBBX4IBA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218759268 (6h51m15.80581548s) [noise=true , address=45.91.116.14:4190 , piece_id=VN7ZRT4NTNUNQNKN2OCNILBFMN2QNGZOOY6B6EFDPET3TIB42A7A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218945636 (6h51m15.806001857s) [noise=true , address=77.90.31.11:4553 , piece_id=SJUK6H5G242A3QOXRARMJGQP6IGRQ2WJOUHBK24BMT2M3DYKGKAQ, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219088874 (6h51m15.806145095s) [noise=true , address=77.90.31.19:4842 , piece_id=2K54EHW7ZB22YIUFSMKZBKD66T4NLCPLSAPR344HPTF7WPX537JA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219280120 (6h51m15.806336341s) [noise=true , address=104.36.82.3:4767 , piece_id=D6EYZVFHGH7J3ABUMJBQPTJUDNA6GPMONDLQY7WL6NTVLYEEZM4Q, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219439768 (6h51m15.806495989s) [noise=true , address=77.90.31.14:4662 , piece_id=FWDHERWXZUN4WAURKVGP6UC7PC57XBLLIDHJFAPEP6YUUHK7XSGA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219602472 (6h51m15.806658693s) [noise=true , address=77.90.31.18:4806 , piece_id=2L4OYP2PJ6LBOBZUUI52QNVFWAOMU7E25SJ7XB6C2CQCAREMFVQA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219943268 (6h51m15.806999489s) [noise=true , address=104.36.82.9:4006 , piece_id=3F6A7S5CWE2TOBI6R77DHSEIWGL3HBLYRWSJKC7PGTN747XL77NQ, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.309104326 (6h51m15.896160547s) [upload=7028]: Next returned nil segment Aug 7 23:18:02.309119103 (6h51m15.896175324s) [upload=7028]: Waiting for error group managing segments... Aug 7 23:18:02.326561371 (6h51m15.913617592s) [noise=true , address=45.91.116.14:4190 , piece_id=VN7ZRT4NTNUNQNKN2OCNILBFMN2QNGZOOY6B6EFDPET3TIB42A7A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.327084597 (6h51m15.914140818s) [noise=true , address=45.91.116.7:4009 , piece_id=VMY27LD3MBT4O65CXJJEYBIP7QVB543GSJOTQLYBELK4G7DOBQRA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.329365706 (6h51m15.916421916s) [noise=true , address=45.91.116.6:4974 , piece_id=MQWUSYBCOBJPCBLKUWG5R2YGX2UXHJAYTFXLU5G45I7ADEEJOSIA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.329710399 (6h51m15.916766611s) [noise=true , address=77.90.31.14:4662 , piece_id=FWDHERWXZUN4WAURKVGP6UC7PC57XBLLIDHJFAPEP6YUUHK7XSGA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.330040726 (6h51m15.917096936s) [noise=true , address=77.90.31.17:4772 , piece_id=A7XUBHQKTJRVAMBQAO24KOU4GFGENHMFY6DBETNJDZ7J5YR7VO6Q, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.330501876 (6h51m15.917558097s) [noise=true , address=77.90.31.18:4806 , piece_id=2L4OYP2PJ6LBOBZUUI52QNVFWAOMU7E25SJ7XB6C2CQCAREMFVQA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.330860886 (6h51m15.917917107s) [noise=true , address=77.90.31.16:4735 , piece_id=QBT3GKFTCJBCONV4W4E3V74IQBXZO2R4M3SQBJBGRPEXMN4L4V2A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.331216931 (6h51m15.918273152s) [noise=true , address=77.90.31.12:4592 , piece_id=X4RIDUL5RUENCF4ZVRNIBWIUAHGQYW6XLBZWDMOUYGEIWQL5ZH7A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.332064072 (6h51m15.919120293s) [noise=true , address=77.90.31.21:4877 , piece_id=C7HY6GFRIWXX43OOGKBM3EXJVIMPUQP36KHHSVSGOVFJYBBX4IBA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.332119265 (6h51m15.919175476s) [noise=true , address=77.90.31.11:4553 , piece_id=SJUK6H5G242A3QOXRARMJGQP6IGRQ2WJOUHBK24BMT2M3DYKGKAQ, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.332931742 (6h51m15.919987963s) [noise=true , address=104.36.82.3:4767 , piece_id=D6EYZVFHGH7J3ABUMJBQPTJUDNA6GPMONDLQY7WL6NTVLYEEZM4Q, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.340533561 (6h51m15.927589782s) [noise=true , address=45.91.116.10:4082 , piece_id=GCFQIVXLPEIWQXT4A5Y6Z4N2E54HKSLGB2QDXR62SYQ4AISIXIMA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.345624544 (6h51m15.932680765s) [noise=true , address=104.36.82.14:4222 , piece_id=BJCIBUJTQDIS64F2XTL6XYNMICZJUXBP2NUA3UXOCORROZWONL4Q, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.345641816 (6h51m15.932698037s) [seg_pos={0 0}, upload=7028]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:02.345736042 (6h51m15.932792253s) [noise=true , address=104.36.82.11:4115 , piece_id=MIW4TCCVX7O3KTEBU2RQRHD4GKPBYQOBTDKXGLH3B3NYCKTBDQ3A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:02.345762481 (6h51m15.932818693s) [noise=true , address=77.90.31.19:4842 , piece_id=2K54EHW7ZB22YIUFSMKZBKD66T4NLCPLSAPR344HPTF7WPX537JA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:02.345881584 (6h51m15.932937805s) [noise=true , address=104.36.82.9:4006 , piece_id=3F6A7S5CWE2TOBI6R77DHSEIWGL3HBLYRWSJKC7PGTN747XL77NQ, seg_pos={0 0}, upload=7028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: context canceled; context canceled Aug 7 23:18:02.345898606 (6h51m15.932954827s) [seg_pos={0 0}, upload=7028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:02.345971481 (6h51m15.933027702s) [upload=7028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:02.345975489 (6h51m15.93303171s) [upload=7028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:22:03.557058290 (6h55m17.144114511s) [upload=7028]: Done uploading segments... Aug 7 23:22:03.557078478 (6h55m17.144134699s) [upload=7028]: Upload finished. err: metaclient: context deadline exceeded Duration: 19.85309612s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................................X.XX.X....X.X.......] duration: 424.385693ms FAIL (2025-08-07 23:27:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 424.41156ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 183.833662ms FAIL (2025-08-07 23:26:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 183.849061ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........................................XXXXX..X...........] duration: 1.934060377s FAIL (2025-08-07 23:26:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.93408375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................................XXXXXX.X.X..........] duration: 1.856466779s FAIL (2025-08-07 23:26:38 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.856484854s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.........................................XXXXX.XX...........] duration: 177.865102ms FAIL (2025-08-07 23:26:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.277141003 (6h47m50.864197244s) [upload=6974]: Starting upload Aug 7 23:14:37.277320197 (6h47m50.864376439s) [upload=6974]: Done starting upload Aug 7 23:14:37.277421095 (6h47m50.864477337s) [upload=6974]: Uploading segments... Aug 7 23:14:37.277519849 (6h47m50.864576091s) [upload=6974]: Got next segment. Inline: true Aug 7 23:14:37.277527635 (6h47m50.864583875s) [upload=6974]: Waiting for error group managing segments... Aug 7 23:14:37.277578299 (6h47m50.86463454s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.277584181 (6h47m50.864640421s) [upload=6974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.277587807 (6h47m50.864644049s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.263997045 (6h48m20.851053287s) [upload=6974]: Done uploading segments... Aug 7 23:15:07.264036289 (6h48m20.85109253s) [upload=6974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:40.767391975 (6h48m54.354448226s) [upload=6993]: Starting upload Aug 7 23:15:40.767481722 (6h48m54.354537963s) [upload=6993]: Done starting upload Aug 7 23:15:40.767544320 (6h48m54.35460057s) [upload=6993]: Uploading segments... Aug 7 23:15:40.767585375 (6h48m54.354641617s) [upload=6993]: Got next segment. Inline: true Aug 7 23:15:40.767591397 (6h48m54.354647638s) [upload=6993]: Waiting for error group managing segments... Aug 7 23:15:40.767637002 (6h48m54.354693244s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:40.767642423 (6h48m54.354698664s) [upload=6993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:40.767646530 (6h48m54.354702782s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:10.754586120 (6h49m24.341642341s) [upload=6993]: Done uploading segments... Aug 7 23:16:10.754601419 (6h49m24.34165763s) [upload=6993]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:26.719060301 (6h49m40.306116542s) [upload=7005]: Starting upload Aug 7 23:16:26.719141191 (6h49m40.306197433s) [upload=7005]: Done starting upload Aug 7 23:16:26.719210051 (6h49m40.306266292s) [upload=7005]: Uploading segments... Aug 7 23:16:26.719285351 (6h49m40.306341592s) [upload=7005]: Got next segment. Inline: true Aug 7 23:16:26.719292685 (6h49m40.306348926s) [upload=7005]: Waiting for error group managing segments... Aug 7 23:16:26.719340825 (6h49m40.306397066s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:26.719347277 (6h49m40.306403518s) [upload=7005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:26.719351064 (6h49m40.306407305s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:26.802515608 (6h49m40.389571849s) [upload=7005]: Done uploading segments... Aug 7 23:16:26.802535986 (6h49m40.389592227s) [upload=7005]: Upload finished. err: <nil> Aug 7 23:16:42.191377275 (6h49m55.778433516s) [upload=7011]: Starting upload Aug 7 23:16:42.191466421 (6h49m55.778522662s) [upload=7011]: Done starting upload Aug 7 23:16:42.191531582 (6h49m55.778587803s) [upload=7011]: Uploading segments... Aug 7 23:16:42.191607003 (6h49m55.778663224s) [upload=7011]: Got next segment. Inline: true Aug 7 23:16:42.191612414 (6h49m55.778668635s) [upload=7011]: Waiting for error group managing segments... Aug 7 23:16:42.191651346 (6h49m55.778707556s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:42.191656575 (6h49m55.778712787s) [upload=7011]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:42.191659481 (6h49m55.778715702s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.173345160 (6h50m25.760401401s) [upload=7011]: Done uploading segments... Aug 7 23:17:12.173364576 (6h50m25.760420816s) [upload=7011]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:12.261509015 (6h50m25.848565257s) [upload=7016]: Starting upload Aug 7 23:17:12.261597881 (6h50m25.848654122s) [upload=7016]: Done starting upload Aug 7 23:17:12.261665168 (6h50m25.848721419s) [upload=7016]: Uploading segments... Aug 7 23:17:12.261708137 (6h50m25.848764388s) [upload=7016]: Got next segment. Inline: true Aug 7 23:17:12.261713487 (6h50m25.848769728s) [upload=7016]: Waiting for error group managing segments... Aug 7 23:17:12.261755746 (6h50m25.848811988s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:12.261761196 (6h50m25.848817448s) [upload=7016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:12.261764593 (6h50m25.848820834s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:42.241895867 (6h50m55.828952078s) [upload=7016]: Done uploading segments... Aug 7 23:17:42.241913931 (6h50m55.828970141s) [upload=7016]: Upload finished. err: metaclient: context deadline exceeded Duration: 177.883526ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [.........................................XXXXXX.X...........] duration: 172.343745ms FAIL (2025-08-07 23:26:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.018385587 (6h43m49.605441827s) [upload=6905]: Upload finished. err: <nil> Aug 7 23:10:57.038963659 (6h44m10.62601988s) [upload=6913]: Starting upload Aug 7 23:10:57.039017149 (6h44m10.62607337s) [upload=6913]: Done starting upload Aug 7 23:10:57.039065159 (6h44m10.62612137s) [upload=6913]: Uploading segments... Aug 7 23:10:57.039093831 (6h44m10.626150042s) [upload=6913]: Got next segment. Inline: true Aug 7 23:10:57.039097308 (6h44m10.626153519s) [upload=6913]: Waiting for error group managing segments... Aug 7 23:10:57.039142663 (6h44m10.626198874s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:57.039146320 (6h44m10.626202531s) [upload=6913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:57.039148564 (6h44m10.626204785s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:57.092133855 (6h44m10.679190086s) [upload=6913]: Done uploading segments... Aug 7 23:10:57.092152090 (6h44m10.67920833s) [upload=6913]: Upload finished. err: <nil> Aug 7 23:11:47.690012005 (6h45m1.277068256s) [upload=6926]: Starting upload Aug 7 23:11:47.690127641 (6h45m1.277183882s) [upload=6926]: Done starting upload Aug 7 23:11:47.690277249 (6h45m1.27733349s) [upload=6926]: Uploading segments... Aug 7 23:11:47.690316773 (6h45m1.277373025s) [upload=6926]: Got next segment. Inline: true Aug 7 23:11:47.690360005 (6h45m1.277416245s) [upload=6926]: Waiting for error group managing segments... Aug 7 23:11:47.690405779 (6h45m1.277462031s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:47.690411880 (6h45m1.277468122s) [upload=6926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:47.690415508 (6h45m1.277471748s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:47.740982296 (6h45m1.328038537s) [upload=6926]: Done uploading segments... Aug 7 23:11:47.740995380 (6h45m1.328051622s) [upload=6926]: Upload finished. err: <nil> Aug 7 23:12:15.388374517 (6h45m28.975430749s) [upload=6935]: Starting upload Aug 7 23:12:15.388464206 (6h45m28.975520437s) [upload=6935]: Done starting upload Aug 7 23:12:15.388550818 (6h45m28.975607039s) [upload=6935]: Uploading segments... Aug 7 23:12:15.388651916 (6h45m28.975708127s) [upload=6935]: Got next segment. Inline: true Aug 7 23:12:15.388660222 (6h45m28.975716443s) [upload=6935]: Waiting for error group managing segments... Aug 7 23:12:15.388716747 (6h45m28.975772968s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:15.388723729 (6h45m28.97577995s) [upload=6935]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:15.388727838 (6h45m28.975784059s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:15.375897715 (6h46m28.962953947s) [upload=6935]: Done uploading segments... Aug 7 23:13:15.375942599 (6h46m28.962998841s) [upload=6935]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:15.452799710 (6h46m29.039855931s) [upload=6951]: Starting upload Aug 7 23:13:15.452856376 (6h46m29.039912597s) [upload=6951]: Done starting upload Aug 7 23:13:15.452901079 (6h46m29.0399573s) [upload=6951]: Uploading segments... Aug 7 23:13:15.452938739 (6h46m29.03999495s) [upload=6951]: Got next segment. Inline: true Aug 7 23:13:15.452943217 (6h46m29.039999439s) [upload=6951]: Waiting for error group managing segments... Aug 7 23:13:15.452974977 (6h46m29.040031188s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:15.452979305 (6h46m29.040035526s) [upload=6951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:15.452982110 (6h46m29.040038331s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:15.442587501 (6h47m29.029643742s) [upload=6951]: Done uploading segments... Aug 7 23:14:15.442608670 (6h47m29.029664922s) [upload=6951]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:15.516163727 (6h47m29.103219978s) [upload=6968]: Starting upload Aug 7 23:14:15.516314939 (6h47m29.10337124s) [upload=6968]: Done starting upload Aug 7 23:14:15.516411479 (6h47m29.103467721s) [upload=6968]: Uploading segments... Aug 7 23:14:15.516477332 (6h47m29.103533573s) [upload=6968]: Got next segment. Inline: true Aug 7 23:14:15.516487672 (6h47m29.103543913s) [upload=6968]: Waiting for error group managing segments... Aug 7 23:14:15.516556882 (6h47m29.103613162s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:15.516567169 (6h47m29.103623411s) [upload=6968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:15.516573491 (6h47m29.103629733s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:15.501845849 (6h48m29.08890207s) [upload=6968]: Done uploading segments... Aug 7 23:15:15.501865025 (6h48m29.088921247s) [upload=6968]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:15.591828711 (6h48m29.178884922s) [upload=6986]: Starting upload Aug 7 23:15:15.591883403 (6h48m29.178939613s) [upload=6986]: Done starting upload Aug 7 23:15:15.591959133 (6h48m29.179015344s) [upload=6986]: Uploading segments... Aug 7 23:15:15.592001994 (6h48m29.179058205s) [upload=6986]: Got next segment. Inline: true Aug 7 23:15:15.592022743 (6h48m29.179078964s) [upload=6986]: Waiting for error group managing segments... Aug 7 23:15:15.592060944 (6h48m29.179117156s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:15.592065653 (6h48m29.179121874s) [upload=6986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:15.592067957 (6h48m29.179124178s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:15.578451341 (6h49m29.165507562s) [upload=6986]: Done uploading segments... Aug 7 23:16:15.578469606 (6h49m29.165525816s) [upload=6986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:23.278937820 (6h49m36.865994061s) [upload=7004]: Starting upload Aug 7 23:16:23.279020784 (6h49m36.866077025s) [upload=7004]: Done starting upload Aug 7 23:16:23.279084963 (6h49m36.866141205s) [upload=7004]: Uploading segments... Aug 7 23:16:23.279128285 (6h49m36.866184526s) [upload=7004]: Got next segment. Inline: true Aug 7 23:16:23.279135057 (6h49m36.866191299s) [upload=7004]: Waiting for error group managing segments... Aug 7 23:16:23.279179039 (6h49m36.86623528s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:23.279184509 (6h49m36.866240751s) [upload=7004]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:23.279188327 (6h49m36.866244558s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:23.264696428 (6h50m36.851752649s) [upload=7004]: Done uploading segments... Aug 7 23:17:23.264712168 (6h50m36.851768379s) [upload=7004]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.354485739 (6h50m36.94154196s) [upload=7020]: Starting upload Aug 7 23:17:23.354537225 (6h50m36.941593446s) [upload=7020]: Done starting upload Aug 7 23:17:23.354582019 (6h50m36.94163824s) [upload=7020]: Uploading segments... Aug 7 23:17:23.354609931 (6h50m36.941666142s) [upload=7020]: Got next segment. Inline: true Aug 7 23:17:23.354613586 (6h50m36.941669807s) [upload=7020]: Waiting for error group managing segments... Aug 7 23:17:23.354647150 (6h50m36.941703371s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.354650656 (6h50m36.941706877s) [upload=7020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.354652961 (6h50m36.941709172s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:23.336455483 (6h51m36.923511723s) [upload=7020]: Done uploading segments... Aug 7 23:18:23.336475711 (6h51m36.923531952s) [upload=7020]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:34.755118608 (6h51m48.342174818s) [upload=7039]: Starting upload Aug 7 23:18:34.755161578 (6h51m48.342217799s) [upload=7039]: Done starting upload Aug 7 23:18:34.755200670 (6h51m48.342256881s) [upload=7039]: Uploading segments... Aug 7 23:18:34.755239954 (6h51m48.342296176s) [upload=7039]: Got next segment. Inline: true Aug 7 23:18:34.755243802 (6h51m48.342300023s) [upload=7039]: Waiting for error group managing segments... Aug 7 23:18:34.755274309 (6h51m48.34233052s) [upload=7039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:34.755277996 (6h51m48.342334207s) [upload=7039]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:34.755280250 (6h51m48.342336471s) [upload=7039]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:34.875891722 (6h51m48.462947943s) [upload=7039]: Done uploading segments... Aug 7 23:18:34.875937538 (6h51m48.462993759s) [upload=7039]: Upload finished. err: <nil> Aug 7 23:19:14.105729045 (6h52m27.692785266s) [upload=7050]: Starting upload Aug 7 23:19:14.105780641 (6h52m27.692836852s) [upload=7050]: Done starting upload Aug 7 23:19:14.105815075 (6h52m27.692871286s) [upload=7050]: Uploading segments... Aug 7 23:19:14.105851563 (6h52m27.692907773s) [upload=7050]: Got next segment. Inline: true Aug 7 23:19:14.105856422 (6h52m27.692912643s) [upload=7050]: Waiting for error group managing segments... Aug 7 23:19:14.105888662 (6h52m27.692944883s) [upload=7050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:14.105892429 (6h52m27.69294864s) [upload=7050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:14.105894724 (6h52m27.692950934s) [upload=7050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:14.092509460 (6h53m27.679565681s) [upload=7050]: Done uploading segments... Aug 7 23:20:14.092523326 (6h53m27.679579547s) [upload=7050]: Upload finished. err: metaclient: context deadline exceeded Duration: 172.366477ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................................XXXXXX.X.X.........] duration: 1.958033715s FAIL (2025-08-07 23:26:21 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.958054103s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................................XXX.X.X........] duration: 1.509390433s FAIL (2025-08-07 23:26:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== A5NM4A, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.091791202 (6h51m59.678847453s) [noise=true , address=104.36.82.59:4307 , piece_id=DGJPGA6ETYMP34JBQE6WXC32PIXFWLUUKZNNWB2NXBWS727MJRPQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.092306323 (6h51m59.679362574s) [noise=true , address=104.36.82.3:4794 , piece_id=2M55ORW4IODVHZGVFIAOG6SDOHBT7NWX3X74ANGSUHIKY7ZLAFJA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.092699598 (6h51m59.679755839s) [noise=true , address=104.36.82.58:4299 , piece_id=25QYBX42LIC7HWREJDDQWECRZFNZV3PBM2JITVH6R52TGPDEMQDA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093097069 (6h51m59.68015331s) [noise=true , address=64.49.13.155:4922 , piece_id=XWJAU4RZXJAW2IM5CUGFI3MAK5GYEM6AMWMO6AO7PK6MYA47ABQQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093493069 (6h51m59.68054931s) [noise=true , address=104.36.82.9:4031 , piece_id=TRXII4JD2BKKTYJ3EVICMM7WGPELQNRU7Z7I23KKY2NFT4KN3JNA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093847771 (6h51m59.680904022s) [noise=true , address=104.36.82.7:4935 , piece_id=LVI53GXPJGQCYU5QXT2MRTPZJOQETBU3RUGVV2DGSN26NP6I3XVQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094203605 (6h51m59.681259855s) [noise=true , address=45.91.116.13:4171 , piece_id=UNI5ZZNWXKPIKQXVVWNE3S7FUGM2XWNO7PSJDRWGBSOERCY22OPQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094597400 (6h51m59.681653641s) [noise=true , address=45.91.116.9:4290 , piece_id=S3USZPGNHC7SX3HHEM6LBII54CGKRCQGLMQNRE7ZQXM3ZRLMJMSQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094954427 (6h51m59.682010668s) [noise=true , address=104.36.82.62:4439 , piece_id=P66ZY2SI3GROYBUTKW7HEEUYT6VD4WWY53PQEL43J25DQWHHGK5Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.095373879 (6h51m59.68243012s) [noise=true , address=104.36.82.46:4512 , piece_id=X5SF67DUK4DXJICZCZVYBVTDFSS3HHFHXUEYJMMOMSVFORXUBS2A, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.095742858 (6h51m59.6827991s) [noise=true , address=64.49.13.154:4930 , piece_id=4D74XBIPQHETEYR7WQSZHGFHW42EGNAVYDO5NSBKAFP44BK4JR2Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096084166 (6h51m59.683140377s) [noise=true , address=79.127.201.215:4350 , piece_id=WSQK2BLHIRJ4J6FH22IEV64A3VV7N7RFNR2WI2XJXWNMSW2H765Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096399684 (6h51m59.683455905s) [noise=true , address=104.36.82.61:4379 , piece_id=W2XB5LCHE7BIGVF53UOKTBS2E2ZQJ2FZHLRDUKQHPUENZQELPU2Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096721615 (6h51m59.683777826s) [noise=true , address=104.36.82.42:4815 , piece_id=NEGWCQ6XEFYUU5XIXZRYB5WVDRVF6ARBF6LMGC65AQHNGGJRJIXA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.097044478 (6h51m59.684100699s) [noise=true , address=104.36.82.16:4315 , piece_id=YDQ3SD75HCNENS667QTX2U5WCHQUFEEOX3WW6CXNVHKPJ7ZAE2YA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.097416412 (6h51m59.684472633s) [noise=true , address=104.36.82.9:4036 , piece_id=KSI3DLD6I3TZUKTH7KQTLLBEIMILZ2GUHVNP5JU6MWNVRIEARHGQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.172312403 (6h51m59.759368613s) [upload=7040]: Next returned nil segment Aug 7 23:18:46.172323192 (6h51m59.759379403s) [upload=7040]: Waiting for error group managing segments... Aug 7 23:18:46.182690655 (6h51m59.769746865s) [noise=true , address=45.91.116.9:4290 , piece_id=S3USZPGNHC7SX3HHEM6LBII54CGKRCQGLMQNRE7ZQXM3ZRLMJMSQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.188105241 (6h51m59.775161462s) [noise=true , address=104.36.82.7:4935 , piece_id=LVI53GXPJGQCYU5QXT2MRTPZJOQETBU3RUGVV2DGSN26NP6I3XVQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.196654078 (6h51m59.783710289s) [noise=true , address=45.91.116.13:4171 , piece_id=UNI5ZZNWXKPIKQXVVWNE3S7FUGM2XWNO7PSJDRWGBSOERCY22OPQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.198492931 (6h51m59.785549152s) [noise=true , address=104.36.82.58:4299 , piece_id=25QYBX42LIC7HWREJDDQWECRZFNZV3PBM2JITVH6R52TGPDEMQDA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.198779515 (6h51m59.785835726s) [noise=true , address=104.36.82.59:4307 , piece_id=DGJPGA6ETYMP34JBQE6WXC32PIXFWLUUKZNNWB2NXBWS727MJRPQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.204927762 (6h51m59.791983983s) [noise=true , address=104.36.82.52:4099 , piece_id=H2WVX26COCLZ7KH6AJLYBN4H6JRIUF5IXYED6DRSCINOVGA5NM4A, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.208251176 (6h51m59.795307387s) [noise=true , address=104.36.82.16:4315 , piece_id=YDQ3SD75HCNENS667QTX2U5WCHQUFEEOX3WW6CXNVHKPJ7ZAE2YA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.225240327 (6h51m59.812296549s) [noise=true , address=104.36.82.62:4439 , piece_id=P66ZY2SI3GROYBUTKW7HEEUYT6VD4WWY53PQEL43J25DQWHHGK5Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.225308985 (6h51m59.812365196s) [noise=true , address=104.36.82.3:4794 , piece_id=2M55ORW4IODVHZGVFIAOG6SDOHBT7NWX3X74ANGSUHIKY7ZLAFJA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.245212838 (6h51m59.832269059s) [noise=true , address=104.36.82.4:4818 , piece_id=U5SIEBGYEKIRZ4FD46ZT3L3AVH7MIVC3JRKPC7V46F6MGFT5QFXQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.262860569 (6h51m59.84991678s) [noise=true , address=104.36.82.9:4031 , piece_id=TRXII4JD2BKKTYJ3EVICMM7WGPELQNRU7Z7I23KKY2NFT4KN3JNA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.287460286 (6h51m59.874516498s) [noise=true , address=104.36.82.9:4036 , piece_id=KSI3DLD6I3TZUKTH7KQTLLBEIMILZ2GUHVNP5JU6MWNVRIEARHGQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.323021195 (6h51m59.910077436s) [noise=true , address=104.36.82.42:4815 , piece_id=NEGWCQ6XEFYUU5XIXZRYB5WVDRVF6ARBF6LMGC65AQHNGGJRJIXA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.337567646 (6h51m59.924623857s) [noise=true , address=79.127.201.215:4350 , piece_id=WSQK2BLHIRJ4J6FH22IEV64A3VV7N7RFNR2WI2XJXWNMSW2H765Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.337650712 (6h51m59.924706923s) [seg_pos={0 0}, upload=7040]: Segment reached optimal threshold of 14 pieces. Aug 7 23:18:46.339154849 (6h51m59.92621106s) [noise=true , address=104.36.82.61:4379 , piece_id=W2XB5LCHE7BIGVF53UOKTBS2E2ZQJ2FZHLRDUKQHPUENZQELPU2Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lp3H6533e4oUt4mbiWJMjLRi5fb2wQr7RrntSpRm9xqF1TcEi): protocol: rpc: dial tcp 104.36.82.61:4379: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339496567 (6h51m59.926552778s) [noise=true , address=64.49.13.154:4930 , piece_id=4D74XBIPQHETEYR7WQSZHGFHW42EGNAVYDO5NSBKAFP44BK4JR2Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kNK8y5et7MiiWNDT6iaC8YBAk3PX6WX1XSuU4QMSFBp3KymBb): protocol: read tcp 5.161.89.252:19812->64.49.13.154:4930: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339638873 (6h51m59.926695084s) [noise=true , address=104.36.82.46:4512 , piece_id=X5SF67DUK4DXJICZCZVYBVTDFSS3HHFHXUEYJMMOMSVFORXUBS2A, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2): protocol: read tcp 5.161.89.252:12938->104.36.82.46:4512: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339695828 (6h51m59.92675204s) [noise=true , address=64.49.13.155:4922 , piece_id=XWJAU4RZXJAW2IM5CUGFI3MAK5GYEM6AMWMO6AO7PK6MYA47ABQQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: read tcp 5.161.89.252:28596->64.49.13.155:4922: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339725374 (6h51m59.926781605s) [seg_pos={0 0}, upload=7040]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:18:46.339804101 (6h51m59.926860312s) [upload=7040]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:46.339836361 (6h51m59.926892572s) [upload=7040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:46.393522662 (6h51m59.980578873s) [upload=7040]: Done uploading segments... Aug 7 23:18:46.393540715 (6h51m59.980596927s) [upload=7040]: Upload finished. err: <nil> Aug 7 23:20:00.141045360 (6h53m13.728101581s) [upload=7062]: Starting upload Aug 7 23:20:00.141098730 (6h53m13.728154951s) [upload=7062]: Done starting upload Aug 7 23:20:00.141249572 (6h53m13.728305793s) [upload=7062]: Uploading segments... Aug 7 23:20:00.141576813 (6h53m13.728633034s) [upload=7062]: Got next segment. Inline: false Aug 7 23:20:00.141621376 (6h53m13.728677597s) [upload=7062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:00.141625283 (6h53m13.728681494s) [upload=7062]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:00.122447463 (6h54m13.709503674s) [upload=7062]: Next segment err: context deadline exceeded Aug 7 23:21:00.122599688 (6h54m13.709655909s) [upload=7062]: Done uploading segments... Aug 7 23:21:00.122624775 (6h54m13.709680986s) [upload=7062]: Upload finished. err: context deadline exceeded Duration: 1.509419604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [..........................................XXXXXX.X..........] duration: 220.27331ms FAIL (2025-08-07 23:26:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.018385587 (6h43m49.605441827s) [upload=6905]: Upload finished. err: <nil> Aug 7 23:10:57.038963659 (6h44m10.62601988s) [upload=6913]: Starting upload Aug 7 23:10:57.039017149 (6h44m10.62607337s) [upload=6913]: Done starting upload Aug 7 23:10:57.039065159 (6h44m10.62612137s) [upload=6913]: Uploading segments... Aug 7 23:10:57.039093831 (6h44m10.626150042s) [upload=6913]: Got next segment. Inline: true Aug 7 23:10:57.039097308 (6h44m10.626153519s) [upload=6913]: Waiting for error group managing segments... Aug 7 23:10:57.039142663 (6h44m10.626198874s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:57.039146320 (6h44m10.626202531s) [upload=6913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:57.039148564 (6h44m10.626204785s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:57.092133855 (6h44m10.679190086s) [upload=6913]: Done uploading segments... Aug 7 23:10:57.092152090 (6h44m10.67920833s) [upload=6913]: Upload finished. err: <nil> Aug 7 23:11:47.690012005 (6h45m1.277068256s) [upload=6926]: Starting upload Aug 7 23:11:47.690127641 (6h45m1.277183882s) [upload=6926]: Done starting upload Aug 7 23:11:47.690277249 (6h45m1.27733349s) [upload=6926]: Uploading segments... Aug 7 23:11:47.690316773 (6h45m1.277373025s) [upload=6926]: Got next segment. Inline: true Aug 7 23:11:47.690360005 (6h45m1.277416245s) [upload=6926]: Waiting for error group managing segments... Aug 7 23:11:47.690405779 (6h45m1.277462031s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:47.690411880 (6h45m1.277468122s) [upload=6926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:47.690415508 (6h45m1.277471748s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:47.740982296 (6h45m1.328038537s) [upload=6926]: Done uploading segments... Aug 7 23:11:47.740995380 (6h45m1.328051622s) [upload=6926]: Upload finished. err: <nil> Aug 7 23:12:15.388374517 (6h45m28.975430749s) [upload=6935]: Starting upload Aug 7 23:12:15.388464206 (6h45m28.975520437s) [upload=6935]: Done starting upload Aug 7 23:12:15.388550818 (6h45m28.975607039s) [upload=6935]: Uploading segments... Aug 7 23:12:15.388651916 (6h45m28.975708127s) [upload=6935]: Got next segment. Inline: true Aug 7 23:12:15.388660222 (6h45m28.975716443s) [upload=6935]: Waiting for error group managing segments... Aug 7 23:12:15.388716747 (6h45m28.975772968s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:15.388723729 (6h45m28.97577995s) [upload=6935]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:15.388727838 (6h45m28.975784059s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:15.375897715 (6h46m28.962953947s) [upload=6935]: Done uploading segments... Aug 7 23:13:15.375942599 (6h46m28.962998841s) [upload=6935]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:15.452799710 (6h46m29.039855931s) [upload=6951]: Starting upload Aug 7 23:13:15.452856376 (6h46m29.039912597s) [upload=6951]: Done starting upload Aug 7 23:13:15.452901079 (6h46m29.0399573s) [upload=6951]: Uploading segments... Aug 7 23:13:15.452938739 (6h46m29.03999495s) [upload=6951]: Got next segment. Inline: true Aug 7 23:13:15.452943217 (6h46m29.039999439s) [upload=6951]: Waiting for error group managing segments... Aug 7 23:13:15.452974977 (6h46m29.040031188s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:15.452979305 (6h46m29.040035526s) [upload=6951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:15.452982110 (6h46m29.040038331s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:15.442587501 (6h47m29.029643742s) [upload=6951]: Done uploading segments... Aug 7 23:14:15.442608670 (6h47m29.029664922s) [upload=6951]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:15.516163727 (6h47m29.103219978s) [upload=6968]: Starting upload Aug 7 23:14:15.516314939 (6h47m29.10337124s) [upload=6968]: Done starting upload Aug 7 23:14:15.516411479 (6h47m29.103467721s) [upload=6968]: Uploading segments... Aug 7 23:14:15.516477332 (6h47m29.103533573s) [upload=6968]: Got next segment. Inline: true Aug 7 23:14:15.516487672 (6h47m29.103543913s) [upload=6968]: Waiting for error group managing segments... Aug 7 23:14:15.516556882 (6h47m29.103613162s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:15.516567169 (6h47m29.103623411s) [upload=6968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:15.516573491 (6h47m29.103629733s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:15.501845849 (6h48m29.08890207s) [upload=6968]: Done uploading segments... Aug 7 23:15:15.501865025 (6h48m29.088921247s) [upload=6968]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:15.591828711 (6h48m29.178884922s) [upload=6986]: Starting upload Aug 7 23:15:15.591883403 (6h48m29.178939613s) [upload=6986]: Done starting upload Aug 7 23:15:15.591959133 (6h48m29.179015344s) [upload=6986]: Uploading segments... Aug 7 23:15:15.592001994 (6h48m29.179058205s) [upload=6986]: Got next segment. Inline: true Aug 7 23:15:15.592022743 (6h48m29.179078964s) [upload=6986]: Waiting for error group managing segments... Aug 7 23:15:15.592060944 (6h48m29.179117156s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:15.592065653 (6h48m29.179121874s) [upload=6986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:15.592067957 (6h48m29.179124178s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:15.578451341 (6h49m29.165507562s) [upload=6986]: Done uploading segments... Aug 7 23:16:15.578469606 (6h49m29.165525816s) [upload=6986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:23.278937820 (6h49m36.865994061s) [upload=7004]: Starting upload Aug 7 23:16:23.279020784 (6h49m36.866077025s) [upload=7004]: Done starting upload Aug 7 23:16:23.279084963 (6h49m36.866141205s) [upload=7004]: Uploading segments... Aug 7 23:16:23.279128285 (6h49m36.866184526s) [upload=7004]: Got next segment. Inline: true Aug 7 23:16:23.279135057 (6h49m36.866191299s) [upload=7004]: Waiting for error group managing segments... Aug 7 23:16:23.279179039 (6h49m36.86623528s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:23.279184509 (6h49m36.866240751s) [upload=7004]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:23.279188327 (6h49m36.866244558s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:23.264696428 (6h50m36.851752649s) [upload=7004]: Done uploading segments... Aug 7 23:17:23.264712168 (6h50m36.851768379s) [upload=7004]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.354485739 (6h50m36.94154196s) [upload=7020]: Starting upload Aug 7 23:17:23.354537225 (6h50m36.941593446s) [upload=7020]: Done starting upload Aug 7 23:17:23.354582019 (6h50m36.94163824s) [upload=7020]: Uploading segments... Aug 7 23:17:23.354609931 (6h50m36.941666142s) [upload=7020]: Got next segment. Inline: true Aug 7 23:17:23.354613586 (6h50m36.941669807s) [upload=7020]: Waiting for error group managing segments... Aug 7 23:17:23.354647150 (6h50m36.941703371s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.354650656 (6h50m36.941706877s) [upload=7020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.354652961 (6h50m36.941709172s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:23.336455483 (6h51m36.923511723s) [upload=7020]: Done uploading segments... Aug 7 23:18:23.336475711 (6h51m36.923531952s) [upload=7020]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:34.755118608 (6h51m48.342174818s) [upload=7039]: Starting upload Aug 7 23:18:34.755161578 (6h51m48.342217799s) [upload=7039]: Done starting upload Aug 7 23:18:34.755200670 (6h51m48.342256881s) [upload=7039]: Uploading segments... Aug 7 23:18:34.755239954 (6h51m48.342296176s) [upload=7039]: Got next segment. Inline: true Aug 7 23:18:34.755243802 (6h51m48.342300023s) [upload=7039]: Waiting for error group managing segments... Aug 7 23:18:34.755274309 (6h51m48.34233052s) [upload=7039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:34.755277996 (6h51m48.342334207s) [upload=7039]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:34.755280250 (6h51m48.342336471s) [upload=7039]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:34.875891722 (6h51m48.462947943s) [upload=7039]: Done uploading segments... Aug 7 23:18:34.875937538 (6h51m48.462993759s) [upload=7039]: Upload finished. err: <nil> Aug 7 23:19:14.105729045 (6h52m27.692785266s) [upload=7050]: Starting upload Aug 7 23:19:14.105780641 (6h52m27.692836852s) [upload=7050]: Done starting upload Aug 7 23:19:14.105815075 (6h52m27.692871286s) [upload=7050]: Uploading segments... Aug 7 23:19:14.105851563 (6h52m27.692907773s) [upload=7050]: Got next segment. Inline: true Aug 7 23:19:14.105856422 (6h52m27.692912643s) [upload=7050]: Waiting for error group managing segments... Aug 7 23:19:14.105888662 (6h52m27.692944883s) [upload=7050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:14.105892429 (6h52m27.69294864s) [upload=7050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:14.105894724 (6h52m27.692950934s) [upload=7050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:14.092509460 (6h53m27.679565681s) [upload=7050]: Done uploading segments... Aug 7 23:20:14.092523326 (6h53m27.679579547s) [upload=7050]: Upload finished. err: metaclient: context deadline exceeded Duration: 220.291413ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................................X.XX.X....X.X......] duration: 731.687751ms FAIL (2025-08-07 23:26:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 731.706135ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [...........................................X.XXXXX..X.......] duration: 194.595009ms FAIL (2025-08-07 23:26:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 194.63317ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.362955918s FAIL (2025-08-07 23:26:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 1.362975466s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................................X....XXX.X......] duration: 1.941400638s FAIL (2025-08-07 23:26:13 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ite tcp 5.161.89.252:59232->104.36.82.13:4187: use of closed network connection Aug 7 23:17:27.468792451 (6h50m41.055848663s) [noise=true , address=104.36.82.15:4257 , piece_id=YSUUNUNXXR4XZZRQBZVTYSP7WZNQVDHHS7KVVRKIAY5AQI3EMGCQ, seg_pos={0 0}, upload=7021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: read tcp 5.161.89.252:16228->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:27.468805356 (6h50m41.055861567s) [seg_pos={0 0}, upload=7021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:17:27.468898489 (6h50m41.055954701s) [upload=7021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:27.468902838 (6h50m41.055959049s) [upload=7021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:27.512149772 (6h50m41.099205993s) [upload=7021]: Done uploading segments... Aug 7 23:17:27.512163628 (6h50m41.099219849s) [upload=7021]: Upload finished. err: <nil> Aug 7 23:17:51.618188257 (6h51m5.205244499s) [upload=7028]: Starting upload Aug 7 23:17:51.618315174 (6h51m5.205371416s) [upload=7028]: Done starting upload Aug 7 23:17:51.618432584 (6h51m5.205488824s) [upload=7028]: Uploading segments... Aug 7 23:17:51.618706434 (6h51m5.205762675s) [upload=7028]: Got next segment. Inline: false Aug 7 23:17:51.618753392 (6h51m5.205809634s) [upload=7028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:51.618759304 (6h51m5.205815554s) [upload=7028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:02.217185080 (6h51m15.804241291s) [seg_pos={0 0}, upload=7028]: Begin upload segment... Aug 7 23:18:02.217213693 (6h51m15.804269904s) [seg_pos={0 0}, upload=7028]: Done begin upload segment. Aug 7 23:18:02.217258727 (6h51m15.804314948s) [noise=true , address=45.91.116.7:4009 , piece_id=VMY27LD3MBT4O65CXJJEYBIP7QVB543GSJOTQLYBELK4G7DOBQRA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217278985 (6h51m15.804335206s) [noise=true , address=77.90.31.12:4592 , piece_id=X4RIDUL5RUENCF4ZVRNIBWIUAHGQYW6XLBZWDMOUYGEIWQL5ZH7A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217653263 (6h51m15.804709484s) [noise=true , address=104.36.82.11:4115 , piece_id=MIW4TCCVX7O3KTEBU2RQRHD4GKPBYQOBTDKXGLH3B3NYCKTBDQ3A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217754012 (6h51m15.804810233s) [noise=true , address=77.90.31.17:4772 , piece_id=A7XUBHQKTJRVAMBQAO24KOU4GFGENHMFY6DBETNJDZ7J5YR7VO6Q, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217977880 (6h51m15.805034101s) [noise=true , address=104.36.82.14:4222 , piece_id=BJCIBUJTQDIS64F2XTL6XYNMICZJUXBP2NUA3UXOCORROZWONL4Q, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218105467 (6h51m15.805161678s) [noise=true , address=77.90.31.16:4735 , piece_id=QBT3GKFTCJBCONV4W4E3V74IQBXZO2R4M3SQBJBGRPEXMN4L4V2A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218314438 (6h51m15.805370659s) [noise=true , address=45.91.116.10:4082 , piece_id=GCFQIVXLPEIWQXT4A5Y6Z4N2E54HKSLGB2QDXR62SYQ4AISIXIMA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218477351 (6h51m15.805533563s) [noise=true , address=45.91.116.6:4974 , piece_id=MQWUSYBCOBJPCBLKUWG5R2YGX2UXHJAYTFXLU5G45I7ADEEJOSIA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218633864 (6h51m15.805690076s) [noise=true , address=77.90.31.21:4877 , piece_id=C7HY6GFRIWXX43OOGKBM3EXJVIMPUQP36KHHSVSGOVFJYBBX4IBA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218759268 (6h51m15.80581548s) [noise=true , address=45.91.116.14:4190 , piece_id=VN7ZRT4NTNUNQNKN2OCNILBFMN2QNGZOOY6B6EFDPET3TIB42A7A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218945636 (6h51m15.806001857s) [noise=true , address=77.90.31.11:4553 , piece_id=SJUK6H5G242A3QOXRARMJGQP6IGRQ2WJOUHBK24BMT2M3DYKGKAQ, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219088874 (6h51m15.806145095s) [noise=true , address=77.90.31.19:4842 , piece_id=2K54EHW7ZB22YIUFSMKZBKD66T4NLCPLSAPR344HPTF7WPX537JA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219280120 (6h51m15.806336341s) [noise=true , address=104.36.82.3:4767 , piece_id=D6EYZVFHGH7J3ABUMJBQPTJUDNA6GPMONDLQY7WL6NTVLYEEZM4Q, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219439768 (6h51m15.806495989s) [noise=true , address=77.90.31.14:4662 , piece_id=FWDHERWXZUN4WAURKVGP6UC7PC57XBLLIDHJFAPEP6YUUHK7XSGA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219602472 (6h51m15.806658693s) [noise=true , address=77.90.31.18:4806 , piece_id=2L4OYP2PJ6LBOBZUUI52QNVFWAOMU7E25SJ7XB6C2CQCAREMFVQA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219943268 (6h51m15.806999489s) [noise=true , address=104.36.82.9:4006 , piece_id=3F6A7S5CWE2TOBI6R77DHSEIWGL3HBLYRWSJKC7PGTN747XL77NQ, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.309104326 (6h51m15.896160547s) [upload=7028]: Next returned nil segment Aug 7 23:18:02.309119103 (6h51m15.896175324s) [upload=7028]: Waiting for error group managing segments... Aug 7 23:18:02.326561371 (6h51m15.913617592s) [noise=true , address=45.91.116.14:4190 , piece_id=VN7ZRT4NTNUNQNKN2OCNILBFMN2QNGZOOY6B6EFDPET3TIB42A7A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.327084597 (6h51m15.914140818s) [noise=true , address=45.91.116.7:4009 , piece_id=VMY27LD3MBT4O65CXJJEYBIP7QVB543GSJOTQLYBELK4G7DOBQRA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.329365706 (6h51m15.916421916s) [noise=true , address=45.91.116.6:4974 , piece_id=MQWUSYBCOBJPCBLKUWG5R2YGX2UXHJAYTFXLU5G45I7ADEEJOSIA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.329710399 (6h51m15.916766611s) [noise=true , address=77.90.31.14:4662 , piece_id=FWDHERWXZUN4WAURKVGP6UC7PC57XBLLIDHJFAPEP6YUUHK7XSGA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.330040726 (6h51m15.917096936s) [noise=true , address=77.90.31.17:4772 , piece_id=A7XUBHQKTJRVAMBQAO24KOU4GFGENHMFY6DBETNJDZ7J5YR7VO6Q, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.330501876 (6h51m15.917558097s) [noise=true , address=77.90.31.18:4806 , piece_id=2L4OYP2PJ6LBOBZUUI52QNVFWAOMU7E25SJ7XB6C2CQCAREMFVQA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.330860886 (6h51m15.917917107s) [noise=true , address=77.90.31.16:4735 , piece_id=QBT3GKFTCJBCONV4W4E3V74IQBXZO2R4M3SQBJBGRPEXMN4L4V2A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.331216931 (6h51m15.918273152s) [noise=true , address=77.90.31.12:4592 , piece_id=X4RIDUL5RUENCF4ZVRNIBWIUAHGQYW6XLBZWDMOUYGEIWQL5ZH7A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.332064072 (6h51m15.919120293s) [noise=true , address=77.90.31.21:4877 , piece_id=C7HY6GFRIWXX43OOGKBM3EXJVIMPUQP36KHHSVSGOVFJYBBX4IBA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.332119265 (6h51m15.919175476s) [noise=true , address=77.90.31.11:4553 , piece_id=SJUK6H5G242A3QOXRARMJGQP6IGRQ2WJOUHBK24BMT2M3DYKGKAQ, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.332931742 (6h51m15.919987963s) [noise=true , address=104.36.82.3:4767 , piece_id=D6EYZVFHGH7J3ABUMJBQPTJUDNA6GPMONDLQY7WL6NTVLYEEZM4Q, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.340533561 (6h51m15.927589782s) [noise=true , address=45.91.116.10:4082 , piece_id=GCFQIVXLPEIWQXT4A5Y6Z4N2E54HKSLGB2QDXR62SYQ4AISIXIMA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.345624544 (6h51m15.932680765s) [noise=true , address=104.36.82.14:4222 , piece_id=BJCIBUJTQDIS64F2XTL6XYNMICZJUXBP2NUA3UXOCORROZWONL4Q, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.345641816 (6h51m15.932698037s) [seg_pos={0 0}, upload=7028]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:02.345736042 (6h51m15.932792253s) [noise=true , address=104.36.82.11:4115 , piece_id=MIW4TCCVX7O3KTEBU2RQRHD4GKPBYQOBTDKXGLH3B3NYCKTBDQ3A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:02.345762481 (6h51m15.932818693s) [noise=true , address=77.90.31.19:4842 , piece_id=2K54EHW7ZB22YIUFSMKZBKD66T4NLCPLSAPR344HPTF7WPX537JA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:02.345881584 (6h51m15.932937805s) [noise=true , address=104.36.82.9:4006 , piece_id=3F6A7S5CWE2TOBI6R77DHSEIWGL3HBLYRWSJKC7PGTN747XL77NQ, seg_pos={0 0}, upload=7028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: context canceled; context canceled Aug 7 23:18:02.345898606 (6h51m15.932954827s) [seg_pos={0 0}, upload=7028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:02.345971481 (6h51m15.933027702s) [upload=7028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:02.345975489 (6h51m15.93303171s) [upload=7028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:22:03.557058290 (6h55m17.144114511s) [upload=7028]: Done uploading segments... Aug 7 23:22:03.557078478 (6h55m17.144134699s) [upload=7028]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.941422109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [........................................XXXX.X..X..X........] duration: 326.992482ms FAIL (2025-08-07 23:26:09 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 327.030222ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..........................................XXXXX..X..........] duration: 1.775284806s FAIL (2025-08-07 23:26:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.775313209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [..........................................XXXXX.XX..........] duration: 195.212902ms FAIL (2025-08-07 23:25:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.277141003 (6h47m50.864197244s) [upload=6974]: Starting upload Aug 7 23:14:37.277320197 (6h47m50.864376439s) [upload=6974]: Done starting upload Aug 7 23:14:37.277421095 (6h47m50.864477337s) [upload=6974]: Uploading segments... Aug 7 23:14:37.277519849 (6h47m50.864576091s) [upload=6974]: Got next segment. Inline: true Aug 7 23:14:37.277527635 (6h47m50.864583875s) [upload=6974]: Waiting for error group managing segments... Aug 7 23:14:37.277578299 (6h47m50.86463454s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.277584181 (6h47m50.864640421s) [upload=6974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.277587807 (6h47m50.864644049s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.263997045 (6h48m20.851053287s) [upload=6974]: Done uploading segments... Aug 7 23:15:07.264036289 (6h48m20.85109253s) [upload=6974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:40.767391975 (6h48m54.354448226s) [upload=6993]: Starting upload Aug 7 23:15:40.767481722 (6h48m54.354537963s) [upload=6993]: Done starting upload Aug 7 23:15:40.767544320 (6h48m54.35460057s) [upload=6993]: Uploading segments... Aug 7 23:15:40.767585375 (6h48m54.354641617s) [upload=6993]: Got next segment. Inline: true Aug 7 23:15:40.767591397 (6h48m54.354647638s) [upload=6993]: Waiting for error group managing segments... Aug 7 23:15:40.767637002 (6h48m54.354693244s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:40.767642423 (6h48m54.354698664s) [upload=6993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:40.767646530 (6h48m54.354702782s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:10.754586120 (6h49m24.341642341s) [upload=6993]: Done uploading segments... Aug 7 23:16:10.754601419 (6h49m24.34165763s) [upload=6993]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:26.719060301 (6h49m40.306116542s) [upload=7005]: Starting upload Aug 7 23:16:26.719141191 (6h49m40.306197433s) [upload=7005]: Done starting upload Aug 7 23:16:26.719210051 (6h49m40.306266292s) [upload=7005]: Uploading segments... Aug 7 23:16:26.719285351 (6h49m40.306341592s) [upload=7005]: Got next segment. Inline: true Aug 7 23:16:26.719292685 (6h49m40.306348926s) [upload=7005]: Waiting for error group managing segments... Aug 7 23:16:26.719340825 (6h49m40.306397066s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:26.719347277 (6h49m40.306403518s) [upload=7005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:26.719351064 (6h49m40.306407305s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:26.802515608 (6h49m40.389571849s) [upload=7005]: Done uploading segments... Aug 7 23:16:26.802535986 (6h49m40.389592227s) [upload=7005]: Upload finished. err: <nil> Aug 7 23:16:42.191377275 (6h49m55.778433516s) [upload=7011]: Starting upload Aug 7 23:16:42.191466421 (6h49m55.778522662s) [upload=7011]: Done starting upload Aug 7 23:16:42.191531582 (6h49m55.778587803s) [upload=7011]: Uploading segments... Aug 7 23:16:42.191607003 (6h49m55.778663224s) [upload=7011]: Got next segment. Inline: true Aug 7 23:16:42.191612414 (6h49m55.778668635s) [upload=7011]: Waiting for error group managing segments... Aug 7 23:16:42.191651346 (6h49m55.778707556s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:42.191656575 (6h49m55.778712787s) [upload=7011]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:42.191659481 (6h49m55.778715702s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.173345160 (6h50m25.760401401s) [upload=7011]: Done uploading segments... Aug 7 23:17:12.173364576 (6h50m25.760420816s) [upload=7011]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:12.261509015 (6h50m25.848565257s) [upload=7016]: Starting upload Aug 7 23:17:12.261597881 (6h50m25.848654122s) [upload=7016]: Done starting upload Aug 7 23:17:12.261665168 (6h50m25.848721419s) [upload=7016]: Uploading segments... Aug 7 23:17:12.261708137 (6h50m25.848764388s) [upload=7016]: Got next segment. Inline: true Aug 7 23:17:12.261713487 (6h50m25.848769728s) [upload=7016]: Waiting for error group managing segments... Aug 7 23:17:12.261755746 (6h50m25.848811988s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:12.261761196 (6h50m25.848817448s) [upload=7016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:12.261764593 (6h50m25.848820834s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:42.241895867 (6h50m55.828952078s) [upload=7016]: Done uploading segments... Aug 7 23:17:42.241913931 (6h50m55.828970141s) [upload=7016]: Upload finished. err: metaclient: context deadline exceeded Duration: 195.234994ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [...........................................XXXXXX.X.........] duration: 172.466273ms FAIL (2025-08-07 23:25:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.018385587 (6h43m49.605441827s) [upload=6905]: Upload finished. err: <nil> Aug 7 23:10:57.038963659 (6h44m10.62601988s) [upload=6913]: Starting upload Aug 7 23:10:57.039017149 (6h44m10.62607337s) [upload=6913]: Done starting upload Aug 7 23:10:57.039065159 (6h44m10.62612137s) [upload=6913]: Uploading segments... Aug 7 23:10:57.039093831 (6h44m10.626150042s) [upload=6913]: Got next segment. Inline: true Aug 7 23:10:57.039097308 (6h44m10.626153519s) [upload=6913]: Waiting for error group managing segments... Aug 7 23:10:57.039142663 (6h44m10.626198874s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:57.039146320 (6h44m10.626202531s) [upload=6913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:57.039148564 (6h44m10.626204785s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:57.092133855 (6h44m10.679190086s) [upload=6913]: Done uploading segments... Aug 7 23:10:57.092152090 (6h44m10.67920833s) [upload=6913]: Upload finished. err: <nil> Aug 7 23:11:47.690012005 (6h45m1.277068256s) [upload=6926]: Starting upload Aug 7 23:11:47.690127641 (6h45m1.277183882s) [upload=6926]: Done starting upload Aug 7 23:11:47.690277249 (6h45m1.27733349s) [upload=6926]: Uploading segments... Aug 7 23:11:47.690316773 (6h45m1.277373025s) [upload=6926]: Got next segment. Inline: true Aug 7 23:11:47.690360005 (6h45m1.277416245s) [upload=6926]: Waiting for error group managing segments... Aug 7 23:11:47.690405779 (6h45m1.277462031s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:47.690411880 (6h45m1.277468122s) [upload=6926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:47.690415508 (6h45m1.277471748s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:47.740982296 (6h45m1.328038537s) [upload=6926]: Done uploading segments... Aug 7 23:11:47.740995380 (6h45m1.328051622s) [upload=6926]: Upload finished. err: <nil> Aug 7 23:12:15.388374517 (6h45m28.975430749s) [upload=6935]: Starting upload Aug 7 23:12:15.388464206 (6h45m28.975520437s) [upload=6935]: Done starting upload Aug 7 23:12:15.388550818 (6h45m28.975607039s) [upload=6935]: Uploading segments... Aug 7 23:12:15.388651916 (6h45m28.975708127s) [upload=6935]: Got next segment. Inline: true Aug 7 23:12:15.388660222 (6h45m28.975716443s) [upload=6935]: Waiting for error group managing segments... Aug 7 23:12:15.388716747 (6h45m28.975772968s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:15.388723729 (6h45m28.97577995s) [upload=6935]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:15.388727838 (6h45m28.975784059s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:15.375897715 (6h46m28.962953947s) [upload=6935]: Done uploading segments... Aug 7 23:13:15.375942599 (6h46m28.962998841s) [upload=6935]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:15.452799710 (6h46m29.039855931s) [upload=6951]: Starting upload Aug 7 23:13:15.452856376 (6h46m29.039912597s) [upload=6951]: Done starting upload Aug 7 23:13:15.452901079 (6h46m29.0399573s) [upload=6951]: Uploading segments... Aug 7 23:13:15.452938739 (6h46m29.03999495s) [upload=6951]: Got next segment. Inline: true Aug 7 23:13:15.452943217 (6h46m29.039999439s) [upload=6951]: Waiting for error group managing segments... Aug 7 23:13:15.452974977 (6h46m29.040031188s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:15.452979305 (6h46m29.040035526s) [upload=6951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:15.452982110 (6h46m29.040038331s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:15.442587501 (6h47m29.029643742s) [upload=6951]: Done uploading segments... Aug 7 23:14:15.442608670 (6h47m29.029664922s) [upload=6951]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:15.516163727 (6h47m29.103219978s) [upload=6968]: Starting upload Aug 7 23:14:15.516314939 (6h47m29.10337124s) [upload=6968]: Done starting upload Aug 7 23:14:15.516411479 (6h47m29.103467721s) [upload=6968]: Uploading segments... Aug 7 23:14:15.516477332 (6h47m29.103533573s) [upload=6968]: Got next segment. Inline: true Aug 7 23:14:15.516487672 (6h47m29.103543913s) [upload=6968]: Waiting for error group managing segments... Aug 7 23:14:15.516556882 (6h47m29.103613162s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:15.516567169 (6h47m29.103623411s) [upload=6968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:15.516573491 (6h47m29.103629733s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:15.501845849 (6h48m29.08890207s) [upload=6968]: Done uploading segments... Aug 7 23:15:15.501865025 (6h48m29.088921247s) [upload=6968]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:15.591828711 (6h48m29.178884922s) [upload=6986]: Starting upload Aug 7 23:15:15.591883403 (6h48m29.178939613s) [upload=6986]: Done starting upload Aug 7 23:15:15.591959133 (6h48m29.179015344s) [upload=6986]: Uploading segments... Aug 7 23:15:15.592001994 (6h48m29.179058205s) [upload=6986]: Got next segment. Inline: true Aug 7 23:15:15.592022743 (6h48m29.179078964s) [upload=6986]: Waiting for error group managing segments... Aug 7 23:15:15.592060944 (6h48m29.179117156s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:15.592065653 (6h48m29.179121874s) [upload=6986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:15.592067957 (6h48m29.179124178s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:15.578451341 (6h49m29.165507562s) [upload=6986]: Done uploading segments... Aug 7 23:16:15.578469606 (6h49m29.165525816s) [upload=6986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:23.278937820 (6h49m36.865994061s) [upload=7004]: Starting upload Aug 7 23:16:23.279020784 (6h49m36.866077025s) [upload=7004]: Done starting upload Aug 7 23:16:23.279084963 (6h49m36.866141205s) [upload=7004]: Uploading segments... Aug 7 23:16:23.279128285 (6h49m36.866184526s) [upload=7004]: Got next segment. Inline: true Aug 7 23:16:23.279135057 (6h49m36.866191299s) [upload=7004]: Waiting for error group managing segments... Aug 7 23:16:23.279179039 (6h49m36.86623528s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:23.279184509 (6h49m36.866240751s) [upload=7004]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:23.279188327 (6h49m36.866244558s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:23.264696428 (6h50m36.851752649s) [upload=7004]: Done uploading segments... Aug 7 23:17:23.264712168 (6h50m36.851768379s) [upload=7004]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.354485739 (6h50m36.94154196s) [upload=7020]: Starting upload Aug 7 23:17:23.354537225 (6h50m36.941593446s) [upload=7020]: Done starting upload Aug 7 23:17:23.354582019 (6h50m36.94163824s) [upload=7020]: Uploading segments... Aug 7 23:17:23.354609931 (6h50m36.941666142s) [upload=7020]: Got next segment. Inline: true Aug 7 23:17:23.354613586 (6h50m36.941669807s) [upload=7020]: Waiting for error group managing segments... Aug 7 23:17:23.354647150 (6h50m36.941703371s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.354650656 (6h50m36.941706877s) [upload=7020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.354652961 (6h50m36.941709172s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:23.336455483 (6h51m36.923511723s) [upload=7020]: Done uploading segments... Aug 7 23:18:23.336475711 (6h51m36.923531952s) [upload=7020]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:34.755118608 (6h51m48.342174818s) [upload=7039]: Starting upload Aug 7 23:18:34.755161578 (6h51m48.342217799s) [upload=7039]: Done starting upload Aug 7 23:18:34.755200670 (6h51m48.342256881s) [upload=7039]: Uploading segments... Aug 7 23:18:34.755239954 (6h51m48.342296176s) [upload=7039]: Got next segment. Inline: true Aug 7 23:18:34.755243802 (6h51m48.342300023s) [upload=7039]: Waiting for error group managing segments... Aug 7 23:18:34.755274309 (6h51m48.34233052s) [upload=7039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:34.755277996 (6h51m48.342334207s) [upload=7039]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:34.755280250 (6h51m48.342336471s) [upload=7039]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:34.875891722 (6h51m48.462947943s) [upload=7039]: Done uploading segments... Aug 7 23:18:34.875937538 (6h51m48.462993759s) [upload=7039]: Upload finished. err: <nil> Aug 7 23:19:14.105729045 (6h52m27.692785266s) [upload=7050]: Starting upload Aug 7 23:19:14.105780641 (6h52m27.692836852s) [upload=7050]: Done starting upload Aug 7 23:19:14.105815075 (6h52m27.692871286s) [upload=7050]: Uploading segments... Aug 7 23:19:14.105851563 (6h52m27.692907773s) [upload=7050]: Got next segment. Inline: true Aug 7 23:19:14.105856422 (6h52m27.692912643s) [upload=7050]: Waiting for error group managing segments... Aug 7 23:19:14.105888662 (6h52m27.692944883s) [upload=7050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:14.105892429 (6h52m27.69294864s) [upload=7050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:14.105894724 (6h52m27.692950934s) [upload=7050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:14.092509460 (6h53m27.679565681s) [upload=7050]: Done uploading segments... Aug 7 23:20:14.092523326 (6h53m27.679579547s) [upload=7050]: Upload finished. err: metaclient: context deadline exceeded Duration: 172.486441ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 142.590338ms FAIL (2025-08-07 23:25:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 142.613432ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.........................................XXXX.X..X..X.......] duration: 261.879802ms FAIL (2025-08-07 23:25:46 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 261.902314ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................................XXXXXX.X.X........] duration: 1.808108765s FAIL (2025-08-07 23:25:41 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.808133412s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [............................................XXXXXX.X........] duration: 200.268489ms FAIL (2025-08-07 23:25:36 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.018385587 (6h43m49.605441827s) [upload=6905]: Upload finished. err: <nil> Aug 7 23:10:57.038963659 (6h44m10.62601988s) [upload=6913]: Starting upload Aug 7 23:10:57.039017149 (6h44m10.62607337s) [upload=6913]: Done starting upload Aug 7 23:10:57.039065159 (6h44m10.62612137s) [upload=6913]: Uploading segments... Aug 7 23:10:57.039093831 (6h44m10.626150042s) [upload=6913]: Got next segment. Inline: true Aug 7 23:10:57.039097308 (6h44m10.626153519s) [upload=6913]: Waiting for error group managing segments... Aug 7 23:10:57.039142663 (6h44m10.626198874s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:57.039146320 (6h44m10.626202531s) [upload=6913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:57.039148564 (6h44m10.626204785s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:57.092133855 (6h44m10.679190086s) [upload=6913]: Done uploading segments... Aug 7 23:10:57.092152090 (6h44m10.67920833s) [upload=6913]: Upload finished. err: <nil> Aug 7 23:11:47.690012005 (6h45m1.277068256s) [upload=6926]: Starting upload Aug 7 23:11:47.690127641 (6h45m1.277183882s) [upload=6926]: Done starting upload Aug 7 23:11:47.690277249 (6h45m1.27733349s) [upload=6926]: Uploading segments... Aug 7 23:11:47.690316773 (6h45m1.277373025s) [upload=6926]: Got next segment. Inline: true Aug 7 23:11:47.690360005 (6h45m1.277416245s) [upload=6926]: Waiting for error group managing segments... Aug 7 23:11:47.690405779 (6h45m1.277462031s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:47.690411880 (6h45m1.277468122s) [upload=6926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:47.690415508 (6h45m1.277471748s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:47.740982296 (6h45m1.328038537s) [upload=6926]: Done uploading segments... Aug 7 23:11:47.740995380 (6h45m1.328051622s) [upload=6926]: Upload finished. err: <nil> Aug 7 23:12:15.388374517 (6h45m28.975430749s) [upload=6935]: Starting upload Aug 7 23:12:15.388464206 (6h45m28.975520437s) [upload=6935]: Done starting upload Aug 7 23:12:15.388550818 (6h45m28.975607039s) [upload=6935]: Uploading segments... Aug 7 23:12:15.388651916 (6h45m28.975708127s) [upload=6935]: Got next segment. Inline: true Aug 7 23:12:15.388660222 (6h45m28.975716443s) [upload=6935]: Waiting for error group managing segments... Aug 7 23:12:15.388716747 (6h45m28.975772968s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:15.388723729 (6h45m28.97577995s) [upload=6935]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:15.388727838 (6h45m28.975784059s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:15.375897715 (6h46m28.962953947s) [upload=6935]: Done uploading segments... Aug 7 23:13:15.375942599 (6h46m28.962998841s) [upload=6935]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:15.452799710 (6h46m29.039855931s) [upload=6951]: Starting upload Aug 7 23:13:15.452856376 (6h46m29.039912597s) [upload=6951]: Done starting upload Aug 7 23:13:15.452901079 (6h46m29.0399573s) [upload=6951]: Uploading segments... Aug 7 23:13:15.452938739 (6h46m29.03999495s) [upload=6951]: Got next segment. Inline: true Aug 7 23:13:15.452943217 (6h46m29.039999439s) [upload=6951]: Waiting for error group managing segments... Aug 7 23:13:15.452974977 (6h46m29.040031188s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:15.452979305 (6h46m29.040035526s) [upload=6951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:15.452982110 (6h46m29.040038331s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:15.442587501 (6h47m29.029643742s) [upload=6951]: Done uploading segments... Aug 7 23:14:15.442608670 (6h47m29.029664922s) [upload=6951]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:15.516163727 (6h47m29.103219978s) [upload=6968]: Starting upload Aug 7 23:14:15.516314939 (6h47m29.10337124s) [upload=6968]: Done starting upload Aug 7 23:14:15.516411479 (6h47m29.103467721s) [upload=6968]: Uploading segments... Aug 7 23:14:15.516477332 (6h47m29.103533573s) [upload=6968]: Got next segment. Inline: true Aug 7 23:14:15.516487672 (6h47m29.103543913s) [upload=6968]: Waiting for error group managing segments... Aug 7 23:14:15.516556882 (6h47m29.103613162s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:15.516567169 (6h47m29.103623411s) [upload=6968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:15.516573491 (6h47m29.103629733s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:15.501845849 (6h48m29.08890207s) [upload=6968]: Done uploading segments... Aug 7 23:15:15.501865025 (6h48m29.088921247s) [upload=6968]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:15.591828711 (6h48m29.178884922s) [upload=6986]: Starting upload Aug 7 23:15:15.591883403 (6h48m29.178939613s) [upload=6986]: Done starting upload Aug 7 23:15:15.591959133 (6h48m29.179015344s) [upload=6986]: Uploading segments... Aug 7 23:15:15.592001994 (6h48m29.179058205s) [upload=6986]: Got next segment. Inline: true Aug 7 23:15:15.592022743 (6h48m29.179078964s) [upload=6986]: Waiting for error group managing segments... Aug 7 23:15:15.592060944 (6h48m29.179117156s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:15.592065653 (6h48m29.179121874s) [upload=6986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:15.592067957 (6h48m29.179124178s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:15.578451341 (6h49m29.165507562s) [upload=6986]: Done uploading segments... Aug 7 23:16:15.578469606 (6h49m29.165525816s) [upload=6986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:23.278937820 (6h49m36.865994061s) [upload=7004]: Starting upload Aug 7 23:16:23.279020784 (6h49m36.866077025s) [upload=7004]: Done starting upload Aug 7 23:16:23.279084963 (6h49m36.866141205s) [upload=7004]: Uploading segments... Aug 7 23:16:23.279128285 (6h49m36.866184526s) [upload=7004]: Got next segment. Inline: true Aug 7 23:16:23.279135057 (6h49m36.866191299s) [upload=7004]: Waiting for error group managing segments... Aug 7 23:16:23.279179039 (6h49m36.86623528s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:23.279184509 (6h49m36.866240751s) [upload=7004]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:23.279188327 (6h49m36.866244558s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:23.264696428 (6h50m36.851752649s) [upload=7004]: Done uploading segments... Aug 7 23:17:23.264712168 (6h50m36.851768379s) [upload=7004]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.354485739 (6h50m36.94154196s) [upload=7020]: Starting upload Aug 7 23:17:23.354537225 (6h50m36.941593446s) [upload=7020]: Done starting upload Aug 7 23:17:23.354582019 (6h50m36.94163824s) [upload=7020]: Uploading segments... Aug 7 23:17:23.354609931 (6h50m36.941666142s) [upload=7020]: Got next segment. Inline: true Aug 7 23:17:23.354613586 (6h50m36.941669807s) [upload=7020]: Waiting for error group managing segments... Aug 7 23:17:23.354647150 (6h50m36.941703371s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.354650656 (6h50m36.941706877s) [upload=7020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.354652961 (6h50m36.941709172s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:23.336455483 (6h51m36.923511723s) [upload=7020]: Done uploading segments... Aug 7 23:18:23.336475711 (6h51m36.923531952s) [upload=7020]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:34.755118608 (6h51m48.342174818s) [upload=7039]: Starting upload Aug 7 23:18:34.755161578 (6h51m48.342217799s) [upload=7039]: Done starting upload Aug 7 23:18:34.755200670 (6h51m48.342256881s) [upload=7039]: Uploading segments... Aug 7 23:18:34.755239954 (6h51m48.342296176s) [upload=7039]: Got next segment. Inline: true Aug 7 23:18:34.755243802 (6h51m48.342300023s) [upload=7039]: Waiting for error group managing segments... Aug 7 23:18:34.755274309 (6h51m48.34233052s) [upload=7039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:34.755277996 (6h51m48.342334207s) [upload=7039]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:34.755280250 (6h51m48.342336471s) [upload=7039]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:34.875891722 (6h51m48.462947943s) [upload=7039]: Done uploading segments... Aug 7 23:18:34.875937538 (6h51m48.462993759s) [upload=7039]: Upload finished. err: <nil> Aug 7 23:19:14.105729045 (6h52m27.692785266s) [upload=7050]: Starting upload Aug 7 23:19:14.105780641 (6h52m27.692836852s) [upload=7050]: Done starting upload Aug 7 23:19:14.105815075 (6h52m27.692871286s) [upload=7050]: Uploading segments... Aug 7 23:19:14.105851563 (6h52m27.692907773s) [upload=7050]: Got next segment. Inline: true Aug 7 23:19:14.105856422 (6h52m27.692912643s) [upload=7050]: Waiting for error group managing segments... Aug 7 23:19:14.105888662 (6h52m27.692944883s) [upload=7050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:14.105892429 (6h52m27.69294864s) [upload=7050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:14.105894724 (6h52m27.692950934s) [upload=7050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:14.092509460 (6h53m27.679565681s) [upload=7050]: Done uploading segments... Aug 7 23:20:14.092523326 (6h53m27.679579547s) [upload=7050]: Upload finished. err: metaclient: context deadline exceeded Duration: 200.340654ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................................XXX.X.X.......] duration: 1.261322901s FAIL (2025-08-07 23:25:34 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== A5NM4A, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.091791202 (6h51m59.678847453s) [noise=true , address=104.36.82.59:4307 , piece_id=DGJPGA6ETYMP34JBQE6WXC32PIXFWLUUKZNNWB2NXBWS727MJRPQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.092306323 (6h51m59.679362574s) [noise=true , address=104.36.82.3:4794 , piece_id=2M55ORW4IODVHZGVFIAOG6SDOHBT7NWX3X74ANGSUHIKY7ZLAFJA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.092699598 (6h51m59.679755839s) [noise=true , address=104.36.82.58:4299 , piece_id=25QYBX42LIC7HWREJDDQWECRZFNZV3PBM2JITVH6R52TGPDEMQDA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093097069 (6h51m59.68015331s) [noise=true , address=64.49.13.155:4922 , piece_id=XWJAU4RZXJAW2IM5CUGFI3MAK5GYEM6AMWMO6AO7PK6MYA47ABQQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093493069 (6h51m59.68054931s) [noise=true , address=104.36.82.9:4031 , piece_id=TRXII4JD2BKKTYJ3EVICMM7WGPELQNRU7Z7I23KKY2NFT4KN3JNA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093847771 (6h51m59.680904022s) [noise=true , address=104.36.82.7:4935 , piece_id=LVI53GXPJGQCYU5QXT2MRTPZJOQETBU3RUGVV2DGSN26NP6I3XVQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094203605 (6h51m59.681259855s) [noise=true , address=45.91.116.13:4171 , piece_id=UNI5ZZNWXKPIKQXVVWNE3S7FUGM2XWNO7PSJDRWGBSOERCY22OPQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094597400 (6h51m59.681653641s) [noise=true , address=45.91.116.9:4290 , piece_id=S3USZPGNHC7SX3HHEM6LBII54CGKRCQGLMQNRE7ZQXM3ZRLMJMSQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094954427 (6h51m59.682010668s) [noise=true , address=104.36.82.62:4439 , piece_id=P66ZY2SI3GROYBUTKW7HEEUYT6VD4WWY53PQEL43J25DQWHHGK5Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.095373879 (6h51m59.68243012s) [noise=true , address=104.36.82.46:4512 , piece_id=X5SF67DUK4DXJICZCZVYBVTDFSS3HHFHXUEYJMMOMSVFORXUBS2A, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.095742858 (6h51m59.6827991s) [noise=true , address=64.49.13.154:4930 , piece_id=4D74XBIPQHETEYR7WQSZHGFHW42EGNAVYDO5NSBKAFP44BK4JR2Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096084166 (6h51m59.683140377s) [noise=true , address=79.127.201.215:4350 , piece_id=WSQK2BLHIRJ4J6FH22IEV64A3VV7N7RFNR2WI2XJXWNMSW2H765Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096399684 (6h51m59.683455905s) [noise=true , address=104.36.82.61:4379 , piece_id=W2XB5LCHE7BIGVF53UOKTBS2E2ZQJ2FZHLRDUKQHPUENZQELPU2Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096721615 (6h51m59.683777826s) [noise=true , address=104.36.82.42:4815 , piece_id=NEGWCQ6XEFYUU5XIXZRYB5WVDRVF6ARBF6LMGC65AQHNGGJRJIXA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.097044478 (6h51m59.684100699s) [noise=true , address=104.36.82.16:4315 , piece_id=YDQ3SD75HCNENS667QTX2U5WCHQUFEEOX3WW6CXNVHKPJ7ZAE2YA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.097416412 (6h51m59.684472633s) [noise=true , address=104.36.82.9:4036 , piece_id=KSI3DLD6I3TZUKTH7KQTLLBEIMILZ2GUHVNP5JU6MWNVRIEARHGQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.172312403 (6h51m59.759368613s) [upload=7040]: Next returned nil segment Aug 7 23:18:46.172323192 (6h51m59.759379403s) [upload=7040]: Waiting for error group managing segments... Aug 7 23:18:46.182690655 (6h51m59.769746865s) [noise=true , address=45.91.116.9:4290 , piece_id=S3USZPGNHC7SX3HHEM6LBII54CGKRCQGLMQNRE7ZQXM3ZRLMJMSQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.188105241 (6h51m59.775161462s) [noise=true , address=104.36.82.7:4935 , piece_id=LVI53GXPJGQCYU5QXT2MRTPZJOQETBU3RUGVV2DGSN26NP6I3XVQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.196654078 (6h51m59.783710289s) [noise=true , address=45.91.116.13:4171 , piece_id=UNI5ZZNWXKPIKQXVVWNE3S7FUGM2XWNO7PSJDRWGBSOERCY22OPQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.198492931 (6h51m59.785549152s) [noise=true , address=104.36.82.58:4299 , piece_id=25QYBX42LIC7HWREJDDQWECRZFNZV3PBM2JITVH6R52TGPDEMQDA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.198779515 (6h51m59.785835726s) [noise=true , address=104.36.82.59:4307 , piece_id=DGJPGA6ETYMP34JBQE6WXC32PIXFWLUUKZNNWB2NXBWS727MJRPQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.204927762 (6h51m59.791983983s) [noise=true , address=104.36.82.52:4099 , piece_id=H2WVX26COCLZ7KH6AJLYBN4H6JRIUF5IXYED6DRSCINOVGA5NM4A, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.208251176 (6h51m59.795307387s) [noise=true , address=104.36.82.16:4315 , piece_id=YDQ3SD75HCNENS667QTX2U5WCHQUFEEOX3WW6CXNVHKPJ7ZAE2YA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.225240327 (6h51m59.812296549s) [noise=true , address=104.36.82.62:4439 , piece_id=P66ZY2SI3GROYBUTKW7HEEUYT6VD4WWY53PQEL43J25DQWHHGK5Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.225308985 (6h51m59.812365196s) [noise=true , address=104.36.82.3:4794 , piece_id=2M55ORW4IODVHZGVFIAOG6SDOHBT7NWX3X74ANGSUHIKY7ZLAFJA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.245212838 (6h51m59.832269059s) [noise=true , address=104.36.82.4:4818 , piece_id=U5SIEBGYEKIRZ4FD46ZT3L3AVH7MIVC3JRKPC7V46F6MGFT5QFXQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.262860569 (6h51m59.84991678s) [noise=true , address=104.36.82.9:4031 , piece_id=TRXII4JD2BKKTYJ3EVICMM7WGPELQNRU7Z7I23KKY2NFT4KN3JNA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.287460286 (6h51m59.874516498s) [noise=true , address=104.36.82.9:4036 , piece_id=KSI3DLD6I3TZUKTH7KQTLLBEIMILZ2GUHVNP5JU6MWNVRIEARHGQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.323021195 (6h51m59.910077436s) [noise=true , address=104.36.82.42:4815 , piece_id=NEGWCQ6XEFYUU5XIXZRYB5WVDRVF6ARBF6LMGC65AQHNGGJRJIXA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.337567646 (6h51m59.924623857s) [noise=true , address=79.127.201.215:4350 , piece_id=WSQK2BLHIRJ4J6FH22IEV64A3VV7N7RFNR2WI2XJXWNMSW2H765Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.337650712 (6h51m59.924706923s) [seg_pos={0 0}, upload=7040]: Segment reached optimal threshold of 14 pieces. Aug 7 23:18:46.339154849 (6h51m59.92621106s) [noise=true , address=104.36.82.61:4379 , piece_id=W2XB5LCHE7BIGVF53UOKTBS2E2ZQJ2FZHLRDUKQHPUENZQELPU2Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lp3H6533e4oUt4mbiWJMjLRi5fb2wQr7RrntSpRm9xqF1TcEi): protocol: rpc: dial tcp 104.36.82.61:4379: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339496567 (6h51m59.926552778s) [noise=true , address=64.49.13.154:4930 , piece_id=4D74XBIPQHETEYR7WQSZHGFHW42EGNAVYDO5NSBKAFP44BK4JR2Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kNK8y5et7MiiWNDT6iaC8YBAk3PX6WX1XSuU4QMSFBp3KymBb): protocol: read tcp 5.161.89.252:19812->64.49.13.154:4930: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339638873 (6h51m59.926695084s) [noise=true , address=104.36.82.46:4512 , piece_id=X5SF67DUK4DXJICZCZVYBVTDFSS3HHFHXUEYJMMOMSVFORXUBS2A, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2): protocol: read tcp 5.161.89.252:12938->104.36.82.46:4512: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339695828 (6h51m59.92675204s) [noise=true , address=64.49.13.155:4922 , piece_id=XWJAU4RZXJAW2IM5CUGFI3MAK5GYEM6AMWMO6AO7PK6MYA47ABQQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: read tcp 5.161.89.252:28596->64.49.13.155:4922: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339725374 (6h51m59.926781605s) [seg_pos={0 0}, upload=7040]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:18:46.339804101 (6h51m59.926860312s) [upload=7040]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:46.339836361 (6h51m59.926892572s) [upload=7040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:46.393522662 (6h51m59.980578873s) [upload=7040]: Done uploading segments... Aug 7 23:18:46.393540715 (6h51m59.980596927s) [upload=7040]: Upload finished. err: <nil> Aug 7 23:20:00.141045360 (6h53m13.728101581s) [upload=7062]: Starting upload Aug 7 23:20:00.141098730 (6h53m13.728154951s) [upload=7062]: Done starting upload Aug 7 23:20:00.141249572 (6h53m13.728305793s) [upload=7062]: Uploading segments... Aug 7 23:20:00.141576813 (6h53m13.728633034s) [upload=7062]: Got next segment. Inline: false Aug 7 23:20:00.141621376 (6h53m13.728677597s) [upload=7062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:00.141625283 (6h53m13.728681494s) [upload=7062]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:00.122447463 (6h54m13.709503674s) [upload=7062]: Next segment err: context deadline exceeded Aug 7 23:21:00.122599688 (6h54m13.709655909s) [upload=7062]: Done uploading segments... Aug 7 23:21:00.122624775 (6h54m13.709680986s) [upload=7062]: Upload finished. err: context deadline exceeded Duration: 1.261348919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 356.740798ms FAIL (2025-08-07 23:25:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 356.75873ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 942.070656ms FAIL (2025-08-07 23:25:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 942.09393ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [............................................X.XXXXX..X......] duration: 232.870203ms FAIL (2025-08-07 23:25:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 232.886463ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........................................XXXXXX.X.X.......] duration: 1.969121652s FAIL (2025-08-07 23:25:22 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.9691646s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................................X....XXX.X.....] duration: 587.723788ms FAIL (2025-08-07 23:25:21 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ite tcp 5.161.89.252:59232->104.36.82.13:4187: use of closed network connection Aug 7 23:17:27.468792451 (6h50m41.055848663s) [noise=true , address=104.36.82.15:4257 , piece_id=YSUUNUNXXR4XZZRQBZVTYSP7WZNQVDHHS7KVVRKIAY5AQI3EMGCQ, seg_pos={0 0}, upload=7021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: read tcp 5.161.89.252:16228->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:27.468805356 (6h50m41.055861567s) [seg_pos={0 0}, upload=7021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:17:27.468898489 (6h50m41.055954701s) [upload=7021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:27.468902838 (6h50m41.055959049s) [upload=7021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:27.512149772 (6h50m41.099205993s) [upload=7021]: Done uploading segments... Aug 7 23:17:27.512163628 (6h50m41.099219849s) [upload=7021]: Upload finished. err: <nil> Aug 7 23:17:51.618188257 (6h51m5.205244499s) [upload=7028]: Starting upload Aug 7 23:17:51.618315174 (6h51m5.205371416s) [upload=7028]: Done starting upload Aug 7 23:17:51.618432584 (6h51m5.205488824s) [upload=7028]: Uploading segments... Aug 7 23:17:51.618706434 (6h51m5.205762675s) [upload=7028]: Got next segment. Inline: false Aug 7 23:17:51.618753392 (6h51m5.205809634s) [upload=7028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:51.618759304 (6h51m5.205815554s) [upload=7028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:02.217185080 (6h51m15.804241291s) [seg_pos={0 0}, upload=7028]: Begin upload segment... Aug 7 23:18:02.217213693 (6h51m15.804269904s) [seg_pos={0 0}, upload=7028]: Done begin upload segment. Aug 7 23:18:02.217258727 (6h51m15.804314948s) [noise=true , address=45.91.116.7:4009 , piece_id=VMY27LD3MBT4O65CXJJEYBIP7QVB543GSJOTQLYBELK4G7DOBQRA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217278985 (6h51m15.804335206s) [noise=true , address=77.90.31.12:4592 , piece_id=X4RIDUL5RUENCF4ZVRNIBWIUAHGQYW6XLBZWDMOUYGEIWQL5ZH7A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217653263 (6h51m15.804709484s) [noise=true , address=104.36.82.11:4115 , piece_id=MIW4TCCVX7O3KTEBU2RQRHD4GKPBYQOBTDKXGLH3B3NYCKTBDQ3A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217754012 (6h51m15.804810233s) [noise=true , address=77.90.31.17:4772 , piece_id=A7XUBHQKTJRVAMBQAO24KOU4GFGENHMFY6DBETNJDZ7J5YR7VO6Q, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217977880 (6h51m15.805034101s) [noise=true , address=104.36.82.14:4222 , piece_id=BJCIBUJTQDIS64F2XTL6XYNMICZJUXBP2NUA3UXOCORROZWONL4Q, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218105467 (6h51m15.805161678s) [noise=true , address=77.90.31.16:4735 , piece_id=QBT3GKFTCJBCONV4W4E3V74IQBXZO2R4M3SQBJBGRPEXMN4L4V2A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218314438 (6h51m15.805370659s) [noise=true , address=45.91.116.10:4082 , piece_id=GCFQIVXLPEIWQXT4A5Y6Z4N2E54HKSLGB2QDXR62SYQ4AISIXIMA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218477351 (6h51m15.805533563s) [noise=true , address=45.91.116.6:4974 , piece_id=MQWUSYBCOBJPCBLKUWG5R2YGX2UXHJAYTFXLU5G45I7ADEEJOSIA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218633864 (6h51m15.805690076s) [noise=true , address=77.90.31.21:4877 , piece_id=C7HY6GFRIWXX43OOGKBM3EXJVIMPUQP36KHHSVSGOVFJYBBX4IBA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218759268 (6h51m15.80581548s) [noise=true , address=45.91.116.14:4190 , piece_id=VN7ZRT4NTNUNQNKN2OCNILBFMN2QNGZOOY6B6EFDPET3TIB42A7A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218945636 (6h51m15.806001857s) [noise=true , address=77.90.31.11:4553 , piece_id=SJUK6H5G242A3QOXRARMJGQP6IGRQ2WJOUHBK24BMT2M3DYKGKAQ, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219088874 (6h51m15.806145095s) [noise=true , address=77.90.31.19:4842 , piece_id=2K54EHW7ZB22YIUFSMKZBKD66T4NLCPLSAPR344HPTF7WPX537JA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219280120 (6h51m15.806336341s) [noise=true , address=104.36.82.3:4767 , piece_id=D6EYZVFHGH7J3ABUMJBQPTJUDNA6GPMONDLQY7WL6NTVLYEEZM4Q, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219439768 (6h51m15.806495989s) [noise=true , address=77.90.31.14:4662 , piece_id=FWDHERWXZUN4WAURKVGP6UC7PC57XBLLIDHJFAPEP6YUUHK7XSGA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219602472 (6h51m15.806658693s) [noise=true , address=77.90.31.18:4806 , piece_id=2L4OYP2PJ6LBOBZUUI52QNVFWAOMU7E25SJ7XB6C2CQCAREMFVQA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219943268 (6h51m15.806999489s) [noise=true , address=104.36.82.9:4006 , piece_id=3F6A7S5CWE2TOBI6R77DHSEIWGL3HBLYRWSJKC7PGTN747XL77NQ, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.309104326 (6h51m15.896160547s) [upload=7028]: Next returned nil segment Aug 7 23:18:02.309119103 (6h51m15.896175324s) [upload=7028]: Waiting for error group managing segments... Aug 7 23:18:02.326561371 (6h51m15.913617592s) [noise=true , address=45.91.116.14:4190 , piece_id=VN7ZRT4NTNUNQNKN2OCNILBFMN2QNGZOOY6B6EFDPET3TIB42A7A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.327084597 (6h51m15.914140818s) [noise=true , address=45.91.116.7:4009 , piece_id=VMY27LD3MBT4O65CXJJEYBIP7QVB543GSJOTQLYBELK4G7DOBQRA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.329365706 (6h51m15.916421916s) [noise=true , address=45.91.116.6:4974 , piece_id=MQWUSYBCOBJPCBLKUWG5R2YGX2UXHJAYTFXLU5G45I7ADEEJOSIA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.329710399 (6h51m15.916766611s) [noise=true , address=77.90.31.14:4662 , piece_id=FWDHERWXZUN4WAURKVGP6UC7PC57XBLLIDHJFAPEP6YUUHK7XSGA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.330040726 (6h51m15.917096936s) [noise=true , address=77.90.31.17:4772 , piece_id=A7XUBHQKTJRVAMBQAO24KOU4GFGENHMFY6DBETNJDZ7J5YR7VO6Q, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.330501876 (6h51m15.917558097s) [noise=true , address=77.90.31.18:4806 , piece_id=2L4OYP2PJ6LBOBZUUI52QNVFWAOMU7E25SJ7XB6C2CQCAREMFVQA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.330860886 (6h51m15.917917107s) [noise=true , address=77.90.31.16:4735 , piece_id=QBT3GKFTCJBCONV4W4E3V74IQBXZO2R4M3SQBJBGRPEXMN4L4V2A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.331216931 (6h51m15.918273152s) [noise=true , address=77.90.31.12:4592 , piece_id=X4RIDUL5RUENCF4ZVRNIBWIUAHGQYW6XLBZWDMOUYGEIWQL5ZH7A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.332064072 (6h51m15.919120293s) [noise=true , address=77.90.31.21:4877 , piece_id=C7HY6GFRIWXX43OOGKBM3EXJVIMPUQP36KHHSVSGOVFJYBBX4IBA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.332119265 (6h51m15.919175476s) [noise=true , address=77.90.31.11:4553 , piece_id=SJUK6H5G242A3QOXRARMJGQP6IGRQ2WJOUHBK24BMT2M3DYKGKAQ, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.332931742 (6h51m15.919987963s) [noise=true , address=104.36.82.3:4767 , piece_id=D6EYZVFHGH7J3ABUMJBQPTJUDNA6GPMONDLQY7WL6NTVLYEEZM4Q, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.340533561 (6h51m15.927589782s) [noise=true , address=45.91.116.10:4082 , piece_id=GCFQIVXLPEIWQXT4A5Y6Z4N2E54HKSLGB2QDXR62SYQ4AISIXIMA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.345624544 (6h51m15.932680765s) [noise=true , address=104.36.82.14:4222 , piece_id=BJCIBUJTQDIS64F2XTL6XYNMICZJUXBP2NUA3UXOCORROZWONL4Q, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.345641816 (6h51m15.932698037s) [seg_pos={0 0}, upload=7028]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:02.345736042 (6h51m15.932792253s) [noise=true , address=104.36.82.11:4115 , piece_id=MIW4TCCVX7O3KTEBU2RQRHD4GKPBYQOBTDKXGLH3B3NYCKTBDQ3A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:02.345762481 (6h51m15.932818693s) [noise=true , address=77.90.31.19:4842 , piece_id=2K54EHW7ZB22YIUFSMKZBKD66T4NLCPLSAPR344HPTF7WPX537JA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:02.345881584 (6h51m15.932937805s) [noise=true , address=104.36.82.9:4006 , piece_id=3F6A7S5CWE2TOBI6R77DHSEIWGL3HBLYRWSJKC7PGTN747XL77NQ, seg_pos={0 0}, upload=7028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: context canceled; context canceled Aug 7 23:18:02.345898606 (6h51m15.932954827s) [seg_pos={0 0}, upload=7028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:02.345971481 (6h51m15.933027702s) [upload=7028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:02.345975489 (6h51m15.93303171s) [upload=7028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:22:03.557058290 (6h55m17.144114511s) [upload=7028]: Done uploading segments... Aug 7 23:22:03.557078478 (6h55m17.144134699s) [upload=7028]: Upload finished. err: metaclient: context deadline exceeded Duration: 587.741481ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...........................................XXXXX..X.........] duration: 1.872970434s FAIL (2025-08-07 23:25:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.872996132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [..........................................XXXX.X..X..X......] duration: 267.995337ms FAIL (2025-08-07 23:25:18 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 268.024631ms core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................X.XX.X....X.X.....] duration: 579.359605ms FAIL (2025-08-07 23:25:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 579.384271ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [...........................................XXXXX.XX.........] duration: 253.95005ms FAIL (2025-08-07 23:25:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.277141003 (6h47m50.864197244s) [upload=6974]: Starting upload Aug 7 23:14:37.277320197 (6h47m50.864376439s) [upload=6974]: Done starting upload Aug 7 23:14:37.277421095 (6h47m50.864477337s) [upload=6974]: Uploading segments... Aug 7 23:14:37.277519849 (6h47m50.864576091s) [upload=6974]: Got next segment. Inline: true Aug 7 23:14:37.277527635 (6h47m50.864583875s) [upload=6974]: Waiting for error group managing segments... Aug 7 23:14:37.277578299 (6h47m50.86463454s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.277584181 (6h47m50.864640421s) [upload=6974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.277587807 (6h47m50.864644049s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.263997045 (6h48m20.851053287s) [upload=6974]: Done uploading segments... Aug 7 23:15:07.264036289 (6h48m20.85109253s) [upload=6974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:40.767391975 (6h48m54.354448226s) [upload=6993]: Starting upload Aug 7 23:15:40.767481722 (6h48m54.354537963s) [upload=6993]: Done starting upload Aug 7 23:15:40.767544320 (6h48m54.35460057s) [upload=6993]: Uploading segments... Aug 7 23:15:40.767585375 (6h48m54.354641617s) [upload=6993]: Got next segment. Inline: true Aug 7 23:15:40.767591397 (6h48m54.354647638s) [upload=6993]: Waiting for error group managing segments... Aug 7 23:15:40.767637002 (6h48m54.354693244s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:40.767642423 (6h48m54.354698664s) [upload=6993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:40.767646530 (6h48m54.354702782s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:10.754586120 (6h49m24.341642341s) [upload=6993]: Done uploading segments... Aug 7 23:16:10.754601419 (6h49m24.34165763s) [upload=6993]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:26.719060301 (6h49m40.306116542s) [upload=7005]: Starting upload Aug 7 23:16:26.719141191 (6h49m40.306197433s) [upload=7005]: Done starting upload Aug 7 23:16:26.719210051 (6h49m40.306266292s) [upload=7005]: Uploading segments... Aug 7 23:16:26.719285351 (6h49m40.306341592s) [upload=7005]: Got next segment. Inline: true Aug 7 23:16:26.719292685 (6h49m40.306348926s) [upload=7005]: Waiting for error group managing segments... Aug 7 23:16:26.719340825 (6h49m40.306397066s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:26.719347277 (6h49m40.306403518s) [upload=7005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:26.719351064 (6h49m40.306407305s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:26.802515608 (6h49m40.389571849s) [upload=7005]: Done uploading segments... Aug 7 23:16:26.802535986 (6h49m40.389592227s) [upload=7005]: Upload finished. err: <nil> Aug 7 23:16:42.191377275 (6h49m55.778433516s) [upload=7011]: Starting upload Aug 7 23:16:42.191466421 (6h49m55.778522662s) [upload=7011]: Done starting upload Aug 7 23:16:42.191531582 (6h49m55.778587803s) [upload=7011]: Uploading segments... Aug 7 23:16:42.191607003 (6h49m55.778663224s) [upload=7011]: Got next segment. Inline: true Aug 7 23:16:42.191612414 (6h49m55.778668635s) [upload=7011]: Waiting for error group managing segments... Aug 7 23:16:42.191651346 (6h49m55.778707556s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:42.191656575 (6h49m55.778712787s) [upload=7011]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:42.191659481 (6h49m55.778715702s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.173345160 (6h50m25.760401401s) [upload=7011]: Done uploading segments... Aug 7 23:17:12.173364576 (6h50m25.760420816s) [upload=7011]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:12.261509015 (6h50m25.848565257s) [upload=7016]: Starting upload Aug 7 23:17:12.261597881 (6h50m25.848654122s) [upload=7016]: Done starting upload Aug 7 23:17:12.261665168 (6h50m25.848721419s) [upload=7016]: Uploading segments... Aug 7 23:17:12.261708137 (6h50m25.848764388s) [upload=7016]: Got next segment. Inline: true Aug 7 23:17:12.261713487 (6h50m25.848769728s) [upload=7016]: Waiting for error group managing segments... Aug 7 23:17:12.261755746 (6h50m25.848811988s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:12.261761196 (6h50m25.848817448s) [upload=7016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:12.261764593 (6h50m25.848820834s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:42.241895867 (6h50m55.828952078s) [upload=7016]: Done uploading segments... Aug 7 23:17:42.241913931 (6h50m55.828970141s) [upload=7016]: Upload finished. err: metaclient: context deadline exceeded Duration: 253.976198ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [............................................XXXXX..X........] duration: 1.510918401s FAIL (2025-08-07 23:25:02 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.510942787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 957.065939ms FAIL (2025-08-07 23:24:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 957.084464ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [...........................................XXXX.X..X..X.....] duration: 183.478909ms FAIL (2025-08-07 23:24:52 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 183.500699ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [.............................................XXXXXX.X.......] duration: 171.994193ms FAIL (2025-08-07 23:24:41 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.018385587 (6h43m49.605441827s) [upload=6905]: Upload finished. err: <nil> Aug 7 23:10:57.038963659 (6h44m10.62601988s) [upload=6913]: Starting upload Aug 7 23:10:57.039017149 (6h44m10.62607337s) [upload=6913]: Done starting upload Aug 7 23:10:57.039065159 (6h44m10.62612137s) [upload=6913]: Uploading segments... Aug 7 23:10:57.039093831 (6h44m10.626150042s) [upload=6913]: Got next segment. Inline: true Aug 7 23:10:57.039097308 (6h44m10.626153519s) [upload=6913]: Waiting for error group managing segments... Aug 7 23:10:57.039142663 (6h44m10.626198874s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:57.039146320 (6h44m10.626202531s) [upload=6913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:57.039148564 (6h44m10.626204785s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:57.092133855 (6h44m10.679190086s) [upload=6913]: Done uploading segments... Aug 7 23:10:57.092152090 (6h44m10.67920833s) [upload=6913]: Upload finished. err: <nil> Aug 7 23:11:47.690012005 (6h45m1.277068256s) [upload=6926]: Starting upload Aug 7 23:11:47.690127641 (6h45m1.277183882s) [upload=6926]: Done starting upload Aug 7 23:11:47.690277249 (6h45m1.27733349s) [upload=6926]: Uploading segments... Aug 7 23:11:47.690316773 (6h45m1.277373025s) [upload=6926]: Got next segment. Inline: true Aug 7 23:11:47.690360005 (6h45m1.277416245s) [upload=6926]: Waiting for error group managing segments... Aug 7 23:11:47.690405779 (6h45m1.277462031s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:47.690411880 (6h45m1.277468122s) [upload=6926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:47.690415508 (6h45m1.277471748s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:47.740982296 (6h45m1.328038537s) [upload=6926]: Done uploading segments... Aug 7 23:11:47.740995380 (6h45m1.328051622s) [upload=6926]: Upload finished. err: <nil> Aug 7 23:12:15.388374517 (6h45m28.975430749s) [upload=6935]: Starting upload Aug 7 23:12:15.388464206 (6h45m28.975520437s) [upload=6935]: Done starting upload Aug 7 23:12:15.388550818 (6h45m28.975607039s) [upload=6935]: Uploading segments... Aug 7 23:12:15.388651916 (6h45m28.975708127s) [upload=6935]: Got next segment. Inline: true Aug 7 23:12:15.388660222 (6h45m28.975716443s) [upload=6935]: Waiting for error group managing segments... Aug 7 23:12:15.388716747 (6h45m28.975772968s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:15.388723729 (6h45m28.97577995s) [upload=6935]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:15.388727838 (6h45m28.975784059s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:15.375897715 (6h46m28.962953947s) [upload=6935]: Done uploading segments... Aug 7 23:13:15.375942599 (6h46m28.962998841s) [upload=6935]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:15.452799710 (6h46m29.039855931s) [upload=6951]: Starting upload Aug 7 23:13:15.452856376 (6h46m29.039912597s) [upload=6951]: Done starting upload Aug 7 23:13:15.452901079 (6h46m29.0399573s) [upload=6951]: Uploading segments... Aug 7 23:13:15.452938739 (6h46m29.03999495s) [upload=6951]: Got next segment. Inline: true Aug 7 23:13:15.452943217 (6h46m29.039999439s) [upload=6951]: Waiting for error group managing segments... Aug 7 23:13:15.452974977 (6h46m29.040031188s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:15.452979305 (6h46m29.040035526s) [upload=6951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:15.452982110 (6h46m29.040038331s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:15.442587501 (6h47m29.029643742s) [upload=6951]: Done uploading segments... Aug 7 23:14:15.442608670 (6h47m29.029664922s) [upload=6951]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:15.516163727 (6h47m29.103219978s) [upload=6968]: Starting upload Aug 7 23:14:15.516314939 (6h47m29.10337124s) [upload=6968]: Done starting upload Aug 7 23:14:15.516411479 (6h47m29.103467721s) [upload=6968]: Uploading segments... Aug 7 23:14:15.516477332 (6h47m29.103533573s) [upload=6968]: Got next segment. Inline: true Aug 7 23:14:15.516487672 (6h47m29.103543913s) [upload=6968]: Waiting for error group managing segments... Aug 7 23:14:15.516556882 (6h47m29.103613162s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:15.516567169 (6h47m29.103623411s) [upload=6968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:15.516573491 (6h47m29.103629733s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:15.501845849 (6h48m29.08890207s) [upload=6968]: Done uploading segments... Aug 7 23:15:15.501865025 (6h48m29.088921247s) [upload=6968]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:15.591828711 (6h48m29.178884922s) [upload=6986]: Starting upload Aug 7 23:15:15.591883403 (6h48m29.178939613s) [upload=6986]: Done starting upload Aug 7 23:15:15.591959133 (6h48m29.179015344s) [upload=6986]: Uploading segments... Aug 7 23:15:15.592001994 (6h48m29.179058205s) [upload=6986]: Got next segment. Inline: true Aug 7 23:15:15.592022743 (6h48m29.179078964s) [upload=6986]: Waiting for error group managing segments... Aug 7 23:15:15.592060944 (6h48m29.179117156s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:15.592065653 (6h48m29.179121874s) [upload=6986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:15.592067957 (6h48m29.179124178s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:15.578451341 (6h49m29.165507562s) [upload=6986]: Done uploading segments... Aug 7 23:16:15.578469606 (6h49m29.165525816s) [upload=6986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:23.278937820 (6h49m36.865994061s) [upload=7004]: Starting upload Aug 7 23:16:23.279020784 (6h49m36.866077025s) [upload=7004]: Done starting upload Aug 7 23:16:23.279084963 (6h49m36.866141205s) [upload=7004]: Uploading segments... Aug 7 23:16:23.279128285 (6h49m36.866184526s) [upload=7004]: Got next segment. Inline: true Aug 7 23:16:23.279135057 (6h49m36.866191299s) [upload=7004]: Waiting for error group managing segments... Aug 7 23:16:23.279179039 (6h49m36.86623528s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:23.279184509 (6h49m36.866240751s) [upload=7004]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:23.279188327 (6h49m36.866244558s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:23.264696428 (6h50m36.851752649s) [upload=7004]: Done uploading segments... Aug 7 23:17:23.264712168 (6h50m36.851768379s) [upload=7004]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.354485739 (6h50m36.94154196s) [upload=7020]: Starting upload Aug 7 23:17:23.354537225 (6h50m36.941593446s) [upload=7020]: Done starting upload Aug 7 23:17:23.354582019 (6h50m36.94163824s) [upload=7020]: Uploading segments... Aug 7 23:17:23.354609931 (6h50m36.941666142s) [upload=7020]: Got next segment. Inline: true Aug 7 23:17:23.354613586 (6h50m36.941669807s) [upload=7020]: Waiting for error group managing segments... Aug 7 23:17:23.354647150 (6h50m36.941703371s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.354650656 (6h50m36.941706877s) [upload=7020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.354652961 (6h50m36.941709172s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:23.336455483 (6h51m36.923511723s) [upload=7020]: Done uploading segments... Aug 7 23:18:23.336475711 (6h51m36.923531952s) [upload=7020]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:34.755118608 (6h51m48.342174818s) [upload=7039]: Starting upload Aug 7 23:18:34.755161578 (6h51m48.342217799s) [upload=7039]: Done starting upload Aug 7 23:18:34.755200670 (6h51m48.342256881s) [upload=7039]: Uploading segments... Aug 7 23:18:34.755239954 (6h51m48.342296176s) [upload=7039]: Got next segment. Inline: true Aug 7 23:18:34.755243802 (6h51m48.342300023s) [upload=7039]: Waiting for error group managing segments... Aug 7 23:18:34.755274309 (6h51m48.34233052s) [upload=7039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:34.755277996 (6h51m48.342334207s) [upload=7039]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:34.755280250 (6h51m48.342336471s) [upload=7039]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:34.875891722 (6h51m48.462947943s) [upload=7039]: Done uploading segments... Aug 7 23:18:34.875937538 (6h51m48.462993759s) [upload=7039]: Upload finished. err: <nil> Aug 7 23:19:14.105729045 (6h52m27.692785266s) [upload=7050]: Starting upload Aug 7 23:19:14.105780641 (6h52m27.692836852s) [upload=7050]: Done starting upload Aug 7 23:19:14.105815075 (6h52m27.692871286s) [upload=7050]: Uploading segments... Aug 7 23:19:14.105851563 (6h52m27.692907773s) [upload=7050]: Got next segment. Inline: true Aug 7 23:19:14.105856422 (6h52m27.692912643s) [upload=7050]: Waiting for error group managing segments... Aug 7 23:19:14.105888662 (6h52m27.692944883s) [upload=7050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:14.105892429 (6h52m27.69294864s) [upload=7050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:14.105894724 (6h52m27.692950934s) [upload=7050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:14.092509460 (6h53m27.679565681s) [upload=7050]: Done uploading segments... Aug 7 23:20:14.092523326 (6h53m27.679579547s) [upload=7050]: Upload finished. err: metaclient: context deadline exceeded Duration: 172.022014ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................................X....XXX.X....] duration: 547.963652ms FAIL (2025-08-07 23:24:32 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ite tcp 5.161.89.252:59232->104.36.82.13:4187: use of closed network connection Aug 7 23:17:27.468792451 (6h50m41.055848663s) [noise=true , address=104.36.82.15:4257 , piece_id=YSUUNUNXXR4XZZRQBZVTYSP7WZNQVDHHS7KVVRKIAY5AQI3EMGCQ, seg_pos={0 0}, upload=7021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: read tcp 5.161.89.252:16228->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:27.468805356 (6h50m41.055861567s) [seg_pos={0 0}, upload=7021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:17:27.468898489 (6h50m41.055954701s) [upload=7021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:27.468902838 (6h50m41.055959049s) [upload=7021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:27.512149772 (6h50m41.099205993s) [upload=7021]: Done uploading segments... Aug 7 23:17:27.512163628 (6h50m41.099219849s) [upload=7021]: Upload finished. err: <nil> Aug 7 23:17:51.618188257 (6h51m5.205244499s) [upload=7028]: Starting upload Aug 7 23:17:51.618315174 (6h51m5.205371416s) [upload=7028]: Done starting upload Aug 7 23:17:51.618432584 (6h51m5.205488824s) [upload=7028]: Uploading segments... Aug 7 23:17:51.618706434 (6h51m5.205762675s) [upload=7028]: Got next segment. Inline: false Aug 7 23:17:51.618753392 (6h51m5.205809634s) [upload=7028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:51.618759304 (6h51m5.205815554s) [upload=7028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:02.217185080 (6h51m15.804241291s) [seg_pos={0 0}, upload=7028]: Begin upload segment... Aug 7 23:18:02.217213693 (6h51m15.804269904s) [seg_pos={0 0}, upload=7028]: Done begin upload segment. Aug 7 23:18:02.217258727 (6h51m15.804314948s) [noise=true , address=45.91.116.7:4009 , piece_id=VMY27LD3MBT4O65CXJJEYBIP7QVB543GSJOTQLYBELK4G7DOBQRA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217278985 (6h51m15.804335206s) [noise=true , address=77.90.31.12:4592 , piece_id=X4RIDUL5RUENCF4ZVRNIBWIUAHGQYW6XLBZWDMOUYGEIWQL5ZH7A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217653263 (6h51m15.804709484s) [noise=true , address=104.36.82.11:4115 , piece_id=MIW4TCCVX7O3KTEBU2RQRHD4GKPBYQOBTDKXGLH3B3NYCKTBDQ3A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217754012 (6h51m15.804810233s) [noise=true , address=77.90.31.17:4772 , piece_id=A7XUBHQKTJRVAMBQAO24KOU4GFGENHMFY6DBETNJDZ7J5YR7VO6Q, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.217977880 (6h51m15.805034101s) [noise=true , address=104.36.82.14:4222 , piece_id=BJCIBUJTQDIS64F2XTL6XYNMICZJUXBP2NUA3UXOCORROZWONL4Q, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218105467 (6h51m15.805161678s) [noise=true , address=77.90.31.16:4735 , piece_id=QBT3GKFTCJBCONV4W4E3V74IQBXZO2R4M3SQBJBGRPEXMN4L4V2A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218314438 (6h51m15.805370659s) [noise=true , address=45.91.116.10:4082 , piece_id=GCFQIVXLPEIWQXT4A5Y6Z4N2E54HKSLGB2QDXR62SYQ4AISIXIMA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218477351 (6h51m15.805533563s) [noise=true , address=45.91.116.6:4974 , piece_id=MQWUSYBCOBJPCBLKUWG5R2YGX2UXHJAYTFXLU5G45I7ADEEJOSIA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218633864 (6h51m15.805690076s) [noise=true , address=77.90.31.21:4877 , piece_id=C7HY6GFRIWXX43OOGKBM3EXJVIMPUQP36KHHSVSGOVFJYBBX4IBA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218759268 (6h51m15.80581548s) [noise=true , address=45.91.116.14:4190 , piece_id=VN7ZRT4NTNUNQNKN2OCNILBFMN2QNGZOOY6B6EFDPET3TIB42A7A, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.218945636 (6h51m15.806001857s) [noise=true , address=77.90.31.11:4553 , piece_id=SJUK6H5G242A3QOXRARMJGQP6IGRQ2WJOUHBK24BMT2M3DYKGKAQ, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219088874 (6h51m15.806145095s) [noise=true , address=77.90.31.19:4842 , piece_id=2K54EHW7ZB22YIUFSMKZBKD66T4NLCPLSAPR344HPTF7WPX537JA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219280120 (6h51m15.806336341s) [noise=true , address=104.36.82.3:4767 , piece_id=D6EYZVFHGH7J3ABUMJBQPTJUDNA6GPMONDLQY7WL6NTVLYEEZM4Q, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219439768 (6h51m15.806495989s) [noise=true , address=77.90.31.14:4662 , piece_id=FWDHERWXZUN4WAURKVGP6UC7PC57XBLLIDHJFAPEP6YUUHK7XSGA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219602472 (6h51m15.806658693s) [noise=true , address=77.90.31.18:4806 , piece_id=2L4OYP2PJ6LBOBZUUI52QNVFWAOMU7E25SJ7XB6C2CQCAREMFVQA, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.219943268 (6h51m15.806999489s) [noise=true , address=104.36.82.9:4006 , piece_id=3F6A7S5CWE2TOBI6R77DHSEIWGL3HBLYRWSJKC7PGTN747XL77NQ, seg_pos={0 0}, upload=7028]: Uploading piece... Aug 7 23:18:02.309104326 (6h51m15.896160547s) [upload=7028]: Next returned nil segment Aug 7 23:18:02.309119103 (6h51m15.896175324s) [upload=7028]: Waiting for error group managing segments... Aug 7 23:18:02.326561371 (6h51m15.913617592s) [noise=true , address=45.91.116.14:4190 , piece_id=VN7ZRT4NTNUNQNKN2OCNILBFMN2QNGZOOY6B6EFDPET3TIB42A7A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.327084597 (6h51m15.914140818s) [noise=true , address=45.91.116.7:4009 , piece_id=VMY27LD3MBT4O65CXJJEYBIP7QVB543GSJOTQLYBELK4G7DOBQRA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.329365706 (6h51m15.916421916s) [noise=true , address=45.91.116.6:4974 , piece_id=MQWUSYBCOBJPCBLKUWG5R2YGX2UXHJAYTFXLU5G45I7ADEEJOSIA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.329710399 (6h51m15.916766611s) [noise=true , address=77.90.31.14:4662 , piece_id=FWDHERWXZUN4WAURKVGP6UC7PC57XBLLIDHJFAPEP6YUUHK7XSGA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.330040726 (6h51m15.917096936s) [noise=true , address=77.90.31.17:4772 , piece_id=A7XUBHQKTJRVAMBQAO24KOU4GFGENHMFY6DBETNJDZ7J5YR7VO6Q, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.330501876 (6h51m15.917558097s) [noise=true , address=77.90.31.18:4806 , piece_id=2L4OYP2PJ6LBOBZUUI52QNVFWAOMU7E25SJ7XB6C2CQCAREMFVQA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.330860886 (6h51m15.917917107s) [noise=true , address=77.90.31.16:4735 , piece_id=QBT3GKFTCJBCONV4W4E3V74IQBXZO2R4M3SQBJBGRPEXMN4L4V2A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.331216931 (6h51m15.918273152s) [noise=true , address=77.90.31.12:4592 , piece_id=X4RIDUL5RUENCF4ZVRNIBWIUAHGQYW6XLBZWDMOUYGEIWQL5ZH7A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.332064072 (6h51m15.919120293s) [noise=true , address=77.90.31.21:4877 , piece_id=C7HY6GFRIWXX43OOGKBM3EXJVIMPUQP36KHHSVSGOVFJYBBX4IBA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.332119265 (6h51m15.919175476s) [noise=true , address=77.90.31.11:4553 , piece_id=SJUK6H5G242A3QOXRARMJGQP6IGRQ2WJOUHBK24BMT2M3DYKGKAQ, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.332931742 (6h51m15.919987963s) [noise=true , address=104.36.82.3:4767 , piece_id=D6EYZVFHGH7J3ABUMJBQPTJUDNA6GPMONDLQY7WL6NTVLYEEZM4Q, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.340533561 (6h51m15.927589782s) [noise=true , address=45.91.116.10:4082 , piece_id=GCFQIVXLPEIWQXT4A5Y6Z4N2E54HKSLGB2QDXR62SYQ4AISIXIMA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.345624544 (6h51m15.932680765s) [noise=true , address=104.36.82.14:4222 , piece_id=BJCIBUJTQDIS64F2XTL6XYNMICZJUXBP2NUA3UXOCORROZWONL4Q, seg_pos={0 0}, upload=7028]: Done uploading piece. err: <nil> Aug 7 23:18:02.345641816 (6h51m15.932698037s) [seg_pos={0 0}, upload=7028]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:02.345736042 (6h51m15.932792253s) [noise=true , address=104.36.82.11:4115 , piece_id=MIW4TCCVX7O3KTEBU2RQRHD4GKPBYQOBTDKXGLH3B3NYCKTBDQ3A, seg_pos={0 0}, upload=7028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:02.345762481 (6h51m15.932818693s) [noise=true , address=77.90.31.19:4842 , piece_id=2K54EHW7ZB22YIUFSMKZBKD66T4NLCPLSAPR344HPTF7WPX537JA, seg_pos={0 0}, upload=7028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:02.345881584 (6h51m15.932937805s) [noise=true , address=104.36.82.9:4006 , piece_id=3F6A7S5CWE2TOBI6R77DHSEIWGL3HBLYRWSJKC7PGTN747XL77NQ, seg_pos={0 0}, upload=7028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: context canceled; context canceled Aug 7 23:18:02.345898606 (6h51m15.932954827s) [seg_pos={0 0}, upload=7028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:02.345971481 (6h51m15.933027702s) [upload=7028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:02.345975489 (6h51m15.93303171s) [upload=7028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:22:03.557058290 (6h55m17.144114511s) [upload=7028]: Done uploading segments... Aug 7 23:22:03.557078478 (6h55m17.144134699s) [upload=7028]: Upload finished. err: metaclient: context deadline exceeded Duration: 547.991984ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [............................................XXXXX.XX........] duration: 525.168536ms FAIL (2025-08-07 23:24:30 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.277141003 (6h47m50.864197244s) [upload=6974]: Starting upload Aug 7 23:14:37.277320197 (6h47m50.864376439s) [upload=6974]: Done starting upload Aug 7 23:14:37.277421095 (6h47m50.864477337s) [upload=6974]: Uploading segments... Aug 7 23:14:37.277519849 (6h47m50.864576091s) [upload=6974]: Got next segment. Inline: true Aug 7 23:14:37.277527635 (6h47m50.864583875s) [upload=6974]: Waiting for error group managing segments... Aug 7 23:14:37.277578299 (6h47m50.86463454s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.277584181 (6h47m50.864640421s) [upload=6974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.277587807 (6h47m50.864644049s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.263997045 (6h48m20.851053287s) [upload=6974]: Done uploading segments... Aug 7 23:15:07.264036289 (6h48m20.85109253s) [upload=6974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:40.767391975 (6h48m54.354448226s) [upload=6993]: Starting upload Aug 7 23:15:40.767481722 (6h48m54.354537963s) [upload=6993]: Done starting upload Aug 7 23:15:40.767544320 (6h48m54.35460057s) [upload=6993]: Uploading segments... Aug 7 23:15:40.767585375 (6h48m54.354641617s) [upload=6993]: Got next segment. Inline: true Aug 7 23:15:40.767591397 (6h48m54.354647638s) [upload=6993]: Waiting for error group managing segments... Aug 7 23:15:40.767637002 (6h48m54.354693244s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:40.767642423 (6h48m54.354698664s) [upload=6993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:40.767646530 (6h48m54.354702782s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:10.754586120 (6h49m24.341642341s) [upload=6993]: Done uploading segments... Aug 7 23:16:10.754601419 (6h49m24.34165763s) [upload=6993]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:26.719060301 (6h49m40.306116542s) [upload=7005]: Starting upload Aug 7 23:16:26.719141191 (6h49m40.306197433s) [upload=7005]: Done starting upload Aug 7 23:16:26.719210051 (6h49m40.306266292s) [upload=7005]: Uploading segments... Aug 7 23:16:26.719285351 (6h49m40.306341592s) [upload=7005]: Got next segment. Inline: true Aug 7 23:16:26.719292685 (6h49m40.306348926s) [upload=7005]: Waiting for error group managing segments... Aug 7 23:16:26.719340825 (6h49m40.306397066s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:26.719347277 (6h49m40.306403518s) [upload=7005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:26.719351064 (6h49m40.306407305s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:26.802515608 (6h49m40.389571849s) [upload=7005]: Done uploading segments... Aug 7 23:16:26.802535986 (6h49m40.389592227s) [upload=7005]: Upload finished. err: <nil> Aug 7 23:16:42.191377275 (6h49m55.778433516s) [upload=7011]: Starting upload Aug 7 23:16:42.191466421 (6h49m55.778522662s) [upload=7011]: Done starting upload Aug 7 23:16:42.191531582 (6h49m55.778587803s) [upload=7011]: Uploading segments... Aug 7 23:16:42.191607003 (6h49m55.778663224s) [upload=7011]: Got next segment. Inline: true Aug 7 23:16:42.191612414 (6h49m55.778668635s) [upload=7011]: Waiting for error group managing segments... Aug 7 23:16:42.191651346 (6h49m55.778707556s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:42.191656575 (6h49m55.778712787s) [upload=7011]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:42.191659481 (6h49m55.778715702s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.173345160 (6h50m25.760401401s) [upload=7011]: Done uploading segments... Aug 7 23:17:12.173364576 (6h50m25.760420816s) [upload=7011]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:12.261509015 (6h50m25.848565257s) [upload=7016]: Starting upload Aug 7 23:17:12.261597881 (6h50m25.848654122s) [upload=7016]: Done starting upload Aug 7 23:17:12.261665168 (6h50m25.848721419s) [upload=7016]: Uploading segments... Aug 7 23:17:12.261708137 (6h50m25.848764388s) [upload=7016]: Got next segment. Inline: true Aug 7 23:17:12.261713487 (6h50m25.848769728s) [upload=7016]: Waiting for error group managing segments... Aug 7 23:17:12.261755746 (6h50m25.848811988s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:12.261761196 (6h50m25.848817448s) [upload=7016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:12.261764593 (6h50m25.848820834s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:42.241895867 (6h50m55.828952078s) [upload=7016]: Done uploading segments... Aug 7 23:17:42.241913931 (6h50m55.828970141s) [upload=7016]: Upload finished. err: metaclient: context deadline exceeded Duration: 525.197521ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [............................................XXXX.X..X..X....] duration: 206.936205ms FAIL (2025-08-07 23:24:30 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 206.955421ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 513.072254ms FAIL (2025-08-07 23:24:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 513.115194ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................XXX.X.X......] duration: 859.892646ms FAIL (2025-08-07 23:24:27 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== A5NM4A, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.091791202 (6h51m59.678847453s) [noise=true , address=104.36.82.59:4307 , piece_id=DGJPGA6ETYMP34JBQE6WXC32PIXFWLUUKZNNWB2NXBWS727MJRPQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.092306323 (6h51m59.679362574s) [noise=true , address=104.36.82.3:4794 , piece_id=2M55ORW4IODVHZGVFIAOG6SDOHBT7NWX3X74ANGSUHIKY7ZLAFJA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.092699598 (6h51m59.679755839s) [noise=true , address=104.36.82.58:4299 , piece_id=25QYBX42LIC7HWREJDDQWECRZFNZV3PBM2JITVH6R52TGPDEMQDA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093097069 (6h51m59.68015331s) [noise=true , address=64.49.13.155:4922 , piece_id=XWJAU4RZXJAW2IM5CUGFI3MAK5GYEM6AMWMO6AO7PK6MYA47ABQQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093493069 (6h51m59.68054931s) [noise=true , address=104.36.82.9:4031 , piece_id=TRXII4JD2BKKTYJ3EVICMM7WGPELQNRU7Z7I23KKY2NFT4KN3JNA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093847771 (6h51m59.680904022s) [noise=true , address=104.36.82.7:4935 , piece_id=LVI53GXPJGQCYU5QXT2MRTPZJOQETBU3RUGVV2DGSN26NP6I3XVQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094203605 (6h51m59.681259855s) [noise=true , address=45.91.116.13:4171 , piece_id=UNI5ZZNWXKPIKQXVVWNE3S7FUGM2XWNO7PSJDRWGBSOERCY22OPQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094597400 (6h51m59.681653641s) [noise=true , address=45.91.116.9:4290 , piece_id=S3USZPGNHC7SX3HHEM6LBII54CGKRCQGLMQNRE7ZQXM3ZRLMJMSQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094954427 (6h51m59.682010668s) [noise=true , address=104.36.82.62:4439 , piece_id=P66ZY2SI3GROYBUTKW7HEEUYT6VD4WWY53PQEL43J25DQWHHGK5Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.095373879 (6h51m59.68243012s) [noise=true , address=104.36.82.46:4512 , piece_id=X5SF67DUK4DXJICZCZVYBVTDFSS3HHFHXUEYJMMOMSVFORXUBS2A, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.095742858 (6h51m59.6827991s) [noise=true , address=64.49.13.154:4930 , piece_id=4D74XBIPQHETEYR7WQSZHGFHW42EGNAVYDO5NSBKAFP44BK4JR2Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096084166 (6h51m59.683140377s) [noise=true , address=79.127.201.215:4350 , piece_id=WSQK2BLHIRJ4J6FH22IEV64A3VV7N7RFNR2WI2XJXWNMSW2H765Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096399684 (6h51m59.683455905s) [noise=true , address=104.36.82.61:4379 , piece_id=W2XB5LCHE7BIGVF53UOKTBS2E2ZQJ2FZHLRDUKQHPUENZQELPU2Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096721615 (6h51m59.683777826s) [noise=true , address=104.36.82.42:4815 , piece_id=NEGWCQ6XEFYUU5XIXZRYB5WVDRVF6ARBF6LMGC65AQHNGGJRJIXA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.097044478 (6h51m59.684100699s) [noise=true , address=104.36.82.16:4315 , piece_id=YDQ3SD75HCNENS667QTX2U5WCHQUFEEOX3WW6CXNVHKPJ7ZAE2YA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.097416412 (6h51m59.684472633s) [noise=true , address=104.36.82.9:4036 , piece_id=KSI3DLD6I3TZUKTH7KQTLLBEIMILZ2GUHVNP5JU6MWNVRIEARHGQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.172312403 (6h51m59.759368613s) [upload=7040]: Next returned nil segment Aug 7 23:18:46.172323192 (6h51m59.759379403s) [upload=7040]: Waiting for error group managing segments... Aug 7 23:18:46.182690655 (6h51m59.769746865s) [noise=true , address=45.91.116.9:4290 , piece_id=S3USZPGNHC7SX3HHEM6LBII54CGKRCQGLMQNRE7ZQXM3ZRLMJMSQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.188105241 (6h51m59.775161462s) [noise=true , address=104.36.82.7:4935 , piece_id=LVI53GXPJGQCYU5QXT2MRTPZJOQETBU3RUGVV2DGSN26NP6I3XVQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.196654078 (6h51m59.783710289s) [noise=true , address=45.91.116.13:4171 , piece_id=UNI5ZZNWXKPIKQXVVWNE3S7FUGM2XWNO7PSJDRWGBSOERCY22OPQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.198492931 (6h51m59.785549152s) [noise=true , address=104.36.82.58:4299 , piece_id=25QYBX42LIC7HWREJDDQWECRZFNZV3PBM2JITVH6R52TGPDEMQDA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.198779515 (6h51m59.785835726s) [noise=true , address=104.36.82.59:4307 , piece_id=DGJPGA6ETYMP34JBQE6WXC32PIXFWLUUKZNNWB2NXBWS727MJRPQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.204927762 (6h51m59.791983983s) [noise=true , address=104.36.82.52:4099 , piece_id=H2WVX26COCLZ7KH6AJLYBN4H6JRIUF5IXYED6DRSCINOVGA5NM4A, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.208251176 (6h51m59.795307387s) [noise=true , address=104.36.82.16:4315 , piece_id=YDQ3SD75HCNENS667QTX2U5WCHQUFEEOX3WW6CXNVHKPJ7ZAE2YA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.225240327 (6h51m59.812296549s) [noise=true , address=104.36.82.62:4439 , piece_id=P66ZY2SI3GROYBUTKW7HEEUYT6VD4WWY53PQEL43J25DQWHHGK5Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.225308985 (6h51m59.812365196s) [noise=true , address=104.36.82.3:4794 , piece_id=2M55ORW4IODVHZGVFIAOG6SDOHBT7NWX3X74ANGSUHIKY7ZLAFJA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.245212838 (6h51m59.832269059s) [noise=true , address=104.36.82.4:4818 , piece_id=U5SIEBGYEKIRZ4FD46ZT3L3AVH7MIVC3JRKPC7V46F6MGFT5QFXQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.262860569 (6h51m59.84991678s) [noise=true , address=104.36.82.9:4031 , piece_id=TRXII4JD2BKKTYJ3EVICMM7WGPELQNRU7Z7I23KKY2NFT4KN3JNA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.287460286 (6h51m59.874516498s) [noise=true , address=104.36.82.9:4036 , piece_id=KSI3DLD6I3TZUKTH7KQTLLBEIMILZ2GUHVNP5JU6MWNVRIEARHGQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.323021195 (6h51m59.910077436s) [noise=true , address=104.36.82.42:4815 , piece_id=NEGWCQ6XEFYUU5XIXZRYB5WVDRVF6ARBF6LMGC65AQHNGGJRJIXA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.337567646 (6h51m59.924623857s) [noise=true , address=79.127.201.215:4350 , piece_id=WSQK2BLHIRJ4J6FH22IEV64A3VV7N7RFNR2WI2XJXWNMSW2H765Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.337650712 (6h51m59.924706923s) [seg_pos={0 0}, upload=7040]: Segment reached optimal threshold of 14 pieces. Aug 7 23:18:46.339154849 (6h51m59.92621106s) [noise=true , address=104.36.82.61:4379 , piece_id=W2XB5LCHE7BIGVF53UOKTBS2E2ZQJ2FZHLRDUKQHPUENZQELPU2Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lp3H6533e4oUt4mbiWJMjLRi5fb2wQr7RrntSpRm9xqF1TcEi): protocol: rpc: dial tcp 104.36.82.61:4379: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339496567 (6h51m59.926552778s) [noise=true , address=64.49.13.154:4930 , piece_id=4D74XBIPQHETEYR7WQSZHGFHW42EGNAVYDO5NSBKAFP44BK4JR2Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kNK8y5et7MiiWNDT6iaC8YBAk3PX6WX1XSuU4QMSFBp3KymBb): protocol: read tcp 5.161.89.252:19812->64.49.13.154:4930: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339638873 (6h51m59.926695084s) [noise=true , address=104.36.82.46:4512 , piece_id=X5SF67DUK4DXJICZCZVYBVTDFSS3HHFHXUEYJMMOMSVFORXUBS2A, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2): protocol: read tcp 5.161.89.252:12938->104.36.82.46:4512: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339695828 (6h51m59.92675204s) [noise=true , address=64.49.13.155:4922 , piece_id=XWJAU4RZXJAW2IM5CUGFI3MAK5GYEM6AMWMO6AO7PK6MYA47ABQQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: read tcp 5.161.89.252:28596->64.49.13.155:4922: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339725374 (6h51m59.926781605s) [seg_pos={0 0}, upload=7040]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:18:46.339804101 (6h51m59.926860312s) [upload=7040]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:46.339836361 (6h51m59.926892572s) [upload=7040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:46.393522662 (6h51m59.980578873s) [upload=7040]: Done uploading segments... Aug 7 23:18:46.393540715 (6h51m59.980596927s) [upload=7040]: Upload finished. err: <nil> Aug 7 23:20:00.141045360 (6h53m13.728101581s) [upload=7062]: Starting upload Aug 7 23:20:00.141098730 (6h53m13.728154951s) [upload=7062]: Done starting upload Aug 7 23:20:00.141249572 (6h53m13.728305793s) [upload=7062]: Uploading segments... Aug 7 23:20:00.141576813 (6h53m13.728633034s) [upload=7062]: Got next segment. Inline: false Aug 7 23:20:00.141621376 (6h53m13.728677597s) [upload=7062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:00.141625283 (6h53m13.728681494s) [upload=7062]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:00.122447463 (6h54m13.709503674s) [upload=7062]: Next segment err: context deadline exceeded Aug 7 23:21:00.122599688 (6h54m13.709655909s) [upload=7062]: Done uploading segments... Aug 7 23:21:00.122624775 (6h54m13.709680986s) [upload=7062]: Upload finished. err: context deadline exceeded Duration: 859.909488ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.............................................X.XXXXX..X.....] duration: 459.644077ms FAIL (2025-08-07 23:24:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 459.664785ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................................X.XX.X....X.X....] duration: 519.499701ms FAIL (2025-08-07 23:24:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 519.521833ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............................................XXXXXX.X.X......] duration: 1.968949327s FAIL (2025-08-07 23:24:10 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.96897286s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................XXX.X.X.....] duration: 1.642315937s FAIL (2025-08-07 23:24:10 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== A5NM4A, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.091791202 (6h51m59.678847453s) [noise=true , address=104.36.82.59:4307 , piece_id=DGJPGA6ETYMP34JBQE6WXC32PIXFWLUUKZNNWB2NXBWS727MJRPQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.092306323 (6h51m59.679362574s) [noise=true , address=104.36.82.3:4794 , piece_id=2M55ORW4IODVHZGVFIAOG6SDOHBT7NWX3X74ANGSUHIKY7ZLAFJA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.092699598 (6h51m59.679755839s) [noise=true , address=104.36.82.58:4299 , piece_id=25QYBX42LIC7HWREJDDQWECRZFNZV3PBM2JITVH6R52TGPDEMQDA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093097069 (6h51m59.68015331s) [noise=true , address=64.49.13.155:4922 , piece_id=XWJAU4RZXJAW2IM5CUGFI3MAK5GYEM6AMWMO6AO7PK6MYA47ABQQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093493069 (6h51m59.68054931s) [noise=true , address=104.36.82.9:4031 , piece_id=TRXII4JD2BKKTYJ3EVICMM7WGPELQNRU7Z7I23KKY2NFT4KN3JNA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093847771 (6h51m59.680904022s) [noise=true , address=104.36.82.7:4935 , piece_id=LVI53GXPJGQCYU5QXT2MRTPZJOQETBU3RUGVV2DGSN26NP6I3XVQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094203605 (6h51m59.681259855s) [noise=true , address=45.91.116.13:4171 , piece_id=UNI5ZZNWXKPIKQXVVWNE3S7FUGM2XWNO7PSJDRWGBSOERCY22OPQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094597400 (6h51m59.681653641s) [noise=true , address=45.91.116.9:4290 , piece_id=S3USZPGNHC7SX3HHEM6LBII54CGKRCQGLMQNRE7ZQXM3ZRLMJMSQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094954427 (6h51m59.682010668s) [noise=true , address=104.36.82.62:4439 , piece_id=P66ZY2SI3GROYBUTKW7HEEUYT6VD4WWY53PQEL43J25DQWHHGK5Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.095373879 (6h51m59.68243012s) [noise=true , address=104.36.82.46:4512 , piece_id=X5SF67DUK4DXJICZCZVYBVTDFSS3HHFHXUEYJMMOMSVFORXUBS2A, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.095742858 (6h51m59.6827991s) [noise=true , address=64.49.13.154:4930 , piece_id=4D74XBIPQHETEYR7WQSZHGFHW42EGNAVYDO5NSBKAFP44BK4JR2Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096084166 (6h51m59.683140377s) [noise=true , address=79.127.201.215:4350 , piece_id=WSQK2BLHIRJ4J6FH22IEV64A3VV7N7RFNR2WI2XJXWNMSW2H765Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096399684 (6h51m59.683455905s) [noise=true , address=104.36.82.61:4379 , piece_id=W2XB5LCHE7BIGVF53UOKTBS2E2ZQJ2FZHLRDUKQHPUENZQELPU2Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096721615 (6h51m59.683777826s) [noise=true , address=104.36.82.42:4815 , piece_id=NEGWCQ6XEFYUU5XIXZRYB5WVDRVF6ARBF6LMGC65AQHNGGJRJIXA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.097044478 (6h51m59.684100699s) [noise=true , address=104.36.82.16:4315 , piece_id=YDQ3SD75HCNENS667QTX2U5WCHQUFEEOX3WW6CXNVHKPJ7ZAE2YA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.097416412 (6h51m59.684472633s) [noise=true , address=104.36.82.9:4036 , piece_id=KSI3DLD6I3TZUKTH7KQTLLBEIMILZ2GUHVNP5JU6MWNVRIEARHGQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.172312403 (6h51m59.759368613s) [upload=7040]: Next returned nil segment Aug 7 23:18:46.172323192 (6h51m59.759379403s) [upload=7040]: Waiting for error group managing segments... Aug 7 23:18:46.182690655 (6h51m59.769746865s) [noise=true , address=45.91.116.9:4290 , piece_id=S3USZPGNHC7SX3HHEM6LBII54CGKRCQGLMQNRE7ZQXM3ZRLMJMSQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.188105241 (6h51m59.775161462s) [noise=true , address=104.36.82.7:4935 , piece_id=LVI53GXPJGQCYU5QXT2MRTPZJOQETBU3RUGVV2DGSN26NP6I3XVQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.196654078 (6h51m59.783710289s) [noise=true , address=45.91.116.13:4171 , piece_id=UNI5ZZNWXKPIKQXVVWNE3S7FUGM2XWNO7PSJDRWGBSOERCY22OPQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.198492931 (6h51m59.785549152s) [noise=true , address=104.36.82.58:4299 , piece_id=25QYBX42LIC7HWREJDDQWECRZFNZV3PBM2JITVH6R52TGPDEMQDA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.198779515 (6h51m59.785835726s) [noise=true , address=104.36.82.59:4307 , piece_id=DGJPGA6ETYMP34JBQE6WXC32PIXFWLUUKZNNWB2NXBWS727MJRPQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.204927762 (6h51m59.791983983s) [noise=true , address=104.36.82.52:4099 , piece_id=H2WVX26COCLZ7KH6AJLYBN4H6JRIUF5IXYED6DRSCINOVGA5NM4A, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.208251176 (6h51m59.795307387s) [noise=true , address=104.36.82.16:4315 , piece_id=YDQ3SD75HCNENS667QTX2U5WCHQUFEEOX3WW6CXNVHKPJ7ZAE2YA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.225240327 (6h51m59.812296549s) [noise=true , address=104.36.82.62:4439 , piece_id=P66ZY2SI3GROYBUTKW7HEEUYT6VD4WWY53PQEL43J25DQWHHGK5Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.225308985 (6h51m59.812365196s) [noise=true , address=104.36.82.3:4794 , piece_id=2M55ORW4IODVHZGVFIAOG6SDOHBT7NWX3X74ANGSUHIKY7ZLAFJA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.245212838 (6h51m59.832269059s) [noise=true , address=104.36.82.4:4818 , piece_id=U5SIEBGYEKIRZ4FD46ZT3L3AVH7MIVC3JRKPC7V46F6MGFT5QFXQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.262860569 (6h51m59.84991678s) [noise=true , address=104.36.82.9:4031 , piece_id=TRXII4JD2BKKTYJ3EVICMM7WGPELQNRU7Z7I23KKY2NFT4KN3JNA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.287460286 (6h51m59.874516498s) [noise=true , address=104.36.82.9:4036 , piece_id=KSI3DLD6I3TZUKTH7KQTLLBEIMILZ2GUHVNP5JU6MWNVRIEARHGQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.323021195 (6h51m59.910077436s) [noise=true , address=104.36.82.42:4815 , piece_id=NEGWCQ6XEFYUU5XIXZRYB5WVDRVF6ARBF6LMGC65AQHNGGJRJIXA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.337567646 (6h51m59.924623857s) [noise=true , address=79.127.201.215:4350 , piece_id=WSQK2BLHIRJ4J6FH22IEV64A3VV7N7RFNR2WI2XJXWNMSW2H765Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.337650712 (6h51m59.924706923s) [seg_pos={0 0}, upload=7040]: Segment reached optimal threshold of 14 pieces. Aug 7 23:18:46.339154849 (6h51m59.92621106s) [noise=true , address=104.36.82.61:4379 , piece_id=W2XB5LCHE7BIGVF53UOKTBS2E2ZQJ2FZHLRDUKQHPUENZQELPU2Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lp3H6533e4oUt4mbiWJMjLRi5fb2wQr7RrntSpRm9xqF1TcEi): protocol: rpc: dial tcp 104.36.82.61:4379: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339496567 (6h51m59.926552778s) [noise=true , address=64.49.13.154:4930 , piece_id=4D74XBIPQHETEYR7WQSZHGFHW42EGNAVYDO5NSBKAFP44BK4JR2Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kNK8y5et7MiiWNDT6iaC8YBAk3PX6WX1XSuU4QMSFBp3KymBb): protocol: read tcp 5.161.89.252:19812->64.49.13.154:4930: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339638873 (6h51m59.926695084s) [noise=true , address=104.36.82.46:4512 , piece_id=X5SF67DUK4DXJICZCZVYBVTDFSS3HHFHXUEYJMMOMSVFORXUBS2A, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2): protocol: read tcp 5.161.89.252:12938->104.36.82.46:4512: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339695828 (6h51m59.92675204s) [noise=true , address=64.49.13.155:4922 , piece_id=XWJAU4RZXJAW2IM5CUGFI3MAK5GYEM6AMWMO6AO7PK6MYA47ABQQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: read tcp 5.161.89.252:28596->64.49.13.155:4922: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339725374 (6h51m59.926781605s) [seg_pos={0 0}, upload=7040]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:18:46.339804101 (6h51m59.926860312s) [upload=7040]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:46.339836361 (6h51m59.926892572s) [upload=7040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:46.393522662 (6h51m59.980578873s) [upload=7040]: Done uploading segments... Aug 7 23:18:46.393540715 (6h51m59.980596927s) [upload=7040]: Upload finished. err: <nil> Aug 7 23:20:00.141045360 (6h53m13.728101581s) [upload=7062]: Starting upload Aug 7 23:20:00.141098730 (6h53m13.728154951s) [upload=7062]: Done starting upload Aug 7 23:20:00.141249572 (6h53m13.728305793s) [upload=7062]: Uploading segments... Aug 7 23:20:00.141576813 (6h53m13.728633034s) [upload=7062]: Got next segment. Inline: false Aug 7 23:20:00.141621376 (6h53m13.728677597s) [upload=7062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:00.141625283 (6h53m13.728681494s) [upload=7062]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:00.122447463 (6h54m13.709503674s) [upload=7062]: Next segment err: context deadline exceeded Aug 7 23:21:00.122599688 (6h54m13.709655909s) [upload=7062]: Done uploading segments... Aug 7 23:21:00.122624775 (6h54m13.709680986s) [upload=7062]: Upload finished. err: context deadline exceeded Duration: 1.642355491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.288359108s FAIL (2025-08-07 23:24:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 2.288384454s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.............................................XXXXX..X.......] duration: 1.470113826s FAIL (2025-08-07 23:24:00 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.470133042s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..............................................XXXXX..X......] duration: 13.342320186s FAIL (2025-08-07 23:23:51 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.342339762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XXX.X.X....] duration: 1.135124811s FAIL (2025-08-07 23:23:50 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== A5NM4A, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.091791202 (6h51m59.678847453s) [noise=true , address=104.36.82.59:4307 , piece_id=DGJPGA6ETYMP34JBQE6WXC32PIXFWLUUKZNNWB2NXBWS727MJRPQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.092306323 (6h51m59.679362574s) [noise=true , address=104.36.82.3:4794 , piece_id=2M55ORW4IODVHZGVFIAOG6SDOHBT7NWX3X74ANGSUHIKY7ZLAFJA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.092699598 (6h51m59.679755839s) [noise=true , address=104.36.82.58:4299 , piece_id=25QYBX42LIC7HWREJDDQWECRZFNZV3PBM2JITVH6R52TGPDEMQDA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093097069 (6h51m59.68015331s) [noise=true , address=64.49.13.155:4922 , piece_id=XWJAU4RZXJAW2IM5CUGFI3MAK5GYEM6AMWMO6AO7PK6MYA47ABQQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093493069 (6h51m59.68054931s) [noise=true , address=104.36.82.9:4031 , piece_id=TRXII4JD2BKKTYJ3EVICMM7WGPELQNRU7Z7I23KKY2NFT4KN3JNA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.093847771 (6h51m59.680904022s) [noise=true , address=104.36.82.7:4935 , piece_id=LVI53GXPJGQCYU5QXT2MRTPZJOQETBU3RUGVV2DGSN26NP6I3XVQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094203605 (6h51m59.681259855s) [noise=true , address=45.91.116.13:4171 , piece_id=UNI5ZZNWXKPIKQXVVWNE3S7FUGM2XWNO7PSJDRWGBSOERCY22OPQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094597400 (6h51m59.681653641s) [noise=true , address=45.91.116.9:4290 , piece_id=S3USZPGNHC7SX3HHEM6LBII54CGKRCQGLMQNRE7ZQXM3ZRLMJMSQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.094954427 (6h51m59.682010668s) [noise=true , address=104.36.82.62:4439 , piece_id=P66ZY2SI3GROYBUTKW7HEEUYT6VD4WWY53PQEL43J25DQWHHGK5Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.095373879 (6h51m59.68243012s) [noise=true , address=104.36.82.46:4512 , piece_id=X5SF67DUK4DXJICZCZVYBVTDFSS3HHFHXUEYJMMOMSVFORXUBS2A, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.095742858 (6h51m59.6827991s) [noise=true , address=64.49.13.154:4930 , piece_id=4D74XBIPQHETEYR7WQSZHGFHW42EGNAVYDO5NSBKAFP44BK4JR2Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096084166 (6h51m59.683140377s) [noise=true , address=79.127.201.215:4350 , piece_id=WSQK2BLHIRJ4J6FH22IEV64A3VV7N7RFNR2WI2XJXWNMSW2H765Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096399684 (6h51m59.683455905s) [noise=true , address=104.36.82.61:4379 , piece_id=W2XB5LCHE7BIGVF53UOKTBS2E2ZQJ2FZHLRDUKQHPUENZQELPU2Q, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.096721615 (6h51m59.683777826s) [noise=true , address=104.36.82.42:4815 , piece_id=NEGWCQ6XEFYUU5XIXZRYB5WVDRVF6ARBF6LMGC65AQHNGGJRJIXA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.097044478 (6h51m59.684100699s) [noise=true , address=104.36.82.16:4315 , piece_id=YDQ3SD75HCNENS667QTX2U5WCHQUFEEOX3WW6CXNVHKPJ7ZAE2YA, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.097416412 (6h51m59.684472633s) [noise=true , address=104.36.82.9:4036 , piece_id=KSI3DLD6I3TZUKTH7KQTLLBEIMILZ2GUHVNP5JU6MWNVRIEARHGQ, seg_pos={0 0}, upload=7040]: Uploading piece... Aug 7 23:18:46.172312403 (6h51m59.759368613s) [upload=7040]: Next returned nil segment Aug 7 23:18:46.172323192 (6h51m59.759379403s) [upload=7040]: Waiting for error group managing segments... Aug 7 23:18:46.182690655 (6h51m59.769746865s) [noise=true , address=45.91.116.9:4290 , piece_id=S3USZPGNHC7SX3HHEM6LBII54CGKRCQGLMQNRE7ZQXM3ZRLMJMSQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.188105241 (6h51m59.775161462s) [noise=true , address=104.36.82.7:4935 , piece_id=LVI53GXPJGQCYU5QXT2MRTPZJOQETBU3RUGVV2DGSN26NP6I3XVQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.196654078 (6h51m59.783710289s) [noise=true , address=45.91.116.13:4171 , piece_id=UNI5ZZNWXKPIKQXVVWNE3S7FUGM2XWNO7PSJDRWGBSOERCY22OPQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.198492931 (6h51m59.785549152s) [noise=true , address=104.36.82.58:4299 , piece_id=25QYBX42LIC7HWREJDDQWECRZFNZV3PBM2JITVH6R52TGPDEMQDA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.198779515 (6h51m59.785835726s) [noise=true , address=104.36.82.59:4307 , piece_id=DGJPGA6ETYMP34JBQE6WXC32PIXFWLUUKZNNWB2NXBWS727MJRPQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.204927762 (6h51m59.791983983s) [noise=true , address=104.36.82.52:4099 , piece_id=H2WVX26COCLZ7KH6AJLYBN4H6JRIUF5IXYED6DRSCINOVGA5NM4A, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.208251176 (6h51m59.795307387s) [noise=true , address=104.36.82.16:4315 , piece_id=YDQ3SD75HCNENS667QTX2U5WCHQUFEEOX3WW6CXNVHKPJ7ZAE2YA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.225240327 (6h51m59.812296549s) [noise=true , address=104.36.82.62:4439 , piece_id=P66ZY2SI3GROYBUTKW7HEEUYT6VD4WWY53PQEL43J25DQWHHGK5Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.225308985 (6h51m59.812365196s) [noise=true , address=104.36.82.3:4794 , piece_id=2M55ORW4IODVHZGVFIAOG6SDOHBT7NWX3X74ANGSUHIKY7ZLAFJA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.245212838 (6h51m59.832269059s) [noise=true , address=104.36.82.4:4818 , piece_id=U5SIEBGYEKIRZ4FD46ZT3L3AVH7MIVC3JRKPC7V46F6MGFT5QFXQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.262860569 (6h51m59.84991678s) [noise=true , address=104.36.82.9:4031 , piece_id=TRXII4JD2BKKTYJ3EVICMM7WGPELQNRU7Z7I23KKY2NFT4KN3JNA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.287460286 (6h51m59.874516498s) [noise=true , address=104.36.82.9:4036 , piece_id=KSI3DLD6I3TZUKTH7KQTLLBEIMILZ2GUHVNP5JU6MWNVRIEARHGQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.323021195 (6h51m59.910077436s) [noise=true , address=104.36.82.42:4815 , piece_id=NEGWCQ6XEFYUU5XIXZRYB5WVDRVF6ARBF6LMGC65AQHNGGJRJIXA, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.337567646 (6h51m59.924623857s) [noise=true , address=79.127.201.215:4350 , piece_id=WSQK2BLHIRJ4J6FH22IEV64A3VV7N7RFNR2WI2XJXWNMSW2H765Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: <nil> Aug 7 23:18:46.337650712 (6h51m59.924706923s) [seg_pos={0 0}, upload=7040]: Segment reached optimal threshold of 14 pieces. Aug 7 23:18:46.339154849 (6h51m59.92621106s) [noise=true , address=104.36.82.61:4379 , piece_id=W2XB5LCHE7BIGVF53UOKTBS2E2ZQJ2FZHLRDUKQHPUENZQELPU2Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lp3H6533e4oUt4mbiWJMjLRi5fb2wQr7RrntSpRm9xqF1TcEi): protocol: rpc: dial tcp 104.36.82.61:4379: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339496567 (6h51m59.926552778s) [noise=true , address=64.49.13.154:4930 , piece_id=4D74XBIPQHETEYR7WQSZHGFHW42EGNAVYDO5NSBKAFP44BK4JR2Q, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kNK8y5et7MiiWNDT6iaC8YBAk3PX6WX1XSuU4QMSFBp3KymBb): protocol: read tcp 5.161.89.252:19812->64.49.13.154:4930: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339638873 (6h51m59.926695084s) [noise=true , address=104.36.82.46:4512 , piece_id=X5SF67DUK4DXJICZCZVYBVTDFSS3HHFHXUEYJMMOMSVFORXUBS2A, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2): protocol: read tcp 5.161.89.252:12938->104.36.82.46:4512: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339695828 (6h51m59.92675204s) [noise=true , address=64.49.13.155:4922 , piece_id=XWJAU4RZXJAW2IM5CUGFI3MAK5GYEM6AMWMO6AO7PK6MYA47ABQQ, seg_pos={0 0}, upload=7040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: read tcp 5.161.89.252:28596->64.49.13.155:4922: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:46.339725374 (6h51m59.926781605s) [seg_pos={0 0}, upload=7040]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:18:46.339804101 (6h51m59.926860312s) [upload=7040]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:46.339836361 (6h51m59.926892572s) [upload=7040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:46.393522662 (6h51m59.980578873s) [upload=7040]: Done uploading segments... Aug 7 23:18:46.393540715 (6h51m59.980596927s) [upload=7040]: Upload finished. err: <nil> Aug 7 23:20:00.141045360 (6h53m13.728101581s) [upload=7062]: Starting upload Aug 7 23:20:00.141098730 (6h53m13.728154951s) [upload=7062]: Done starting upload Aug 7 23:20:00.141249572 (6h53m13.728305793s) [upload=7062]: Uploading segments... Aug 7 23:20:00.141576813 (6h53m13.728633034s) [upload=7062]: Got next segment. Inline: false Aug 7 23:20:00.141621376 (6h53m13.728677597s) [upload=7062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:00.141625283 (6h53m13.728681494s) [upload=7062]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:00.122447463 (6h54m13.709503674s) [upload=7062]: Next segment err: context deadline exceeded Aug 7 23:21:00.122599688 (6h54m13.709655909s) [upload=7062]: Done uploading segments... Aug 7 23:21:00.122624775 (6h54m13.709680986s) [upload=7062]: Upload finished. err: context deadline exceeded Duration: 1.1351459s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [..............................................X.XXXXX..X....] duration: 205.620138ms FAIL (2025-08-07 23:23:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 205.67451ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)