Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.................XXXXX......................................] duration:
578.709011ms
FAIL (2025-11-30 10:32:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 578.729529ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
362.928945ms
FAIL (2025-11-30 10:31:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 362.945916ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..................XXXXX.....................................] duration:
533.849807ms
FAIL (2025-11-30 10:31:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 533.873181ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [............XXXXXXXX........................................] duration:
468.72311ms
FAIL (2025-11-30 10:31:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 468.740432ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [............XXXXX...........................................] duration:
831.26561ms
FAIL (2025-11-30 10:31:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 831.288322ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
289.606797ms
FAIL (2025-11-30 10:31:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 289.626954ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [.............XXXXXXXX.......................................] duration:
531.622015ms
FAIL (2025-11-30 10:31:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 531.639608ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
304.343484ms
FAIL (2025-11-30 10:30:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 304.363151ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [..............XXXXXXXX......................................] duration:
413.929583ms
FAIL (2025-11-30 10:30:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 413.945754ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............XXXXX..........................................] duration:
305.053655ms
FAIL (2025-11-30 10:30:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 305.07274ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...................XXXXX....................................] duration:
529.29957ms
FAIL (2025-11-30 10:30:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 529.315881ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
377.273716ms
FAIL (2025-11-30 10:30:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 377.309402ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [...............XXXXXXXX.....................................] duration:
448.149739ms
FAIL (2025-11-30 10:30:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 448.167373ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
318.965253ms
FAIL (2025-11-30 10:30:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 318.992443ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............XXXXX.........................................] duration:
810.350579ms
FAIL (2025-11-30 10:29:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 810.372179ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
401.797065ms
FAIL (2025-11-30 10:29:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 401.861607ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [....................XXXXX...................................] duration:
561.891831ms
FAIL (2025-11-30 10:29:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 562.094622ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
364.927606ms
FAIL (2025-11-30 10:29:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 364.950559ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [................XXXXXXXX....................................] duration:
477.229533ms
FAIL (2025-11-30 10:29:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 477.417095ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............XXXXX........................................] duration:
355.185684ms
FAIL (2025-11-30 10:29:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 355.206464ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.....................XXXXX..................................] duration:
545.784955ms
FAIL (2025-11-30 10:29:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 545.807949ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
309.380305ms
FAIL (2025-11-30 10:28:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 309.399992ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
316.802565ms
FAIL (2025-11-30 10:28:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 316.825528ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [.................XXXXXXXX...................................] duration:
433.445234ms
FAIL (2025-11-30 10:28:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 433.461565ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
337.340406ms
FAIL (2025-11-30 10:28:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 337.356557ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [................XXXXX.......................................] duration:
354.424122ms
FAIL (2025-11-30 10:28:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 354.448329ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [......................XXXXX.................................] duration:
595.739429ms
FAIL (2025-11-30 10:28:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 595.753896ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [..................XXXXXXXX..................................] duration:
423.547392ms
FAIL (2025-11-30 10:28:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 423.560206ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................XXXXX......................................] duration:
352.298496ms
FAIL (2025-11-30 10:27:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 352.320127ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
320.720496ms
FAIL (2025-11-30 10:27:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 320.739171ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.......................XXXXX................................] duration:
750.116655ms
FAIL (2025-11-30 10:27:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 750.18407ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................XXXXX.....................................] duration:
439.880272ms
FAIL (2025-11-30 10:27:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 439.904027ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
312.414937ms
FAIL (2025-11-30 10:27:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 312.433152ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
324.776085ms
FAIL (2025-11-30 10:27:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 324.793107ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................XXXXX....................................] duration:
845.917333ms
FAIL (2025-11-30 10:27:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 845.946439ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
303.666224ms
FAIL (2025-11-30 10:26:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 303.694686ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [...................XXXXXXXX.................................] duration:
404.90626ms
FAIL (2025-11-30 10:26:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 404.926809ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [........................XXXXX...............................] duration:
543.983292ms
FAIL (2025-11-30 10:26:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 543.999963ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................XXXXX...................................] duration:
532.302054ms
FAIL (2025-11-30 10:26:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 532.315197ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
379.612805ms
FAIL (2025-11-30 10:26:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 379.630588ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [....................XXXXXXXX................................] duration:
407.216466ms
FAIL (2025-11-30 10:26:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 407.235621ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
360.611304ms
FAIL (2025-11-30 10:26:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 360.635331ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................XXXXX..................................] duration:
820.665637ms
FAIL (2025-11-30 10:26:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 820.692698ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [.....................XXXXXXXX...............................] duration:
421.903598ms
FAIL (2025-11-30 10:26:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 422.041367ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.........................XXXXX..............................] duration:
609.117974ms
FAIL (2025-11-30 10:25:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 609.137831ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................XXXXX.................................] duration:
341.809172ms
FAIL (2025-11-30 10:25:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 341.829471ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
338.688095ms
FAIL (2025-11-30 10:25:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 338.712369ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..........................XXXXX.............................] duration:
557.170629ms
FAIL (2025-11-30 10:25:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 557.188913ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [......................XXXXXXXX..............................] duration:
638.425595ms
FAIL (2025-11-30 10:25:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 638.442526ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
348.969539ms
FAIL (2025-11-30 10:25:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 348.98647ms
edge
gateway
us-select-1
|