Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................................XXXXX....................] duration:
307.754163ms
FAIL (2025-11-30 10:17:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 307.776686ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
406.707363ms
FAIL (2025-11-30 10:16:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 406.733502ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [...............................XXXXXXXX.....................] duration:
435.644361ms
FAIL (2025-11-30 10:16:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 435.662575ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
331.043541ms
FAIL (2025-11-30 10:16:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 331.060373ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.....................................XXXXX..................] duration:
574.417962ms
FAIL (2025-11-30 10:16:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 574.438582ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
309.214727ms
FAIL (2025-11-30 10:16:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 309.238101ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [......................................XXXXX.................] duration:
549.537592ms
FAIL (2025-11-30 10:16:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 549.557138ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
497.019246ms
FAIL (2025-11-30 10:15:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 497.035587ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
321.584299ms
FAIL (2025-11-30 10:15:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 321.607872ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................................XXXXX...................] duration:
418.248466ms
FAIL (2025-11-30 10:15:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 418.264076ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [................................XXXXXXXX....................] duration:
422.749514ms
FAIL (2025-11-30 10:15:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 422.765083ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.......................................XXXXX................] duration:
558.422793ms
FAIL (2025-11-30 10:15:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 558.443282ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................................XXXXX..................] duration:
363.988574ms
FAIL (2025-11-30 10:15:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 364.007238ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
330.833275ms
FAIL (2025-11-30 10:15:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 330.846861ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [.................................XXXXXXXX...................] duration:
469.761246ms
FAIL (2025-11-30 10:14:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 469.785641ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [........................................XXXXX...............] duration:
557.761864ms
FAIL (2025-11-30 10:14:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 557.781069ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
375.164677ms
FAIL (2025-11-30 10:14:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 375.191627ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................................XXXXX.................] duration:
332.435622ms
FAIL (2025-11-30 10:14:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 332.455619ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
313.30937ms
FAIL (2025-11-30 10:14:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 313.33556ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.........................................XXXXX..............] duration:
551.918109ms
FAIL (2025-11-30 10:14:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 551.969715ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
385.321236ms
FAIL (2025-11-30 10:13:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 385.343698ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [..................................XXXXXXXX..................] duration:
453.472328ms
FAIL (2025-11-30 10:13:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 453.490412ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
327.983737ms
FAIL (2025-11-30 10:13:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 328.025686ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [...................................XXXXXXXX.................] duration:
587.571905ms
FAIL (2025-11-30 10:13:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 587.590239ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................................XXXXX................] duration:
336.183285ms
FAIL (2025-11-30 10:13:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 336.200668ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
387.10822ms
FAIL (2025-11-30 10:13:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 387.131874ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
334.454049ms
FAIL (2025-11-30 10:13:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 334.473516ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..........................................XXXXX.............] duration:
777.182937ms
FAIL (2025-11-30 10:13:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 777.202082ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
427.610574ms
FAIL (2025-11-30 10:12:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 427.638087ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
314.953726ms
FAIL (2025-11-30 10:12:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 315.145045ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
387.249516ms
FAIL (2025-11-30 10:12:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 387.370834ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [....................................XXXXXXXX................] duration:
444.391424ms
FAIL (2025-11-30 10:12:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 444.409459ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [........................................XXXXX...............] duration:
426.459697ms
FAIL (2025-11-30 10:12:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 426.485194ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...........................................XXXXX............] duration:
556.905978ms
FAIL (2025-11-30 10:11:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 556.921128ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [.....................................XXXXXXXX...............] duration:
424.359454ms
FAIL (2025-11-30 10:11:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 424.377218ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
319.094537ms
FAIL (2025-11-30 10:11:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 319.109376ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........................................XXXXX..............] duration:
362.136778ms
FAIL (2025-11-30 10:11:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 362.225446ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
354.771375ms
FAIL (2025-11-30 10:11:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 354.785442ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [......................................XXXXXXXX..............] duration:
474.973355ms
FAIL (2025-11-30 10:11:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 474.994545ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
329.129428ms
FAIL (2025-11-30 10:11:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 329.154384ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........................................XXXXX.............] duration:
394.339931ms
FAIL (2025-11-30 10:10:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 394.364306ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [.......................................XXXXXXXX.............] duration:
500.514816ms
FAIL (2025-11-30 10:10:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 500.530605ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [............................................XXXXX...........] duration:
629.575029ms
FAIL (2025-11-30 10:10:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 629.672491ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........................................XXXXX............] duration:
359.512124ms
FAIL (2025-11-30 10:10:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 359.537431ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
323.000165ms
FAIL (2025-11-30 10:10:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 323.014051ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.............................................XXXXX..........] duration:
532.717526ms
FAIL (2025-11-30 10:10:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 532.73029ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
312.742927ms
FAIL (2025-11-30 10:10:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 312.762494ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [............................................XXXXX...........] duration:
360.371296ms
FAIL (2025-11-30 10:09:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 360.395751ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
357.894177ms
FAIL (2025-11-30 10:09:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 357.91186ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [........................................XXXXXXXX............] duration:
471.697188ms
FAIL (2025-11-30 10:09:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 471.717005ms
core
uplink
us-select-1
us1-placement-20
|