Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..............................................XXXXX.........] duration:
593.165774ms
FAIL (2025-11-30 10:09:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 593.189398ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
454.245649ms
FAIL (2025-11-30 10:09:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 454.271426ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............................................XXXXX..........] duration:
399.091227ms
FAIL (2025-11-30 10:09:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 399.112567ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
346.595956ms
FAIL (2025-11-30 10:08:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 346.61961ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...............................................XXXXX........] duration:
562.482824ms
FAIL (2025-11-30 10:08:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 562.500337ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
332.079393ms
FAIL (2025-11-30 10:08:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 332.11463ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............................................XXXXX.........] duration:
340.195255ms
FAIL (2025-11-30 10:08:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 340.228827ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [.........................................XXXXXXXX...........] duration:
469.620731ms
FAIL (2025-11-30 10:08:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 469.632113ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
340.670687ms
FAIL (2025-11-30 10:08:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 340.690934ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [................................................XXXXX.......] duration:
555.022143ms
FAIL (2025-11-30 10:07:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 555.034837ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
352.7164ms
FAIL (2025-11-30 10:07:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 352.735626ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [..........................................XXXXXXXX..........] duration:
444.172155ms
FAIL (2025-11-30 10:07:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 444.195437ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............................................XXXXX........] duration:
412.254097ms
FAIL (2025-11-30 10:07:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 412.27687ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
339.122961ms
FAIL (2025-11-30 10:07:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 339.142358ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
449.702742ms
FAIL (2025-11-30 10:07:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 449.718262ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [................................................XXXXX.......] duration:
383.679935ms
FAIL (2025-11-30 10:06:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 383.707757ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.................................................XXXXX......] duration:
626.350937ms
FAIL (2025-11-30 10:06:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 626.406753ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [...........................................XXXXXXXX.........] duration:
450.061807ms
FAIL (2025-11-30 10:06:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 450.077968ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................................................XXXXX......] duration:
357.621126ms
FAIL (2025-11-30 10:06:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 357.646522ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
376.800215ms
FAIL (2025-11-30 10:06:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 376.823218ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
351.683883ms
FAIL (2025-11-30 10:06:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 351.699913ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [............................................XXXXXXXX........] duration:
448.147805ms
FAIL (2025-11-30 10:06:18 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 448.162482ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..................................................XXXXX.....] duration:
557.363016ms
FAIL (2025-11-30 10:05:49 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 557.385448ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [.............................................XXXXXXXX.......] duration:
444.800553ms
FAIL (2025-11-30 10:05:46 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 444.814499ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
345.60188ms
FAIL (2025-11-30 10:05:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 345.68189ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................................................XXXXX.....] duration:
323.510604ms
FAIL (2025-11-30 10:05:30 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 323.527837ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [..............................................XXXXXXXX......] duration:
444.450017ms
FAIL (2025-11-30 10:05:28 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 444.466558ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
317.473945ms
FAIL (2025-11-30 10:05:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 317.490857ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...................................................XXXXX....] duration:
575.257766ms
FAIL (2025-11-30 10:05:11 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 575.277704ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................................................XXXXX....] duration:
364.670974ms
FAIL (2025-11-30 10:04:49 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 364.693236ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [...............................................XXXXXXXX.....] duration:
462.513554ms
FAIL (2025-11-30 10:04:45 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 462.533591ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
404.566741ms
FAIL (2025-11-30 10:04:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 404.588291ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [................................................XXXXXXXX....] duration:
408.863554ms
FAIL (2025-11-30 10:04:28 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 408.878312ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [....................................................XXXXX...] duration:
567.358473ms
FAIL (2025-11-30 10:04:28 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 567.374523ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
331.132426ms
FAIL (2025-11-30 10:04:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 331.15042ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................................................XXXXX...] duration:
343.169175ms
FAIL (2025-11-30 10:04:16 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 343.183191ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
384.315619ms
FAIL (2025-11-30 10:04:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 384.332249ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [.................................................XXXXXXXX...] duration:
444.956597ms
FAIL (2025-11-30 10:04:09 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 444.983477ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
386.504498ms
FAIL (2025-11-30 10:03:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 386.547628ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [..................................................XXXXXXXX..] duration:
439.906119ms
FAIL (2025-11-30 10:03:36 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 440.013221ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................................................XXXXX..] duration:
356.637879ms
FAIL (2025-11-30 10:03:35 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 356.662316ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
316.549429ms
FAIL (2025-11-30 10:03:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 316.569716ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.....................................................XXXXX..] duration:
566.711948ms
FAIL (2025-11-30 10:03:25 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 566.732538ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [......................................................XXXXX.] duration:
16.401872431s
FAIL (2025-11-30 10:03:09 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 16.401884464s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
343.894846ms
FAIL (2025-11-30 10:03:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 343.911547ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................................................XXXXX.] duration:
23.636338161s
FAIL (2025-11-30 10:03:05 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 23.636352838s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [...................................................XXXXXXXX.] duration:
1.462128842s
FAIL (2025-11-30 10:03:04 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.462142208s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
1m0.001348231s
FAIL (2025-11-30 10:02:52 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001361685s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration:
5.001507517s
FAIL (2025-11-30 10:02:46 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001528957s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
313.217798ms
FAIL (2025-11-30 10:02:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 313.230312ms
edge
gateway
us-select-1
|