Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
344.927585ms
FAIL (2025-11-30 10:39:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 344.948083ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
312.756113ms
FAIL (2025-11-30 10:39:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 312.776271ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
304.90565ms
FAIL (2025-11-30 10:39:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 304.925277ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.XXXXX......................................................] duration:
321.696229ms
FAIL (2025-11-30 10:39:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 321.710837ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [...XXXXXXXX.................................................] duration:
422.987735ms
FAIL (2025-11-30 10:39:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 423.008234ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
313.273022ms
FAIL (2025-11-30 10:38:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 313.290104ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [........XXXXX...............................................] duration:
748.750502ms
FAIL (2025-11-30 10:38:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 748.7711ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [....XXXXXXXX................................................] duration:
432.242347ms
FAIL (2025-11-30 10:38:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 432.263927ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
393.394767ms
FAIL (2025-11-30 10:38:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 393.41762ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..XXXXX.....................................................] duration:
314.601886ms
FAIL (2025-11-30 10:38:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 314.621554ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
320.065229ms
FAIL (2025-11-30 10:38:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 320.084425ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [.....XXXXXXXX...............................................] duration:
663.371702ms
FAIL (2025-11-30 10:37:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 663.389766ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...XXXXX....................................................] duration:
389.417014ms
FAIL (2025-11-30 10:37:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 389.436531ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
386.622921ms
FAIL (2025-11-30 10:37:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 386.644371ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.........XXXXX..............................................] duration:
544.03763ms
FAIL (2025-11-30 10:37:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 544.062607ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
386.787229ms
FAIL (2025-11-30 10:37:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 386.807356ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [......XXXXXXXX..............................................] duration:
454.759664ms
FAIL (2025-11-30 10:37:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 454.774111ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
320.413151ms
FAIL (2025-11-30 10:36:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 320.437487ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
447.453705ms
FAIL (2025-11-30 10:36:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 447.478912ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....XXXXX...................................................] duration:
311.772516ms
FAIL (2025-11-30 10:36:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 311.79043ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..........XXXXX.............................................] duration:
558.51031ms
FAIL (2025-11-30 10:36:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 558.536851ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
367.146682ms
FAIL (2025-11-30 10:36:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 367.16707ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
319.972604ms
FAIL (2025-11-30 10:36:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 319.984537ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [.......XXXXXXXX.............................................] duration:
431.874591ms
FAIL (2025-11-30 10:35:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 431.903335ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....XXXXX..................................................] duration:
680.600224ms
FAIL (2025-11-30 10:35:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 680.621703ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
388.305567ms
FAIL (2025-11-30 10:35:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 388.332548ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...........XXXXX............................................] duration:
559.498283ms
FAIL (2025-11-30 10:35:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 559.519524ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [........XXXXXXXX............................................] duration:
425.775161ms
FAIL (2025-11-30 10:35:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 425.793795ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [............XXXXX...........................................] duration:
592.471209ms
FAIL (2025-11-30 10:35:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 592.491847ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [......XXXXX.................................................] duration:
369.372187ms
FAIL (2025-11-30 10:35:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 369.394538ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
404.754997ms
FAIL (2025-11-30 10:35:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 404.778791ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [.........XXXXXXXX...........................................] duration:
487.950069ms
FAIL (2025-11-30 10:34:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 487.969145ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
564.769357ms
FAIL (2025-11-30 10:34:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 564.784705ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......XXXXX................................................] duration:
403.985922ms
FAIL (2025-11-30 10:34:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 404.011981ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.............XXXXX..........................................] duration:
637.352786ms
FAIL (2025-11-30 10:34:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 637.374225ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..............XXXXX.........................................] duration:
585.202366ms
FAIL (2025-11-30 10:34:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 585.218696ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
530.246784ms
FAIL (2025-11-30 10:34:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 530.267834ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [........XXXXX...............................................] duration:
433.558829ms
FAIL (2025-11-30 10:34:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 433.577474ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
402.529159ms
FAIL (2025-11-30 10:34:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 402.575135ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
406.227591ms
FAIL (2025-11-30 10:33:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 406.24823ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [..........XXXXXXXX..........................................] duration:
466.479008ms
FAIL (2025-11-30 10:33:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 466.533561ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........XXXXX..............................................] duration:
315.099057ms
FAIL (2025-11-30 10:33:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 315.121779ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
374.352032ms
FAIL (2025-11-30 10:32:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 374.381547ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...............XXXXX........................................] duration:
524.816863ms
FAIL (2025-11-30 10:32:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 524.83631ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........XXXXX.............................................] duration:
383.318709ms
FAIL (2025-11-30 10:32:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 383.34082ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [................XXXXX.......................................] duration:
560.181195ms
FAIL (2025-11-30 10:32:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 560.211632ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
302.298265ms
FAIL (2025-11-30 10:32:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 302.319345ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=3) [...........XXXXXXXX.........................................] duration:
463.994456ms
FAIL (2025-11-30 10:32:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 464.014774ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........XXXXX............................................] duration:
324.087474ms
FAIL (2025-11-30 10:32:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
165596 (814h48m8.109110511s) [seg_pos={0 0}, upload=716889]: Done begin upload segment.
Nov 30 10:01:41.408194700 (814h48m8.109139616s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408224707 (814h48m8.109169622s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408698056 (814h48m8.109642971s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.408704358 (814h48m8.109649272s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409120197 (814h48m8.110065113s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409130086 (814h48m8.110075001s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409504428 (814h48m8.110449343s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.409543391 (814h48m8.110488316s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410342341 (814h48m8.111287255s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410374791 (814h48m8.111319756s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.410970980 (814h48m8.111915896s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412047490 (814h48m8.112992406s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.412199065 (814h48m8.11314398s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413134330 (814h48m8.114079245s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.413280624 (814h48m8.114225539s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414080115 (814h48m8.11502503s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414219066 (814h48m8.115163981s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.414913329 (814h48m8.115858243s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Uploading piece...
Nov 30 10:01:41.497117191 (814h48m8.198062086s) [upload=716889]: Next returned nil segment
Nov 30 10:01:41.497127249 (814h48m8.198072144s) [upload=716889]: Waiting for error group managing segments...
Nov 30 10:01:41.505081407 (814h48m8.206026291s) [noise=true , address=45.91.116.10:4104 , piece_id=RIGUL7KSO2JK5M7TNDVKQZD6FLZDEWYDXZRFHEWILUNBZTJYUW2Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505294346 (814h48m8.206239231s) [noise=true , address=45.91.116.12:4247 , piece_id=SJCO2XP7AUZUN3JLM5NHX5DRDWMR6TBVXTMWSSX53KYHRDIBDYIA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505597595 (814h48m8.206542521s) [noise=true , address=45.91.116.7:4034 , piece_id=3TGU5SC64E3SJURS2FJNBMNHH6HCTIZUGP6XZGZNJDSH5YHHZVNA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.505940729 (814h48m8.206885644s) [noise=true , address=45.91.116.9:4286 , piece_id=LJ5HJED3MRX7PBBPVUIIFPQCXKOZYUUYNKTTU6QFC3TJ2VKRCRDQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.506595328 (814h48m8.207540253s) [noise=true , address=145.79.227.13:4807 , piece_id=VU4WQ423XOJWRKJRMKEKVJSMP3QLR5HYQG3GMJS7ZEJDHYILULAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507083102 (814h48m8.208028018s) [noise=true , address=145.79.227.5:4235 , piece_id=HUAIZFMCVATSUHT2B7M62H6CUZ6MVZFCBKAJ6U6QWX62LG632RAQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.507415977 (814h48m8.208360892s) [noise=true , address=45.91.116.13:4162 , piece_id=QFNCJIWNSQUT2VYQGBKZZTHU7YM3XO33XZYZI5O7FT3WQBK36T3Q, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.520800485 (814h48m8.22174537s) [noise=true , address=45.91.116.14:4192 , piece_id=USKGGDF5655GRJ2JX6JZFGWRR6JME3KMQGRI6PFJH42HLIV4RVYQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.974912342 (814h48m8.675857227s) [noise=true , address=64.49.13.170:4717 , piece_id=QFYCR6LHBUW277EU62RII3WYKCWH6NCMMBJ72MHQZJI3UUFOS5GQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:41.997957398 (814h48m8.698902293s) [noise=true , address=64.49.13.154:4934 , piece_id=HJVI7UG2JQCOP2TV4BDKJUDFT5X55TFFQVC3O47AETFPEJHZEXBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:01:42.860910619 (814h48m9.561855514s) [noise=true , address=64.49.13.169:4716 , piece_id=DBG2O7KRPUVWYKWDNQI5VKBUL46FTJUBJP5ZWDHWCVZAPKCXQAPA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: <nil>
Nov 30 10:02:41.372086706 (814h49m8.073031601s) [noise=true , address=77.90.31.10:4548 , piece_id=NFF3IVBCV6UPVW3OXTP7WIO6IIQJZTSFXJXZCZVNR2HBCRH4EH3A, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372182455 (814h49m8.073127341s) [noise=true , address=77.90.31.13:4637 , piece_id=BXAN7NRCXCMNORCJRUDS6PR3MSOZ4MJD3HZOLD4PS7WNB63WHBKA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 5.161.89.252:54802->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372220236 (814h49m8.073165131s) [noise=true , address=77.90.31.11:4574 , piece_id=KOAG4T3FABZHOUKKKHU3D5HAUAXZPIZ7OPKNX3XHHHDAI3KILYQQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372279337 (814h49m8.073224233s) [noise=true , address=77.90.31.17:4794 , piece_id=2A533NKQB2W4X3EXNBHE3M3THMDOT77QKYLHNMJXCBYNYFILNHYA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: write tcp 5.161.89.252:23588->77.90.31.17:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372309224 (814h49m8.073254119s) [noise=true , address=77.90.31.21:4891 , piece_id=JPRZXCFQO4VCYJJY5PV6MET4ZF64NBU7CWMAGISCBTYFXVLARPBQ, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372345802 (814h49m8.073290697s) [noise=true , address=77.90.31.18:4812 , piece_id=MRSFSFJXTW5GXC4XHFR6Q2X3QSGXKGT2O4TL32JFCDMXZFJ6SVDA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372411244 (814h49m8.07335613s) [noise=true , address=77.90.31.14:4668 , piece_id=WJEEZHF4QOWYMHQCW3V43E6OO63ELRZJ5E3XLI3KIIXV2LLF4WGA, seg_pos={0 0}, upload=716889]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: write tcp 5.161.89.252:11036->77.90.31.14:4668: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:41.372429860 (814h49m8.073374745s) [seg_pos={0 0}, upload=716889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:41.388787807 (814h49m8.089732692s) [upload=716889]: Done uploading segments...
Nov 30 10:02:41.388801623 (814h49m8.089746508s) [upload=716889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 324.10698ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
431.150711ms
FAIL (2025-11-30 10:32:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 30 09:57:54.338190879 (814h44m21.039135763s) [upload=716835]: Starting upload
Nov 30 09:57:54.338244719 (814h44m21.039189614s) [upload=716835]: Done starting upload
Nov 30 09:57:54.338384973 (814h44m21.039329868s) [upload=716835]: Uploading segments...
Nov 30 09:57:54.338487656 (814h44m21.039432541s) [upload=716835]: Got next segment. Inline: false
Nov 30 09:57:54.338526589 (814h44m21.039471494s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 09:57:54.338559952 (814h44m21.039504877s) [upload=716835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 09:57:54.361097355 (814h44m21.06204224s) [seg_pos={0 0}, upload=716835]: Begin upload segment...
Nov 30 09:57:54.361157588 (814h44m21.062102462s) [seg_pos={0 0}, upload=716835]: Done begin upload segment.
Nov 30 09:57:54.361186011 (814h44m21.062130916s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361205928 (814h44m21.062150823s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361626347 (814h44m21.062571232s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.361930398 (814h44m21.062875293s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362176009 (814h44m21.063120904s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362428763 (814h44m21.063373648s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362677220 (814h44m21.063622105s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.362933340 (814h44m21.063878225s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363160577 (814h44m21.064105472s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.363400687 (814h44m21.064345582s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364097124 (814h44m21.065042009s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364511481 (814h44m21.065456386s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.364999117 (814h44m21.065944012s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365062575 (814h44m21.06600747s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365334095 (814h44m21.06627898s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.365625122 (814h44m21.066570006s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Uploading piece...
Nov 30 09:57:54.437461809 (814h44m21.138406694s) [upload=716835]: Next returned nil segment
Nov 30 09:57:54.437472619 (814h44m21.138417504s) [upload=716835]: Waiting for error group managing segments...
Nov 30 09:57:54.446112432 (814h44m21.147057317s) [noise=true , address=45.91.116.8:4045 , piece_id=I5IJZ7VBOWMXUAPBUJP5YYMGX3GSS4SOWZUWOOMM7XAZRTM7AUMQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.446603695 (814h44m21.147548609s) [noise=true , address=45.91.116.10:4081 , piece_id=WUQEQDP2VS2EHLLO7RGS462V4FKUOVXGLZRX25ZVVTUT3HGOIA5Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447195905 (814h44m21.1481408s) [noise=true , address=145.79.227.39:4015 , piece_id=KAB5SQBQPWGHKGEYMTBD2VTCAURBYEZ2LT7B4XESCIUG5PMCHJQA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447430665 (814h44m21.148375551s) [noise=true , address=145.79.227.42:4121 , piece_id=5OICIQHCUXLBKC7VOHNSNF7FM3EWDLM7QZG74EGUY7OSWDGTCLKQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447592850 (814h44m21.148537736s) [noise=true , address=145.79.227.24:4582 , piece_id=RAZUHVJ7XPITPIENN46YV5KO2LXIXHW4L7ZRMIURYJXDT2SYJWWA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.447971461 (814h44m21.148916346s) [noise=true , address=145.79.227.26:4655 , piece_id=CE4V4WNY6N6XLC6FCFLVENKPMEXBZJWCFJKCJ2N7LQOOAUQLU2LQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448046792 (814h44m21.148991688s) [noise=true , address=145.79.227.4:4186 , piece_id=NKYJRESISGAEEOBIIKHFBHMRXB74WRBPJORJJH7WQG4APO7SD7LA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448347225 (814h44m21.149292111s) [noise=true , address=45.91.116.11:4119 , piece_id=SJ7S6JCLNY3MID3DLIXWRM7RF2GNSGXXWSMMMMY62JB7O42UA6EQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448489934 (814h44m21.149434819s) [noise=true , address=145.79.227.28:4726 , piece_id=RXBLNITDMQMXDL5YCDYE5QYZOS6CUO35JHXYPBRIY7SJ4M7ESXTA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448666054 (814h44m21.14961095s) [noise=true , address=45.91.116.5:4299 , piece_id=WWCR5BCEITUBHUHYQQWZ3XUJCVYZ2PRQYBLPDQSPZLP732M33MDA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.448904582 (814h44m21.149849467s) [noise=true , address=145.79.227.30:4761 , piece_id=CGG3PQB52EIVJH7IMVQTWQ2APIJ5SATHWQIJDMRYSW4ULIKYHZLQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:57:54.449463692 (814h44m21.150408587s) [noise=true , address=145.79.227.38:4977 , piece_id=VV64FHNNAX7ZX6ITWO7UBLWQEAK4YBRKPZHDEQ27FY4G3VSUXOVQ, seg_pos={0 0}, upload=716835]: Done uploading piece. err: <nil>
Nov 30 09:58:24.327129404 (814h44m51.028074289s) [noise=true , address=77.90.31.10:4520 , piece_id=IS2R6IMSFXSVW7QBYVEZA36VU3QAQQ77RH6ZEFXRD5B2ZPT7GG6Q, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:27704->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327148590 (814h44m51.028093476s) [noise=true , address=77.90.31.19:4842 , piece_id=XJT5ZQA3UKQ64FMC2K2JSHADLHUUWIGINSZ5S6CMEWU6YPPIGYBA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327211238 (814h44m51.028156123s) [noise=true , address=77.90.31.16:4734 , piece_id=Y7ZFZGGBGXMDKOIKVA5IDOEKWAO6O5UW4LUBFWQYUC74NZ5DY7CA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:45892->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327233088 (814h44m51.028177984s) [noise=true , address=77.90.31.15:4699 , piece_id=VVNU63DNNIDA2SL3KWGRB4LETBZ2F5MPSFIRKRHUMWNUDOOCB5IA, seg_pos={0 0}, upload=716835]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 5.161.89.252:21530->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 09:58:24.327250742 (814h44m51.028195637s) [seg_pos={0 0}, upload=716835]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 30 09:58:24.345660629 (814h44m51.046605515s) [upload=716835]: Done uploading segments...
Nov 30 09:58:24.345673454 (814h44m51.046618349s) [upload=716835]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 431.177671ms
core
uplink
us-select-1
|