Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [....................................................XXXX....] duration:
3.696155751s
FAIL (2025-08-31 15:27:33 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.69617629s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................................................XX.XX....] duration:
1.392208348s
FAIL (2025-08-31 15:27:32 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 1.392229859s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.....................................................XXXX...] duration:
39.249150125s
FAIL (2025-08-31 15:27:27 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 39.24916392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X.X......] duration:
8.710478257s
FAIL (2025-08-31 15:27:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.710498535s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................................................XX.XX...] duration:
541.065877ms
FAIL (2025-08-31 15:26:27 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 541.091307ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................................................XXXX..] duration:
1.027778758s
FAIL (2025-08-31 15:26:18 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 1.027796862s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration:
351.426442ms
FAIL (2025-08-31 15:26:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 351.442822ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................................................XXXX.] duration:
24.862482427s
FAIL (2025-08-31 15:26:17 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 24.862498558s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................................................XX.XX..] duration:
2.433217456s
FAIL (2025-08-31 15:26:13 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.433243324s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration:
838.40744ms
FAIL (2025-08-31 15:25:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 838.432146ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
1m0.023487325s
FAIL (2025-08-31 15:25:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 1m0.02350654s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [......................................................XXXX..] duration:
4.292754736s
FAIL (2025-08-31 15:25:47 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.292767041s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.......................................................XXXX.] duration:
51.585813274s
FAIL (2025-08-31 15:25:43 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 51.585826989s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X.X...] duration:
466.234037ms
FAIL (2025-08-31 15:25:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 466.362772ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................................................XX.XX.] duration:
375.51258ms
FAIL (2025-08-31 15:25:14 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 375.529401ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................................................XX.XX] duration:
30.016888778s
FAIL (2025-08-31 15:25:13 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.016910228s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
1m0.001025718s
FAIL (2025-08-31 15:24:51 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001152848s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
1m0.016104654s
FAIL (2025-08-31 15:24:48 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
upload=43377]: Uploading piece...
Aug 31 15:23:48.123228148 (49h15m23.476013276s) [noise=true , address=104.36.82.5:4848 , piece_id=QTHDV362WEJYNC4YIMUJQUPE52K5PBPS4CFEYTAAZM6BBSIF6LHQ, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.123436520 (49h15m23.476221648s) [noise=true , address=104.36.82.2:4743 , piece_id=4HNOSY35LQWBEH6LN675WYORVFTYTBWK6DPJHD33KWFQCPAEZ4WA, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.123638991 (49h15m23.47642412s) [noise=true , address=104.36.82.7:4923 , piece_id=A6PQLQS57XGFB5VOMBWQOFYRE3FEUHQDSCGYFZZQJ3D3H2WWDOMQ, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.124001403 (49h15m23.476786531s) [noise=true , address=104.36.82.48:4598 , piece_id=XP4IF3LA7WA6JNPUE43TSZZDKXBWM4KO5VUPF6VP5AFVLXUZAV4A, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.124106421 (49h15m23.4768916s) [noise=true , address=104.36.82.23:4451 , piece_id=T5XFABZIHP6HEUYYBOABWESGGC3TDTW6W24PD2ESC6XTI3G7VHRA, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.124538565 (49h15m23.477323693s) [noise=true , address=104.36.82.47:4542 , piece_id=Z2DEPZ4C5QIUG7PDSD55VO5LS3TFPQP5LLP3DYITBQCYV6EVQ2ZA, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.124595763 (49h15m23.477380891s) [noise=true , address=104.36.82.44:4881 , piece_id=VZZEQYM26JHJOPN3JTH5RTZMJA24AQ24PK5C4SEQYBG4LN6D6OAQ, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.124950521 (49h15m23.477735648s) [noise=true , address=104.36.82.10:4110 , piece_id=IRVFUHNE2JI7IM6OV645733B5GOFZE4LMNJF6U7363JCLLM75W2Q, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.125146870 (49h15m23.477931999s) [noise=true , address=104.36.82.6:4888 , piece_id=2MMFERO2H3K6ANU6UVVXRERWLMUQAPM6H2MKR5IW7O6FI4WEXWNQ, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.125334644 (49h15m23.478119772s) [noise=true , address=45.91.116.4:4961 , piece_id=ZTUCNKQZJ5SPK4YGPHHFTYPTKE2IKUUKR5LRURXVONDT6IXLF4ZA, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.126027709 (49h15m23.478812846s) [noise=true , address=77.90.31.11:4583 , piece_id=5VUYBYT6VGRDOLMEI3TJ5OUKXWZX5E2MKKIYRTK55R63QNSXR6OA, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.126973670 (49h15m23.479758818s) [noise=true , address=77.90.31.18:4836 , piece_id=WHFQYEFIQI57Y7APUZIN7B3RY3LH6CWDQFVQUQN45IFGKKP2N35Q, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.127832106 (49h15m23.480617234s) [noise=true , address=79.127.205.245:4245 , piece_id=BHUAHI5PFSPAUVQEAD4EFWBIK5UBNSCBJVI5B64PI4XCWJ4SGERQ, seg_pos={0 0}, upload=43377]: Uploading piece...
Aug 31 15:23:48.212434445 (49h15m23.565219553s) [upload=43377]: Next returned nil segment
Aug 31 15:23:48.212456947 (49h15m23.565242045s) [upload=43377]: Waiting for error group managing segments...
Aug 31 15:23:48.220007112 (49h15m23.572792221s) [noise=true , address=45.91.116.4:4961 , piece_id=ZTUCNKQZJ5SPK4YGPHHFTYPTKE2IKUUKR5LRURXVONDT6IXLF4ZA, seg_pos={0 0}, upload=43377]: Done uploading piece. err: <nil>
Aug 31 15:23:48.221052089 (49h15m23.573837197s) [noise=true , address=77.90.31.18:4836 , piece_id=WHFQYEFIQI57Y7APUZIN7B3RY3LH6CWDQFVQUQN45IFGKKP2N35Q, seg_pos={0 0}, upload=43377]: Done uploading piece. err: <nil>
Aug 31 15:23:48.241327054 (49h15m23.594112152s) [noise=true , address=77.90.31.17:4799 , piece_id=YCFBXPYRJDSMFGAUMHSRX2ULXUMQVQS3SPOSFCT62QRU2BYWX3DQ, seg_pos={0 0}, upload=43377]: Done uploading piece. err: <nil>
Aug 31 15:23:48.258290271 (49h15m23.611075379s) [noise=true , address=77.90.31.13:4655 , piece_id=BEM2HK47AZU5LERZLEJG6D2UHYTRTKPOJCQWT25FXQDS6P3L4FUQ, seg_pos={0 0}, upload=43377]: Done uploading piece. err: <nil>
Aug 31 15:23:48.267224594 (49h15m23.620009692s) [noise=true , address=77.90.31.11:4583 , piece_id=5VUYBYT6VGRDOLMEI3TJ5OUKXWZX5E2MKKIYRTK55R63QNSXR6OA, seg_pos={0 0}, upload=43377]: Done uploading piece. err: <nil>
Aug 31 15:23:48.666512229 (49h15m24.019297337s) [noise=true , address=79.127.205.245:4245 , piece_id=BHUAHI5PFSPAUVQEAD4EFWBIK5UBNSCBJVI5B64PI4XCWJ4SGERQ, seg_pos={0 0}, upload=43377]: Done uploading piece. err: <nil>
Aug 31 15:24:48.080851636 (49h16m23.433636733s) [noise=true , address=104.36.82.10:4110 , piece_id=IRVFUHNE2JI7IM6OV645733B5GOFZE4LMNJF6U7363JCLLM75W2Q, seg_pos={0 0}, upload=43377]: Done uploading piece. err: ecclient: upload failed (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi, address:104.36.82.10:4110): protocol: rpc: dial tcp 104.36.82.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:48.080887843 (49h16m23.433672942s) [noise=true , address=104.36.82.47:4542 , piece_id=Z2DEPZ4C5QIUG7PDSD55VO5LS3TFPQP5LLP3DYITBQCYV6EVQ2ZA, seg_pos={0 0}, upload=43377]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: rpc: dial tcp 104.36.82.47:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:48.080903253 (49h16m23.43368835s) [noise=true , address=104.36.82.2:4743 , piece_id=4HNOSY35LQWBEH6LN675WYORVFTYTBWK6DPJHD33KWFQCPAEZ4WA, seg_pos={0 0}, upload=43377]: Done uploading piece. err: ecclient: upload failed (node:1UMjJdYFqUjDcAEamv69RrmqtBPT2wiAk585jBsrv6yxFESjNy, address:104.36.82.2:4743): protocol: rpc: dial tcp 104.36.82.2:4743: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:48.080926506 (49h16m23.433711604s) [noise=true , address=104.36.82.55:4219 , piece_id=SUFWLG75G5GSUYN34SXGDYEIVB3OULXU26KUEZFNJ3OTAH7YJ5OQ, seg_pos={0 0}, upload=43377]: Done uploading piece. err: ecclient: upload failed (node:12TqSxiG3LYYc3W5T6wqiuuKwoh1LvCdntou7ve6TAEbXYf2QKn, address:104.36.82.55:4219): protocol: rpc: dial tcp 104.36.82.55:4219: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:48.080940181 (49h16m23.43372528s) [noise=true , address=104.36.82.48:4598 , piece_id=XP4IF3LA7WA6JNPUE43TSZZDKXBWM4KO5VUPF6VP5AFVLXUZAV4A, seg_pos={0 0}, upload=43377]: Done uploading piece. err: ecclient: upload failed (node:1AtHr6PvnJUPpR96cfiYRc4JkwtBB56HJvso76YCQU47JQns8n, address:104.36.82.48:4598): protocol: rpc: dial tcp 104.36.82.48:4598: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:48.080962764 (49h16m23.433747872s) [noise=true , address=104.36.82.5:4848 , piece_id=QTHDV362WEJYNC4YIMUJQUPE52K5PBPS4CFEYTAAZM6BBSIF6LHQ, seg_pos={0 0}, upload=43377]: Done uploading piece. err: ecclient: upload failed (node:1xvod7N8eEwunLDYWAvVUP1Cp1CdEr3quJUMG5kWyBrwzzr9vp, address:104.36.82.5:4848): protocol: rpc: dial tcp 104.36.82.5:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:48.080980738 (49h16m23.433765837s) [noise=true , address=104.36.82.23:4451 , piece_id=T5XFABZIHP6HEUYYBOABWESGGC3TDTW6W24PD2ESC6XTI3G7VHRA, seg_pos={0 0}, upload=43377]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:104.36.82.23:4451): protocol: rpc: dial tcp 104.36.82.23:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:48.081029319 (49h16m23.433814418s) [noise=true , address=104.36.82.4:4836 , piece_id=OE3TDGFKHZWGYJ22G4TQFCU47UX4SGNGPAHNQPK6JEBGHOUUSXBQ, seg_pos={0 0}, upload=43377]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:104.36.82.4:4836): protocol: write tcp 5.161.89.252:14098->104.36.82.4:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:48.081039098 (49h16m23.433824196s) [noise=true , address=104.36.82.56:4259 , piece_id=HB2EHGDMR3EDHJLISPU54L46VRMCI5LWAINT3W6FAKQQGR5RS6EQ, seg_pos={0 0}, upload=43377]: Done uploading piece. err: ecclient: upload failed (node:12BCJsPfe1NNUxVUfQWKbvkzviQWYgMxkRZjDhKJ7KFZc1GmnjV, address:104.36.82.56:4259): protocol: write tcp 5.161.89.252:19816->104.36.82.56:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:48.081047083 (49h16m23.433832181s) [noise=true , address=104.36.82.7:4923 , piece_id=A6PQLQS57XGFB5VOMBWQOFYRE3FEUHQDSCGYFZZQJ3D3H2WWDOMQ, seg_pos={0 0}, upload=43377]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:104.36.82.7:4923): protocol: rpc: dial tcp 104.36.82.7:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:48.081074965 (49h16m23.433860073s) [noise=true , address=104.36.82.6:4888 , piece_id=2MMFERO2H3K6ANU6UVVXRERWLMUQAPM6H2MKR5IW7O6FI4WEXWNQ, seg_pos={0 0}, upload=43377]: Done uploading piece. err: ecclient: upload failed (node:12HwYszCTcXEHqSNnWJpT11NpKg8cFUvdAqU2wUigsLbxmDFkg6, address:104.36.82.6:4888): protocol: rpc: dial tcp 104.36.82.6:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:48.081100063 (49h16m23.43388516s) [noise=true , address=104.36.82.44:4881 , piece_id=VZZEQYM26JHJOPN3JTH5RTZMJA24AQ24PK5C4SEQYBG4LN6D6OAQ, seg_pos={0 0}, upload=43377]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:104.36.82.44:4881): protocol: write tcp 5.161.89.252:34060->104.36.82.44:4881: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:48.081123006 (49h16m23.433908114s) [seg_pos={0 0}, upload=43377]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Aug 31 15:24:48.094739413 (49h16m23.447524521s) [upload=43377]: Done uploading segments...
Aug 31 15:24:48.094761694 (49h16m23.447546802s) [upload=43377]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 1m0.016119943s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration:
391.829205ms
FAIL (2025-08-31 15:24:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 391.851247ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................................................XX.X] duration:
30.022606694s
FAIL (2025-08-31 15:24:14 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Aug 31 15:23:44.316904015 (49h15m19.669689144s) [upload=43376]: Starting upload
Aug 31 15:23:44.316985108 (49h15m19.669770235s) [upload=43376]: Done starting upload
Aug 31 15:23:44.317128969 (49h15m19.669914096s) [upload=43376]: Uploading segments...
Aug 31 15:23:44.317373008 (49h15m19.670158137s) [upload=43376]: Got next segment. Inline: false
Aug 31 15:23:44.317420928 (49h15m19.670206056s) [upload=43376]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:23:44.317426359 (49h15m19.670211487s) [upload=43376]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:23:44.338423402 (49h15m19.69120854s) [seg_pos={0 0}, upload=43376]: Begin upload segment...
Aug 31 15:23:44.338493383 (49h15m19.691278521s) [seg_pos={0 0}, upload=43376]: Done begin upload segment.
Aug 31 15:23:44.338548127 (49h15m19.691333255s) [noise=true , address=45.91.116.7:4009 , piece_id=DVAAAQDMFI27KFQBHTDRGO4UFGWFJT436I5QLTSZF5IKE4Y3XZDA, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.338593171 (49h15m19.691378309s) [noise=true , address=77.90.31.11:4553 , piece_id=HAH4V5KZIA26YHFDZX4OZR3OPPRNXVEQ3R55ZPU5MXXLIPXRRBPQ, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.339136655 (49h15m19.691921783s) [noise=true , address=45.91.116.10:4082 , piece_id=V6K2NRRIHEZVODLHJ2LWGY5ODHSDYVFTT5W23ICC4VOMAK5ESSHQ, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.339198581 (49h15m19.691983709s) [noise=true , address=104.36.82.4:4801 , piece_id=44KJ7KO54AQO42IOUA4EFHNRJUGWWX5CF7JQETHQHBWIREQIRC3A, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.339588015 (49h15m19.692373152s) [noise=true , address=104.36.82.15:4258 , piece_id=E3GTT223EDE63AWEVQW5ID4WPF7MXSNY4IADOEYVLNIPH6A6NOTA, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.339882158 (49h15m19.692667286s) [noise=true , address=77.90.31.19:4841 , piece_id=FDSM3MUG3YWMTCCAWPMICTL6HTC5JONGALSFSG6RD67D4PLHDZYA, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.340053892 (49h15m19.69283902s) [noise=true , address=45.91.116.8:4047 , piece_id=AXKK7OFQELPRJ3FP7WBPC3PJPSJBGOQ5HNSZZHNI64JIOJKTP4DQ, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.340459285 (49h15m19.693244414s) [noise=true , address=45.91.116.11:4117 , piece_id=LED7ZCTUUBERJ7L4O4Q2DUGKXQJRVWBINYQZABDN6YYLHXTQN42Q, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.340550256 (49h15m19.693335394s) [noise=true , address=77.90.31.12:4589 , piece_id=YQ53UC4CSDCAXE3TONO4IWZXATETOHIFFELBQ35WZYCBATWTDGOQ, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.340875038 (49h15m19.693660166s) [noise=true , address=104.36.82.5:4837 , piece_id=DZMSUGHSU5H7VLZN3LL3EMDV35OVJ2VQCJKEVHBDYLK2X3VR42HQ, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.341194169 (49h15m19.693979296s) [noise=true , address=45.91.116.9:4261 , piece_id=OS4TJKPOWDISLO5I4RGU45G5KBAIJHANNOGQ4DGCR3GNWH4OR5HQ, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.341277005 (49h15m19.694062133s) [noise=true , address=77.90.31.16:4735 , piece_id=QNXR465ZQUKHNX2WWJDYNT6WGLBDC42JFZNS4R573MXIDNSDWYIQ, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.341829395 (49h15m19.694614523s) [noise=true , address=45.91.116.12:4226 , piece_id=HS6JZYMRQM35QTFESYNJXASYCWWOQQTWFG4AVJILERBRYU3BJCEA, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.342409347 (49h15m19.695194485s) [noise=true , address=104.36.82.15:4259 , piece_id=OP2YGF6AZ2KAUSVBQCZOEIGFS3NSTV462CSJD5BXZ4RZET4PTQPQ, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.343071734 (49h15m19.695856872s) [noise=true , address=77.90.31.21:4878 , piece_id=IH655YSGIPIKBX3Y23M7BDL5S4H35Y53GROCKQOPOVA3QOHA3L3Q, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.343791078 (49h15m19.696576216s) [noise=true , address=45.91.116.10:4083 , piece_id=H2O3FQPURUX67IJBG6D46MI7JPE65RWMMN4LWV7TFXPGCAPD624Q, seg_pos={0 0}, upload=43376]: Uploading piece...
Aug 31 15:23:44.466603039 (49h15m19.819388167s) [upload=43376]: Next returned nil segment
Aug 31 15:23:44.466621113 (49h15m19.81940624s) [upload=43376]: Waiting for error group managing segments...
Aug 31 15:23:44.474190364 (49h15m19.826975492s) [noise=true , address=45.91.116.10:4082 , piece_id=V6K2NRRIHEZVODLHJ2LWGY5ODHSDYVFTT5W23ICC4VOMAK5ESSHQ, seg_pos={0 0}, upload=43376]: Done uploading piece. err: <nil>
Aug 31 15:23:44.474515325 (49h15m19.827300423s) [noise=true , address=45.91.116.7:4009 , piece_id=DVAAAQDMFI27KFQBHTDRGO4UFGWFJT436I5QLTSZF5IKE4Y3XZDA, seg_pos={0 0}, upload=43376]: Done uploading piece. err: <nil>
Aug 31 15:23:44.475021889 (49h15m19.827806997s) [noise=true , address=45.91.116.8:4047 , piece_id=AXKK7OFQELPRJ3FP7WBPC3PJPSJBGOQ5HNSZZHNI64JIOJKTP4DQ, seg_pos={0 0}, upload=43376]: Done uploading piece. err: <nil>
Aug 31 15:23:44.475411703 (49h15m19.828196831s) [noise=true , address=45.91.116.12:4226 , piece_id=HS6JZYMRQM35QTFESYNJXASYCWWOQQTWFG4AVJILERBRYU3BJCEA, seg_pos={0 0}, upload=43376]: Done uploading piece. err: <nil>
Aug 31 15:23:44.475443072 (49h15m19.82822818s) [noise=true , address=45.91.116.9:4261 , piece_id=OS4TJKPOWDISLO5I4RGU45G5KBAIJHANNOGQ4DGCR3GNWH4OR5HQ, seg_pos={0 0}, upload=43376]: Done uploading piece. err: <nil>
Aug 31 15:23:44.475875005 (49h15m19.828660113s) [noise=true , address=45.91.116.11:4117 , piece_id=LED7ZCTUUBERJ7L4O4Q2DUGKXQJRVWBINYQZABDN6YYLHXTQN42Q, seg_pos={0 0}, upload=43376]: Done uploading piece. err: <nil>
Aug 31 15:23:44.476263557 (49h15m19.829048695s) [noise=true , address=77.90.31.21:4878 , piece_id=IH655YSGIPIKBX3Y23M7BDL5S4H35Y53GROCKQOPOVA3QOHA3L3Q, seg_pos={0 0}, upload=43376]: Done uploading piece. err: <nil>
Aug 31 15:23:44.476400695 (49h15m19.829185793s) [noise=true , address=45.91.116.10:4083 , piece_id=H2O3FQPURUX67IJBG6D46MI7JPE65RWMMN4LWV7TFXPGCAPD624Q, seg_pos={0 0}, upload=43376]: Done uploading piece. err: <nil>
Aug 31 15:23:44.476914583 (49h15m19.829699721s) [noise=true , address=77.90.31.11:4553 , piece_id=HAH4V5KZIA26YHFDZX4OZR3OPPRNXVEQ3R55ZPU5MXXLIPXRRBPQ, seg_pos={0 0}, upload=43376]: Done uploading piece. err: <nil>
Aug 31 15:23:44.476970879 (49h15m19.829755987s) [noise=true , address=77.90.31.12:4589 , piece_id=YQ53UC4CSDCAXE3TONO4IWZXATETOHIFFELBQ35WZYCBATWTDGOQ, seg_pos={0 0}, upload=43376]: Done uploading piece. err: <nil>
Aug 31 15:23:44.477405556 (49h15m19.830190665s) [noise=true , address=77.90.31.19:4841 , piece_id=FDSM3MUG3YWMTCCAWPMICTL6HTC5JONGALSFSG6RD67D4PLHDZYA, seg_pos={0 0}, upload=43376]: Done uploading piece. err: <nil>
Aug 31 15:23:44.477515323 (49h15m19.830300452s) [noise=true , address=77.90.31.16:4735 , piece_id=QNXR465ZQUKHNX2WWJDYNT6WGLBDC42JFZNS4R573MXIDNSDWYIQ, seg_pos={0 0}, upload=43376]: Done uploading piece. err: <nil>
Aug 31 15:24:14.304301578 (49h15m49.657086705s) [noise=true , address=104.36.82.4:4801 , piece_id=44KJ7KO54AQO42IOUA4EFHNRJUGWWX5CF7JQETHQHBWIREQIRC3A, seg_pos={0 0}, upload=43376]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:14.304344188 (49h15m49.657129317s) [noise=true , address=104.36.82.5:4837 , piece_id=DZMSUGHSU5H7VLZN3LL3EMDV35OVJ2VQCJKEVHBDYLK2X3VR42HQ, seg_pos={0 0}, upload=43376]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:14.304443876 (49h15m49.657229013s) [noise=true , address=104.36.82.15:4259 , piece_id=OP2YGF6AZ2KAUSVBQCZOEIGFS3NSTV462CSJD5BXZ4RZET4PTQPQ, seg_pos={0 0}, upload=43376]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 5.161.89.252:31370->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:14.304499481 (49h15m49.657284619s) [noise=true , address=104.36.82.15:4258 , piece_id=E3GTT223EDE63AWEVQW5ID4WPF7MXSNY4IADOEYVLNIPH6A6NOTA, seg_pos={0 0}, upload=43376]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:24:14.304556317 (49h15m49.657341445s) [seg_pos={0 0}, upload=43376]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Aug 31 15:24:14.322904543 (49h15m49.675689671s) [upload=43376]: Done uploading segments...
Aug 31 15:24:14.322925622 (49h15m49.675710751s) [upload=43376]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.022628696s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................................X.X.] duration:
426.834451ms
FAIL (2025-08-31 15:23:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 426.85514ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
1m0.001779584s
FAIL (2025-08-31 15:23:51 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001797778s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
1m0.017191183s
FAIL (2025-08-31 15:23:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
, address=104.36.82.13:4189 , piece_id=4XOVTZEYAO7KUV7BSBXG5JKD6OVAUOJ2MXCVVOFPIMTHPDFKSTTA, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.052675595 (49h14m23.405460724s) [noise=true , address=104.36.82.4:4806 , piece_id=AQZNLHMRDSJW3FJDGMCUZE7IRKYULE2Z2EDOICQ3YXB5UEE6XEEQ, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.052912221 (49h14m23.40569736s) [noise=true , address=77.90.31.19:4873 , piece_id=DAYYYWKHTXG5ILZU3ZJBOS5RBB3PMSON57W7WRYL75IN3LXIXYAA, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.053209120 (49h14m23.405994248s) [noise=true , address=104.36.82.7:4944 , piece_id=SQBTPTQ6AXUVGNKKJE5M2NLQ2QPWOMUIJ57NETL3MJ7ZSIZ3V3XA, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.053318506 (49h14m23.406103634s) [noise=true , address=104.36.82.5:4857 , piece_id=XQ74MZALDTT7BBXVTMA2PQFCGVDYMJYU6FW3YBC27UOQRBFZZSNA, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.053746662 (49h14m23.40653179s) [noise=true , address=77.90.31.16:4751 , piece_id=HHUKYTQDGIO5N2YP7PNA4LJCRV2RR656LBO4Z5W23E3GMXJUJ4SQ, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.053798400 (49h14m23.406583527s) [noise=true , address=77.90.31.15:4728 , piece_id=4DPWJBY6USSMCHFADTRB4WJXWGWCDIVQ24WR2TRQKYDYL62EFLOA, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.054230062 (49h14m23.4070152s) [noise=true , address=45.91.116.11:4121 , piece_id=2AAHWORYGEZMVSPI5XNPAUXNAAEBZS4KTQIWVBQ2YX2XZQP6IQWA, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.054292941 (49h14m23.407078079s) [noise=true , address=104.36.82.44:4901 , piece_id=VZSAXZUYR5I5H44RT6BGAGOEWAR2MJFSOVQE43EBXZVTKXW7WV5Q, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.054751343 (49h14m23.407536472s) [noise=true , address=104.36.82.23:4457 , piece_id=MXL3RSFELPH3L7B7AXE3OPBOWJIVGVONWNVJFZYEVXJ62LVOIHQQ, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.055484144 (49h14m23.408269282s) [noise=true , address=77.90.31.21:4887 , piece_id=6OJ4KSZKCE4ORM6IZ72LZVWGU2Z7TPVKPB7OI744XP3VFEWCXH7A, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.055947225 (49h14m23.408732354s) [noise=true , address=77.90.31.17:4802 , piece_id=LLMSAGB52LMPWMFSWD5I5BEJNXHTO4FMC4X2DVAJJ7KSODRHH6IA, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.056407011 (49h14m23.409192139s) [noise=true , address=79.127.205.246:4284 , piece_id=QI6MTK4NGCY65GHAHYB6C5ABSVQEM2WO7PTZKCGUNQXI2N3U25DQ, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.056991502 (49h14m23.40977664s) [noise=true , address=104.36.82.48:4576 , piece_id=HR5GRSUA7C4V7MS7CI42GVC5IWNTLJPJ7AF2R3CN45MKFTJYTPSA, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.057502343 (49h14m23.410287481s) [noise=true , address=104.36.82.60:4372 , piece_id=E7YZHJJYTESR6WB5UX7DSGSJCIAGYL23GRLPCYHTDGVHU3VTKLTQ, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.058219343 (49h14m23.411004471s) [noise=true , address=104.36.82.14:4224 , piece_id=IPTRHEWNWEBEL5WGQKATNMM4M7BK2A5QMRGRUYEDLT3LPRAYNHKQ, seg_pos={0 0}, upload=43362]: Uploading piece...
Aug 31 15:22:48.149420386 (49h14m23.502205514s) [upload=43362]: Next returned nil segment
Aug 31 15:22:48.149434633 (49h14m23.502219751s) [upload=43362]: Waiting for error group managing segments...
Aug 31 15:22:48.158576785 (49h14m23.511361884s) [noise=true , address=45.91.116.11:4121 , piece_id=2AAHWORYGEZMVSPI5XNPAUXNAAEBZS4KTQIWVBQ2YX2XZQP6IQWA, seg_pos={0 0}, upload=43362]: Done uploading piece. err: <nil>
Aug 31 15:22:48.161230672 (49h14m23.5140158s) [noise=true , address=77.90.31.15:4728 , piece_id=4DPWJBY6USSMCHFADTRB4WJXWGWCDIVQ24WR2TRQKYDYL62EFLOA, seg_pos={0 0}, upload=43362]: Done uploading piece. err: <nil>
Aug 31 15:22:48.168600188 (49h14m23.521385285s) [noise=true , address=77.90.31.19:4873 , piece_id=DAYYYWKHTXG5ILZU3ZJBOS5RBB3PMSON57W7WRYL75IN3LXIXYAA, seg_pos={0 0}, upload=43362]: Done uploading piece. err: <nil>
Aug 31 15:22:48.187621932 (49h14m23.54040706s) [noise=true , address=77.90.31.21:4887 , piece_id=6OJ4KSZKCE4ORM6IZ72LZVWGU2Z7TPVKPB7OI744XP3VFEWCXH7A, seg_pos={0 0}, upload=43362]: Done uploading piece. err: <nil>
Aug 31 15:22:48.194366971 (49h14m23.547152069s) [noise=true , address=77.90.31.17:4802 , piece_id=LLMSAGB52LMPWMFSWD5I5BEJNXHTO4FMC4X2DVAJJ7KSODRHH6IA, seg_pos={0 0}, upload=43362]: Done uploading piece. err: <nil>
Aug 31 15:22:48.217866707 (49h14m23.570651835s) [noise=true , address=77.90.31.16:4751 , piece_id=HHUKYTQDGIO5N2YP7PNA4LJCRV2RR656LBO4Z5W23E3GMXJUJ4SQ, seg_pos={0 0}, upload=43362]: Done uploading piece. err: <nil>
Aug 31 15:22:48.585301251 (49h14m23.938086359s) [noise=true , address=79.127.205.246:4284 , piece_id=QI6MTK4NGCY65GHAHYB6C5ABSVQEM2WO7PTZKCGUNQXI2N3U25DQ, seg_pos={0 0}, upload=43362]: Done uploading piece. err: <nil>
Aug 31 15:22:48.603494667 (49h14m23.956279795s) [noise=true , address=79.127.205.245:4242 , piece_id=7FR3I7YOBMO5CKY356MMNDHC64CR2XTKHLUYLAMZW6MXX6VARTWQ, seg_pos={0 0}, upload=43362]: Done uploading piece. err: <nil>
Aug 31 15:23:48.008617950 (49h15m23.361403078s) [noise=true , address=104.36.82.44:4901 , piece_id=VZSAXZUYR5I5H44RT6BGAGOEWAR2MJFSOVQE43EBXZVTKXW7WV5Q, seg_pos={0 0}, upload=43362]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:23:48.008693442 (49h15m23.361478569s) [noise=true , address=104.36.82.48:4576 , piece_id=HR5GRSUA7C4V7MS7CI42GVC5IWNTLJPJ7AF2R3CN45MKFTJYTPSA, seg_pos={0 0}, upload=43362]: Done uploading piece. err: ecclient: upload failed (node:1Ckqi1hws3KtSMmHzNt1SppBgWduvow9Aic2Zap2UdMf2W3FZC, address:104.36.82.48:4576): protocol: rpc: dial tcp 104.36.82.48:4576: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:23:48.008783311 (49h15m23.361568439s) [noise=true , address=104.36.82.60:4372 , piece_id=E7YZHJJYTESR6WB5UX7DSGSJCIAGYL23GRLPCYHTDGVHU3VTKLTQ, seg_pos={0 0}, upload=43362]: Done uploading piece. err: ecclient: upload failed (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3, address:104.36.82.60:4372): protocol: rpc: dial tcp 104.36.82.60:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:23:48.008865355 (49h15m23.361650493s) [noise=true , address=104.36.82.13:4189 , piece_id=4XOVTZEYAO7KUV7BSBXG5JKD6OVAUOJ2MXCVVOFPIMTHPDFKSTTA, seg_pos={0 0}, upload=43362]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:104.36.82.13:4189): protocol: rpc: dial tcp 104.36.82.13:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:23:48.008946028 (49h15m23.361731155s) [noise=true , address=104.36.82.7:4944 , piece_id=SQBTPTQ6AXUVGNKKJE5M2NLQ2QPWOMUIJ57NETL3MJ7ZSIZ3V3XA, seg_pos={0 0}, upload=43362]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: write tcp 5.161.89.252:23770->104.36.82.7:4944: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:23:48.009058659 (49h15m23.361843798s) [noise=true , address=104.36.82.4:4806 , piece_id=AQZNLHMRDSJW3FJDGMCUZE7IRKYULE2Z2EDOICQ3YXB5UEE6XEEQ, seg_pos={0 0}, upload=43362]: Done uploading piece. err: ecclient: upload failed (node:12qUyweKQhw15RjewCgKi1oLyCvN3VUB1VMpeGhj2Q2hszvJBsQ, address:104.36.82.4:4806): protocol: rpc: dial tcp 104.36.82.4:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:23:48.009089116 (49h15m23.361874245s) [noise=true , address=104.36.82.5:4857 , piece_id=XQ74MZALDTT7BBXVTMA2PQFCGVDYMJYU6FW3YBC27UOQRBFZZSNA, seg_pos={0 0}, upload=43362]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:23:48.009220894 (49h15m23.362006033s) [noise=true , address=104.36.82.14:4224 , piece_id=IPTRHEWNWEBEL5WGQKATNMM4M7BK2A5QMRGRUYEDLT3LPRAYNHKQ, seg_pos={0 0}, upload=43362]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:104.36.82.14:4224): protocol: rpc: dial tcp 104.36.82.14:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:23:48.009255500 (49h15m23.362040627s) [noise=true , address=104.36.82.23:4457 , piece_id=MXL3RSFELPH3L7B7AXE3OPBOWJIVGVONWNVJFZYEVXJ62LVOIHQQ, seg_pos={0 0}, upload=43362]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: rpc: dial tcp 104.36.82.23:4457: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:23:48.009320693 (49h15m23.362105821s) [noise=true , address=104.36.82.11:4134 , piece_id=BFLO5EIALPGWXDGXVFTLVM7SQJJHQX2NMBHG6D6KGHN4KXNUNIHQ, seg_pos={0 0}, upload=43362]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:23:48.009359636 (49h15m23.362144764s) [seg_pos={0 0}, upload=43362]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Aug 31 15:23:48.023607041 (49h15m23.376392149s) [upload=43362]: Done uploading segments...
Aug 31 15:23:48.023618813 (49h15m23.376403911s) [upload=43362]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 1m0.017208586s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................X.X] duration:
30.00069844s
FAIL (2025-08-31 15:23:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000716917s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
329.079919ms
FAIL (2025-08-31 15:23:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Aug 31 15:22:12.666186445 (49h13m48.018971544s) [upload=43354]: Starting upload
Aug 31 15:22:12.666234656 (49h13m48.019019764s) [upload=43354]: Done starting upload
Aug 31 15:22:12.666270643 (49h13m48.019055751s) [upload=43354]: Uploading segments...
Aug 31 15:22:12.666444441 (49h13m48.01922954s) [upload=43354]: Got next segment. Inline: false
Aug 31 15:22:12.666469468 (49h13m48.019254576s) [upload=43354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:22:12.666473015 (49h13m48.019258113s) [upload=43354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:22:12.691386522 (49h13m48.04417163s) [seg_pos={0 0}, upload=43354]: Begin upload segment...
Aug 31 15:22:12.691422941 (49h13m48.044208049s) [seg_pos={0 0}, upload=43354]: Done begin upload segment.
Aug 31 15:22:12.691448109 (49h13m48.044233217s) [noise=true , address=77.90.31.14:4662 , piece_id=VA7FDJHW2KDPR7ADXOSSMQXVMBD7FA63FDV6HATZR3WCBLHR3UGA, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.691469268 (49h13m48.044254377s) [noise=true , address=104.36.82.12:4150 , piece_id=CZHD2JUUYZNNP6GFR6773UGM5644R5ARXMQSOGVW2RYVY3TJEOOQ, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.691843752 (49h13m48.044628851s) [noise=true , address=77.90.31.15:4699 , piece_id=4JPISIZYKWY4VLNAQR4ZKALW5NRRUHSTO3RHHEKWAIFAUOVWWIIQ, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.691877707 (49h13m48.044662815s) [noise=true , address=77.90.31.14:4663 , piece_id=PXEREAUW7JBERYKKN3JFSE5OYUXKSVNXQD7WD4XD2JG7BBX5GL6Q, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692082102 (49h13m48.044867199s) [noise=true , address=45.91.116.6:4974 , piece_id=AKMATT24QDEMIO34SFZFVDU4AJO2Z32SFTLBZGOD4M3Z6TGMF2RA, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692139701 (49h13m48.044924809s) [noise=true , address=45.91.116.8:4045 , piece_id=DMWRCVK6JCEXVAH3VODUPE227ICHLXXXZAXFMZXUOD6JWZM3MUAQ, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692292819 (49h13m48.045077927s) [noise=true , address=104.36.82.14:4223 , piece_id=7RZ5MOKK2GS6COP3EES3GU55UGBMO2DXYL7PNDXFR33H3NN3VEDA, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692385302 (49h13m48.04517041s) [noise=true , address=45.91.116.8:4048 , piece_id=6IRXHH562GUYXJCYFJMNTRM56FJEIKC3APM7V77GZP4UF6AOLH4A, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692501551 (49h13m48.045286649s) [noise=true , address=45.91.116.14:4190 , piece_id=56ZBVR37LHOWEHW55UUYEQ7KO47UJ4ZZOBBV6324S5LEWOELJSYQ, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692596080 (49h13m48.045381188s) [noise=true , address=104.36.82.7:4912 , piece_id=MQSBI7OOGJAOXGJPP2CBOLJULUMEEVLHM5KXKKVHQUAJGXTTNV4Q, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692716737 (49h13m48.045501845s) [noise=true , address=45.91.116.4:4938 , piece_id=O3VWPQCXYK523J5QC6F52SXVGZDMTPX3P5B4F4JTNUWQXYHNME7A, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692814911 (49h13m48.045600009s) [noise=true , address=104.36.82.10:4079 , piece_id=FFQKRXKFUE2TT47CT4UFEJ2MNSVFXNNQTIWL33OOKG4A5CRCEH4A, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692938684 (49h13m48.045723792s) [noise=true , address=77.90.31.16:4735 , piece_id=6YRPM2NBNJ73MYVPNYQQL33CJW4SOVNXFWCYYC4MIHJHI55ZNXLQ, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.693156875 (49h13m48.045941973s) [noise=true , address=104.36.82.8:4972 , piece_id=EU7WHNC5JICA5AFP2AEJ2UGZONBDQLFDUF5HL2IC7EZ7LNCS3UVA, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.693362652 (49h13m48.04614775s) [noise=true , address=45.91.116.13:4155 , piece_id=LSDSQLNQYEK2RO7AVDMUERTAYM4TN3VJQZ5AGZWNV3OMEUA7SUCQ, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.693557860 (49h13m48.046342968s) [noise=true , address=45.91.116.9:4262 , piece_id=KEPQ2NZEOOIBABBNZ6XOZWUWY5AFLZMHQ3A2TUSAJ3BRVAG53IUA, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.763827152 (49h13m48.116612251s) [upload=43354]: Next returned nil segment
Aug 31 15:22:12.763848723 (49h13m48.116633831s) [upload=43354]: Waiting for error group managing segments...
Aug 31 15:22:12.772609198 (49h13m48.125394296s) [noise=true , address=45.91.116.13:4155 , piece_id=LSDSQLNQYEK2RO7AVDMUERTAYM4TN3VJQZ5AGZWNV3OMEUA7SUCQ, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.772757357 (49h13m48.125542465s) [noise=true , address=45.91.116.8:4048 , piece_id=6IRXHH562GUYXJCYFJMNTRM56FJEIKC3APM7V77GZP4UF6AOLH4A, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.773046571 (49h13m48.125831679s) [noise=true , address=45.91.116.9:4262 , piece_id=KEPQ2NZEOOIBABBNZ6XOZWUWY5AFLZMHQ3A2TUSAJ3BRVAG53IUA, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.773116973 (49h13m48.125902081s) [noise=true , address=45.91.116.4:4938 , piece_id=O3VWPQCXYK523J5QC6F52SXVGZDMTPX3P5B4F4JTNUWQXYHNME7A, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.773410215 (49h13m48.126195323s) [noise=true , address=45.91.116.8:4045 , piece_id=DMWRCVK6JCEXVAH3VODUPE227ICHLXXXZAXFMZXUOD6JWZM3MUAQ, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.773487230 (49h13m48.126272339s) [noise=true , address=45.91.116.6:4974 , piece_id=AKMATT24QDEMIO34SFZFVDU4AJO2Z32SFTLBZGOD4M3Z6TGMF2RA, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.773842580 (49h13m48.126627678s) [noise=true , address=77.90.31.14:4663 , piece_id=PXEREAUW7JBERYKKN3JFSE5OYUXKSVNXQD7WD4XD2JG7BBX5GL6Q, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.773868890 (49h13m48.126653998s) [noise=true , address=45.91.116.14:4190 , piece_id=56ZBVR37LHOWEHW55UUYEQ7KO47UJ4ZZOBBV6324S5LEWOELJSYQ, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.774180255 (49h13m48.126965363s) [noise=true , address=77.90.31.16:4735 , piece_id=6YRPM2NBNJ73MYVPNYQQL33CJW4SOVNXFWCYYC4MIHJHI55ZNXLQ, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.774316472 (49h13m48.12710158s) [noise=true , address=77.90.31.14:4662 , piece_id=VA7FDJHW2KDPR7ADXOSSMQXVMBD7FA63FDV6HATZR3WCBLHR3UGA, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.775768005 (49h13m48.128553113s) [noise=true , address=77.90.31.15:4699 , piece_id=4JPISIZYKWY4VLNAQR4ZKALW5NRRUHSTO3RHHEKWAIFAUOVWWIIQ, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:42.650345379 (49h14m18.003130527s) [noise=true , address=104.36.82.7:4912 , piece_id=MQSBI7OOGJAOXGJPP2CBOLJULUMEEVLHM5KXKKVHQUAJGXTTNV4Q, seg_pos={0 0}, upload=43354]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:42.650401756 (49h14m18.003186854s) [noise=true , address=104.36.82.12:4150 , piece_id=CZHD2JUUYZNNP6GFR6773UGM5644R5ARXMQSOGVW2RYVY3TJEOOQ, seg_pos={0 0}, upload=43354]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 5.161.89.252:51496->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:42.650446029 (49h14m18.003231137s) [noise=true , address=104.36.82.8:4972 , piece_id=EU7WHNC5JICA5AFP2AEJ2UGZONBDQLFDUF5HL2IC7EZ7LNCS3UVA, seg_pos={0 0}, upload=43354]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 5.161.89.252:41344->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:42.650491115 (49h14m18.003276223s) [noise=true , address=104.36.82.14:4223 , piece_id=7RZ5MOKK2GS6COP3EES3GU55UGBMO2DXYL7PNDXFR33H3NN3VEDA, seg_pos={0 0}, upload=43354]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 5.161.89.252:61462->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:42.650494972 (49h14m18.00328008s) [noise=true , address=104.36.82.10:4079 , piece_id=FFQKRXKFUE2TT47CT4UFEJ2MNSVFXNNQTIWL33OOKG4A5CRCEH4A, seg_pos={0 0}, upload=43354]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:42.650552409 (49h14m18.003337508s) [seg_pos={0 0}, upload=43354]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Aug 31 15:22:42.666217263 (49h14m18.019002361s) [upload=43354]: Done uploading segments...
Aug 31 15:22:42.666234134 (49h14m18.019019233s) [upload=43354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 329.109835ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
1m0.00091168s
FAIL (2025-08-31 15:22:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000925246s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.018401274s
FAIL (2025-08-31 15:22:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
E4NNMPISFQ, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.974990363 (49h13m23.327775492s) [noise=true , address=79.127.205.246:4282 , piece_id=57ONSLHDYEKNHOYK73II2VHBO6OYEBKLDIJD6CVBSV4SFS4R6RKQ, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.975247808 (49h13m23.328032916s) [noise=true , address=104.36.82.14:4240 , piece_id=2RD3URPEI7WC7RFNDJG4OMLKSM7XZG3GXKLG2JUQAH5XZNBRZBIA, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.975419552 (49h13m23.328204689s) [noise=true , address=104.36.82.40:4944 , piece_id=2D243TAZUE65HFG7P73MWK3ITO43CGUVGPOVO5AINRJJ3HVHM7HQ, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.975477841 (49h13m23.328262949s) [noise=true , address=77.90.31.21:4888 , piece_id=3BCQM76O6C3CXZY4W55I4LRBLL4QJGXA6XVMP6WH4UMUO4MI6KCA, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.975729835 (49h13m23.328514934s) [noise=true , address=104.36.82.48:4571 , piece_id=UKZ7YODSXTR2ZMZTDZ6NOPHPXIXTTHARX7AASJO2B4XP7BNQP4BA, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.975935142 (49h13m23.328720271s) [noise=true , address=104.36.82.44:4883 , piece_id=FM4F4CLZGH77ETQWPUZIQ2J2CKGDQHILR4D5B6PYLF7GYI5D2WDQ, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.975967503 (49h13m23.328752601s) [noise=true , address=77.90.31.17:4778 , piece_id=LI2E6CD44YDG4PCJ4CRPBVIJJUADC2HCAB32TL4DST7ST75BY6TQ, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.976177619 (49h13m23.328962727s) [noise=true , address=104.36.82.43:4848 , piece_id=SC6DQ3LAOCPIXLIAU4LLJG753DOBSE6T5LV4AVAYKEGTY4FOWTYA, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.976345365 (49h13m23.329130493s) [noise=true , address=77.90.31.16:4753 , piece_id=VNDOTSF2A6NSY5QOEOX7EHD75Z2S5AXCOY4WEMOI7UPWUFFMYN4A, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.976392393 (49h13m23.329177501s) [noise=true , address=104.36.82.46:4506 , piece_id=PEVRRYGHG2XGMVDUO5VXT7PVOFEPNB4XISJWYIVD4OXI76CEQVYQ, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.976742964 (49h13m23.329528101s) [noise=true , address=104.36.82.4:4826 , piece_id=GAKV5MMP36D3S6SFFEYK3DCFBAHHPO6TCDADPAWNUHCBS7AKBADA, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.977141634 (49h13m23.329926761s) [noise=true , address=104.36.82.10:4101 , piece_id=E7YPL2VDA3WBB3JIQL3NJWT7SQPD367SJX63XOS42F2TSNU6FC3Q, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:47.977512031 (49h13m23.330297169s) [noise=true , address=104.36.82.23:4445 , piece_id=RRHELKFL3RIERN2ZOJQUKO4UQ4SJ57UAXBGGO7D6VJSY7NCK4A6Q, seg_pos={0 0}, upload=43349]: Uploading piece...
Aug 31 15:21:48.048819588 (49h13m23.401604686s) [upload=43349]: Next returned nil segment
Aug 31 15:21:48.048829877 (49h13m23.401614985s) [upload=43349]: Waiting for error group managing segments...
Aug 31 15:21:48.057550397 (49h13m23.410335505s) [noise=true , address=77.90.31.16:4753 , piece_id=VNDOTSF2A6NSY5QOEOX7EHD75Z2S5AXCOY4WEMOI7UPWUFFMYN4A, seg_pos={0 0}, upload=43349]: Done uploading piece. err: <nil>
Aug 31 15:21:48.058148503 (49h13m23.410933632s) [noise=true , address=77.90.31.21:4888 , piece_id=3BCQM76O6C3CXZY4W55I4LRBLL4QJGXA6XVMP6WH4UMUO4MI6KCA, seg_pos={0 0}, upload=43349]: Done uploading piece. err: <nil>
Aug 31 15:21:48.065839072 (49h13m23.41862417s) [noise=true , address=77.90.31.17:4778 , piece_id=LI2E6CD44YDG4PCJ4CRPBVIJJUADC2HCAB32TL4DST7ST75BY6TQ, seg_pos={0 0}, upload=43349]: Done uploading piece. err: <nil>
Aug 31 15:21:48.075459605 (49h13m23.428244743s) [noise=true , address=45.91.116.11:4121 , piece_id=QBJTKFXG5TA3UWNGTUBFGC2DN3CJ7J27SL5FFYMPW7RJTMGGCAXA, seg_pos={0 0}, upload=43349]: Done uploading piece. err: <nil>
Aug 31 15:21:48.510111696 (49h13m23.862896825s) [noise=true , address=79.127.205.246:4282 , piece_id=57ONSLHDYEKNHOYK73II2VHBO6OYEBKLDIJD6CVBSV4SFS4R6RKQ, seg_pos={0 0}, upload=43349]: Done uploading piece. err: <nil>
Aug 31 15:21:48.551788294 (49h13m23.904573432s) [noise=true , address=79.127.205.245:4249 , piece_id=LIJLDBONX2OKUSTDKDRRBZZ3O5ZJMIYJULXCWKCKMBU562BP5XHQ, seg_pos={0 0}, upload=43349]: Done uploading piece. err: <nil>
Aug 31 15:22:47.931078654 (49h14m23.283863793s) [noise=true , address=104.36.82.14:4240 , piece_id=2RD3URPEI7WC7RFNDJG4OMLKSM7XZG3GXKLG2JUQAH5XZNBRZBIA, seg_pos={0 0}, upload=43349]: Done uploading piece. err: ecclient: upload failed (node:13SaC5StbguPT3abAT9oTX3mZAyGhJcpAA6b7ueyiDFHa8CCEo, address:104.36.82.14:4240): protocol: rpc: dial tcp 104.36.82.14:4240: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:47.931148197 (49h14m23.283933324s) [noise=true , address=104.36.82.7:4922 , piece_id=K6EH463NLZMU2N7NCRWMS5L5JOLUBFU77P3PWF22VFSQ5JPKTIYQ, seg_pos={0 0}, upload=43349]: Done uploading piece. err: ecclient: upload failed (node:12KrMgT4Pwj9UFE88q6tt65D7o6q68NhsmEXmifNmvzJ6eEkm8R, address:104.36.82.7:4922): protocol: rpc: dial tcp 104.36.82.7:4922: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:47.931204903 (49h14m23.283990031s) [noise=true , address=104.36.82.42:4829 , piece_id=OOOWM2RYRB2LXHISHDZGHIGY6EQZXYBX6BVMP2PKEPE4NNMPISFQ, seg_pos={0 0}, upload=43349]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:47.931254967 (49h14m23.284040094s) [noise=true , address=104.36.82.44:4883 , piece_id=FM4F4CLZGH77ETQWPUZIQ2J2CKGDQHILR4D5B6PYLF7GYI5D2WDQ, seg_pos={0 0}, upload=43349]: Done uploading piece. err: ecclient: upload failed (node:12cx1avL4umDm2pzJS1rNZqeDZjZMWnMtNsyNTdMdkxhpa1ByUN, address:104.36.82.44:4883): protocol: write tcp 5.161.89.252:54072->104.36.82.44:4883: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:47.931316262 (49h14m23.284101391s) [noise=true , address=104.36.82.23:4445 , piece_id=RRHELKFL3RIERN2ZOJQUKO4UQ4SJ57UAXBGGO7D6VJSY7NCK4A6Q, seg_pos={0 0}, upload=43349]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:104.36.82.23:4445): protocol: rpc: dial tcp 104.36.82.23:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:47.931366817 (49h14m23.284151945s) [noise=true , address=104.36.82.40:4944 , piece_id=2D243TAZUE65HFG7P73MWK3ITO43CGUVGPOVO5AINRJJ3HVHM7HQ, seg_pos={0 0}, upload=43349]: Done uploading piece. err: ecclient: upload failed (node:12KMFsxzp7SfMiFBuMTaWGVyJoRTMw7ZRcfEpPDv1BmbViGF6K9, address:104.36.82.40:4944): protocol: rpc: dial tcp 104.36.82.40:4944: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:47.931412293 (49h14m23.284197421s) [noise=true , address=104.36.82.48:4571 , piece_id=UKZ7YODSXTR2ZMZTDZ6NOPHPXIXTTHARX7AASJO2B4XP7BNQP4BA, seg_pos={0 0}, upload=43349]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:104.36.82.48:4571): protocol: rpc: dial tcp 104.36.82.48:4571: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:47.931449413 (49h14m23.284234541s) [noise=true , address=104.36.82.46:4506 , piece_id=PEVRRYGHG2XGMVDUO5VXT7PVOFEPNB4XISJWYIVD4OXI76CEQVYQ, seg_pos={0 0}, upload=43349]: Done uploading piece. err: ecclient: upload failed (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9, address:104.36.82.46:4506): protocol: rpc: dial tcp 104.36.82.46:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:47.931497784 (49h14m23.284282922s) [noise=true , address=104.36.82.43:4848 , piece_id=SC6DQ3LAOCPIXLIAU4LLJG753DOBSE6T5LV4AVAYKEGTY4FOWTYA, seg_pos={0 0}, upload=43349]: Done uploading piece. err: ecclient: upload failed (node:1DPLv5Kfsx9RLVWeRrxbdkaLGJc4dfrzGnPLwc8us9hT5uiPbj, address:104.36.82.43:4848): protocol: rpc: dial tcp 104.36.82.43:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:47.931538661 (49h14m23.284323799s) [noise=true , address=104.36.82.4:4826 , piece_id=GAKV5MMP36D3S6SFFEYK3DCFBAHHPO6TCDADPAWNUHCBS7AKBADA, seg_pos={0 0}, upload=43349]: Done uploading piece. err: ecclient: upload failed (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548, address:104.36.82.4:4826): protocol: rpc: dial tcp 104.36.82.4:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:47.931573808 (49h14m23.284358935s) [noise=true , address=104.36.82.43:4868 , piece_id=OSCWUCWO6MHQ3PZWKLQJSP7YWG3SM56OOW3JHUZLMU6RDPENOWBQ, seg_pos={0 0}, upload=43349]: Done uploading piece. err: ecclient: upload failed (node:124LJtYshPwu1uZJwVyjrjUMk3htyVnp9ntnZWSUb7X3bNkwU6t, address:104.36.82.43:4868): protocol: rpc: dial tcp 104.36.82.43:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:47.931600458 (49h14m23.284385585s) [noise=true , address=104.36.82.10:4101 , piece_id=E7YPL2VDA3WBB3JIQL3NJWT7SQPD367SJX63XOS42F2TSNU6FC3Q, seg_pos={0 0}, upload=43349]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 5.161.89.252:63354->104.36.82.10:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:47.931647786 (49h14m23.284432914s) [seg_pos={0 0}, upload=43349]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Aug 31 15:22:47.947465537 (49h14m23.300250635s) [upload=43349]: Done uploading segments...
Aug 31 15:22:47.947485805 (49h14m23.300270914s) [upload=43349]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 1m0.018417835s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.017156242s
FAIL (2025-08-31 15:22:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Aug 31 15:22:12.666186445 (49h13m48.018971544s) [upload=43354]: Starting upload
Aug 31 15:22:12.666234656 (49h13m48.019019764s) [upload=43354]: Done starting upload
Aug 31 15:22:12.666270643 (49h13m48.019055751s) [upload=43354]: Uploading segments...
Aug 31 15:22:12.666444441 (49h13m48.01922954s) [upload=43354]: Got next segment. Inline: false
Aug 31 15:22:12.666469468 (49h13m48.019254576s) [upload=43354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:22:12.666473015 (49h13m48.019258113s) [upload=43354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:22:12.691386522 (49h13m48.04417163s) [seg_pos={0 0}, upload=43354]: Begin upload segment...
Aug 31 15:22:12.691422941 (49h13m48.044208049s) [seg_pos={0 0}, upload=43354]: Done begin upload segment.
Aug 31 15:22:12.691448109 (49h13m48.044233217s) [noise=true , address=77.90.31.14:4662 , piece_id=VA7FDJHW2KDPR7ADXOSSMQXVMBD7FA63FDV6HATZR3WCBLHR3UGA, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.691469268 (49h13m48.044254377s) [noise=true , address=104.36.82.12:4150 , piece_id=CZHD2JUUYZNNP6GFR6773UGM5644R5ARXMQSOGVW2RYVY3TJEOOQ, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.691843752 (49h13m48.044628851s) [noise=true , address=77.90.31.15:4699 , piece_id=4JPISIZYKWY4VLNAQR4ZKALW5NRRUHSTO3RHHEKWAIFAUOVWWIIQ, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.691877707 (49h13m48.044662815s) [noise=true , address=77.90.31.14:4663 , piece_id=PXEREAUW7JBERYKKN3JFSE5OYUXKSVNXQD7WD4XD2JG7BBX5GL6Q, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692082102 (49h13m48.044867199s) [noise=true , address=45.91.116.6:4974 , piece_id=AKMATT24QDEMIO34SFZFVDU4AJO2Z32SFTLBZGOD4M3Z6TGMF2RA, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692139701 (49h13m48.044924809s) [noise=true , address=45.91.116.8:4045 , piece_id=DMWRCVK6JCEXVAH3VODUPE227ICHLXXXZAXFMZXUOD6JWZM3MUAQ, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692292819 (49h13m48.045077927s) [noise=true , address=104.36.82.14:4223 , piece_id=7RZ5MOKK2GS6COP3EES3GU55UGBMO2DXYL7PNDXFR33H3NN3VEDA, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692385302 (49h13m48.04517041s) [noise=true , address=45.91.116.8:4048 , piece_id=6IRXHH562GUYXJCYFJMNTRM56FJEIKC3APM7V77GZP4UF6AOLH4A, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692501551 (49h13m48.045286649s) [noise=true , address=45.91.116.14:4190 , piece_id=56ZBVR37LHOWEHW55UUYEQ7KO47UJ4ZZOBBV6324S5LEWOELJSYQ, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692596080 (49h13m48.045381188s) [noise=true , address=104.36.82.7:4912 , piece_id=MQSBI7OOGJAOXGJPP2CBOLJULUMEEVLHM5KXKKVHQUAJGXTTNV4Q, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692716737 (49h13m48.045501845s) [noise=true , address=45.91.116.4:4938 , piece_id=O3VWPQCXYK523J5QC6F52SXVGZDMTPX3P5B4F4JTNUWQXYHNME7A, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692814911 (49h13m48.045600009s) [noise=true , address=104.36.82.10:4079 , piece_id=FFQKRXKFUE2TT47CT4UFEJ2MNSVFXNNQTIWL33OOKG4A5CRCEH4A, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.692938684 (49h13m48.045723792s) [noise=true , address=77.90.31.16:4735 , piece_id=6YRPM2NBNJ73MYVPNYQQL33CJW4SOVNXFWCYYC4MIHJHI55ZNXLQ, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.693156875 (49h13m48.045941973s) [noise=true , address=104.36.82.8:4972 , piece_id=EU7WHNC5JICA5AFP2AEJ2UGZONBDQLFDUF5HL2IC7EZ7LNCS3UVA, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.693362652 (49h13m48.04614775s) [noise=true , address=45.91.116.13:4155 , piece_id=LSDSQLNQYEK2RO7AVDMUERTAYM4TN3VJQZ5AGZWNV3OMEUA7SUCQ, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.693557860 (49h13m48.046342968s) [noise=true , address=45.91.116.9:4262 , piece_id=KEPQ2NZEOOIBABBNZ6XOZWUWY5AFLZMHQ3A2TUSAJ3BRVAG53IUA, seg_pos={0 0}, upload=43354]: Uploading piece...
Aug 31 15:22:12.763827152 (49h13m48.116612251s) [upload=43354]: Next returned nil segment
Aug 31 15:22:12.763848723 (49h13m48.116633831s) [upload=43354]: Waiting for error group managing segments...
Aug 31 15:22:12.772609198 (49h13m48.125394296s) [noise=true , address=45.91.116.13:4155 , piece_id=LSDSQLNQYEK2RO7AVDMUERTAYM4TN3VJQZ5AGZWNV3OMEUA7SUCQ, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.772757357 (49h13m48.125542465s) [noise=true , address=45.91.116.8:4048 , piece_id=6IRXHH562GUYXJCYFJMNTRM56FJEIKC3APM7V77GZP4UF6AOLH4A, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.773046571 (49h13m48.125831679s) [noise=true , address=45.91.116.9:4262 , piece_id=KEPQ2NZEOOIBABBNZ6XOZWUWY5AFLZMHQ3A2TUSAJ3BRVAG53IUA, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.773116973 (49h13m48.125902081s) [noise=true , address=45.91.116.4:4938 , piece_id=O3VWPQCXYK523J5QC6F52SXVGZDMTPX3P5B4F4JTNUWQXYHNME7A, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.773410215 (49h13m48.126195323s) [noise=true , address=45.91.116.8:4045 , piece_id=DMWRCVK6JCEXVAH3VODUPE227ICHLXXXZAXFMZXUOD6JWZM3MUAQ, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.773487230 (49h13m48.126272339s) [noise=true , address=45.91.116.6:4974 , piece_id=AKMATT24QDEMIO34SFZFVDU4AJO2Z32SFTLBZGOD4M3Z6TGMF2RA, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.773842580 (49h13m48.126627678s) [noise=true , address=77.90.31.14:4663 , piece_id=PXEREAUW7JBERYKKN3JFSE5OYUXKSVNXQD7WD4XD2JG7BBX5GL6Q, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.773868890 (49h13m48.126653998s) [noise=true , address=45.91.116.14:4190 , piece_id=56ZBVR37LHOWEHW55UUYEQ7KO47UJ4ZZOBBV6324S5LEWOELJSYQ, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.774180255 (49h13m48.126965363s) [noise=true , address=77.90.31.16:4735 , piece_id=6YRPM2NBNJ73MYVPNYQQL33CJW4SOVNXFWCYYC4MIHJHI55ZNXLQ, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.774316472 (49h13m48.12710158s) [noise=true , address=77.90.31.14:4662 , piece_id=VA7FDJHW2KDPR7ADXOSSMQXVMBD7FA63FDV6HATZR3WCBLHR3UGA, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:12.775768005 (49h13m48.128553113s) [noise=true , address=77.90.31.15:4699 , piece_id=4JPISIZYKWY4VLNAQR4ZKALW5NRRUHSTO3RHHEKWAIFAUOVWWIIQ, seg_pos={0 0}, upload=43354]: Done uploading piece. err: <nil>
Aug 31 15:22:42.650345379 (49h14m18.003130527s) [noise=true , address=104.36.82.7:4912 , piece_id=MQSBI7OOGJAOXGJPP2CBOLJULUMEEVLHM5KXKKVHQUAJGXTTNV4Q, seg_pos={0 0}, upload=43354]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:42.650401756 (49h14m18.003186854s) [noise=true , address=104.36.82.12:4150 , piece_id=CZHD2JUUYZNNP6GFR6773UGM5644R5ARXMQSOGVW2RYVY3TJEOOQ, seg_pos={0 0}, upload=43354]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 5.161.89.252:51496->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:42.650446029 (49h14m18.003231137s) [noise=true , address=104.36.82.8:4972 , piece_id=EU7WHNC5JICA5AFP2AEJ2UGZONBDQLFDUF5HL2IC7EZ7LNCS3UVA, seg_pos={0 0}, upload=43354]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 5.161.89.252:41344->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:42.650491115 (49h14m18.003276223s) [noise=true , address=104.36.82.14:4223 , piece_id=7RZ5MOKK2GS6COP3EES3GU55UGBMO2DXYL7PNDXFR33H3NN3VEDA, seg_pos={0 0}, upload=43354]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 5.161.89.252:61462->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:42.650494972 (49h14m18.00328008s) [noise=true , address=104.36.82.10:4079 , piece_id=FFQKRXKFUE2TT47CT4UFEJ2MNSVFXNNQTIWL33OOKG4A5CRCEH4A, seg_pos={0 0}, upload=43354]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:42.650552409 (49h14m18.003337508s) [seg_pos={0 0}, upload=43354]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Aug 31 15:22:42.666217263 (49h14m18.019002361s) [upload=43354]: Done uploading segments...
Aug 31 15:22:42.666234134 (49h14m18.019019233s) [upload=43354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.017177782s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
334.757431ms
FAIL (2025-08-31 15:22:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 334.7782ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000759842s
FAIL (2025-08-31 15:22:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000779659s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.016814056s
FAIL (2025-08-31 15:22:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Aug 31 15:21:37.640010254 (49h13m12.992795382s) [upload=43348]: Starting upload
Aug 31 15:21:37.640075837 (49h13m12.992860965s) [upload=43348]: Done starting upload
Aug 31 15:21:37.640259593 (49h13m12.993044721s) [upload=43348]: Uploading segments...
Aug 31 15:21:37.640557624 (49h13m12.993342763s) [upload=43348]: Got next segment. Inline: false
Aug 31 15:21:37.640609922 (49h13m12.993395051s) [upload=43348]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:21:37.640619500 (49h13m12.993404629s) [upload=43348]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:21:37.669375402 (49h13m13.02216053s) [seg_pos={0 0}, upload=43348]: Begin upload segment...
Aug 31 15:21:37.669414045 (49h13m13.022199173s) [seg_pos={0 0}, upload=43348]: Done begin upload segment.
Aug 31 15:21:37.669441466 (49h13m13.022226594s) [noise=true , address=77.90.31.13:4627 , piece_id=B3MMBQYE24IVKU5FHFWB5SBHOPINYBFNYODB3QQP6QDCTHFZ7IOA, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.669536125 (49h13m13.022321263s) [noise=true , address=104.36.82.7:4912 , piece_id=WJGWPKCXL3KVRA7SSV3O64VQCP32SEEYA2V5QFVIYNVEKUCPMFHA, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.670111467 (49h13m13.022896596s) [noise=true , address=45.91.116.13:4153 , piece_id=M2QJWF4JNA75LLDRKOMVZJP2EM6KZC5B5YXDD5S4H76KXSCE6CBQ, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.670330380 (49h13m13.023115508s) [noise=true , address=45.91.116.11:4117 , piece_id=B34AVC5GX57WECZIRF3662NNVB5KVZVI2U5N3BSGOTEMQAKTGKLA, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.670601310 (49h13m13.023386439s) [noise=true , address=45.91.116.9:4263 , piece_id=CYY6LLVZRZBH6DM4XCI2RQWNTKB32JU64MZJI5KVOEGHXWIZJ3BA, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.670796247 (49h13m13.023581375s) [noise=true , address=45.91.116.5:4297 , piece_id=BHJDR22TOFN43MWBZMWBDKWUPV3UKZDRYC6FALNFIL7OA3SSJ2ZA, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.671124635 (49h13m13.023909763s) [noise=true , address=104.36.82.8:4969 , piece_id=IYLQJ5TWIJTKBZSRMOTKTIK4IW4J7RJCUG3RF7F3LRZ4NVBHGVDQ, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.671463594 (49h13m13.024248721s) [noise=true , address=45.91.116.10:4083 , piece_id=M4LOBRW5I7TZ4WHUXORL3OBHWBXODGOA73JI5KS6UTIU62VFDRTQ, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.671566298 (49h13m13.024351425s) [noise=true , address=104.36.82.2:4730 , piece_id=DYM23BAH2CSXRCSI5QVGBX56SU76HDXQI5ASOB6RBSLB6IJNVV5A, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.672087067 (49h13m13.024872205s) [noise=true , address=77.90.31.14:4661 , piece_id=SN4ZFQDDECJTYIHXRBRQ7DWBCYRKETOTYEYFC6RS3SXQBD7YTXOQ, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.672531023 (49h13m13.025316152s) [noise=true , address=104.36.82.12:4151 , piece_id=EDYPGVUDEU5JG4TRD4HAYRS4CAM5WIU4XBGLEFGVXIZSMMHGIQDA, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.672561921 (49h13m13.02534705s) [noise=true , address=104.36.82.2:4732 , piece_id=YTQ4B33GQQWIZ3B3EHJ4NZ2QTDCBS5CMMSWQBCY64SJWHYM5XFCQ, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.673086389 (49h13m13.025871517s) [noise=true , address=77.90.31.12:4589 , piece_id=Q235LCEIHDYSXUJNKQV3VAY6Z42SSYUVGYPGTTKHBOZBQ6FVEUVQ, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.673282157 (49h13m13.026067286s) [noise=true , address=45.91.116.4:4937 , piece_id=TIZLSCLSU4QKOVYV5VQMNPJEESBE3C35E3HSUTEDVTCFF3C6ZWDQ, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.673833235 (49h13m13.026618363s) [noise=true , address=77.90.31.18:4807 , piece_id=7L6W3KAJHHZCT4VZRE6WHWABQX2YYMZY7WENYD4TLIOWSNMVD76A, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.674303711 (49h13m13.02708884s) [noise=true , address=77.90.31.15:4697 , piece_id=2AKAALMZIAU3N7UYICJQMSWD3AHHYAORK6XNCRF7PIWJEBS5Y4OQ, seg_pos={0 0}, upload=43348]: Uploading piece...
Aug 31 15:21:37.800256584 (49h13m13.153041682s) [upload=43348]: Next returned nil segment
Aug 31 15:21:37.800267875 (49h13m13.153052984s) [upload=43348]: Waiting for error group managing segments...
Aug 31 15:21:37.807903883 (49h13m13.160688991s) [noise=true , address=45.91.116.11:4117 , piece_id=B34AVC5GX57WECZIRF3662NNVB5KVZVI2U5N3BSGOTEMQAKTGKLA, seg_pos={0 0}, upload=43348]: Done uploading piece. err: <nil>
Aug 31 15:21:37.808575117 (49h13m13.161360215s) [noise=true , address=45.91.116.4:4937 , piece_id=TIZLSCLSU4QKOVYV5VQMNPJEESBE3C35E3HSUTEDVTCFF3C6ZWDQ, seg_pos={0 0}, upload=43348]: Done uploading piece. err: <nil>
Aug 31 15:21:37.808614110 (49h13m13.161399218s) [noise=true , address=45.91.116.10:4083 , piece_id=M4LOBRW5I7TZ4WHUXORL3OBHWBXODGOA73JI5KS6UTIU62VFDRTQ, seg_pos={0 0}, upload=43348]: Done uploading piece. err: <nil>
Aug 31 15:21:37.809059819 (49h13m13.161844917s) [noise=true , address=45.91.116.13:4153 , piece_id=M2QJWF4JNA75LLDRKOMVZJP2EM6KZC5B5YXDD5S4H76KXSCE6CBQ, seg_pos={0 0}, upload=43348]: Done uploading piece. err: <nil>
Aug 31 15:21:37.809610025 (49h13m13.162395133s) [noise=true , address=77.90.31.18:4807 , piece_id=7L6W3KAJHHZCT4VZRE6WHWABQX2YYMZY7WENYD4TLIOWSNMVD76A, seg_pos={0 0}, upload=43348]: Done uploading piece. err: <nil>
Aug 31 15:21:37.809723990 (49h13m13.162509098s) [noise=true , address=45.91.116.9:4263 , piece_id=CYY6LLVZRZBH6DM4XCI2RQWNTKB32JU64MZJI5KVOEGHXWIZJ3BA, seg_pos={0 0}, upload=43348]: Done uploading piece. err: <nil>
Aug 31 15:21:37.810073849 (49h13m13.162858947s) [noise=true , address=77.90.31.14:4661 , piece_id=SN4ZFQDDECJTYIHXRBRQ7DWBCYRKETOTYEYFC6RS3SXQBD7YTXOQ, seg_pos={0 0}, upload=43348]: Done uploading piece. err: <nil>
Aug 31 15:21:37.810478429 (49h13m13.163263537s) [noise=true , address=45.91.116.5:4297 , piece_id=BHJDR22TOFN43MWBZMWBDKWUPV3UKZDRYC6FALNFIL7OA3SSJ2ZA, seg_pos={0 0}, upload=43348]: Done uploading piece. err: <nil>
Aug 31 15:21:37.810500061 (49h13m13.163285169s) [noise=true , address=77.90.31.12:4589 , piece_id=Q235LCEIHDYSXUJNKQV3VAY6Z42SSYUVGYPGTTKHBOZBQ6FVEUVQ, seg_pos={0 0}, upload=43348]: Done uploading piece. err: <nil>
Aug 31 15:21:37.811772918 (49h13m13.164558026s) [noise=true , address=77.90.31.13:4627 , piece_id=B3MMBQYE24IVKU5FHFWB5SBHOPINYBFNYODB3QQP6QDCTHFZ7IOA, seg_pos={0 0}, upload=43348]: Done uploading piece. err: <nil>
Aug 31 15:21:37.812540764 (49h13m13.165325872s) [noise=true , address=77.90.31.15:4697 , piece_id=2AKAALMZIAU3N7UYICJQMSWD3AHHYAORK6XNCRF7PIWJEBS5Y4OQ, seg_pos={0 0}, upload=43348]: Done uploading piece. err: <nil>
Aug 31 15:22:07.615483633 (49h13m42.968268741s) [noise=true , address=104.36.82.8:4969 , piece_id=IYLQJ5TWIJTKBZSRMOTKTIK4IW4J7RJCUG3RF7F3LRZ4NVBHGVDQ, seg_pos={0 0}, upload=43348]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:07.615565306 (49h13m42.968350405s) [noise=true , address=104.36.82.2:4732 , piece_id=YTQ4B33GQQWIZ3B3EHJ4NZ2QTDCBS5CMMSWQBCY64SJWHYM5XFCQ, seg_pos={0 0}, upload=43348]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:07.615604070 (49h13m42.968389178s) [noise=true , address=104.36.82.7:4912 , piece_id=WJGWPKCXL3KVRA7SSV3O64VQCP32SEEYA2V5QFVIYNVEKUCPMFHA, seg_pos={0 0}, upload=43348]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:07.615612005 (49h13m42.968397113s) [noise=true , address=104.36.82.12:4151 , piece_id=EDYPGVUDEU5JG4TRD4HAYRS4CAM5WIU4XBGLEFGVXIZSMMHGIQDA, seg_pos={0 0}, upload=43348]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: context deadline exceeded; context deadline exceeded
Aug 31 15:22:07.615649696 (49h13m42.968434804s) [noise=true , address=104.36.82.2:4730 , piece_id=DYM23BAH2CSXRCSI5QVGBX56SU76HDXQI5ASOB6RBSLB6IJNVV5A, seg_pos={0 0}, upload=43348]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 5.161.89.252:17638->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:22:07.615670515 (49h13m42.968455623s) [seg_pos={0 0}, upload=43348]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Aug 31 15:22:07.630426857 (49h13m42.983211965s) [upload=43348]: Done uploading segments...
Aug 31 15:22:07.630438369 (49h13m42.983223468s) [upload=43348]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.016826348s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001634546s
FAIL (2025-08-31 15:21:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001659864s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
626.038743ms
FAIL (2025-08-31 13:35:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 626.064192ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
778.656079ms
FAIL (2025-08-31 13:34:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 778.670146ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
928.778207ms
FAIL (2025-08-31 13:33:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 928.797964ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
915.879711ms
FAIL (2025-08-31 13:32:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 916.15494ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
734.460735ms
FAIL (2025-08-31 13:31:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 734.698854ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
733.019292ms
FAIL (2025-08-31 13:30:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 733.0344ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
724.306797ms
FAIL (2025-08-31 13:30:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 724.330431ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
622.229159ms
FAIL (2025-08-31 13:29:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 622.254317ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
744.324154ms
FAIL (2025-08-31 13:28:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 744.339553ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
781.120408ms
FAIL (2025-08-31 13:27:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 781.171766ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
671.403167ms
FAIL (2025-08-31 13:27:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 671.449845ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
827.399466ms
FAIL (2025-08-31 13:26:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 827.420414ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
998.647533ms
FAIL (2025-08-31 13:25:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 998.671498ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
627.66236ms
FAIL (2025-08-31 13:25:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 627.682627ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
890.40684ms
FAIL (2025-08-31 13:24:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 890.43281ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
610.544102ms
FAIL (2025-08-31 13:23:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 610.563128ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
644.081167ms
FAIL (2025-08-31 13:23:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 644.101235ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
685.226638ms
FAIL (2025-08-31 13:22:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 685.250382ms
core
uplink
us-select-1
us1-placement-20
|