Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 1m0.018110438s FAIL (2025-09-22 05:05:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== oise=true , address=145.79.227.30:4775 , piece_id=6Y5W2DO3TNAYW4UYCM7W5D56F57JLE5WSTQQT4WPMPAJE7ZQKIUA, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.869236202 (566h56m8.22202133s) [noise=true , address=145.79.227.39:4013 , piece_id=OISIFQIDVNAG6G77B2EOUYM7WC2NENG3DGS64K5WEPUN5EQISC2Q, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.869380733 (566h56m8.222165862s) [noise=true , address=145.79.227.38:4980 , piece_id=TT7VT2MZW2TMOYSU325BB2MFQH7BIQQ75GT6M56JWSZBPGBVAJAA, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.869618289 (566h56m8.222403427s) [noise=true , address=145.79.227.17:4977 , piece_id=QW6HIOFGC47BVNYUMZTNWUN7GCQSWZKRCMKG6BNM3GLVLQ66UCWA, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.869937808 (566h56m8.222722946s) [noise=true , address=145.79.227.12:4794 , piece_id=TX3XR7UNCENTL3C4A5O2ACH2ST6BSQEAD2R7Y4W5EXIHZQF6IBXA, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.869996529 (566h56m8.222781658s) [noise=true , address=145.79.227.41:4088 , piece_id=YQ33LISFICOHVI37GJOWRYEGKE3YIXOBNEARYHUFWJFVT377OV4A, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.870352486 (566h56m8.223137625s) [noise=true , address=77.90.31.15:4705 , piece_id=YNII3RGVISQN5FBTHCGJRANX7S3JBJIWQ7DRYCFJQWYL7KHDLL4Q, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.870447775 (566h56m8.223232904s) [noise=true , address=145.79.227.6:4291 , piece_id=C2WV6SUTWERDVJFLR5C7MARPLUFZDOD4QIEP5BYZZ5QGMMRWD36A, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.870728131 (566h56m8.22351327s) [noise=true , address=145.79.227.2:4142 , piece_id=3VMCECSE6AHD5CQ3XNMLB5OWGCQ2Y5TOU2KT3U3ZYBNU32UXAXEA, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.871095151 (566h56m8.223880279s) [noise=true , address=145.79.227.23:4554 , piece_id=I24ISPJY5NVYO5NZEECJTGKYRXIZBA3PSG2XCNAYCAJUOCV7AY6A, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.871460155 (566h56m8.224245284s) [noise=true , address=145.79.227.14:4849 , piece_id=EJM6IRSKW7ODLA7CN7RFVNJKAG7RMOBKRPGLMT4XQTDPYELVPYEQ, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.871858163 (566h56m8.2246433s) [noise=true , address=145.79.227.16:4923 , piece_id=VZFQUIA44MHDJ2DBGCXX3M3RDPEXTVQH7NDBOCQQMNP3JACRRXHQ, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.872170248 (566h56m8.224955386s) [noise=true , address=145.79.227.28:4757 , piece_id=7L64UR364ZCJYG4FXSJ4C7MKHR33VWAHGSSJZ26LGAGLPWY6FPRA, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.872560089 (566h56m8.225345217s) [noise=true , address=145.79.227.4:4194 , piece_id=54FCM3DO55SH7GFEWH7QXL2TW5FMPPHCJCBQYEVZPEE7ZDYGU4LA, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.872950993 (566h56m8.225736122s) [noise=true , address=145.79.227.18:4034 , piece_id=MY2SIGVQE4FFETRO37ULKLIXNGEL4IJSNQIYVKN3T6JCXAZ2PWFQ, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.873309075 (566h56m8.226094203s) [noise=true , address=145.79.227.24:4612 , piece_id=NE5QKM62C2VO4P77XXIX2GQ46WSMJE3JRSUMTCAIKDJCFCPDVXNQ, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.873498290 (566h56m8.226283418s) [noise=true , address=145.79.227.26:4656 , piece_id=5K3BFPPGLXVZZ3KF2R64TZD6GLFEOLE32ALH3JLSHQRHL6GJ4YTQ, seg_pos={0 0}, upload=498957]: Uploading piece... Sep 22 05:04:32.964399004 (566h56m8.317184132s) [upload=498957]: Next returned nil segment Sep 22 05:04:32.964414894 (566h56m8.317200022s) [upload=498957]: Waiting for error group managing segments... Sep 22 05:04:32.973347836 (566h56m8.326132965s) [noise=true , address=145.79.227.26:4656 , piece_id=5K3BFPPGLXVZZ3KF2R64TZD6GLFEOLE32ALH3JLSHQRHL6GJ4YTQ, seg_pos={0 0}, upload=498957]: Done uploading piece. err: <nil> Sep 22 05:04:32.974027452 (566h56m8.32681258s) [noise=true , address=145.79.227.30:4775 , piece_id=6Y5W2DO3TNAYW4UYCM7W5D56F57JLE5WSTQQT4WPMPAJE7ZQKIUA, seg_pos={0 0}, upload=498957]: Done uploading piece. err: <nil> Sep 22 05:04:32.974372609 (566h56m8.327157747s) [noise=true , address=145.79.227.28:4757 , piece_id=7L64UR364ZCJYG4FXSJ4C7MKHR33VWAHGSSJZ26LGAGLPWY6FPRA, seg_pos={0 0}, upload=498957]: Done uploading piece. err: <nil> Sep 22 05:04:32.974976603 (566h56m8.327761731s) [noise=true , address=145.79.227.24:4612 , piece_id=NE5QKM62C2VO4P77XXIX2GQ46WSMJE3JRSUMTCAIKDJCFCPDVXNQ, seg_pos={0 0}, upload=498957]: Done uploading piece. err: <nil> Sep 22 05:04:32.975003984 (566h56m8.327789123s) [noise=true , address=145.79.227.38:4980 , piece_id=TT7VT2MZW2TMOYSU325BB2MFQH7BIQQ75GT6M56JWSZBPGBVAJAA, seg_pos={0 0}, upload=498957]: Done uploading piece. err: <nil> Sep 22 05:04:32.975556542 (566h56m8.32834167s) [noise=true , address=77.90.31.15:4705 , piece_id=YNII3RGVISQN5FBTHCGJRANX7S3JBJIWQ7DRYCFJQWYL7KHDLL4Q, seg_pos={0 0}, upload=498957]: Done uploading piece. err: <nil> Sep 22 05:04:32.975661399 (566h56m8.328446526s) [noise=true , address=145.79.227.41:4088 , piece_id=YQ33LISFICOHVI37GJOWRYEGKE3YIXOBNEARYHUFWJFVT377OV4A, seg_pos={0 0}, upload=498957]: Done uploading piece. err: <nil> Sep 22 05:04:32.976217391 (566h56m8.329002519s) [noise=true , address=145.79.227.39:4013 , piece_id=OISIFQIDVNAG6G77B2EOUYM7WC2NENG3DGS64K5WEPUN5EQISC2Q, seg_pos={0 0}, upload=498957]: Done uploading piece. err: <nil> Sep 22 05:04:32.976274820 (566h56m8.329059948s) [noise=true , address=145.79.227.23:4554 , piece_id=I24ISPJY5NVYO5NZEECJTGKYRXIZBA3PSG2XCNAYCAJUOCV7AY6A, seg_pos={0 0}, upload=498957]: Done uploading piece. err: <nil> Sep 22 05:04:34.588452708 (566h56m9.941237805s) [noise=true , address=145.79.227.4:4194 , piece_id=54FCM3DO55SH7GFEWH7QXL2TW5FMPPHCJCBQYEVZPEE7ZDYGU4LA, seg_pos={0 0}, upload=498957]: Done uploading piece. err: ecclient: upload failed (node:1u6tGwzk56SwgUqzpyRFLNTTysUtk6C2Ls77UkjFryeuF9D8pV, address:145.79.227.4:4194): protocol: rpc: dial tcp 145.79.227.4:4194: connect: no route to host; manager closed: conn closed: op Sep 22 05:05:32.833224105 (566h57m8.186009232s) [noise=true , address=145.79.227.12:4794 , piece_id=TX3XR7UNCENTL3C4A5O2ACH2ST6BSQEAD2R7Y4W5EXIHZQF6IBXA, seg_pos={0 0}, upload=498957]: Done uploading piece. err: ecclient: upload failed (node:12ctLLD5skk4t5i3BFYgoTmuZ2aqsX23wDGDMnKriQ7U2KXyPAo, address:145.79.227.12:4794): protocol: rpc: dial tcp 145.79.227.12:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:05:32.833501003 (566h57m8.186286132s) [noise=true , address=145.79.227.2:4142 , piece_id=3VMCECSE6AHD5CQ3XNMLB5OWGCQ2Y5TOU2KT3U3ZYBNU32UXAXEA, seg_pos={0 0}, upload=498957]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:145.79.227.2:4142): protocol: rpc: dial tcp 145.79.227.2:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:05:32.833572338 (566h57m8.186357467s) [noise=true , address=145.79.227.13:4809 , piece_id=NYNY3JSUNABLAIUIMVGJ5BFBIQAX23XQUXWV6I7G2KLSLDQSF3ZQ, seg_pos={0 0}, upload=498957]: Done uploading piece. err: ecclient: upload failed (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9, address:145.79.227.13:4809): protocol: rpc: dial tcp 145.79.227.13:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:05:32.833623324 (566h57m8.186408452s) [noise=true , address=145.79.227.18:4034 , piece_id=MY2SIGVQE4FFETRO37ULKLIXNGEL4IJSNQIYVKN3T6JCXAZ2PWFQ, seg_pos={0 0}, upload=498957]: Done uploading piece. err: ecclient: upload failed (node:12jHHvRhcKfV1ufn7PRfnPHJ3ZxWJ8JUutf772G7YcKQjbpE3ZY, address:145.79.227.18:4034): protocol: rpc: dial tcp 145.79.227.18:4034: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:05:32.833716739 (566h57m8.186501867s) [noise=true , address=145.79.227.17:4977 , piece_id=QW6HIOFGC47BVNYUMZTNWUN7GCQSWZKRCMKG6BNM3GLVLQ66UCWA, seg_pos={0 0}, upload=498957]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:145.79.227.17:4977): protocol: rpc: dial tcp 145.79.227.17:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:05:32.833859096 (566h57m8.186644224s) [noise=true , address=145.79.227.6:4291 , piece_id=C2WV6SUTWERDVJFLR5C7MARPLUFZDOD4QIEP5BYZZ5QGMMRWD36A, seg_pos={0 0}, upload=498957]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:145.79.227.6:4291): protocol: rpc: dial tcp 145.79.227.6:4291: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:05:32.833980744 (566h57m8.186765873s) [noise=true , address=145.79.227.14:4849 , piece_id=EJM6IRSKW7ODLA7CN7RFVNJKAG7RMOBKRPGLMT4XQTDPYELVPYEQ, seg_pos={0 0}, upload=498957]: Done uploading piece. err: ecclient: upload failed (node:1n2gSzVmNCEZcfVsFKibjXHs2jMr82D3SWQEykFQQ4uyhhnrj1, address:145.79.227.14:4849): protocol: rpc: dial tcp 145.79.227.14:4849: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:05:32.833988539 (566h57m8.186773667s) [noise=true , address=145.79.227.16:4923 , piece_id=VZFQUIA44MHDJ2DBGCXX3M3RDPEXTVQH7NDBOCQQMNP3JACRRXHQ, seg_pos={0 0}, upload=498957]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:05:32.834038853 (566h57m8.186823982s) [seg_pos={0 0}, upload=498957]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Sep 22 05:05:32.847776423 (566h57m8.200561541s) [upload=498957]: Done uploading segments... Sep 22 05:05:32.847795129 (566h57m8.200580256s) [upload=498957]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.018158368s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.................XXX.XXXXX.........X........X...............] duration: 405.188031ms FAIL (2025-09-22 05:05:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 405.208179ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.175089543s FAIL (2025-09-22 05:04:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.17511431s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................XXX.XXXXX.........X........X..............] duration: 527.60634ms FAIL (2025-09-22 05:04:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 527.629985ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X........................X...X.......X.] duration: 410.872329ms FAIL (2025-09-22 05:04:05 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Sep 22 05:03:34.837319866 (566h55m10.190104974s) [upload=498944]: Starting upload Sep 22 05:03:34.837366693 (566h55m10.190151792s) [upload=498944]: Done starting upload Sep 22 05:03:34.837445321 (566h55m10.190230419s) [upload=498944]: Uploading segments... Sep 22 05:03:34.837576247 (566h55m10.190361356s) [upload=498944]: Got next segment. Inline: false Sep 22 05:03:34.837608789 (566h55m10.190393897s) [upload=498944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:03:34.837612175 (566h55m10.190397283s) [upload=498944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:03:34.858573992 (566h55m10.21135912s) [seg_pos={0 0}, upload=498944]: Begin upload segment... Sep 22 05:03:34.858623325 (566h55m10.211408453s) [seg_pos={0 0}, upload=498944]: Done begin upload segment. Sep 22 05:03:34.858654734 (566h55m10.211439862s) [noise=true , address=45.91.116.14:4191 , piece_id=6QZT2Y3YBQYDSONWAHYMLPPWFQ4GES2SSTMNPEDE7LAK3WXBJYQA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.858695580 (566h55m10.211480708s) [noise=true , address=45.91.116.12:4225 , piece_id=AUAEFQC4HF7S7T2UPXOX3LS2B2WYIW7FYOYWDIODR5Y3UUI2CR7A, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.859160693 (566h55m10.21194582s) [noise=true , address=45.91.116.5:4297 , piece_id=O7XG6O6CITLENXSOSU7AX2REZELQOB3XOFU6NQY5XM3J33ZSJ36Q, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.859318559 (566h55m10.212103697s) [noise=true , address=145.79.227.6:4258 , piece_id=U73TSZ2PYVB4XH3QKWXBZNXFHVCI3P267V5SQH4WLADKU7DHOIFQ, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.859544543 (566h55m10.212329682s) [noise=true , address=145.79.227.12:4765 , piece_id=Z6PTNQ4QFMQ7KCQNHDSKYMFATXFYQUS7UDFXJXW4ILIP7NGXYVVQ, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.859972176 (566h55m10.212757303s) [noise=true , address=145.79.227.16:4912 , piece_id=TP7X3LBIVM5TOGXDF6CM3GEPEE3WF77AZBGSUMVN64HNPIG6KWEA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.860858389 (566h55m10.213643517s) [noise=true , address=77.90.31.16:4733 , piece_id=XR6QLMXYB7AEWGGBRMEHF6HHR66O7ZWECACXJVF6KOZIP4VWC5XA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.861144285 (566h55m10.213929413s) [noise=true , address=145.79.227.15:4875 , piece_id=6QQMXAGJF3PDFZS2WI7BL3EG5QJILFDABK6NZPI2EHN2YA7A4V7Q, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.861693677 (566h55m10.214478804s) [noise=true , address=145.79.227.18:4007 , piece_id=VBTKKUBATNEHK5UKI5VJLNIPNLVXOMJ36XKRNNT4PRNSPF66KKBA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.862473079 (566h55m10.215258207s) [noise=true , address=77.90.31.11:4553 , piece_id=GGQRFO7VMKAHXXZ6XLGS5CXG3PRMRLK5KJHH67AP5JW3MM56S26Q, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.862863982 (566h55m10.21564911s) [noise=true , address=77.90.31.10:4518 , piece_id=P75F2CQBRDPISRK4BOP27PV6IKOQTYXGVKZ4IBHVEGLZ373VD4VA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.863451656 (566h55m10.216236783s) [noise=true , address=145.79.227.7:4294 , piece_id=E34AXC73UWE5WWDOKAGJ3HAOH27SB7N2WAR7EQRVFFTKXMFQ6XKA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.864065677 (566h55m10.216850805s) [noise=true , address=45.91.116.11:4119 , piece_id=PAUULQH2HNLP2ZFJMVIQT4KDJQUXPM225WQ5VZC5BPMWUYCI63PA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.864759539 (566h55m10.217544677s) [noise=true , address=145.79.227.3:4149 , piece_id=GHYM6OR2QRK5W5CE6AZLJVODUQUXSBSRHUVHCCQRZVDUFM5Z3WSA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.864874314 (566h55m10.217659443s) [noise=true , address=145.79.227.11:4731 , piece_id=KRVM7E63DMMGMVWJL7QUI667Y2HI5L76Y2BCJVZSF44VODHGGV2A, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.865482116 (566h55m10.218267243s) [noise=true , address=45.91.116.9:4263 , piece_id=2MIU6LXV6NADHII5QBXZPHCO23E5K5K7DDEJKVWFDCQR5BALOURQ, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.962766068 (566h55m10.315551195s) [upload=498944]: Next returned nil segment Sep 22 05:03:34.962796785 (566h55m10.315581923s) [upload=498944]: Waiting for error group managing segments... Sep 22 05:03:34.970671914 (566h55m10.323457032s) [noise=true , address=45.91.116.12:4225 , piece_id=AUAEFQC4HF7S7T2UPXOX3LS2B2WYIW7FYOYWDIODR5Y3UUI2CR7A, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.971133479 (566h55m10.323918608s) [noise=true , address=45.91.116.5:4297 , piece_id=O7XG6O6CITLENXSOSU7AX2REZELQOB3XOFU6NQY5XM3J33ZSJ36Q, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.971212848 (566h55m10.323997966s) [noise=true , address=45.91.116.9:4263 , piece_id=2MIU6LXV6NADHII5QBXZPHCO23E5K5K7DDEJKVWFDCQR5BALOURQ, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.972164915 (566h55m10.324950033s) [noise=true , address=45.91.116.14:4191 , piece_id=6QZT2Y3YBQYDSONWAHYMLPPWFQ4GES2SSTMNPEDE7LAK3WXBJYQA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.972671416 (566h55m10.325456544s) [noise=true , address=77.90.31.16:4733 , piece_id=XR6QLMXYB7AEWGGBRMEHF6HHR66O7ZWECACXJVF6KOZIP4VWC5XA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.972710429 (566h55m10.325495557s) [noise=true , address=45.91.116.11:4119 , piece_id=PAUULQH2HNLP2ZFJMVIQT4KDJQUXPM225WQ5VZC5BPMWUYCI63PA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.973560494 (566h55m10.326345622s) [noise=true , address=77.90.31.10:4518 , piece_id=P75F2CQBRDPISRK4BOP27PV6IKOQTYXGVKZ4IBHVEGLZ373VD4VA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.973925850 (566h55m10.326710978s) [noise=true , address=77.90.31.11:4553 , piece_id=GGQRFO7VMKAHXXZ6XLGS5CXG3PRMRLK5KJHH67AP5JW3MM56S26Q, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:35.903641765 (566h55m11.256426894s) [noise=true , address=145.79.227.16:4912 , piece_id=TP7X3LBIVM5TOGXDF6CM3GEPEE3WF77AZBGSUMVN64HNPIG6KWEA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op Sep 22 05:03:36.892292359 (566h55m12.245077456s) [noise=true , address=145.79.227.18:4007 , piece_id=VBTKKUBATNEHK5UKI5VJLNIPNLVXOMJ36XKRNNT4PRNSPF66KKBA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: connect: no route to host; manager closed: conn closed: op Sep 22 05:04:04.826700554 (566h55m40.179485662s) [noise=true , address=145.79.227.7:4294 , piece_id=E34AXC73UWE5WWDOKAGJ3HAOH27SB7N2WAR7EQRVFFTKXMFQ6XKA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:04:04.826772068 (566h55m40.179557166s) [noise=true , address=145.79.227.6:4258 , piece_id=U73TSZ2PYVB4XH3QKWXBZNXFHVCI3P267V5SQH4WLADKU7DHOIFQ, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:04:04.826812443 (566h55m40.179597552s) [noise=true , address=145.79.227.15:4875 , piece_id=6QQMXAGJF3PDFZS2WI7BL3EG5QJILFDABK6NZPI2EHN2YA7A4V7Q, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:04:04.826869972 (566h55m40.17965507s) [noise=true , address=145.79.227.11:4731 , piece_id=KRVM7E63DMMGMVWJL7QUI667Y2HI5L76Y2BCJVZSF44VODHGGV2A, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:04:04.826922791 (566h55m40.179707889s) [noise=true , address=145.79.227.12:4765 , piece_id=Z6PTNQ4QFMQ7KCQNHDSKYMFATXFYQUS7UDFXJXW4ILIP7NGXYVVQ, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:04:04.826935955 (566h55m40.179721063s) [noise=true , address=145.79.227.3:4149 , piece_id=GHYM6OR2QRK5W5CE6AZLJVODUQUXSBSRHUVHCCQRZVDUFM5Z3WSA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:04:04.826957676 (566h55m40.179742774s) [seg_pos={0 0}, upload=498944]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:04:04.838601738 (566h55m40.191386846s) [upload=498944]: Done uploading segments... Sep 22 05:04:04.838622938 (566h55m40.191408035s) [upload=498944]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 410.89443ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X........................X...X.......X] duration: 30.01378748s FAIL (2025-09-22 05:04:04 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Sep 22 05:03:34.837319866 (566h55m10.190104974s) [upload=498944]: Starting upload Sep 22 05:03:34.837366693 (566h55m10.190151792s) [upload=498944]: Done starting upload Sep 22 05:03:34.837445321 (566h55m10.190230419s) [upload=498944]: Uploading segments... Sep 22 05:03:34.837576247 (566h55m10.190361356s) [upload=498944]: Got next segment. Inline: false Sep 22 05:03:34.837608789 (566h55m10.190393897s) [upload=498944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:03:34.837612175 (566h55m10.190397283s) [upload=498944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:03:34.858573992 (566h55m10.21135912s) [seg_pos={0 0}, upload=498944]: Begin upload segment... Sep 22 05:03:34.858623325 (566h55m10.211408453s) [seg_pos={0 0}, upload=498944]: Done begin upload segment. Sep 22 05:03:34.858654734 (566h55m10.211439862s) [noise=true , address=45.91.116.14:4191 , piece_id=6QZT2Y3YBQYDSONWAHYMLPPWFQ4GES2SSTMNPEDE7LAK3WXBJYQA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.858695580 (566h55m10.211480708s) [noise=true , address=45.91.116.12:4225 , piece_id=AUAEFQC4HF7S7T2UPXOX3LS2B2WYIW7FYOYWDIODR5Y3UUI2CR7A, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.859160693 (566h55m10.21194582s) [noise=true , address=45.91.116.5:4297 , piece_id=O7XG6O6CITLENXSOSU7AX2REZELQOB3XOFU6NQY5XM3J33ZSJ36Q, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.859318559 (566h55m10.212103697s) [noise=true , address=145.79.227.6:4258 , piece_id=U73TSZ2PYVB4XH3QKWXBZNXFHVCI3P267V5SQH4WLADKU7DHOIFQ, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.859544543 (566h55m10.212329682s) [noise=true , address=145.79.227.12:4765 , piece_id=Z6PTNQ4QFMQ7KCQNHDSKYMFATXFYQUS7UDFXJXW4ILIP7NGXYVVQ, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.859972176 (566h55m10.212757303s) [noise=true , address=145.79.227.16:4912 , piece_id=TP7X3LBIVM5TOGXDF6CM3GEPEE3WF77AZBGSUMVN64HNPIG6KWEA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.860858389 (566h55m10.213643517s) [noise=true , address=77.90.31.16:4733 , piece_id=XR6QLMXYB7AEWGGBRMEHF6HHR66O7ZWECACXJVF6KOZIP4VWC5XA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.861144285 (566h55m10.213929413s) [noise=true , address=145.79.227.15:4875 , piece_id=6QQMXAGJF3PDFZS2WI7BL3EG5QJILFDABK6NZPI2EHN2YA7A4V7Q, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.861693677 (566h55m10.214478804s) [noise=true , address=145.79.227.18:4007 , piece_id=VBTKKUBATNEHK5UKI5VJLNIPNLVXOMJ36XKRNNT4PRNSPF66KKBA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.862473079 (566h55m10.215258207s) [noise=true , address=77.90.31.11:4553 , piece_id=GGQRFO7VMKAHXXZ6XLGS5CXG3PRMRLK5KJHH67AP5JW3MM56S26Q, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.862863982 (566h55m10.21564911s) [noise=true , address=77.90.31.10:4518 , piece_id=P75F2CQBRDPISRK4BOP27PV6IKOQTYXGVKZ4IBHVEGLZ373VD4VA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.863451656 (566h55m10.216236783s) [noise=true , address=145.79.227.7:4294 , piece_id=E34AXC73UWE5WWDOKAGJ3HAOH27SB7N2WAR7EQRVFFTKXMFQ6XKA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.864065677 (566h55m10.216850805s) [noise=true , address=45.91.116.11:4119 , piece_id=PAUULQH2HNLP2ZFJMVIQT4KDJQUXPM225WQ5VZC5BPMWUYCI63PA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.864759539 (566h55m10.217544677s) [noise=true , address=145.79.227.3:4149 , piece_id=GHYM6OR2QRK5W5CE6AZLJVODUQUXSBSRHUVHCCQRZVDUFM5Z3WSA, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.864874314 (566h55m10.217659443s) [noise=true , address=145.79.227.11:4731 , piece_id=KRVM7E63DMMGMVWJL7QUI667Y2HI5L76Y2BCJVZSF44VODHGGV2A, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.865482116 (566h55m10.218267243s) [noise=true , address=45.91.116.9:4263 , piece_id=2MIU6LXV6NADHII5QBXZPHCO23E5K5K7DDEJKVWFDCQR5BALOURQ, seg_pos={0 0}, upload=498944]: Uploading piece... Sep 22 05:03:34.962766068 (566h55m10.315551195s) [upload=498944]: Next returned nil segment Sep 22 05:03:34.962796785 (566h55m10.315581923s) [upload=498944]: Waiting for error group managing segments... Sep 22 05:03:34.970671914 (566h55m10.323457032s) [noise=true , address=45.91.116.12:4225 , piece_id=AUAEFQC4HF7S7T2UPXOX3LS2B2WYIW7FYOYWDIODR5Y3UUI2CR7A, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.971133479 (566h55m10.323918608s) [noise=true , address=45.91.116.5:4297 , piece_id=O7XG6O6CITLENXSOSU7AX2REZELQOB3XOFU6NQY5XM3J33ZSJ36Q, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.971212848 (566h55m10.323997966s) [noise=true , address=45.91.116.9:4263 , piece_id=2MIU6LXV6NADHII5QBXZPHCO23E5K5K7DDEJKVWFDCQR5BALOURQ, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.972164915 (566h55m10.324950033s) [noise=true , address=45.91.116.14:4191 , piece_id=6QZT2Y3YBQYDSONWAHYMLPPWFQ4GES2SSTMNPEDE7LAK3WXBJYQA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.972671416 (566h55m10.325456544s) [noise=true , address=77.90.31.16:4733 , piece_id=XR6QLMXYB7AEWGGBRMEHF6HHR66O7ZWECACXJVF6KOZIP4VWC5XA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.972710429 (566h55m10.325495557s) [noise=true , address=45.91.116.11:4119 , piece_id=PAUULQH2HNLP2ZFJMVIQT4KDJQUXPM225WQ5VZC5BPMWUYCI63PA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.973560494 (566h55m10.326345622s) [noise=true , address=77.90.31.10:4518 , piece_id=P75F2CQBRDPISRK4BOP27PV6IKOQTYXGVKZ4IBHVEGLZ373VD4VA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:34.973925850 (566h55m10.326710978s) [noise=true , address=77.90.31.11:4553 , piece_id=GGQRFO7VMKAHXXZ6XLGS5CXG3PRMRLK5KJHH67AP5JW3MM56S26Q, seg_pos={0 0}, upload=498944]: Done uploading piece. err: <nil> Sep 22 05:03:35.903641765 (566h55m11.256426894s) [noise=true , address=145.79.227.16:4912 , piece_id=TP7X3LBIVM5TOGXDF6CM3GEPEE3WF77AZBGSUMVN64HNPIG6KWEA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: connect: no route to host; conn closed: op Sep 22 05:03:36.892292359 (566h55m12.245077456s) [noise=true , address=145.79.227.18:4007 , piece_id=VBTKKUBATNEHK5UKI5VJLNIPNLVXOMJ36XKRNNT4PRNSPF66KKBA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: connect: no route to host; manager closed: conn closed: op Sep 22 05:04:04.826700554 (566h55m40.179485662s) [noise=true , address=145.79.227.7:4294 , piece_id=E34AXC73UWE5WWDOKAGJ3HAOH27SB7N2WAR7EQRVFFTKXMFQ6XKA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:04:04.826772068 (566h55m40.179557166s) [noise=true , address=145.79.227.6:4258 , piece_id=U73TSZ2PYVB4XH3QKWXBZNXFHVCI3P267V5SQH4WLADKU7DHOIFQ, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:04:04.826812443 (566h55m40.179597552s) [noise=true , address=145.79.227.15:4875 , piece_id=6QQMXAGJF3PDFZS2WI7BL3EG5QJILFDABK6NZPI2EHN2YA7A4V7Q, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:04:04.826869972 (566h55m40.17965507s) [noise=true , address=145.79.227.11:4731 , piece_id=KRVM7E63DMMGMVWJL7QUI667Y2HI5L76Y2BCJVZSF44VODHGGV2A, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: rpc: dial tcp 145.79.227.11:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:04:04.826922791 (566h55m40.179707889s) [noise=true , address=145.79.227.12:4765 , piece_id=Z6PTNQ4QFMQ7KCQNHDSKYMFATXFYQUS7UDFXJXW4ILIP7NGXYVVQ, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:04:04.826935955 (566h55m40.179721063s) [noise=true , address=145.79.227.3:4149 , piece_id=GHYM6OR2QRK5W5CE6AZLJVODUQUXSBSRHUVHCCQRZVDUFM5Z3WSA, seg_pos={0 0}, upload=498944]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:04:04.826957676 (566h55m40.179742774s) [seg_pos={0 0}, upload=498944]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:04:04.838601738 (566h55m40.191386846s) [upload=498944]: Done uploading segments... Sep 22 05:04:04.838622938 (566h55m40.191408035s) [upload=498944]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.013803579s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 4.937152315s FAIL (2025-09-22 05:04:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.937173254s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..................XXX.XXXXX.........X........X.............] duration: 521.878324ms FAIL (2025-09-22 05:03:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 521.901456ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 749.930222ms FAIL (2025-09-22 05:03:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [seg_pos={0 0}, upload=498934]: Begin upload segment... Sep 22 05:02:52.553768154 (566h54m27.906553262s) [seg_pos={0 0}, upload=498934]: Done begin upload segment. Sep 22 05:02:52.553787660 (566h54m27.906572768s) [noise=true , address=145.79.227.5:4231 , piece_id=KEAYN3TNPR56Q7ZX7Q2H6HLP52UHMGV3LRPTFYA73LIEBE2TMT4A, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.553840329 (566h54m27.906625437s) [noise=true , address=145.79.227.18:4039 , piece_id=4WHPNRFSOK3ZXJCSYEPPJL46FNW2IR5762ACWRSZRECTFJPYVTNQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.554271207 (566h54m27.907056315s) [noise=true , address=145.79.227.24:4605 , piece_id=R7KER72NENMHE6RN4N6CXC5VLTI2UK6J5CKU7GIHPXRFP4OVCT2Q, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.554280946 (566h54m27.907066054s) [noise=true , address=145.79.227.4:4205 , piece_id=4F7VWNKARIEOBY2ROYSTVAMFKLZQWKG2FJS5JBSHVWHGVGQPLR6Q, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.554639238 (566h54m27.907424336s) [noise=true , address=145.79.227.38:4978 , piece_id=QJWYZHCACV6GB56NCXN4NFW7QI4YE3O7J7JDD5CPN7BQM77ADLXA, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.554650179 (566h54m27.907435287s) [noise=true , address=145.79.227.2:4127 , piece_id=ZH6PIAR7BPZZXSXDCSSBVOPUGYGOFVWHJLIU6RAST3XI66EDSATQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.555015685 (566h54m27.907800793s) [noise=true , address=145.79.227.49:4340 , piece_id=MKWQRHWNKMXQWCVUB2YTMNSJ3KJFDW6PCXX5ZJQODULRMYKPSZ2A, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.555025412 (566h54m27.90781052s) [noise=true , address=77.90.31.16:4759 , piece_id=PRRYSC7UN5CUVNNSUOQ2SENYM27IGGVQMJ4GRA4PX2K7WN4SZUKA, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.555370481 (566h54m27.908155589s) [noise=true , address=145.79.227.33:4871 , piece_id=UGQQEJJNB2N73DQLXEFAVD7ZQEJGNIHKGAHMP2KXZPXJ3PFE54OA, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.555706000 (566h54m27.908491108s) [noise=true , address=77.90.31.21:4900 , piece_id=BDWLUW2Y4OUBZHQT452TCCA5X6G6JCBYSAJH2ESBRPIHLGIRR7IQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.556059172 (566h54m27.908844271s) [noise=true , address=145.79.227.28:4736 , piece_id=O6UD4IDEDNMS6F6FPRXE7COX442MSYH6VC3WPG66UJNXPAPKCVNQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.556396456 (566h54m27.909181564s) [noise=true , address=145.79.227.6:4292 , piece_id=GD74NH4GQDULU5O5W6GYAASSYSCKHIDGY27GFYUSVZR4LE3L3SDQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.556960043 (566h54m27.909745151s) [noise=true , address=64.49.13.27:4960 , piece_id=OGEBPE6WHKYEDOKWVWOCYZEVTP45VHTCVF6JXFVQFNMATRECGOWQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.557304239 (566h54m27.910089347s) [noise=true , address=145.79.227.11:4744 , piece_id=TKTM7CMZYHCXALNJ3HA2IUV5MHRR4GSZNQXXSHJMUUVDWGFYRDEQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.558093730 (566h54m27.910878838s) [noise=true , address=77.90.31.11:4578 , piece_id=72YKT46IUQHXHVEUKQCCUL7L6W2GZBD4DDGZ4IA6OIU33AF7Z3LA, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.558459607 (566h54m27.911244715s) [noise=true , address=145.79.227.13:4825 , piece_id=5R2TH7DS4ZK62R77PEYY2H3WXLRCOABO2FOVCLSCHLRSOY43MSTQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.559383371 (566h54m27.912168479s) [noise=true , address=145.79.227.46:4603 , piece_id=NHBLJ2PDPWAWQ25DAUUBTQ2EGDFNRJ6DTJG3RPCJUXLRPSUTMO6A, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.559734519 (566h54m27.912519618s) [noise=true , address=145.79.227.47:4293 , piece_id=5OUIUGVXV7AYZ2WRVUUDGIK3SJ2SDEW7JSP4D22LRULJFEJFJBEQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.620194758 (566h54m27.972979866s) [upload=498934]: Next returned nil segment Sep 22 05:02:52.620208534 (566h54m27.972993632s) [upload=498934]: Waiting for error group managing segments... Sep 22 05:02:52.628328663 (566h54m27.981113771s) [noise=true , address=145.79.227.46:4603 , piece_id=NHBLJ2PDPWAWQ25DAUUBTQ2EGDFNRJ6DTJG3RPCJUXLRPSUTMO6A, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.628347418 (566h54m27.981132515s) [noise=true , address=145.79.227.33:4871 , piece_id=UGQQEJJNB2N73DQLXEFAVD7ZQEJGNIHKGAHMP2KXZPXJ3PFE54OA, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.629186633 (566h54m27.981971741s) [noise=true , address=145.79.227.28:4736 , piece_id=O6UD4IDEDNMS6F6FPRXE7COX442MSYH6VC3WPG66UJNXPAPKCVNQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.629332917 (566h54m27.982118015s) [noise=true , address=77.90.31.16:4759 , piece_id=PRRYSC7UN5CUVNNSUOQ2SENYM27IGGVQMJ4GRA4PX2K7WN4SZUKA, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.629579369 (566h54m27.982364467s) [noise=true , address=145.79.227.47:4293 , piece_id=5OUIUGVXV7AYZ2WRVUUDGIK3SJ2SDEW7JSP4D22LRULJFEJFJBEQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.629748586 (566h54m27.982533685s) [noise=true , address=145.79.227.38:4978 , piece_id=QJWYZHCACV6GB56NCXN4NFW7QI4YE3O7J7JDD5CPN7BQM77ADLXA, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.630433231 (566h54m27.983218339s) [noise=true , address=145.79.227.49:4340 , piece_id=MKWQRHWNKMXQWCVUB2YTMNSJ3KJFDW6PCXX5ZJQODULRMYKPSZ2A, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.630446847 (566h54m27.983231955s) [noise=true , address=145.79.227.24:4605 , piece_id=R7KER72NENMHE6RN4N6CXC5VLTI2UK6J5CKU7GIHPXRFP4OVCT2Q, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.630748593 (566h54m27.983533701s) [noise=true , address=77.90.31.21:4900 , piece_id=BDWLUW2Y4OUBZHQT452TCCA5X6G6JCBYSAJH2ESBRPIHLGIRR7IQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.631103630 (566h54m27.983888738s) [noise=true , address=77.90.31.11:4578 , piece_id=72YKT46IUQHXHVEUKQCCUL7L6W2GZBD4DDGZ4IA6OIU33AF7Z3LA, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:53.160192173 (566h54m28.512977281s) [noise=true , address=64.49.13.27:4960 , piece_id=OGEBPE6WHKYEDOKWVWOCYZEVTP45VHTCVF6JXFVQFNMATRECGOWQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:56.959055014 (566h54m32.311840153s) [noise=true , address=145.79.227.18:4039 , piece_id=4WHPNRFSOK3ZXJCSYEPPJL46FNW2IR5762ACWRSZRECTFJPYVTNQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: rpc: dial tcp 145.79.227.18:4039: connect: no route to host; conn closed: op Sep 22 05:03:52.518590460 (566h55m27.871375589s) [noise=true , address=145.79.227.2:4127 , piece_id=ZH6PIAR7BPZZXSXDCSSBVOPUGYGOFVWHJLIU6RAST3XI66EDSATQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:145.79.227.2:4127): protocol: rpc: dial tcp 145.79.227.2:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:03:52.518655643 (566h55m27.871440771s) [noise=true , address=145.79.227.13:4825 , piece_id=5R2TH7DS4ZK62R77PEYY2H3WXLRCOABO2FOVCLSCHLRSOY43MSTQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:03:52.518703162 (566h55m27.87148829s) [noise=true , address=145.79.227.4:4205 , piece_id=4F7VWNKARIEOBY2ROYSTVAMFKLZQWKG2FJS5JBSHVWHGVGQPLR6Q, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr, address:145.79.227.4:4205): protocol: rpc: dial tcp 145.79.227.4:4205: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:03:52.518747355 (566h55m27.871532473s) [noise=true , address=145.79.227.6:4292 , piece_id=GD74NH4GQDULU5O5W6GYAASSYSCKHIDGY27GFYUSVZR4LE3L3SDQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:03:52.518795986 (566h55m27.871581114s) [noise=true , address=145.79.227.11:4744 , piece_id=TKTM7CMZYHCXALNJ3HA2IUV5MHRR4GSZNQXXSHJMUUVDWGFYRDEQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH, address:145.79.227.11:4744): protocol: rpc: dial tcp 145.79.227.11:4744: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:03:52.518837965 (566h55m27.871623082s) [noise=true , address=145.79.227.5:4231 , piece_id=KEAYN3TNPR56Q7ZX7Q2H6HLP52UHMGV3LRPTFYA73LIEBE2TMT4A, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:1ixY5pokE6MtrVH9FpKc1tGZhEAaW5brbzBeeGmYUczK3z7tCQ, address:145.79.227.5:4231): protocol: rpc: dial tcp 145.79.227.5:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:03:52.518854966 (566h55m27.871640095s) [seg_pos={0 0}, upload=498934]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:03:52.532557471 (566h55m27.885342599s) [upload=498934]: Done uploading segments... Sep 22 05:03:52.532574512 (566h55m27.885359641s) [upload=498934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 749.954789ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.016466733s FAIL (2025-09-22 05:03:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [seg_pos={0 0}, upload=498934]: Begin upload segment... Sep 22 05:02:52.553768154 (566h54m27.906553262s) [seg_pos={0 0}, upload=498934]: Done begin upload segment. Sep 22 05:02:52.553787660 (566h54m27.906572768s) [noise=true , address=145.79.227.5:4231 , piece_id=KEAYN3TNPR56Q7ZX7Q2H6HLP52UHMGV3LRPTFYA73LIEBE2TMT4A, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.553840329 (566h54m27.906625437s) [noise=true , address=145.79.227.18:4039 , piece_id=4WHPNRFSOK3ZXJCSYEPPJL46FNW2IR5762ACWRSZRECTFJPYVTNQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.554271207 (566h54m27.907056315s) [noise=true , address=145.79.227.24:4605 , piece_id=R7KER72NENMHE6RN4N6CXC5VLTI2UK6J5CKU7GIHPXRFP4OVCT2Q, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.554280946 (566h54m27.907066054s) [noise=true , address=145.79.227.4:4205 , piece_id=4F7VWNKARIEOBY2ROYSTVAMFKLZQWKG2FJS5JBSHVWHGVGQPLR6Q, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.554639238 (566h54m27.907424336s) [noise=true , address=145.79.227.38:4978 , piece_id=QJWYZHCACV6GB56NCXN4NFW7QI4YE3O7J7JDD5CPN7BQM77ADLXA, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.554650179 (566h54m27.907435287s) [noise=true , address=145.79.227.2:4127 , piece_id=ZH6PIAR7BPZZXSXDCSSBVOPUGYGOFVWHJLIU6RAST3XI66EDSATQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.555015685 (566h54m27.907800793s) [noise=true , address=145.79.227.49:4340 , piece_id=MKWQRHWNKMXQWCVUB2YTMNSJ3KJFDW6PCXX5ZJQODULRMYKPSZ2A, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.555025412 (566h54m27.90781052s) [noise=true , address=77.90.31.16:4759 , piece_id=PRRYSC7UN5CUVNNSUOQ2SENYM27IGGVQMJ4GRA4PX2K7WN4SZUKA, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.555370481 (566h54m27.908155589s) [noise=true , address=145.79.227.33:4871 , piece_id=UGQQEJJNB2N73DQLXEFAVD7ZQEJGNIHKGAHMP2KXZPXJ3PFE54OA, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.555706000 (566h54m27.908491108s) [noise=true , address=77.90.31.21:4900 , piece_id=BDWLUW2Y4OUBZHQT452TCCA5X6G6JCBYSAJH2ESBRPIHLGIRR7IQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.556059172 (566h54m27.908844271s) [noise=true , address=145.79.227.28:4736 , piece_id=O6UD4IDEDNMS6F6FPRXE7COX442MSYH6VC3WPG66UJNXPAPKCVNQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.556396456 (566h54m27.909181564s) [noise=true , address=145.79.227.6:4292 , piece_id=GD74NH4GQDULU5O5W6GYAASSYSCKHIDGY27GFYUSVZR4LE3L3SDQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.556960043 (566h54m27.909745151s) [noise=true , address=64.49.13.27:4960 , piece_id=OGEBPE6WHKYEDOKWVWOCYZEVTP45VHTCVF6JXFVQFNMATRECGOWQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.557304239 (566h54m27.910089347s) [noise=true , address=145.79.227.11:4744 , piece_id=TKTM7CMZYHCXALNJ3HA2IUV5MHRR4GSZNQXXSHJMUUVDWGFYRDEQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.558093730 (566h54m27.910878838s) [noise=true , address=77.90.31.11:4578 , piece_id=72YKT46IUQHXHVEUKQCCUL7L6W2GZBD4DDGZ4IA6OIU33AF7Z3LA, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.558459607 (566h54m27.911244715s) [noise=true , address=145.79.227.13:4825 , piece_id=5R2TH7DS4ZK62R77PEYY2H3WXLRCOABO2FOVCLSCHLRSOY43MSTQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.559383371 (566h54m27.912168479s) [noise=true , address=145.79.227.46:4603 , piece_id=NHBLJ2PDPWAWQ25DAUUBTQ2EGDFNRJ6DTJG3RPCJUXLRPSUTMO6A, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.559734519 (566h54m27.912519618s) [noise=true , address=145.79.227.47:4293 , piece_id=5OUIUGVXV7AYZ2WRVUUDGIK3SJ2SDEW7JSP4D22LRULJFEJFJBEQ, seg_pos={0 0}, upload=498934]: Uploading piece... Sep 22 05:02:52.620194758 (566h54m27.972979866s) [upload=498934]: Next returned nil segment Sep 22 05:02:52.620208534 (566h54m27.972993632s) [upload=498934]: Waiting for error group managing segments... Sep 22 05:02:52.628328663 (566h54m27.981113771s) [noise=true , address=145.79.227.46:4603 , piece_id=NHBLJ2PDPWAWQ25DAUUBTQ2EGDFNRJ6DTJG3RPCJUXLRPSUTMO6A, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.628347418 (566h54m27.981132515s) [noise=true , address=145.79.227.33:4871 , piece_id=UGQQEJJNB2N73DQLXEFAVD7ZQEJGNIHKGAHMP2KXZPXJ3PFE54OA, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.629186633 (566h54m27.981971741s) [noise=true , address=145.79.227.28:4736 , piece_id=O6UD4IDEDNMS6F6FPRXE7COX442MSYH6VC3WPG66UJNXPAPKCVNQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.629332917 (566h54m27.982118015s) [noise=true , address=77.90.31.16:4759 , piece_id=PRRYSC7UN5CUVNNSUOQ2SENYM27IGGVQMJ4GRA4PX2K7WN4SZUKA, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.629579369 (566h54m27.982364467s) [noise=true , address=145.79.227.47:4293 , piece_id=5OUIUGVXV7AYZ2WRVUUDGIK3SJ2SDEW7JSP4D22LRULJFEJFJBEQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.629748586 (566h54m27.982533685s) [noise=true , address=145.79.227.38:4978 , piece_id=QJWYZHCACV6GB56NCXN4NFW7QI4YE3O7J7JDD5CPN7BQM77ADLXA, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.630433231 (566h54m27.983218339s) [noise=true , address=145.79.227.49:4340 , piece_id=MKWQRHWNKMXQWCVUB2YTMNSJ3KJFDW6PCXX5ZJQODULRMYKPSZ2A, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.630446847 (566h54m27.983231955s) [noise=true , address=145.79.227.24:4605 , piece_id=R7KER72NENMHE6RN4N6CXC5VLTI2UK6J5CKU7GIHPXRFP4OVCT2Q, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.630748593 (566h54m27.983533701s) [noise=true , address=77.90.31.21:4900 , piece_id=BDWLUW2Y4OUBZHQT452TCCA5X6G6JCBYSAJH2ESBRPIHLGIRR7IQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:52.631103630 (566h54m27.983888738s) [noise=true , address=77.90.31.11:4578 , piece_id=72YKT46IUQHXHVEUKQCCUL7L6W2GZBD4DDGZ4IA6OIU33AF7Z3LA, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:53.160192173 (566h54m28.512977281s) [noise=true , address=64.49.13.27:4960 , piece_id=OGEBPE6WHKYEDOKWVWOCYZEVTP45VHTCVF6JXFVQFNMATRECGOWQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: <nil> Sep 22 05:02:56.959055014 (566h54m32.311840153s) [noise=true , address=145.79.227.18:4039 , piece_id=4WHPNRFSOK3ZXJCSYEPPJL46FNW2IR5762ACWRSZRECTFJPYVTNQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: rpc: dial tcp 145.79.227.18:4039: connect: no route to host; conn closed: op Sep 22 05:03:52.518590460 (566h55m27.871375589s) [noise=true , address=145.79.227.2:4127 , piece_id=ZH6PIAR7BPZZXSXDCSSBVOPUGYGOFVWHJLIU6RAST3XI66EDSATQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:145.79.227.2:4127): protocol: rpc: dial tcp 145.79.227.2:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:03:52.518655643 (566h55m27.871440771s) [noise=true , address=145.79.227.13:4825 , piece_id=5R2TH7DS4ZK62R77PEYY2H3WXLRCOABO2FOVCLSCHLRSOY43MSTQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:145.79.227.13:4825): protocol: rpc: dial tcp 145.79.227.13:4825: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:03:52.518703162 (566h55m27.87148829s) [noise=true , address=145.79.227.4:4205 , piece_id=4F7VWNKARIEOBY2ROYSTVAMFKLZQWKG2FJS5JBSHVWHGVGQPLR6Q, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr, address:145.79.227.4:4205): protocol: rpc: dial tcp 145.79.227.4:4205: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:03:52.518747355 (566h55m27.871532473s) [noise=true , address=145.79.227.6:4292 , piece_id=GD74NH4GQDULU5O5W6GYAASSYSCKHIDGY27GFYUSVZR4LE3L3SDQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:03:52.518795986 (566h55m27.871581114s) [noise=true , address=145.79.227.11:4744 , piece_id=TKTM7CMZYHCXALNJ3HA2IUV5MHRR4GSZNQXXSHJMUUVDWGFYRDEQ, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH, address:145.79.227.11:4744): protocol: rpc: dial tcp 145.79.227.11:4744: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:03:52.518837965 (566h55m27.871623082s) [noise=true , address=145.79.227.5:4231 , piece_id=KEAYN3TNPR56Q7ZX7Q2H6HLP52UHMGV3LRPTFYA73LIEBE2TMT4A, seg_pos={0 0}, upload=498934]: Done uploading piece. err: ecclient: upload failed (node:1ixY5pokE6MtrVH9FpKc1tGZhEAaW5brbzBeeGmYUczK3z7tCQ, address:145.79.227.5:4231): protocol: rpc: dial tcp 145.79.227.5:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:03:52.518854966 (566h55m27.871640095s) [seg_pos={0 0}, upload=498934]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:03:52.532557471 (566h55m27.885342599s) [upload=498934]: Done uploading segments... Sep 22 05:03:52.532574512 (566h55m27.885359641s) [upload=498934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.016495026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.107526887s FAIL (2025-09-22 05:03:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.107551633s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X..................XXX.XXXXX.........X........X............] duration: 771.771908ms FAIL (2025-09-22 05:03:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 772.008321ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.123395136s FAIL (2025-09-22 05:02:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.123425221s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X........................X...X.......] duration: 443.866825ms FAIL (2025-09-22 05:02:33 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:57:09.597529116 (566h48m44.950314224s) [upload=498847]: Starting upload Sep 22 04:57:09.597575052 (566h48m44.95036016s) [upload=498847]: Done starting upload Sep 22 04:57:09.597633201 (566h48m44.9504183s) [upload=498847]: Uploading segments... Sep 22 04:57:09.597786539 (566h48m44.950571637s) [upload=498847]: Got next segment. Inline: false Sep 22 04:57:09.597821665 (566h48m44.950606773s) [upload=498847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:57:09.597826965 (566h48m44.950612092s) [upload=498847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:57:09.619411660 (566h48m44.972196778s) [seg_pos={0 0}, upload=498847]: Begin upload segment... Sep 22 04:57:09.619441175 (566h48m44.972226293s) [seg_pos={0 0}, upload=498847]: Done begin upload segment. Sep 22 04:57:09.619461974 (566h48m44.972247092s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619509433 (566h48m44.972294552s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619801252 (566h48m44.97258635s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619932799 (566h48m44.972717907s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620051050 (566h48m44.972836158s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620209487 (566h48m44.972994586s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620262406 (566h48m44.973047504s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620479183 (566h48m44.973264291s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620494562 (566h48m44.97327966s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620732109 (566h48m44.973517226s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620850641 (566h48m44.973635759s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621048552 (566h48m44.97383367s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621269727 (566h48m44.974054845s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621360306 (566h48m44.974145414s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621630594 (566h48m44.974415712s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621636906 (566h48m44.974422025s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.753712959 (566h48m45.106498087s) [upload=498847]: Next returned nil segment Sep 22 04:57:09.753730412 (566h48m45.10651554s) [upload=498847]: Waiting for error group managing segments... Sep 22 04:57:09.763163654 (566h48m45.115948782s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.763797522 (566h48m45.116582651s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764432855 (566h48m45.117217983s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764704334 (566h48m45.117489473s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765034515 (566h48m45.117819643s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765593363 (566h48m45.118378491s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766236961 (566h48m45.119022089s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766522516 (566h48m45.119307645s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766574073 (566h48m45.119359201s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766795570 (566h48m45.119580697s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767395605 (566h48m45.120180733s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767423678 (566h48m45.120208806s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:11.771267705 (566h48m47.124052832s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586132480 (566h49m14.938917588s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:57:39.586142679 (566h49m14.938927778s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586195388 (566h49m14.938980486s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: rpc: dial tcp 145.79.227.17:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586220555 (566h49m14.939005653s) [seg_pos={0 0}, upload=498847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:57:39.600066628 (566h49m14.952851727s) [upload=498847]: Done uploading segments... Sep 22 04:57:39.600082829 (566h49m14.952867937s) [upload=498847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 443.884508ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X........................X...X......] duration: 502.908432ms FAIL (2025-09-22 05:02:00 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:57:09.597529116 (566h48m44.950314224s) [upload=498847]: Starting upload Sep 22 04:57:09.597575052 (566h48m44.95036016s) [upload=498847]: Done starting upload Sep 22 04:57:09.597633201 (566h48m44.9504183s) [upload=498847]: Uploading segments... Sep 22 04:57:09.597786539 (566h48m44.950571637s) [upload=498847]: Got next segment. Inline: false Sep 22 04:57:09.597821665 (566h48m44.950606773s) [upload=498847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:57:09.597826965 (566h48m44.950612092s) [upload=498847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:57:09.619411660 (566h48m44.972196778s) [seg_pos={0 0}, upload=498847]: Begin upload segment... Sep 22 04:57:09.619441175 (566h48m44.972226293s) [seg_pos={0 0}, upload=498847]: Done begin upload segment. Sep 22 04:57:09.619461974 (566h48m44.972247092s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619509433 (566h48m44.972294552s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619801252 (566h48m44.97258635s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619932799 (566h48m44.972717907s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620051050 (566h48m44.972836158s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620209487 (566h48m44.972994586s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620262406 (566h48m44.973047504s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620479183 (566h48m44.973264291s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620494562 (566h48m44.97327966s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620732109 (566h48m44.973517226s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620850641 (566h48m44.973635759s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621048552 (566h48m44.97383367s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621269727 (566h48m44.974054845s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621360306 (566h48m44.974145414s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621630594 (566h48m44.974415712s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621636906 (566h48m44.974422025s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.753712959 (566h48m45.106498087s) [upload=498847]: Next returned nil segment Sep 22 04:57:09.753730412 (566h48m45.10651554s) [upload=498847]: Waiting for error group managing segments... Sep 22 04:57:09.763163654 (566h48m45.115948782s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.763797522 (566h48m45.116582651s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764432855 (566h48m45.117217983s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764704334 (566h48m45.117489473s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765034515 (566h48m45.117819643s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765593363 (566h48m45.118378491s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766236961 (566h48m45.119022089s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766522516 (566h48m45.119307645s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766574073 (566h48m45.119359201s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766795570 (566h48m45.119580697s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767395605 (566h48m45.120180733s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767423678 (566h48m45.120208806s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:11.771267705 (566h48m47.124052832s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586132480 (566h49m14.938917588s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:57:39.586142679 (566h49m14.938927778s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586195388 (566h49m14.938980486s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: rpc: dial tcp 145.79.227.17:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586220555 (566h49m14.939005653s) [seg_pos={0 0}, upload=498847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:57:39.600066628 (566h49m14.952851727s) [upload=498847]: Done uploading segments... Sep 22 04:57:39.600082829 (566h49m14.952867937s) [upload=498847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 502.963727ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X..................XXX.XXXXX.........X........X...........] duration: 1.051254504s FAIL (2025-09-22 05:01:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.051278961s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.146555205s FAIL (2025-09-22 05:01:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.146582498s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X..................XXX.XXXXX.........X........X..........] duration: 770.485837ms FAIL (2025-09-22 05:01:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 770.502359ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X........................X...X.....] duration: 401.255302ms FAIL (2025-09-22 05:01:27 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:57:09.597529116 (566h48m44.950314224s) [upload=498847]: Starting upload Sep 22 04:57:09.597575052 (566h48m44.95036016s) [upload=498847]: Done starting upload Sep 22 04:57:09.597633201 (566h48m44.9504183s) [upload=498847]: Uploading segments... Sep 22 04:57:09.597786539 (566h48m44.950571637s) [upload=498847]: Got next segment. Inline: false Sep 22 04:57:09.597821665 (566h48m44.950606773s) [upload=498847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:57:09.597826965 (566h48m44.950612092s) [upload=498847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:57:09.619411660 (566h48m44.972196778s) [seg_pos={0 0}, upload=498847]: Begin upload segment... Sep 22 04:57:09.619441175 (566h48m44.972226293s) [seg_pos={0 0}, upload=498847]: Done begin upload segment. Sep 22 04:57:09.619461974 (566h48m44.972247092s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619509433 (566h48m44.972294552s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619801252 (566h48m44.97258635s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619932799 (566h48m44.972717907s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620051050 (566h48m44.972836158s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620209487 (566h48m44.972994586s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620262406 (566h48m44.973047504s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620479183 (566h48m44.973264291s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620494562 (566h48m44.97327966s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620732109 (566h48m44.973517226s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620850641 (566h48m44.973635759s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621048552 (566h48m44.97383367s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621269727 (566h48m44.974054845s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621360306 (566h48m44.974145414s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621630594 (566h48m44.974415712s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621636906 (566h48m44.974422025s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.753712959 (566h48m45.106498087s) [upload=498847]: Next returned nil segment Sep 22 04:57:09.753730412 (566h48m45.10651554s) [upload=498847]: Waiting for error group managing segments... Sep 22 04:57:09.763163654 (566h48m45.115948782s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.763797522 (566h48m45.116582651s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764432855 (566h48m45.117217983s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764704334 (566h48m45.117489473s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765034515 (566h48m45.117819643s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765593363 (566h48m45.118378491s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766236961 (566h48m45.119022089s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766522516 (566h48m45.119307645s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766574073 (566h48m45.119359201s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766795570 (566h48m45.119580697s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767395605 (566h48m45.120180733s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767423678 (566h48m45.120208806s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:11.771267705 (566h48m47.124052832s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586132480 (566h49m14.938917588s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:57:39.586142679 (566h49m14.938927778s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586195388 (566h49m14.938980486s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: rpc: dial tcp 145.79.227.17:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586220555 (566h49m14.939005653s) [seg_pos={0 0}, upload=498847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:57:39.600066628 (566h49m14.952851727s) [upload=498847]: Done uploading segments... Sep 22 04:57:39.600082829 (566h49m14.952867937s) [upload=498847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 401.271342ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X..................XXX.XXXXX.........X........X.........] duration: 679.177817ms FAIL (2025-09-22 05:01:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 679.198235ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X........................X...X....] duration: 463.381865ms FAIL (2025-09-22 05:01:03 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:57:09.597529116 (566h48m44.950314224s) [upload=498847]: Starting upload Sep 22 04:57:09.597575052 (566h48m44.95036016s) [upload=498847]: Done starting upload Sep 22 04:57:09.597633201 (566h48m44.9504183s) [upload=498847]: Uploading segments... Sep 22 04:57:09.597786539 (566h48m44.950571637s) [upload=498847]: Got next segment. Inline: false Sep 22 04:57:09.597821665 (566h48m44.950606773s) [upload=498847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:57:09.597826965 (566h48m44.950612092s) [upload=498847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:57:09.619411660 (566h48m44.972196778s) [seg_pos={0 0}, upload=498847]: Begin upload segment... Sep 22 04:57:09.619441175 (566h48m44.972226293s) [seg_pos={0 0}, upload=498847]: Done begin upload segment. Sep 22 04:57:09.619461974 (566h48m44.972247092s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619509433 (566h48m44.972294552s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619801252 (566h48m44.97258635s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619932799 (566h48m44.972717907s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620051050 (566h48m44.972836158s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620209487 (566h48m44.972994586s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620262406 (566h48m44.973047504s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620479183 (566h48m44.973264291s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620494562 (566h48m44.97327966s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620732109 (566h48m44.973517226s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620850641 (566h48m44.973635759s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621048552 (566h48m44.97383367s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621269727 (566h48m44.974054845s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621360306 (566h48m44.974145414s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621630594 (566h48m44.974415712s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621636906 (566h48m44.974422025s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.753712959 (566h48m45.106498087s) [upload=498847]: Next returned nil segment Sep 22 04:57:09.753730412 (566h48m45.10651554s) [upload=498847]: Waiting for error group managing segments... Sep 22 04:57:09.763163654 (566h48m45.115948782s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.763797522 (566h48m45.116582651s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764432855 (566h48m45.117217983s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764704334 (566h48m45.117489473s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765034515 (566h48m45.117819643s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765593363 (566h48m45.118378491s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766236961 (566h48m45.119022089s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766522516 (566h48m45.119307645s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766574073 (566h48m45.119359201s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766795570 (566h48m45.119580697s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767395605 (566h48m45.120180733s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767423678 (566h48m45.120208806s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:11.771267705 (566h48m47.124052832s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586132480 (566h49m14.938917588s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:57:39.586142679 (566h49m14.938927778s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586195388 (566h49m14.938980486s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: rpc: dial tcp 145.79.227.17:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586220555 (566h49m14.939005653s) [seg_pos={0 0}, upload=498847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:57:39.600066628 (566h49m14.952851727s) [upload=498847]: Done uploading segments... Sep 22 04:57:39.600082829 (566h49m14.952867937s) [upload=498847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.417341ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 4.949476641s FAIL (2025-09-22 05:00:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.949506138s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X..................XXX.XXXXX.........X........X........] duration: 716.81093ms FAIL (2025-09-22 04:59:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 716.829154ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X........................X...X...] duration: 421.988465ms FAIL (2025-09-22 04:59:48 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:57:09.597529116 (566h48m44.950314224s) [upload=498847]: Starting upload Sep 22 04:57:09.597575052 (566h48m44.95036016s) [upload=498847]: Done starting upload Sep 22 04:57:09.597633201 (566h48m44.9504183s) [upload=498847]: Uploading segments... Sep 22 04:57:09.597786539 (566h48m44.950571637s) [upload=498847]: Got next segment. Inline: false Sep 22 04:57:09.597821665 (566h48m44.950606773s) [upload=498847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:57:09.597826965 (566h48m44.950612092s) [upload=498847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:57:09.619411660 (566h48m44.972196778s) [seg_pos={0 0}, upload=498847]: Begin upload segment... Sep 22 04:57:09.619441175 (566h48m44.972226293s) [seg_pos={0 0}, upload=498847]: Done begin upload segment. Sep 22 04:57:09.619461974 (566h48m44.972247092s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619509433 (566h48m44.972294552s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619801252 (566h48m44.97258635s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619932799 (566h48m44.972717907s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620051050 (566h48m44.972836158s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620209487 (566h48m44.972994586s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620262406 (566h48m44.973047504s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620479183 (566h48m44.973264291s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620494562 (566h48m44.97327966s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620732109 (566h48m44.973517226s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620850641 (566h48m44.973635759s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621048552 (566h48m44.97383367s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621269727 (566h48m44.974054845s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621360306 (566h48m44.974145414s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621630594 (566h48m44.974415712s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621636906 (566h48m44.974422025s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.753712959 (566h48m45.106498087s) [upload=498847]: Next returned nil segment Sep 22 04:57:09.753730412 (566h48m45.10651554s) [upload=498847]: Waiting for error group managing segments... Sep 22 04:57:09.763163654 (566h48m45.115948782s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.763797522 (566h48m45.116582651s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764432855 (566h48m45.117217983s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764704334 (566h48m45.117489473s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765034515 (566h48m45.117819643s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765593363 (566h48m45.118378491s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766236961 (566h48m45.119022089s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766522516 (566h48m45.119307645s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766574073 (566h48m45.119359201s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766795570 (566h48m45.119580697s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767395605 (566h48m45.120180733s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767423678 (566h48m45.120208806s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:11.771267705 (566h48m47.124052832s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586132480 (566h49m14.938917588s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:57:39.586142679 (566h49m14.938927778s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586195388 (566h49m14.938980486s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: rpc: dial tcp 145.79.227.17:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586220555 (566h49m14.939005653s) [seg_pos={0 0}, upload=498847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:57:39.600066628 (566h49m14.952851727s) [upload=498847]: Done uploading segments... Sep 22 04:57:39.600082829 (566h49m14.952867937s) [upload=498847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 422.013813ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 4.953198346s FAIL (2025-09-22 04:59:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.953224435s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X..................XXX.XXXXX.........X........X.......] duration: 376.535132ms FAIL (2025-09-22 04:59:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 376.553928ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.262203792s FAIL (2025-09-22 04:59:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.262236984s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 4.883516747s FAIL (2025-09-22 04:58:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.883535182s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X........................X...X..] duration: 453.400187ms FAIL (2025-09-22 04:58:36 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:57:09.597529116 (566h48m44.950314224s) [upload=498847]: Starting upload Sep 22 04:57:09.597575052 (566h48m44.95036016s) [upload=498847]: Done starting upload Sep 22 04:57:09.597633201 (566h48m44.9504183s) [upload=498847]: Uploading segments... Sep 22 04:57:09.597786539 (566h48m44.950571637s) [upload=498847]: Got next segment. Inline: false Sep 22 04:57:09.597821665 (566h48m44.950606773s) [upload=498847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:57:09.597826965 (566h48m44.950612092s) [upload=498847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:57:09.619411660 (566h48m44.972196778s) [seg_pos={0 0}, upload=498847]: Begin upload segment... Sep 22 04:57:09.619441175 (566h48m44.972226293s) [seg_pos={0 0}, upload=498847]: Done begin upload segment. Sep 22 04:57:09.619461974 (566h48m44.972247092s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619509433 (566h48m44.972294552s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619801252 (566h48m44.97258635s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619932799 (566h48m44.972717907s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620051050 (566h48m44.972836158s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620209487 (566h48m44.972994586s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620262406 (566h48m44.973047504s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620479183 (566h48m44.973264291s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620494562 (566h48m44.97327966s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620732109 (566h48m44.973517226s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620850641 (566h48m44.973635759s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621048552 (566h48m44.97383367s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621269727 (566h48m44.974054845s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621360306 (566h48m44.974145414s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621630594 (566h48m44.974415712s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621636906 (566h48m44.974422025s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.753712959 (566h48m45.106498087s) [upload=498847]: Next returned nil segment Sep 22 04:57:09.753730412 (566h48m45.10651554s) [upload=498847]: Waiting for error group managing segments... Sep 22 04:57:09.763163654 (566h48m45.115948782s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.763797522 (566h48m45.116582651s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764432855 (566h48m45.117217983s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764704334 (566h48m45.117489473s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765034515 (566h48m45.117819643s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765593363 (566h48m45.118378491s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766236961 (566h48m45.119022089s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766522516 (566h48m45.119307645s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766574073 (566h48m45.119359201s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766795570 (566h48m45.119580697s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767395605 (566h48m45.120180733s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767423678 (566h48m45.120208806s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:11.771267705 (566h48m47.124052832s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586132480 (566h49m14.938917588s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:57:39.586142679 (566h49m14.938927778s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586195388 (566h49m14.938980486s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: rpc: dial tcp 145.79.227.17:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586220555 (566h49m14.939005653s) [seg_pos={0 0}, upload=498847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:57:39.600066628 (566h49m14.952851727s) [upload=498847]: Done uploading segments... Sep 22 04:57:39.600082829 (566h49m14.952867937s) [upload=498847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.417849ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X......X..................XXX.XXXXX.........X........X......] duration: 428.03334ms FAIL (2025-09-22 04:58:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 428.053177ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X........................X...X.] duration: 411.380975ms FAIL (2025-09-22 04:58:14 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:57:09.597529116 (566h48m44.950314224s) [upload=498847]: Starting upload Sep 22 04:57:09.597575052 (566h48m44.95036016s) [upload=498847]: Done starting upload Sep 22 04:57:09.597633201 (566h48m44.9504183s) [upload=498847]: Uploading segments... Sep 22 04:57:09.597786539 (566h48m44.950571637s) [upload=498847]: Got next segment. Inline: false Sep 22 04:57:09.597821665 (566h48m44.950606773s) [upload=498847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:57:09.597826965 (566h48m44.950612092s) [upload=498847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:57:09.619411660 (566h48m44.972196778s) [seg_pos={0 0}, upload=498847]: Begin upload segment... Sep 22 04:57:09.619441175 (566h48m44.972226293s) [seg_pos={0 0}, upload=498847]: Done begin upload segment. Sep 22 04:57:09.619461974 (566h48m44.972247092s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619509433 (566h48m44.972294552s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619801252 (566h48m44.97258635s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619932799 (566h48m44.972717907s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620051050 (566h48m44.972836158s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620209487 (566h48m44.972994586s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620262406 (566h48m44.973047504s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620479183 (566h48m44.973264291s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620494562 (566h48m44.97327966s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620732109 (566h48m44.973517226s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620850641 (566h48m44.973635759s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621048552 (566h48m44.97383367s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621269727 (566h48m44.974054845s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621360306 (566h48m44.974145414s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621630594 (566h48m44.974415712s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621636906 (566h48m44.974422025s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.753712959 (566h48m45.106498087s) [upload=498847]: Next returned nil segment Sep 22 04:57:09.753730412 (566h48m45.10651554s) [upload=498847]: Waiting for error group managing segments... Sep 22 04:57:09.763163654 (566h48m45.115948782s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.763797522 (566h48m45.116582651s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764432855 (566h48m45.117217983s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764704334 (566h48m45.117489473s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765034515 (566h48m45.117819643s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765593363 (566h48m45.118378491s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766236961 (566h48m45.119022089s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766522516 (566h48m45.119307645s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766574073 (566h48m45.119359201s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766795570 (566h48m45.119580697s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767395605 (566h48m45.120180733s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767423678 (566h48m45.120208806s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:11.771267705 (566h48m47.124052832s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586132480 (566h49m14.938917588s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:57:39.586142679 (566h49m14.938927778s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586195388 (566h49m14.938980486s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: rpc: dial tcp 145.79.227.17:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586220555 (566h49m14.939005653s) [seg_pos={0 0}, upload=498847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:57:39.600066628 (566h49m14.952851727s) [upload=498847]: Done uploading segments... Sep 22 04:57:39.600082829 (566h49m14.952867937s) [upload=498847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 411.399028ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X........................X...X] duration: 30.014969184s FAIL (2025-09-22 04:57:39 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:57:09.597529116 (566h48m44.950314224s) [upload=498847]: Starting upload Sep 22 04:57:09.597575052 (566h48m44.95036016s) [upload=498847]: Done starting upload Sep 22 04:57:09.597633201 (566h48m44.9504183s) [upload=498847]: Uploading segments... Sep 22 04:57:09.597786539 (566h48m44.950571637s) [upload=498847]: Got next segment. Inline: false Sep 22 04:57:09.597821665 (566h48m44.950606773s) [upload=498847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:57:09.597826965 (566h48m44.950612092s) [upload=498847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:57:09.619411660 (566h48m44.972196778s) [seg_pos={0 0}, upload=498847]: Begin upload segment... Sep 22 04:57:09.619441175 (566h48m44.972226293s) [seg_pos={0 0}, upload=498847]: Done begin upload segment. Sep 22 04:57:09.619461974 (566h48m44.972247092s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619509433 (566h48m44.972294552s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619801252 (566h48m44.97258635s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.619932799 (566h48m44.972717907s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620051050 (566h48m44.972836158s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620209487 (566h48m44.972994586s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620262406 (566h48m44.973047504s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620479183 (566h48m44.973264291s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620494562 (566h48m44.97327966s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620732109 (566h48m44.973517226s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.620850641 (566h48m44.973635759s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621048552 (566h48m44.97383367s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621269727 (566h48m44.974054845s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621360306 (566h48m44.974145414s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621630594 (566h48m44.974415712s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.621636906 (566h48m44.974422025s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Uploading piece... Sep 22 04:57:09.753712959 (566h48m45.106498087s) [upload=498847]: Next returned nil segment Sep 22 04:57:09.753730412 (566h48m45.10651554s) [upload=498847]: Waiting for error group managing segments... Sep 22 04:57:09.763163654 (566h48m45.115948782s) [noise=true , address=45.91.116.10:4082 , piece_id=4DJUU5YL24Q4VGVHXWZLRZQ7TVB4CMJGR66JFQ5XTVIRT2344PHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.763797522 (566h48m45.116582651s) [noise=true , address=45.91.116.13:4154 , piece_id=GBLGQMZUAAA4FGUIDOLCLSWG6BYY2GLHKHCK2B7YMR3CY5U5ZKBQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764432855 (566h48m45.117217983s) [noise=true , address=77.90.31.16:4735 , piece_id=NFOB2YJHNUIBWGNB6QTXZ7EUA23XOE4DAUPUOUUJEPWMF7Y2IQBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.764704334 (566h48m45.117489473s) [noise=true , address=45.91.116.8:4047 , piece_id=BW6OBMKEE3UKVCHZWKEKUB5XPE5ES3KJDWVAVHOL6PISYVS5ZT6Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765034515 (566h48m45.117819643s) [noise=true , address=45.91.116.4:4939 , piece_id=6BUQGJOKTYGSVHQFRZDXCZAX6SEFBW6VVO6JG6QNRYLLHWR4Z5VQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.765593363 (566h48m45.118378491s) [noise=true , address=45.91.116.14:4191 , piece_id=QX77PWQENDNVGPMWSHUKWGLJR55HCR4PI2N5KNJXPUZD7LF3OKTQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766236961 (566h48m45.119022089s) [noise=true , address=45.91.116.10:4081 , piece_id=S2C6SOPKT5JMDTJMW725SZ66TSGM3S3GBDLN2ESO2JT24352FX2Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766522516 (566h48m45.119307645s) [noise=true , address=45.91.116.5:4297 , piece_id=GT7N2CFHZ5GHI2BHL4HATGSOMJYZHB24CXEK7UV4ZMUO2NR26T5Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766574073 (566h48m45.119359201s) [noise=true , address=45.91.116.6:4975 , piece_id=P2IWUHC4JYSWB7NLW4CKMMULJXV3YNZCX4EYGS5IWRC4V552WDBA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.766795570 (566h48m45.119580697s) [noise=true , address=45.91.116.11:4118 , piece_id=HZTZXWVK3SWMVU7J2EZ7JERSF52AOLDMTRPAY7V3HYG3X46JQWUQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767395605 (566h48m45.120180733s) [noise=true , address=77.90.31.10:4517 , piece_id=F4FWWEH5VJUS24E2ILFQUA2E2GXFQL32SZCQXFJ5M2E5RMPOMUHQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:09.767423678 (566h48m45.120208806s) [noise=true , address=45.91.116.12:4227 , piece_id=DOKIAHVOKI7JYFWU6HGQKDPVZAH52XCPQUD2R2DJ5UPI3C2HYSXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: <nil> Sep 22 04:57:11.771267705 (566h48m47.124052832s) [noise=true , address=145.79.227.16:4909 , piece_id=6O6BXCAB35CHGX5KJUNMX4Q632CYUSU2JTRYXX5K6FIBOVL2XEFQ, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586132480 (566h49m14.938917588s) [noise=true , address=145.79.227.15:4874 , piece_id=ZDT47DQMODDZN7YXK4Z7MVE4TCU3DNA2AWLHKV67DIP2LZ7YBSOA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:57:39.586142679 (566h49m14.938927778s) [noise=true , address=145.79.227.19:4079 , piece_id=5C5TDWAX4PZPQ6U3F6PC6LBK7PZEQJ6PPHIQC77PAQZELH4PYZXA, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586195388 (566h49m14.938980486s) [noise=true , address=145.79.227.17:4969 , piece_id=K3JWYXH7IVNEME63MDHHIVAOCKROFW64O5XGGATTWW55MTRGKT3Q, seg_pos={0 0}, upload=498847]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: rpc: dial tcp 145.79.227.17:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:57:39.586220555 (566h49m14.939005653s) [seg_pos={0 0}, upload=498847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:57:39.600066628 (566h49m14.952851727s) [upload=498847]: Done uploading segments... Sep 22 04:57:39.600082829 (566h49m14.952867937s) [upload=498847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.014982809s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 4.977748312s FAIL (2025-09-22 04:57:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.977774651s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X......X..................XXX.XXXXX.........X........X.....] duration: 390.731414ms FAIL (2025-09-22 04:57:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 390.751592ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X........................X...] duration: 489.026282ms FAIL (2025-09-22 04:56:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:54:28.279017060 (566h46m3.631802168s) [upload=498810]: Starting upload Sep 22 04:54:28.279088584 (566h46m3.631873682s) [upload=498810]: Done starting upload Sep 22 04:54:28.279286835 (566h46m3.632071943s) [upload=498810]: Uploading segments... Sep 22 04:54:28.279404757 (566h46m3.632189865s) [upload=498810]: Got next segment. Inline: false Sep 22 04:54:28.279440063 (566h46m3.632225171s) [upload=498810]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:54:28.279444170 (566h46m3.632229269s) [upload=498810]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:54:28.316395062 (566h46m3.66918017s) [seg_pos={0 0}, upload=498810]: Begin upload segment... Sep 22 04:54:28.316425259 (566h46m3.669210357s) [seg_pos={0 0}, upload=498810]: Done begin upload segment. Sep 22 04:54:28.316444585 (566h46m3.669229694s) [noise=true , address=77.90.31.18:4805 , piece_id=SYYYGZJNPXL62CD2EDJ7HTDZVAGFV23BSNMXLL5U4VMVRECZI3LQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.316494850 (566h46m3.669279948s) [noise=true , address=45.91.116.5:4297 , piece_id=WIAZ24OOOJGIS57A3VALOEINHJ4LCKLGDQ3V2LWOYATY4XDVE6RA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.316733016 (566h46m3.669518124s) [noise=true , address=77.90.31.12:4590 , piece_id=6YYVSTSRZ5CENFV6BRYOPSSAX22ME6BXCORJUND5YYY5OXQU2F3Q, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.316751130 (566h46m3.669536238s) [noise=true , address=145.79.227.19:4078 , piece_id=YSMSYFBS2QHK6ZLQXHOMQUN4IYLR4HQHW42WF4HMOKIKE6YZRJOA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317000808 (566h46m3.669785916s) [noise=true , address=77.90.31.19:4841 , piece_id=Z4RWZXXICIXKQUVXBDDFAIIMPPRUCKZUXGYB3D7IHEBTTZUW3FAQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317013212 (566h46m3.66979832s) [noise=true , address=45.91.116.4:4940 , piece_id=A6YHS7R72YYXHZEGOKNHSBSL6PAIYW4ZEPAZQ7BYDQCV5FS46M4Q, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317235951 (566h46m3.670021049s) [noise=true , address=145.79.227.13:4801 , piece_id=TTVJGJQX2UZVLYAKSVD6CF6VY2UL6OQR5GMLQUQOLL5DTVO3RG4A, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317449061 (566h46m3.670234158s) [noise=true , address=77.90.31.14:4662 , piece_id=3IPX6TBM3DJIAXKIRQWK5ZMAG4PJBQAEQCCLABA337F77ZLSHE5A, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317532486 (566h46m3.670317594s) [noise=true , address=45.91.116.11:4117 , piece_id=XTF6SNJ7EZGHJYU75YCCOY27XCOALI4ARN5U5IGP5LG5CJH67XGQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317654906 (566h46m3.670440004s) [noise=true , address=77.90.31.17:4771 , piece_id=STZJUQPM52ZAILBLDOS3ID2NMTX5SN7OMEBINAKBUN5HF67WYKXQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317780301 (566h46m3.670565409s) [noise=true , address=45.91.116.12:4227 , piece_id=XJIWHHR3MMOOZU7NA7KYKB3Y7LNULX265E3SN3V2W5DHL2HVCRHQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317858769 (566h46m3.670643877s) [noise=true , address=45.91.116.12:4225 , piece_id=JD2BMT5ZPMXHBZJEM5IBQ5F722RZKHUB2DL3UWHQAYA247HCYK6A, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318024650 (566h46m3.670809749s) [noise=true , address=145.79.227.3:4149 , piece_id=D7QK4CHIM6JCJSD525OL76UOCWOHRWHHSSURCF5VRXYOE26ME7DQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318076999 (566h46m3.670862097s) [noise=true , address=145.79.227.16:4909 , piece_id=237F3T43VTPPCFP2L55YSFGW4WJ4ORCGUG5FHNYL6CCMR3I7H6DA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318270001 (566h46m3.671055109s) [noise=true , address=45.91.116.13:4155 , piece_id=HARQMYJGTUMGH43D3RIJ7JDTSWG3ZAFOQGFK7X7OBDOVRVLNWNSA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318296210 (566h46m3.671081318s) [noise=true , address=77.90.31.13:4626 , piece_id=GRHQM2NK5KHAG5TVBG6MQXQYOJOCFMMJS4HESCTBOBRAOVWEBPGA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.395620925 (566h46m3.748406033s) [upload=498810]: Next returned nil segment Sep 22 04:54:28.395632406 (566h46m3.748417515s) [upload=498810]: Waiting for error group managing segments... Sep 22 04:54:28.404132007 (566h46m3.756917115s) [noise=true , address=45.91.116.12:4227 , piece_id=XJIWHHR3MMOOZU7NA7KYKB3Y7LNULX265E3SN3V2W5DHL2HVCRHQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404327825 (566h46m3.757112923s) [noise=true , address=45.91.116.12:4225 , piece_id=JD2BMT5ZPMXHBZJEM5IBQ5F722RZKHUB2DL3UWHQAYA247HCYK6A, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404472256 (566h46m3.757257364s) [noise=true , address=45.91.116.5:4297 , piece_id=WIAZ24OOOJGIS57A3VALOEINHJ4LCKLGDQ3V2LWOYATY4XDVE6RA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404908104 (566h46m3.757693201s) [noise=true , address=77.90.31.12:4590 , piece_id=6YYVSTSRZ5CENFV6BRYOPSSAX22ME6BXCORJUND5YYY5OXQU2F3Q, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404995728 (566h46m3.757780836s) [noise=true , address=45.91.116.13:4155 , piece_id=HARQMYJGTUMGH43D3RIJ7JDTSWG3ZAFOQGFK7X7OBDOVRVLNWNSA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405266737 (566h46m3.758051845s) [noise=true , address=45.91.116.11:4117 , piece_id=XTF6SNJ7EZGHJYU75YCCOY27XCOALI4ARN5U5IGP5LG5CJH67XGQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405618116 (566h46m3.758403224s) [noise=true , address=77.90.31.19:4841 , piece_id=Z4RWZXXICIXKQUVXBDDFAIIMPPRUCKZUXGYB3D7IHEBTTZUW3FAQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405646719 (566h46m3.758431827s) [noise=true , address=45.91.116.4:4940 , piece_id=A6YHS7R72YYXHZEGOKNHSBSL6PAIYW4ZEPAZQ7BYDQCV5FS46M4Q, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405977380 (566h46m3.758762488s) [noise=true , address=77.90.31.13:4626 , piece_id=GRHQM2NK5KHAG5TVBG6MQXQYOJOCFMMJS4HESCTBOBRAOVWEBPGA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.406215007 (566h46m3.759000115s) [noise=true , address=77.90.31.18:4805 , piece_id=SYYYGZJNPXL62CD2EDJ7HTDZVAGFV23BSNMXLL5U4VMVRECZI3LQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.408093832 (566h46m3.76087893s) [noise=true , address=77.90.31.14:4662 , piece_id=3IPX6TBM3DJIAXKIRQWK5ZMAG4PJBQAEQCCLABA337F77ZLSHE5A, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.409582205 (566h46m3.762367313s) [noise=true , address=77.90.31.17:4771 , piece_id=STZJUQPM52ZAILBLDOS3ID2NMTX5SN7OMEBINAKBUN5HF67WYKXQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:32.565058717 (566h46m7.917843815s) [noise=true , address=145.79.227.3:4149 , piece_id=D7QK4CHIM6JCJSD525OL76UOCWOHRWHHSSURCF5VRXYOE26ME7DQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: connect: no route to host; manager closed: conn closed: op Sep 22 04:54:58.264859545 (566h46m33.617644653s) [noise=true , address=145.79.227.19:4078 , piece_id=YSMSYFBS2QHK6ZLQXHOMQUN4IYLR4HQHW42WF4HMOKIKE6YZRJOA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:54:58.264933745 (566h46m33.617718853s) [noise=true , address=145.79.227.13:4801 , piece_id=TTVJGJQX2UZVLYAKSVD6CF6VY2UL6OQR5GMLQUQOLL5DTVO3RG4A, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:54:58.264952610 (566h46m33.617737718s) [noise=true , address=145.79.227.16:4909 , piece_id=237F3T43VTPPCFP2L55YSFGW4WJ4ORCGUG5FHNYL6CCMR3I7H6DA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:54:58.264980422 (566h46m33.61776553s) [seg_pos={0 0}, upload=498810]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:54:58.290835962 (566h46m33.64362106s) [upload=498810]: Done uploading segments... Sep 22 04:54:58.290853404 (566h46m33.643638512s) [upload=498810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 489.051408ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 4.979610321s FAIL (2025-09-22 04:56:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.979634687s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X......X..................XXX.XXXXX.........X........X....] duration: 509.798356ms FAIL (2025-09-22 04:56:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 509.815118ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X..X......X..................XXX.XXXXX.........X........X...] duration: 828.787484ms FAIL (2025-09-22 04:56:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 828.808964ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X........................X..] duration: 583.078963ms FAIL (2025-09-22 04:56:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:54:28.279017060 (566h46m3.631802168s) [upload=498810]: Starting upload Sep 22 04:54:28.279088584 (566h46m3.631873682s) [upload=498810]: Done starting upload Sep 22 04:54:28.279286835 (566h46m3.632071943s) [upload=498810]: Uploading segments... Sep 22 04:54:28.279404757 (566h46m3.632189865s) [upload=498810]: Got next segment. Inline: false Sep 22 04:54:28.279440063 (566h46m3.632225171s) [upload=498810]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:54:28.279444170 (566h46m3.632229269s) [upload=498810]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:54:28.316395062 (566h46m3.66918017s) [seg_pos={0 0}, upload=498810]: Begin upload segment... Sep 22 04:54:28.316425259 (566h46m3.669210357s) [seg_pos={0 0}, upload=498810]: Done begin upload segment. Sep 22 04:54:28.316444585 (566h46m3.669229694s) [noise=true , address=77.90.31.18:4805 , piece_id=SYYYGZJNPXL62CD2EDJ7HTDZVAGFV23BSNMXLL5U4VMVRECZI3LQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.316494850 (566h46m3.669279948s) [noise=true , address=45.91.116.5:4297 , piece_id=WIAZ24OOOJGIS57A3VALOEINHJ4LCKLGDQ3V2LWOYATY4XDVE6RA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.316733016 (566h46m3.669518124s) [noise=true , address=77.90.31.12:4590 , piece_id=6YYVSTSRZ5CENFV6BRYOPSSAX22ME6BXCORJUND5YYY5OXQU2F3Q, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.316751130 (566h46m3.669536238s) [noise=true , address=145.79.227.19:4078 , piece_id=YSMSYFBS2QHK6ZLQXHOMQUN4IYLR4HQHW42WF4HMOKIKE6YZRJOA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317000808 (566h46m3.669785916s) [noise=true , address=77.90.31.19:4841 , piece_id=Z4RWZXXICIXKQUVXBDDFAIIMPPRUCKZUXGYB3D7IHEBTTZUW3FAQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317013212 (566h46m3.66979832s) [noise=true , address=45.91.116.4:4940 , piece_id=A6YHS7R72YYXHZEGOKNHSBSL6PAIYW4ZEPAZQ7BYDQCV5FS46M4Q, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317235951 (566h46m3.670021049s) [noise=true , address=145.79.227.13:4801 , piece_id=TTVJGJQX2UZVLYAKSVD6CF6VY2UL6OQR5GMLQUQOLL5DTVO3RG4A, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317449061 (566h46m3.670234158s) [noise=true , address=77.90.31.14:4662 , piece_id=3IPX6TBM3DJIAXKIRQWK5ZMAG4PJBQAEQCCLABA337F77ZLSHE5A, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317532486 (566h46m3.670317594s) [noise=true , address=45.91.116.11:4117 , piece_id=XTF6SNJ7EZGHJYU75YCCOY27XCOALI4ARN5U5IGP5LG5CJH67XGQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317654906 (566h46m3.670440004s) [noise=true , address=77.90.31.17:4771 , piece_id=STZJUQPM52ZAILBLDOS3ID2NMTX5SN7OMEBINAKBUN5HF67WYKXQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317780301 (566h46m3.670565409s) [noise=true , address=45.91.116.12:4227 , piece_id=XJIWHHR3MMOOZU7NA7KYKB3Y7LNULX265E3SN3V2W5DHL2HVCRHQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317858769 (566h46m3.670643877s) [noise=true , address=45.91.116.12:4225 , piece_id=JD2BMT5ZPMXHBZJEM5IBQ5F722RZKHUB2DL3UWHQAYA247HCYK6A, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318024650 (566h46m3.670809749s) [noise=true , address=145.79.227.3:4149 , piece_id=D7QK4CHIM6JCJSD525OL76UOCWOHRWHHSSURCF5VRXYOE26ME7DQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318076999 (566h46m3.670862097s) [noise=true , address=145.79.227.16:4909 , piece_id=237F3T43VTPPCFP2L55YSFGW4WJ4ORCGUG5FHNYL6CCMR3I7H6DA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318270001 (566h46m3.671055109s) [noise=true , address=45.91.116.13:4155 , piece_id=HARQMYJGTUMGH43D3RIJ7JDTSWG3ZAFOQGFK7X7OBDOVRVLNWNSA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318296210 (566h46m3.671081318s) [noise=true , address=77.90.31.13:4626 , piece_id=GRHQM2NK5KHAG5TVBG6MQXQYOJOCFMMJS4HESCTBOBRAOVWEBPGA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.395620925 (566h46m3.748406033s) [upload=498810]: Next returned nil segment Sep 22 04:54:28.395632406 (566h46m3.748417515s) [upload=498810]: Waiting for error group managing segments... Sep 22 04:54:28.404132007 (566h46m3.756917115s) [noise=true , address=45.91.116.12:4227 , piece_id=XJIWHHR3MMOOZU7NA7KYKB3Y7LNULX265E3SN3V2W5DHL2HVCRHQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404327825 (566h46m3.757112923s) [noise=true , address=45.91.116.12:4225 , piece_id=JD2BMT5ZPMXHBZJEM5IBQ5F722RZKHUB2DL3UWHQAYA247HCYK6A, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404472256 (566h46m3.757257364s) [noise=true , address=45.91.116.5:4297 , piece_id=WIAZ24OOOJGIS57A3VALOEINHJ4LCKLGDQ3V2LWOYATY4XDVE6RA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404908104 (566h46m3.757693201s) [noise=true , address=77.90.31.12:4590 , piece_id=6YYVSTSRZ5CENFV6BRYOPSSAX22ME6BXCORJUND5YYY5OXQU2F3Q, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404995728 (566h46m3.757780836s) [noise=true , address=45.91.116.13:4155 , piece_id=HARQMYJGTUMGH43D3RIJ7JDTSWG3ZAFOQGFK7X7OBDOVRVLNWNSA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405266737 (566h46m3.758051845s) [noise=true , address=45.91.116.11:4117 , piece_id=XTF6SNJ7EZGHJYU75YCCOY27XCOALI4ARN5U5IGP5LG5CJH67XGQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405618116 (566h46m3.758403224s) [noise=true , address=77.90.31.19:4841 , piece_id=Z4RWZXXICIXKQUVXBDDFAIIMPPRUCKZUXGYB3D7IHEBTTZUW3FAQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405646719 (566h46m3.758431827s) [noise=true , address=45.91.116.4:4940 , piece_id=A6YHS7R72YYXHZEGOKNHSBSL6PAIYW4ZEPAZQ7BYDQCV5FS46M4Q, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405977380 (566h46m3.758762488s) [noise=true , address=77.90.31.13:4626 , piece_id=GRHQM2NK5KHAG5TVBG6MQXQYOJOCFMMJS4HESCTBOBRAOVWEBPGA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.406215007 (566h46m3.759000115s) [noise=true , address=77.90.31.18:4805 , piece_id=SYYYGZJNPXL62CD2EDJ7HTDZVAGFV23BSNMXLL5U4VMVRECZI3LQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.408093832 (566h46m3.76087893s) [noise=true , address=77.90.31.14:4662 , piece_id=3IPX6TBM3DJIAXKIRQWK5ZMAG4PJBQAEQCCLABA337F77ZLSHE5A, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.409582205 (566h46m3.762367313s) [noise=true , address=77.90.31.17:4771 , piece_id=STZJUQPM52ZAILBLDOS3ID2NMTX5SN7OMEBINAKBUN5HF67WYKXQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:32.565058717 (566h46m7.917843815s) [noise=true , address=145.79.227.3:4149 , piece_id=D7QK4CHIM6JCJSD525OL76UOCWOHRWHHSSURCF5VRXYOE26ME7DQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: connect: no route to host; manager closed: conn closed: op Sep 22 04:54:58.264859545 (566h46m33.617644653s) [noise=true , address=145.79.227.19:4078 , piece_id=YSMSYFBS2QHK6ZLQXHOMQUN4IYLR4HQHW42WF4HMOKIKE6YZRJOA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:54:58.264933745 (566h46m33.617718853s) [noise=true , address=145.79.227.13:4801 , piece_id=TTVJGJQX2UZVLYAKSVD6CF6VY2UL6OQR5GMLQUQOLL5DTVO3RG4A, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:54:58.264952610 (566h46m33.617737718s) [noise=true , address=145.79.227.16:4909 , piece_id=237F3T43VTPPCFP2L55YSFGW4WJ4ORCGUG5FHNYL6CCMR3I7H6DA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:54:58.264980422 (566h46m33.61776553s) [seg_pos={0 0}, upload=498810]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:54:58.290835962 (566h46m33.64362106s) [upload=498810]: Done uploading segments... Sep 22 04:54:58.290853404 (566h46m33.643638512s) [upload=498810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 583.105855ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 4.875640307s FAIL (2025-09-22 04:55:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.875662008s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X........................X.] duration: 499.140435ms FAIL (2025-09-22 04:55:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:54:28.279017060 (566h46m3.631802168s) [upload=498810]: Starting upload Sep 22 04:54:28.279088584 (566h46m3.631873682s) [upload=498810]: Done starting upload Sep 22 04:54:28.279286835 (566h46m3.632071943s) [upload=498810]: Uploading segments... Sep 22 04:54:28.279404757 (566h46m3.632189865s) [upload=498810]: Got next segment. Inline: false Sep 22 04:54:28.279440063 (566h46m3.632225171s) [upload=498810]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:54:28.279444170 (566h46m3.632229269s) [upload=498810]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:54:28.316395062 (566h46m3.66918017s) [seg_pos={0 0}, upload=498810]: Begin upload segment... Sep 22 04:54:28.316425259 (566h46m3.669210357s) [seg_pos={0 0}, upload=498810]: Done begin upload segment. Sep 22 04:54:28.316444585 (566h46m3.669229694s) [noise=true , address=77.90.31.18:4805 , piece_id=SYYYGZJNPXL62CD2EDJ7HTDZVAGFV23BSNMXLL5U4VMVRECZI3LQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.316494850 (566h46m3.669279948s) [noise=true , address=45.91.116.5:4297 , piece_id=WIAZ24OOOJGIS57A3VALOEINHJ4LCKLGDQ3V2LWOYATY4XDVE6RA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.316733016 (566h46m3.669518124s) [noise=true , address=77.90.31.12:4590 , piece_id=6YYVSTSRZ5CENFV6BRYOPSSAX22ME6BXCORJUND5YYY5OXQU2F3Q, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.316751130 (566h46m3.669536238s) [noise=true , address=145.79.227.19:4078 , piece_id=YSMSYFBS2QHK6ZLQXHOMQUN4IYLR4HQHW42WF4HMOKIKE6YZRJOA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317000808 (566h46m3.669785916s) [noise=true , address=77.90.31.19:4841 , piece_id=Z4RWZXXICIXKQUVXBDDFAIIMPPRUCKZUXGYB3D7IHEBTTZUW3FAQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317013212 (566h46m3.66979832s) [noise=true , address=45.91.116.4:4940 , piece_id=A6YHS7R72YYXHZEGOKNHSBSL6PAIYW4ZEPAZQ7BYDQCV5FS46M4Q, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317235951 (566h46m3.670021049s) [noise=true , address=145.79.227.13:4801 , piece_id=TTVJGJQX2UZVLYAKSVD6CF6VY2UL6OQR5GMLQUQOLL5DTVO3RG4A, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317449061 (566h46m3.670234158s) [noise=true , address=77.90.31.14:4662 , piece_id=3IPX6TBM3DJIAXKIRQWK5ZMAG4PJBQAEQCCLABA337F77ZLSHE5A, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317532486 (566h46m3.670317594s) [noise=true , address=45.91.116.11:4117 , piece_id=XTF6SNJ7EZGHJYU75YCCOY27XCOALI4ARN5U5IGP5LG5CJH67XGQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317654906 (566h46m3.670440004s) [noise=true , address=77.90.31.17:4771 , piece_id=STZJUQPM52ZAILBLDOS3ID2NMTX5SN7OMEBINAKBUN5HF67WYKXQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317780301 (566h46m3.670565409s) [noise=true , address=45.91.116.12:4227 , piece_id=XJIWHHR3MMOOZU7NA7KYKB3Y7LNULX265E3SN3V2W5DHL2HVCRHQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317858769 (566h46m3.670643877s) [noise=true , address=45.91.116.12:4225 , piece_id=JD2BMT5ZPMXHBZJEM5IBQ5F722RZKHUB2DL3UWHQAYA247HCYK6A, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318024650 (566h46m3.670809749s) [noise=true , address=145.79.227.3:4149 , piece_id=D7QK4CHIM6JCJSD525OL76UOCWOHRWHHSSURCF5VRXYOE26ME7DQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318076999 (566h46m3.670862097s) [noise=true , address=145.79.227.16:4909 , piece_id=237F3T43VTPPCFP2L55YSFGW4WJ4ORCGUG5FHNYL6CCMR3I7H6DA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318270001 (566h46m3.671055109s) [noise=true , address=45.91.116.13:4155 , piece_id=HARQMYJGTUMGH43D3RIJ7JDTSWG3ZAFOQGFK7X7OBDOVRVLNWNSA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318296210 (566h46m3.671081318s) [noise=true , address=77.90.31.13:4626 , piece_id=GRHQM2NK5KHAG5TVBG6MQXQYOJOCFMMJS4HESCTBOBRAOVWEBPGA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.395620925 (566h46m3.748406033s) [upload=498810]: Next returned nil segment Sep 22 04:54:28.395632406 (566h46m3.748417515s) [upload=498810]: Waiting for error group managing segments... Sep 22 04:54:28.404132007 (566h46m3.756917115s) [noise=true , address=45.91.116.12:4227 , piece_id=XJIWHHR3MMOOZU7NA7KYKB3Y7LNULX265E3SN3V2W5DHL2HVCRHQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404327825 (566h46m3.757112923s) [noise=true , address=45.91.116.12:4225 , piece_id=JD2BMT5ZPMXHBZJEM5IBQ5F722RZKHUB2DL3UWHQAYA247HCYK6A, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404472256 (566h46m3.757257364s) [noise=true , address=45.91.116.5:4297 , piece_id=WIAZ24OOOJGIS57A3VALOEINHJ4LCKLGDQ3V2LWOYATY4XDVE6RA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404908104 (566h46m3.757693201s) [noise=true , address=77.90.31.12:4590 , piece_id=6YYVSTSRZ5CENFV6BRYOPSSAX22ME6BXCORJUND5YYY5OXQU2F3Q, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404995728 (566h46m3.757780836s) [noise=true , address=45.91.116.13:4155 , piece_id=HARQMYJGTUMGH43D3RIJ7JDTSWG3ZAFOQGFK7X7OBDOVRVLNWNSA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405266737 (566h46m3.758051845s) [noise=true , address=45.91.116.11:4117 , piece_id=XTF6SNJ7EZGHJYU75YCCOY27XCOALI4ARN5U5IGP5LG5CJH67XGQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405618116 (566h46m3.758403224s) [noise=true , address=77.90.31.19:4841 , piece_id=Z4RWZXXICIXKQUVXBDDFAIIMPPRUCKZUXGYB3D7IHEBTTZUW3FAQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405646719 (566h46m3.758431827s) [noise=true , address=45.91.116.4:4940 , piece_id=A6YHS7R72YYXHZEGOKNHSBSL6PAIYW4ZEPAZQ7BYDQCV5FS46M4Q, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405977380 (566h46m3.758762488s) [noise=true , address=77.90.31.13:4626 , piece_id=GRHQM2NK5KHAG5TVBG6MQXQYOJOCFMMJS4HESCTBOBRAOVWEBPGA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.406215007 (566h46m3.759000115s) [noise=true , address=77.90.31.18:4805 , piece_id=SYYYGZJNPXL62CD2EDJ7HTDZVAGFV23BSNMXLL5U4VMVRECZI3LQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.408093832 (566h46m3.76087893s) [noise=true , address=77.90.31.14:4662 , piece_id=3IPX6TBM3DJIAXKIRQWK5ZMAG4PJBQAEQCCLABA337F77ZLSHE5A, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.409582205 (566h46m3.762367313s) [noise=true , address=77.90.31.17:4771 , piece_id=STZJUQPM52ZAILBLDOS3ID2NMTX5SN7OMEBINAKBUN5HF67WYKXQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:32.565058717 (566h46m7.917843815s) [noise=true , address=145.79.227.3:4149 , piece_id=D7QK4CHIM6JCJSD525OL76UOCWOHRWHHSSURCF5VRXYOE26ME7DQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: connect: no route to host; manager closed: conn closed: op Sep 22 04:54:58.264859545 (566h46m33.617644653s) [noise=true , address=145.79.227.19:4078 , piece_id=YSMSYFBS2QHK6ZLQXHOMQUN4IYLR4HQHW42WF4HMOKIKE6YZRJOA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:54:58.264933745 (566h46m33.617718853s) [noise=true , address=145.79.227.13:4801 , piece_id=TTVJGJQX2UZVLYAKSVD6CF6VY2UL6OQR5GMLQUQOLL5DTVO3RG4A, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:54:58.264952610 (566h46m33.617737718s) [noise=true , address=145.79.227.16:4909 , piece_id=237F3T43VTPPCFP2L55YSFGW4WJ4ORCGUG5FHNYL6CCMR3I7H6DA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:54:58.264980422 (566h46m33.61776553s) [seg_pos={0 0}, upload=498810]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:54:58.290835962 (566h46m33.64362106s) [upload=498810]: Done uploading segments... Sep 22 04:54:58.290853404 (566h46m33.643638512s) [upload=498810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 499.200447ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X........................X] duration: 30.027039253s FAIL (2025-09-22 04:54:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:54:28.279017060 (566h46m3.631802168s) [upload=498810]: Starting upload Sep 22 04:54:28.279088584 (566h46m3.631873682s) [upload=498810]: Done starting upload Sep 22 04:54:28.279286835 (566h46m3.632071943s) [upload=498810]: Uploading segments... Sep 22 04:54:28.279404757 (566h46m3.632189865s) [upload=498810]: Got next segment. Inline: false Sep 22 04:54:28.279440063 (566h46m3.632225171s) [upload=498810]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:54:28.279444170 (566h46m3.632229269s) [upload=498810]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:54:28.316395062 (566h46m3.66918017s) [seg_pos={0 0}, upload=498810]: Begin upload segment... Sep 22 04:54:28.316425259 (566h46m3.669210357s) [seg_pos={0 0}, upload=498810]: Done begin upload segment. Sep 22 04:54:28.316444585 (566h46m3.669229694s) [noise=true , address=77.90.31.18:4805 , piece_id=SYYYGZJNPXL62CD2EDJ7HTDZVAGFV23BSNMXLL5U4VMVRECZI3LQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.316494850 (566h46m3.669279948s) [noise=true , address=45.91.116.5:4297 , piece_id=WIAZ24OOOJGIS57A3VALOEINHJ4LCKLGDQ3V2LWOYATY4XDVE6RA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.316733016 (566h46m3.669518124s) [noise=true , address=77.90.31.12:4590 , piece_id=6YYVSTSRZ5CENFV6BRYOPSSAX22ME6BXCORJUND5YYY5OXQU2F3Q, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.316751130 (566h46m3.669536238s) [noise=true , address=145.79.227.19:4078 , piece_id=YSMSYFBS2QHK6ZLQXHOMQUN4IYLR4HQHW42WF4HMOKIKE6YZRJOA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317000808 (566h46m3.669785916s) [noise=true , address=77.90.31.19:4841 , piece_id=Z4RWZXXICIXKQUVXBDDFAIIMPPRUCKZUXGYB3D7IHEBTTZUW3FAQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317013212 (566h46m3.66979832s) [noise=true , address=45.91.116.4:4940 , piece_id=A6YHS7R72YYXHZEGOKNHSBSL6PAIYW4ZEPAZQ7BYDQCV5FS46M4Q, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317235951 (566h46m3.670021049s) [noise=true , address=145.79.227.13:4801 , piece_id=TTVJGJQX2UZVLYAKSVD6CF6VY2UL6OQR5GMLQUQOLL5DTVO3RG4A, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317449061 (566h46m3.670234158s) [noise=true , address=77.90.31.14:4662 , piece_id=3IPX6TBM3DJIAXKIRQWK5ZMAG4PJBQAEQCCLABA337F77ZLSHE5A, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317532486 (566h46m3.670317594s) [noise=true , address=45.91.116.11:4117 , piece_id=XTF6SNJ7EZGHJYU75YCCOY27XCOALI4ARN5U5IGP5LG5CJH67XGQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317654906 (566h46m3.670440004s) [noise=true , address=77.90.31.17:4771 , piece_id=STZJUQPM52ZAILBLDOS3ID2NMTX5SN7OMEBINAKBUN5HF67WYKXQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317780301 (566h46m3.670565409s) [noise=true , address=45.91.116.12:4227 , piece_id=XJIWHHR3MMOOZU7NA7KYKB3Y7LNULX265E3SN3V2W5DHL2HVCRHQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.317858769 (566h46m3.670643877s) [noise=true , address=45.91.116.12:4225 , piece_id=JD2BMT5ZPMXHBZJEM5IBQ5F722RZKHUB2DL3UWHQAYA247HCYK6A, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318024650 (566h46m3.670809749s) [noise=true , address=145.79.227.3:4149 , piece_id=D7QK4CHIM6JCJSD525OL76UOCWOHRWHHSSURCF5VRXYOE26ME7DQ, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318076999 (566h46m3.670862097s) [noise=true , address=145.79.227.16:4909 , piece_id=237F3T43VTPPCFP2L55YSFGW4WJ4ORCGUG5FHNYL6CCMR3I7H6DA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318270001 (566h46m3.671055109s) [noise=true , address=45.91.116.13:4155 , piece_id=HARQMYJGTUMGH43D3RIJ7JDTSWG3ZAFOQGFK7X7OBDOVRVLNWNSA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.318296210 (566h46m3.671081318s) [noise=true , address=77.90.31.13:4626 , piece_id=GRHQM2NK5KHAG5TVBG6MQXQYOJOCFMMJS4HESCTBOBRAOVWEBPGA, seg_pos={0 0}, upload=498810]: Uploading piece... Sep 22 04:54:28.395620925 (566h46m3.748406033s) [upload=498810]: Next returned nil segment Sep 22 04:54:28.395632406 (566h46m3.748417515s) [upload=498810]: Waiting for error group managing segments... Sep 22 04:54:28.404132007 (566h46m3.756917115s) [noise=true , address=45.91.116.12:4227 , piece_id=XJIWHHR3MMOOZU7NA7KYKB3Y7LNULX265E3SN3V2W5DHL2HVCRHQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404327825 (566h46m3.757112923s) [noise=true , address=45.91.116.12:4225 , piece_id=JD2BMT5ZPMXHBZJEM5IBQ5F722RZKHUB2DL3UWHQAYA247HCYK6A, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404472256 (566h46m3.757257364s) [noise=true , address=45.91.116.5:4297 , piece_id=WIAZ24OOOJGIS57A3VALOEINHJ4LCKLGDQ3V2LWOYATY4XDVE6RA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404908104 (566h46m3.757693201s) [noise=true , address=77.90.31.12:4590 , piece_id=6YYVSTSRZ5CENFV6BRYOPSSAX22ME6BXCORJUND5YYY5OXQU2F3Q, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.404995728 (566h46m3.757780836s) [noise=true , address=45.91.116.13:4155 , piece_id=HARQMYJGTUMGH43D3RIJ7JDTSWG3ZAFOQGFK7X7OBDOVRVLNWNSA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405266737 (566h46m3.758051845s) [noise=true , address=45.91.116.11:4117 , piece_id=XTF6SNJ7EZGHJYU75YCCOY27XCOALI4ARN5U5IGP5LG5CJH67XGQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405618116 (566h46m3.758403224s) [noise=true , address=77.90.31.19:4841 , piece_id=Z4RWZXXICIXKQUVXBDDFAIIMPPRUCKZUXGYB3D7IHEBTTZUW3FAQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405646719 (566h46m3.758431827s) [noise=true , address=45.91.116.4:4940 , piece_id=A6YHS7R72YYXHZEGOKNHSBSL6PAIYW4ZEPAZQ7BYDQCV5FS46M4Q, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.405977380 (566h46m3.758762488s) [noise=true , address=77.90.31.13:4626 , piece_id=GRHQM2NK5KHAG5TVBG6MQXQYOJOCFMMJS4HESCTBOBRAOVWEBPGA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.406215007 (566h46m3.759000115s) [noise=true , address=77.90.31.18:4805 , piece_id=SYYYGZJNPXL62CD2EDJ7HTDZVAGFV23BSNMXLL5U4VMVRECZI3LQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.408093832 (566h46m3.76087893s) [noise=true , address=77.90.31.14:4662 , piece_id=3IPX6TBM3DJIAXKIRQWK5ZMAG4PJBQAEQCCLABA337F77ZLSHE5A, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:28.409582205 (566h46m3.762367313s) [noise=true , address=77.90.31.17:4771 , piece_id=STZJUQPM52ZAILBLDOS3ID2NMTX5SN7OMEBINAKBUN5HF67WYKXQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: <nil> Sep 22 04:54:32.565058717 (566h46m7.917843815s) [noise=true , address=145.79.227.3:4149 , piece_id=D7QK4CHIM6JCJSD525OL76UOCWOHRWHHSSURCF5VRXYOE26ME7DQ, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: connect: no route to host; manager closed: conn closed: op Sep 22 04:54:58.264859545 (566h46m33.617644653s) [noise=true , address=145.79.227.19:4078 , piece_id=YSMSYFBS2QHK6ZLQXHOMQUN4IYLR4HQHW42WF4HMOKIKE6YZRJOA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:54:58.264933745 (566h46m33.617718853s) [noise=true , address=145.79.227.13:4801 , piece_id=TTVJGJQX2UZVLYAKSVD6CF6VY2UL6OQR5GMLQUQOLL5DTVO3RG4A, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:54:58.264952610 (566h46m33.617737718s) [noise=true , address=145.79.227.16:4909 , piece_id=237F3T43VTPPCFP2L55YSFGW4WJ4ORCGUG5FHNYL6CCMR3I7H6DA, seg_pos={0 0}, upload=498810]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:54:58.264980422 (566h46m33.61776553s) [seg_pos={0 0}, upload=498810]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:54:58.290835962 (566h46m33.64362106s) [upload=498810]: Done uploading segments... Sep 22 04:54:58.290853404 (566h46m33.643638512s) [upload=498810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.027057428s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X..X......X..................XXX.XXXXX.........X........X..] duration: 479.525637ms FAIL (2025-09-22 04:54:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.536698ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 30.004378772s FAIL (2025-09-22 04:54:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.004396194s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 32.271389007s FAIL (2025-09-22 04:54:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 32.271415288s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X..X......X..................XXX.XXXXX.........X........X.] duration: 599.947238ms FAIL (2025-09-22 04:54:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 599.968637ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 427.784991ms FAIL (2025-09-22 04:54:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:36:12.644832629 (566h27m47.997617726s) [upload=498538]: Starting upload Sep 22 04:36:12.644902188 (566h27m47.997687296s) [upload=498538]: Done starting upload Sep 22 04:36:12.644968543 (566h27m47.997753651s) [upload=498538]: Uploading segments... Sep 22 04:36:12.645120949 (566h27m47.997906047s) [upload=498538]: Got next segment. Inline: false Sep 22 04:36:12.645149963 (566h27m47.997935071s) [upload=498538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:36:12.645153330 (566h27m47.997938438s) [upload=498538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:36:12.671594820 (566h27m48.024379958s) [seg_pos={0 0}, upload=498538]: Begin upload segment... Sep 22 04:36:12.671637109 (566h27m48.024422217s) [seg_pos={0 0}, upload=498538]: Done begin upload segment. Sep 22 04:36:12.671683396 (566h27m48.024468494s) [noise=true , address=145.79.227.7:4294 , piece_id=GH2PNKFYARMA67TLPFQZS5CMCLP6ONK3TIH4T3UFKDJGTTUKXNDQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.671789284 (566h27m48.024574392s) [noise=true , address=145.79.227.15:4873 , piece_id=WCQ6GQRQDOXNLG7SMZVLKQ2RUQRR4GYSROVIUUYVQ3UGSP3ZS5QA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.672167214 (566h27m48.024952352s) [noise=true , address=77.90.31.15:4698 , piece_id=ZBIDGJBBGYB63KYMS6HK4H534T6VJJIYTONCTD3TXLCLHURBP3BQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.672317737 (566h27m48.025102835s) [noise=true , address=145.79.227.16:4910 , piece_id=IGZZ3OIBLBS4T3GLGDBOMJYLYEREU3OISWPZO22BXMLLQI2VN7DQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.672532309 (566h27m48.025317417s) [noise=true , address=145.79.227.3:4151 , piece_id=HEEJZ5WRA3XODAVULTWCYJ3MLI7DAD2DTOFYVKVSJ7AEEAIJPPPQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.672939864 (566h27m48.025724962s) [noise=true , address=77.90.31.18:4806 , piece_id=23A3S3W6IXULKCW6EILAMAII353EH4V7BIBPRAPZGGE4DHVKNYMQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.673170226 (566h27m48.025955334s) [noise=true , address=77.90.31.19:4841 , piece_id=RZKK73XV5E4VBZ73QIPV7UN2Q3HLWS55GOIVCTHCGNQS2FQEWLUA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.673501818 (566h27m48.026286926s) [noise=true , address=77.90.31.16:4735 , piece_id=LOZIHPZLJBLPG4VLFU443EH3M2IC5TE7RZ5BM7LHQL4GVXROI2BQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.673873606 (566h27m48.026658714s) [noise=true , address=77.90.31.14:4663 , piece_id=KXOH7EYNYSGFOT4WWO4LZ3EJPJSN3TV326EJ2N3O3GR4QQDQ3ZMQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.673883694 (566h27m48.026668802s) [noise=true , address=45.91.116.6:4974 , piece_id=VCGDW3XLL2EE3FEMJZDLCEJ54GLG73EQBPWZ62TKJ54SYY5MMDBQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674173469 (566h27m48.026958577s) [noise=true , address=45.91.116.9:4261 , piece_id=O2HJ5NUG2I7IRF6MVMFH4VWY47R3L7GF4EA7ZCYLATHQGMI7DYDA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674192495 (566h27m48.026977593s) [noise=true , address=45.91.116.8:4047 , piece_id=HIN5PY4OCEVZULCCZRA2IB52BSHM7SOZ6IMMAB6OGIZ4VMWKUQ2Q, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674502076 (566h27m48.027287184s) [noise=true , address=77.90.31.16:4734 , piece_id=ID2PAF5VIXHG2FX4XN3GFECPM43HCQ5VUQPFDNKHNR34UXGIIT5Q, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674523706 (566h27m48.027308814s) [noise=true , address=45.91.116.10:4083 , piece_id=7UHK5QBWHPR47YNPXPWB4P2NFNSZH2RXACWSXW26TNB2CZR2LAMA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674899711 (566h27m48.027684819s) [noise=true , address=77.90.31.17:4772 , piece_id=I6EOUCOVQABAQM2DCS7QGXOGRYAAYWMLWUDRXDP4EUME32GVDUJA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.675288862 (566h27m48.02807396s) [noise=true , address=45.91.116.4:4939 , piece_id=B2KT7EFM7BMVLHPQD35UWN352VYWNQQ7C66IZTXGN2VNFCQPWQXA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.754015098 (566h27m48.106800206s) [upload=498538]: Next returned nil segment Sep 22 04:36:12.754026128 (566h27m48.106811236s) [upload=498538]: Waiting for error group managing segments... Sep 22 04:36:12.762538753 (566h27m48.115323851s) [noise=true , address=45.91.116.4:4939 , piece_id=B2KT7EFM7BMVLHPQD35UWN352VYWNQQ7C66IZTXGN2VNFCQPWQXA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.763691737 (566h27m48.116476826s) [noise=true , address=45.91.116.10:4083 , piece_id=7UHK5QBWHPR47YNPXPWB4P2NFNSZH2RXACWSXW26TNB2CZR2LAMA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.763851938 (566h27m48.116637036s) [noise=true , address=77.90.31.17:4772 , piece_id=I6EOUCOVQABAQM2DCS7QGXOGRYAAYWMLWUDRXDP4EUME32GVDUJA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764119320 (566h27m48.116904407s) [noise=true , address=45.91.116.8:4047 , piece_id=HIN5PY4OCEVZULCCZRA2IB52BSHM7SOZ6IMMAB6OGIZ4VMWKUQ2Q, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764344512 (566h27m48.11712962s) [noise=true , address=45.91.116.6:4974 , piece_id=VCGDW3XLL2EE3FEMJZDLCEJ54GLG73EQBPWZ62TKJ54SYY5MMDBQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764450341 (566h27m48.117235428s) [noise=true , address=45.91.116.9:4261 , piece_id=O2HJ5NUG2I7IRF6MVMFH4VWY47R3L7GF4EA7ZCYLATHQGMI7DYDA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764690291 (566h27m48.117475399s) [noise=true , address=77.90.31.16:4735 , piece_id=LOZIHPZLJBLPG4VLFU443EH3M2IC5TE7RZ5BM7LHQL4GVXROI2BQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764770231 (566h27m48.117555319s) [noise=true , address=77.90.31.16:4734 , piece_id=ID2PAF5VIXHG2FX4XN3GFECPM43HCQ5VUQPFDNKHNR34UXGIIT5Q, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.765165502 (566h27m48.117950601s) [noise=true , address=77.90.31.14:4663 , piece_id=KXOH7EYNYSGFOT4WWO4LZ3EJPJSN3TV326EJ2N3O3GR4QQDQ3ZMQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.765188967 (566h27m48.117974075s) [noise=true , address=77.90.31.18:4806 , piece_id=23A3S3W6IXULKCW6EILAMAII353EH4V7BIBPRAPZGGE4DHVKNYMQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.765508888 (566h27m48.118293985s) [noise=true , address=77.90.31.15:4698 , piece_id=ZBIDGJBBGYB63KYMS6HK4H534T6VJJIYTONCTD3TXLCLHURBP3BQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.765532983 (566h27m48.118318081s) [noise=true , address=77.90.31.19:4841 , piece_id=RZKK73XV5E4VBZ73QIPV7UN2Q3HLWS55GOIVCTHCGNQS2FQEWLUA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:42.632309187 (566h28m17.985094306s) [noise=true , address=145.79.227.3:4151 , piece_id=HEEJZ5WRA3XODAVULTWCYJ3MLI7DAD2DTOFYVKVSJ7AEEAIJPPPQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:42.632392333 (566h28m17.985177462s) [noise=true , address=145.79.227.15:4873 , piece_id=WCQ6GQRQDOXNLG7SMZVLKQ2RUQRR4GYSROVIUUYVQ3UGSP3ZS5QA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:42.632448048 (566h28m17.985233176s) [noise=true , address=145.79.227.7:4294 , piece_id=GH2PNKFYARMA67TLPFQZS5CMCLP6ONK3TIH4T3UFKDJGTTUKXNDQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:42.632483284 (566h28m17.985268402s) [noise=true , address=145.79.227.16:4910 , piece_id=IGZZ3OIBLBS4T3GLGDBOMJYLYEREU3OISWPZO22BXMLLQI2VN7DQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:42.632529260 (566h28m17.985314378s) [seg_pos={0 0}, upload=498538]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:36:42.646011962 (566h28m17.99879709s) [upload=498538]: Done uploading segments... Sep 22 04:36:42.646038512 (566h28m17.99882365s) [upload=498538]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.811011ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X..X......X..................XXX.XXXXX.........X........X] duration: 30.000483414s FAIL (2025-09-22 04:53:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000496478s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...................................X.......................] duration: 442.257869ms FAIL (2025-09-22 04:53:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:36:12.644832629 (566h27m47.997617726s) [upload=498538]: Starting upload Sep 22 04:36:12.644902188 (566h27m47.997687296s) [upload=498538]: Done starting upload Sep 22 04:36:12.644968543 (566h27m47.997753651s) [upload=498538]: Uploading segments... Sep 22 04:36:12.645120949 (566h27m47.997906047s) [upload=498538]: Got next segment. Inline: false Sep 22 04:36:12.645149963 (566h27m47.997935071s) [upload=498538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:36:12.645153330 (566h27m47.997938438s) [upload=498538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:36:12.671594820 (566h27m48.024379958s) [seg_pos={0 0}, upload=498538]: Begin upload segment... Sep 22 04:36:12.671637109 (566h27m48.024422217s) [seg_pos={0 0}, upload=498538]: Done begin upload segment. Sep 22 04:36:12.671683396 (566h27m48.024468494s) [noise=true , address=145.79.227.7:4294 , piece_id=GH2PNKFYARMA67TLPFQZS5CMCLP6ONK3TIH4T3UFKDJGTTUKXNDQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.671789284 (566h27m48.024574392s) [noise=true , address=145.79.227.15:4873 , piece_id=WCQ6GQRQDOXNLG7SMZVLKQ2RUQRR4GYSROVIUUYVQ3UGSP3ZS5QA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.672167214 (566h27m48.024952352s) [noise=true , address=77.90.31.15:4698 , piece_id=ZBIDGJBBGYB63KYMS6HK4H534T6VJJIYTONCTD3TXLCLHURBP3BQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.672317737 (566h27m48.025102835s) [noise=true , address=145.79.227.16:4910 , piece_id=IGZZ3OIBLBS4T3GLGDBOMJYLYEREU3OISWPZO22BXMLLQI2VN7DQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.672532309 (566h27m48.025317417s) [noise=true , address=145.79.227.3:4151 , piece_id=HEEJZ5WRA3XODAVULTWCYJ3MLI7DAD2DTOFYVKVSJ7AEEAIJPPPQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.672939864 (566h27m48.025724962s) [noise=true , address=77.90.31.18:4806 , piece_id=23A3S3W6IXULKCW6EILAMAII353EH4V7BIBPRAPZGGE4DHVKNYMQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.673170226 (566h27m48.025955334s) [noise=true , address=77.90.31.19:4841 , piece_id=RZKK73XV5E4VBZ73QIPV7UN2Q3HLWS55GOIVCTHCGNQS2FQEWLUA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.673501818 (566h27m48.026286926s) [noise=true , address=77.90.31.16:4735 , piece_id=LOZIHPZLJBLPG4VLFU443EH3M2IC5TE7RZ5BM7LHQL4GVXROI2BQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.673873606 (566h27m48.026658714s) [noise=true , address=77.90.31.14:4663 , piece_id=KXOH7EYNYSGFOT4WWO4LZ3EJPJSN3TV326EJ2N3O3GR4QQDQ3ZMQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.673883694 (566h27m48.026668802s) [noise=true , address=45.91.116.6:4974 , piece_id=VCGDW3XLL2EE3FEMJZDLCEJ54GLG73EQBPWZ62TKJ54SYY5MMDBQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674173469 (566h27m48.026958577s) [noise=true , address=45.91.116.9:4261 , piece_id=O2HJ5NUG2I7IRF6MVMFH4VWY47R3L7GF4EA7ZCYLATHQGMI7DYDA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674192495 (566h27m48.026977593s) [noise=true , address=45.91.116.8:4047 , piece_id=HIN5PY4OCEVZULCCZRA2IB52BSHM7SOZ6IMMAB6OGIZ4VMWKUQ2Q, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674502076 (566h27m48.027287184s) [noise=true , address=77.90.31.16:4734 , piece_id=ID2PAF5VIXHG2FX4XN3GFECPM43HCQ5VUQPFDNKHNR34UXGIIT5Q, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674523706 (566h27m48.027308814s) [noise=true , address=45.91.116.10:4083 , piece_id=7UHK5QBWHPR47YNPXPWB4P2NFNSZH2RXACWSXW26TNB2CZR2LAMA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674899711 (566h27m48.027684819s) [noise=true , address=77.90.31.17:4772 , piece_id=I6EOUCOVQABAQM2DCS7QGXOGRYAAYWMLWUDRXDP4EUME32GVDUJA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.675288862 (566h27m48.02807396s) [noise=true , address=45.91.116.4:4939 , piece_id=B2KT7EFM7BMVLHPQD35UWN352VYWNQQ7C66IZTXGN2VNFCQPWQXA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.754015098 (566h27m48.106800206s) [upload=498538]: Next returned nil segment Sep 22 04:36:12.754026128 (566h27m48.106811236s) [upload=498538]: Waiting for error group managing segments... Sep 22 04:36:12.762538753 (566h27m48.115323851s) [noise=true , address=45.91.116.4:4939 , piece_id=B2KT7EFM7BMVLHPQD35UWN352VYWNQQ7C66IZTXGN2VNFCQPWQXA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.763691737 (566h27m48.116476826s) [noise=true , address=45.91.116.10:4083 , piece_id=7UHK5QBWHPR47YNPXPWB4P2NFNSZH2RXACWSXW26TNB2CZR2LAMA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.763851938 (566h27m48.116637036s) [noise=true , address=77.90.31.17:4772 , piece_id=I6EOUCOVQABAQM2DCS7QGXOGRYAAYWMLWUDRXDP4EUME32GVDUJA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764119320 (566h27m48.116904407s) [noise=true , address=45.91.116.8:4047 , piece_id=HIN5PY4OCEVZULCCZRA2IB52BSHM7SOZ6IMMAB6OGIZ4VMWKUQ2Q, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764344512 (566h27m48.11712962s) [noise=true , address=45.91.116.6:4974 , piece_id=VCGDW3XLL2EE3FEMJZDLCEJ54GLG73EQBPWZ62TKJ54SYY5MMDBQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764450341 (566h27m48.117235428s) [noise=true , address=45.91.116.9:4261 , piece_id=O2HJ5NUG2I7IRF6MVMFH4VWY47R3L7GF4EA7ZCYLATHQGMI7DYDA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764690291 (566h27m48.117475399s) [noise=true , address=77.90.31.16:4735 , piece_id=LOZIHPZLJBLPG4VLFU443EH3M2IC5TE7RZ5BM7LHQL4GVXROI2BQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764770231 (566h27m48.117555319s) [noise=true , address=77.90.31.16:4734 , piece_id=ID2PAF5VIXHG2FX4XN3GFECPM43HCQ5VUQPFDNKHNR34UXGIIT5Q, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.765165502 (566h27m48.117950601s) [noise=true , address=77.90.31.14:4663 , piece_id=KXOH7EYNYSGFOT4WWO4LZ3EJPJSN3TV326EJ2N3O3GR4QQDQ3ZMQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.765188967 (566h27m48.117974075s) [noise=true , address=77.90.31.18:4806 , piece_id=23A3S3W6IXULKCW6EILAMAII353EH4V7BIBPRAPZGGE4DHVKNYMQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.765508888 (566h27m48.118293985s) [noise=true , address=77.90.31.15:4698 , piece_id=ZBIDGJBBGYB63KYMS6HK4H534T6VJJIYTONCTD3TXLCLHURBP3BQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.765532983 (566h27m48.118318081s) [noise=true , address=77.90.31.19:4841 , piece_id=RZKK73XV5E4VBZ73QIPV7UN2Q3HLWS55GOIVCTHCGNQS2FQEWLUA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:42.632309187 (566h28m17.985094306s) [noise=true , address=145.79.227.3:4151 , piece_id=HEEJZ5WRA3XODAVULTWCYJ3MLI7DAD2DTOFYVKVSJ7AEEAIJPPPQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:42.632392333 (566h28m17.985177462s) [noise=true , address=145.79.227.15:4873 , piece_id=WCQ6GQRQDOXNLG7SMZVLKQ2RUQRR4GYSROVIUUYVQ3UGSP3ZS5QA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:42.632448048 (566h28m17.985233176s) [noise=true , address=145.79.227.7:4294 , piece_id=GH2PNKFYARMA67TLPFQZS5CMCLP6ONK3TIH4T3UFKDJGTTUKXNDQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:42.632483284 (566h28m17.985268402s) [noise=true , address=145.79.227.16:4910 , piece_id=IGZZ3OIBLBS4T3GLGDBOMJYLYEREU3OISWPZO22BXMLLQI2VN7DQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:42.632529260 (566h28m17.985314378s) [seg_pos={0 0}, upload=498538]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:36:42.646011962 (566h28m17.99879709s) [upload=498538]: Done uploading segments... Sep 22 04:36:42.646038512 (566h28m17.99882365s) [upload=498538]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.283128ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX...................................X......................] duration: 703.80336ms FAIL (2025-09-22 04:52:59 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 04:36:12.644832629 (566h27m47.997617726s) [upload=498538]: Starting upload Sep 22 04:36:12.644902188 (566h27m47.997687296s) [upload=498538]: Done starting upload Sep 22 04:36:12.644968543 (566h27m47.997753651s) [upload=498538]: Uploading segments... Sep 22 04:36:12.645120949 (566h27m47.997906047s) [upload=498538]: Got next segment. Inline: false Sep 22 04:36:12.645149963 (566h27m47.997935071s) [upload=498538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:36:12.645153330 (566h27m47.997938438s) [upload=498538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:36:12.671594820 (566h27m48.024379958s) [seg_pos={0 0}, upload=498538]: Begin upload segment... Sep 22 04:36:12.671637109 (566h27m48.024422217s) [seg_pos={0 0}, upload=498538]: Done begin upload segment. Sep 22 04:36:12.671683396 (566h27m48.024468494s) [noise=true , address=145.79.227.7:4294 , piece_id=GH2PNKFYARMA67TLPFQZS5CMCLP6ONK3TIH4T3UFKDJGTTUKXNDQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.671789284 (566h27m48.024574392s) [noise=true , address=145.79.227.15:4873 , piece_id=WCQ6GQRQDOXNLG7SMZVLKQ2RUQRR4GYSROVIUUYVQ3UGSP3ZS5QA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.672167214 (566h27m48.024952352s) [noise=true , address=77.90.31.15:4698 , piece_id=ZBIDGJBBGYB63KYMS6HK4H534T6VJJIYTONCTD3TXLCLHURBP3BQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.672317737 (566h27m48.025102835s) [noise=true , address=145.79.227.16:4910 , piece_id=IGZZ3OIBLBS4T3GLGDBOMJYLYEREU3OISWPZO22BXMLLQI2VN7DQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.672532309 (566h27m48.025317417s) [noise=true , address=145.79.227.3:4151 , piece_id=HEEJZ5WRA3XODAVULTWCYJ3MLI7DAD2DTOFYVKVSJ7AEEAIJPPPQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.672939864 (566h27m48.025724962s) [noise=true , address=77.90.31.18:4806 , piece_id=23A3S3W6IXULKCW6EILAMAII353EH4V7BIBPRAPZGGE4DHVKNYMQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.673170226 (566h27m48.025955334s) [noise=true , address=77.90.31.19:4841 , piece_id=RZKK73XV5E4VBZ73QIPV7UN2Q3HLWS55GOIVCTHCGNQS2FQEWLUA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.673501818 (566h27m48.026286926s) [noise=true , address=77.90.31.16:4735 , piece_id=LOZIHPZLJBLPG4VLFU443EH3M2IC5TE7RZ5BM7LHQL4GVXROI2BQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.673873606 (566h27m48.026658714s) [noise=true , address=77.90.31.14:4663 , piece_id=KXOH7EYNYSGFOT4WWO4LZ3EJPJSN3TV326EJ2N3O3GR4QQDQ3ZMQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.673883694 (566h27m48.026668802s) [noise=true , address=45.91.116.6:4974 , piece_id=VCGDW3XLL2EE3FEMJZDLCEJ54GLG73EQBPWZ62TKJ54SYY5MMDBQ, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674173469 (566h27m48.026958577s) [noise=true , address=45.91.116.9:4261 , piece_id=O2HJ5NUG2I7IRF6MVMFH4VWY47R3L7GF4EA7ZCYLATHQGMI7DYDA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674192495 (566h27m48.026977593s) [noise=true , address=45.91.116.8:4047 , piece_id=HIN5PY4OCEVZULCCZRA2IB52BSHM7SOZ6IMMAB6OGIZ4VMWKUQ2Q, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674502076 (566h27m48.027287184s) [noise=true , address=77.90.31.16:4734 , piece_id=ID2PAF5VIXHG2FX4XN3GFECPM43HCQ5VUQPFDNKHNR34UXGIIT5Q, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674523706 (566h27m48.027308814s) [noise=true , address=45.91.116.10:4083 , piece_id=7UHK5QBWHPR47YNPXPWB4P2NFNSZH2RXACWSXW26TNB2CZR2LAMA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.674899711 (566h27m48.027684819s) [noise=true , address=77.90.31.17:4772 , piece_id=I6EOUCOVQABAQM2DCS7QGXOGRYAAYWMLWUDRXDP4EUME32GVDUJA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.675288862 (566h27m48.02807396s) [noise=true , address=45.91.116.4:4939 , piece_id=B2KT7EFM7BMVLHPQD35UWN352VYWNQQ7C66IZTXGN2VNFCQPWQXA, seg_pos={0 0}, upload=498538]: Uploading piece... Sep 22 04:36:12.754015098 (566h27m48.106800206s) [upload=498538]: Next returned nil segment Sep 22 04:36:12.754026128 (566h27m48.106811236s) [upload=498538]: Waiting for error group managing segments... Sep 22 04:36:12.762538753 (566h27m48.115323851s) [noise=true , address=45.91.116.4:4939 , piece_id=B2KT7EFM7BMVLHPQD35UWN352VYWNQQ7C66IZTXGN2VNFCQPWQXA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.763691737 (566h27m48.116476826s) [noise=true , address=45.91.116.10:4083 , piece_id=7UHK5QBWHPR47YNPXPWB4P2NFNSZH2RXACWSXW26TNB2CZR2LAMA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.763851938 (566h27m48.116637036s) [noise=true , address=77.90.31.17:4772 , piece_id=I6EOUCOVQABAQM2DCS7QGXOGRYAAYWMLWUDRXDP4EUME32GVDUJA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764119320 (566h27m48.116904407s) [noise=true , address=45.91.116.8:4047 , piece_id=HIN5PY4OCEVZULCCZRA2IB52BSHM7SOZ6IMMAB6OGIZ4VMWKUQ2Q, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764344512 (566h27m48.11712962s) [noise=true , address=45.91.116.6:4974 , piece_id=VCGDW3XLL2EE3FEMJZDLCEJ54GLG73EQBPWZ62TKJ54SYY5MMDBQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764450341 (566h27m48.117235428s) [noise=true , address=45.91.116.9:4261 , piece_id=O2HJ5NUG2I7IRF6MVMFH4VWY47R3L7GF4EA7ZCYLATHQGMI7DYDA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764690291 (566h27m48.117475399s) [noise=true , address=77.90.31.16:4735 , piece_id=LOZIHPZLJBLPG4VLFU443EH3M2IC5TE7RZ5BM7LHQL4GVXROI2BQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.764770231 (566h27m48.117555319s) [noise=true , address=77.90.31.16:4734 , piece_id=ID2PAF5VIXHG2FX4XN3GFECPM43HCQ5VUQPFDNKHNR34UXGIIT5Q, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.765165502 (566h27m48.117950601s) [noise=true , address=77.90.31.14:4663 , piece_id=KXOH7EYNYSGFOT4WWO4LZ3EJPJSN3TV326EJ2N3O3GR4QQDQ3ZMQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.765188967 (566h27m48.117974075s) [noise=true , address=77.90.31.18:4806 , piece_id=23A3S3W6IXULKCW6EILAMAII353EH4V7BIBPRAPZGGE4DHVKNYMQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.765508888 (566h27m48.118293985s) [noise=true , address=77.90.31.15:4698 , piece_id=ZBIDGJBBGYB63KYMS6HK4H534T6VJJIYTONCTD3TXLCLHURBP3BQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:12.765532983 (566h27m48.118318081s) [noise=true , address=77.90.31.19:4841 , piece_id=RZKK73XV5E4VBZ73QIPV7UN2Q3HLWS55GOIVCTHCGNQS2FQEWLUA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: <nil> Sep 22 04:36:42.632309187 (566h28m17.985094306s) [noise=true , address=145.79.227.3:4151 , piece_id=HEEJZ5WRA3XODAVULTWCYJ3MLI7DAD2DTOFYVKVSJ7AEEAIJPPPQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:42.632392333 (566h28m17.985177462s) [noise=true , address=145.79.227.15:4873 , piece_id=WCQ6GQRQDOXNLG7SMZVLKQ2RUQRR4GYSROVIUUYVQ3UGSP3ZS5QA, seg_pos={0 0}, upload=498538]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:42.632448048 (566h28m17.985233176s) [noise=true , address=145.79.227.7:4294 , piece_id=GH2PNKFYARMA67TLPFQZS5CMCLP6ONK3TIH4T3UFKDJGTTUKXNDQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:42.632483284 (566h28m17.985268402s) [noise=true , address=145.79.227.16:4910 , piece_id=IGZZ3OIBLBS4T3GLGDBOMJYLYEREU3OISWPZO22BXMLLQI2VN7DQ, seg_pos={0 0}, upload=498538]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:36:42.632529260 (566h28m17.985314378s) [seg_pos={0 0}, upload=498538]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 04:36:42.646011962 (566h28m17.99879709s) [upload=498538]: Done uploading segments... Sep 22 04:36:42.646038512 (566h28m17.99882365s) [upload=498538]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 703.831113ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)