Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....XX..................X.X...X..X......X..................] duration: 455.021456ms FAIL (2025-09-22 04:33:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 455.042495ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X...........X........X..XXX.................................] duration: 836.083709ms FAIL (2025-09-22 04:33:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 836.114566ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......XX..................X.X...X..X......X.................] duration: 445.990086ms FAIL (2025-09-22 04:33:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.042223ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X...........X........X..XXX................................] duration: 503.960793ms FAIL (2025-09-22 04:32:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 503.985099ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......XX..................X.X...X..X......X................] duration: 425.101828ms FAIL (2025-09-22 04:32:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 425.125713ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X...........X........X..XXX...............................] duration: 453.216297ms FAIL (2025-09-22 04:32:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 453.23964ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........XX..................X.X...X..X......X...............] duration: 851.674437ms FAIL (2025-09-22 04:31:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 851.6979ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X...........X........X..XXX..............................] duration: 569.157509ms FAIL (2025-09-22 04:31:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 569.179882ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X...........X........X..XXX.............................] duration: 774.07235ms FAIL (2025-09-22 04:30:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 774.10509ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........XX..................X.X...X..X......X..............] duration: 683.261235ms FAIL (2025-09-22 04:30:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 683.282575ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X...........X........X..XXX............................] duration: 464.434839ms FAIL (2025-09-22 04:30:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 464.460036ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.........XX..................X.X...X..X......X.............] duration: 412.031861ms FAIL (2025-09-22 04:30:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 412.055124ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X...........X........X..XXX...........................] duration: 745.643961ms FAIL (2025-09-22 04:29:34 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 745.664861ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.........XX..................X.X...X..X......X............] duration: 409.749569ms FAIL (2025-09-22 04:29:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 409.769837ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X...........X........X..XXX..........................] duration: 568.404467ms FAIL (2025-09-22 04:28:58 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 568.481381ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X...........X........X..XXX.........................] duration: 769.294643ms FAIL (2025-09-22 04:28:34 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 769.313269ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.........XX..................X.X...X..X......X...........] duration: 434.974175ms FAIL (2025-09-22 04:28:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 434.989744ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X........X...........X........X..XXX........................] duration: 537.746055ms FAIL (2025-09-22 04:27:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 537.762374ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X.........XX..................X.X...X..X......X..........] duration: 473.604276ms FAIL (2025-09-22 04:27:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 473.633281ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X........X...........X........X..XXX.......................] duration: 433.766157ms FAIL (2025-09-22 04:27:08 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 433.793308ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X...X.........XX..................X.X...X..X......X.........] duration: 1.105561696s FAIL (2025-09-22 04:26:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.105584289s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X........X...........X........X..XXX......................] duration: 430.055775ms FAIL (2025-09-22 04:26:21 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 430.079259ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X...X.........XX..................X.X...X..X......X........] duration: 509.691017ms FAIL (2025-09-22 04:25:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 509.709321ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X........X...........X........X..XXX.....................] duration: 712.335204ms FAIL (2025-09-22 04:25:40 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 712.356093ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X...X.........XX..................X.X...X..X......X.......] duration: 450.852842ms FAIL (2025-09-22 04:25:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 450.879152ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X........X...........X........X..XXX....................] duration: 490.91435ms FAIL (2025-09-22 04:25:23 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 490.942653ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X........X...........X........X..XXX...................] duration: 508.101934ms FAIL (2025-09-22 04:24:44 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 508.119137ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X...X.........XX..................X.X...X..X......X......] duration: 661.268294ms FAIL (2025-09-22 04:24:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 661.290155ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.....X........X...........X........X..XXX..................] duration: 457.144169ms FAIL (2025-09-22 04:24:02 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 457.168875ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X...X.........XX..................X.X...X..X......X.....] duration: 631.590801ms FAIL (2025-09-22 04:23:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 631.61075ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX.....X........X...........X........X..XXX.................] duration: 521.559119ms FAIL (2025-09-22 04:23:22 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 521.600336ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X...X.........XX..................X.X...X..X......X....] duration: 440.267728ms FAIL (2025-09-22 04:23:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 440.289037ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX.....X........X...........X........X..XXX................] duration: 545.100545ms FAIL (2025-09-22 04:22:46 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 545.121555ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X...X.........XX..................X.X...X..X......X...] duration: 467.349078ms FAIL (2025-09-22 04:22:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 467.365088ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X...X.........XX..................X.X...X..X......X..] duration: 827.327478ms FAIL (2025-09-22 04:21:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 827.531901ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XX.....X........X...........X........X..XXX...............] duration: 386.070944ms FAIL (2025-09-22 04:21:57 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 386.091411ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...XX.....X........X...........X........X..XXX..............] duration: 437.125462ms FAIL (2025-09-22 04:21:20 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 437.145119ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....XX.....X........X...........X........X..XXX.............] duration: 385.089874ms FAIL (2025-09-22 04:21:01 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 385.112656ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.......X...X.........XX..................X.X...X..X......X.] duration: 434.03382ms FAIL (2025-09-22 04:21:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 434.051843ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.......X...X.........XX..................X.X...X..X......X] duration: 30.00063628s FAIL (2025-09-22 04:20:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000660616s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....XX.....X........X...........X........X..XXX............] duration: 415.443534ms FAIL (2025-09-22 04:19:48 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 415.466647ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.......X...X.........XX..................X.X...X..X......] duration: 872.712707ms FAIL (2025-09-22 04:19:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 872.732224ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.......X...X.........XX..................X.X...X..X.....] duration: 470.180141ms FAIL (2025-09-22 04:18:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 470.198545ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.....XX.....X........X...........X........X..XXX...........] duration: 588.739698ms FAIL (2025-09-22 04:18:38 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 588.792807ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.....XX.....X........X...........X........X..XXX..........] duration: 396.942494ms FAIL (2025-09-22 04:18:18 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 396.970336ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X...X.......X...X.........XX..................X.X...X..X....] duration: 447.619223ms FAIL (2025-09-22 04:17:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 447.636366ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX...X.......X...X.........XX..................X.X...X..X...] duration: 752.062778ms FAIL (2025-09-22 04:17:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 752.083407ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.X.....XX.....X........X...........X........X..XXX.........] duration: 438.318289ms FAIL (2025-09-22 04:17:11 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 22 04:10:24.100440694 (566h1m59.453225822s) [upload=498156]: Starting upload Sep 22 04:10:24.100520314 (566h1m59.453305442s) [upload=498156]: Done starting upload Sep 22 04:10:24.100735217 (566h1m59.453520345s) [upload=498156]: Uploading segments... Sep 22 04:10:24.101026633 (566h1m59.453811761s) [upload=498156]: Got next segment. Inline: false Sep 22 04:10:24.101075957 (566h1m59.453861095s) [upload=498156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 04:10:24.101081457 (566h1m59.453866585s) [upload=498156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 04:10:24.141321941 (566h1m59.49410704s) [seg_pos={0 0}, upload=498156]: Begin upload segment... Sep 22 04:10:24.141357568 (566h1m59.494142666s) [seg_pos={0 0}, upload=498156]: Done begin upload segment. Sep 22 04:10:24.141385170 (566h1m59.494170278s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141445864 (566h1m59.494231002s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141678511 (566h1m59.494463608s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141927137 (566h1m59.494712245s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.141990595 (566h1m59.494775724s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142153943 (566h1m59.494939051s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142381640 (566h1m59.495166768s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142409462 (566h1m59.49519456s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142691211 (566h1m59.495476308s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142793723 (566h1m59.495578851s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.142933555 (566h1m59.495718663s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143158097 (566h1m59.495943205s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143165702 (566h1m59.495950829s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143383330 (566h1m59.496168438s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143536176 (566h1m59.496321315s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.143973899 (566h1m59.496759026s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Uploading piece... Sep 22 04:10:24.239968480 (566h1m59.592753588s) [upload=498156]: Next returned nil segment Sep 22 04:10:24.239980803 (566h1m59.592765911s) [upload=498156]: Waiting for error group managing segments... Sep 22 04:10:24.248891123 (566h1m59.601676222s) [noise=true , address=45.91.116.6:4975 , piece_id=76USR62GWBSXUU2MS342YKEKS4J3OMRCRMFZMKHJ6MPC3IL23NDA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249484668 (566h1m59.602269806s) [noise=true , address=77.90.31.16:4733 , piece_id=HAZAR4TOACTT3ASTIHB5246ENO5HYLS2GNFHNV7XH65JGGMIYUQQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.249704590 (566h1m59.602489698s) [noise=true , address=45.91.116.14:4189 , piece_id=I37DGFASGWXDN4MJB5A62HJET7EUHZ732ZXFEI6RDRTLUU4M4MOQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250108327 (566h1m59.602893456s) [noise=true , address=45.91.116.14:4190 , piece_id=KC567SOI5VQXAHLS6Y4SHGU5VXEK2U4MSUPTUNW6I2OLHKWOJWRQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250133304 (566h1m59.602918402s) [noise=true , address=77.90.31.14:4662 , piece_id=CICQLONUXLNZHKXQDF4K57YFRG3OYLVYJCA2RSGMRMPE2N7OEEAQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250596404 (566h1m59.603381502s) [noise=true , address=45.91.116.9:4262 , piece_id=O2ZZYVZF2DZFNQZ37S2RIBNMQEXWYINCT4QVIMNRWRWMOQRWZWLQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250743099 (566h1m59.603528227s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMQGD6R76PGVJZ6IZCHQCKU3NSPG7LFTRHPXBNUPDTCES2GGD6Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.250985423 (566h1m59.603770521s) [noise=true , address=77.90.31.21:4877 , piece_id=FB2TADVIS6CEVVAAPTCZCUCHIKRJUASN53J6BBTORBEMJYHJTGYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.251295306 (566h1m59.604080433s) [noise=true , address=45.91.116.13:4155 , piece_id=X64HHNW74DGDPDYB5MDT57D5SMMDVSXD6JGGJ6BAK4OPHCQMZZMA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:24.252309960 (566h1m59.605095057s) [noise=true , address=77.90.31.15:4697 , piece_id=KARHFNGBGATUZD5UROSKUSPJDHLBAVJYPZLETQVZYH62B5RZQGWA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: <nil> Sep 22 04:10:25.114124764 (566h2m0.466909893s) [noise=true , address=145.79.227.5:4223 , piece_id=3QABJHOGWIA4YA6SEWNGWTCF3SMPMYSIS2JWRQY2FNZ5QA7OSL7Q, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 04:10:25.338224247 (566h2m0.691009384s) [noise=true , address=145.79.227.18:4005 , piece_id=TPPQ4QW4UO5DIVPZFKWHHUI23YFVFOVTOQ7EVYDVJ7NTOYDPKPCQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: connect: no route to host; conn closed: op Sep 22 04:10:26.389719283 (566h2m1.742504391s) [noise=true , address=145.79.227.11:4730 , piece_id=JIFGUBU5GSE2ADQVP77HV4QDQKAPZN7ER2Y2ULD4PPYCIRXQ7UYQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088552400 (566h2m29.441337538s) [noise=true , address=145.79.227.7:4293 , piece_id=QGQBZIXR5U6XGPWFZOBKD24P4GRYDOYSMCCBCOVHG7DERFINQ6AQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 04:10:54.088698834 (566h2m29.441483963s) [noise=true , address=145.79.227.6:4259 , piece_id=5LHCOTXW3VW2UAEX2WL5TNFF6FN6NMY5PLSAS3ITWDPLHPRDOXGA, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088742276 (566h2m29.441527404s) [noise=true , address=145.79.227.15:4873 , piece_id=S3KVWWHCCEOECHSU3L3NZ54FOATSPC55WC6V3RXJDK73YGEDQOHQ, seg_pos={0 0}, upload=498156]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: context deadline exceeded; context deadline exceeded Sep 22 04:10:54.088842734 (566h2m29.441627862s) [seg_pos={0 0}, upload=498156]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 22 04:10:54.101418053 (566h2m29.45420318s) [upload=498156]: Done uploading segments... Sep 22 04:10:54.101435134 (566h2m29.454220273s) [upload=498156]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 438.336594ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXX...X.......X...X.........XX..................X.X...X..X..] duration: 408.726038ms FAIL (2025-09-22 04:16:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 408.741377ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XXX...X.......X...X.........XX..................X.X...X..X.] duration: 423.256144ms FAIL (2025-09-22 04:16:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 423.273497ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)